The DeHavilland Blog

Tuesday, May 01, 2007

Two views on NCLB

I’ve received a couple of thoughtful and detailed comments recently, and wanted to highlight them and respond. The first comes from an education professor at Cleveland State University in response to my post, The NCLB ‘Resistance Movement’ – I’ll share Dr. Wheatley’s comments below and intersperse my thoughts.

I understand why Ohanian sounds crazy to you, so why DOES she get such applause? And why are so many leading educational researchers and even statisticians clapping along, and writing their own books to say NCLB is a disaster?

Right off the bat, we get to the core difference between me and Dr. Wheatley: we simply see public education differently. Dr. Wheatley (as he’ll affirm later in this note) gives primacy to the people inside the system – NCLB is a disaster because educators (who are Ohanian’s audience) and others within the industry say it’s a disaster. In this world view, the public education system is an end in itself, and it is the people inside the system who should set the objectives and methods of learning. The people outside the system should have no voice in determining why we educate, how we educate, what we teach, or how we measure progress and success.

For me, public education is a means to an end – external stakeholders (citizens, business, etc.) pay for the public education system and therefore have the right and responsibility to state what we want it to accomplish. The people inside the system are paid to fulfill the objectives set forth by the citizen taxpayers who fund the system.

NCLB came about because the people who fund public education were not satisfied with the system’s outcomes and wanted some way to start gathering independent information on what’s going on in our schools. And while there are any number of flaws in this first effort at accountability (and yes, everyone thinks it can be improved), it has succeeded in providing new information to stakeholders, and more importantly, a focus on outcomes to everyone involved in education.

Quite a difference in perspectives, I’d say – and one that sets the stage for marked disagreements over every aspect of education.

Because there has never been an education law that was so brilliantly marketed (complete with bullying into silence people who suggested that poverty matters!), and yet was designed in so many ways to do the opposite of what is effective in the long run.

As I understand it, NCLB was created because poverty does matter, and children in poverty were being swept under the rug. Information on their performance was hidden within averages (in the case of test scores) or with deceptive figures (dropout rates). We can’t address a challenge like ensuring every child in poverty has a shot at a good education unless we can identify the problem and establish a way to gauge progress.

As an educational researcher and teacher educator, I teach teachers, future teachers and doctoral students about curriculum, lesson plans, content standards, child development, and motivation. NCLB undermines every one of those areas.

Most researchers assume that NCLB is really designed to privatize education--why else set goals that are flatly impossible to achieve (This isn't Lake Wobegone--all children will never be on grade level), then send the money to (poorly supervised) private vendors when schools inevitably fail.

I’ve talked to a lot of businesspeople about NCLB, and yes, there are some who want privatization. However, most people fail to grasp why that is. They’re not interested in privatization from a profit perspective – they’re interested in it because they see how a competitive marketplace (which privatization would create) forces improvement and innovation in an industry, and they see that as the best path to improving our education system. If control over all schools went from the government to a single company, the monopoly would remain, and there would be no improvement.

I would also say that NCLB and school choice (which is where the privatization concept falls in) are simply two different approaches to increasing school quality. If you had a free market for schools, you wouldn’t have NCLB – competition would ensure quality. And if you had strong performance on independent accountability (a la NCLB), there would be little interest in school choice.

Next, according to our professional standards (APA, AERA, etc) , high-stakes testing is educational malpractice. Even the testmakers say tests are not designed to be used this way.

I’d like to hear more on this. Which testmakers? What do they say?

Also, most of what matters is simply not on the tests, and will never be something you can bubble in. Tests have their place, but they're probably good for about 30-40% of what matters in education, and their validity drops when you attach high-stakes to them.

This is a common argument in this debate: you can’t test what we’re teaching. It’s the ultimate unassailable position, but I believe it to be patently false. Knowledge can be tested. Skills can be tested. What can we not evaluate? What is the 60-70% of what matters in education that we cannot assess? (And I’ll of course follow up on any response with: matters to whom?)

Also, let’s look at the concept of “high stakes” testing – it sounds scary, but deserves to be broken down.

To be meaningful, any system of assessment must have consequences. If a student cannot demonstrate that he or she has learned some material, there should be consequences: the student should be re-taught, whether that means redoing a unit, being held back a grade, or not graduating. This is not punishment: if we say that students have to know X, Y, and Z in order to succeed in life, we’re doing them a disservice by allowing them to go forward without that knowledge or those skills. We’re setting them up for failure. And if there are no consequences – if we’re just going to advance those students anyway – what’s the point of assessing them? Why measure if you don’t use the measurements for something?

And in terms of validity - how can the validity of a test drop when we attach consequences? Either a test is a valid measure of knowledge and skills, or it is not. It certainly becomes more important due to the addition of consequences – and it may cause anxiety – but it does not affect validity. (And note that in the case of NCLB, the consequences are the school’s, not the students, so the anxiety should reside there, and not with the students – despite what is so often reported.)

Education is about learning, and unlike business, it's not much about performance, and not at all about products or profits. Whatever you want to use in business, carrots, sticks and competition have a dreadful track record in education. Sure, you can get short term obedience, but in the long run, they work about as well as a fad diet.

Yes, it sounds reasonable to say teachers should teach to tests, but it's more complicated than that. If we're talking about knowledge, teaching to tests is a very bad idea--tests are designed to sample 15-20 of what we teach. To teach to the test often becomes teaching mostly the 20% that will be tested, and not teaching the other 80% that is equally important, and without which that 20% is useless trivia.

It also sounds reasonable to say that tests can't be preventing teachers from teaching high-level content if the scores reveal kids haven't mastered the lower-level stuff. It SOUNDS reasonable, but it's wrong. There is an infinite amount of lower-level content we can teach and test, but much of it is trivial, and is not necessary to master in order to move onto more complex subject matter. The tests have education stuck in 1st gear on this low-level content. Exemplary magnet schools have had to dumb down their content because "higher-standards" often just means learning a LOT of lower level material. This is the nonsense that high-stakes testing creates in schools.

Dr. Wheatley’s arguments in the previous two paragraphs seem to focus entirely on the quality of the tests being administered, not on whether we should have independent assessment. Considering that the states (not the federal government) determine what is measured and how it is assessed, surely there’s at least one state doing it right? And if not even one of 50 states can get assessment right, then how can we trust them on the other elements of education?

And I would also argue with the notion (in the last paragraph above) that we don’t need to master lower-level material in order to tackle more complex subject matter. This may be another of those intractable differences between me and Dr. Wheatley – I thought it was pretty well established that one has to master foundational knowledge and processes in order to grasp more advanced material. (See here, and the original Scientific American article here.)

Cockroach analogies aside, educators DO want meaningful accountability, but standardized tests can only be a piece of that, and local control must be a big piece of that process.

One thing that would be helpful would be to see an alternate proposal for accountability. If you want to eliminate NCLB but believe in establishing a meaningful accountability system, what would that look like? (And when I say meaningful, I would include words like ‘independent’ and ‘objective’ in that definition – I don’t believe there’s much meaning in having the same person who teaches the child also oversee evaluation – there’s too much conflict of interest.)

I teach teachers in the poorest major city in America, and many schools for poor children have become absolutely dreadful. If rich folks won't put up with scripted instruction for their children, why are we subjecting poor children to this mind-numbing nonsense? Talk about increasing the learning gap!

Many schools for poor children are dreadful. But this has nothing to do with scripted instruction in reading – it’s the only thing with hard evidence behind it showing its effectiveness (see here for information on Project Follow Through).

Never mind that the public has been misled about the state of American education for decades, and has been misled about the need for high-skill jobs, and have been misled about the tests themselves, and about how students and NCLB are doing (Increases in NAEP scores came almost entirely before NCLB kicked in). We're tired of the politicians and CEOs misleading Americans about education (I assume they just don't know any better), and we researchers are finally speaking up.

Again, this comes back to our respective views of the reason for public education. I would argue that citizens (represented by politicians) and businesspeople are the ones who are paying for the system, and who are doing so for a reason: to give children the tools they need to succeed in life. Clearly, those people believe that changing workforce requirements and global competition mean that our children will graduate into a very different world than existed in times past, and that our current education practices are not providing them with the tools they need to live successfully in that world. I think it’s perfectly reasonable for these people to insist that education change accordingly.

And a minor point: NAEP scores have been stagnant for decades. See here for results in math, reading, and science between 1971 and 2004. Pay particular attention to the scores for 17 year olds - the people just about to graduate.

If America's teachers weren't afraid of losing their jobs for speaking up, NCLB would have been dropped like a bad habit two years ago--because teachers would be telling everyone just what is going on.

I require teachers to learn about content standards, and the deepest irony of all is that teacher are not supposed to focus on some of the very sensible outcomes that have been dreamed up by the business community (What we look for in new workers) because they are not in the content standards-because they are hard to quantify on tests.

America is the drunk looking for the keys to its educational future under the streetlamp of high stakes testing, but that's not where the keys are.

And no, I'm not some left-wing nut. The U.S. flag is waving out front right now, I attend church every Sunday, and yes, I teach Sunday School.

God Bless You, and please help some of your friends wake up to NCLB. We were duped.

I don’t have any intention of casting aspersions on Dr. Wheatley, although I can understand his concern - there's a lot of flaming in this debate, and I'm glad he's not interested in participating at that level (I'm certainly not). I’m sure he’s a patriot, and when talking about NCLB, it doesn’t seem that political leanings make any difference. (Consider that Kennedy and Miller spearheaded the bill in the Senate and House and continue to argue for it, and that Republicans now want to gut it.) But clearly we see this issue differently.

I’ll wrap by disagreeing with his last statement. Dr.Wheatley says that we’ve all been duped by the promise of NCLB – I would argue that we’ve been duped for decades with faulty information on school performance, and that NCLB (or at least the spirit of NCLB) is an important first step on the road to a solution.

I’ve asked Dr. Wheatley to respond – I’ll be happy to post anything that he cares to send and make public.


  • I've posted copiously about NCLB on my own blog, but I'll be linking to and quoting from yours this afternoon. Well done.

    Should you be interested in my thoughts on the subject, surfk over to my blog and click on the NCLB label in the left column.

    By Blogger Darren, at 11:48 AM  

  • Hmm, it looks like you didn’t get my original reply. I’ll have to send a shorter version.

    Thanks so much for such a thoughtful response.

    A few points.

    1) I couldn’t agree with you more that schools are often very unresponsive to parents and local community. For example, if my child is in your school, I expect to be welcome to come in and see what you’re doing. Let’s have dramatically more local control—but that means bean counters in D.C. or Columbus can’t be calling all the shots. (Nor can the Business Round Table).

    2) Many groups are upset about NCLB, including parents who overwhelmingly think too much testing is going on. Mental health professionals are writing about mental health problems related to increased school stress, pediatricians have been critical—which is part of why they came out with a statement supporting unstructured play. Independent researchers (i.e., not part of DOE, Achieve, Ed Trust, etc.) have been very critical of NCLB—many from the very beginning.

    3) I agree—Disaggregated data is great. However, I see no evidence that poor kids are getting any better education now. I teach in America’s poorest major city, and in the panic to somehow pass those tests, many schools are doing some of the most mind-numbing parroting instruction imaginable. For some, they often just practice old tests. This turns education into Trivial Pursuit—cram answers you don’t understand for the test, forget, cram again. It’s a brilliant way to prepare kids for the 1800s, and ensure that poor kids can’t think well enough to compete for good jobs.

    4) I teach a lot about competition and competitiveness, especially in my master’s and doctoral courses on motivation. We like to pay competitive prices, want our kids to go to competitive colleges, and “compete” with ourselves to improve ourselves. The first problem with competition and competitive is that those words mean a lot of different things, and we tend to lump it all together, and assume it’s all beneficial. In the real world, the win-lose type of competition has various benefits and costs—sometimes the costs are greater than the benefits.

    Competitive markets bring us great, inexpensive cars—competitive classrooms create all sorts of problems for motivation, learning, and behavior—especially if we’re serious about educating all kids. Education is about learning; business is primarily about performance, products, and profits. Different dynamics apply in education vs. business.

    There’s no evidence I know of that competition between schools is beneficial overall. Private schools serve a legitimate function, but don’t do any better than public schools on apples to apples comparisons. Where charter schools are very loosely regulated, as here in Ohio, we’ve had disasters. Many of my students teach in charter schools, and their stories are just as disturbing as what I hear from Cleveland public schools. Once you distinguish between the various meanings of compete/competitive, the research on true competition in education is pretty depressing.

    5) It has been a long-accepted professional and ethical standard that no single assessment be used for high-stakes decisions. That’s one reason dozens of professional, civil rights and religious organizations have signed FairTests’s petition regarding changing NCLB. I believe you could get a disclaimer from every major testmaker—either on the web or on their materials—that tests are not designed to be used this way. (This is a big reason academics are seething—this is educational malpractice.)

    A sampling of quotes:

    "When these tests are used exclusively for graduation, I think that's wrong."
    —Eugene Paslov, President, Harcourt Brace
    ". . .Even if the SOL tests were beyond reproach, the use of test scores as the ultimate criterion for graduation decision violates professional standards for test use. Test scores should inform professional opinion, not override it."
    —Dr. Laurence H. Cross, Professor of Educational Research, Evaluation & Policy Studies
    "High-stakes decisions based on school-mean proficiency are scientifically indefensible. We cannot regard differences in school-mean proficiency as reflecting differences in school effectiveness. . . . To reward schools for high mean achievement is tantamount to rewarding those schools for serving students who were doing well prior to school entry." —Stephen Raudenbush, Schooling, Statistics, & Poverty (he’s a top expert on HLM)
    "There is no date by which all (or even nearly all) students in any subgroup, even middle-class white students, can achieve proficiency. Proficiency for all is an oxymoron, as the term 'proficiency' is commonly understood and properly used." —R. Rothstein, R. Jacobsen, & T. Wilder, 11/06
    " School leaders have a duty to undo the harm being perpetrated on schools today. " —W. James Popham (The Mismeasurement of Educational Quality)
    (Perhaps America’s best-known statistician. Has another book—The Truth About Testing—heavily critical of how we’re using tests. He’s written tens of thousands of test items, even helped Texas design their tests, but he knows what tests are and aren’t for).
    "NCLB is martial law. States that were doing innovative stuff took a step backwards when NCLB came along." —John Katzman, founder, Princeton Review

    6) I agree that assessment should be used for something—especially to inform altered or improved teaching. Most people will interpret many of the consequences of NCLB as punishments, and motivationally, the effects of something depend a great deal on how it is interpreted. Unfortunately, while punishments can effectively bring short-term compliance, substantial reliance on them is an inferior approach to motivation, what Stephen Covey calls the primitive carrot-and-stick motivational paradigm.

    7) High stakes are well known to affect validity. Think of two people—one who knows all subjects well, and one who is not very knowledgeable, but memorizes a stack of Trivial Pursuit cards. Both can answer many of the questions, but we get fooled into believing that both people are very knowledgeable, when only one is. Under high-stakes conditions, we see something akin to memorizing Trivial Pursuit cards. To really understand something well, you have to understand a million pieces, and how the pieces fit together. Tests can only target so many pieces and a few relationships, and teachers with their feet to the fire start teaching only to those things most likely to be on the tests. The easiest way is to just teach the low-level standards in your state—that’s what’s most likely to be on the test. Unfortunately, the teacher is now targeting a lot of dots, but not enough to make a meaningful picture of history or science, or whatever. In reading, they actually sometimes assess ability to decode nonsense syllables. No one needs this skill in real life, but it’s easy to test, and it leads to people drilling kids on nonsense syllables. This inflates our judgment of whether kids can really read. Also, students who learn something in order to pass a test are more likely to forget material quickly after a test when compared to students who learn something just to learn it.

    8) That a fair question--if the states can’t get it right, why trust them? Unfortunately, DOE has bullied states into doing assessment their way—by not accepting states’ plans. I believe only Maine and Nebraska have stuck it out and kept something interesting. Maybe Rhode Island. Many states doing interesting assessments returned to bubble sheets when pressured. If we treat people like sheep, we won’t get creative solutions.

    9) Fair enough—what’s the alternative assessment plan? Linda Darling-Hammond provides some examples in Right to Learn. You need multiple types of assessments, and multiple folks looking at the data—definitely including people outside the school.

    10) What do you mean by objective? Standardized? We say “objective test” a lot in education, but there is no such thing as objectivity in testing. In constructing tests, human subjectivity creeps in at every single step. Sack’s book Standardized Minds is a good source on this.

    11) I believe they re-evaluated Follow Through a while ago, using newer statistical techniques, and there was no longer any clear advantage for Direct Instruction. However, cooperative learning has a lot of research support, as the NRP report notes.

    12) People have been saying the sky is falling on American education for a century. Since a Nation at Risk, we had our two biggest economic expansions in history. When do we put the Chicken Little reports in some perspective? It’s raining in some places, even damaging hail in some places, but U.S. kids have been middling on tests for decades, yet those same kids keep going out into the world and outperforming kids with higher test scores from other countries. The tests have a place, but we’re trusting them way too much. I saw somewhere recently that among the most competitive countries economically, test scores are negatively correlated with economic competitiveness.

    We were cleverly sold on the idea schools were failing terribly and schools and teachers were behaving irresponsibly. That’s convenient spin, but it misleads the public about the complex story. We have “failing police stations” in most of the same neighborhoods, but we haven’t framed it that way, or gone after cops the way we went after teachers, because we know that’s a misleading way to frame it.

    13) More assessment data can be very helpful, but do we want meaningful evidence of real-world competence, or high-stakes testing? You can’t have both. With all this testing, no one in DC or Columbus really knows how well your local teacher is teaching. When I say fraud, I mean the gov’t is taking our tax dollars to pay for all this testing, and there are so many holes in the system, it’s essentially useless. And value added will just add another layer to the illusion that we have meaningful accountability—and watch your wallet on that.

    14) There are all sorts of important outcomes that cannot be bubbled in, including ability to carry out scientific experiments, and all sorts of technology applications. Paper-and-pencil testing is extraordinarily limiting, and even if you do essays, that has been reduced to pat formulas for writing a good 5-paragraph essay. Paper-and-pencil testing works best for math—which is why they always drag out math tests—but is much weaker for science, technology, etc. If you look at the outcomes most valued by parents and the business community, including items from SCANS report, many of them are not assessed on these tests, as well as the more complex content standards states have identified. The same thing is true in teacher ed.—what students do on some paper and pencil tests is both limited and misleading. I have them role play all sorts of things and we watch them in the classroom. What they can do on paper is just the beginning.

    By Anonymous Anonymous, at 12:09 AM  

  • Brett,

    Strictly speaking, tests are not valid or invalid--it's the SCORES from those tests that are valid or invalid. They are only reasonably valid if the test is used as intended.

    We have two identical thermometers on our house. They both produce relatively valid indications of the temperature in the yard--until its afternoon in the summer--when the sun hits one of them. Then the readings from that thermometer (inflated by 10-20 degrees) are no longer valid indicators of what I want it to tell me--the backyard temperature.

    The thermometer isn't broken--it just can't tell me what I want it to tell me when it's used under these conditions. The same is already true of high-stakes tests--especially where stakes are really high. We have no idea, for example, how well kids are being educated in Texas.

    Tests are not designed to be used in these ways.

    - Karl Wheatley, Ph.D.

    By Anonymous Anonymous, at 8:05 AM  

  • Dear Brett,

    Thank you for your extremely infuriating, but kind post supporting NCLB. Now, please come to my classroom in the the hood of Las Vegas and take over for me. I want to leave, but it will ruin the rest of my career and job opportunities for me to teach in a private school. The topic starting off next semester is rational expressions and properties of rational exponents for Algebra II. I have some interesting ways to present the material, but if I do that, I will not be able to cover the next set of objectives in the allotted amount of time. The students already failed the last semester test because I tried to teach them meaningful mathematics so they could understand the purpose of what they are learning. Now I realize I am forced in an unreasonable time limit to teach these kids abstract concepts without any connection to anything in their lives and have them pass these tests. Secondly, the students already hate me because they feel like failures, the parents think I stink, and the administration has already given me a bad evaluation and will probably keep me because they've been using permanent subs because no one wants to go through this torture. If I ever met you in person, I would hurl every insult and swear word at your dog because I am a human being. I know you have probably never set foot in a situation like this, and probably never will, and will continue to give your insights on NCLB. And I am not one of those just get by math teachers. If I could these students would know about Newton and Gauss and the significance of their discoveries. I hate to say this, but you are PATHETIC until you step into a situation like this. I HATE you!!!

    By Anonymous Anonymous, at 10:37 PM  

Post a Comment

<< Home