What's the Score? Holding Florida's Education Accountability System Accountable

Introduction

Last week was significant for education policy in Florida. While the Florida Department of Education held three public hearings around the state regarding how to score the new Florida Standards Assessment (FSA), the Senate and House Education Committees also met, to discuss the FSA validity study and its implications for Florida’s school accountability policies.

FSA Cut Scores

Soon, Education Commissioner Pam Stewart will make cut score recommendations to the State Board of Education, which will then have the final say on the scores. Two panels that were convened, one comprised of teachers and one of education and community leaders (the reactor panel), made their recommendations, which would lead to tougher scoring than on the previous state tests. State Board members have already voiced their desire to have even higher cut scores, echoing the position of the Foundation for Florida’s Future and the Foundation for Excellence in Education, the education foundations established by former Governor Jeb Bush.

FSA vs. NAEP

The Foundations and Board members frequently reference the National Assessment of Educational Progress (NAEP) exams, and the need for Florida’s state test cut scores to lead to comparable achievement rates with Florida’s NAEP achievement levels. This comparison is problematic though, given that the NAEP is not aligned with our state’s standards and uses a different system of achievement levels. It is also only given to a sample of students in the state, about 2500 to 3000 from about 100 schools, and participants each take only a portion of the full assessment. Basically, comparing FSA scores to NAEP scores is comparing apples and oranges: different purposes, different standards, and different methods of testing. More information on this can be found at https://www.nagb.org/toolbar/faqs.html.

An Objective Measure?

Some legislators, State Board members, and advocacy groups have emphasized the importance of the FSA scores as the only objective indicator we have about student academic performance in Florida. While it is an indicator, to call it objective is fallacious. Which standards are appropriate for which grade levels is a subjective decision, and one that ultimately varies by the child. Whether or not the test scores are a valid indicator of a student’s actual knowledge is also subjective, encompassing whether the student is able to “show what they know” on the tests under ideal circumstances, and then also knowing the many problems with the state test administration this year.

Finally, the process of setting cut scores is subjective. It is disturbing how much of the conversation around what the cut scores should be in Florida is centered on what percentage of students would pass under different scores. The panel of teachers was specifically instructed to focus on test item content and what student achievement would look like on each item when making their cut score recommendations. The reactor panel then reviewed the teacher panel’s recommendations and made their own recommendations based on the impact of the scores, meaning what percentage of students will fall into each level. As we await Commissioner Stewart and the State Board’s final decision on scores, we don’t know explicitly what criteria they are considering, but the fact that this is a subjective, political process is indisputable. These cut scores have a very real impact on the lives of students, teachers, families, and school communities. For legislators, State Board members, and advocacy groups to claim the FSA test scores are an objective measure and comparable to NAEP scores is intellectually dishonest. More information on the state’s cut score setting process is available at http://www.fldoe.org/core/fileparse.php/5663/urlt/FSAAchievementLevelCutScores.pdf

Given their actions over the past several years, Commissioner Stewart and the State Board of Education’s ability to set reliable and valid cut scores is suspect. Unfortunately, this is one of the least of our problems with Florida’s testing and accountability regime, because even if the cut scores were fair, the rest of the system is not.

In Committee

Following the Senate Education Committee meeting last Thursday, September 17, where several Senators showed leadership by critically engaging with the problems our current education policies have created, Committee Chairman John Legg, R-Lutz, indicated that his concerns over the FSA validity study had been assuaged and he is moving on to other issues. Many legislators, educators, and parents disagree, and as the Pre-K-12 Education Committee Chairman, he needs to acknowledge their concerns.

During the love-fest that was the House Education Committee meeting, critical inquiry was in short supply, but at least the members took the time to all sign a thank you note to Commissioner Stewart, presumably for putting up with the Senators the day before. Committee Chairwoman Marlene O’Toole shared how she always promotes school choice to parents, but her enthusiasm for parental involvement does not extend to parents trying to improve their children’s current schools. Answering a question about parents who disapprove of the state’s testing and accountability system and would like to have their children opt out of taking the FSA, O’Toole said, “You’re not allowed to do that and keep your child in public school. So if you want your child to learn another way or do something different, you always have the option to take’em.” Of course! Every family has the option to homeschool or pay for private school, right? The hubris in these comments is mind-boggling. O’Toole is effectively saying that the children of taxpaying families in Florida will be denied the “efficient and high quality” education they are entitled to under Florida Statute because their parents decline having them participate in a flawed state testing system. So much for parental rights. I would also suggest that O’Toole and the other committee members read the response to Question 11, in this document from Seminole County Public Schools, which lays out a common sense approach to dealing with students who opt not to take the tests. I wonder what Rep. O’Toole would recommend doing differently? Arresting parents? Expelling students?

Florida’s Condescending Legislature

The problem with bad policies is that when you put them all together, they become even worse. The “move along” mentality we’ve seen from Commissioner Stewart and Chairman Legg has grown old. Two years ago it was the constantly manipulated school grading formula and the abrupt resignation of the Education Commissioner at the time, Tony Bennett, over a prior school grading scandal in Indiana. One year ago it was the renaming of the Common Core standards as the Florida Standards after making a few changes, mostly involving calculus and cursive. It’s a good thing the standards didn’t change too much though- that’s why we were able to buy test questions off Utah, whose standards are also based on the Common Core. Now it is the rushed implementation of both the standards and the new tests, and the insistence on still attaching high stakes to them. None of the problems of the past have been solved. They’re still here, and growing.

Besides being told to move along and not worry, the other things citizens are getting tired of hearing is how the reason they disagree with policies is because they don't understand them. This is the case with the Value-Added Models (VAM) that Florida is implementing to measure teachers' contributions to student learning. This is the formula for calculating VAM: 

I’d love to hear a legislator or State Board member explain this model in layman’s terms. I’ve earned A’s in doctoral level statistics classes that focused on education data, and I wouldn’t even know where to begin. Yet despite organizations such as the American Statistical Association cautioning against using VAM scores for high stakes purposes, the score will count for a third of a teacher’s evaluation in Florida. Evaluations guide decisions on hiring, firing, and salary. Also of note, Florida’s VAM formula controls for several different student characteristics, but fails to take socioeconomic status into account. Because, you know, poverty- how could that possibly impact student learning?

Then again, this month, when parents, educators, and legislators questioned the FSA Validity Study, what was the response from the FLDOE and Foundations? The only problem is, you just don’t understand. Even the study’s lead author, Andrew Wiley, a Senior Psychometrician with Alpine Testing Solutions, acknowledged during the Senate Education Committee meeting that “there is room for professional disagreement” regarding the study’s conclusions. Wiley further stated, “I think there is data in the report that can be looked at and pointed to that says maybe the use of these test scores would not be appropriate, and quite frankly there was rigorous debate within our own group of people. This was not an easy decision.” I, for one, would be very interested to hear the other perspectives in this rigorous debate. I doubt the Commissioner would share that sentiment.

The FSA Validity Study

Regarding the validity study, much has been made over the FSA’s alignment with our state’s standards. I don’t know that this is a major issue, since most items, if they didn’t match a particular standard, matched another standard that was similar. The statistics on the Grade 3 ELA exam were particularly troublesome though, with review panelists selecting a different standard than the intended standard for 33% of the 60 test items, without even a partial standard match. Given Florida’s 3rd grade retention policy for students not passing this test, it has particularly high stakes.

Students also had to contend with a computerized testing system that malfunctioned numerous times, from students being unable to log-on, to students being unexpectedly timed out and losing their work. The circumstances under which students took the tests were far from standardized. Yet, the validity study supported using the scores for teacher evaluations and school grades because those are aggregate, not individual, uses. Across all test takers, the percentage that experienced problems with the administration was relatively small. However, those problems weren't evenly distributed across test takers. Problems tended to be concentrated in particular districts and schools, and for particular grade levels and tests. So for some teachers, maybe a lot of their students experienced problems that impacted their test score and for other teachers, none of their students had these problems. This applies to school grades as well. Teachers and schools aren’t operating on an even playing field.

Some school districts and legislators have advocated replacing the FSA with national exams such as the Iowa Tests and the SAT. Commissioner Stewart has said that won’t work since those tests aren’t aligned with the Florida Standards, but that didn’t stop us from using test questions from Utah this year. The Iowa Tests and the SAT are aligned with the Common Core standards, which are remarkably similar to the Florida Standards. Plus, the SAT is already approved as an alternative test for high school students who don’t pass the FSA.

Who Benefits?

The time for actual learning that is lost and the taxpayer money that is wasted on these tests is obscene. And for what? I don’t think any parent, teacher, or student would say that the contribution these test scores make to student learning outweighs what they take away through lost learning time and sending students the message that the most important thing about them as learners is their test scores.

Do these tests serve our students, their families, and their teachers? No. They primarily serve the bureaucratic nightmare that our state’s education system has become. We must have the test scores so we can grade our schools and evaluate our teachers. Does anyone really think school grades tell us much about student learning anymore? The formula has been changed numerous times and the grades are based on a testing system that is highly flawed. Furthermore, this year’s grades will be calculated based on an incomplete formula that does not include student learning gains, since with the new tests, there is no prior year comparison for growth. Students, teachers, and schools are the ones that are hurt from this unfairness.

Despite the protestations of superintendents, school administrators, teachers, and parents, and despite the fact that using the FSA scores could hurt teachers (through unfairly low evaluations), schools (through unfairly low school grades), and students (through having to repeat a grade, having to take additional tests, and so on), the Florida legislature and the FLDOE have been insistent on the importance of releasing school grades this year.

Why? Perhaps because if there were a year without grades, people would realize how little they truly signify about a school. Without the grades there to label school communities as D’s and F’s, how would we continue to funnel students into voucher programs and for-profit charter schools? You know, the ones run by people and businesses wealthy enough to afford big donations to our legislators. Goodness knows, the average teacher can’t afford to make campaign donations. For-profit education companies donated $1.8 million to Florida candidates and political parties in 2012, and I expect the amount has grown since, commensurate with the growth of the industry. Why so generous? They rely on legislators passing bills that will support their businesses, which rely on public schools looking bad to justify their existence. We also have multiple legislators with direct connections to the for-profit academic industry. How might this impact their judgment as they vote on bills related to their own or their family members’ livelihood?

When those who are actually directly involved in education are recommending one path and only legislators, lobbyists, and state-level bureaucrats are recommending another, it is a shame that it is the latter path that the citizens of this state are being forced to follow. It would almost be funny if the stakes weren’t so high, that now we have politicians trying to downplay the importance of school grades. All we’ve heard for the past fifteen years is how crucial and vital the grades are to tell us how students and schools are performing; and how parents need a clear and objective letter grade to understand the overall performance of a school, just as our students are graded. If a teacher assigned semester grades to students based on the outcomes of one test, it would be grossly unfair and students and parents would be up in arms. Yet that is essentially what our state grading system does. Grading schools should not be equated with grading students because if we graded students the way we grade schools it would be considered educational malpractice.

Educational malpractice is an apt description for what is happening in Florida right now, and citizens should demand better from our legislators. Opting out of this flawed testing system is one option, starving the beast, so to speak, although this is not a long-term solution. Voting out those who play politics with our children’s futures is another. Citizens who disagree with the direction of our state’s educational system need to make their voices heard.

On a Personal Note

Although I am certified to teach in Florida, as an educator who valued my ability to work autonomously and creatively, who expected and enjoyed making my own lesson plans and tests, and who would hope to be able to do my job without a micromanaging and timewasting bureaucracy getting in the way, I would not want to teach in a public school in this state. The current tangle of dysfunctional policies is going to keep many intelligent and talented people from going into the education profession.

Wait- what’s that you say? The legislature has decided to give teachers bonuses for having high SAT and ACT scores? It makes sense, in light of their high regard for standardized test scores, but that is the only way in which this policy makes sense. This legislation is just the latest insult to teachers in this state.

I have also “opted out” of the public school system for my own children, something I never expected to do. I did so to insure they would be seen by their school and teachers for who they really are, not labeled by levels and scores. I want to know that their teachers are free to teach in the manner they deem best and that the time they spend in school is used wisely. I am paying to have these things that were once a part of the Florida public school system where I was educated. That’s a sad truth. As a former teacher with two master’s degrees in education, I say with confidence that I have lost, and will lose, nothing by not having this type of test score for my children. I talk with my children’s teachers, I look over their schoolwork, I read to them, and, as a former English teacher and avid reader, I am glad to know my children will be choosing books based not upon their Accelerated Reader level, but according to their interests and curiosity.  

Conclusion

At this point, the State Board of Education, Commissioner Stewart, and legislators are all trying to pass the buck, claiming they are powerless to stop the release of school grades because they are currently required by law.

Gov. Scott, this is an opportunity for you to demonstrate leadership by issuing an Executive Order halting the use of FSA scores for all high stakes purposes at least until that time when we have unequivocally valid tests and the capacity to administer them in a standardized manner. You can begin the process of returning control over Florida’s schools to the citizens of this state.

Legislators, please continue to work toward fixing this mess and stop spinning the web of policies that serve political and corporate interests over students, schools, and communities. There are better policy directions that would lead to greater student learning and preparation for the future. I hope and suspect that you already know this. If you don’t, email me at jenny@learningbeyondmeasure.com. I would be happy to talk with you. For our conservative legislators, I also encourage you to read my blog posts, On Thrift and On Freedom. They address the ways that many education reform policies are actually not conservative at all.