74 Interview: Researcher Susan Dynarski on Boston Charters, Student Poverty, and Data in Education
;)
Untangle Your Mind!
Sign up for our free newsletter and start your day with in-depth reporting on the latest topics in education.
See previous interviews: Secretary of Education Arne Duncan, former U.S. Sen. Tom Harkin, current U.S. Senator and education committee Chairman Lamar Alexander and Professor Chris Emdin, who talked about “white teachers in the hood.” Full 74 Interview archive here.
When not writing prolifically for academic journals on both K–12 and higher education policy, University of Michigan economist Susan Dynarski is also a prominent Tweeter and a regular New York Times contributor.
We spoke at length earlier this month, about the use of research in the mainstream education debate, the enduring legacy of No Child Left Behind, and Dynarski’s own research that focuses on Boston charter schools and measuring student poverty. (The interview has been lightly edited for clarity and length.)
The 74: How do you feel like educational research is used in terms of informing policymakers and journalists?
Dynarski: It is a bigger issue, like how is research used to inform any policy in the U.S. or any country, for that matter. And education policy is so diffuse. I think you can probably say something about medical policy and health policy because I think there’s a more centralized mechanism for creating it. Medicaid and Medicare by what they’ll pay for, for example, push medicine in certain directions, and that tends to be based on randomized trials. There’s no such central body that decides education policy in the US.
And randomized trials are such a luxury in education policy.
What do you mean by luxury?
Meaning that you don’t get them as frequently as you would like and policymakers don’t explicitly build them in a lot of the time.
That’s right. In fact, they’re almost never built in. It’s easy enough to do and it’s starting to take hold in some places, but it’s very much dependent on individual organizations and their personalities and who their leaders are.
I think it’s certainly getting better over time. I think there’s more data now that helps us understand what’s happening in schools and districts than ever before in history. I actually think that’s the main legacy of No Child Left Behind — not accountability but the fact that we now can measure test scores in every state on an annual basis. That was not true before NCLB. Go through the list of research papers, statements that are made about children and performance — most of them could not be made without those data.
Part of what’s been going on too is that federal policy has been pushing the states and districts towards using more evidence in their policy and that’s been a good development. First through NCLB and then during the recession, actually, various strings were attached to monies in the federal government to really expand data analysis and data capabilities in the states.
Is there a lot variation between states in how much data they collect and how accessible it is to educational researchers? I’m asking because I reported on California, where they’ve totally gutted their educational data collection systems, and they also make it a real pain, if not impossible, for researchers to collect data. But it seems like other states are doing a much better job of that.
There is huge variation. California I would say is an outlier, so you just cited the worst example.
And it’s a small state, so we shouldn’t worry about that!
It’s astounding. All the data actually exists, and what they canned and defunded was pulling it together. California is a mess. Researchers are dealing with it by creating partnerships and creating data sets at a more local level. Some cities are getting together — I know there’s at least one consortium of large cities that have pooled their data for analyses. The college system is forging on with research despite the state not creating a state-wide data set.
There are a lot of different models out there. This really is a case of the states being a laboratory. North Carolina makes its data very freely available to researchers who are conducting education research. And when I say freely available, that’s under all of the conditions of security and privacy that are necessary to keep these data secure. But if there is a valid research question to be asked by a university-based researcher, North Carolina will help that happen.
Do you have any advice for researchers in terms of translating their research in a way that’s more understandable and practical for journalists and for policymakers?
I think if an idea can’t be expressed in plain English, it’s not a clear idea. If you start out by writing your papers in such a way that they’re comprehensible by researchers from many fields, then you’re on the right track. I think an awful lot of academics don’t pay sufficient attention to the quality of their writing and in fact, sometimes it’s almost encouraged to write in as opaque a way as possible in order to make it look professional and part of the clique. That really minimizes the impact of the work. So first it’s just making your own papers accessible.
Then if you’ve got the resources, if you’ve got a university that can help you in writing more accessible briefs, that’s also terrific. Twitter and blogs are a great way to take your own work and translate it in a short form so others can get hold of it. I’d be careful though of working with press offices at universities; they often are looking for the headline and they’re not experts in your topic and you can lose control of the message very easily. If you look out there and you see a lot of wilder claims about cocoa curing dementia and so forth, these tend to be university press offices run amok — sometimes it’s the researcher who’s guilty — but their job is to get press and your job is to get the message out that you know is the right one. Those can be in conflict.
Can you talk to me about yourself and how you balance the role of being an objective, rigorous researcher, while also being a public intellectual, writing in places like the New York Times, and expressing strong opinions? Do you see a tension there or do you think those two things go hand in hand?
There is a tension. I think student loans is a good example. Student loans are a policy area that was pretty much running on a data free-basis. There wasn’t much good academic research on student loans — I’m thinking back five years — and there wasn’t even much good data. When you lack data and research, you see a lot of theorizing and anecdotes and that’s what policy was running on. A willingness to even promulgate and call attention to basic facts can certainly be useful.
I rarely have taken positions on very specific pieces of legislation but rather just tried to get the information out there that I hope can steer and inform the conversation. I don’t find the roles to be in conflict. I do think this is a role for senior faculty; it’s a tough one for junior faculty to balance. The protection of tenure certainly helps.
One thing I think about being a “public intellectual” — a researcher who talks about their research in the public arena — is I try to stick to my topics of expertise. My position does not give me expertise in everything. Many people go wrong where they become “pundits at large.”
Journalists struggle sometimes where we try to figure out very earnestly, “Well what does research show?” And we talk to one expert who says X and the other who says Y that is completely the opposite, and so it’s very hard to parse. Sometimes the research is mixed. But it’s hard to parse how two very smart people can be telling you the complete opposite thing.
And maybe that has to be what’s written about: that this is an unsettled question. Of course, we want to avoid the false equivalence, if the weight of the research is pretty clear, and there are just a couple dissenters, then it should be expressed as such. Climate change being sort of the classic example of where I think the media was too balanced for a long time.
Let’s talk a little about some of your specific research interests. Can you talk to me about your research on Boston charter schools and what you found?
Along with researchers at Harvard and MIT I’ve been engaged in a long-term research project looking at the performance of charter schools in Boston. We have found that they have some of the largest positive effects on achievement that I have seen in any setting.
The middle schools and the high schools increase math scores in particular enough that in a few years students attending one of these schools can close the black–white test score gap for example. We use lotteries to reach these conclusions. The key challenge in any research into school effects is that students are not randomly assigned to schools. They choose which schools to go to, so it’s very difficult to get past the selection bias issue.
But when charter schools are oversubscribed, they run lotteries and that lets us compare outcomes for those who win and lose the lottery. That’s how we’ve reached these conclusions about charter performance in Boston. We’ve also found that the high schools increase college attendance, increase the quality of college attended by the student, increase SAT scores, increase AP course taking. The charter schools in Boston seem to be doing quite a good job.
Our research has set into the policy debate, which is good. Our first papers on Boston charters contributed, I think, to changing the mayor’s mind about charter schools. There’s a very pitched battle under way right now in Massachusetts about raising the cap. Other states don’t have caps, but in Massachusetts that’s the operative cap.
How would you recommend a policymaker or even a voter in Massachusetts — where there’s a ballot initiative — look at your research in the context of thinking about raising the cap? On the one hand you find very positive results in Boston. On the other hand, that may not answer the question of whether the cap should be lifted. For instance, we don’t necessarily know that the new schools that would exist if there were no cap would be as good as the schools that you’re looking at under the cap.
Also the Boston results are very positive, but if you step outside of Boston to non-urban areas, suburban areas, the effects are null. There’s nothing going on. That would suggest to me that you lift the cap in the urban areas, and, if you choose to, have a cap in suburban areas.
The research on the effects of a program is not the last word on whether somebody should vote for a program. It enters into their assessment. There are also distributional questions. Maybe it’s more important to you that schools are governed by elected school boards that students are performing better. What we want to do is force people to weigh that decision appropriately. The discussion in Massachusetts, and in other places, often comes down to denying what the research says as opposed to saying that’s not my highest priority, I think something else is more important than that. That’s what I find discouraging.
What about peer effects? Could the positive effects of Boston charters be explained by peer effects? That is, the students who apply to charters are more “motivated” and so the winners, by definition, will attend schools with other more motivated peers.
I suspect that might be part of what’s going on, though our best efforts to try to determine this suggest it’s not the operative channel. We have a recent paper that looks at the high schools in Boston. We had heard a lot of questions about whether the low achievers in the charter schools were being pushed out so that perhaps the peer achievement in the charters schools was getting more and more rarefied. We actually looked at this.
We took the baseline achievement of students who won and who lost and then tracked the baseline achievement of their peers over time. What we found was that the baseline achievement of peers in the public schools rose more rapidly than it did in the charter schools. I think what’s going on is in the public schools they’re getting sorted out into the dropout recovery schools, because the dropout rate is no higher in the public schools than in the charter schools, so that’s not what’s going on. Somehow the lower achievers are disappearing more rapidly from the public schools than they are from the charter schools. If anything it’s the opposite story: The low achievers are getting pushed out more rapidly from the public schools than the charter schools.
Do you know if the Boston charter schools are spending more money per pupil than the traditional public schools?
Best we can see, no. Massachusetts keeps admirably good records on this stuff. If you look at our Quarterly Journal of Economics paper we’ve got links to this. There are both revenue and expenditure data that’s very detailed including grant resources and it looks about the same. They spend it differently. In particular, charters don’t have the same pension and health costs that the traditional public schools do, so they can deploy their resources differently.
What do you make of the argument that the gains that you’re finding are largely test scores gains, aside from the increased attendance to four-year universities? Maybe test scores just aren’t a very good proxy for educational achievement and attainment.
Well the college stuff is our answer to that. There are only so many things you can measure about children. You’ve got to look at what we can measure. I get pretty frustrated by those who say, “Well, but you’re not showing effects on ‘Y’” — where Y is something that goes unmeasured. That’s unanswerable — research cannot answer that question.
I think (Will) Dobbie and (Roland) Fryer in their work on the Harlem’s Children’s Zone poured millions into doing surveys of the applicants to the Harlem Children’s Zone. They actually therefore were able to get at stuff like teen pregnancy and crime and so forth, and they showed there were effects beyond the test scores.
Did you see the study that Dobbie and Fryer just released on Texas charter schools, showing that in no-excuses charter schools there were big gains in test scores and college attainment but the gains for income were quite small, not even statistically significant?
No, I haven’t. I’ve got to think that they’re pretty young as adults however. A quick response is that when you’re comparing people who do and don’t go to college, earnings differences tend to be quite small until you get to their thirties because when you look at people in their twenties, the people who didn’t go to college have more work experience. This is a well-known fact among labor economists. There’s a cross-over point, traditionally around age 28 and it could be getting later.
Do you have any theories for why the Boston charter sector seems to be so much better than other charter sectors in the country?
Boston’s got a particularly rich intellectual environment, and you can draw on a lot of high-energy, well-educated young people coming out of colleges. Potentially that’s part of it. The schools there tend to be start ups, as opposed to part of national chains, though that can cut both ways. KIPP is an amazing chain, in part because it diffuses its practices.
So, I’ve got hypotheses but I don’t have an answer.
Massachusetts in general has a well-functioning policy environment. The state does a really good job of monitoring the charter sector. The flip side of this cap is that the applications that are required are thorough, you need to have a serious plan, they review schools quite thoroughly as well. And I’m contrasting this with Michigan where we’ve got the wild West here: any post-secondary institution in Michigan can charter a school. There’s very little control on the front end and very little supervision on the back end. The key for charters improving education is that you get both autonomy, but also accountability. Massachusetts has found the secret balance there.
The one thing that strikes me about Boston is that I might have thought that charters would be doing best where traditional public schools were the worst, but Boston and Massachusetts cut against that theory.
That’s right.
Let’s talk about your recent research on the use of free and reduced-price lunch as a proxy for poverty. Can you tell me what your study found?
The traditional way to measure gaps between poor kids and other kids in school is to compare children who are eligible and not eligible for subsidized federal meals. The problem is that at this point about half of children are eligible for subsidized meals, so that’s clearly a pretty blunt measure of disadvantage. By contrast only about a quarter of children actually live in households in poverty. The reason why it’s a poor proxy is because of the eligibility rules. It’s for children who live in households with up to 185 percent of the federal poverty line.
We wanted to see if we could leverage the administrative data sets that so many people use for research, to develop a finer measure of economic disadvantage. What we found in Michigan is that while half of children are currently eligible for subsidized meals, just 14 percent have been consistently eligible since they entered the school systems. These children who are persistently disadvantaged have test scores that are well below the children who are never disadvantaged and also well below the scores those who are occasionally economically disadvantaged. The score gap between the persistently and the never disadvantaged is a full standard deviation, about 40 percent larger than what you get if you use the traditional measures.
Why should anyone other than researchers care about this finding?
Policymakers use eligibility for subsidized meals to distribute funds to schools. Many states, districts, and the feds, use this measure as a proxy for the disadvantage of the school to target resources to them. What we found in Michigan is that if you compare schools that have the same shares of currently disadvantaged kids, they have vastly different shares of persistently disadvantaged kids. We’re masking real variation in the needs of schools and the needs of kids. We’re failing to target resources as well as we could.
You also suggest that this may have implications for the use of test scores in teacher evaluation.
Yeah, so if persistently disadvantaged children are concentrated among certain teachers — and given the history of segregation in the U.S., I tend to think this is not implausible — then value-added test score measures will be biased against those teachers. Basically some teachers and schools — and we’ve known this intuitively for a while — have students who are in deep poverty and it’s not captured by the traditional measures. This helps us to measure that depth of poverty. The next step is for some researchers of value-added to see practically what difference this makes.
What other K–12 education research are you currently working on?
I’m working on a randomized trial in Tennessee that is testing the effect of dual credit courses in high schools. The dual credit course we know best are Advanced Placement courses, but the Tennessee legislature mandated a few years ago that high schools start creating courses for students who are less academically elite than students who tend to take AP courses. They wanted to see courses in advanced algebra, in greenhouse management.
Since the person who runs research in Tennessee is a former student of mine, Nate Schwartz, we decided, since it had to be ramped up somehow, that we would roll it out as a randomized trial — or rather I suggested and the state decided to take this approach. I don’t get to decide anything.
We’re watching what the effect is. The students who take these courses and pass the relevant tests automatically get credit in Tennessee's public post-secondary institutions. We want to see what the impact is on college preparation, college going and remedial education.
We also have an ongoing partnership with the state of Michigan and have been examining the effects of ‘college for all’ curriculum on student achievement. That’s out as a working paper, and we’ve got a new one coming out shortly looking at the effect of a Michigan scholarship program on college going.
The college for all working paper found null effects, right?
That’s right — we got a zero.
A lot of what I’m doing right now is working with the state of Michigan to further develop research and data capacity, trying to make the data more accessible to other researchers, and to link to other outcomes besides test scores. We’d like to know if part of the effect of a charter school is that it increases the safety of students. We certainly hear that from parents, so to get at that question, you’d want to see information about contact with the juvenile justice system. So how do we move forward with that research agenda while maintaining the privacy of those children but answering important questions?
One thing that frustrates me is that it tends to be the very people who say that test scores aren’t everything are then the same ones who don’t want to see you gathering more data.
Are you trying to follow the Boston charter school students through and out of college to also measure those long-run outcomes of attending a charter school?
Yeah, we’re impatiently waiting for time to pass. Basically, those charter school applicants age a year, each year that passes!
You haven’t been able to get around that?
Nope! The paper we just published looked at college entry and choice, but we need a few more years to pass. Median time to a degree is six years at a postsecondary institution, so we’ve got a few years to wait. And of course we’d love to be able to match on information about earnings and other outcomes.
Get stories like these delivered straight to your inbox. Sign up for The 74 Newsletter
