Charter schools, on average, do no better or worse than District schools.

Charter schools are better than District schools in educating poor children.

Charter schools don’t play fair.

The battlefield over charter schools success is littered with these simple refrains. Year after year, a new study or a review of charters or a School Reform Commission hearing brings out a new interpretation of charter schools’ value—each one presented as though it is the end of the discussion.

The latest came just a few weeks ago, with the release of a study of urban charter schools from Stanford’s Center for Research on Education Outcomes (CREDO), which found that poor students in charters learned more than their comparable peers at traditional public schools. In Philadelphia, the results were striking: Based on test scores, the study indicated that, on average, charter school students received the equivalent of 40 more days of math learning and 28 more days of reading. School reformers (of a certain stripe) touted the results. As Sharmain Matlock-Turner and Michael Person wrote in The Inquirer: “The debate over charter schools’ ability to produce better results is over.”

That’s it. End of discussion.

Except that, of course, it’s not the end of the discussion—or even any real discussion at all. Sweeping statements about charter schools—or any set of schools—do not account for the complications and nuances of public education. They are more akin to the sort of low-minded discourse of the mayoral race, where candidates were asked in the first televised debate to answer “yes” or “no” to the question: Do you support charter schools?

One recent study found that poor students in charters learned more than their comparable peers at traditional public schools. In Philadelphia, on average, charter school students received the equivalent of 40 more days of math learning and 28 more days of reading.

It would be nice for the issue to be so simple: Digging into what is really behind the bumper sticker sloganeering takes you down a rabbit hole lined with funhouse mirrors, where every seemingly definitive statement has a series of what Jon Cetel of state education advocacy organization PennCAN calls “yeah, buts…”

“The charter school system is not a monolith,” says Adam Schott, Director of Policy Research for education think tank Research for Action. “When you’re asking the question of how charters perform for different groups of kids, you have to ask, What are the characteristics of that group? What counts as making gains? What do you consider success?”

So what do we know about charter versus District performance in Philly?

• The CREDO study offers a deeper dive into what happens at charters versus District schools than we have seen. Like other studies over the years, Stanford uses what it calls “Virtual Control Records” at traditional public schools to make as close to an exact comparison among city students as possible. For every one charter school student’s scores, the study uses an average of up to seven District school students’ scores, making sure that the demographics in certain areas are virtually the same: gender; race; grade; if they qualify for free or reduced lunch; English Language Learner; Special Ed; test scores at the start of the study. (This is a good breakdown of how they make the match.) But it still ignores nuances that can make a difference. Similarly, the District released its latest School Progress Report last week, grouping schools by demographics and other categorizations, to allow for comparisons within  similar cohorts. It too showed success at many charters, while highlighting great progress at some District schools. Unfortunately, not all charters participated in the study. And the rating system puts successful schools at a bit of a disadvantage: One important category in the SPR is student growth, which is naturally less at schools that are already doing well, academically.

“Because charter schools are schools of choice they may not have a student population that exactly mirrors the districts from which they draw students,” the CREDO study notes. “These differences are important for understanding which families elect to enroll their students in charter schools.”

• Individual stories of improvement are disproportionately taking place at charter schools in Philadelphia, in particular those run by Mastery, KIPP, Young Scholars and a few other stand-alones, like Folk Arts Cultural Treasures. (Mastery was the top-ranked school in four School Progress Report groups.) These are schools with open and relatively simple lotteries that are finding success not just in higher test scores, but are also cultivating school environments that are conducive to learning. And these charters tend to produce (at least the beginning of) turn around at the hardest schools to manage: Comprehensive high schools. This doesn’t mean even well-managed charter schools are the answer for every child, in every neighborhood. Different families in Philly have different needs. But for many, it is undeniably true that a Mastery school will provide a better education than the neighborhood public in poor neighborhoods around the city.

• The cyber charter schools—located around the state, but enrolling hundreds of Philadelphians—do a terrible job of educating children. They, and several other local brick and mortars, should be closed. But state law makes it difficult to shut down even poorly-performing charters, except in instances of galling financial mismanagement or corruption. That could be changing: The School Reform Commission is likely to vote for the closure of some poor-performing charter and/or Renaissance schools at its meeting this month. The School District this year is also instituting an annual rating system to make charters more accountable, both academically and financially. (The first report should be out in early 2016.) And there is pending state legislation that might make it easier to close failing charters. If this had been the case all along, at least some of the debate over charter schools could have been upended years ago. Instead, thousands of students have swapped their struggling neighborhood schools for struggling charters—which hurts everyone in the system.

• There is poor, and there is poor. CREDO and other researchers have reported  that charters do a better job of educating low-income children, based on the number of students who meet the federal standard to qualify for free or reduced lunch. As the CREDO study’s authors note, District schools in Philadelphia enroll more disadvantaged children than charters—87 percent versus 77 percent. But even those numbers can be deceptive. Not all District schools have so high a percentage of poor students; many, in middle class neighborhoods, have far fewer than the charter school average. Even within the population of poor students, there are variables: There are those with two working parents at home who make minimum wage, and those who are in foster care, homeless or otherwise enmeshed in the justice system. A recent CHOP study found that there are significantly more students in neighborhood schools who are involved in the child welfare or juvenile justice system, which brings with it a whole other set of social and psychological issues that can affect learning. “Those are very different kids,” says Schott, “with very different needs.” Even the act of enrolling in a charter school requires parental involvement and a stable address. “Because charter schools are schools of choice they may not have a student population that exactly mirrors the districts from which they draw students,” the CREDO study notes. “These differences are important for understanding which families elect to enroll their students in charter schools.”

“When the District asked for people to take over neighborhood schools, a relatively small number applied,” says Lapp. “When it came to starting a new school, you had all these people signing up. That’s when you can control everything—but that’s not the ballgame.”

• We also need to consider English Language Learners and Special Ed students. Every study shows that charter schools in general have fewer Special Ed students and fewer non-native speakers than District schools. In the CREDO study, 11 percent of charter students were in Special Ed, compared to 13 percent in District schools. The difference among ELL students was even greater: 3 percent in charters, compared to 7 percent in District schools. This matters. Special Ed and ELL take more resources, more teachers, more hours—something, ironically, that charter schools often have more of than traditional publics. Again, though, this is not the whole story. Even among District schools the number of ELL students greatly varies. Like some charters, magnet schools and schools in neighborhoods with few immigrants have no ELL students to speak of. Around 20 schools in immigrant-heavy communities, on the other hand, have a huge percentage of ELL students. As for Special Ed: “No public school can say that we don’t serve these kinds of kids,” says David Lapp, staff attorney at the Education Law Center. But Lapp says parents don’t always know that, and that he often gets calls from parents who have been turned away from a charter because their children have complex needs. “That’s illegal,” he says.

Studies show that charters do a better job of educating low-income children, based on the number of students who qualify for free or reduced lunch—which in Philadelphia is 87 percent. But District schools in Philadelphia enroll more disadvantaged children than charters—87 percent versus 77 percent. And even those numbers can be deceptive because not all District schools have so high a percentage of poor students.

• Not all “Special Ed” is the same. The term Special Education encompasses many things: those children who have a speech impediment, those who have ADHD, those who have severe mental or physical disabilities. The public school system is required to educate them all. In the District, that often means assigning children with particular needs—those who are blind, or have cerebral palsy, for example—to certain schools that are set up to work with those students. This can be an advantage to everyone: Students can get specialized services, in a community designed for them; and the District can pool its resources in a few locations, rather than scatter them throughout. Charters, for all their touted nimbleness, are not designed to take on these challenges. And, as a group, they have fewer high-needs Special Ed students than traditional publics. This affects tests—the more children there are with difficult learning issues, the lower the test scores—and also funding. In the state’s bizarre funding formula, schools get a lump sum for each Special Ed student, essentially the amount needed to pay for services for a child in the middle of the Special Ed spectrum. That means schools with lower-need children—like most charters—end up with extra funds, while schools with higher-need students—like many neighborhood schools—get far less than they need. “We have really screwed up the funding mechanism,” says Lapp. “The incentive is built in not to serve a lot of kids with severe needs.”

• Some charters stack the decks by refusing to “backfill.” Kevin McCorry at Newsworks did a phenomenal job laying out the details of this earlier this month. Unlike District schools, which have to take any child from the neighborhood at any time, many charters have a policy of not replacing students who have left for any reason. McCorry uses the example of Freire Charter School, where the graduating class was half the size of the incoming 9th grade class four years earlier. The ones who left? Probably students who couldn’t handle the tough curriculum—and boys, lots of boys. This allows those charters the ability to maintain a culture that is honed on day one of school, part of what is valuable about successful charters—but also gives them a clear advantage. (As McCorry notes, some of the most successful charters—notably KIPP, Mastery and Young Scholars—fill seats as they are emptied.) District schools have to accept kids throughout the year, even if they walk in the door in April (right before testing), and some see a 50 percent turnover throughout the year. Inevitably, this makes educating them (and their classmates) more difficult. As part of its deal with the new five charters it approved in February, the SRC mandated that they have to backfill, and some—including Freire— have started to do that on their own. Superintendent Hite has also called for charters to track their students, so the District can get a clearer picture of who is at the school, something Public Citizens for Children and Youth and others have called for for years.

• Renaissance schools can work. Philadelphia is unique in having Ren Schools, which are neighborhood schools that are run by charter companies. Many are showing great progress with the same population of children as the schools they replaced. (Although some, notably Universal schools, are faring poorly.) So it is possible, at least in some cases. But it is slower progress, and after five years, none are what you would call a great school—yet. This year, the District is going to consider the charter renewal for the first group of Renaissance schools—likely shutting some down—which will provide an in-depth look at how the city’s charters can run neighborhood schools. This work is harder—and charter companies know it. “When the District asked for people to take over neighborhood schools, a relatively small number applied,” says Lapp. “When it came to starting a new school, you had all these people signing up. That’s when you can control everything—but that’s not the ballgame.”

Overwhelmed yet? That’s exactly why we tend towards the sweeping generalizations. But broad statements—and overly-broad comparisons—don’t solve anything in an educational crisis like the one we’re in. The hard work is understanding what happens at individual schools beyond test scores and simple stats:  Is Samuel Fels High School making progress in areas that are important for its particular students? Has Mastery Smedley figured out the tools needed to bring its population up to standards? Can what works at one neighborhood school—or at a charter—work at another? Is anyone trying to make that assessment? If you care about educating kids, you need to spend time in the fine print. What matters is not how charters in general compare to District schools in general—because, really, there is no in general when it comes to our school system.