Behind The Rankings

1. How does the Challenge Index work?
We take the total number of Advanced Placement, International Baccalaureate or Cambridge tests given at a school in May, and divide by the number of seniors graduating in May or June. All public schools that NEWSWEEK researchers Amy Novak and Dan Brillman and I found that achieved a ratio of at least 1.000, meaning they had as many tests in 2008 as they had graduates, are put on the list on the NEWSWEEK Web site, Newsweek.com.

NEWSWEEK published national lists based on this formula in 1998, 2000, 2003, 2005, 2006, 2007 and 2008. In The Washington Post, I have reported the Challenge Index ratings for every public school in the Washington area every year since 1998. I think 1.000 is a modest standard. A school can reach that level if only half of its students take one AP, IB or Cambridge test in their junior year and one in their senior year. But this year, less than 6 percent of the approximately 27,000 U.S. public high schools managed to reach that standard and be placed on the NEWSWEEK list.

2. Why does the number of schools on the Newsweek.com list get larger, and the ranks of most of the schools drop, after the list first comes out?
On the Web site we invite all qualifying schools we may have missed to e-mail us their data so that we can put them on the list. There is no national database that has the number of AP, IB and Cambridge tests and number of graduates for each public high school, so we have had to build our own. We are happy to capture the few schools we missed by using the publicity generated by publication of a new list. When you add more schools to any ranked list, most of the schools already on the list see their ranks drop. See questions No. 7 and 12 for why we rank, and why you should not pay much attention to ranks.

3. Why do you count only the number of tests given, and not how well the students do on the tests?
In the past, schools have often bragged of their high passing rates on AP or IB as a sign of how well their programs were doing. When I say passing rate, I mean the percentage of students who scored 3, 4 or 5 on the 5-point AP test or 4, 5, 6 or 7 on the 7-point IB test. (The Cambridge tests, although similar to AP and IB, are used in very few schools, and rarely appear in school assessments.) Some IB exams are composed of several separate sections, called "papers," but we only count one exam per IB course. Passing AP or IB scores are the rough equivalent of a C or C-plus in a college course and make the student eligible for credit at most colleges.

I decided not to count passing rates in the way schools had done in the past because I found that most American high schools kept those rates artificially high by allowing only top students to take the courses. In other instances, they opened the courses to all but encouraged only the best students to take the tests.

AP, IB and Cambridge are important because they give average students a chance to experience the trauma of heavy college reading lists and long, analytical college examinations. Studies by U.S. Department of Education senior researcher Clifford Adelman in 1999 and 2005 showed that the best predictors of college graduation were not good high school grades or test scores, but whether or not a student had an intense academic experience in high school. Such experiences were produced by taking higher-level math and English courses and struggling with the demands of college-level courses like AP or IB. Several other studies looked at hundreds of thousands of students in California and Texas and found if they had passing scores on AP exams they were more likely to do well academically in college. In the latest Texas study, even low-performing, low-income students who got only a 2 on an AP test did significantly better in college than similar students who did not take AP in high school.

To send a student off to college without having had an AP, IB or Cambridge course and test is like insisting that a child learn to ride a bike without ever taking off the training wheels. It is dumb, and in my view a form of educational malpractice. But most American high schools still do it. I don't think such schools should be rewarded because they have artificially high AP or IB passing rates achieved by making certain just their best students take the tests.

NEWSWEEK and The Washington Post, however, have added a new statistic developed by the College Board that indicates how well students are doing on the exams at each school while still recognizing the importance of increasing student participation. It is the Equity and Excellence rate, the percentage of all graduating seniors, including those who never got near an AP course, who had at least one score of 3 or above on at least one AP test sometime in high school. That is the "E&E" on our list. "Subs. Lunch" on the list stands for the percentage of students who qualify for federally subsidized lunches, the best measure of the percentage of low-income students at each school.

The average Equity and Excellence rate in 2008 was 15.2 percent. In the 2008 NEWSWEEK list, we give the Equity and Excellence percentage for those schools that have the necessary data. We ask IB schools to calculate their IB, or combined AP-IB, Equity and Excellence rate, using a 4 on the 7-point IB test as the equivalent of a 3 on the AP.

4. Why do you divide by the number of graduating seniors, and does that mean you only count tests taken by seniors? Don't you know that juniors, and sometimes even sophomores and freshman take AP tests?
We divide by May or June graduates as a convenient measure of the relative size of each school. That way a big school like Troy High in Fullerton, Calif., which gave 2,871 AP and IB tests and graduated 502 seniors in 2008 for a rating of 5.442 this year, will not have an advantage over Northcoast Preparatory and Performing Arts Academy in Arcata, Calif., which gave only 98 AP tests but also graduated only 18 seniors for a rating of 5.444. On the 2009 NEWSWEEK list they are right next to each other at No. 30 and 31.

We count all tests given at the school, not just those taken by seniors.

5. How can you call these the best schools or the top schools if you are using just one narrow measure? High school is more than just AP or IB tests.
Indeed it is, and if I could quantify all those other things in a meaningful way, I would give it a try. But teacher quality, extracurricular activities and other important factors are too subjective for a ranked list. Participation in challenging courses and tests, on the other hand, can be counted, and the results expose a significant failing in most high schools—SO far less than 6 percent of the public high schools in the United States qualify for the NEWSWEEK list. I think that this is the most useful quantitative measure of a high school, and one of its strengths is the narrowness of the criteria. Everyone can understand the simple arithmetic that produces a school's Challenge Index rating and discuss it intelligently, as opposed to ranked lists like U.S. News & World Report's America's Best Colleges, which has too many factors for me to comprehend.

As for the words "top" and "best," they are always based on criteria chosen by the listmaker. My list of best film directors may depend on Academy Award nominations. Yours may be based on ticket sales. I have been very clear about what I am measuring in these schools. You may not like my criteria, but I have not found anyone who understands how high schools work and does not think AP, IB or Cambridge test participation is important. I often ask people to tell me what quantitative measure of high schools they think is more important than this one. Such discussions can be interesting and productive.

I have been having such a debate with Andy Rotherham, codirector of the Education Sector think tank. He argues that some of the schools on the NEWSWEEK list have low average test scores and high dropout rates and do not belong on any best-high-schools list. My response is that these are all schools with lots of low-income students and great teachers who have found ways to get them involved in college-level courses. We have as yet no proven way for educators in low-income schools to improve significantly their average tests scores or graduation rates. Until we do, I don't see any point in making them play a game that, no matter how energetic or smart they are, they can't win.