C. ANDERSON/BLEND IMAGES/CORBIS
Before deciding which doctoral astrophysics programme to apply for, Nick Lee perused the published rankings. He found a grab bag of results. The US News & World Report's numbers offered little substantive insight about the institutions he was interested in. The statistics from the US National Research Council (NRC) were updated too infrequently to help him. And the Times Higher Education World University Rankings offered data about the best universities for natural sciences, but not specifically PhD programmes in astronomy. Lee was left confused.
Despite their flaws, prospective graduate students frequently use such ranking schemes to assess doctoral programmes. But the various criteria in use make it hard to know which scheme to rely on. Students need to identify the most desirable programme, where they will not only invest the next several years of their lives but also start their research careers. For universities trying to attract top talent, the stakes are just as high.
Competition for a place in a top graduate school is fierce, and the number of applicants continues to swell. According to the US National Science Foundation (NSF), more than 150,000 science and engineering doctoral degrees were awarded worldwide in 2005, compared with 125,000 in 2002. In Britain, for example, the number of final-year PhD students at universities grew by 31% between 1999 and 2003.
Brian Maple: "high ranking attracts top-quality faculty members."
Despite the limitations of ranking schemes, many professors and administrators admit to using them to plan recruiting strategies and gauge where their institutions stand. "It is easier to attract higher-quality faculty members to a highly ranked institution," says Brian Maple, chair of the department of physics at the University of California, San Diego.
And the tables can play a major part in applicants' decision processes. One tenure-track faculty member in physics at a highly ranked university says he is worried about accepting a faculty job offer from a lower-ranked institution. (He asked to remain anonymous, as he is still negotiating.) Despite the department's positive attributes, including access to top research facilities, he worries that this early-career move could hurt his prospects later on. "Once faculty step down the rankings ladder, it's often difficult to climb back up," he says. "I have concerns about the loss of prestige, possibly leading to less funding and less competitive students."
Playing the system
With so much as stake for faculty members and students, it's no wonder that some universities attempt to manipulate the system.
Geoff Davis, mathematician and founder of the online resource PhDs.org, says he was once hired as a consultant to a university disappointed with the fall in its rankings. "My job was to figure out strategies they could use to regain their standing," says Davis.
One of those strategies is to manipulate 'admissions yield', the ratio of students offered a place to those who actually take up the offer. A higher yield boosts the rankings. To keep yields high and avoid being perceived as a safety net, upper-tier institutions may, for example, decide not to offer a spot to students with perfect test scores if they are considered likely to go to a higher-ranked institution. According to Davis, most rankings are vulnerable to this kind of manipulation. But it creates difficulties for a student trying to make the best choice.
Disparities among rankings criteria add to the confusion. Consider, for example, how the physics department at the Massachusetts Institute of Technology (MIT) fared in various schemes. US News & World Report's 2008 rankings of best US graduate schools in physics place MIT firmly at the top with a perfect 5.0 score. Bob Morse, data research director at US News & World Report, explains that the science rankings are based solely on a peer survey of deans and department heads. Small changes in the ranks are the norm because people's opinions tend to change slowly.
The Times Higher Education table of the top 50 natural-sciences universities (it does not offer a ranking for physics alone) places MIT second to the University of California, Berkeley, scoring 97.6 out of 100. The Times Higher Education combines six criteria to produce an overall table. Peer review accounts for 40% of the possible score. It also factors in citations and staff-to-student ratios, and asks employers at which institutions they seek future employees.
CLARE COLLEGE, CAMBRIDGE
Jerry Ostriker: "We asked faculty which metrics are important for their field, and are using those metrics to determine the new rankings."
In the category of natural sciences and mathematics, rankings drawn up by China's Shanghai Jiao Tong University rank MIT in sixth place, with a score of 90.4 out of 100, behind four other US institutions and the University of Cambridge, UK. This scheme does not use peer-review surveys but ranks according to academic or research performance. It uses criteria such as whether staff have won Nobel prizes or Fields medals, and the percentage of articles published in top journals. Clearly, varying criteria mean varying results.
Most of these schemes have multiple caveats. For example, basing scores on opinions could confer serious bias. "Assessing programmes this way comes down to measurement of the views in the field by peer review," says Jerry Ostriker, chair of the NRC's committee to examine the methodology for the assessment of research-doctorate programmes. This means that an institution with a good reputation is likely to keep it, Ostriker notes, whereas up-and-coming programmes are overlooked.
Ostriker calls this the 'halo effect'. The reputation of a highly ranked university such as MIT could buoy the ratings of an otherwise weak programme in one particular field. The halo effect also works in reverse. A university's overall low ranking could mask one stellar department.
Davis cites research done by the US National Bureau of Economic Research, showing that student decisions are a more reliable source of preference rankings than expert opinion, citation rates or admission yields. The bureau used students' choices of university as the measure of competitiveness, while controlling for biasing factors such as free tuition, alumni parents and distance from home. This scheme, according to the rationale, is more telling and less subject to manipulation by the universities.
In 2004, an initiative by the European Centre for Higher Education and the US Institute for Higher Education Policy attempted to address rankings inequities. The International Rankings Expert Group (IREG) sought a broad international standard. It published the Berlin Principles on Ranking of Higher Education Institutions in 2006, urging that rankings systems use rigorous data and transparent data collection and analysis methods. Some have heeded this recommendation.
Tailor-made rankings
J. FAHERTY
Dagny Looper: "it's what you do at the institution that really matters."
Part of the solution, as the IREG report asserts, may be league tables tailor-made for the user. At the PhDs.org site, which Lee eventually turned to for help, users seek the US doctoral programme of their dreams by sorting through dozens of attributes and choosing only the ones that matter most, such as sub-field, size, cost and prestige. Users input their priorities and assign a value from one to five, leaving fields blank if they are not important. Users can further customize the search by weighing and adjusting specific details, such as faculty citations and job placement rate. The site returns a list of ranked matches. Data come from numerous US sources including the NRC, NSF and the National Center for Education Statistics.
"We don't declare any programme 'the best'," says Davis. "Rather, we provide a tool to help people find the programmes that are the best for them."
Ostriker and others at the NRC recognize the need for better data. "We've asked faculty, in all the different fields, which metrics are important for that field," says Ostriker. "Now we're using those metrics to determine the new rankings."
In the past, the NRC relied on survey questions and statistics published by the universities. Its new rankings, to be released in September, seek to remedy flaws. The new metrics include citation counts for science programme rankings as well as data about the other graduate students, the mix of faculty members and student funding opportunities. Ostriker anticipates that some institutions will be upset, although he declines to name names.
Using the 'random halves' method to analyse the data, the NRC plans to provide a more honest assessment of the uncertainty involved. This requires comparing the results of one half of the data to the other half and then repeating that step using different sets of halves multiple times. It's a way of estimating how much the answers to questions depend on who is asked.
"We're hoping that once we have all the methodology in place we can update it every two years," says Ostriker. The NRC is also planning a website that allows users to rate institutions on the basis of variables that they consider important. Following the Berlin Principles, users will be able to disagree with the NRC committee's assigned weights.
See for yourself
Of course, looking at tables is no substitute for visiting a place in person. Dagney Looper, a third-year doctoral student in astrophysics at the University of Hawaii, recalls hosting a clutch of undecided prospective doctoral students this spring. She showed off everything from the school's world-class observing facilities to its local surf spots. Looper says that many of the students used rankings to decide where to apply. They were torn between offers from higher-ranked schools and from the University of Hawaii, which has a strong astronomy programme but less overall prestige.
"I told the prospective graduate students that getting into a good doctoral programme is important," says Looper. "But it's what you do there that really matters."