Gatekeepers of Science

The work of scientific research depends absolutely on a reliable system of verification. For newly announced findings to be trusted, researchers (and the public, for that matter) need some assurance that proper procedures have been followed, and that the work has been checked and rechecked by people who know what they’re doing.

For a great many years, the community of scientists has relied upon the system of “peer review” to perform this function. When a new paper is proposed for publication in a journal, the editors have it reviewed anonymously by a number of prominent researchers in the field, to be sure that everything is in order. All the major scientific journals, including those that are highly specialized in one field or another, offer their readers the imprimatur of peer review, and scientists simply do not take seriously any findings that have not been published in a peer-reviewed journal.

In recent months, however, two rather different controversies have arisen over peer review — one in Britain and one in the United States.

Over the summer, Britain’s Royal Society — the ancient and esteemed national scientific academy — opened a major public inquiry into the ways in which research results are published in the U.K. Sir Patrick Bateson, who is chairing the inquiry, said that peer review “has been criticized for being too secretive, conducted behind closed doors, and assessed by anonymous referees,” and for being used by the scientific establishment “to prevent unorthodox ideas, methods, and views, regardless of their merit, from being made public.” Others worry that the system is not working properly to assure the reliability of studies, particularly when those studies are funded by the government, by corporations, or by interest groups with a political agenda. Such conflicts of interest are of increasing concern, particularly as more science, even academic science, is done under the aegis of companies seeking patents and marketability.

The panel took public submissions and testimony in the summer and early fall, and is at work on a report, which may take the form of a “best practices” guide for journals and researchers.

Meanwhile, in the United States, a new proposal by the White House budget office has raised concerns about the misuse of peer review for political purposes.

In the fall, the Office of Management and Budget (OMB) proposed that any new regulation that relies on scientific data — such as data about pollution, the health effects of consumer products, deforestation, work-related injuries, and countless other matters — be subject to a centralized peer review process overseen by OMB. The Bush administration argues that the new process would ensure that regulations rely on good science, standardizing the process by which government science is reviewed.

Under the current system, which has been in place for decades, each individual federal agency establishes its own process of scientific review, which typically involves inviting outside experts to review the data used in making decisions. The proposed OMB rules would largely take over this process, by establishing rules about who may be hired to review data, and by exerting final authority about whether a particular peer review process has been sufficiently rigorous.

The new rules have drawn fire from a variety of sources, including the nation’s largest and most prominent scientific organizations. One might expect scientists to approve of an extension of the peer-review “gold standard,” but the OMB proposal raises two concerns for the critics.

First, many scientists and special interest groups have argued that the new process would allow the executive branch to exert direct political pressure to suppress scientific data that might lead to politically undesirable policies. By giving OMB — a White House office — the final say in peer review, the rules could give the White House control over the government’s version of science. Indeed, the rules would explicitly allow OMB to impose an even higher standard of scrutiny over areas of science that affect “an administration policy priority.” Critics, including the American Association for the Advancement of Science, worry that this would allow the White House to protect corporate polluters or other favored industry players when the scientific facts show that they need to be regulated.

Second, some worry that the new rules would slow down the regulatory process, making it difficult for regulators to react to changing circumstances. Joan Claybrook, who heads the liberal special interest group Public Citizen, told the Washington Post that the new rules were a form of “paralysis by analysis,” which might obstruct needed new regulation under the pretense of scientific integrity, and might even delay federal response in cases of health or environmental emergencies. The proposed rules would give the OMB final control over emergency declarations by agencies like the Food and Drug Administration and the Environmental Protection Agency. In a letter to the White House in early January, a group of 20 former top regulators — Republicans and Democrats — complained that such an exertion of control “could damage the federal system for protecting public health and the environment.”

The Bush administration contends that the proposed system would simply be a way to bulk up peer review procedures, which in many federal agencies are insufficient or too weak. “Even agencies that have peer review policies have not been found to implement them consistently,” an unnamed administration official told the St. Louis Post-Dispatch. But to this point, the administration has offered few direct answers to the particular contentions of the critics.

It seems clear from these examples on both sides of the pond that peer review remains a useful — if imperfect — part of the scientific process, and one that needs to be applied very differently in different circumstances. A federal agency, after all, has few “peers” who might review its work, and scientific advances may not always benefit from a secretive system that grants established scientists a veto over new research.