Wait, Federal Agency Charge Questions To Advisory Committees Can Demonstrate Loss of Scientific Integrity?

One way to lose scientific integrity is to engage in flawed peer reviews. And EPA did just that with the National Academies of Science, Engineering, and Medicine's committee charged with reviewing the IRIS Program's Formaldehyde assessment.

Yes — US Government Agencies can lose scientific integrity through their charge questions. In fact, they can also lose scientific integrity through undue influence of the peer review process, limiting the scope of the peer review, and through manipulation of the charge questions.

I argue that’s exactly what happened with the US EPA’s Charge to the National Academy of Science, Engineering, and Medicine’s Peer Review of the IRIS Toxicological Review of Formaldehyde.

But first, before we get into all of these details, let’s take a step back. Let’s talk about what Scientific Integrity is.

What is Scientific Integrity?

Scientific Integrity is the idea that we can trust science when it was done properly. And it’s not hard to do science properly. It’s also not hard to do science improperly.

And Scientific Integrity is the bedrock, the foundation, of how many democratic nations build their policies — at least in theory. In the United States, Congress has made their expectation well known that chemical regulations are based on high quality science in their report on the Lautenberg Amendment to the Toxic Substances Control Act (TSCA).

The Biden-Harris White House made Scientific Integrity a big deal in the early days of their administration. They created the Scientific Integrity Fast-Track Action Committee, which in turn created the report Protecting the Integrity of Government Science.

Scientific Integrity and Federal Advisory Committees: A Story of Formaldehyde

Now, let’s turn our attention to Federal Advisory Committee Act (FACA) Committees. These committees, especially the science-based ones, provide scientific advice to Federal agencies, especially regulatory agencies, like the US Environmental Protection Agency. The science-based FACA committees, like the Toxic Substances Control Act (TSCA) Science Advisory Committee on Chemicals (SACC) have scientists on them.

The Agency will ask the FACA committee (e.g., SACC) to respond to a series of charge questions. Let’s take the example of the National Academy of Science, Engineering, and Medicine (NASEM) review of the US EPA’s Integrated Risk Information System (IRIS) Assessment of Formaldehyde. In this case, the US EPA provided the following charge to the NASEM peer review panel:

An ad hoc committee under the auspices of the National Academies of Sciences, Engineering, and Medicine will conduct a scientific review of EPA’s draft document referred to as the Integrated Risk Information System (IRIS) Toxicological Review of Formaldehyde, plus appendices. The committee will assess whether EPA’s draft document adequately and transparently evaluated the scientific literature, used appropriate methods to synthesize the current state-of-the science, and presented conclusions regarding the hazard identification analysis and dose-response analysis of formaldehyde that are supported by the scientific evidence. The committee will not conduct its own hazard assessment of formaldehyde, nor will the committee address the broader aspects of the IRIS Program.
Recommendations about the IRIS assessment will be prioritized as follows:
Tier 1: recommended revisions that are important for EPA to consider and address to improve critical scientific concepts, issues, or narrative in the assessment.
Tier 2: suggested revisions that are encouraged to strengthen or clarify the scientific concepts, issues, or narrative in the assessment but are not critical. Other factors, such as agency practices and resources, might need to be considered by EPA before undertaking the revisions.
Tier 3: considerations that might inform future evaluations of key science issues or inform development of future assessments.

https://nap.nationalacademies.org/read/27153/chapter/3#17

Okay, I’m going to assume you read that. If you didn’t, go back and actually read it. It’s important to understand what’s going on here, and I don’t want to spoil it for you.

For the uninitiated, that looks kind of straightforward, right? This sentence looks innocuous enough: “The committee will assess whether EPA’s draft document adequately and transparently evaluated the scientific literature, used appropriate methods to synthesize the current state-of-the science, and presented conclusions regarding the hazard identification analysis and dose-response analysis of formaldehyde that are supported by the scientific evidence.”

But what about the next sentence? “The committee will not conduct its own hazard assessment of formaldehyde, nor will the committee address the broader aspects of the IRIS Program.”

Taken together, what the US EPA is saying to the NASEM peer reviewers, is that US EPA is giving NASEM a very narrow charge. Let’s break this down:

  • “adequately and transparently evaluated the scientific literature” — US EPA is asking NASEM to assess if US EPA did an adequate job of evaluating the scientific literature. The problem here is that US EPA never defines “evaluate”. So it is unclear what US EPA is actually asking NASEM to do.
  • “used appropriate methods to synthesize the current state-of-the-science” — US EPA is asking NASEM to assess if appropriate methods were used in synthesizing the science. This means NASEM needs to weigh in on whether or not US EPA’s methods for combining the scientific information together to draw conclusions were appropriate.
  • “presented conclusions regarding the hazard identification analysis and dose-response analysis of formaldehyde that are supported by the scientific evidence” — here US EPA is asking NASEM to assess if the conclusions are supported by the scientific evidence.
  • But in doing all of these things, US EPA does NOT want NASEM to perform its own hazard assessment of formaldehyde.

That’s not a great charge.

But here’s the rub — a Federal Advisory Committee or NASEM Peer Review Committee can only do what the sponsoring Federal Agency charges them with.

Don’t believe me? Look at what NASEM says in their own report, on page 16:

The committee also was not charged with commenting on other interpretations of scientific information relevant to the hazards and risks of formaldehyde, or with reviewing alternative opinions of EPA’s assessment.

Review of EPA’s 2022 Draft Formaldehyde Assessment (2023); https://nap.nationalacademies.org/read/27153/chapter/3

What gives? The US EPA did not charge NASEM with identifying alternative interpretations of the scientific studies they relied upon? That’s right — and the NASEM Committee points this out in their report. Because US EPA did not charge them to look at alternative interpretations, they cannot.

What else didn’t the US EPA do? US EPA did not charge NASEM with reviewing alternative opinions of the US EPA’s formaldehyde assessment.

US EPA also forbade NASEM from creating an alternative formaldehyde assessment. Look carefully again at that last point in the main part of the charge. Recall that US EPA said, “The committee will not conduct its own hazard assessment of formaldehyde…”

Did US EPA Violate Their Own Peer Review Policy?

According to the US EPA’s Peer Review Handbook:

Peer review is an in-depth assessment of the assumptions, calculations, extrapolations, alternate interpretations, methodology, acceptance criteria and conclusions pertaining to the scientific or technical work product…

US EPA Peer Review Handbook, 4th Edition

Wait, wait. So the US EPA’s Peer Review Handbook says that peer review is supposed to be looking for alternate interpretations, right? But the US EPA told NASEM, essentially that [paraphrasing] ‘Thou shalt not make alternative interpretations of the science we relied upon in our assessment.’ They said this by 1) not explicitly charging NASEM with the ability to make alternative interpretations of the science that US EPA relied upon, and 2) when US EPA stated that “The committee will not conduct its own hazard assessment of formaldehyde.”

So, the answer is:

Yes, the US EPA violated their own Peer Review policy stipulated in the Peer Review Handbook, when US EPA failed to charge NASEM with the ability to make alternative interpretations of the science that US EPA relied upon, and when US EPA stated that “The committee will not conduct its own hazard assessment of formaldehyde.”

Got it. But how is this evidence of a violation of scientific integrity policies with respect to how US EPA is handling formaldehyde?

Flawed Peer Review Leads to Scientific Integrity Problems

We are going back to the Biden-Harris White House, and the Scientific Integrity Fast-Track Action Committee’s report Protecting the Integrity of Government Science. Turn to page 49 (PDF page 63). Look at table C-1. Go down to row 3: Flawed Review. It says:

Undue influence or inadequate technical or peer review…limiting scope of a review or peer review charge.

White House Office of Science and Technology Policy, Scientific Integrity Fast-Track Action Committee Report, Protecting the Integrity of Government Science, p49.

Okay, so do we have any of these?

Undue Influence? Oh yeah, lots of undue influence. US EPA told NASEM to not consider alternative interpretations to US EPA’s interpretations of scientific studies that were used. US EPA told NASEM to not conduct its own alternative hazard assessment of formaldehyde. NASEM was not allowed to construct alternative interpretations of US EPA’s assessment or the science it relied upon.

Limiting scope of a peer review charge? Oh yeah, lots of this, too. All of the same points as before.

Altogether, is this an inadequate peer review? Oh yeah! I sound like Kool-Aid man in the 80s (lots of oh yeahs!).

Why Is Open, Independent, and Unrestricted Peer Review Critical for Science Used to Inform Public Policy?

The purpose of peer review in reviewing the science that will become the basis of public policy is simple: it ensures that all voices are able to critically analyze, point out the flaws, point out the strengths and weaknesses, identify the knowledge gaps, and characterize the uncertainty in the science. It helps the government make better decisions by avoiding the biases inherent in its employees. It provides sunshine to help mitigate fraud, waste, and abuse.

As the Scientific Integrity Fast-Track Action Committee’s report Protecting the Integrity of Government Science says:

The scientific process benefits from a culture of rigorous scientific debate that guards against inadequate methods and analysis that welcomes open discussion across potentially disparate perspectives.

White House Office of Science and Technology Policy, Scientific Integrity Fast-Track Action Committee Report, Protecting the Integrity of Government Science, p23.

The problem is that Agencies have figured out how to game the peer review system. By relying upon FACA Advisory Committees and Peer Review panels, the Agencies have figured out how to manipulate the charge questions they ask to get the responses they want.

Also, Agencies are the ones who appoint FACA Advisory Committee members. Agencies can also be involved in assigning their own peer reviewers at NASEM. This means that Agencies have the opportunity to ensure individuals who are more likely to share the same biases as the authors of their assessments.

But what about FACA’s rules on viewpoint balance? When it comes to US EPA FACA committees, rarely have I seen true viewpoint balance. There are typically far fewer scientists that believe in thresholds (the idea that there are doses below which chemicals are not toxic) than there are public health scientists that believe there are no thresholds (in other words, all chemicals are toxic at all exposure levels).

So What Is the Prescription to Fix the Peer Review Issues at EPA?

Simple: Congress needs to force the US EPA to only use fully open and independent peer review. What this means is that when US EPA uses FACA to charge a Federal Advisory Committee, or a NASEM Peer Review Committee, the US EPA’s charge will be simplified to the following language:

“The Committee is charged to perform a full and independent peer review. The Committee is specifically charged with adjudicating any valid alternative interpretations of the science relied upon in the assessment, and to identify any additional studies that were not assessed by the Agency. In addition, the Committee is charged with identifying any differences that the Committee may have with any scientific decision made by the Agency, and to identify the impacts of these differences. The Committee is charged with adjudicating these differences and offering scientifically viable alternatives to the Agency’s assessment. Nothing in this charge should be seen as constraining the Committee from performing a full and independent peer review. The Agency would especially be interested in the Committee’s thoughts on the following items: [list those items].”

How can Congress force the US EPA to use this type of simplified charge language?

Simple: Congress can put this into an appropriation bill. Or Congress could create a new bill that specifically directs the US EPA to do this. Alternatively, Congress could amend the Toxic Substances Control Act, the Community Right to Know Act, CERCLA, the Clean Air Act, the Safe Drinking Water Act, the Clean Water Act, and the Environmental Research, Development, and Demonstration Authorization Act (Section 7 might be a good starting point). There are a lot of ways Congress could accomplish this.

Or, another option is for the White House to execute an Executive Order to direct the Administrator of the US EPA to adopt this simplified charge language as a universal template for all science-related charges to FACA and NASEM Peer Review Committees when they are reviewing assessment products.

Seriously, Though, EPA’s Not Okay and Needs Help

Unfortunately, EPA is demonstrating lately that it either is ignorant about the flaws in its approach to peer review, or it simply does not care. I have not looked at enough of the US EPA’s FACA committees to say how wide-spread this issue is.

What I do know is that there is evidence that US EPA’s charge to the NASEM Peer Review Committee reviewing the formaldehyde IRIS assessment demonstrates a lack of scientific integrity. And I also know that because of the importance of the IRIS program to US EPA’s program offices, that this is simply unacceptable.

IRIS needs help. US EPA needs help. Congress is not full of scientists, and it’s not filled with scientific integrity experts. If we don’t hold the US EPA and IRIS accountable, who will? Should we trust that the Agency can police itself? That doesn’t seem likely.

Lyle D. Burgoon, Ph.D., ATS
Lyle D. Burgoon, Ph.D., ATShttps://www.raptorpharmtox.com
Dr. Burgoon is a pharmacologist/toxicologist, biostatistician, ethicist and risk assessor. Dr. Burgoon writes on chemical safety, biostatistics, biosecurity, sustainability, and scientific ethics. He is the President and CEO of Raptor Pharm & Tox, Ltd, a consulting firm.

Latest articles

Related articles

Leave a reply

Please enter your comment!
Please enter your name here