Study Quality100503538

Study Design100505821

CONTROLLED HUMAN EXPOSURE:

Studies should clearly describe the primary and any secondary objectives of the study, or specific hypotheses being tested. Study subjects should be randomly exposed without knowledge of the exposure condition (i.e., blinding of participants is ideal). Preference is given to balanced crossover (repeated measures) or parallel design studies which include control exposures (e.g., to clean filtered air). In crossover studies, a sufficient and specified time between exposure days should be provided to avoid carry over effects from prior exposure days. In parallel design studies, all arms should be matched for individual characteristics such as age, sex, race, anthropometric properties, and health status. In studies evaluating effects of disease, appropriately matched healthy controls are desired for interpretative purposes. 

TOXICOLOGY:

1) Studies should clearly describe the primary and any secondary objectives of the study, or specific hypotheses being tested. 2) Studies should include appropriately matched control exposures (e.g., to clean filtered air, time matched). 3) Studies should use methods to limit differences in baseline characteristics of control and exposure groups. 4) Studies should randomize assignment to exposure groups and where possible conceal allocation to research personnel. 5) Groups should be subjected to identical experimental procedures and conditions; animal care including housing, husbandry, etc. should be identical between groups. 6) Blinding of research personnel to study group may not be possible due to animal welfare and experimental considerations; however, differences in the monitoring or handling of animals in all groups by research personnel should be minimized.

PROMPTING QUESTIONS:

• Are there readily identifiable study weaknesses due to deficiencies in study design?

Study Population100505822

CONTROLLED HUMAN EXPOSURE:

In general, the subjects recruited into study groups should be similarly matched for age, sex, race, anthropometric properties, and health status. In studies evaluating effects of specific subject characteristics (e.g., disease, genetic polymorphism, etc.), appropriately matched healthy controls are preferred. Relevant characteristics and health status should be reported for each experimental group. Criteria for including and excluding subjects should be clearly indicated. For the examination of populations with an underlying health condition (e.g., asthma), independent, clinical assessment of the health condition is ideal, but self-report of physician diagnosis generally is considered to be reliable for respiratory and cardiovascular disease outcomes. The loss or withdrawal of recruited subjects during the course of a study should be reported. Specific rationale for excluding subject(s) from any portion of a protocol should be explained.

TOXICOLOGY:

Ideally, studies should report species, strain, substrain, genetic background, age, sex, and weight. Unless data indicate otherwise, all animal species and strains are considered appropriate for evaluating effects of ozone exposure. It is preferred that the authors test for effects in both sexes and multiple lifestages and report the result for each group separately. All animals used in a study should be accounted for, and rationale for exclusion of animals or data should be specified.

PROMPTING QUESTIONS:

• Are there readily identifiable study weaknesses due to the populations included in the study?

Exposure Assessment100505817

CONTROLLED HUMAN EXPOSURE:

Studies should have well-characterized pollutant concentration, temperature, and relative humidity and/or have measures in place to adequately control the exposure conditions. Preference is given to balanced crossover or parallel design studies which include control exposures (e.g., to clean filtered air). Study subjects should be randomly exposed without knowledge of the exposure condition. Method of exposure (e.g., chamber, facemask, etc.) should be specified and activity level of subjects during exposures should be well characterized.

TOXICOLOGY:

Studies should characterize pollutant concentration, temperature, and relative humidity and/or have measures in place to adequately control the exposure conditions. All studies should include exposure control groups (e.g., clean filtered air).

PROMPTING QUESTIONS:

• Is the exposure protocol adequately described?
• Do the study results provide data in a manner sufficient to characterize the effect of pollutant exposure itself on the endpoints of interest?

Outcome Assessment100505823

CONTROLLED HUMAN EXPOSURE AND TOXICOLOGY:

Endpoints should be assessed in the same manner for control and exposure groups (e.g., time after exposure, methods, endpoint evaluator) using valid, reliable methods. Blinding of endpoint evaluators is ideal, especially for qualitative endpoints (e.g., histopathology). For each experiment and each experimental group, including controls, precise details of all procedures carried out should be provided including how, when, and where. Time of the endpoint evaluations is a key consideration that will vary depending on endpoint evaluated. Endpoints should be assessed at time points that are appropriate for the research questions.

PROMPTING QUESTIONS:

• Do the methods clearly describe the manner and timing of endpoint measurements?
• Do the methods allow for determination of the effect of the pollutant itself on the endpoints of interest?
• Are the endpoints analyzed and described in a manner that makes it difficult to adequately understand or generalize the results?

Potential Confounding Factors100505824

CONTROLLED HUMAN EXPOSURE:

Preference is given to studies utilizing experimental and control groups that are matched for individual level characteristics (e.g., race/ethnicity, sex, body weight, smoking history, age) and time varying factors (e.g., seasonal and diurnal patterns).                          

TOXICOLOGY: 

Preference is given to studies utilizing experimental and control groups that are matched for individual level characteristics (e.g., strain, sex, body weight, litter size, food and water consumption) and time varying factors (e.g., seasonal and diurnal patterns).                             

PROMPTING QUESTIONS:

• Have factors that could potentially confound results such as those mentioned above been largely been taken into account?

Statistical Analysis100505819

CONTROLLED HUMAN EXPOSURE AND TOXICOLOGY:

Statistical methods should be clearly described and appropriate for the study design and research question (e.g., correction for multiple comparisons). Generally, statistical significance is used to evaluate the findings of controlled human exposure/animal toxicology studies. However, consistent trends are also informative. Detection of statistical significance is influenced by a variety of factors including, but not limited to, the size of the study, exposure and outcome measurement error, and statistical model specifications. Sample size is not a criterion for exclusion; ideally, the sample size should provide adequate power to detect hypothesized effects (e.g., sample sizes less than 3 are considered less informative). Because statistical tests have limitations, consideration is given to both trends in data and reproducibility of results.

PROMPTING QUESTIONS:

• Were the statistics used clearly described?
• Do they appear to be appropriate for the comparisons being made? 

---

Summary100503539

Scope100505832

• Identify the health endpoint and exposure duration of the study (e.g., short-term exposure and hospital admissions for asthma).
• Identify the section relevant ISA section (e.g., short-term exposure and respiratory health) and provide the causality determination from the previous ISA.

Strengths100505830

Study Quality

• Complete the domain specific study quality assessment before summarizing the study quality strengths. Subsequently, pull in the information from the prompting bullets if it meets the criteria for a high study quality designation (e.g., multicity studies, multiple years, large sample size, longitudinal design, etc.).

ISA Relevance

• Identification of analyses examining copollutant confounding; at-risk populations; lag structure; averaging-times; effect modification or confounding by season, weather patterns, or temperature; the shape of the C-R relationship; associations at low concentrations; results from different exposure assignment techniques; or other issues that inform our understanding of the relationship between air pollution and health effects.

Threats to Internal Validity/Risk of Bias100505833

• Complete the domain specific study quality assessment before summarizing the threats to internal validity. Subsequently, pull in information from the prompting bullets if it demonstrates risk of bias concerns. To the extent possible, describe how the bias may affect the conclusions of the study (e.g., the expected direction of the bias). 

Conclusion100505831

• “Based on study quality and applicability considerations, this study does/does not merit inclusion in the ISA [and will contribute to the overall weight of evidence therein]”.
• If the study is critically deficient, include a brief statement highlighting the weaknesses that necessitate its exclusion from the ISA.
• If there are study quality concerns, but the study merits inclusion in the ISA, include a brief statement describing how the study may address uncertainties from the previous review to advance the science.

---

Study Quality100503538

Study Design100505825

Brief description of study design should include the following elements, to the extent possible:

• Epidemiologic study design
• Identification of geographic scale (e.g., single-city, multicity study, national, etc.)
• Length of study (for long-term prospective cohort studies this generally includes separate identification of exposure measurement and follow-up time) 

Study Population100505826

Brief description of study population should include the following elements, to the extent possible:

• Identification of the study population, including age, location, and information on the source of the study population (e.g., specific cohort, administrative database, etc.)
• The number of participants and/or observations. Should be reported based on the appropriate analytic scale (e.g., average number of events per day for time-series studies)
• Description of inclusion/exclusion criteria, if applicable
• Participation rates/loss-to-follow-up, if applicable

Exposure100505820

Brief discussion of exposure assessment should include the following elements, to the extent possible:

• Identification of how exposure(s) were estimated, including information on averaging time
• Description of known errors and uncertainties associated with the exposure model or measurement
• Description of characteristics of the pollutant (generally with respect to spatial or temporal variability) that would influence applicability of the exposure assessment method
• Broad conclusion about whether errors and uncertainties in the exposure assessment would add bias or uncertainty to the effect estimate for the epidemiologic study design used, and if there is bias, in what direction (i.e., does bias related to exposure assessment change the conclusions of the study?)

Outcome Assessment100505818

Brief description of outcome assessment should include the following elements, to the extent possible:

• Identification of the outcome(s) of interest (include ICD codes or clinical, if applicable)
• Description of how the outcome(s) was measured, including information on the validity and reliability of the outcome metric, if available. Note: statements on validity and reliability can be as simple as those provided in the study quality tables (e.g., “validated questionnaires for subjective outcomes such as symptoms are regarded to be reliable, particularly when collected frequently and not subject to long recall.”)
• Description of any selective reporting of outcomes.

Confounding100505827

Brief description of confounding should include the following elements, to the extent possible:

• Identification of variables that were adjusted for in statistical models to control for potential confounders. Additionally, include any variables that the authors considered but left out of the model because they were not associated with the exposure and/or outcome.
• Potential confounders that are accounted for in the study design, if applicable
• Identification of any key potential confounders that the authors do not control for in their model or study design, and/or any variables that could result in inadequate control or notable residual confounding
• Any variables included in the model that could result in over-adjustment
• Description of model specification sensitivity analyses related to potential confounders
• Broad conclusion on the likelihood of unmeasured confounding. Note: might include brief comparison of results from different model specifications, if applicable

Statistical Analysis100505828

Brief description of statistical analysis should include the following elements, to the extent possible:

• The statistical model used to relate the exposure to the health effect.
• The modeled temporal relationship between the exposure and outcome (i.e., exposure lag[s])
• Other model details, including pooling of effect estimates and/or model specification sensitivity analyses not related to potential confounders

Alternative Analyses Conducted100505829

If applicable, brief description of additional analyses should include the following elements:

• Identification of any additional ISA relevant analyses beyond the primary exposure-health effect analysis. Analyses should be listed under relevant sub-headers (e.g., model specification, confounding, effect modification, etc.)
• Description of risk of bias concerns for relevant analyses, as pertaining to any of the domain-specific criteria, that have not already been captured above

---