RATING GUIDANCE: Once the evaluation domains have been classified, these ratings will be combined to reach an overall study confidence classification of High, Medium, Low, or Uninformative. In case of multiple endpoints or group of endpoints, a rationale for each one should be given separately.

This classification will be based on the classifications in the evaluation domains (reporting, methodology), and will include consideration of the likely impact of the noted deficiencies in bias and sensitivity on the results. In general, individual criteria in the reporting quality domain are regarded as less important compared to those in the methodological quality domain and this should be taken into account when arriving at the final confidence rating.

Studies with a high number of critical deficiencies, especially on criteria that are deemed very important (e.g. failure to report the chemical name, route of exposure, concentration used) will be classified as Uninformative.

Other classifications will generally follow a sorting such that High Confidence studies would have the highest evaluation ('Good') for all or most domains; Low Confidence studies would have a 'Deficient' evaluation a high number of criteria (unless the impact of the particular limitation(s) is judged to be unlikely to be severe), and Medium Confidence studies are in between these groups.

Example ratings:

High confidence:

Reproductive and developmental effects other than behavior [High Confidence]: The study was well-designed for the evaluation reproductive and developmental toxicity induced by chemical exposure. The study applied established approaches, recommendations, and best practices, and employed an appropriate exposure design for these endpoints. Evidence was presented clearly and transparently.

Behavioral measures [Low Confidence]: The cursory cage-side observations of motor-related activity are considered to be insensitive methods for detecting behavioral effects, with a strong bias towards the null.

Medium confidence:

Developmental effects [all Medium Confidence]: The study was adequately designed for the evaluation of developmental toxicity. Although the authors failed to describe randomized allocation of animals to exposure groups and some concerns were raised regarding the sensitivity (i.e., timing) and sample sizes (i.e., n=6 litters/group) used for the evaluation of potential effects on male reproductive system development with gestational exposure, these limitations are expected to have a minimal impact on the results.

Lower confidence

Developmental effects [Low Confidence]: Substantial concerns were raised regarding quantitative analyses without addressing potential litter effects. Other significant limitations included incomplete data presentation (sample sizes for outcome assessment were unclear; no information on maternal toxicity was provided), and methods for selection of animals for outcome assessment.

Histopathology [Medium Confidence]: The study authors did not report information on the severity of histological effects for which this is routinely provided. The authors also failed to describe use of methods to reduce potential observational bias.

Sperm Measures [Uninformative]: Issues were identified with the methods used to prepare samples for analysis, which are likely to introduce artifacts. Concerns were also raised regarding results presentation (i.e., lack of group variability), missing information on sample sizes and loss of animals, and a lack of information on the timing of these evaluations. Taken together, the evaluation of this endpoint was considered uninformative.

Uninformative

Example 1: Critical information was not reported. Specifically, the study authors did not report the duration of the exposure or the results (qualitative or quantitative). Given this critical deficiency, the other domains were not evaluated.

Example 2: Concerns were raised over the lack of information on test animal strain and allocation, and chemical source/purity. The lack of information on blinding or other methods to reduce observational blinding is also of significant concern for the endpoints of interest (i.e., follicle counts, ova counts, and evaluation of estrous cyclicity). Finally, concerns were also raised over the apparent self-plagiarism in similar chromium studies published in 1996 by this group of authors. Taken together, this combination of limitations resulted in an interpretation that the results were unreliable.

Scanlan, 2015

Study evaluation