Systematic reviews of randomized controlled trials in literacy research : methodological challenges

Introduction: In this item a 'tertiary' review of systematic reviews in literacy learning
is presented. It explores the methodological quality of the identified systematic reviews
and identifies the primary data that are used for the in-depth methodological work in
Item 3 on the two main threats to the validity of systematic reviews: publication bias
and design bias.
Background: Recent governments in the UK have introduced a number of initiatives
aimed at improving the literacy levels of children. It is important, therefore, that policy
and practice are informed by the most rigorous available evidence, particularly for
questions of effectiveness in literacy learning. It is also important that this evidence is
subjected to rigorous critical scrutiny.
Methods: Systematic reviews undertaken in the field of literacy learning in English in
the years between 1983 and 2003 were searched for, located and quality assessed. The
scope of the review was limited to systematic reviews of experimental research
evaluating literacy interventions with quantifiable literacy outcome measures in English
as a first (not second or additional) language and focusing on children and young people
in school settings up to the age of 18.
Results: A total of 14 systematic reviews containing meta-analyses and meeting all the
inclusion criteria were included in the tertiary review. The following data were
extracted from the reviews: literacy interventions, outcomes evaluated and effect sizes.
The quality of the reviews was examined using an adaptation of the QUORUM
statement. Overall the quality of the meta-analyses included in this tertiary review was
good. When examining the effect sizes of randomized controlled trials (RCTs) and
controlled trials (CTs) separately there was no clear pattern as to whether the RCTs
produced a larger or smaller effect size than the CTs.
Discussion: Overall the quality of the meta-analyses included in this tertiary review was
good. The QUORUM checklist seemed to perform well for the appraisal of educational
meta-analyses. All the reviews clearly stated their research question, and their methods
of searching for and selecting included studies. Most studies described their data
extraction and used some form of quality assessment of included studies. On the other
hand, some reviews did have notable methodological weaknesses. Six of the 14 studies
did not make an assessment of publication bias, which is potentially a major threat to
the validity of any systematic review. In addition, six studies did not provide evidence
for reviewer agreement when synthesising the data. There is, therefore, some room for
improvement in the methodological quality of systematic reviews in literacy learning.
Conclusions: A number of reviews in this tertiary review are judged to be of
sufficiently high quality to provide reliable evidence for the effectiveness of literacy
interventions.