Downloaded from
`
`http://fampra.oxfordjournals.org/
`
` by Margaret Yamin on June 22, 2016
`
`Family Practice
`O Oxford University Press 1991
`
`Vol. 8, No. 4
`Printed in Great Britain
`
`When Plagiarism Becomes Research
`
`DAVID KATERNDAHL
`
`Katerndahl D. When plagiarism becomes research. Family Practice 1991; 8: 382-383.
`There are three possible levels of analysis in clinical research. Primary analysis deals with the original
`analysis of research study data. Secondary analysis is a reanalysis of the original data, either to address the
`original question through better techniques or to address a new question using old data. Meta-analysis is a
`statistical analysis of many studies done to summarize a body of literature. Meta-analysis is particularly
`helpful in an area in which original research studies have produced conflicting results because it enables
`analysis of the impact of study characteristics upon the end result.
`The family practitioner as a consumer of research needs to become familiar with the technique of meta-
`analysis because it is appearing with increasing frequency in the medical literature. Still somewhat con-
`troversial, meta-analysis requires a rigorous approach to ensure its validity: this editorial is written to assist
`the family practitioner in an understanding of the meta-analytic technique and point out important features
`that need to appear in any published meta-analysis.
`
`FEATURES OF META-ANALYSIS
`Meta-analysis involves the pooling of results across
`studies. This differs from data pooling in which indi-
`vidual subjects are pooled. To use data pooling appro-
`priately the studies from which the subjects are
`gathered must be sufficiently similar to warrant such
`pooling. Meta-analysis, on the other hand, does not
`rely on similarities in the study setting or design. The
`unit of analysis for meta-analysis is the individual
`study and the differences in study design are utilized to
`investigate the impact of study characteristics upon
`study outcome.
`Meta-analyses have several essential features. The
`location of studies to be included is essential because
`meta-analysis seeks to review the entire body of litera-
`ture. Inclusion and exclusion criteria for studies must
`be clearly specified in a meta-analysis. Study charac-
`teristics are described in detail for each study and the
`study outcomes are quantified through the use of an
`effect size. The effect size can take several different
`forms, but is used to measure the overall difference
`between experimental and control groups in a standar-
`dized fashion.
`The identification of studies to be included is a
`critical step in a meta-analysis. Published studies
`should be identified through reviews of bibliographies,
`books, and computer searches. However, meta-analysis
`needs to include as many unpublished studies as
`possible. Publication bias—the fact that unpublished
`studies have a smaller effect size than published
`studies—is evident in several areas of research. Conse-
`
`The University of Texas, Health Science Center at San Antonio, 7703
`Floyd Curl Drive, San Antonio, TX 78284-7795, USA.
`
`382
`
`quently, if publication bias is to be eliminated as a
`problem, unpublished studies must be located. Clinical
`reports, dissertations, and conference reports are all
`sources for the identification of unpublished studies.
`Once potential studies are identified, criteria for
`inclusion and exclusion must be applied. Although
`some meta-analysts believe that all studies should be
`included in any meta-analysis, a practical approach is
`to require every included study to clearly address the
`study question and include an adequate documentation
`of exposure as well as an appropriate control group.
`Limiting your meta-analysis to only high quality
`studies may drastically reduce the number of studies
`included and restrict its ability to identify the impact
`which study parameters have upon outcome.
`Because an essential part of meta-analysis is to iden-
`tify the role of study design characteristics upon out-
`come, a detailed description of study characteristics
`should be encoded in any meta-analysis. Such features
`as study site, sampling techniques, sample demo-
`graphics, statistical power, and publication charac-
`teristics should all be included.
`The final key step is the quantification of study out-
`come, as previously mentioned. Classically, the effect
`size index is measured as the standardized difference
`between the means of the experimental and control
`groups. The standardized difference can be obtained
`by dividing the difference by the pooled deviation for
`the entire sample. Other measures of effect size can
`include the natural logarithm of the relative risk, /
`statistics or correlation coefficients. When more than
`one effect size can be derived in any study, the selec-
`tion of a key or 'best' effect size is important. In those
`situations where any selection of a key effect size
`would be arbitrary, the use of a Jack-knife technique
`
`Ex. 2046-0001
`
`

`
`383
`
`WHEN PLAGIARISM BECOMES RESEARCH
`INTERPRETATION OF META-ANALYSIS
`to select an overall representation for the study is
`The statistical interpretation of any meta-analysis is
`possible.
`relatively straight forward if the analyses described
`above are used. For the practitioner, an empirical inter-
`pretation may be more valuable.
`Usually one of three situations exist in any meta-
`analysis. The interpretation differs, depending on
`which of the three situations apply. If most studies in a
`meta-analysis point in the same direction you should
`probably accept that conclusion. In this case, the large
`high quality studies will give the reader a rough
`estimate of the effects of mass implementation of
`whatever treatment is being investigated.
`A second possible outcome is that the effects differ
`between high quality studies and poor studies. In this
`situation, our interpretation should lean in favour of
`results of the high quality studies.
`Finally, it is possible that the high quality studies
`have found inconsistent or conflicting results. In this
`case, you must look at the analysis of the relationship
`between study characteristics and outcome. If our
`multivariate analysis suggests that the better the study
`design, the higher is the effect size then that suggests
`that there is indeed a relationship. On the other hand,
`if the effect size is larger in situations where study
`features are more poorly controlled, then that suggests
`that there may not be a significant relationship and
`that any observed relationship may be due to artifact.
`
`ANALYSIS IN META-ANALYSIS
`Perhaps the first step in the analysis phase of meta-
`analysis is the description of the studies themselves and
`their quality. Presentation of frequencies and means
`for the individual study characteristics is very valuable
`in the description of the overall quality of the literature
`upon which the meta-analysis is based. Because some
`variables involve a subjective judgment on the part of
`the meta-analyst, presentation of interrater agreement
`on these variables is important.
`Summarization of the overall findings of the litera-
`ture can be done in one of three ways. The use of
`voting methods in which the number of 'significant'
`studies is tabulated is notoriously unreliable and a
`poor way of summarizing the literature. The effect size
`can be summarized by calculating the mean key effect
`size across the studies with the presentation of con-
`fidence intervals. This is best done using a weighted
`effect size which adjusts for differences in sample size
`from one study to another. The final method of com-
`bining outcomes is by combining P values of the indi-
`vidual effect sizes in the studies. Although this can be
`done by one of several methods, combining P values
`only gives you a sense of the overall level of signi-
`ficance of the literature rather than the overall effect
`size of the literature.
`A final step in summarization should include an
`assessment of publication bias. This can be done
`graphically through the use of a funnel graph or can be
`done through the calculation of a fail-safe n—the
`number of unpublished studies having no effect that
`are needed to bring the mean effect size to a level of
`non-significance. Every meta-analysis should include
`some assessment of publication bias.
`A final analysis phase in any meta-analysis should be
`an assessment of the relationship between study charac-
`teristics and outcome. This should include not only
`univariate analyses, but also an overall multivariate
`analysis, using techniques such as multiple linear
`regression, logistic regression, etc. Only through these
`analyses can an understanding of the source of conflict
`in a body of literature be truly determined.
`
`Downloaded from
`
`http://fampra.oxfordjournals.org/
`
` by Margaret Yamin on June 22, 2016
`
`CONCLUSION
`The technique of meta-analysis is extremely valuable in
`summarizing a large body of research as well as in
`explaining conflicting results. Many issues of great
`concern to family practitioners could be very appro-
`priately addressed through the use of meta-analysis,
`such as the efficacy of various weight-loss techniques,
`smoking reduction techniques, and therapies in sub-
`stance abuse. The medical literature, in general, and
`the family practice literature, in particular, are begin-
`ning to include meta-analysis in their publications. It is
`extremely important for the family practitioner to
`understand what meta-analysis is, and be able to read
`and interpret a meta-analysis.
`
`Ex. 2046-0002