Open access
`
`Research
`
`Role of supplementary material in
`biomedical journal articles: surveys of
`authors, reviewers and readers
`
`Amy Price,1 Sara Schroter,2 Mike Clarke,3,4 Helen McAneney4
`
`To cite: Price A, Schroter S,
`Clarke M, et al. Role of
`supplementary material in
`biomedical journal articles:
`surveys of authors, reviewers
`and readers. BMJ Open
`2018;8:e021753. doi:10.1136/
`bmjopen-2018-021753
`
` ► Prepublication history and
`additional materials for this
`paper are available online. To
`view these files, please visit the
`journal online ( http:// dx. doi.
`org/ 10. 1136/ bmjopen- 2018-
`021753).
`
`Received 16 January 2018
`Revised 5 June 2018
`Accepted 21 August 2018
`
`© Author(s) (or their
`employer(s)) 2018. Re-use
`permitted under CC BY-NC. No
`commercial re-use. See rights
`and permissions. Published by
`BMJ.
`1Department of Continuing
`Education, University of Oxford,
`Oxford, UK
`2The BMJ, Oxford, UK
`3Northern Ireland Clinical Trials
`Unit, Belfast Health and Social
`Care Trust, Belfast, UK
`4Northern Ireland Methodology
`Hub, Centre for Public Health,
`Queen's University Belfast,
`Belfast, UK
`
`Correspondence to
`Dr Helen McAneney;
` h. mcaneney@ qub. ac. uk
`
`AbstrACt
`Objective Many journals permit authors to submit
`supplementary material for publication alongside the
`article. We explore the value, use and role of this material
`in biomedical journal articles from the perspectives of
`authors, peer reviewers and readers.
`Design and setting We conducted online surveys
`(November–December 2016) of corresponding authors and
`peer reviewers at 17 BMJ Publishing Group journals in a
`range of specialities.
`Participants Participants were asked to respond to one of
`three surveys: as authors, peer reviewers or readers.
`results We received 2872/20340 (14%) responses:
`authors 819/6892 (12%), peer reviewers 1142/6682 (17%)
`and readers 911/6766 (14%). Most authors submitted
`(711/819, 87%) and 80% (724/911) of readers reported
`reading supplementary material with their last article,
`while 95% (1086/1142) of reviewers reported seeing these
`materials sometimes. Additional data tables were the most
`common supplementary material reported (authors: 74%;
`reviewers: 89%; readers: 67%). A majority in each group
`indicated additional tables were most useful to readers
`(61%–77%); 20%–36% and 3%–4% indicated they
`were most useful to peer reviewers and journal editors,
`respectively. Checklists and reporting guidelines showed
`the opposite: higher proportions of each group regarded
`these as most useful to journal editors. All three groups
`favoured the publication of additional tables and figures
`on the journal’s website (80%–83%), with <4% of each
`group responding that these do not need to be available.
`Approximately one-fifth (16%–23%) responded that raw
`study data should be available on the journal’s website,
`while 24%–33% said that these materials should not be
`made available anywhere.
`Conclusions Authors, peer reviewers and readers agree
`that supplementary materials are useful. Supplementary
`tables and figures were favoured over reporting checklists
`or raw data for reading but not for study replication.
`Journals should consider the roles, resource costs and
`strategic placement of supplementary materials to ensure
`optimal usage and minimise waste.
`trial registration number NCT02961036.
`
`bACkgrOunD
`Many journals allow or require authors to
`submit supplementary material along with
`their manuscript. These materials might
`help in deciding about the publication of
`
`strengths and limitations of this study
`
` ► Our large sample from a diverse group of active
`international authors and reviewers from 17 differ-
`ent journals provide evidence for stakeholder views
`on supplementary materials within the biomedical
`literature.
` ► The response rate is comparable to response rates
`for other electronic surveys of researchers.
` ► Participants were asked to respond in the as-
`signed role/perspective of a reader, peer reviewer
`or author, although these are not mutually exclusive
`categories, as academics often engage in all three
`activities.
`
`the article (such as completed checklists
`for reporting guidelines) or provide addi-
`tional information for readers who wish to
`delve deeper into the findings, replicate the
`research or use it for secondary analysis. The
`materials might also help improve access in
`the context of initiatives such as the Find-
`ability, Accessibility, Interoperability and
`Reusability Data Principles for the automatic
`finding and use of scientific data,1 and the
`wish to facilitate automation in the systematic
`review process.2
`The volume of supplementary materials is
`accelerating in step with research complexity
`and multidisciplinary alliances. For example,
`Schriger et al show the percentage of articles
`containing supplementary materials increasing
`from 7% in 2003 to 25% in 2009 with web-only
`supplementary materials doubling in the same
`time period.3 Scientific journals report chal-
`lenges in keeping up, citing reviewer fatigue,
`publishing delays, bloated publishing repos-
`itories and confusion, as it is not unusual for
`articles that occupy 5–7 pages in the journal to
`present with over 140 pages of supplementary
`data or for systematic reviews or trial reports to
`include several hundred pages of information
`that would be needed to replicate, but not to
`report the findings of the research.4–7 Supple-
`mentary materials might provide additional
`
`1
`
`Price A, et al. BMJ Open 2018;8:e021753. doi:10.1136/bmjopen-2018-021753
`
`AAA, Ex. 2001
`Evergreen Theragnostics, Inc. v. AAA SA
`PGR2021-0003
`Page 1 of 7
`
`

`

`Open access
`
`results from a study or the detail needed to replicate the
`methods or present formulas, statistical models, interven-
`tion details or algorithms. Some journals refuse the mate-
`rials as excessive, while others allow ‘reasonable use’ which
`each journal defines individually.4–7 This is set within the
`backdrop of an increasing demand for research transpar-
`ency through the sharing of all findings and corresponding
`data.8 Although standards for supplementary materials were
`suggested in 2012 by the National Information Standards
`Organisation and the National Federation of Advanced
`Information Services,9 the concerns of medical journals
`were not specifically considered and any policy adopted by
`medical journal editors will have implications for readers,
`editors, reviewers and the general public.
`Clinicians and researchers struggle to keep up with
`reading the literature. Nearly a decade ago, Bastian et al
`reported the publication of 75 trials and 11 systematic
`reviews per day and asked ‘how will we ever keep up?’10 The
`numbers have continued to increase since then and the
`challenges have been compounded by the burgeoning
`supplementary material and problems with incompatible
`file systems, bandwidth restrictions and broken weblinks.11
`The increasing volume of supplementary materials
`submitted to journals puts more pressure on journal
`editors and peer reviewers to retrieve relevant informa-
`tion from multiple sources.7 Schaffer and Jackson12 make
`recommendations on how access to supplementary mate-
`rial can be improved. There is concern that the exces-
`sive volume of supplementary materials can influence
`decisions made during peer review and skew the integ-
`rity of the scientific record.6 A recent study of research
`manuscripts submitted to JAMA, JAMA Internal Medicine
`and JAMA Pediatrics found that manuscripts with supple-
`ments were more likely to be peer reviewed and accepted
`than those without supplements.13 The requirements
`and practices of journals around supplementary mate-
`rials vary12 14 and journals’ expectations of peer reviewers
`in terms of supplementary material are often not made
`clear in guidance to reviewers.6 For example, some jour-
`nals explicitly state that supplementary material will not
`be peer reviewed, while others only mention that it will
`not be typeset. This variety of approaches forces authors,
`reviewers and readers to place different degrees of priori-
`tisation and importance on supplementary material when
`including, reading or using them.
`The use of supplementary materials during and after
`submission and publication is patchy, and the perceived
`value to stakeholders of the work involved in producing,
`assessing and using them is unclear.13 15 We conducted a
`survey to help resolve these uncertainties and to inves-
`tigate the role of supplementary material in biomedical
`journal articles from the perspective of authors, peer
`reviewers and readers.
`
`MethODs
`This survey is registered at ClinicalTrials. gov.
`
`2
`
`sampling
`Journal sampling
`Participants were drawn from a sample of 17 of BMJ
`Publishing Group’s biomedical journals (online supple-
`mentary appendix 1). Journals varied in size and impact
`factor but each has a website and publishes supplemen-
`tary material.
`
`Participant sampling
`One author (SS) downloaded contact details of all corre-
`sponding authors who submitted a full-length original
`research submission to one of the 17 journals in 2013
`and all peer reviewers who had completed a review of a
`research submission for one of the journals in 2014 from
`the journal manuscript tracking systems. She used Micro-
`soft Excel to remove duplicates from within each journal
`subsample and then across author/reviewer samples
`for all journals based on the person’s email address. We
`sent each sampled email address an invitation to just
`one of the three surveys, but it is possible some dupli-
`cates remained if an individual had more than one email
`address in the manuscript tracking systems. We excluded
`potential participants if they had previously opted out of
`receiving BMJ communications or had participated in a
`BMJ research survey within the previous 6 months.
`Two-thirds of the authors were randomly assigned to
`receive the Author Survey, two-thirds of the peer reviewers
`were randomly assigned to receive the Reviewer Survey
`and one-third of each sample was randomised to receive
`the Reader Survey, under the assumption that all partici-
`pants were likely to be readers of journal articles.
`
`Questionnaire administration
`The surveys were developed by the researchers and
`piloted with 45 volunteers to check for ambiguous ques-
`tions. The surveys were revised based on this feedback
`before launching.
`Participants were sent an email invitation in November
`2016 to complete an online survey administered using
`SurveyMonkey. Non-respondents were sent up to two
`reminders. Participants were asked to complete the
`survey from the perspective of their allocated role to
`provide information about their use of specific types of
`supplementary material (study protocol, data collection
`or extraction forms, data tables and figures, completed
`reporting guideline checklists and flow diagrams, inter-
`view transcripts and raw study data). Survey questions
`asked who the material is most useful to; the expected
`use of materials by authors, reviewers and readers; the
`preferred option for accessing supplementary mate-
`rial and if and where supplementary material should
`be published. The questions and response categories
`for each of the survey instruments are shown in online
`supplementary appendices 2–4.
`
`statistical analysis
`Data were exported into Excel, cleaned and anonymised
`prior to analysis. All statistical analyses were conducted in
`
`Price A, et al. BMJ Open 2018;8:e021753. doi:10.1136/bmjopen-2018-021753
`
`AAA, Ex. 2001
`Page 2 of 7
`
`

`

`Open access
`
`Table 1 Characteristics of respondents
`
`Authors
`
`Readers
`
`Reviewers
`
`Overall
`
`819 (28.5)
`4.4 (1.96)
`36 (68.5)
`
`Number (%) of sample
`Mean (SD) number of years as an active researcher
`Approximate number of research papers published as
`author or coauthor—median (IQR)
`Number (%) on how frequently they read articles in medical journals
`
` Very frequently
`377 (46.0)
`
` Frequently
`337 (41.1)
`
` Occasionally
`58 (7.1)
`
` Rarely
`3 (0.4)
`
` Never
`1 (0.1)
`
`911 (31.7)
`4.6 (1.98)
`41 (75)
`
`1142 (39.8)
`5.3 (2.89)
`51 (77)
`
`2872 (100)
`4.8 (2.41)
`46 (81)
`
`462 (54.2)
`331 (38.8)
`58 (6.4)
`1 (0.1)
`1 (0.1)
`
`628 (55.0)
`383 (33.5)
`55 (4.8)
`7 (0.6)
`2 (0.2)
`
`1467 (51.1)
`1051 (36.6)
`171 (6.0)
`11 (0.4)
`4 (0.1)
`
`SPSS V.22. Descriptive and summary statistics of interval
`scale variables were calculated using mean and stan-
`dard deviation (or median and IQR for skewed data),
`and categorical data as frequency and percentages. Data
`have been reported from the individual perspectives of
`authors, readers and reviewers, as well as the aggregated
`overall perspective.
`
`Public and patient involvement
`Forty-five volunteers piloted the surveys and shared valu-
`able feedback to make the questions clear and unam-
`biguous. These volunteers were community members,
`physicians, researchers, patients and teachers.
`
`results
`Online supplementary appendix 5 shows which questions
`in the surveys pertain to our findings presented below
`and in the tables and appendices.
`
`respondent characteristics
`We sent the survey by email to 20 340 people and received
`2872 (14%) responses (819 (12%) from authors, 1142
`(17%) from peer reviewers and 911 (14%) from those
`responding as readers); see table 1. The numbers of years
`as an active researcher were comparable across respon-
`dents with a mean of 4.4 years (SD 1.96) for authors, 4.6
`years (SD 1.98) for readers and 5.3 years (SD 2.89) for
`reviewers. The approximate number of research papers
`reported as published by respondents was a median of 46
`overall (36 for authors, 41 for readers, 51 for reviewers,
`which are statistically different across the groups: inde-
`pendent samples Kruskal-Wallis test p<0.001) but with a
`spread of experience (IQR: 81 research papers). More
`than 87% of respondents read articles in medical jour-
`nals either frequently or very frequently. Respondents are
`from an international sample, with authors from 65 coun-
`tries, reviewers from 57 and readers from 53 countries.
`
`respondent’s interaction with supplementary material
`When recalling what supplementary material was
`contained in their last article submitted, authors stated
`including additional tables of data (74%) or additional
`
`figures (57%) most frequently, followed by checklists for
`relevant reporting guidelines (39%). Readers recalled
`reading additional tables of data (67%) or additional
`figures (53%), followed by study protocol (23%). Over
`80% of reviewers recalled the use of additional figures
`and tables of data in articles they peer reviewed some-
`times or often, in contrast to more than 80% reporting
`rarely seeing raw study data or interview transcripts
`(online supplementary appendix 6).
`
`Preferred option for accessing supplementary material
`Overall (n=2872) respondents’ preferred option for
`accessing tables of data and additional figures was as
`supplementary files alongside the article (60% and 59%,
`respectively), while 50% chose this as their preferred
`option for data collection forms and completed check-
`lists for relevant reporting guidelines. In contrast, 40%
`of respondents preferred that interview transcripts and
`raw study data would not be made available (see figure 1
`for overall data and online supplementary appendix 7 for
`responses by group).
`The open-text responses to accessing supplementary
`materials also showed common sentiment across readers,
`reviewers and authors; as illustrated by this quote ‘It
`depends on the type of research and my purpose for accessing
`it. If I am only reading for enjoyment or for an overview of the
`topic I seldom look at supplementary materials but to replicate
`the research or to further verify the authors findings or methods,
`the supplementary materials provide nuances the paper does not.’
`
`Who the material is most useful to
`Figure 2 shows the overall views of who each type of
`supplementary material is most useful to, from the total
`of 2872 respondents. Additional tables of data and addi-
`tional figures were deemed to be most useful to readers
`(>65%), while the study protocol and data collection/
`extraction forms were deemed most useful to peer
`reviewers (>40%), in contrast to the completed check-
`lists which were deemed most relevant to journal editors
`(40%).
`Table 2 (and online supplementary appendix 8)
`further stratifies these opinions by allocated group, which
`
`3
`
`Price A, et al. BMJ Open 2018;8:e021753. doi:10.1136/bmjopen-2018-021753
`
`AAA, Ex. 2001
`Page 3 of 7
`
`

`

`Open access
`
`Figure 1 Overall views of preferred option for providing/reading/receiving supplementary material (n=2872).
`
`reveals similar trends to those given overall. For instance,
`additional tables of data were regarded as most useful to
`readers (58%–72%) by all groups (authors, reviewers and
`readers), while checklists were perceived as more useful
`to journal editors or peer reviewers rather than readers
`(36%–45% versus 12%–16%).
`
`If and where supplementary material should be published
`Figure 3 depicts the overall views on where (each type
`of) supplementary material should be published, be this
`on the website alongside the article, on another website,
`available directly from the authors, or that it does not
`need to be available. The responses are not mutually
`exclusive, but more than 81% preferred to see additional
`tables of data and figures on a website along with the
`article. In contrast, respondents preferred interview tran-
`scripts (37%) and raw study data (39%) to be available
`by contacting the article’s corresponding author, with a
`further 30% and 27% respondents indicating these mate-
`rials did not need to be made available, respectively. Other
`
`forms of supplementary material, for example, checklists,
`were perceived variably with responses of either availability
`on the website along with the article (45%) or of no need
`to be available (23%). Online supplementary appendix 9
`shows that the responses were similar by group.
`In the open-text responses, there were multiple requests
`for inclusion and publication of replicable software codes,
`dynamic models with the modelling results, statistical
`models, videos and models for imaging and genetics while
`others saw no need for supplementary materials stating
`that the responsibility of the authors was to deliver clear
`and concise reporting that would fit within the given word
`limits of a paper. An important consideration noted by
`some respondents was that some data were restricted and
`could not be shared without compromising the identities
`of participants particularly in data linkage sets. Respon-
`dents stressed the need for improved navigation both of
`the website to access the materials and of the materials
`themselves in terms of labelling, ordering and readability.
`
`Figure 2 Overall views on who each type of supplementary material are most useful to (n=2872).
`
`4
`
`Price A, et al. BMJ Open 2018;8:e021753. doi:10.1136/bmjopen-2018-021753
`
`AAA, Ex. 2001
`Page 4 of 7
`
`

`

`Open access
`
`Table 2 Author, reviewer and reader perspectives on the value of additional tables of data, completed checklists for reporting
`guidelines and raw study data by group*†
`No/total no (%) most useful to
`To journal editors
`To peer reviewers
`
`To readers
`
`Group
`
`Additional tables of data
`29/819 (4)
`
` Authors
`32/1142 (3)
`
` Reviewers
`25/911 (3)
`
` Readers
`68/2872 (3)
`
` Overall
`Completed checklists for reporting guidelines
`
` Authors
`365/819 (45)
`
` Reviewers
`453/1142 (40)
`
` Readers
`340/911 (37)
`
` Overall
`1158/2872 (40)
`Raw study data
`
` Authors
`
` Reviewers
`
` Readers
`
` Overall
`
`120/819 (15)
`207/1142 (18)
`119/911 (13)
`446/2872 (16)
`
`187/819 (23)
`384/1142 (34)
`172/911 (19)
`743/2872 (26)
`
`291/819 (36)
`414/1142 (36)
`394/911 (43)
`1099/2872 (38)
`
`309/819 (38)
`767/1142 (35)
`387/911 (42)
`1093/2872 (38)
`
`564/819 (69)
`662/1142 (58)
`659/911 (72)
`1885/2872 (66)
`
`96/819 (12)
`186/1142 (16)
`117/911 (13)
`399/2872 (14)
`
`276/819 (34)
`385/1142 (34)
`283/911 (31)
`944/2872 (33)
`
`*Percentages do not sum to 100% across each row because some respondents did not answer every question.
`†A table showing the responses for all types of supplementary material is given in our online supplementary material.
`
`It was suggested that supplementary materials for an article
`should be downloadable as a single zipped file.
`
`expected use of materials by authors, reviewers and readers
`Almost half the authors who responded expect that peer
`reviewers should routinely read all supplementary mate-
`rial. But on asking reviewers what they do with supple-
`mentary material, 8%–16% ignored completed checklists,
`flow diagrams, interview transcripts and raw study data,
`with 11%–26% saying it depended on the manuscript. We
`found that only additional tables of data and additional
`figures were being routinely read entirely, at approximately
`60%, with other categories below 36%. In response to the
`
`question about what they usually do with supplementary
`materials, no more than 27% of readers responded that they
`routinely read all of any type of supplementary material, with
`30%–40% ignoring completed checklists, flow diagrams,
`interview transcripts and raw study data (see online supple-
`mentary appendix 10–12).
`
`DIsCussIOn
`In general, authors, reviewers and readers expressed a
`preference for supplementary material that provided
`additional tables over completed reporting checklists or
`
`Figure 3 Overall views on where supplementary material should be published (n=2872).
`
`5
`
`Price A, et al. BMJ Open 2018;8:e021753. doi:10.1136/bmjopen-2018-021753
`
`AAA, Ex. 2001
`Page 5 of 7
`
`

`

`Open access
`
`raw data when reading research articles. This may high-
`light a greater desire among these users of research to
`have access to information that has been analysed or
`summarised by the original researchers. A recurring
`theme in free-text comments was how the importance
`and value of supplementary materials depended on the
`purpose for which they were accessed. For example,
`respondents noted that as interested readers they might
`not access any supplementary materials but that they
`would want to be able to access supplementary materials
`for analysis, replication, secondary research or teaching
`purposes. The respondents also expressed concerns
`about data accessibility, security and the persistence of all
`data, as well as concerns about protecting the trustworthi-
`ness and viability of permissions for raw data (particularly
`when made available to third parties). Considering these
`findings, our survey adds impetus to calls to improve the
`quality of reporting, the use of reporting guidelines15–17
`and the evaluation of the impact of initiatives intended to
`improve the quality of the literature and decisions based
`on it. The survey also revealed uncertainty about the use
`and placement of supplementary materials, as illustrated
`by the following representative open-text comment:
`
`A manuscript to be published should be able to stand
`on its own. Journals are making a mistake by making
`article word counts shorter, then having supplemen-
`tary material. If more data are needed to understand
`the study, they should be in the article
`
`In 2009–2011, the journals Cell, The Journal of Neuro-
`science and Science announced that they would not allow
`authors to include supplemental material on submission
`or host supplemental material on their websites. Instead,
`authors were given the option of including a URL to
`direct readers to the supplementary material on a website
`maintained by the authors, along with a short descrip-
`tion of the supplementary material.4 5 18 However, we
`found little support from our respondents for including
`a weblink within the published paper or for requesting
`supplementary material directly from investigators by
`email. Although journals and researchers may feel a social
`responsibility to make data publicly and permanently avail-
`able,18 they often lack the necessary tools or collaborators
`to build and maintain persistent repositories. Private web
`pages and email are not persistent over time and may be
`vulnerable to corruption. Hofner and colleagues recom-
`mend the use of recognised repositories where digital
`object identifiers are supplied as good practice for data
`preservation and to preserve the options to replicate
`the findings.19 There is considerable debate over how to
`make research more transparent and reproducible.20 As
`supplementary material often contains content that helps
`make research more reproducible, it is important for it to
`be accessible in the long term to help improve research
`efficiency. Others argue that the supplementary material
`needs to be better structured to avoid computational
`errors and to enable machine reading, particularly in the
`fields of genomics, neuroscience, chemistry and other
`
`6
`
`basic sciences.21 Pop and Salzberg proposed that specific
`sections of the supplementary material should be directly
`hyperlinked within the text of the article to improve the
`utility of published scientific articles and to increase the
`likelihood that this material is adequately peer reviewed.6
`
`study limitations
`Our response rate of 14% is typical of current response
`rates for electronic surveys to researchers,22 but still
`allowed us to achieve a large sample, with nearly 3000
`responses from a diverse group of international authors
`and reviewers from 17 biomedical journals. As such, our
`findings make a substantial contribution to the evidence
`on stakeholder views on the value of supplementary
`materials within the peer-reviewed biomedical literature.
`Participants were asked to respond in the assigned role/
`perspective of a reader, peer reviewer or author, and these
`are not mutually exclusive categories, as academics often
`engage in all three activities. Participants gave general
`perceptions and were not asked to report on specific
`cases or the purpose of accessing the article and this may
`have influenced responses.
`
`remaining uncertainties and future research
`Some respondents expressed a preference in open-text
`comments for standardised, well-organised supplemen-
`tary materials that could be combined into a single
`zipped file for downloading or offered as a persistent
`link. However, others commented that data protection
`standards and ethical oversight might not be explicitly
`extended to making supplementary materials publicly
`available. These concerns were not directly addressed
`within the survey questions and so it is not known how
`representative or widespread these opinions might be.
`However, the views expressed could be the target of
`further investigation. It may also be worth investigating
`the relationship between the value of supplementary
`material and the cost of production and publication to
`researchers should journals take on the responsibility for
`the state of supplementary materials in terms of perpetual
`availability, typesetting and compatibility.
`
`COnClusIOns
`Our findings provide evidence that should help jour-
`nals, researchers and funders to consider the roles, costs
`and benefits of supplementary materials. The findings
`highlight, for example, a greater desire among users of
`research to have access to information that has already
`been analysed or summarised by the original researchers,
`rather than their raw material. It may be helpful for jour-
`nals to expand file types to allow storage of, and access
`to a variety of file types, including multimedia, computer
`models and working software prototypes. Our survey
`should also add impetus to calls to improve the quality of
`reporting and the use of reporting guidelines,15–17 and we
`hope that it will stimulate greater emphasis on the need
`for evaluation of the impact of all initiatives intended to
`
`Price A, et al. BMJ Open 2018;8:e021753. doi:10.1136/bmjopen-2018-021753
`
`AAA, Ex. 2001
`Page 6 of 7
`
`

`

`improve the quality of health research and the decisions
`that will subsequently be based on this literature.
`
`Acknowledgements The authors thank the 45 volunteers who piloted this
`research and all the researchers who completed the surveys and especially
`those who shared open-text comments. Their perspectives have increased our
`understanding.
`Contributors AP, SS and MC designed the study and drafted the questionnaires.
`AP drafted the protocol with input from SS and MC. SS extracted the samples
`of authors and reviewers from the journals’ manuscript tracking systems and
`managed the surveys on SurveyMonkey. MC randomised participants to their
`allocated roles. HM analysed the anonymised data. All authors interpreted the
`results, wrote this manuscript and approved its final version.
`Funding The authors have not declared a specific grant for this research from any
`funding agency in the public, commercial or not-for-profit sectors.
`Competing interests AP is the patient editor (Research and Evaluation) at the
`BMJ, and SS is a full-time employee of the BMJ. MC reports involvement in many
`clinical trials and systematic reviews and has prepared and used supplementary
`material widely. He seeks funding for these trials and reviews, as well as for
`research into methodology, including dissemination and accessibility. HM has no
`conflicts of interest.
`Patient consent Not required.
`ethics approval The research was reviewed by, and received ethics clearance
`through, the University of Oxford Central University Research Ethics Committee
`(MS-IDREC-C1-2013-174).
`Provenance and peer review Not commissioned; externally peer reviewed.
`Data sharing statement The datasets used and analysed during the current study
`are available from the corresponding author on reasonable request.
`Open access This is an open access article distributed in accordance with the
`Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which
`permits others to distribute, remix, adapt, build upon this work non-commercially,
`and license their derivative works on different terms, provided the original work is
`properly cited, appropriate credit is given, any changes made indicated, and the use
`is non-commercial. See: http:// creativecommons. org/ licenses/ by- nc/ 4. 0/.
`
`reFerenCes
` 1. Wilkinson MD, Dumontier M, Aalbersberg IJ, et al. The FAIR Guiding
`principles for scientific data management and stewardship. Sci Data
`2016;3:160018.
` 2. Adams CE, Polzmacher S, Wolff A. Systematic reviews: work
`that needs to be done and not to be done. J Evid Based Med
`2013;6:232–5.
`
`Open access
`
` 3. Schriger DL, Chehrazi AC, Merchant RM, et al. Use of the Internet by
`print medical journals in 2003 to 2009: a longitudinal observational
`study. Ann Emerg Med 2011;57:153–60.
` 4. Marcus E. Taming supplemental material. Cell 2009;139:11.
` 5. Maunsell J. Announcement regarding supplemental material. Journal
`of Neuroscience 2010;30:10599–600.
` 6. Pop M, Salzberg SL. Use and mis-use of supplementary material in
`science publications. BMC Bioinformatics 2015;16:237.
` 7. Borowski C. Enough is enough. J Exp Med 2011;208:1337.
` 8. Goldacre B. How to get all trials reported: audit, better data, and
`individual accountability. PLoS Med 2015;12:e1001821.
` 9. Beebee L, McVeigh M. Recommended Practices for Online
`Supplemental Journal Article Materials - NISO RP-15-201x, 2012.
` 10. Bastian H, Glasziou P, Chalmers I. Seventy-five trials and eleven
`systematic reviews a day: how will we ever keep up? PLoS Med
`2010;7:e1000326.
` 11. Anderson NR, Tarczy-Hornoch P, Bumgarner RE. On the persistence
`of supplementary resources in biomedical publications. BMC
`Bioinformatics 2006;7:260.
` 12. Schaffer T, Jackson KM. The use of online supplementary material in
`high-impact scientific journals. Sci Technol Libr 2004;25:73–85.
` 13. Flanagin A, Christiansen SL, Borden C, et al. Editorial evaluation,
`peer review, and publication of research reports with and without
`supplementary online content. JAMA 2018;319:410.
` 14. Kenyon J, Sprague NR, 2014. Trends in the use of supplementary
`materials in environmental science journals. Issues in Science and
`Technology Librarianship http://www. istl. org/ 14- winter/ refereed5.
`html (accessed 27 Nov 2017).
` 15. Hirst A, Altman DG. Are peer reviewers encouraged to use reporting
`guidelines? A survey of 116 health research journals. PLoS One
`2012:7:e35621.
` 16. Stevens A, Shamseer L, Weinstein E, et al. Relation of
`completeness of reporting of health research to journals'
`endorsement of reporting guidelines: systematic review. BMJ
`2014;348:g3804.
` 17. Turner L, Shamseer L, Altman DG, et al. Consolidated standards of
`reporting trials (CONSORT) and the completeness of reporting of
`randomised controlled trials (RCTs) published in medical journals.
`Cochrane Database Syst Rev 2012;11:MR000030.
` 18. Han