The Media Doctor website (www.mediadoctor.org.au) was launched in 2004 with the aim of providing an objective analysis of the strengths and weaknesses of the health stories appearing in the Australian mainstream media. A secondary aim was to increase the completeness of health stories and, subsequently, health literacy among journalists and media consumers. To date, Media Doctor has reviewed over 1300 stories and similar sites have been launched in Canada (www.mediadoctor.ca), in the USA (www.healthnewsreview.org) and in Hong Kong (www.mediadoctor.hk).
Media Doctor reviews health news stories published in the Australian general media, including newspapers, online news and transcripts of television and radio broadcasts. Stories are eligible for review if they cover new health interventions for humans, including drugs, surgical procedures, diagnostic tests, and complementary therapies. Most stories are derived from research-based interventions but this is not an inclusion criterion. Relevant material such as media releases or journal articles are sent with the story to two reviewers.
Media Doctor reviewers include clinicians and researchers who conduct the reviews in a voluntary capacity. Biographical details of reviewers are available on the website. Reviewers rate stories independently of each other using validated rating instruments. The instruments contain 10 items (see Table 1). These are the same items used by media Doctor Canada and Health News Review in the USA.
For each news article, the ten criteria are scored as ‘satisfactory’, ‘not satisfactory’ or ‘not applicable’. Total scores (expressed as proportion of items rated ‘satisfactory’) are posted for articles that have seven or more ‘evaluable’ items. Scores are visually depicted on the website using a 1-5 ‘star’ rating along with commentaries from the reviewers. Cumulative scores for the major media outlets are also presented, providing ongoing feedback on their performance compared with other outlets. Reviewers post draft reviews in a password-protected area of the website and discrepancies are resolved by consensus. If necessary, a third reviewer is used to settle disagreements. To ensure objectivity, all reviews are screened by a researcher who checks the scores and edits comments. Both reviewers contribute to the comment section, which is used to highlight the strengths of the story, or aspects that could have been improved, including areas not covered in the rating instrument, such as sensationalist language or inappropriate headlines. The turnaround for reviews is approximately two weeks from locating the news story to having it appear on the website.
Table 1. 10 Criteria used to rate news articles about medical interventions.
|Rating Criteria*: The extent to which the story:|
|1. Reported the novelty of the intervention|
|2. Reported the availability of the intervention|
|3. Described the treatment or diagnostic options that are available|
|4. Avoided elements of disease mongering|
|5. Reported evidence supporting the intervention|
|6. Quantified the benefits of intervention|
|7. Described the harms of intervention|
|8. Reported on the costs of intervention|
|9. Consulted with independent expert sources of information|
|10. Went beyond any available media release.|
*Stories are marked ‘satisfactory’, ‘not satisfactory’ or ‘not applicable’. Criteria used to determine scores are available at the Media Doctor website.