Get 20M+ Full-Text Papers For Less Than $1.50/day. Start a 14-Day Trial for You or Your Team.

Learn More →

Tracking Australian health and medical research expenditure with a PubMed bibliometric method

Tracking Australian health and medical research expenditure with a PubMed bibliometric method T he ‘2013 Strategic Review of Health and Medical Research: Better Health through Research’ (2013 HMR Review) vision is to have the world's best healthcare system for Australia in the next 10 years. 1 The 1998 Wills Report 2 paved the way by identifying priority‐driven and strategic health and medical research (HMR) funded through competitive grants. The 2013 HMR review states that “to deliver optimal returns on HMR investment, it is critical to track and monitor both investment and outcomes”. It highlights that apart from the well documented NHMRC competitive schemes, of around $0.8 billion, “the rest of the total $6 billion in investment is not adequately tracked and its outcomes are unclear”. 1 Even the most promising findings of basic research take a long time to translate into clinical experimentation, and adoption in clinical practice is rare. Only one in four promising technologies resulted in a published randomised control trial and fewer than one in 10 entered routine clinical use within 20 years of the index basic science publication. 3 Publications in peer‐reviewed journals are the primary medium through which research findings are rapidly disseminated to the scientific community and an increase in the number of peer‐reviewed publications should be an early consequence of increased funding. 4 A study that used PubMed to assess links between funding grants from the US National Institutes of Health (NIH) and subsequent publications supports the feasibility of this method to study the link between grant funding and research productivity. 5 Internationally, Australia ranked sixth in terms of the number of citations per research publication in the decade 2001–2010, and produced a high relative proportion of publications (per capita) in key international fundamental science and clinical journals. 1 Although the international standing of Australia's publications and citations are well established, particularly those emanating from NHMRC‐funded research, 6 there has been no published literature exploring the associations between Australian HMR expenditure and Australian HMR publications in their entirety. In 2006, we published a method to track the association between Australian HMR expenditure and Australian HMR publications listed in PubMed and compared Australia's publication output to that of the United Kingdom (UK) and New Zealand (NZ). 7 In this follow‐up study, we refine the method to ascertain whether the hypothesis holds after more than a decade of increased HMR funding and the publication of the 2013 HMR Report. Methods Data on total HMR expenditure, given in constant prices accounting for inflation, were obtained from the Australian Institute of Health and Welfare (AIHW). 8,9 Data for university HMR funding, only available by calendar year and in current prices (unadjusted for inflation), were obtained from the Australian Bureau of Statistics. 10 Data for HMR funding of hospitals was not available and could not be included in our analyses. 1 Population statistics for Australia, the UK and NZ were obtained from the Australian Bureau of Statistics, 11 Office of National Statistics 12 and Statistics New Zealand, 13 respectively. Medline is the largest biomedical bibliometric database accessed using PubMed, 14 and contains more than 22 million publications from more than 5,500 biomedical journals. PubMed queries are used to search Medline and we used advanced PubMed queries to retrieve the HMR publications using MeSH (Medical Subject Headings) and text words/phrases. There is no single query term to retrieve ‘Health and Medical Research’ publications from the database as the phrase/concept is not compatible with Medline MeSH terms. We have therefore refined our previous query 7 for Australian publications to retrieve all Australian HMR publications. Details of the refinement process and queries can be found in the supplementary file (available in the online version of this article). We examined a random sample of 10% of papers retrieved, to compare Australian publications retrieved with a simple query and our refined Australian query (supplementary Table 1 and Table 3). There was an improvement of more than 20% in reducing false positives with our refined Australian query. This validation was carried out per calendar year for the years 2009 through to 2013 (supplementary Table 3). The quality of HMR articles retrieved during the analysis period was assessed using the PubMed ‘article type’ filters to enumerate the number of articles that were letters, editorials, clinical trials/randomised control trials, reviews, systematic reviews or cohort studies. Publications that appeared in the PubMed ‘core clinical journal’ collection were also enumerated as another quality measure. 15 Pearson's correlation coefficient was used to examine the relationship between expenditure and publication numbers on log 10 transformed data. All figures display non‐transformed data. Results From 2000/01 to 2006/07, Australian HMR publications increased from 10,696 to 15,798 (increment of 48%) keeping in line with the total HMR expenditure increase from $1.49 billion to $2.73 billion (83%) (Table ). From 2007/08 to 2011/12, although HMR expenditure increased from $3.09 billion to $4.94 billion (60%), publications only increased from 17,131 to 23,818 (39%) (Figure ). Overall, HMR expenditure increased 232% since 2000/01 and this was associated with a 123% increase in PubMed publication outputs. HMR expenditure and PubMed publications were highly correlated; r = 0.982. Australian HMR Expenditure and the Quantity of Australian HMR Publications Listed in PubMed. Financial year HMR Expenditure (billions)* Total HMR Publications Dollars per publication (‘000)* University HMR Publications Hospital HMR Publications 2000/01 1,490 10,696 139,304 5,883 2,950 2001/02 1,578 11,091 142,256 6,127 3,062 2002/03 1,758 11,397 154,293 6,631 3,050 2003/04 1,861 12,062 154,288 7,074 3,227 2004/05 2,115 13,173 160,525 7,670 3,422 2005/06 2,295 14,806 155,030 8,800 3,674 2006/07 2,730 15,798 172,777 9,688 3,670 2007/08 3,090 17,131 180,368 10,748 3,690 2008/09 4,038 18,165 222,320 11,699 3,757 2009/10 4,494 19,221 233,807 12,459 3,908 2010/11 4,452 22,170 200,812 14,614 4,209 2011/12 4,939 23,818 207,364 15,945 4,353 *Constant dollars, adjusted for inflation Association between Australian HMR Expenditure* and Research Publication Output per Financial Year. We have also tabulated the average HMR dollar investment required for a single PubMed publication from 2000/01 to 2011/12 (Table column 4). The cost of a publication has increased 49%, from $139,304 in 2000/01 to $207,364 in 2011/12. The overall trend was one of a gradual linear increase in cost, except in 2008/09 and 2009/10, where the average HMR investment required for a single PubMed publication spiked at $222,320 and $233,807, respectively. Figure shows the association between university HMR expenditure and publications from 2000 to 2012. With an increase of HMR funding from $670 million to $2,823 million, the university publications increased from 5,685 to 16,273. There is a strong correlation between the increasing university HMR funding and publications, r=0.957; an average of 173,478 university HMR dollars are spent for every research publication indexed in PubMed (2012 figures). Hospital PubMed publications have increased from 2,950 in 2000/01 to 4,353 in 2011/12, an increase of 48% (Table ). Compared to the increase in university publications, this increase is small. Association between Australian University HMR expenditure* and PubMed Publications. Table shows two measures of quality of PubMed publications: core clinical journal publications and publication types. Although the total PubMed publications increased from 10,696 to 23,818 (more than double) during the period from 2000/01 to 2011/12, the total ‘core clinical journal publications’ only increased from 575 to 933 (62% increment). The ‘core clinical journal publications’ decreased from 5.4% (2000/01) to 3.9% (2011/12) of the total PubMed publications. Considering the publication types, the highest percentage increase was seen in systematic reviews (262 to 1,210: 362%), followed by cohort studies (867 to 2,653: 206%) and clinical trial/RCTs (552 to 1,440: 161%). Reviews also increased from 1,579 to 3,148 (99% increment). Letters and editorials did not show any significant increases. Quality Analysis of Australian HMR Publications Listed in PubMed. Quality Measure – A Quality Measure – B Financial year Total HMR Publications PubMed Core Clinical Journals Letter Editorial Clinical Trial/RCT Review Systematic Review Cohort Studies 2000/01 10,696 575 265 153 552 1,579 262 867 2001/02 11,091 680 287 150 575 1,578 299 926 2002/03 11,397 728 223 162 664 1,653 324 1,035 2003/04 12,062 764 266 145 794 1,675 343 1,199 2004/05 13,173 757 287 167 825 2,038 417 1,339 2005/06 14,806 826 324 192 943 2,123 498 1,544 2006/07 15,798 834 274 168 985 2,354 631 1,691 2007/08 17,131 838 286 203 1,072 2,513 643 1,805 2008/09 18,165 867 309 185 1,096 2,575 708 1,829 2009/10 19,221 905 275 216 1,164 2,673 843 1,995 2010/11 22,170 923 313 230 1,319 2,929 1,066 2,358 2011/12 23,818 933 227 174 1,440 3,148 1,210 2,653 A – Number of Publications in Journals Belonging to PubMed's Core Clinical Journal Collection B – Publication Type Figure shows the comparison of publication output between Australia, UK and NZ. On a per capita basis, all three have had a similar HMR productivity measured by PubMed publication output since 2000/01. If the comparison is with gross publication numbers, Australia's output has followed a similar upward trend to that of the UK, while NZ's output has tracked upwards at a slower rate. Research publication output, Australia, United Kingdom (UK) and New Zealand (NZ) 2000/01 to 2011/12. Discussion This bibliometric analysis confirms the previously reported 7 association of increased Australian HMR expenditure leading to an increase in HMR publications from Australian authors indexed in PubMed. This evidence suggests that the 1998 Federal Government decision to almost double the annual NHMRC funding, 2 to become one of the highest in the world on a per capita basis, has delivered a sustained increase in the number of PubMed publications along with an increase in publication quality up to 2011/12. Our analysis showed a robust, ‘bang for buck’ performance from the university sector. Butler and colleagues showed that university scientific and medical research publications comprised two‐thirds to three‐quarters of Australia's total research publication output. 16 A similar trend was seen here with university HMR publications accounting for around 65% of total HMR publications in recent years. Internationally, Australian universities ranked at the top with the UK in a comparison of university research funding and publication performance between eight countries. 17 The reason for the high productivity is thought to be the competitive funding environment that exists in Australia. However, the sharp upward trend in university HMR funding in 2008 was not reflected in the publication output until 2012. This is probably due to the lag time of 2–3 years that generally exists between funding and publication output. 18,19 A similar lag can be observed in total HMR expenditure that spikes between 2008/09 and 2010/11, while the corresponding publication spike appears to lag two years behind. The association between HMR expenditure and publication output can be applied to specific universities for tracking performance. The method will be more useful for specific institutions because of the availability of accurate HMR expenditure and PubMed publications at an institution level. Unlike university publications, hospital‐based research publications appear to have decreased from 27.6% to 18.3% of total HMR publications, despite an increase in funding. 1 A possible explanation for this is attribution bias. The majority of hospital‐based clinicians have university affiliations and may prefer to use that affiliation when publishing. This may underestimate the true numbers of hospital‐based research publications. However, there is no data available to determine whether clinician preferences for stating their affiliations have changed over time. It may also be that additional affiliations are removed when it is being indexed in PubMed, something that is beyond the control of the authors. There are other likely contributing factors that may discourage budding clinician researchers: having to compete against full‐time researchers, uncertainty of recurrent funding, no protected time for research, lack of appropriate infrastructure, and a hospital culture where research is generally not considered to be a priority. 1 We used PubMed, the largest biomedical bibliometric database for our analysis. In a comparison with Scopus, Web of Science and Google Scholar, PubMed emerged as the optimal tool for biomedical electronic research that is readily available to researchers and clinicians. 20 Using PubMed is also important because of its free access for the reproducibility and verification of our methods by independent researchers. However, not all Australian HMR publications may be indexed in PubMed. Due to the high visibility of Australian medical research; however, the incidence of non‐Medline indexed HMR articles should be small. 16 While measuring the quantity of HMR articles indexed in PubMed is relatively simple, assessing their quality is not so easy. PubMed publication types such as systematic reviews and randomised clinical trials allow some form of categorisation of the quality of a journal article according to the study design, which gives a measure of the methodological quality. However, such analyses can be fraught with debate about which article types are ‘quality’ as the qualitative research publications get lesser weightage. Considering ‘publication types’, Australian publications seem to be increasing their quality by a steady increase of systematic reviews, clinical trials and cohort studies. The number of Australian publications in the PubMed ‘core clinical journal’ collection is another quality indicator we measured to ascertain the quality trend over time. From the 5,500 journals indexed in PubMed, the 120 core clinical journals are the top‐rated journals in biomedicine. 14,15 Although the total number of HMR publications more than doubled from 2001 to 2012, the proportion of core clinical journal publications as a percentage reduced from 5.4% to 3.9%. This evidence alone is suggestive of a reduction in quality. However, a recent publication evaluating the relevance of the current core clinical journal collection concluded that a review and update of the collection is now required to better align with current clinical needs. 21 If so, a reduction in publications appearing in the core collection in favour of publication in other potentially more relevant journals could be seen as a quality enhancement for Australian HMR publications. Journal citation index (JCI) is among the factors Lundberg lists that could be considered when assessing quality of articles and journals. 22 The JCI, or impact factor (IF), is a bibliometric tool that was developed to rank journals, but has also been used to infer the quality of the publications therein. 23,24 The JCI reflects how frequently the journal's recent papers in total are cited. We did not use the JCI to measure the quality for several reasons. The process for calculating the JCI seems to be quite inconsistent, as the denominator can vary greatly depending on which articles Thompson Scientific deems as substantive. 25 Australian HMR publications are found in a large number of different PubMed journals and not in a single journal of, say, ‘Health Economics’. Therefore a proxy measure of JCI does not help much in our quest to measure the worth and impact of Australian HMR. Moreover, Lozano et al 23 reported that the relationship between journal impact factors and paper citations is weakening significantly due to online open access availability. In 2008, the NHMRC moved away from using the JCI and now stipulates that NHMRC‐funded research publications be made open access. 26 There have been increasing calls from international groups to find alternatives to the JCI. 20,24 Individual articles with higher citations are a good indicator of quality. However it is not possible to calculate the citation counts for each year for the more than 10,000 to 20,000 articles that are published in a number of different journals for Australian HMR publications. The emerging field of altmetrics, 27 which provides statistics on online shares, downloads, reviews and social media usage, provides yet another alternative holistic measure of assessing the quality, or citability, of individual articles. However, altmetrics cannot currently rapidly assess an entire country's productivity and research quality using these parameters. Lozano 28 suggested ‘impact per dollar’ as a way to assess an individual researcher's productivity by comparing the impact of their publications against the dollar value of the grant used to generate those publications. Here we have used a similar measurement to assess Australian HMR productivity as a whole. The average HMR investment for a PubMed publication in 2011/12 showed a 49% increase on the 2000/01 figures. The reasons for this increase are likely due to multiple factors. An increase in the number of clinical trials being performed, as evidenced by the increase in Australian HMR publications of this type described earlier, would be a contributing factor. Adding to this is the escalating cost of multicentre clinical trials and the high cost of newer technologies. International comparisons showed that the productivity of Australian HMR researchers, on a per capita basis, has kept pace with – if not bettered – that of the UK and NZ over the period examined. Limitations Our method for tracking publications is only an intermediate step in evaluating Australian HMR expenditure, considering the comprehensive research translation process. Journal publications and their related sophisticated indicators such as impact factors and H‐indices can be subjected to manipulations by publishing institutions in many different ways. 23,25 The comment by the 2013 Nobel Laureate Randy Schekman to boycott the so‐called luxury journals in favour of the open access journals gives an indication of the increase and manipulation of indicators by the top‐ranked journals. 29 The method described in this article depends on the accuracy and completeness of author affiliations in the PubMed listing. Variations in the way affiliations are input and how MeSH terms are applied to each article make it difficult to accurately retrieve Australian HMR publications using a simple one‐word query. Our analyses showed a significant reduction in the incidence of false positives when a basic Australia search was replaced with our refined Australia query (see supplementary file). Our searches captured many articles where the Australian author was not the first listed author on the article. However, we cannot rule out that some articles were missed due to the same reason. Although we checked for false positives in a random 5% subset of articles during the query refinement process and in a 10% random subset thereafter, we did not have a method to check for articles we missed (false negatives). Reliable annual HMR expenditure data for the UK and NZ were not available for these analyses and thus a per capita comparison was performed. An analysis of HMR expenditure and resultant publication output for all three countries would be a good measure of Australia's global competitiveness. Because reliable HMR expenditure data was not available for hospitals 1 and it is difficult to accurately retrieve hospital‐affiliated publications, we did not carry out a correlation analysis of publications and expenditure as was carried out for universities. We tested our new queries only for the past five years. There may be a very small effect of using the new query beyond the immediate past five years, extending to ten years, because of the changes in MeSH and the way librarians index the articles. Since the publication of the 2013 HMR review, 1 successive Federal Government budgets have flagged cuts to university funding, deviating from the 1998 initiative of almost doubling NHMRC funding. The Gillard Government's 2013/14 budget cut $2.3 billion from the higher education sector 30 although this cut has been reduced by the Abbott Government in its 2014/15 budget. Despite this, the Abbott budget has also flagged an approximate 20% reduction in Federal contributions to university course fees and the introduction of student contributions to students undertaking higher degrees by research. 31 All these changes will affect HMR expenditure at the institution level and associated publication output. It will be interesting to use our method to monitor future trends as these changes come into play at the federal level and also for institutions at the university and hospital level. Conclusion Our analysis shows that it is possible to have relatively uncomplicated measures of quantity and quality to assess Australian health and medical research (HMR) investment returns by measuring the temporal trends in HMR expenditure and PubMed publications by Australian authors. Tracking HMR investment and PubMed outputs could be used to compare different institutions and universities or hospital clusters over a period of time. This method could also be used by funding organisations and inform the outcome of different funding methods. http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png Australian and New Zealand Journal of Public Health Wiley

Tracking Australian health and medical research expenditure with a PubMed bibliometric method

Download PDF
5 pages

Loading next page...
 
/lp/wiley/tracking-australian-health-and-medical-research-expenditure-with-a-XyRpnlzfFw

References (31)

Publisher
Wiley
Copyright
© 2015 Public Health Association of Australia
ISSN
1326-0200
eISSN
1753-6405
DOI
10.1111/1753-6405.12366
pmid
25904290
Publisher site
See Article on Publisher Site

Abstract

T he ‘2013 Strategic Review of Health and Medical Research: Better Health through Research’ (2013 HMR Review) vision is to have the world's best healthcare system for Australia in the next 10 years. 1 The 1998 Wills Report 2 paved the way by identifying priority‐driven and strategic health and medical research (HMR) funded through competitive grants. The 2013 HMR review states that “to deliver optimal returns on HMR investment, it is critical to track and monitor both investment and outcomes”. It highlights that apart from the well documented NHMRC competitive schemes, of around $0.8 billion, “the rest of the total $6 billion in investment is not adequately tracked and its outcomes are unclear”. 1 Even the most promising findings of basic research take a long time to translate into clinical experimentation, and adoption in clinical practice is rare. Only one in four promising technologies resulted in a published randomised control trial and fewer than one in 10 entered routine clinical use within 20 years of the index basic science publication. 3 Publications in peer‐reviewed journals are the primary medium through which research findings are rapidly disseminated to the scientific community and an increase in the number of peer‐reviewed publications should be an early consequence of increased funding. 4 A study that used PubMed to assess links between funding grants from the US National Institutes of Health (NIH) and subsequent publications supports the feasibility of this method to study the link between grant funding and research productivity. 5 Internationally, Australia ranked sixth in terms of the number of citations per research publication in the decade 2001–2010, and produced a high relative proportion of publications (per capita) in key international fundamental science and clinical journals. 1 Although the international standing of Australia's publications and citations are well established, particularly those emanating from NHMRC‐funded research, 6 there has been no published literature exploring the associations between Australian HMR expenditure and Australian HMR publications in their entirety. In 2006, we published a method to track the association between Australian HMR expenditure and Australian HMR publications listed in PubMed and compared Australia's publication output to that of the United Kingdom (UK) and New Zealand (NZ). 7 In this follow‐up study, we refine the method to ascertain whether the hypothesis holds after more than a decade of increased HMR funding and the publication of the 2013 HMR Report. Methods Data on total HMR expenditure, given in constant prices accounting for inflation, were obtained from the Australian Institute of Health and Welfare (AIHW). 8,9 Data for university HMR funding, only available by calendar year and in current prices (unadjusted for inflation), were obtained from the Australian Bureau of Statistics. 10 Data for HMR funding of hospitals was not available and could not be included in our analyses. 1 Population statistics for Australia, the UK and NZ were obtained from the Australian Bureau of Statistics, 11 Office of National Statistics 12 and Statistics New Zealand, 13 respectively. Medline is the largest biomedical bibliometric database accessed using PubMed, 14 and contains more than 22 million publications from more than 5,500 biomedical journals. PubMed queries are used to search Medline and we used advanced PubMed queries to retrieve the HMR publications using MeSH (Medical Subject Headings) and text words/phrases. There is no single query term to retrieve ‘Health and Medical Research’ publications from the database as the phrase/concept is not compatible with Medline MeSH terms. We have therefore refined our previous query 7 for Australian publications to retrieve all Australian HMR publications. Details of the refinement process and queries can be found in the supplementary file (available in the online version of this article). We examined a random sample of 10% of papers retrieved, to compare Australian publications retrieved with a simple query and our refined Australian query (supplementary Table 1 and Table 3). There was an improvement of more than 20% in reducing false positives with our refined Australian query. This validation was carried out per calendar year for the years 2009 through to 2013 (supplementary Table 3). The quality of HMR articles retrieved during the analysis period was assessed using the PubMed ‘article type’ filters to enumerate the number of articles that were letters, editorials, clinical trials/randomised control trials, reviews, systematic reviews or cohort studies. Publications that appeared in the PubMed ‘core clinical journal’ collection were also enumerated as another quality measure. 15 Pearson's correlation coefficient was used to examine the relationship between expenditure and publication numbers on log 10 transformed data. All figures display non‐transformed data. Results From 2000/01 to 2006/07, Australian HMR publications increased from 10,696 to 15,798 (increment of 48%) keeping in line with the total HMR expenditure increase from $1.49 billion to $2.73 billion (83%) (Table ). From 2007/08 to 2011/12, although HMR expenditure increased from $3.09 billion to $4.94 billion (60%), publications only increased from 17,131 to 23,818 (39%) (Figure ). Overall, HMR expenditure increased 232% since 2000/01 and this was associated with a 123% increase in PubMed publication outputs. HMR expenditure and PubMed publications were highly correlated; r = 0.982. Australian HMR Expenditure and the Quantity of Australian HMR Publications Listed in PubMed. Financial year HMR Expenditure (billions)* Total HMR Publications Dollars per publication (‘000)* University HMR Publications Hospital HMR Publications 2000/01 1,490 10,696 139,304 5,883 2,950 2001/02 1,578 11,091 142,256 6,127 3,062 2002/03 1,758 11,397 154,293 6,631 3,050 2003/04 1,861 12,062 154,288 7,074 3,227 2004/05 2,115 13,173 160,525 7,670 3,422 2005/06 2,295 14,806 155,030 8,800 3,674 2006/07 2,730 15,798 172,777 9,688 3,670 2007/08 3,090 17,131 180,368 10,748 3,690 2008/09 4,038 18,165 222,320 11,699 3,757 2009/10 4,494 19,221 233,807 12,459 3,908 2010/11 4,452 22,170 200,812 14,614 4,209 2011/12 4,939 23,818 207,364 15,945 4,353 *Constant dollars, adjusted for inflation Association between Australian HMR Expenditure* and Research Publication Output per Financial Year. We have also tabulated the average HMR dollar investment required for a single PubMed publication from 2000/01 to 2011/12 (Table column 4). The cost of a publication has increased 49%, from $139,304 in 2000/01 to $207,364 in 2011/12. The overall trend was one of a gradual linear increase in cost, except in 2008/09 and 2009/10, where the average HMR investment required for a single PubMed publication spiked at $222,320 and $233,807, respectively. Figure shows the association between university HMR expenditure and publications from 2000 to 2012. With an increase of HMR funding from $670 million to $2,823 million, the university publications increased from 5,685 to 16,273. There is a strong correlation between the increasing university HMR funding and publications, r=0.957; an average of 173,478 university HMR dollars are spent for every research publication indexed in PubMed (2012 figures). Hospital PubMed publications have increased from 2,950 in 2000/01 to 4,353 in 2011/12, an increase of 48% (Table ). Compared to the increase in university publications, this increase is small. Association between Australian University HMR expenditure* and PubMed Publications. Table shows two measures of quality of PubMed publications: core clinical journal publications and publication types. Although the total PubMed publications increased from 10,696 to 23,818 (more than double) during the period from 2000/01 to 2011/12, the total ‘core clinical journal publications’ only increased from 575 to 933 (62% increment). The ‘core clinical journal publications’ decreased from 5.4% (2000/01) to 3.9% (2011/12) of the total PubMed publications. Considering the publication types, the highest percentage increase was seen in systematic reviews (262 to 1,210: 362%), followed by cohort studies (867 to 2,653: 206%) and clinical trial/RCTs (552 to 1,440: 161%). Reviews also increased from 1,579 to 3,148 (99% increment). Letters and editorials did not show any significant increases. Quality Analysis of Australian HMR Publications Listed in PubMed. Quality Measure – A Quality Measure – B Financial year Total HMR Publications PubMed Core Clinical Journals Letter Editorial Clinical Trial/RCT Review Systematic Review Cohort Studies 2000/01 10,696 575 265 153 552 1,579 262 867 2001/02 11,091 680 287 150 575 1,578 299 926 2002/03 11,397 728 223 162 664 1,653 324 1,035 2003/04 12,062 764 266 145 794 1,675 343 1,199 2004/05 13,173 757 287 167 825 2,038 417 1,339 2005/06 14,806 826 324 192 943 2,123 498 1,544 2006/07 15,798 834 274 168 985 2,354 631 1,691 2007/08 17,131 838 286 203 1,072 2,513 643 1,805 2008/09 18,165 867 309 185 1,096 2,575 708 1,829 2009/10 19,221 905 275 216 1,164 2,673 843 1,995 2010/11 22,170 923 313 230 1,319 2,929 1,066 2,358 2011/12 23,818 933 227 174 1,440 3,148 1,210 2,653 A – Number of Publications in Journals Belonging to PubMed's Core Clinical Journal Collection B – Publication Type Figure shows the comparison of publication output between Australia, UK and NZ. On a per capita basis, all three have had a similar HMR productivity measured by PubMed publication output since 2000/01. If the comparison is with gross publication numbers, Australia's output has followed a similar upward trend to that of the UK, while NZ's output has tracked upwards at a slower rate. Research publication output, Australia, United Kingdom (UK) and New Zealand (NZ) 2000/01 to 2011/12. Discussion This bibliometric analysis confirms the previously reported 7 association of increased Australian HMR expenditure leading to an increase in HMR publications from Australian authors indexed in PubMed. This evidence suggests that the 1998 Federal Government decision to almost double the annual NHMRC funding, 2 to become one of the highest in the world on a per capita basis, has delivered a sustained increase in the number of PubMed publications along with an increase in publication quality up to 2011/12. Our analysis showed a robust, ‘bang for buck’ performance from the university sector. Butler and colleagues showed that university scientific and medical research publications comprised two‐thirds to three‐quarters of Australia's total research publication output. 16 A similar trend was seen here with university HMR publications accounting for around 65% of total HMR publications in recent years. Internationally, Australian universities ranked at the top with the UK in a comparison of university research funding and publication performance between eight countries. 17 The reason for the high productivity is thought to be the competitive funding environment that exists in Australia. However, the sharp upward trend in university HMR funding in 2008 was not reflected in the publication output until 2012. This is probably due to the lag time of 2–3 years that generally exists between funding and publication output. 18,19 A similar lag can be observed in total HMR expenditure that spikes between 2008/09 and 2010/11, while the corresponding publication spike appears to lag two years behind. The association between HMR expenditure and publication output can be applied to specific universities for tracking performance. The method will be more useful for specific institutions because of the availability of accurate HMR expenditure and PubMed publications at an institution level. Unlike university publications, hospital‐based research publications appear to have decreased from 27.6% to 18.3% of total HMR publications, despite an increase in funding. 1 A possible explanation for this is attribution bias. The majority of hospital‐based clinicians have university affiliations and may prefer to use that affiliation when publishing. This may underestimate the true numbers of hospital‐based research publications. However, there is no data available to determine whether clinician preferences for stating their affiliations have changed over time. It may also be that additional affiliations are removed when it is being indexed in PubMed, something that is beyond the control of the authors. There are other likely contributing factors that may discourage budding clinician researchers: having to compete against full‐time researchers, uncertainty of recurrent funding, no protected time for research, lack of appropriate infrastructure, and a hospital culture where research is generally not considered to be a priority. 1 We used PubMed, the largest biomedical bibliometric database for our analysis. In a comparison with Scopus, Web of Science and Google Scholar, PubMed emerged as the optimal tool for biomedical electronic research that is readily available to researchers and clinicians. 20 Using PubMed is also important because of its free access for the reproducibility and verification of our methods by independent researchers. However, not all Australian HMR publications may be indexed in PubMed. Due to the high visibility of Australian medical research; however, the incidence of non‐Medline indexed HMR articles should be small. 16 While measuring the quantity of HMR articles indexed in PubMed is relatively simple, assessing their quality is not so easy. PubMed publication types such as systematic reviews and randomised clinical trials allow some form of categorisation of the quality of a journal article according to the study design, which gives a measure of the methodological quality. However, such analyses can be fraught with debate about which article types are ‘quality’ as the qualitative research publications get lesser weightage. Considering ‘publication types’, Australian publications seem to be increasing their quality by a steady increase of systematic reviews, clinical trials and cohort studies. The number of Australian publications in the PubMed ‘core clinical journal’ collection is another quality indicator we measured to ascertain the quality trend over time. From the 5,500 journals indexed in PubMed, the 120 core clinical journals are the top‐rated journals in biomedicine. 14,15 Although the total number of HMR publications more than doubled from 2001 to 2012, the proportion of core clinical journal publications as a percentage reduced from 5.4% to 3.9%. This evidence alone is suggestive of a reduction in quality. However, a recent publication evaluating the relevance of the current core clinical journal collection concluded that a review and update of the collection is now required to better align with current clinical needs. 21 If so, a reduction in publications appearing in the core collection in favour of publication in other potentially more relevant journals could be seen as a quality enhancement for Australian HMR publications. Journal citation index (JCI) is among the factors Lundberg lists that could be considered when assessing quality of articles and journals. 22 The JCI, or impact factor (IF), is a bibliometric tool that was developed to rank journals, but has also been used to infer the quality of the publications therein. 23,24 The JCI reflects how frequently the journal's recent papers in total are cited. We did not use the JCI to measure the quality for several reasons. The process for calculating the JCI seems to be quite inconsistent, as the denominator can vary greatly depending on which articles Thompson Scientific deems as substantive. 25 Australian HMR publications are found in a large number of different PubMed journals and not in a single journal of, say, ‘Health Economics’. Therefore a proxy measure of JCI does not help much in our quest to measure the worth and impact of Australian HMR. Moreover, Lozano et al 23 reported that the relationship between journal impact factors and paper citations is weakening significantly due to online open access availability. In 2008, the NHMRC moved away from using the JCI and now stipulates that NHMRC‐funded research publications be made open access. 26 There have been increasing calls from international groups to find alternatives to the JCI. 20,24 Individual articles with higher citations are a good indicator of quality. However it is not possible to calculate the citation counts for each year for the more than 10,000 to 20,000 articles that are published in a number of different journals for Australian HMR publications. The emerging field of altmetrics, 27 which provides statistics on online shares, downloads, reviews and social media usage, provides yet another alternative holistic measure of assessing the quality, or citability, of individual articles. However, altmetrics cannot currently rapidly assess an entire country's productivity and research quality using these parameters. Lozano 28 suggested ‘impact per dollar’ as a way to assess an individual researcher's productivity by comparing the impact of their publications against the dollar value of the grant used to generate those publications. Here we have used a similar measurement to assess Australian HMR productivity as a whole. The average HMR investment for a PubMed publication in 2011/12 showed a 49% increase on the 2000/01 figures. The reasons for this increase are likely due to multiple factors. An increase in the number of clinical trials being performed, as evidenced by the increase in Australian HMR publications of this type described earlier, would be a contributing factor. Adding to this is the escalating cost of multicentre clinical trials and the high cost of newer technologies. International comparisons showed that the productivity of Australian HMR researchers, on a per capita basis, has kept pace with – if not bettered – that of the UK and NZ over the period examined. Limitations Our method for tracking publications is only an intermediate step in evaluating Australian HMR expenditure, considering the comprehensive research translation process. Journal publications and their related sophisticated indicators such as impact factors and H‐indices can be subjected to manipulations by publishing institutions in many different ways. 23,25 The comment by the 2013 Nobel Laureate Randy Schekman to boycott the so‐called luxury journals in favour of the open access journals gives an indication of the increase and manipulation of indicators by the top‐ranked journals. 29 The method described in this article depends on the accuracy and completeness of author affiliations in the PubMed listing. Variations in the way affiliations are input and how MeSH terms are applied to each article make it difficult to accurately retrieve Australian HMR publications using a simple one‐word query. Our analyses showed a significant reduction in the incidence of false positives when a basic Australia search was replaced with our refined Australia query (see supplementary file). Our searches captured many articles where the Australian author was not the first listed author on the article. However, we cannot rule out that some articles were missed due to the same reason. Although we checked for false positives in a random 5% subset of articles during the query refinement process and in a 10% random subset thereafter, we did not have a method to check for articles we missed (false negatives). Reliable annual HMR expenditure data for the UK and NZ were not available for these analyses and thus a per capita comparison was performed. An analysis of HMR expenditure and resultant publication output for all three countries would be a good measure of Australia's global competitiveness. Because reliable HMR expenditure data was not available for hospitals 1 and it is difficult to accurately retrieve hospital‐affiliated publications, we did not carry out a correlation analysis of publications and expenditure as was carried out for universities. We tested our new queries only for the past five years. There may be a very small effect of using the new query beyond the immediate past five years, extending to ten years, because of the changes in MeSH and the way librarians index the articles. Since the publication of the 2013 HMR review, 1 successive Federal Government budgets have flagged cuts to university funding, deviating from the 1998 initiative of almost doubling NHMRC funding. The Gillard Government's 2013/14 budget cut $2.3 billion from the higher education sector 30 although this cut has been reduced by the Abbott Government in its 2014/15 budget. Despite this, the Abbott budget has also flagged an approximate 20% reduction in Federal contributions to university course fees and the introduction of student contributions to students undertaking higher degrees by research. 31 All these changes will affect HMR expenditure at the institution level and associated publication output. It will be interesting to use our method to monitor future trends as these changes come into play at the federal level and also for institutions at the university and hospital level. Conclusion Our analysis shows that it is possible to have relatively uncomplicated measures of quantity and quality to assess Australian health and medical research (HMR) investment returns by measuring the temporal trends in HMR expenditure and PubMed publications by Australian authors. Tracking HMR investment and PubMed outputs could be used to compare different institutions and universities or hospital clusters over a period of time. This method could also be used by funding organisations and inform the outcome of different funding methods.

Journal

Australian and New Zealand Journal of Public Health – Wiley

Published: Jun 1, 2015

There are no references for this article.