Get 20M+ Full-Text Papers For Less Than $1.50/day. Start a 14-Day Trial for You or Your Team.

Learn More →

Data envelopment analysis applications in primary health care: a systematic review

Data envelopment analysis applications in primary health care: a systematic review Abstract Background Strategic management of primary health care centres is necessary for creating an efficient global health care system that delivers good care. Objectives To perform a systematic literature review of the use of data envelopment analysis in estimating the relative technical efficiency of primary health care centres, and to identify the inputs, outputs and models used. Methods PubMed, MEDLINE Complete, Embase and Web of Science were searched for papers published before the 25 March 2019. Results Of a total of 4231 search results, 54 studies met the inclusion criteria. The identified inputs included personnel costs, gross expenditures, referrals and days of hospitalization, as well as prescriptions and investigations. Outputs included consultations or visits, registered patients, procedures, treatments and services, prescriptions and investigations. A variety of data envelopment analysis models used was identified, with no standard approach. Conclusions Data envelopment analysis extends the scope of tools used to analyse primary care functioning. It can support health economic analyses when assessing primary care efficiency. The main issues are setting outputs and inputs and selecting a model best suited for the range of products and services in the primary health care sector. This article serves as a step forward in the standardization of data envelopment analysis, but further research is needed. Health economics, health information, primary care, public health, quality of care, systematic review Key Messages We identify various approaches to effective assessment of primary care. Data envelopment analysis is widely used in primary care. A variety of analysis models exist, with no single standard. Introduction Many countries have a growing demand for health care services and this is accompanied by growing expenditure (1–3). Measuring the relative efficiency of health care systems, including primary health care (PHC) centres, creates a baseline for evaluating the management of their resources and for benchmarking their productiveness against others. Moreover, better use of health care resources could lead to the provision of better health services at the basic level. Recent years have seen a growth in interest in the use of quantitative methods for comparing the efficiency of health care systems (2,4,5), and many approaches for estimating the efficiency of various levels of health care organizations are under intensive investigation (6). Data envelopment analysis (DEA) (7) is a non-parametric, deterministic alternative to several efficiency measurement techniques, such as the cost-effectiveness ratio, corrected ordinary least squares or stochastic frontier analysis. DEA (7) uses a linear programming technique that gives a single measure of efficiency. It is based on the principle that an organization can be considered efficient when it is able to obtain the greatest output, in terms of goods, products or feasible services, by using a certain combination of used resources as input or, alternatively, when it produces a certain level of output using the least possible input (8,9). Since its inception 40 years ago, DEA has been extensively investigated and applied as a tool (10). Despite its uncertainty (11), it is a promising way to estimate the efficiency of units in many areas, including health care systems (12,13), reform (14,15), PHC centres (see 54 publications in Supplementary Table S1), regions (16–20), dental units (21), emergency departments (22) and hospitals (10,23–25). Systematic reviews have surveyed the literature associated with DEA and identified the most influential journals in the fields of DEA and its applications in the period 1978–2019 (26–30). DEA is a well-developed method used in health care efficiency assessment, and its use is still being refined (31). It can be used to evaluate operating organizations, establish criteria to improve their functioning and measure their progress. Despite the considerable body of literature surrounding DEA published over the last 20 years (32), including applications in Poland (33), a search of PROSPERO revealed only four protocols for the systematic review of assessments of the efficiency of health services using DEA (34,35). The aim of the present article is to systematically review empirical studies of DEA applications in the field of PHC to identify the most commonly employed group inputs, outputs and models. Our findings will facilitate the development of a standard set of criteria for the design and execution of DEA in PHC and may prove valuable for the standardization of DEA outcomes. This study serves as a step towards the standardization of DEA as the most widely used tool for improving the efficiency of PHC organizations and contributes to the refinement of DEA as a methodology. Methods Our review employs the approach described by the Institute of Health Science in Oxford, adopted for General Practice by Department of General Practice, University of Glasgow, as part of their Critical Appraisal Skills Programme (36). This review was reported according to the Preferred Reporting Item for Systematic Reviews and Meta-Analysis (PRISMA) approach (37). The review corpus comprises studies of health care technical efficiency based in the primary care setting. The list of included studies was restricted to those concerning decision-making units (DMUs) such as PHC centres, physicians, family physicians, primary care physicians, GPs, general practices and health maintenance organizations (HMOs). Search strategy A systematic electronic search was performed according to PRISMA (37) between March 2017 and March 2019, which covered the studies published before 25 March 2019. The literature search was performed by two independent researchers (IZ and MGC) in four electronic databases: PubMed, MEDLINE Complete (Medical Literature Analysis and Retrieval System Online), Embase (Excerpta Medica Database) and Web of Science. The search terms and filters listed in Box 1 were used. The identified papers were limited to full-text original and review articles published in English. Box 1. Search headings – Efficiency [MeSH Major Topic] – Benchmarking [MeSH Major Topic] – Benchmarking [Title/Abstract] – ‘Data Envelopment Analysis’ [Title/Abstract] – DEA[Title/Abstract] – ‘Technical Efficiency’ [Title/Abstract] – #1 OR #2 OR #3 OR #4 OR #5 OR #6 – ‘Primary Health Care’ [MeSH Major Topic] – ‘Physicians, Primary Care’ [MeSH Major Topic] – ‘Primary Health Care’ [Title/Abstract] – ‘General Practice*’ [Title/Abstract] – ‘Physicians*’ [Title/Abstract] – ‘HMO*’ [Title/Abstract] – #8 OR #9 OR #10 OR #11 OR #12 OR #13 – #7 AND #14 Filters: (Journal Article OR Review) AND Full text AND English Study selection The retrieved papers were imported into EndNote X4. Duplicates were identified and removed. The two researchers independently manually screened the titles and Abstracts to select relevant papers. Any disagreements were resolved by discussions with an external expert. When the articles had insufficient information in the title and Abstract to support this screening, a full-text reading was conducted. Following this, all potentially eligible papers were added in full-text form. A manual search was then used to retrieve papers for full-text review. These papers were examined by two authors using a checklist designed for this study, with inclusion and exclusion criteria given in Box 2. The list of included studies was restricted to those concerning DMUs such as PHC centres, physicians, family physicians, primary care physicians, GPs, general practices and HMOs. Box 2. Inclusion and exclusion criteria Inclusion Studies that met the following criteria were included inputs and outputs used to evaluate the technical efficiency of PHC centres or physicians, using the DEA method. The studies concentrated on the efficiency of PHC centres as the organization as a whole, including the physicians’ efficiency working in these centres. Studies that included the following DMU levels: PHC centres, physicians, family physicians, primary care physicians, GPs, general practices, and health maintenance organizations (HMO). Studies written as a full-text journal article, in the English language. Exclusion Studies that were not based on the DEA method: no inputs, no outputs, no models, or no technical efficiency calculations, and not refer to primary health care. In addition, those that did refer to technical efficiency of quality/satisfaction, disease, treatment/drug/therapy, e-health/computer, reform/system, programmes, statistics, or education/training. Studies that were based on the following DMU levels: nurses, specialists, emergency, paediatric, mental/psychology units, systems, hospitals, psychiatric hospitals, nursing homes, veterans integrated service networks, acute care nursing units, ambulatory surgery centres, specialized inpatient cancer centres, dialysis, dialysis centres, dental providers, organ procurement organizations, skilled nursing facilities, community-based youth services, mental health cases, regions and area agencies on ageing. Studies that were not a journal article, book, review or editorial; studies that were not written in the English language; and studies with no full text. All papers that passed the inclusion criteria were subjected to full-text reading. Papers from the reference lists and bibliographies of the retrieved studies were also included. Finally, 54 papers were selected for analysis. Data collection process The key findings and conclusions of the eligible studies were identified by one author. An evidence table was used to extract information relevant to the study aim. As shown in online resource, Supplementary Table S1, this extracted information included the authors and year of publication, link, title, name and number of centres, country of study and key findings of the DEA model used (orientation and type); input and output categories were analysed systematically to ensure consistency between the eligible studies regarding the extracted data characteristics. Further consistency with the primary studies was ensured by sharing data between the authors. Synthesis of results A thematic analysis was performed of the results, tables and graphs of summary data of the studies; this analysis allowed a comparison of the key findings, conclusions and impact of study quality on results, and to identify the potential for publication bias. The inputs, outputs, categories and models were summarized and calculated, as were the descriptive statistics for categories (minimum, maximum, mode and median). The inputs and outputs from eligible papers were classified into categories. The PHC dimensions developed by Kringos et al. for primary care systems (31,38) were included as compound variables (Tables 1 and 2). Table 1. Input categories Number . Input categories . Sum of variables used from 54 publications per category . I Personnel 98 II PHC centre 33 III Consultations or visits 25 IV Referrals or hospitalization days 24 V Pharmaceuticals or prescriptions 23 VI Procedures, treatments, and services 20 VII Patients 11 VIII Investigations (laboratory tests, special examinations and imaging) 11 IX Quality 9 X Equipment and resources 8 XI Vaccination 4 XII Educational interventions 2 XIII Health program 1 Minimum number of used input variables per model 1 Maximum number of used input variables per model 24 Mode (the value that appears most often in the number of used input variables) per model 3 Median (the ‘middle’ value of the number of used input variables) per model 3 Number . Input categories . Sum of variables used from 54 publications per category . I Personnel 98 II PHC centre 33 III Consultations or visits 25 IV Referrals or hospitalization days 24 V Pharmaceuticals or prescriptions 23 VI Procedures, treatments, and services 20 VII Patients 11 VIII Investigations (laboratory tests, special examinations and imaging) 11 IX Quality 9 X Equipment and resources 8 XI Vaccination 4 XII Educational interventions 2 XIII Health program 1 Minimum number of used input variables per model 1 Maximum number of used input variables per model 24 Mode (the value that appears most often in the number of used input variables) per model 3 Median (the ‘middle’ value of the number of used input variables) per model 3 Open in new tab Table 1. Input categories Number . Input categories . Sum of variables used from 54 publications per category . I Personnel 98 II PHC centre 33 III Consultations or visits 25 IV Referrals or hospitalization days 24 V Pharmaceuticals or prescriptions 23 VI Procedures, treatments, and services 20 VII Patients 11 VIII Investigations (laboratory tests, special examinations and imaging) 11 IX Quality 9 X Equipment and resources 8 XI Vaccination 4 XII Educational interventions 2 XIII Health program 1 Minimum number of used input variables per model 1 Maximum number of used input variables per model 24 Mode (the value that appears most often in the number of used input variables) per model 3 Median (the ‘middle’ value of the number of used input variables) per model 3 Number . Input categories . Sum of variables used from 54 publications per category . I Personnel 98 II PHC centre 33 III Consultations or visits 25 IV Referrals or hospitalization days 24 V Pharmaceuticals or prescriptions 23 VI Procedures, treatments, and services 20 VII Patients 11 VIII Investigations (laboratory tests, special examinations and imaging) 11 IX Quality 9 X Equipment and resources 8 XI Vaccination 4 XII Educational interventions 2 XIII Health program 1 Minimum number of used input variables per model 1 Maximum number of used input variables per model 24 Mode (the value that appears most often in the number of used input variables) per model 3 Median (the ‘middle’ value of the number of used input variables) per model 3 Open in new tab Table 2. Output categories Number . Output categories . Sum of variables used from 54 publications per category . I Consultations or visits 83 II Patients 69 III Procedures, treatment, and services 45 IV Quality 43 V Personnel 31 VI Preventive interventions (including vaccinations) 18 VII PHC centre 11 VIII Referrals or hospitalizations 10 IX Educational interventions 8 X Investigations (laboratory tests, special examinations and imaging) 8 XI Pharmaceuticals or prescriptions 2 XII Equipment and resources 2 Minimum number of used output variables per model 1 Maximum number of used output variables per model 21 Mode (the value that appears most often in the number of used output variables) per model 3 Median (the ‘middle’ value of the number of used output variables) per model 4 Number . Output categories . Sum of variables used from 54 publications per category . I Consultations or visits 83 II Patients 69 III Procedures, treatment, and services 45 IV Quality 43 V Personnel 31 VI Preventive interventions (including vaccinations) 18 VII PHC centre 11 VIII Referrals or hospitalizations 10 IX Educational interventions 8 X Investigations (laboratory tests, special examinations and imaging) 8 XI Pharmaceuticals or prescriptions 2 XII Equipment and resources 2 Minimum number of used output variables per model 1 Maximum number of used output variables per model 21 Mode (the value that appears most often in the number of used output variables) per model 3 Median (the ‘middle’ value of the number of used output variables) per model 4 Open in new tab Table 2. Output categories Number . Output categories . Sum of variables used from 54 publications per category . I Consultations or visits 83 II Patients 69 III Procedures, treatment, and services 45 IV Quality 43 V Personnel 31 VI Preventive interventions (including vaccinations) 18 VII PHC centre 11 VIII Referrals or hospitalizations 10 IX Educational interventions 8 X Investigations (laboratory tests, special examinations and imaging) 8 XI Pharmaceuticals or prescriptions 2 XII Equipment and resources 2 Minimum number of used output variables per model 1 Maximum number of used output variables per model 21 Mode (the value that appears most often in the number of used output variables) per model 3 Median (the ‘middle’ value of the number of used output variables) per model 4 Number . Output categories . Sum of variables used from 54 publications per category . I Consultations or visits 83 II Patients 69 III Procedures, treatment, and services 45 IV Quality 43 V Personnel 31 VI Preventive interventions (including vaccinations) 18 VII PHC centre 11 VIII Referrals or hospitalizations 10 IX Educational interventions 8 X Investigations (laboratory tests, special examinations and imaging) 8 XI Pharmaceuticals or prescriptions 2 XII Equipment and resources 2 Minimum number of used output variables per model 1 Maximum number of used output variables per model 21 Mode (the value that appears most often in the number of used output variables) per model 3 Median (the ‘middle’ value of the number of used output variables) per model 4 Open in new tab The details of the included studies of the basic bibliographical information and all inputs and outputs used are presented in online resource (Supplementary Table S1). Risk of bias across studies The overall quality of the studies was assessed using the Quantitative Study Assessment Checklist developed at the Department of Computer Science, University of Auckland (39). All studies described the DEA method in detail; however, some did not include substantive information on the variables used to minimize selection bias. None of the eligible studies reported the theoretical or philosophical bases for methodological choice, which limited the ability to situate and assess methodological relevance. The risk of bias and the quality of individual selected studies were assessed by two members of the team working directly on the review, who independently evaluated each included paper. Doubts were adjudicated by a third, external reviewer. The criteria for assessing research quality were based on the Critical Appraisal Checklist for a Systematic Review adapted by the Department of General Practice, University of Glasgow, from the Critical Appraisal Skills Programme of the Institute of Health Science in Oxford (36). Results Study selection A total of 4231 papers were identified (639 in PubMed, 849 in MEDLINE Complete, 103 in Embase and 2640 in the Web of Science) for title and Abstract screening and manual selection. Eighty-one papers were retrieved from the screenings, with an additional 25 selected from their bibliographies, for a total of 108 papers selected for full-text review. After the full-text review based on a checklist, 54 papers were selected and analysed for inputs, outputs and models. The following numbers of papers were excluded for the following reasons: no full text available (25), no DEA (no input and no output) (17), no primary care (6), review paper (5) and editorial article (1). Figure 1 presents a flowchart of the search strategy results of the DEA method applications in PHC. Figure 1. Open in new tabDownload slide Flow diagram of the search strategy results of the data envelopment analysis method applications in primary health care. Study characteristics Most of the eligible studies were performed in Europe (24 studies), followed by North America (15), Africa (6), South America (6), Asia (2), and Australia and New Zeeland (1). The most eligible publications came from the USA (13), followed by Spain (6), the UK (6), Greece (4), Brazil (3) and Italy (3). Two studies per country were identified in Portugal, Sierra Leone and Burkina Faso, while only one each was identified from the Netherlands, Austria, Canada, Finland, Guatemala, Pakistan, Colombia, Chile, Mexico, New Zealand, South African, Ethiopia and Saudi Arabia. Inputs and outputs Inputs and outputs from the analysed papers were assigned to 13 and 12 thematic categories, respectively. Details of the inputs and outputs included in each category are presented in online resource Supplementary Table S1. The number of inputs used in a single study ranged from 1 (minimum) to 24 (maximum), while the outputs ranged from 1 (minimum) to 21 (maximum), with a modal value of three for both. The most frequently used input categories were personnel (associated with 98 variables), PHC centres (33), consultations or visits (25), referrals or hospitalizations (24), and pharmaceuticals or prescriptions (23) (Table 1). The most frequently used output categories were health care consultations or visits (83 variables from studies), patients (69), procedures, treatment, and services (45), quality (43), personnel (31), preventive interventions (including vaccinations) (18), and PHC centres (11) (Table 2). Eleven categories were represented in both the input and output groups. Models The efficiency of PHC centres was evaluated using various DEA models. The most commonly used single model was the Variable Returns to Scale (VRS) DEA, which was applied in 16 studies. In 13 publications, the efficiency was calculated using both the Constant Returns to Scale (CRS) and VRS DEA models. Fourteen publications used the CRS model. The most widely used DEA model was input orientation, which was applied in 22 papers. The characteristics of the eligible studies are presented in Supplementary Table S1. Discussion Summary of evidence This systematic review showed a number of approaches to quantitative evaluation to PHC activities with a scope of inputs and outputs used. These can be divided into thematic categories, with the variety of models which have been used. There is still room for improvement of the model in PHC applications. A total of 54 studies on DEA applications in PHC were identified with selections of inputs, outputs and models related to patients. This is a potential additional value of the DEA method: it offers researchers a wide selection of potential research questions associated with an adequate choice of the model and analysis parameters. Quantitative DEA-based studies can examine the effectiveness of a wide scope of processes in primary care, such as costs of provided care, medication, patient waiting time or chronic care delivery. We hope that future studies will confirm our expectations. Greater standardization of DEA is needed in further research considering PHC applications. Eleven categories were represented in both the input and output groups: e.g. for category as consultations or visits used as input (e.g. outpatient visits, the annual number of patient consultations with their physicians) and as the most frequently output (e.g. number of visits carried out by the community health workers; annual number of patient visits to each Primary Care Centre) (Table 2 and Supplementary Table S1). The choice of CRS or VRS and model orientation depends on the context. The number of publications related to DEA in all databases has increased over the last 5 years. Various DEA methods were used to estimate the efficiency of organizations in the health sector, with a variety of models being applied. DEA methods do not require any knowledge of the linkage between inputs and outputs to calculate efficiency. DEA use varies geographically, with most studies performed in the USA and the UK. In 1999, Garcia (8) found the efficiency of PHCs to be affected by intermediate outputs, which needed to be improved. These results confirmed that efficiency depends on the number of outputs and inputs and the choice of outputs for a specific unit of measurement (8). According to Pelone et al., primary care outcomes can be determined by general practice discretionary inputs (40). Input and output categories The main input and output categories can be seen from two perspectives. The first concerns PHC centres and patients, which addresses the patients, number of staff (GPs, nurses and administrators), costs, areas, procedures, prescriptions and referrals. The second concerns public health, which looks at health care systems and the optimal organizational achievement of primary care service delivery; their inputs include primary care governance, workforce development and economic conditions, and their outputs include comprehensiveness, access, coordination and service delivery indicators of access continuity and comprehensiveness of care. Ferreira et al. used another approach including staff expenditure as the most common input, with the different kinds of consultations related to the PHC being the most commonly used outputs (41). Models All of the included DEA applications were focussed on technical efficiency. Various DEA models were used to evaluate the efficiency of PHC units (e.g. primary care practices, district health authorities, physician practices, PHC centres). In 13 publications, the efficiency was calculated using both the CRS and VRS DEA models. The choice of a CRS or a VRS should depend on the context and the level of analysis. The CRS model assumes a linear, proportional change in outputs associated with changes in inputs, e.g. Chilingerian and Sherman employed the CRS DEA model, with the DMU as the individual primary care physician (42). The VRS model is appropriate when input or output variables are defined using ratios (43). While 16 studies used VRS, 14 used CRS (see Supplementary Table S1). Model orientations The most widely used DEA model was input orientation, which was applied in 22 papers. When choosing a DEA model, it is necessary to define initially if the input or output-oriented method will be used. In an input-oriented model, the goal is to minimize the use of inputs to maintain a constant level of outputs (input-oriented model producing a given output with minimum inputs). In the health care industry, outputs are less controllable than inputs. The choice of the input model is justified on the fact that managers in health care services tend to have greater control over inputs rather than outputs. From the point of view of the health care executives, it is easier to control inputs than health results, which is why reserchers choosing an input-oriented model (44). Cordero Ferrera et al. used an input orientation in primary care centre efficiency measurements because managers can determine only those resources attributed to each primary care centres and that the demand for health services cannot be controlled (45). However, regarding the reduction of expenditure in PHC services, Stefko et al. conclude that the health care sector is specific and that health care services should concentrate on increasing outputs rather than reducing inputs and costs (20). Oikonomou et al. chose an output-oriented model because the demand for PHC services has a tendency to expand rather than decrease. In an output-oriented DEA model, whose aim is to maximize the outputs with the given level of inputs, it is assumed that greater output is associated with technical efficiency (46). Methods for measuring the efficiency of health care sectors and national innovations most commonly were based on the input- or output-oriented DEA CRS model (47), although the super-efficiency DEA model, the DEA specification for bilateral comparison of two clusters of DMUs, and grey relational analysis with DEA models (48) were also used. Pelone et al. studied PHC efficiency using the DEA method and showed that scale efficiency scores depended on the DEA model orientation, the input–output variables used, and the restrictions incorporated into the DEA model (49). Our analysis revealed a gradual increase in the number of scientific publications related to the use of DEA methods. DEA appears to be the most commonly used tool used for analysing the efficiency of PHC organizations. Nevertheless, there is still room for improvement; further research is needed on DEA analysis, particularly the choice of inputs and outputs, as these affect the efficiency of the organizations examined. An interesting idea concerns the introduction of exogenous variables, which in addition to the allocation efficiency score for all units, also provide information about potential additional reductions in inputs or potential increases in outputs. These can be detected in specific cases by incorporating non-radial inefficiency or slacks to the DEA dual model (43). Finally, it has to be stressed that DEA scores depend on the choice of input and output variables, models and weighting. The efficiency score is relative. Although each organization can be compared with the reference organization, i.e. the best one, within a study, it is not possible to compare scores between separate efficiency studies. Moreover, the DEA technique ignores the noise in the data, and the efficiency measures are very sensitive to the sample size and outliers (50). In addition, there are no diagnostic tests to determine the validity of the model or to improve the model specification (51). Despite rising health care costs and the growing need for the financial sustainability of health care systems, the tools for analysing their efficiency, including DEA models, remain inadequate, and further studies on PHC organizational efficiency are needed. Limitations This review presents a quantitative tool for the assessment of the public domains of PHC, which despite their importance, are costly and prone to risk of shortage. However, this review has limitations. As it was limited to studies published in English in peer-reviewed journals, it is possible that other relevant published or unpublished studies and insights were missed. Some publications could be missed due to lack of access. Moreover, the screening process for some of the eligible studies was conducted by a single author, which may have affected the accuracy, reliability and transparency of the process. Conclusion This article, a review of state-of-the-art research describing the most commonly used groups of outputs and inputs, serves as a step towards the standardization of DEA. It was found that the most widely used model for efficiency orientation was input orientation. Although the number of studies based on DEA methods is gradually increasing and DEA is the most frequently used tool in the efficiency analysis of PHC organizations, there is still room for improvement. Further research is required to identify appropriate input and output variables and a suitable DEA model for assessing PHC. The standardization of DEA could extend the scope of research tools for the analysis of functioning the primary care. This would support health economic analyses of measurements of primary care efficiency. Declaration Funding: Narodowe Centrum Nauki (National Science Centre Poland) (2016/21/B/NZ7/02052). Ethical approval: none. Conflict of interest: none. Data availability The authors declare that the data supporting the study findings are available within the article. Acknowledgements The authors thank Katarzyna Kosiek, PhD, for reviewing the manuscript and Edward Lowczowski for English language assistance. Reference list 1. Xu K , Saksena P. The Determinants of Health Expenditure: A Country-Level Panel Data Analysis . World Health Organization . A Working Paper of the Results for Development Institute (R4D) [Internet], 2011 . http://www.who.int/health_financing/documents/report_en_11_deter-he.pdf?ua=1 (accessed on 5 April 2019 ). 2. WHO . 2016 Global Health Care Outlook. Battling Costs While Improving Care. (2016) “2016 Global Health Care Outlook Battling Costs While Improving Care” DTTL Global Life Sciences and Health Care (LSHC) Industry Group analysis, from [Internet] . 2016 . http://www2.deloitte.com/content/dam/Deloitte/global/Documents/Life-Sciences-Health-Care/gx-lshc-2016-health-care-outlook.pdf (accessed on 5 April 2019). 3. Mark G Shrime MG, Mukhopadhyayb S, Alkirea BC. Health-system-adapted data envelopment analysis for decisionmaking in universal health coverage . Bull World Health Organ 2018 ; 96 : 393 – 401 . Google Scholar Crossref Search ADS PubMed WorldCat 4. Jablonsky J , Emrouznejad A, Toloo M. Editorial: special issue on data envelopment analysis . Cent Eur J Oper Res 2018 ; 26 ( 4 ): 809 – 12 . Google Scholar Crossref Search ADS WorldCat 5. Emrouznejada A , Yang G. A survey and analysis of the first 40 years of scholarly literature in DEA: 1978–2016 . Socio-Econ Plan Sci 2018 ; 61 : 4 – 8 . Google Scholar Crossref Search ADS WorldCat 6. Ozcan YA , editor. Performance Measurement Using Data Envelopment Analysis (DEA) . New York, Heidelberg, Dordrecht and London : Springer , 2014 . Google Scholar Crossref Search ADS Google Preview WorldCat COPAC 7. Charnes A , Cooper WW, Rhodes E. Measuring the efficiency of decision making units . Eur J Oper Res 1978 ; 2 : 429 – 44 . 8. Garcia F , Marcuello C, Serrano D, Urbina O. Evaluation of efficiency in primary health care centres: an application of data envelopment analysis . Financial Acc & Man 1999 ; 15 ( 1 ): 67 – 83 . 9. Farrell MJ . The measurement of productive efficiency . J R Stat Soc Ser A (Gen) 1957 ; 120 ( 3 ): 253 – 81 . Google Scholar Crossref Search ADS WorldCat 10. Cwiakala-Malys A , Nowak W. Classification of data envelopment analysis models [Sposoby klasyfikacji modeli DEA] . Bad Oper Decyzje 2009 ; 3 : 5 – 18 . (in Polish) Google Scholar OpenURL Placeholder Text WorldCat 11. Dyson RG , Shale EA. Data envelopment analysis, operational research and uncertainty . J Oper Res Soc 2010 ; 61 ( 1 ): 25 – 34 . Google Scholar Crossref Search ADS WorldCat 12. Del Rocío Moreno-Enguix M , Gómez-Gallego JC, Gómez Gallego M. Analysis and determination the efficiency of the European health systems . Int J Health Plann Manage 2018 ; 33 : 136 – 54 . Google Scholar Crossref Search ADS PubMed WorldCat 13. Pourreza A , Alipour V, Arabloo J, Bayati M, Ahadinezhad B. Health production and determinants of health systems performance in WHO Eastern Mediterranean Region . East Mediterr Health J 2017 ; 23 : 368 – 74 . Google Scholar Crossref Search ADS PubMed WorldCat 14. Zhang L , Cheng G, Song S et al. Efficiency performance of China’s health care delivery system . Int J Health Plann Manage 2017 ; 32 : 254 – 63 . Google Scholar Crossref Search ADS PubMed WorldCat 15. Erus B , Hatipoglu O. Physician payment schemes and physician productivity: analysis of Turkish healthcare reforms . Health Policy 2017 ; 121 : 553 – 7 . Google Scholar Crossref Search ADS PubMed WorldCat 16. Ding JM , Hu XJ, Zhang XZ, Shang L, Yu M, Chen HL. Equity and efficiency of medical service systems at the provincial level of China’s mainland: a comparative study from 2009 to 2014 . Bmc Public Health 2018 ; 18 ( 1 ): 214 . Google Scholar Crossref Search ADS PubMed WorldCat 17. Zhang Y , Wang Q, Jiang T, Wang J. Equity and efficiency of primary health care resource allocation in mainland China . Int J Equity Health 2018 ; 17 : 140 . Google Scholar Crossref Search ADS PubMed WorldCat 18. Dlouhý M . Measuring geographic inequalities: dealing with multiple health resources by data envelopment analysis . Front Public Health 2018 ; 6 : 53 . Google Scholar Crossref Search ADS PubMed WorldCat 19. Campos-Lucena MSS-D , Sanz Diaz MT, Morente FV, Ovando RY. The efficiency of health services in Spain based on Data Envelopment Analysis (DEA). The singularity of health efficiency . Int J Collab Res Inter Med Public Health 2018 ; 10 ( 1 ): 776 – 91 . Google Scholar OpenURL Placeholder Text WorldCat 20. Stefko R , Gavurova B, Kocisova K. Healthcare efficiency assessment using DEA analysis in the Slovak Republic . Health Econ Rev 2018 ; 8 ( 1 ): 6 . Google Scholar Crossref Search ADS PubMed WorldCat 21. Barouni M , Amiresmaieli MR, Shahravan A, Amini S. The efficiency assessment of dental units using data envelopment analysis approach: the case of Iran . Iran J Public Health 2017 ; 46 : 552 – 9 . Google Scholar PubMed OpenURL Placeholder Text WorldCat 22. Fiallos J , Patrick J, Michalowski W, Farion K. Using data envelopment analysis for assessing the performance of pediatric emergency department physicians . Health Care Manag Sci 2017 ; 20 : 129 – 40 . Google Scholar Crossref Search ADS PubMed WorldCat 23. Sowlati T. Establishing the Practical Frontier in Data Envelopment Analysis . PhD Thesis. Ontario, Canada : University of Toronto , 2001 . Google Scholar Google Preview OpenURL Placeholder Text WorldCat COPAC 24. Ferreira DC , Nunes AM, Marques RC. Doctors, nurses, and the optimal scale size in the Portuguese public hospitals . Health Policy 2018 ; 122 : 1093 – 100 . Google Scholar Crossref Search ADS PubMed WorldCat 25. Johannessen KA , Kittelsen SA, Hagen TP. Assessing physician productivity following Norwegian hospital reform: a panel and data envelopment analysis . Soc Sci Med 2017 ; 175 : 117 – 26 . Google Scholar Crossref Search ADS PubMed WorldCat 26. Liu JS , Lu LYY, Lu WM, Lin BJY. Data envelopment analysis 1978–2010: a citation-based literature survey . Omega Int J Manag Sci 2013 ; 41 ( 1 ): 3 – 15 . Google Scholar Crossref Search ADS WorldCat 27. Liu JS , Lu LYY, Lu WM, Lin BJY. A survey of DEA applications . Omega In J Manag Sci 2013 ; 41 ( 5 ): 893 – 2 . Google Scholar Crossref Search ADS WorldCat 28. Amado CAF , Dyson RG. On comparing the performance of primary care providers . Eur J Oper Res 2008 ; 185 ( 3 ): 915 – 32 . Google Scholar Crossref Search ADS WorldCat 29. Hollingsworth B . The measurement of efficiency and productivity of health care delivery . Health Econ 2008 ; 17 : 1107 – 28 . Google Scholar Crossref Search ADS PubMed WorldCat 30. Hollingsworth B , Dawson PJ, Maniadakis N. Efficiency measurement of health care: a review of non-parametric methods and applications . Health Care Manag Sci 1999 ; 2 : 161 – 72 . Google Scholar Crossref Search ADS PubMed WorldCat 31. Kringos DSB , Boerma WGW, Hutchinson A, Saltman RB. Building Primary Care in a Changing Europe . 2015 . http://www.euro.who.int/__data/assets/pdf_file/0018/271170/BuildingPrimaryCareChangingEurope.pdf?ua=1 (accessed on 14 June 2017 ). 32. Cooper WW , Seiford LM, Zhu J. Data envelopment analysis. history, models and interpretations . In: Handbook on Data Envelopment Analysis . New York : Springer , 2011 , pp. 1 – 39 . Google Scholar Crossref Search ADS Google Preview WorldCat COPAC 33. Domagala A . Data envelopment analysis – a method for measuring the relative technical efficiency [Metoda data envelopment analysis jako narzędzie badania względnej efektywności technicznej] . Bad Oper Decyzje 2007 ; 3–4 : 21 – 34 . (in Polish) Google Scholar OpenURL Placeholder Text WorldCat 34. PROSPERO . International Prospective Register of Systematic Reviews . York, UK : National Institute of Health Research (NIHR), University of York . https://www.crd.york.ac.uk/PROSPERO/ (accessed on 22 September 2019 ). 35. García-Alonso CR , Almeda N, Salinas-Pérez JA, Gutiérrez-Colosía MR, Salvador-Carulla L. Relative Technical Efficiency Assessment of the technical efficiency of Mental Health Services: a systematic review . Adm Policy Ment Health 2019 ; 46 ( 4 ): 429 – 44 . 36. CASP U . Critical Appraisal Skills Programme (CASP) . Oxford, UK : CASP , 2017 . http://www.casp-uk.net/, https://casp-uk.net/wp-content/uploads/2018/03/CASP-Systematic-Review-Checklist-2018_fillable-form.pdf (accessed on 22 September 2019 ). 37. Moher D , Liberati A, Tetzlaff J, Altman DG; PRISMA Group. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement . PLoS Med 2009 ; 6 : e1000097 . Google Scholar Crossref Search ADS PubMed WorldCat 38. Kringos DS , Boerma WGW, Hutchinson A, van der Zee Software J, Groenewegen PP. The breadth of primary care: a systematic literature review of its core dimensions . BMC Health Serv Res 2010 ; 10 : 65 . Google Scholar Crossref Search ADS PubMed WorldCat 39. Riaz M , Mendes E, Tempero E. A systematic review of software maintainability prediction and metrics 2009 3rd International Symposium on Empirical Software Engineering and Measurement . https://ieeexplore.ieee.org/document/5314233 (accessed on 13 September 2019 ). 40. Pelone F , Kringos DS, Valerio L et al. The measurement of relative efficiency of general practice and the implications for policy makers . Health Policy 2012 ; 107 ( 2–3 ): 258 – 68 . Google Scholar Crossref Search ADS PubMed WorldCat 41. Ferreira C , Marques RC, Nicola P. On evaluating health centers groups in Lisbon and Tagus Valley: efficiency, equity and quality . BMC Health Serv Res 2013 ; 13 : 529 . Google Scholar Crossref Search ADS PubMed WorldCat 42. Chilingerian JA , Sherman HD. Benchmarking physician practice patterns with DEA: a multi-stage approach for cost containment . Ann Oper Res 1996 ; 67 ( 1 ): 83 – 116 . Google Scholar Crossref Search ADS WorldCat 43. Manuel Cordero-Ferrera J , Crespo-Cebada E, Murillo-Zamorano LR. Measuring technical efficiency in primary health care: the effect of exogenous variables on results . J Med Syst 2011 ; 35 ( 4 ): 545 – 54 . Google Scholar Crossref Search ADS PubMed WorldCat 44. Bahurmoz A . Measuring efficiency in primary health care centres in Saudi Arabia . J KAU Econ Adm 1998 ; 12 ( 2 ): 3 – 18 . Google Scholar Crossref Search ADS WorldCat 45. Cordero Ferrera JM , Crespo Cebada E, Murillo Zamorano LR. The effect of quality and socio-demographic variables on efficiency measures in primary health care . Eur J Health Econ 2014 ; 15 : 289 – 302 . Google Scholar Crossref Search ADS PubMed WorldCat 46. Oikonomou N , Tountas Y, Mariolis A et al. Measuring the efficiency of the Greek rural primary health care using a restricted DEA model; the case of southern and western Greece . Health Care Manag Sci 2016 ; 19 : 313 – 25 . Google Scholar Crossref Search ADS PubMed WorldCat 47. Cantor VJM , Poh KL. Integrated analysis of healthcare efficiency: a systematic review . J Med Syst 2017 ; 42 : 8 . Google Scholar Crossref Search ADS PubMed WorldCat 48. Kotsemir MN . Research Paper. Measuring National Innovation Systems Efficiency – A Review of DEA Approach . 2013 , p. 24 . https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2304735 (accessed on 5 April 2019 ). 49. Pelone F , Kringos DS, Romaniello A et al. Primary care efficiency measurement using data envelopment analysis: a systematic review . J Med Syst 2015 ; 39 : 156 . Google Scholar Crossref Search ADS PubMed WorldCat 50. Ehrgott M , Holder A, Nohadani O. Uncertain data envelopment analysis . Eur J Oper Res 2018 ; 268 ( 1 ): 231 – 42 . Google Scholar Crossref Search ADS WorldCat 51. vanVeen SHCM. Comparative Efficiency Analysis from the Perspective of the Dutch Health Care Insurer: Determining the Usefulness of Efficiency Measures for Contracting Primary Care Organizations . Rotterdam, The Netherlands : Erasmus University Rotterdam , 2010 . Google Scholar Google Preview OpenURL Placeholder Text WorldCat COPAC © The Author(s) 2019. Published by Oxford University Press. This is an Open Access article distributed under the terms of the Creative Commons Attribution Non-Commercial License (http://creativecommons.org/licenses/by-nc/4.0/), which permits non-commercial re-use, distribution, and reproduction in any medium, provided the original work is properly cited. For commercial re-use, please contact journals.permissions@oup.com © The Author(s) 2019. Published by Oxford University Press. http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png Family Practice Oxford University Press

Data envelopment analysis applications in primary health care: a systematic review

Zakowska, Izabela; Godycki-Cwirko, Maciek
Family Practice , Volume 37 (2) – Mar 25, 2020
Download PDF
7 pages

Loading next page...
 
/lp/oxford-university-press/data-envelopment-analysis-applications-in-primary-health-care-a-zZoPc6UoMI

References (50)

Publisher
Oxford University Press
Copyright
Copyright © 2022 Oxford University Press
ISSN
0263-2136
eISSN
1460-2229
DOI
10.1093/fampra/cmz057
Publisher site
See Article on Publisher Site

Abstract

Abstract Background Strategic management of primary health care centres is necessary for creating an efficient global health care system that delivers good care. Objectives To perform a systematic literature review of the use of data envelopment analysis in estimating the relative technical efficiency of primary health care centres, and to identify the inputs, outputs and models used. Methods PubMed, MEDLINE Complete, Embase and Web of Science were searched for papers published before the 25 March 2019. Results Of a total of 4231 search results, 54 studies met the inclusion criteria. The identified inputs included personnel costs, gross expenditures, referrals and days of hospitalization, as well as prescriptions and investigations. Outputs included consultations or visits, registered patients, procedures, treatments and services, prescriptions and investigations. A variety of data envelopment analysis models used was identified, with no standard approach. Conclusions Data envelopment analysis extends the scope of tools used to analyse primary care functioning. It can support health economic analyses when assessing primary care efficiency. The main issues are setting outputs and inputs and selecting a model best suited for the range of products and services in the primary health care sector. This article serves as a step forward in the standardization of data envelopment analysis, but further research is needed. Health economics, health information, primary care, public health, quality of care, systematic review Key Messages We identify various approaches to effective assessment of primary care. Data envelopment analysis is widely used in primary care. A variety of analysis models exist, with no single standard. Introduction Many countries have a growing demand for health care services and this is accompanied by growing expenditure (1–3). Measuring the relative efficiency of health care systems, including primary health care (PHC) centres, creates a baseline for evaluating the management of their resources and for benchmarking their productiveness against others. Moreover, better use of health care resources could lead to the provision of better health services at the basic level. Recent years have seen a growth in interest in the use of quantitative methods for comparing the efficiency of health care systems (2,4,5), and many approaches for estimating the efficiency of various levels of health care organizations are under intensive investigation (6). Data envelopment analysis (DEA) (7) is a non-parametric, deterministic alternative to several efficiency measurement techniques, such as the cost-effectiveness ratio, corrected ordinary least squares or stochastic frontier analysis. DEA (7) uses a linear programming technique that gives a single measure of efficiency. It is based on the principle that an organization can be considered efficient when it is able to obtain the greatest output, in terms of goods, products or feasible services, by using a certain combination of used resources as input or, alternatively, when it produces a certain level of output using the least possible input (8,9). Since its inception 40 years ago, DEA has been extensively investigated and applied as a tool (10). Despite its uncertainty (11), it is a promising way to estimate the efficiency of units in many areas, including health care systems (12,13), reform (14,15), PHC centres (see 54 publications in Supplementary Table S1), regions (16–20), dental units (21), emergency departments (22) and hospitals (10,23–25). Systematic reviews have surveyed the literature associated with DEA and identified the most influential journals in the fields of DEA and its applications in the period 1978–2019 (26–30). DEA is a well-developed method used in health care efficiency assessment, and its use is still being refined (31). It can be used to evaluate operating organizations, establish criteria to improve their functioning and measure their progress. Despite the considerable body of literature surrounding DEA published over the last 20 years (32), including applications in Poland (33), a search of PROSPERO revealed only four protocols for the systematic review of assessments of the efficiency of health services using DEA (34,35). The aim of the present article is to systematically review empirical studies of DEA applications in the field of PHC to identify the most commonly employed group inputs, outputs and models. Our findings will facilitate the development of a standard set of criteria for the design and execution of DEA in PHC and may prove valuable for the standardization of DEA outcomes. This study serves as a step towards the standardization of DEA as the most widely used tool for improving the efficiency of PHC organizations and contributes to the refinement of DEA as a methodology. Methods Our review employs the approach described by the Institute of Health Science in Oxford, adopted for General Practice by Department of General Practice, University of Glasgow, as part of their Critical Appraisal Skills Programme (36). This review was reported according to the Preferred Reporting Item for Systematic Reviews and Meta-Analysis (PRISMA) approach (37). The review corpus comprises studies of health care technical efficiency based in the primary care setting. The list of included studies was restricted to those concerning decision-making units (DMUs) such as PHC centres, physicians, family physicians, primary care physicians, GPs, general practices and health maintenance organizations (HMOs). Search strategy A systematic electronic search was performed according to PRISMA (37) between March 2017 and March 2019, which covered the studies published before 25 March 2019. The literature search was performed by two independent researchers (IZ and MGC) in four electronic databases: PubMed, MEDLINE Complete (Medical Literature Analysis and Retrieval System Online), Embase (Excerpta Medica Database) and Web of Science. The search terms and filters listed in Box 1 were used. The identified papers were limited to full-text original and review articles published in English. Box 1. Search headings – Efficiency [MeSH Major Topic] – Benchmarking [MeSH Major Topic] – Benchmarking [Title/Abstract] – ‘Data Envelopment Analysis’ [Title/Abstract] – DEA[Title/Abstract] – ‘Technical Efficiency’ [Title/Abstract] – #1 OR #2 OR #3 OR #4 OR #5 OR #6 – ‘Primary Health Care’ [MeSH Major Topic] – ‘Physicians, Primary Care’ [MeSH Major Topic] – ‘Primary Health Care’ [Title/Abstract] – ‘General Practice*’ [Title/Abstract] – ‘Physicians*’ [Title/Abstract] – ‘HMO*’ [Title/Abstract] – #8 OR #9 OR #10 OR #11 OR #12 OR #13 – #7 AND #14 Filters: (Journal Article OR Review) AND Full text AND English Study selection The retrieved papers were imported into EndNote X4. Duplicates were identified and removed. The two researchers independently manually screened the titles and Abstracts to select relevant papers. Any disagreements were resolved by discussions with an external expert. When the articles had insufficient information in the title and Abstract to support this screening, a full-text reading was conducted. Following this, all potentially eligible papers were added in full-text form. A manual search was then used to retrieve papers for full-text review. These papers were examined by two authors using a checklist designed for this study, with inclusion and exclusion criteria given in Box 2. The list of included studies was restricted to those concerning DMUs such as PHC centres, physicians, family physicians, primary care physicians, GPs, general practices and HMOs. Box 2. Inclusion and exclusion criteria Inclusion Studies that met the following criteria were included inputs and outputs used to evaluate the technical efficiency of PHC centres or physicians, using the DEA method. The studies concentrated on the efficiency of PHC centres as the organization as a whole, including the physicians’ efficiency working in these centres. Studies that included the following DMU levels: PHC centres, physicians, family physicians, primary care physicians, GPs, general practices, and health maintenance organizations (HMO). Studies written as a full-text journal article, in the English language. Exclusion Studies that were not based on the DEA method: no inputs, no outputs, no models, or no technical efficiency calculations, and not refer to primary health care. In addition, those that did refer to technical efficiency of quality/satisfaction, disease, treatment/drug/therapy, e-health/computer, reform/system, programmes, statistics, or education/training. Studies that were based on the following DMU levels: nurses, specialists, emergency, paediatric, mental/psychology units, systems, hospitals, psychiatric hospitals, nursing homes, veterans integrated service networks, acute care nursing units, ambulatory surgery centres, specialized inpatient cancer centres, dialysis, dialysis centres, dental providers, organ procurement organizations, skilled nursing facilities, community-based youth services, mental health cases, regions and area agencies on ageing. Studies that were not a journal article, book, review or editorial; studies that were not written in the English language; and studies with no full text. All papers that passed the inclusion criteria were subjected to full-text reading. Papers from the reference lists and bibliographies of the retrieved studies were also included. Finally, 54 papers were selected for analysis. Data collection process The key findings and conclusions of the eligible studies were identified by one author. An evidence table was used to extract information relevant to the study aim. As shown in online resource, Supplementary Table S1, this extracted information included the authors and year of publication, link, title, name and number of centres, country of study and key findings of the DEA model used (orientation and type); input and output categories were analysed systematically to ensure consistency between the eligible studies regarding the extracted data characteristics. Further consistency with the primary studies was ensured by sharing data between the authors. Synthesis of results A thematic analysis was performed of the results, tables and graphs of summary data of the studies; this analysis allowed a comparison of the key findings, conclusions and impact of study quality on results, and to identify the potential for publication bias. The inputs, outputs, categories and models were summarized and calculated, as were the descriptive statistics for categories (minimum, maximum, mode and median). The inputs and outputs from eligible papers were classified into categories. The PHC dimensions developed by Kringos et al. for primary care systems (31,38) were included as compound variables (Tables 1 and 2). Table 1. Input categories Number . Input categories . Sum of variables used from 54 publications per category . I Personnel 98 II PHC centre 33 III Consultations or visits 25 IV Referrals or hospitalization days 24 V Pharmaceuticals or prescriptions 23 VI Procedures, treatments, and services 20 VII Patients 11 VIII Investigations (laboratory tests, special examinations and imaging) 11 IX Quality 9 X Equipment and resources 8 XI Vaccination 4 XII Educational interventions 2 XIII Health program 1 Minimum number of used input variables per model 1 Maximum number of used input variables per model 24 Mode (the value that appears most often in the number of used input variables) per model 3 Median (the ‘middle’ value of the number of used input variables) per model 3 Number . Input categories . Sum of variables used from 54 publications per category . I Personnel 98 II PHC centre 33 III Consultations or visits 25 IV Referrals or hospitalization days 24 V Pharmaceuticals or prescriptions 23 VI Procedures, treatments, and services 20 VII Patients 11 VIII Investigations (laboratory tests, special examinations and imaging) 11 IX Quality 9 X Equipment and resources 8 XI Vaccination 4 XII Educational interventions 2 XIII Health program 1 Minimum number of used input variables per model 1 Maximum number of used input variables per model 24 Mode (the value that appears most often in the number of used input variables) per model 3 Median (the ‘middle’ value of the number of used input variables) per model 3 Open in new tab Table 1. Input categories Number . Input categories . Sum of variables used from 54 publications per category . I Personnel 98 II PHC centre 33 III Consultations or visits 25 IV Referrals or hospitalization days 24 V Pharmaceuticals or prescriptions 23 VI Procedures, treatments, and services 20 VII Patients 11 VIII Investigations (laboratory tests, special examinations and imaging) 11 IX Quality 9 X Equipment and resources 8 XI Vaccination 4 XII Educational interventions 2 XIII Health program 1 Minimum number of used input variables per model 1 Maximum number of used input variables per model 24 Mode (the value that appears most often in the number of used input variables) per model 3 Median (the ‘middle’ value of the number of used input variables) per model 3 Number . Input categories . Sum of variables used from 54 publications per category . I Personnel 98 II PHC centre 33 III Consultations or visits 25 IV Referrals or hospitalization days 24 V Pharmaceuticals or prescriptions 23 VI Procedures, treatments, and services 20 VII Patients 11 VIII Investigations (laboratory tests, special examinations and imaging) 11 IX Quality 9 X Equipment and resources 8 XI Vaccination 4 XII Educational interventions 2 XIII Health program 1 Minimum number of used input variables per model 1 Maximum number of used input variables per model 24 Mode (the value that appears most often in the number of used input variables) per model 3 Median (the ‘middle’ value of the number of used input variables) per model 3 Open in new tab Table 2. Output categories Number . Output categories . Sum of variables used from 54 publications per category . I Consultations or visits 83 II Patients 69 III Procedures, treatment, and services 45 IV Quality 43 V Personnel 31 VI Preventive interventions (including vaccinations) 18 VII PHC centre 11 VIII Referrals or hospitalizations 10 IX Educational interventions 8 X Investigations (laboratory tests, special examinations and imaging) 8 XI Pharmaceuticals or prescriptions 2 XII Equipment and resources 2 Minimum number of used output variables per model 1 Maximum number of used output variables per model 21 Mode (the value that appears most often in the number of used output variables) per model 3 Median (the ‘middle’ value of the number of used output variables) per model 4 Number . Output categories . Sum of variables used from 54 publications per category . I Consultations or visits 83 II Patients 69 III Procedures, treatment, and services 45 IV Quality 43 V Personnel 31 VI Preventive interventions (including vaccinations) 18 VII PHC centre 11 VIII Referrals or hospitalizations 10 IX Educational interventions 8 X Investigations (laboratory tests, special examinations and imaging) 8 XI Pharmaceuticals or prescriptions 2 XII Equipment and resources 2 Minimum number of used output variables per model 1 Maximum number of used output variables per model 21 Mode (the value that appears most often in the number of used output variables) per model 3 Median (the ‘middle’ value of the number of used output variables) per model 4 Open in new tab Table 2. Output categories Number . Output categories . Sum of variables used from 54 publications per category . I Consultations or visits 83 II Patients 69 III Procedures, treatment, and services 45 IV Quality 43 V Personnel 31 VI Preventive interventions (including vaccinations) 18 VII PHC centre 11 VIII Referrals or hospitalizations 10 IX Educational interventions 8 X Investigations (laboratory tests, special examinations and imaging) 8 XI Pharmaceuticals or prescriptions 2 XII Equipment and resources 2 Minimum number of used output variables per model 1 Maximum number of used output variables per model 21 Mode (the value that appears most often in the number of used output variables) per model 3 Median (the ‘middle’ value of the number of used output variables) per model 4 Number . Output categories . Sum of variables used from 54 publications per category . I Consultations or visits 83 II Patients 69 III Procedures, treatment, and services 45 IV Quality 43 V Personnel 31 VI Preventive interventions (including vaccinations) 18 VII PHC centre 11 VIII Referrals or hospitalizations 10 IX Educational interventions 8 X Investigations (laboratory tests, special examinations and imaging) 8 XI Pharmaceuticals or prescriptions 2 XII Equipment and resources 2 Minimum number of used output variables per model 1 Maximum number of used output variables per model 21 Mode (the value that appears most often in the number of used output variables) per model 3 Median (the ‘middle’ value of the number of used output variables) per model 4 Open in new tab The details of the included studies of the basic bibliographical information and all inputs and outputs used are presented in online resource (Supplementary Table S1). Risk of bias across studies The overall quality of the studies was assessed using the Quantitative Study Assessment Checklist developed at the Department of Computer Science, University of Auckland (39). All studies described the DEA method in detail; however, some did not include substantive information on the variables used to minimize selection bias. None of the eligible studies reported the theoretical or philosophical bases for methodological choice, which limited the ability to situate and assess methodological relevance. The risk of bias and the quality of individual selected studies were assessed by two members of the team working directly on the review, who independently evaluated each included paper. Doubts were adjudicated by a third, external reviewer. The criteria for assessing research quality were based on the Critical Appraisal Checklist for a Systematic Review adapted by the Department of General Practice, University of Glasgow, from the Critical Appraisal Skills Programme of the Institute of Health Science in Oxford (36). Results Study selection A total of 4231 papers were identified (639 in PubMed, 849 in MEDLINE Complete, 103 in Embase and 2640 in the Web of Science) for title and Abstract screening and manual selection. Eighty-one papers were retrieved from the screenings, with an additional 25 selected from their bibliographies, for a total of 108 papers selected for full-text review. After the full-text review based on a checklist, 54 papers were selected and analysed for inputs, outputs and models. The following numbers of papers were excluded for the following reasons: no full text available (25), no DEA (no input and no output) (17), no primary care (6), review paper (5) and editorial article (1). Figure 1 presents a flowchart of the search strategy results of the DEA method applications in PHC. Figure 1. Open in new tabDownload slide Flow diagram of the search strategy results of the data envelopment analysis method applications in primary health care. Study characteristics Most of the eligible studies were performed in Europe (24 studies), followed by North America (15), Africa (6), South America (6), Asia (2), and Australia and New Zeeland (1). The most eligible publications came from the USA (13), followed by Spain (6), the UK (6), Greece (4), Brazil (3) and Italy (3). Two studies per country were identified in Portugal, Sierra Leone and Burkina Faso, while only one each was identified from the Netherlands, Austria, Canada, Finland, Guatemala, Pakistan, Colombia, Chile, Mexico, New Zealand, South African, Ethiopia and Saudi Arabia. Inputs and outputs Inputs and outputs from the analysed papers were assigned to 13 and 12 thematic categories, respectively. Details of the inputs and outputs included in each category are presented in online resource Supplementary Table S1. The number of inputs used in a single study ranged from 1 (minimum) to 24 (maximum), while the outputs ranged from 1 (minimum) to 21 (maximum), with a modal value of three for both. The most frequently used input categories were personnel (associated with 98 variables), PHC centres (33), consultations or visits (25), referrals or hospitalizations (24), and pharmaceuticals or prescriptions (23) (Table 1). The most frequently used output categories were health care consultations or visits (83 variables from studies), patients (69), procedures, treatment, and services (45), quality (43), personnel (31), preventive interventions (including vaccinations) (18), and PHC centres (11) (Table 2). Eleven categories were represented in both the input and output groups. Models The efficiency of PHC centres was evaluated using various DEA models. The most commonly used single model was the Variable Returns to Scale (VRS) DEA, which was applied in 16 studies. In 13 publications, the efficiency was calculated using both the Constant Returns to Scale (CRS) and VRS DEA models. Fourteen publications used the CRS model. The most widely used DEA model was input orientation, which was applied in 22 papers. The characteristics of the eligible studies are presented in Supplementary Table S1. Discussion Summary of evidence This systematic review showed a number of approaches to quantitative evaluation to PHC activities with a scope of inputs and outputs used. These can be divided into thematic categories, with the variety of models which have been used. There is still room for improvement of the model in PHC applications. A total of 54 studies on DEA applications in PHC were identified with selections of inputs, outputs and models related to patients. This is a potential additional value of the DEA method: it offers researchers a wide selection of potential research questions associated with an adequate choice of the model and analysis parameters. Quantitative DEA-based studies can examine the effectiveness of a wide scope of processes in primary care, such as costs of provided care, medication, patient waiting time or chronic care delivery. We hope that future studies will confirm our expectations. Greater standardization of DEA is needed in further research considering PHC applications. Eleven categories were represented in both the input and output groups: e.g. for category as consultations or visits used as input (e.g. outpatient visits, the annual number of patient consultations with their physicians) and as the most frequently output (e.g. number of visits carried out by the community health workers; annual number of patient visits to each Primary Care Centre) (Table 2 and Supplementary Table S1). The choice of CRS or VRS and model orientation depends on the context. The number of publications related to DEA in all databases has increased over the last 5 years. Various DEA methods were used to estimate the efficiency of organizations in the health sector, with a variety of models being applied. DEA methods do not require any knowledge of the linkage between inputs and outputs to calculate efficiency. DEA use varies geographically, with most studies performed in the USA and the UK. In 1999, Garcia (8) found the efficiency of PHCs to be affected by intermediate outputs, which needed to be improved. These results confirmed that efficiency depends on the number of outputs and inputs and the choice of outputs for a specific unit of measurement (8). According to Pelone et al., primary care outcomes can be determined by general practice discretionary inputs (40). Input and output categories The main input and output categories can be seen from two perspectives. The first concerns PHC centres and patients, which addresses the patients, number of staff (GPs, nurses and administrators), costs, areas, procedures, prescriptions and referrals. The second concerns public health, which looks at health care systems and the optimal organizational achievement of primary care service delivery; their inputs include primary care governance, workforce development and economic conditions, and their outputs include comprehensiveness, access, coordination and service delivery indicators of access continuity and comprehensiveness of care. Ferreira et al. used another approach including staff expenditure as the most common input, with the different kinds of consultations related to the PHC being the most commonly used outputs (41). Models All of the included DEA applications were focussed on technical efficiency. Various DEA models were used to evaluate the efficiency of PHC units (e.g. primary care practices, district health authorities, physician practices, PHC centres). In 13 publications, the efficiency was calculated using both the CRS and VRS DEA models. The choice of a CRS or a VRS should depend on the context and the level of analysis. The CRS model assumes a linear, proportional change in outputs associated with changes in inputs, e.g. Chilingerian and Sherman employed the CRS DEA model, with the DMU as the individual primary care physician (42). The VRS model is appropriate when input or output variables are defined using ratios (43). While 16 studies used VRS, 14 used CRS (see Supplementary Table S1). Model orientations The most widely used DEA model was input orientation, which was applied in 22 papers. When choosing a DEA model, it is necessary to define initially if the input or output-oriented method will be used. In an input-oriented model, the goal is to minimize the use of inputs to maintain a constant level of outputs (input-oriented model producing a given output with minimum inputs). In the health care industry, outputs are less controllable than inputs. The choice of the input model is justified on the fact that managers in health care services tend to have greater control over inputs rather than outputs. From the point of view of the health care executives, it is easier to control inputs than health results, which is why reserchers choosing an input-oriented model (44). Cordero Ferrera et al. used an input orientation in primary care centre efficiency measurements because managers can determine only those resources attributed to each primary care centres and that the demand for health services cannot be controlled (45). However, regarding the reduction of expenditure in PHC services, Stefko et al. conclude that the health care sector is specific and that health care services should concentrate on increasing outputs rather than reducing inputs and costs (20). Oikonomou et al. chose an output-oriented model because the demand for PHC services has a tendency to expand rather than decrease. In an output-oriented DEA model, whose aim is to maximize the outputs with the given level of inputs, it is assumed that greater output is associated with technical efficiency (46). Methods for measuring the efficiency of health care sectors and national innovations most commonly were based on the input- or output-oriented DEA CRS model (47), although the super-efficiency DEA model, the DEA specification for bilateral comparison of two clusters of DMUs, and grey relational analysis with DEA models (48) were also used. Pelone et al. studied PHC efficiency using the DEA method and showed that scale efficiency scores depended on the DEA model orientation, the input–output variables used, and the restrictions incorporated into the DEA model (49). Our analysis revealed a gradual increase in the number of scientific publications related to the use of DEA methods. DEA appears to be the most commonly used tool used for analysing the efficiency of PHC organizations. Nevertheless, there is still room for improvement; further research is needed on DEA analysis, particularly the choice of inputs and outputs, as these affect the efficiency of the organizations examined. An interesting idea concerns the introduction of exogenous variables, which in addition to the allocation efficiency score for all units, also provide information about potential additional reductions in inputs or potential increases in outputs. These can be detected in specific cases by incorporating non-radial inefficiency or slacks to the DEA dual model (43). Finally, it has to be stressed that DEA scores depend on the choice of input and output variables, models and weighting. The efficiency score is relative. Although each organization can be compared with the reference organization, i.e. the best one, within a study, it is not possible to compare scores between separate efficiency studies. Moreover, the DEA technique ignores the noise in the data, and the efficiency measures are very sensitive to the sample size and outliers (50). In addition, there are no diagnostic tests to determine the validity of the model or to improve the model specification (51). Despite rising health care costs and the growing need for the financial sustainability of health care systems, the tools for analysing their efficiency, including DEA models, remain inadequate, and further studies on PHC organizational efficiency are needed. Limitations This review presents a quantitative tool for the assessment of the public domains of PHC, which despite their importance, are costly and prone to risk of shortage. However, this review has limitations. As it was limited to studies published in English in peer-reviewed journals, it is possible that other relevant published or unpublished studies and insights were missed. Some publications could be missed due to lack of access. Moreover, the screening process for some of the eligible studies was conducted by a single author, which may have affected the accuracy, reliability and transparency of the process. Conclusion This article, a review of state-of-the-art research describing the most commonly used groups of outputs and inputs, serves as a step towards the standardization of DEA. It was found that the most widely used model for efficiency orientation was input orientation. Although the number of studies based on DEA methods is gradually increasing and DEA is the most frequently used tool in the efficiency analysis of PHC organizations, there is still room for improvement. Further research is required to identify appropriate input and output variables and a suitable DEA model for assessing PHC. The standardization of DEA could extend the scope of research tools for the analysis of functioning the primary care. This would support health economic analyses of measurements of primary care efficiency. Declaration Funding: Narodowe Centrum Nauki (National Science Centre Poland) (2016/21/B/NZ7/02052). Ethical approval: none. Conflict of interest: none. Data availability The authors declare that the data supporting the study findings are available within the article. Acknowledgements The authors thank Katarzyna Kosiek, PhD, for reviewing the manuscript and Edward Lowczowski for English language assistance. Reference list 1. Xu K , Saksena P. The Determinants of Health Expenditure: A Country-Level Panel Data Analysis . World Health Organization . A Working Paper of the Results for Development Institute (R4D) [Internet], 2011 . http://www.who.int/health_financing/documents/report_en_11_deter-he.pdf?ua=1 (accessed on 5 April 2019 ). 2. WHO . 2016 Global Health Care Outlook. Battling Costs While Improving Care. (2016) “2016 Global Health Care Outlook Battling Costs While Improving Care” DTTL Global Life Sciences and Health Care (LSHC) Industry Group analysis, from [Internet] . 2016 . http://www2.deloitte.com/content/dam/Deloitte/global/Documents/Life-Sciences-Health-Care/gx-lshc-2016-health-care-outlook.pdf (accessed on 5 April 2019). 3. Mark G Shrime MG, Mukhopadhyayb S, Alkirea BC. Health-system-adapted data envelopment analysis for decisionmaking in universal health coverage . Bull World Health Organ 2018 ; 96 : 393 – 401 . Google Scholar Crossref Search ADS PubMed WorldCat 4. Jablonsky J , Emrouznejad A, Toloo M. Editorial: special issue on data envelopment analysis . Cent Eur J Oper Res 2018 ; 26 ( 4 ): 809 – 12 . Google Scholar Crossref Search ADS WorldCat 5. Emrouznejada A , Yang G. A survey and analysis of the first 40 years of scholarly literature in DEA: 1978–2016 . Socio-Econ Plan Sci 2018 ; 61 : 4 – 8 . Google Scholar Crossref Search ADS WorldCat 6. Ozcan YA , editor. Performance Measurement Using Data Envelopment Analysis (DEA) . New York, Heidelberg, Dordrecht and London : Springer , 2014 . Google Scholar Crossref Search ADS Google Preview WorldCat COPAC 7. Charnes A , Cooper WW, Rhodes E. Measuring the efficiency of decision making units . Eur J Oper Res 1978 ; 2 : 429 – 44 . 8. Garcia F , Marcuello C, Serrano D, Urbina O. Evaluation of efficiency in primary health care centres: an application of data envelopment analysis . Financial Acc & Man 1999 ; 15 ( 1 ): 67 – 83 . 9. Farrell MJ . The measurement of productive efficiency . J R Stat Soc Ser A (Gen) 1957 ; 120 ( 3 ): 253 – 81 . Google Scholar Crossref Search ADS WorldCat 10. Cwiakala-Malys A , Nowak W. Classification of data envelopment analysis models [Sposoby klasyfikacji modeli DEA] . Bad Oper Decyzje 2009 ; 3 : 5 – 18 . (in Polish) Google Scholar OpenURL Placeholder Text WorldCat 11. Dyson RG , Shale EA. Data envelopment analysis, operational research and uncertainty . J Oper Res Soc 2010 ; 61 ( 1 ): 25 – 34 . Google Scholar Crossref Search ADS WorldCat 12. Del Rocío Moreno-Enguix M , Gómez-Gallego JC, Gómez Gallego M. Analysis and determination the efficiency of the European health systems . Int J Health Plann Manage 2018 ; 33 : 136 – 54 . Google Scholar Crossref Search ADS PubMed WorldCat 13. Pourreza A , Alipour V, Arabloo J, Bayati M, Ahadinezhad B. Health production and determinants of health systems performance in WHO Eastern Mediterranean Region . East Mediterr Health J 2017 ; 23 : 368 – 74 . Google Scholar Crossref Search ADS PubMed WorldCat 14. Zhang L , Cheng G, Song S et al. Efficiency performance of China’s health care delivery system . Int J Health Plann Manage 2017 ; 32 : 254 – 63 . Google Scholar Crossref Search ADS PubMed WorldCat 15. Erus B , Hatipoglu O. Physician payment schemes and physician productivity: analysis of Turkish healthcare reforms . Health Policy 2017 ; 121 : 553 – 7 . Google Scholar Crossref Search ADS PubMed WorldCat 16. Ding JM , Hu XJ, Zhang XZ, Shang L, Yu M, Chen HL. Equity and efficiency of medical service systems at the provincial level of China’s mainland: a comparative study from 2009 to 2014 . Bmc Public Health 2018 ; 18 ( 1 ): 214 . Google Scholar Crossref Search ADS PubMed WorldCat 17. Zhang Y , Wang Q, Jiang T, Wang J. Equity and efficiency of primary health care resource allocation in mainland China . Int J Equity Health 2018 ; 17 : 140 . Google Scholar Crossref Search ADS PubMed WorldCat 18. Dlouhý M . Measuring geographic inequalities: dealing with multiple health resources by data envelopment analysis . Front Public Health 2018 ; 6 : 53 . Google Scholar Crossref Search ADS PubMed WorldCat 19. Campos-Lucena MSS-D , Sanz Diaz MT, Morente FV, Ovando RY. The efficiency of health services in Spain based on Data Envelopment Analysis (DEA). The singularity of health efficiency . Int J Collab Res Inter Med Public Health 2018 ; 10 ( 1 ): 776 – 91 . Google Scholar OpenURL Placeholder Text WorldCat 20. Stefko R , Gavurova B, Kocisova K. Healthcare efficiency assessment using DEA analysis in the Slovak Republic . Health Econ Rev 2018 ; 8 ( 1 ): 6 . Google Scholar Crossref Search ADS PubMed WorldCat 21. Barouni M , Amiresmaieli MR, Shahravan A, Amini S. The efficiency assessment of dental units using data envelopment analysis approach: the case of Iran . Iran J Public Health 2017 ; 46 : 552 – 9 . Google Scholar PubMed OpenURL Placeholder Text WorldCat 22. Fiallos J , Patrick J, Michalowski W, Farion K. Using data envelopment analysis for assessing the performance of pediatric emergency department physicians . Health Care Manag Sci 2017 ; 20 : 129 – 40 . Google Scholar Crossref Search ADS PubMed WorldCat 23. Sowlati T. Establishing the Practical Frontier in Data Envelopment Analysis . PhD Thesis. Ontario, Canada : University of Toronto , 2001 . Google Scholar Google Preview OpenURL Placeholder Text WorldCat COPAC 24. Ferreira DC , Nunes AM, Marques RC. Doctors, nurses, and the optimal scale size in the Portuguese public hospitals . Health Policy 2018 ; 122 : 1093 – 100 . Google Scholar Crossref Search ADS PubMed WorldCat 25. Johannessen KA , Kittelsen SA, Hagen TP. Assessing physician productivity following Norwegian hospital reform: a panel and data envelopment analysis . Soc Sci Med 2017 ; 175 : 117 – 26 . Google Scholar Crossref Search ADS PubMed WorldCat 26. Liu JS , Lu LYY, Lu WM, Lin BJY. Data envelopment analysis 1978–2010: a citation-based literature survey . Omega Int J Manag Sci 2013 ; 41 ( 1 ): 3 – 15 . Google Scholar Crossref Search ADS WorldCat 27. Liu JS , Lu LYY, Lu WM, Lin BJY. A survey of DEA applications . Omega In J Manag Sci 2013 ; 41 ( 5 ): 893 – 2 . Google Scholar Crossref Search ADS WorldCat 28. Amado CAF , Dyson RG. On comparing the performance of primary care providers . Eur J Oper Res 2008 ; 185 ( 3 ): 915 – 32 . Google Scholar Crossref Search ADS WorldCat 29. Hollingsworth B . The measurement of efficiency and productivity of health care delivery . Health Econ 2008 ; 17 : 1107 – 28 . Google Scholar Crossref Search ADS PubMed WorldCat 30. Hollingsworth B , Dawson PJ, Maniadakis N. Efficiency measurement of health care: a review of non-parametric methods and applications . Health Care Manag Sci 1999 ; 2 : 161 – 72 . Google Scholar Crossref Search ADS PubMed WorldCat 31. Kringos DSB , Boerma WGW, Hutchinson A, Saltman RB. Building Primary Care in a Changing Europe . 2015 . http://www.euro.who.int/__data/assets/pdf_file/0018/271170/BuildingPrimaryCareChangingEurope.pdf?ua=1 (accessed on 14 June 2017 ). 32. Cooper WW , Seiford LM, Zhu J. Data envelopment analysis. history, models and interpretations . In: Handbook on Data Envelopment Analysis . New York : Springer , 2011 , pp. 1 – 39 . Google Scholar Crossref Search ADS Google Preview WorldCat COPAC 33. Domagala A . Data envelopment analysis – a method for measuring the relative technical efficiency [Metoda data envelopment analysis jako narzędzie badania względnej efektywności technicznej] . Bad Oper Decyzje 2007 ; 3–4 : 21 – 34 . (in Polish) Google Scholar OpenURL Placeholder Text WorldCat 34. PROSPERO . International Prospective Register of Systematic Reviews . York, UK : National Institute of Health Research (NIHR), University of York . https://www.crd.york.ac.uk/PROSPERO/ (accessed on 22 September 2019 ). 35. García-Alonso CR , Almeda N, Salinas-Pérez JA, Gutiérrez-Colosía MR, Salvador-Carulla L. Relative Technical Efficiency Assessment of the technical efficiency of Mental Health Services: a systematic review . Adm Policy Ment Health 2019 ; 46 ( 4 ): 429 – 44 . 36. CASP U . Critical Appraisal Skills Programme (CASP) . Oxford, UK : CASP , 2017 . http://www.casp-uk.net/, https://casp-uk.net/wp-content/uploads/2018/03/CASP-Systematic-Review-Checklist-2018_fillable-form.pdf (accessed on 22 September 2019 ). 37. Moher D , Liberati A, Tetzlaff J, Altman DG; PRISMA Group. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement . PLoS Med 2009 ; 6 : e1000097 . Google Scholar Crossref Search ADS PubMed WorldCat 38. Kringos DS , Boerma WGW, Hutchinson A, van der Zee Software J, Groenewegen PP. The breadth of primary care: a systematic literature review of its core dimensions . BMC Health Serv Res 2010 ; 10 : 65 . Google Scholar Crossref Search ADS PubMed WorldCat 39. Riaz M , Mendes E, Tempero E. A systematic review of software maintainability prediction and metrics 2009 3rd International Symposium on Empirical Software Engineering and Measurement . https://ieeexplore.ieee.org/document/5314233 (accessed on 13 September 2019 ). 40. Pelone F , Kringos DS, Valerio L et al. The measurement of relative efficiency of general practice and the implications for policy makers . Health Policy 2012 ; 107 ( 2–3 ): 258 – 68 . Google Scholar Crossref Search ADS PubMed WorldCat 41. Ferreira C , Marques RC, Nicola P. On evaluating health centers groups in Lisbon and Tagus Valley: efficiency, equity and quality . BMC Health Serv Res 2013 ; 13 : 529 . Google Scholar Crossref Search ADS PubMed WorldCat 42. Chilingerian JA , Sherman HD. Benchmarking physician practice patterns with DEA: a multi-stage approach for cost containment . Ann Oper Res 1996 ; 67 ( 1 ): 83 – 116 . Google Scholar Crossref Search ADS WorldCat 43. Manuel Cordero-Ferrera J , Crespo-Cebada E, Murillo-Zamorano LR. Measuring technical efficiency in primary health care: the effect of exogenous variables on results . J Med Syst 2011 ; 35 ( 4 ): 545 – 54 . Google Scholar Crossref Search ADS PubMed WorldCat 44. Bahurmoz A . Measuring efficiency in primary health care centres in Saudi Arabia . J KAU Econ Adm 1998 ; 12 ( 2 ): 3 – 18 . Google Scholar Crossref Search ADS WorldCat 45. Cordero Ferrera JM , Crespo Cebada E, Murillo Zamorano LR. The effect of quality and socio-demographic variables on efficiency measures in primary health care . Eur J Health Econ 2014 ; 15 : 289 – 302 . Google Scholar Crossref Search ADS PubMed WorldCat 46. Oikonomou N , Tountas Y, Mariolis A et al. Measuring the efficiency of the Greek rural primary health care using a restricted DEA model; the case of southern and western Greece . Health Care Manag Sci 2016 ; 19 : 313 – 25 . Google Scholar Crossref Search ADS PubMed WorldCat 47. Cantor VJM , Poh KL. Integrated analysis of healthcare efficiency: a systematic review . J Med Syst 2017 ; 42 : 8 . Google Scholar Crossref Search ADS PubMed WorldCat 48. Kotsemir MN . Research Paper. Measuring National Innovation Systems Efficiency – A Review of DEA Approach . 2013 , p. 24 . https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2304735 (accessed on 5 April 2019 ). 49. Pelone F , Kringos DS, Romaniello A et al. Primary care efficiency measurement using data envelopment analysis: a systematic review . J Med Syst 2015 ; 39 : 156 . Google Scholar Crossref Search ADS PubMed WorldCat 50. Ehrgott M , Holder A, Nohadani O. Uncertain data envelopment analysis . Eur J Oper Res 2018 ; 268 ( 1 ): 231 – 42 . Google Scholar Crossref Search ADS WorldCat 51. vanVeen SHCM. Comparative Efficiency Analysis from the Perspective of the Dutch Health Care Insurer: Determining the Usefulness of Efficiency Measures for Contracting Primary Care Organizations . Rotterdam, The Netherlands : Erasmus University Rotterdam , 2010 . Google Scholar Google Preview OpenURL Placeholder Text WorldCat COPAC © The Author(s) 2019. Published by Oxford University Press. This is an Open Access article distributed under the terms of the Creative Commons Attribution Non-Commercial License (http://creativecommons.org/licenses/by-nc/4.0/), which permits non-commercial re-use, distribution, and reproduction in any medium, provided the original work is properly cited. For commercial re-use, please contact journals.permissions@oup.com © The Author(s) 2019. Published by Oxford University Press.

Journal

Family Practice – Oxford University Press

Published: Mar 25, 2020

Keywords: primary health care; drug enforcement administration

There are no references for this article.