Get 20M+ Full-Text Papers For Less Than $1.50/day. Start a 14-Day Trial for You or Your Team.

Learn More →

Ensemble learning to predict opioid-related overdose using statewide prescription drug monitoring program and hospital discharge data in the state of Tennessee

Ensemble learning to predict opioid-related overdose using statewide prescription drug monitoring... Journal of the American Medical Informatics Association, 29(1), 2022, 22–32 doi: 10.1093/jamia/ocab218 Advance Access Publication Date: 19 October 2021 Research and Applications Research and Applications Ensemble learning to predict opioid-related overdose using statewide prescription drug monitoring program and hospital discharge data in the state of Tennessee 1 2 1 3 Michael Ripperger , Sarah C. Lotspeich , Drew Wilimitis , Carrie E. Fry , Allison 4 1 4 4 1 Roberts , Matthew Lenert , Charlotte Cherry , Sanura Latham , Katelyn Robinson , 1,2 1,3 4 1,5,6 Qingxia Chen , Melissa L. McPheeters , Ben Tyndall , and Colin G. Walsh 1 2 Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, Tennessee, USA, Department of Biosta- tistics, Vanderbilt University Medical Center, Nashville, Tennessee, USA, Department of Health Policy, Vanderbilt University Medical Center, Nashville, Tennessee, USA, Office of Informatics and Analytics, Tennessee Department of Health, Nashville, 5 6 Tennessee, USA, Department of Medicine, Vanderbilt University Medical Center, Nashville, Tennessee, USA, and Department of Psychiatry, Vanderbilt University Medical Center, Nashville, Tennessee, USA Corresponding Author: Colin G. Walsh, MD, MA, Department of Biomedical Informatics, Vanderbilt University Medical Center, 2525 West End Avenue, Suite 1475, Nashville, TN 37203, USA; colin.walsh@vumc.org Received 4 April 2021; Revised 3 September 2021; Editorial Decision 25 September 2021 ABSTRACT Objective: To develop and validate algorithms for predicting 30-day fatal and nonfatal opioid-related overdose using statewide data sources including prescription drug monitoring program data, Hospital Discharge Data System data, and Tennessee (TN) vital records. Current overdose prevention efforts in TN rely on descriptive and retrospective analyses without prognostication. Materials and Methods: Study data included 3 041 668 TN patients with 71 479 191 controlled substance pre- scriptions from 2012 to 2017. Statewide data and socioeconomic indicators were used to train, ensemble, and calibrate 10 nonparametric “weak learner” models. Validation was performed using area under the receiver operating curve (AUROC), area under the precision recall curve, risk concentration, and Spiegelhalter z-test statistic. Results: Within 30 days, 2574 fatal overdoses occurred after 4912 prescriptions (0.0069%) and 8455 nonfatal overdoses occurred after 19 460 prescriptions (0.027%). Discrimination and calibration improved after ensem- bling (AUROC: 0.79–0.83; Spiegelhalter P value: 0–.12). Risk concentration captured 47–52% of cases in the top quantiles of predicted probabilities. Discussion: Partitioning and ensembling enabled all study data to be used given computational limits and helped mediate case imbalance. Predicting risk at the prescription level can aggregate risk to the patient, pro- vider, pharmacy, county, and regional levels. Implementing these models into Tennessee Department of Health systems might enable more granular risk quantification. Prospective validation with more recent data is needed. Conclusion: Predicting opioid-related overdose risk at statewide scales remains difficult and models like these, which required a partnership between an academic institution and state health agency to develop, may comple- ment traditional epidemiological methods of risk identification and inform public health decisions. Key words: drug overdose, opioid epidemic, machine learning, prescription drug monitoring programs, vital statistics V The Author(s) 2021. Published by Oxford University Press on behalf of the American Medical Informatics Association. This is an Open Access article distributed under the terms of the Creative Commons Attribution-NonCommercial-NoDerivs licence (https://creativecommons.org/licenses/ by-nc-nd/4.0/), which permits non-commercial reproduction and distribution of the work, in any medium, provided the original work is not altered or transformed in any way, and that the work is properly cited. For commercial re-use, please contact journals.permissions@oup.com 22 Journal of the American Medical Informatics Association, 2022, Vol. 29, No. 1 23 and characterizing risk with precise and automated predictive mod- INTRODUCTION els. Part of our efforts was to leverage known sociodemographic and We sought to develop and validate implementable predictive models economic factors relating to mental and physical health. Community for the state of Tennessee (TN) to predict (1) fatal and (2) nonfatal characteristics have been known to be predictive of OPR overdose opioid-related overdose risk by leveraging statewide data sources risk. provided by the Tennessee Department of Health (TDH). Through Seeking to predict future risk by combining linked PDMP and our academic-state partnership, we applied ensemble learning to fa- overdose data, TDH partnered with VUMC to help the state under- tal and nonfatal overdose prediction using statewide controlled sub- stand the opioid epidemic statewide, target interventions, and allo- stance prescription data, hospital discharge diagnoses, and causes of cate scarce resources accordingly. Adhering to the architectural and death from vital records. implementation requirements of TDH, the VUMC team derived a data management strategy, sourced a wide array of social determi- nant variables to help quantify risk, and evaluated our approach. BACKGROUND AND SIGNIFICANCE Once implemented in TDH systems, such models might allow TN to further support the greatest at-risk communities and identify inter- The link between the current opioid epidemic in the United States vention touch points within the community health system. and the over-prescribing of opioid pain relievers (OPRs) has been well established. Over-prescribing and OPR-related harms were first observed in the 1990s and some states including TN have experi- 2,3 MATERIALS AND METHODS enced higher rates of prescribing and the subsequent harms. Near the opioid prescribing peak in 2010, TN providers wrote more OPR This study was approved by the VUMC Institutional Review Board prescriptions than there were residents in the state. Between 2014 (#171323). and 2018, OPR-related deaths rose 49% to an annual cost of 1307 5,6 lives. The United States meanwhile has seen a near-universal Data sources adoption of prescription drug monitoring programs (PDMPs) with Controlled Substance Monitoring Database (CSMD, TN’s PDMP) intentions to combat the opioid epidemic by monitoring prescribing data, Hospital Discharge Data System (HDDS) data, and TN death histories, informing providers, and identifying concerns with varying certificates were combined to produce a 6-year observational cohort 7–10 success. Although PDMPs have seldomly been used to predict that spanned from the beginning of 2012 through the end of 2017. imminent risk at the patient level, prevention at the practice, county, Publicly available socioeconomic indicators relating to health, or regional levels might be possible if accurate algorithms are devel- healthcare utilization, and treatment access were compiled and 11–14 oped, validated, and implemented. Severely affected by the opi- mapped to either ZIP codes or counties. oid crisis, TN has already linked its controlled substance PDMP (II– The following were mapped to residential ZIP codes: Area Dep- V scheduled and gabapentin) to statewide mortality data and hospi- rivation Index (ADI); statistics on employment from the U.S. Census 15,16 tal discharge data. In this study, researchers at TDH and Van- Bureau; and Medication-Assisted Treatment (MAT) locations in- derbilt University Medical Center (VUMC) partnered to develop cluding buprenorphine providers, methadone clinics, and Opioid and validate the first scalable predictive models from statewide data- 31,32 Treatment Programs (OTPs) from data aggregated by TDH. sets in TN for the related but disparate outcomes: (1) fatal and (2) TN age-adjusted morbidity rates from TDH; the Tennessee Vulnera- 17,18 nonfatal opioid overdose. bility Index (TVI) from TDH; statistics on income, poverty, college The application of machine learning to predict individual risk is education, crowding, and private insurance from the American not new in the biomedical literature nor in OPR overdose preven- Community Survey (ACS); Rural–Urban Continuity Codes (RUCC) tion. Prior studies have predicted overdose risk using Medicare from the U.S. Department of Agriculture; the Social Vulnerability claims, self-reported substance use patterns, and demo- Index (SVI) from the Centers for Disease Control and Prevention 13,19–21 graphics. Many studies have also utilized electronic health (CDC); and Anti-Drug Abuse Coalition services from TDH were records with or without vital records including at Mt. Sinai, in the 33–36 mapped to individual counties. A full list of sourced data is state of Colorado, and at the Veteran’s Health Administration available within the Supplementary Material. 22–25 (VHA). Few US states, however, have specifically used PDMP data to predict overdose—namely Maine, Oregon, and Mary- 11,26,27 Outcome ascertainment land. In Maryland, hospital discharge, healthcare utilization, The outcomes of interest in this study were fatal and nonfatal and criminal justice data have been linked to predict future OPR 12,14,28 opioid-related overdose events that occurred within 30 days of a overdose risk for individuals. Our study likewise combines controlled substance prescription fill. The 30-day time window was predictive modeling with comprehensive statewide data. No previ- chosen after plotting the accumulation of overdoses over time after ous studies to our knowledge have assembled these kinds of data for a prescription fill (Supplementary Material). Fatal and nonfatal a large, southern US state like TN where the rates of OPR prescrib- overdoses were identified consistent with methods used by TDH in ing are much higher than the national average. their annual Prescription Drug Overdose Reports. Fatal overdoses In 2019, a federal investigation led by the Department of Justice were identified from TN death certificates using International Clas- (DOJ) uncovered fraud and inappropriate opioid prescribing in TN sification of Disease, revision 10 (ICD-10) codes. Nonfatal over- and resulted in the arrests of multiple physicians, pharmacists, and 29 doses were identified in the HDDS with specified opioid-related other health professionals. Such measures relied upon descriptive diagnostic codes (Supplementary Material). analytics for harms that had already occurred years prior. While monitoring and descriptive analytics may provide a lens into the cur- rent state of the opioid epidemic, they cannot identify the next pa- Predictive modeling details tient, practice, or community at risk. The goal of this work was to Our modeling choices were as follows: (1) establish a vector of so- supplement these traditional epidemiological methods of identifying cioeconomic indicators based on a patient’s last reported location 24 Journal of the American Medical Informatics Association, 2022, Vol. 29, No. 1 from the PDMP (from the time of the previous prescription); (2) each random forest. In total, 20 random forests were developed count the cumulative number of prior medications, diagnostic codes, from the 20 training subsets—10 for each of the 2 outcomes. and hospital visits by type a patient has accumulated thus far; and During training, each training subset itself was split into a 90% (3) add age, sex, and derived variables that represent a patient’s pre- training set and a 10% testing set to allow predictions to be made scription history for controlled substances. Variables chosen in- for each case. Each case was placed in the testing set of each subset cluded the sums of distinct practitioners, distinct pharmacies, exactly one time which guarantied all case data were used in training distinct hospital identifiers, total prescriptions, total morphine milli- and at least one prediction for each associated record was generated. gram equivalents, short/long-acting OPR prescriptions, overlapping After the weak learners were ensembled and calibrated in the devel- OPR and benzodiazepine prescriptions, prior medications for opioid opment set, the resulting ensembled models were validated in a final use disorder, and opioid-naı ¨ve prescriptions as defined as not having held-out testing set. A conceptual diagram of this training scheme is an OPR prescription within the last 45 days (Supplementary Mate- shown (Figure 1). rial). Race and ethnicity were not explicitly represented in our mod- els. Modeling at the prescription level was done to create time- dependent and granular risk predictions which could then be aggre- gated to practice, pharmacy, local, county, and regional levels. This Calibration approach intended to potentially guide planning and response activi- A development set consisting of 5% of the data was reserved to cor- ties at varying levels of detail. rect the miscalibrations from the under-sampled controls in the training subsets. We compared 7 methods of ensembling and cali- bration. Either the minimum, maximum, mean, or median predic- Data preprocessing tion was taken from the 10 weak learner predictions and passed Patient linkage across our datasets relied on TDH-determined mas- through logistic calibration, or the 10 weak learner predictions were ter patient indexing. Only records with valid person identifiers used as inputs for ridge regression, random forest, or penalized re- were retained, and records determined to be related to a nonhuman 43–46 gression (LASSO). patient (ie, veterinary prescription records) were removed. Hospital Logistic calibration, when applied, was defined by training a uni- records from the HDDS were limited to verified inpatient variate logistic regression in the calibration set where the sole pre- encounters. dictor was the aggregate in question (eg, max) and the outcome was Precise ADI and RUCC features were developed from the mini- either fatal or nonfatal overdose. The resulting generalized linear mum, maximum, and mean values of each ZIP code. Other ZIP models along with the aggregation methods were then considered as code features were developed from county data using the TN county ensemblers. The more complex ensembling methods trained multi- that contained the majority area of each ZIP code. OTP and metha- variate models using the 10 weak learners as predictors. Random done clinic availability were modeled using a 60-mile radius, repre- forest was used for comparison for 2 types of penalized logistic re- senting a practical range for driving a normal distance in TN (90– gression: L1-regularized (LASSO) and L2-regularized (RIDGE) re- 120 minutes driving time). gression. All resulting models were expected to be calibrated as they To reduce the dimensionality of PDMP and HDDS features, were either trained on the calibration set or calibrated via logistic prior medications and diagnoses were grouped to higher-order cate- calibration. gories using the National Drug File-Reference Terminology (NDF- Final ensembled and calibrated algorithms were then tested on FT), Pharmacologic Classes and Clinical Classification Software the test set. Weak learners were tested on the calibration set. We (CCS), Level 2 groupings from National Drug Codes (NDCs) and note that no additional calibration was performed on the test set, International Classification of Disease, revision 10, Clinical Modifi- making it a pure test of calibration as well as discrimination. 40–42 cation (ICD-10-CM) codes. In total, 342 features were used for model training after this dimensionality reduction and only entries in patient records prior to prediction dates were used. Performance assessment methods Sampling strategy and model training Discrimination performance metrics included area under the re- We separated the data into 75% training, 5% development, and ceiver operating curve (AUROC), area under the precision recall 20% testing partitions to ultimately derive one model for fatal over- curve (AUPRC), and risk concentration. Risk concentration was per- dose and one model for nonfatal. All prescriptions in the data that formed by dividing the predictions from the test set into 10 quantiles were associated with an individual were added together to only one and calculating the proportion of all the cases those quantiles held. set to prevent leak between training and testing within individuals. Calibration was assessed using Spiegelhalter z-test. The ridge re- Models were trained in the training set and then calibrated, gression ensembles were further assessed for performance differen- ensembled, and evaluated in the development set. ces by subgroups consisting of race, ethnicity, and gender as The training set was equally divided into 10 smaller training par- determined by hospital records as well as age and RUCC codes from titions or subsets due to computational limits. To help combat case residential ZIP codes for urbanicity/rurality. To test how perfor- imbalance, all cases and their associated records were added to each mance varied when the number of partitions in the training set was training set, but only 10% of all the controls from the entire training changed, additional models were trained using N¼ 5or N¼ 15 and set were included in an individual training set (ie, only one training compared using AUPRC. For both fatal and nonfatal overdose, we set contained any one control). Ten random regression forest “weak ranked each feature by taking the mean of the important values learners” were then developed from the training subsets using the from the 10 weak learners—determined by the variance of responses ranger R package with an estimated response variance splitting crite- from each random forest. A full list is available within the Supple- ria. To help limit memory consolidation, 200 trees were used for mentary Material. Journal of the American Medical Informatics Association, 2022, Vol. 29, No. 1 25 Figure 1. Conceptual diagram of training data splits, weak learners, the ensembling/calibration development step, and the testing step. RESULTS LASSO). Random forest performed worse compared to other meth- ods of aggregation. The top 2 performing ensembles, mean and ridge Study data regression, were further evaluated in the risk concentration and cali- Study data included 71 479 191 controlled substance prescriptions bration analyses. across 3 041 668 TN patients. As sourced from hospital records, when available: 1 409 556 (46.3%) patients were Female; 958 440 (31.5%) patients were Male; and 673 672 (22.1%) patients were Risk concentration and calibration performance Unknown. Patients by coded race showed 7104 (0.23%) patients Risk concentration showed that, in the test set, the mean and ridge were Asian-American; 360 314 (11.8%) patients were Black; 704 regression ensembling methods concentrated 47–52% of the over- (0.023%) patients were Native American; 20 147 (0.66%) patients dose outcomes within the top quantiles of predicted probabilities were Other; 1 851 324 (61.0%) patients were White; and 802 075 (Table 2). Both top quantiles contained 10% of the test set predic- (26.4%) patients were Unknown. Patients by coded ethnicity also tions. Overlapping quantiles where the predictions had the same val- showed 16 061 (0.53%) patients were Hispanic; 2 064 654 (67.8%) ues were combined as seen by the number of prescriptions in the patients were non-Hispanic; and 960 953 (32.0%) patients were Un- first quantile of the fatal mean ensembling method. known. Within 30 days, 2574 fatal overdoses occurred after 4912 Calibration measured the degree to which the predictions (0.0069%) prescriptions and 8455 nonfatal overdoses occurred after reflected the true outcome prevalences. The ensembled models pre- 19 460 (0.027%) prescriptions. Nearly 60% of all fatal and nonfatal dicting fatal overdose showed nonsignificant calibration from mean overdoses in the data occurred within 30 days of a prescription (Sup- ensembling and significant calibration from ridge regression as indi- plementary Material). cated by the nonsignificant Spiegelhalter z-test. The ensembled mod- els for nonfatal overdose showed better calibration for ridge regression than for mean ensembling although both were nonsignifi- Weak learner and ensembling model performance cantly calibrated (Table 3). The ridge regression ensembling method Both the fatal and nonfatal weak learner models had similar preva- was subsequently used to analyze performance variations by sub- lence rates throughout the training set and showed consistent groups. AUROC and AUPRC values when applied to the development set (Table 1). AUROC was useful to compare these models simply de- Subgroup performance differences and partition spite having known problems when assessing absolute performance with case imbalance. The total number of cases and controls in the variation training set were 3725 and 53 591 596 (0.0069%) for fatal and 14 Both the fatal and nonfatal ridge regression ensembles were tested 695 and 53 580 626 (0.027%) for nonfatal overdose. on subgroups in the test set. AUROC and AUPRC values varied by Discrimination varied by ensembling method when applied to subgroup in age, sex, race, ethnicity, and RUCC values of residential the test set for both fatal and nonfatal overdose (Table 1). Averaging ZIP codes (Table 4). Case and control percentages among the sub- or selecting the minimum or maximum predictions from the 10 groups also varied. weak learner models for both fatal and nonfatal produced similar Repeating the modeling experiments for N¼ 5 and N¼ 15 results to using more complex methods of aggregation (eg, ridge, showed no differences in AUPRC values when the number of parti- 26 Journal of the American Medical Informatics Association, 2022, Vol. 29, No. 1 Table 1. Characteristics of both the 20 weak learner models in the development set and the 14 ensemble models in the test set for fatal and nonfatal overdose Fatal overdose Nonfatal overdose Weak learner/en- AUROC AUPRC Cases Controls % Outcomes AUROC AUPRC Cases Controls % Outcomes semble WL1 0.77 0.00024 224 3 566 077 0.0063 0.79 0.0018 1131 3 580 452 0.032 WL2 0.73 0.00023 0.78 0.0016 WL3 0.76 0.00023 0.79 0.0019 WL4 0.75 0.00024 0.78 0.0016 WL5 0.72 0.00026 0.79 0.0021 WL6 0.73 0.00027 0.80 0.0019 WL7 0.71 0.00023 0.79 0.0017 WL8 0.78 0.00024 0.80 0.0017 WL9 0.75 0.00025 0.79 0.0017 WL10 0.72 0.00026 0.78 0.0015 Maximum 0.83 0.00040 963 14 316 606 0.0067 0.82 0.0014 4031 14 309 753 0.028 Minimum 0.67 0.00032 0.76 0.0014 Mean 0.83 0.00042 0.83 0.0015 Median 0.80 0.00041 0.82 0.0015 LASSO 0.79 0.00038 0.82 0.0015 Ridge 0.83 0.00042 0.83 0.0016 Random forest 0.38 0.00007 0.49 0.0004 Note: Ensemble models combined and calibrated weak learner model predictions from the development set. AUPRC: area under the precision recall curve; AUROC: area under the receiver operating curve. tions was changed (Figure 2). Absolute change by partition choice The subgroup performance analysis showed that the ridge regres- was minimal as evidenced by the small absolute differences in y-axes sion models resulted in disparate performance in terms of AUPRC shown (eg, <0.0001 change in AUPRC by number of folds for the and AUROC for race and age despite small absolute AUPRC differ- fatal model). ences (Table 4). Case imbalance may be driving these differences. Correcting performance differences is necessary for accurately assessing risk in the state. When the number of training partitions Weak learner feature importances was varied, AUPRCs varied minimally if at all (Figure 2). The top 15 model features from the 10 weak learner models for fatal In the fatal overdose model, the top predictors were face valid as and nonfatal overdose were determined by ranking their mean re- known risk factors for opioid-related overdose (Figure 3). The total sponse variances (Figures 3 and 4). Twelve features were within the quantity of controlled substances prescribed was close to the top of top 15 of both the fatal and nonfatal overdose models. the list. Notably, overlapping benzodiazepine prescriptions were more important in the prediction of fatal opioid-related overdose than nonfatal. Multidrug combinations have been known to play a large role in the fatality potential of opioid-related overdoses and DISCUSSION benzodiazepines have a synergistic respiratory depressant effect This study supports the validity of combining statewide PDMP data when taken with opioids. with clinical discharge and socioeconomic data to predict fatal and Informatics implications of this study include the importance of nonfatal opioid overdose within 30 days of a controlled substance partitioning and sampling to lessen overfitting in settings with high prescription fill. Partitioning and ensembling the data allowed us to stake, but rare (at state scale), outcomes. Efforts to predict risk at an use all study data despite computational limits. We modeled risk at actionable timepoint, for example, a prescription fill event, do not the prescription level, making these models applicable to any indi- obviate aggregating risk analyses to levels relevant for public health vidual prescription with historical data. Aggregating these predic- intervention such as the community and regional levels. US states tions enables risk to be calculated at varying levels of detail for have long implemented PDMPs, but most have not disseminated better informed public health decision-making. predictive modeling approaches at this scale and none of the nearby AUROCs and AUPRCs of the fatal and nonfatal models in the states in the southern United States have done so. Characterizing development set improved in the test set after ensembling (Table 1). OPR risk in our state might inform better prevention both in TN Risk concentration analyses consistently captured half the outcomes and in neighbor states, as the overdose crisis varies considerably of interest in the top quantiles of risk (Table 2). Given the presence near and across state lines. of case imbalance, the highest risk quantiles may enable TN to focus Several attributes of this overdose modeling problem increased prevention efforts more efficiently. Both ensembles were miscali- its complexity. First, extreme case imbalance resulted from the rarity brated when predicting nonfatal overdose, but the ridge regression of fatal and nonfatal overdoses at statewide scale—prevalence less ensemble was calibrated when predicting fatal overdose (Table 3). than a fraction of 1%. Second, person disambiguation in data that Future recalibration efforts should reduce these gaps. Predicting fa- were manually entered by pharmacists into the CSMD resulted in re- tal overdose in the future may enable better prevention. Prospective liance on constructed, probabilistic patient mapping indices. Ongo- evaluation with more recent data is needed. ing work within TDH continues to refine and improve this Journal of the American Medical Informatics Association, 2022, Vol. 29, No. 1 27 Table 2. Risk concentration of the ensembled fatal and nonfatal prediction models which were validated in the test set Fatal/Nonfatal Ensembling method Quantile Prescriptions Cases Proportion of cases Inclusive lower bound Exclusive upper bound Fatal Mean 1 4 106 507 32 0.033 0.00Eþ00 1.65E08 2 210 474 4 0.004 1.65E08 6.28E08 3 1 412 596 14 0.015 6.28E08 3.33E05 4 1 429 211 33 0.034 3.33E05 3.53E04 5 1 431 757 40 0.042 3.53E04 7.07E04 6 1 432 104 66 0.069 7.07E04 1.46E03 7 1 434 688 100 0.104 1.46E03 2.92E03 8 1 428 476 171 0.178 2.92E03 6.80E03 9 1 431 756 503 0.522 6.80E03 3.34E01 Ridge regression 1 1 431 758 81 0.084 3.85E05 5.47E05 2 4 776 940 43 0.045 5.47E05 5.48E05 3 950 091 4 0.004 5.48E05 5.48E05 4 1 443 236 55 0.057 5.48E05 5.54E05 5 1 420 274 60 0.062 5.54E05 5.66E05 6 1 431 800 103 0.107 5.66E05 6.00E05 7 1 431 716 159 0.165 6.00E05 6.73E05 8 1 431 754 458 0.476 6.73E05 3.19E01 Nonfatal Mean 1 1 929 336 43 0.011 0.00Eþ00 1.93E08 2 933 421 25 0.006 1.93E08 8.68E06 3 1 437 802 67 0.017 8.68E06 2.50E04 4 1 425 055 81 0.020 2.50E04 6.91E04 5 1 432 474 123 0.031 6.91E04 1.41E03 6 1 430 183 143 0.035 1.41E03 2.55E03 7 1 431 883 290 0.072 2.55E03 4.50E03 8 1 431 048 415 0.103 4.50E03 8.04E03 9 1 431 205 835 0.207 8.04E03 1.59E02 10 1 431 377 2009 0.498 1.59E02 2.86E01 Ridge regression 1 1 431 493 106 0.026 1.33E04 2.11E04 2 2 073 804 45 0.011 2.11E04 2.11E04 3 788 932 19 0.005 2.11E04 2.11E04 4 1 431 285 96 0.024 2.11E04 2.14E04 5 1 432 043 143 0.035 2.14E04 2.21E04 6 1 430 714 172 0.042 2.21E04 2.31E04 7 1 431 378 239 0.059 2.31E04 2.49E04 8 1 431 379 478 0.119 2.49E04 2.85E04 9 1 431 378 807 0.200 2.85E04 3.85E04 10 1 431 378 1926 0.478 3.85E04 9.97E01 Table 3. Calibration statistics for the mean and ridge regression ceptably high false positives. Current actionability of these models ensembling methods for the fatal and nonfatal overdose models rests upon their ability to ascribe relative risk geographically within after application in the test set TN. Studies of their ability to predict counties and regions at highest risk in need of public health resource allocation are underway. Over- Ensembled model Brier score Intercept Slope Sz Sp dose prevention is currently directed after harm has already oc- Fatal mean 0.0001305 5.5329 0.6205 191.59 0.00 curred—for example, basing “high impact area” designations on Fatal ridge regression 0.0000673 0.3313 0.9599 1.55 0.120 deaths that have already occurred, not those we seek to prevent. Nonfatal mean 0.0004239 4.0305 0.7625 272.14 0.00 Nonfatal ridge regres- 0.0002923 1.7524 0.7942 9.34 0.00 Strengths sion The training-development-test framework in this study enriched case data in the presence of case imbalance without discarding valu- disambiguation. Third, CSMD data in TN contain human and non- able noncase comparator data. Our weak learner approach over- human controlled substance prescription data. Removing those pre- came computational constraints which may apply to other groups scriptions known to be nonhuman was straightforward but ensuring attempting similarly scaled experiments. Our academic-public part- nonhuman data are not miskeyed as human was not. nership catalyzed and made possible a modeling study at this scale Neither the fatal nor nonfatal models are suitable for direct clini- coupled with design choices to enable implementation at TDH. cal application. Given the resulting model AUPRCs, high false-posi- This study included the use of comprehensive real-world data de- tive rates are expected at virtually every cutoff. While it is possible rived from statewide operational datasets. Vital records, validated that clinically actionable subgroups may exist within the high-risk by medical examiners, and certified hospital discharge records were tiers, given the size of this study, most localized clinical interventions used in the context of our partnership with stakeholders at TDH to would likely see highly variable calculated individual risk and unac- ensure modeling decisions reflected the implementation environment 28 Journal of the American Medical Informatics Association, 2022, Vol. 29, No. 1 Table 4. AUROC and AUPRC for various subgroups in the test set for the fatal and nonfatal ridge regression ensembled models Characteristic Subgroup Fatal Nonfatal AUROC AUPRC Cases (%) Controls AUROC AUPRC Cases (%) Controls (%) (%) Age 20–29 0.83 0.00030 16 (1.74) 393 406 0.75 0.0014 141 (3.50) 381 438 (3.22) (3.12) 30–39 0.79 0.00036 126 (13.74) 1 417 833 0.75 0.0011 502 (12.48) 1 421 592 (11.60) (11.63) 40–49 0.79 0.00054 233 (25.41) 1 934 651 0.80 0.0013 590 (14.66) 1 945 431 (15.83) (15.91) 50–59 0.80 0.00062 351 (38.28) 2 583 639 0.82 0.0021 966 (24.01) 2 606 848 (21.14) (21.32) 60–69 0.84 0.00045 172 (18.76) 2 695 746 0.82 0.0019 1036 (25.75) 2 704 083 (22.06) (22.12) 70–79 0.91 0.00022 14 (1.53) 1 876 554 0.83 0.0015 546 (13.57) 1 856 381 (15.36) (15.18) 80–89 0.95 0.00020 5 (0.55) 948 381 0.78 0.0013 217 (5.39) 945 839 (7.76) (7.74) Sex F 0.84 0.00048 470 (48.81) 7 633 488 0.81 0.0016 2439 (60.61) 7 657 711 (53.42) (53.61) M 0.81 0.00044 447 (46.42) 4 556 799 0.80 0.0015 1570 (39.02) 4 542 227 (31.89) (31.80) U 0.74 0.00006 46 (4.78) 2 100 560 0.99 0.0009 15 (0.37) 2 084 944 (14.70) (14.60) Race Asian-American N/A N/A 0 (0.00) 12 253 N/A N/A 0 (0.00) 11 428 (0.090) (0.080) Black 0.86 0.00023 35 (3.63) 1 101 369 0.79 0.0010 198 (4.92) 1 105 227 (7.71) (7.74) Native American N/A N/A 0 (0.00) 1 888 N/A N/A 0 (0.00) 1947 (0.010) (0.010) Other 0.78 0.00021 2 (0.21) 32 659 0.88 0.0006 4 (0.10) 32 400 (0.23) (0.23) Unknown 0.79 0.00041 105 (10.90) 2 794 866 0.92 0.0031 413 (10.26) 2 763 247 (19.56) (19.34) White 0.83 0.00045 821 (85.25) 10 347 812 0.80 0.0015 3409 (84.72) 10 370 633 (72.41) (72.60) Ethnicity Hispanic 0.83 0.00034 2 (0.21) 25 870 0.81 0.0009 5 (0.12) 25 665 (0.18) (0.18) Non-Hispanic 0.81 0.00035 709 (73.62) 10 404 402 0.80 0.0014 3080 (76.54) 10 448 420 (72.80) (73.14) Unknown 0.86 0.00066 252 (26.17) 3 860 575 0.89 0.0021 939 (23.33) 3 810 797 (27.01) (26.68) RUCC 1, metro, >1 000 000 0.86 0.00064 357 (37.07) 4 856 038 0.84 0.0021 1593 (39.59) 4 898 834 (33.98) (34.29) 2, metro, 250 000–1 0.83 0.00042 301 (31.26) 3 852 307 0.83 0.0015 1006 (25.00) 3 829 032 000 000 (26.96) (26.8) 3, metro, <250 000 0.78 0.00021 76 (7.89) 1 491 989 0.83 0.0010 325 (8.08) 1 491 254 (10.44) (10.44) 4, urban, >20 000þ 0.80 0.00046 67 (6.96) 1 346 081 0.81 0.0016 398 (9.89) 1 330 476 metro adjacent (9.42) (9.31) 5, urban, >20 000þ N/A N/A 0 (0.00) 94 940 0.88 0.0007 13 (0.32) 101 859 (0.66) (0.71) 6, urban, 2500–19 0.82 0.00030 101 (10.49) 1 714 426 0.80 0.0013 459 (11.41) 1 709 034 999 metro adjacent (12.00) (11.96) 7, urban, 2500–19 0.71 0.00014 26 (2.70) 453 522 0.82 0.0016 108 (2.68) 442 912 999 (3.17) (3.10) 8, rural, <2500, 0.68 0.00020 21 (2.18) 320 187 0.82 0.0011 86 (2.14) 312 688 metro adjacent (2.24) (2.19) 9, rural, <2500 0.80 0.00510 14 (1.45) 161 357 0.82 0.0011 36 (0.89) 168 793 (1.13) (1.18) AUPRC: area under the precision recall curve; AUROC: area under the receiver operating curve; RUCC: Rural–Urban Continuity Codes. Journal of the American Medical Informatics Association, 2022, Vol. 29, No. 1 29 Figure 2. AUPRC of the LASSO, max, mean, median, min, and ridge regression ensembling methods for fatal and nonfatal overdose models when the number of partitions was changed. Note: compressed y-axes used to visualize minimal differences in models by number of partitions. AUPRC: area under the precision re- call curve. Figure 3. Top 15 predictive features by mean rank of importance for the fatal opioid overdose model. and were responsive to public health informatics requirements for validation with more recent data is needed. Carceral, other criminal overdose prevention. We leveraged the broad expertise among our justice data, and ambulatory clinical data were not available here, TDH and VUMC multidisciplinary partnership, working in close but have been previously used to predict opioid overdose risk. Our communication throughout. decision to predict overdose within 30 days, supported by measuring outcomes over time was chosen empirically and in discussion with TDH (Supplementary Material). Tools to identify patients at longer- Limitations term risk may be important for future prevention efforts. Statewide data used in this study were limited to a 6-year time pe- While our models did not explicitly use race as a predictor, other riod ending in 2017. Given the changing face of the opioid epidemic, variables were still likely proxies for race and health inequalities in 30 Journal of the American Medical Informatics Association, 2022, Vol. 29, No. 1 Figure 4. Top 15 predictive features by mean rank of importance for the nonfatal opioid overdose model. our predictions. Our subgroup analysis showed that race and age tors to produce ensembled opioid-related overdose risk models for vary in both AUROC and AUPRC (Table 4). Understanding the TN. Through an academic-state partnership, our models we able to cause and impact of inaccurately calculating risk for different sub- granularly predict fatal and nonfatal overdose risk within 30 days of groups may have critical policy implications. Sampling may improve receiving a controlled substance prescription. These predictions this disparity. More data are needed and a dedicated investigation in when aggregated may lead to more informed prevention efforts at collaboration with experts in health inequalities is indicated. A large the local, county, and regional levels. percentage of prescriptions had unknown race and gender given a lack of hospital discharge data for those individuals. FUNDING In addition, our outcome ascertainment strategy did not seek to determine if the patient’s last prescription was the actual cause of This work was supported by the Harold Rogers Prescription Drug Monitor- the overdose outcome nor was it used in those risk calculations. His- ing Program Grant No. 2016-PM-BX-K002 and Comprehensive Opioid Abuse Site-based Program Grant No. 2018-PM-BX-0007 awarded by the Bu- torical clinical and demographic information were also added to reau of Justice Assistance. The Bureau of Justice Assistance is a component of these models from batched HDDS data. Calculating risk in real-time the Department of Justice’s Office of Justice Programs, which also includes remains challenging given the additional steps necessary to incorpo- the Bureau of Justice Statistics, the National Institute of Justice, the Office of rate data entered close to the time of prediction. Juvenile Justice and Delinquency Prevention, the Office for Victims of Crime, and the SMART Office. Points of view or opinions in this document are those Future work of the author and do not necessarily represent the official position or policies Implementation of these models into internal state systems is cur- of the U.S. Department of Justice. rently being reviewed. The choice to do so may provide a platform The above funders had no role in any of the following: design and con- for prospective validation opportunities and public health perspec- duct of the study; collection, management, analysis, and interpretation of the tives unprecedented in TN. While a small number of proprietary data; preparation, review, or approval of the manuscript; or decision to sub- risk scores exist in this domain, none are being used at the state level mit the manuscript for publication (authors BT, AR, CC, and SL). in TN. Implementing these models would complement traditional RFS #34301-29519—Predicting opioid overdose in TN using controlled substance monitoring data and vital statistics (authors CGW, MR, DW, KR, epidemiologic methods that identify risk and guide planning for pre- QC, and CEF). vention. Future work includes a need to study the interpretability of Funding for the Research Derivative and BioVU Synthetic Derivative is these models and the need to assess for drift and apply recalibration through UL1 RR024975/RR/NCRR, PI: Gordon Bernard. prospectively. Outcome rates and prescription rates have changed since 2017. More advanced feature engineering and additional ex- ternal data sources might improve these models further. AUTHOR CONTRIBUTIONS MLM, BT, and CGW outlined the study. QC investigated study validity. KR CONCLUSION coordinated the work as performed. SCL, CEF, and AR compiled and ana- Historical statewide PDMP data, hospital discharge data, and death lyzed prior research. MR, SCL, DW, and ML processed all reference data. certificates from vital records were linked to socioeconomic indica- DW delineated study outcomes. MR transformed study data, trained the Journal of the American Medical Informatics Association, 2022, Vol. 29, No. 1 31 10. Martins SS, Ponicki W, Smith N, et al. Prescription drug monitoring pro- weak learners, and performed the subgroup analysis. CGW performed the grams operational characteristics and fatal heroin poisoning. Int J Drug weak learner ensembling. DW analyzed the modeling results and repeated the Policy 2019; 74: 174–80. analysis with varying partitions. MR, DW, and CEF produced figures. All 11. Geissert P, Hallvik S, Van Otterloo J, et al. High risk prescribing and opi- authors contributed to writing and revisions. MR and CGW take responsibil- oid overdose: prospects for prescription drug monitoring program based ity for the integrity and accuracy of the study. proactive alerts. Pain 2018; 159 (1): 150–6. 12. Chang H-Y, Krawczyk N, Schneider KE, et al. A predictive risk model for nonfatal opioid overdose in a statewide population of buprenorphine SUPPLEMENTARY MATERIAL patients. Drug Alcohol Depend 2019; 201: 127–33. Supplementary material is available at Journal of the American Medical Infor- 13. Hastings JS, Howison M, Inman SE. Predicting high-risk opioid prescrip- matics Association online. tions before they are given. Proc Natl Acad Sci U S A 2020; 117 (4): 1917–23. 14. Saloner B, Chang H-Y, Krawczyk N, et al. Predictive modeling of opioid overdose using linked statewide medical and criminal justice data. JAMA ACKNOWLEDGMENTS Psychiatry 2020; 77 (11): 1155. Thank you to the leadership of VUMC and of TDH for enabling this work 15. Nechuta SJ, Tyndall BD, Mukhopadhyay S, et al. Sociodemographic fac- and for their continued efforts toward combatting the opioid crisis in Tennes- tors, prescription history and opioid overdose deaths: a statewide analysis see. using linked PDMP and mortality data. Drug Alcohol Depend 2018; 190: 62–71. 16. Krishnaswami S, Mukhopadhyay S, McPheeters M, et al. Prescribing pat- terns before and after a non-fatal drug overdose using Tennessee’s Con- CONFLICT OF INTEREST STATEMENT trolled Substance Monitoring Database linked to hospital discharge data. None declared. Prev Med 2020; 130: 105883. 17. Kelty E, Hulse G. Fatal and non-fatal opioid overdose in opioid dependent patients treated with methadone, buprenorphine or implant naltrexone. Int J Drug Policy 2017; 46: 54–60. DATA AVAILABILITY 18. Park TW, Lin LA, Hosanagar A, et al. Understanding risk factors for opi- Study data including CSMD, HDDS, and vital statistics were provided by oid overdose in clinical populations to inform treatment and policy. J Ad- TDH under contract and a memorandum of understanding. These data are dict Med 2016; 10 (6): 369–81. not available for sharing by study authors but interested parties are welcome 19. Lo-Ciganic W-H, Huang JL, Zhang HH, et al. Evaluation of machine- to request these data from TDH. Reference data derived from online reposito- learning algorithms for predicting opioid overdose risk among medicare ries open to the public are shareable upon request. beneficiaries with opioid prescriptions. JAMA Netw Open 2019; 2 (3): e190968. 20. Mitra G, Wood E, Nguyen P, et al. Drug use patterns predict risk of non- REFERENCES fatal overdose among street-involved youth in a Canadian setting. Drug Alcohol Depend 2015; 153: 135–9. 1. Controlled Substance Monitoring Database: 2020 Report to the 111th 21. Phalen P, Ray B, Watson DP, et al. Fentanyl related overdose in Indianap- Tennessee General Assembly. Tennessee Department of Health; 2020. olis: estimating trends using multilevel Bayesian models. Addict Behav https://www.tn.gov/content/dam/tn/health/healthprofboards/csmd/2020_ 2018; 86: 4–10. CSMD_Annual_Report.pdf Accessed March 29, 2021. 22. Ellis RJ, Wang Z, Genes N, et al. Predicting opioid dependence from elec- 2. Chakravarthy B, Shah S, Lotfipour S. Prescription drug monitoring pro- tronic health records with machine learning. BioData Min 2019; 12: 3. grams and other interventions to combat prescription opioid abuse. West J 23. Glanz JM, Narwaney KJ, Mueller SR, et al. Prediction model for two-year Emerg Med 2012; 13 (5): 422–5. risk of opioid overdose among patients prescribed chronic opioid therapy. 3. Rolheiser LA, Cordes J, Subramanian SV. Opioid prescribing rates by con- J Gen Intern Med 2018; 33 (10): 1646–53. gressional districts, United States, 2016. Am J Public Health 2018; 108 24. Zedler B, Xie L, Wang L, et al. Development of a risk index for serious (9): 1214–9. prescription opioid-induced respiratory depression or overdose in Veter- 4. National Center for Injury Prevention and Control. U.S. Opioid Dispens- ans’ Health Administration Patients. Pain Med 2015; 16 (8): 1566–79. ing Rate Maps. Drug Overdose CDC Injury Center; 2020. https://www. 25. Zedler BK, Saunders WB, Joyce AR, et al. Validation of a screening risk in- cdc.gov/drugoverdose/maps/rxrate-maps.html Accessed March 29, 2021. dex for serious prescription opioid-induced respiratory depression or over- 5. Tennessee’s Annual Overdose Report 2021: Report on Epidemiologic dose in a US Commercial Health Plan Claims Database. Pain Med 2018; Data, Efforts, and Collaborations to Address the Overdose Epidemic. 19 (1): 68–78. Tennessee Department of Health; 2021. https://www.tn.gov/content/dam/ 26. Holt CT, McCall KL, Cattabriga G, et al. Using controlled substance re- tn/health/documents/pdo/2021%20TN%20Annual%20Overdose% ceipt patterns to predict prescription overdose death. Pharmacology 2018; 20Report.pdf Accessed March 29, 2021. 101 (3–4): 140–7. 6. National Institute on Drug Abuse. Tennessee: Opioid-Involved Deaths 27. Ferris LM, Saloner B, Krawczyk N, et al. Predicting opioid overdose and Related Harms. 2020. https://www.drugabuse.gov/drug-topics/ deaths using prescription drug monitoring program data. Am J Prev Med opioids/opioid-summaries-by-state/tennessee-opioid-involved-deaths-re- 2019; 57 (6): e211–7. lated-harms Accessed March 29, 2021. 28. Eisenberg MD, Saloner B, Krawczyk N, et al. Use of opioid overdose deaths 7. Patrick SW, Fry CE, Jones TF, et al. Implementation of prescription drug reported in one state’s criminal justice, hospital, and prescription databases monitoring programs associated with reductions in opioid-related death to identify risk of opioid fatalities. JAMA Intern Med 2019; 179 (7): 980–2. rates. Health Aff (Millwood) 2016; 35 (7): 1324–32. 29. Johnson C. Nearly 60 Doctors, Other Medical Workers Charged In Fede- 8. Lin DH, Lucas E, Murimi IB, et al. Physician attitudes and experiences ral Opioid Sting. NPR.org. 2019. https://www.npr.org/2019/04/17/ with Maryland’s prescription drug monitoring program (PDMP). Addic- 714014919/nearly-60-docs-other-medical-workers-face-charges-in-fede- tion 2017; 112 (2): 311–9. ral-opioid-sting Accessed March 29, 2021. 9. Strickler GK, Zhang K, Halpin JF, et al. Effects of mandatory pre- 30. Cerd a M, Ransome Y, Keyes KM, et al. Revisiting the role of the urban en- scription drug monitoring program (PDMP) use laws on prescriber vironment in substance use: the case of analgesic overdose fatalities. Am J registration and use and on risky prescribing. Drug Alcohol Depend Public Health 2013; 103 (12): 2252–60. 2019; 199: 1–9. 32 Journal of the American Medical Informatics Association, 2022, Vol. 29, No. 1 31. Kind AJ, Jencks S, Brock J, et al. Neighborhood socioeconomic disadvan- 39. Nechuta S, Mukhopadhyay S, Krishnaswami S, et al. Record linkage tage and 30 day rehospitalizations: an analysis of Medicare data. Ann In- approaches using prescription drug monitoring program and mortality tern Med 2014; 161 (11): 765–74. data for public health analyses and epidemiologic studies. Epidemiology 32. United States Census Bureau. Economic Census Tables: 2012–2017. Eco- 2020; 31 (1): 22–31. nomic Census Tables. https://www.census.gov/programs-surveys/eco- 40. Brown S, Elkin P, Rosenbloom S, et al. VA National Drug File Reference nomic-census/data/tables.html Accessed March 30, 2021. Terminology: a cross-institutional content coverage study. Stud Health 33. Rickles M, Rebeiro PF, Sizemore L, et al. Tennessee’s in-state vulnerability Technol Inform 2004; 107 (Pt 1): 477–81. assessment for a “rapid dissemination of human immunodeficiency virus 41. Clinical Classifications Software (CCS) 2015. Agency for Healthcare Re- or hepatitis C virus infection” event utilizing data about the opioid epi- search and Quality; 2016. https://www.hcup-us.ahrq.gov/toolssoftware/ demic. Clin Infect Dis 2018; 66 (11): 1722–32. ccs/CCSUsersGuide.pdf Accessed March 30, 2021. 34. United States Census Bureau. 2012–2016 ACS 5-Year Data Profile. 2016 42. American Medical Association. ICD-10-CM 2019 the Complete Offi- Data Profiles American Community Survey. US Census Bureau. https:// cial Codebook. Geneva, Switzerland: American Medical Association; www.census.gov/acs/www/data/data-tables-and-tools/data-profiles/2016/ 2018. Accessed March 30, 2021. 43. Wright MN, Ziegler A. ranger: a fast implementation of random for- 35. United States Department of Agriculture: Economic Research Service. ests for high dimensional data in Cþþ and R. J Stat Softw 2017; 77: Rural-Urban Continuum Codes. USDA ERS—Rural-Urban Continuum 1–17. Codes; 2020. https://www.ers.usda.gov/data-products/rural-urban-con- 44. Walsh CG, Sharman K, Hripcsak G. Beyond discrimination: a comparison tinuum-codes.aspx Accessed March 30, 2021. of calibration methods and clinical usefulness of predictive models of 36. Flanagan BE, Gregory EW, Hallisey EJ, et al. A social vulnerability index for readmission risk. J Biomed Inform 2017; 76: 9–18. disaster management. JHomelSecurEmerg Manag 2011; 8: Article 3, 1–24. 45. Steyerberg EW, Borsboom GJJM, Houwelingen H. V, et al. Validation 37. McPheeters M, Nechuta S, Miller S, et al. Prescription Drug Overdose Re- and updating of predictive logistic regression models: a study on sample port, 2018. TN Department of Health; 2018. https://www.tn.gov/content/ size and shrinkage. Statist Med 2004; 23 (16): 2567–86. dam/tn/health/documents/pdo/PDO_2018_Report_02.06.18.pdf 46. Tibshirani R. Regression shrinkage and selection via the lasso. J R Stat Soc Accessed March 30, 2021. Ser B Methodol 1996; 58 (1): 267–88. 38. World Health Organization. The International Statistical Classification of 47. Park TW, Saitz R, Ganoczy D, et al. Benzodiazepine prescribing patterns Diseases and Health Related Problems ICD-10: Tenth Revision. Volume 1: and deaths from drug overdose among US veterans receiving opioid anal- Tabular List. Chicago, IL: World Health Organization; 2004. gesics: case-cohort study. BMJ 2015; 350: h2698. http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png Journal of the American Medical Informatics Association Oxford University Press

Ensemble learning to predict opioid-related overdose using statewide prescription drug monitoring program and hospital discharge data in the state of Tennessee

Download PDF
11 pages

Loading next page...
 
/lp/oxford-university-press/ensemble-learning-to-predict-opioid-related-overdose-using-statewide-q4SQO8OLl5

References (44)

Publisher
Oxford University Press
Copyright
© The Author(s) 2021. Published by Oxford University Press on behalf of the American Medical Informatics Association.
ISSN
1067-5027
eISSN
1527-974X
DOI
10.1093/jamia/ocab218
Publisher site
See Article on Publisher Site

Abstract

Journal of the American Medical Informatics Association, 29(1), 2022, 22–32 doi: 10.1093/jamia/ocab218 Advance Access Publication Date: 19 October 2021 Research and Applications Research and Applications Ensemble learning to predict opioid-related overdose using statewide prescription drug monitoring program and hospital discharge data in the state of Tennessee 1 2 1 3 Michael Ripperger , Sarah C. Lotspeich , Drew Wilimitis , Carrie E. Fry , Allison 4 1 4 4 1 Roberts , Matthew Lenert , Charlotte Cherry , Sanura Latham , Katelyn Robinson , 1,2 1,3 4 1,5,6 Qingxia Chen , Melissa L. McPheeters , Ben Tyndall , and Colin G. Walsh 1 2 Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, Tennessee, USA, Department of Biosta- tistics, Vanderbilt University Medical Center, Nashville, Tennessee, USA, Department of Health Policy, Vanderbilt University Medical Center, Nashville, Tennessee, USA, Office of Informatics and Analytics, Tennessee Department of Health, Nashville, 5 6 Tennessee, USA, Department of Medicine, Vanderbilt University Medical Center, Nashville, Tennessee, USA, and Department of Psychiatry, Vanderbilt University Medical Center, Nashville, Tennessee, USA Corresponding Author: Colin G. Walsh, MD, MA, Department of Biomedical Informatics, Vanderbilt University Medical Center, 2525 West End Avenue, Suite 1475, Nashville, TN 37203, USA; colin.walsh@vumc.org Received 4 April 2021; Revised 3 September 2021; Editorial Decision 25 September 2021 ABSTRACT Objective: To develop and validate algorithms for predicting 30-day fatal and nonfatal opioid-related overdose using statewide data sources including prescription drug monitoring program data, Hospital Discharge Data System data, and Tennessee (TN) vital records. Current overdose prevention efforts in TN rely on descriptive and retrospective analyses without prognostication. Materials and Methods: Study data included 3 041 668 TN patients with 71 479 191 controlled substance pre- scriptions from 2012 to 2017. Statewide data and socioeconomic indicators were used to train, ensemble, and calibrate 10 nonparametric “weak learner” models. Validation was performed using area under the receiver operating curve (AUROC), area under the precision recall curve, risk concentration, and Spiegelhalter z-test statistic. Results: Within 30 days, 2574 fatal overdoses occurred after 4912 prescriptions (0.0069%) and 8455 nonfatal overdoses occurred after 19 460 prescriptions (0.027%). Discrimination and calibration improved after ensem- bling (AUROC: 0.79–0.83; Spiegelhalter P value: 0–.12). Risk concentration captured 47–52% of cases in the top quantiles of predicted probabilities. Discussion: Partitioning and ensembling enabled all study data to be used given computational limits and helped mediate case imbalance. Predicting risk at the prescription level can aggregate risk to the patient, pro- vider, pharmacy, county, and regional levels. Implementing these models into Tennessee Department of Health systems might enable more granular risk quantification. Prospective validation with more recent data is needed. Conclusion: Predicting opioid-related overdose risk at statewide scales remains difficult and models like these, which required a partnership between an academic institution and state health agency to develop, may comple- ment traditional epidemiological methods of risk identification and inform public health decisions. Key words: drug overdose, opioid epidemic, machine learning, prescription drug monitoring programs, vital statistics V The Author(s) 2021. Published by Oxford University Press on behalf of the American Medical Informatics Association. This is an Open Access article distributed under the terms of the Creative Commons Attribution-NonCommercial-NoDerivs licence (https://creativecommons.org/licenses/ by-nc-nd/4.0/), which permits non-commercial reproduction and distribution of the work, in any medium, provided the original work is not altered or transformed in any way, and that the work is properly cited. For commercial re-use, please contact journals.permissions@oup.com 22 Journal of the American Medical Informatics Association, 2022, Vol. 29, No. 1 23 and characterizing risk with precise and automated predictive mod- INTRODUCTION els. Part of our efforts was to leverage known sociodemographic and We sought to develop and validate implementable predictive models economic factors relating to mental and physical health. Community for the state of Tennessee (TN) to predict (1) fatal and (2) nonfatal characteristics have been known to be predictive of OPR overdose opioid-related overdose risk by leveraging statewide data sources risk. provided by the Tennessee Department of Health (TDH). Through Seeking to predict future risk by combining linked PDMP and our academic-state partnership, we applied ensemble learning to fa- overdose data, TDH partnered with VUMC to help the state under- tal and nonfatal overdose prediction using statewide controlled sub- stand the opioid epidemic statewide, target interventions, and allo- stance prescription data, hospital discharge diagnoses, and causes of cate scarce resources accordingly. Adhering to the architectural and death from vital records. implementation requirements of TDH, the VUMC team derived a data management strategy, sourced a wide array of social determi- nant variables to help quantify risk, and evaluated our approach. BACKGROUND AND SIGNIFICANCE Once implemented in TDH systems, such models might allow TN to further support the greatest at-risk communities and identify inter- The link between the current opioid epidemic in the United States vention touch points within the community health system. and the over-prescribing of opioid pain relievers (OPRs) has been well established. Over-prescribing and OPR-related harms were first observed in the 1990s and some states including TN have experi- 2,3 MATERIALS AND METHODS enced higher rates of prescribing and the subsequent harms. Near the opioid prescribing peak in 2010, TN providers wrote more OPR This study was approved by the VUMC Institutional Review Board prescriptions than there were residents in the state. Between 2014 (#171323). and 2018, OPR-related deaths rose 49% to an annual cost of 1307 5,6 lives. The United States meanwhile has seen a near-universal Data sources adoption of prescription drug monitoring programs (PDMPs) with Controlled Substance Monitoring Database (CSMD, TN’s PDMP) intentions to combat the opioid epidemic by monitoring prescribing data, Hospital Discharge Data System (HDDS) data, and TN death histories, informing providers, and identifying concerns with varying certificates were combined to produce a 6-year observational cohort 7–10 success. Although PDMPs have seldomly been used to predict that spanned from the beginning of 2012 through the end of 2017. imminent risk at the patient level, prevention at the practice, county, Publicly available socioeconomic indicators relating to health, or regional levels might be possible if accurate algorithms are devel- healthcare utilization, and treatment access were compiled and 11–14 oped, validated, and implemented. Severely affected by the opi- mapped to either ZIP codes or counties. oid crisis, TN has already linked its controlled substance PDMP (II– The following were mapped to residential ZIP codes: Area Dep- V scheduled and gabapentin) to statewide mortality data and hospi- rivation Index (ADI); statistics on employment from the U.S. Census 15,16 tal discharge data. In this study, researchers at TDH and Van- Bureau; and Medication-Assisted Treatment (MAT) locations in- derbilt University Medical Center (VUMC) partnered to develop cluding buprenorphine providers, methadone clinics, and Opioid and validate the first scalable predictive models from statewide data- 31,32 Treatment Programs (OTPs) from data aggregated by TDH. sets in TN for the related but disparate outcomes: (1) fatal and (2) TN age-adjusted morbidity rates from TDH; the Tennessee Vulnera- 17,18 nonfatal opioid overdose. bility Index (TVI) from TDH; statistics on income, poverty, college The application of machine learning to predict individual risk is education, crowding, and private insurance from the American not new in the biomedical literature nor in OPR overdose preven- Community Survey (ACS); Rural–Urban Continuity Codes (RUCC) tion. Prior studies have predicted overdose risk using Medicare from the U.S. Department of Agriculture; the Social Vulnerability claims, self-reported substance use patterns, and demo- Index (SVI) from the Centers for Disease Control and Prevention 13,19–21 graphics. Many studies have also utilized electronic health (CDC); and Anti-Drug Abuse Coalition services from TDH were records with or without vital records including at Mt. Sinai, in the 33–36 mapped to individual counties. A full list of sourced data is state of Colorado, and at the Veteran’s Health Administration available within the Supplementary Material. 22–25 (VHA). Few US states, however, have specifically used PDMP data to predict overdose—namely Maine, Oregon, and Mary- 11,26,27 Outcome ascertainment land. In Maryland, hospital discharge, healthcare utilization, The outcomes of interest in this study were fatal and nonfatal and criminal justice data have been linked to predict future OPR 12,14,28 opioid-related overdose events that occurred within 30 days of a overdose risk for individuals. Our study likewise combines controlled substance prescription fill. The 30-day time window was predictive modeling with comprehensive statewide data. No previ- chosen after plotting the accumulation of overdoses over time after ous studies to our knowledge have assembled these kinds of data for a prescription fill (Supplementary Material). Fatal and nonfatal a large, southern US state like TN where the rates of OPR prescrib- overdoses were identified consistent with methods used by TDH in ing are much higher than the national average. their annual Prescription Drug Overdose Reports. Fatal overdoses In 2019, a federal investigation led by the Department of Justice were identified from TN death certificates using International Clas- (DOJ) uncovered fraud and inappropriate opioid prescribing in TN sification of Disease, revision 10 (ICD-10) codes. Nonfatal over- and resulted in the arrests of multiple physicians, pharmacists, and 29 doses were identified in the HDDS with specified opioid-related other health professionals. Such measures relied upon descriptive diagnostic codes (Supplementary Material). analytics for harms that had already occurred years prior. While monitoring and descriptive analytics may provide a lens into the cur- rent state of the opioid epidemic, they cannot identify the next pa- Predictive modeling details tient, practice, or community at risk. The goal of this work was to Our modeling choices were as follows: (1) establish a vector of so- supplement these traditional epidemiological methods of identifying cioeconomic indicators based on a patient’s last reported location 24 Journal of the American Medical Informatics Association, 2022, Vol. 29, No. 1 from the PDMP (from the time of the previous prescription); (2) each random forest. In total, 20 random forests were developed count the cumulative number of prior medications, diagnostic codes, from the 20 training subsets—10 for each of the 2 outcomes. and hospital visits by type a patient has accumulated thus far; and During training, each training subset itself was split into a 90% (3) add age, sex, and derived variables that represent a patient’s pre- training set and a 10% testing set to allow predictions to be made scription history for controlled substances. Variables chosen in- for each case. Each case was placed in the testing set of each subset cluded the sums of distinct practitioners, distinct pharmacies, exactly one time which guarantied all case data were used in training distinct hospital identifiers, total prescriptions, total morphine milli- and at least one prediction for each associated record was generated. gram equivalents, short/long-acting OPR prescriptions, overlapping After the weak learners were ensembled and calibrated in the devel- OPR and benzodiazepine prescriptions, prior medications for opioid opment set, the resulting ensembled models were validated in a final use disorder, and opioid-naı ¨ve prescriptions as defined as not having held-out testing set. A conceptual diagram of this training scheme is an OPR prescription within the last 45 days (Supplementary Mate- shown (Figure 1). rial). Race and ethnicity were not explicitly represented in our mod- els. Modeling at the prescription level was done to create time- dependent and granular risk predictions which could then be aggre- gated to practice, pharmacy, local, county, and regional levels. This Calibration approach intended to potentially guide planning and response activi- A development set consisting of 5% of the data was reserved to cor- ties at varying levels of detail. rect the miscalibrations from the under-sampled controls in the training subsets. We compared 7 methods of ensembling and cali- bration. Either the minimum, maximum, mean, or median predic- Data preprocessing tion was taken from the 10 weak learner predictions and passed Patient linkage across our datasets relied on TDH-determined mas- through logistic calibration, or the 10 weak learner predictions were ter patient indexing. Only records with valid person identifiers used as inputs for ridge regression, random forest, or penalized re- were retained, and records determined to be related to a nonhuman 43–46 gression (LASSO). patient (ie, veterinary prescription records) were removed. Hospital Logistic calibration, when applied, was defined by training a uni- records from the HDDS were limited to verified inpatient variate logistic regression in the calibration set where the sole pre- encounters. dictor was the aggregate in question (eg, max) and the outcome was Precise ADI and RUCC features were developed from the mini- either fatal or nonfatal overdose. The resulting generalized linear mum, maximum, and mean values of each ZIP code. Other ZIP models along with the aggregation methods were then considered as code features were developed from county data using the TN county ensemblers. The more complex ensembling methods trained multi- that contained the majority area of each ZIP code. OTP and metha- variate models using the 10 weak learners as predictors. Random done clinic availability were modeled using a 60-mile radius, repre- forest was used for comparison for 2 types of penalized logistic re- senting a practical range for driving a normal distance in TN (90– gression: L1-regularized (LASSO) and L2-regularized (RIDGE) re- 120 minutes driving time). gression. All resulting models were expected to be calibrated as they To reduce the dimensionality of PDMP and HDDS features, were either trained on the calibration set or calibrated via logistic prior medications and diagnoses were grouped to higher-order cate- calibration. gories using the National Drug File-Reference Terminology (NDF- Final ensembled and calibrated algorithms were then tested on FT), Pharmacologic Classes and Clinical Classification Software the test set. Weak learners were tested on the calibration set. We (CCS), Level 2 groupings from National Drug Codes (NDCs) and note that no additional calibration was performed on the test set, International Classification of Disease, revision 10, Clinical Modifi- making it a pure test of calibration as well as discrimination. 40–42 cation (ICD-10-CM) codes. In total, 342 features were used for model training after this dimensionality reduction and only entries in patient records prior to prediction dates were used. Performance assessment methods Sampling strategy and model training Discrimination performance metrics included area under the re- We separated the data into 75% training, 5% development, and ceiver operating curve (AUROC), area under the precision recall 20% testing partitions to ultimately derive one model for fatal over- curve (AUPRC), and risk concentration. Risk concentration was per- dose and one model for nonfatal. All prescriptions in the data that formed by dividing the predictions from the test set into 10 quantiles were associated with an individual were added together to only one and calculating the proportion of all the cases those quantiles held. set to prevent leak between training and testing within individuals. Calibration was assessed using Spiegelhalter z-test. The ridge re- Models were trained in the training set and then calibrated, gression ensembles were further assessed for performance differen- ensembled, and evaluated in the development set. ces by subgroups consisting of race, ethnicity, and gender as The training set was equally divided into 10 smaller training par- determined by hospital records as well as age and RUCC codes from titions or subsets due to computational limits. To help combat case residential ZIP codes for urbanicity/rurality. To test how perfor- imbalance, all cases and their associated records were added to each mance varied when the number of partitions in the training set was training set, but only 10% of all the controls from the entire training changed, additional models were trained using N¼ 5or N¼ 15 and set were included in an individual training set (ie, only one training compared using AUPRC. For both fatal and nonfatal overdose, we set contained any one control). Ten random regression forest “weak ranked each feature by taking the mean of the important values learners” were then developed from the training subsets using the from the 10 weak learners—determined by the variance of responses ranger R package with an estimated response variance splitting crite- from each random forest. A full list is available within the Supple- ria. To help limit memory consolidation, 200 trees were used for mentary Material. Journal of the American Medical Informatics Association, 2022, Vol. 29, No. 1 25 Figure 1. Conceptual diagram of training data splits, weak learners, the ensembling/calibration development step, and the testing step. RESULTS LASSO). Random forest performed worse compared to other meth- ods of aggregation. The top 2 performing ensembles, mean and ridge Study data regression, were further evaluated in the risk concentration and cali- Study data included 71 479 191 controlled substance prescriptions bration analyses. across 3 041 668 TN patients. As sourced from hospital records, when available: 1 409 556 (46.3%) patients were Female; 958 440 (31.5%) patients were Male; and 673 672 (22.1%) patients were Risk concentration and calibration performance Unknown. Patients by coded race showed 7104 (0.23%) patients Risk concentration showed that, in the test set, the mean and ridge were Asian-American; 360 314 (11.8%) patients were Black; 704 regression ensembling methods concentrated 47–52% of the over- (0.023%) patients were Native American; 20 147 (0.66%) patients dose outcomes within the top quantiles of predicted probabilities were Other; 1 851 324 (61.0%) patients were White; and 802 075 (Table 2). Both top quantiles contained 10% of the test set predic- (26.4%) patients were Unknown. Patients by coded ethnicity also tions. Overlapping quantiles where the predictions had the same val- showed 16 061 (0.53%) patients were Hispanic; 2 064 654 (67.8%) ues were combined as seen by the number of prescriptions in the patients were non-Hispanic; and 960 953 (32.0%) patients were Un- first quantile of the fatal mean ensembling method. known. Within 30 days, 2574 fatal overdoses occurred after 4912 Calibration measured the degree to which the predictions (0.0069%) prescriptions and 8455 nonfatal overdoses occurred after reflected the true outcome prevalences. The ensembled models pre- 19 460 (0.027%) prescriptions. Nearly 60% of all fatal and nonfatal dicting fatal overdose showed nonsignificant calibration from mean overdoses in the data occurred within 30 days of a prescription (Sup- ensembling and significant calibration from ridge regression as indi- plementary Material). cated by the nonsignificant Spiegelhalter z-test. The ensembled mod- els for nonfatal overdose showed better calibration for ridge regression than for mean ensembling although both were nonsignifi- Weak learner and ensembling model performance cantly calibrated (Table 3). The ridge regression ensembling method Both the fatal and nonfatal weak learner models had similar preva- was subsequently used to analyze performance variations by sub- lence rates throughout the training set and showed consistent groups. AUROC and AUPRC values when applied to the development set (Table 1). AUROC was useful to compare these models simply de- Subgroup performance differences and partition spite having known problems when assessing absolute performance with case imbalance. The total number of cases and controls in the variation training set were 3725 and 53 591 596 (0.0069%) for fatal and 14 Both the fatal and nonfatal ridge regression ensembles were tested 695 and 53 580 626 (0.027%) for nonfatal overdose. on subgroups in the test set. AUROC and AUPRC values varied by Discrimination varied by ensembling method when applied to subgroup in age, sex, race, ethnicity, and RUCC values of residential the test set for both fatal and nonfatal overdose (Table 1). Averaging ZIP codes (Table 4). Case and control percentages among the sub- or selecting the minimum or maximum predictions from the 10 groups also varied. weak learner models for both fatal and nonfatal produced similar Repeating the modeling experiments for N¼ 5 and N¼ 15 results to using more complex methods of aggregation (eg, ridge, showed no differences in AUPRC values when the number of parti- 26 Journal of the American Medical Informatics Association, 2022, Vol. 29, No. 1 Table 1. Characteristics of both the 20 weak learner models in the development set and the 14 ensemble models in the test set for fatal and nonfatal overdose Fatal overdose Nonfatal overdose Weak learner/en- AUROC AUPRC Cases Controls % Outcomes AUROC AUPRC Cases Controls % Outcomes semble WL1 0.77 0.00024 224 3 566 077 0.0063 0.79 0.0018 1131 3 580 452 0.032 WL2 0.73 0.00023 0.78 0.0016 WL3 0.76 0.00023 0.79 0.0019 WL4 0.75 0.00024 0.78 0.0016 WL5 0.72 0.00026 0.79 0.0021 WL6 0.73 0.00027 0.80 0.0019 WL7 0.71 0.00023 0.79 0.0017 WL8 0.78 0.00024 0.80 0.0017 WL9 0.75 0.00025 0.79 0.0017 WL10 0.72 0.00026 0.78 0.0015 Maximum 0.83 0.00040 963 14 316 606 0.0067 0.82 0.0014 4031 14 309 753 0.028 Minimum 0.67 0.00032 0.76 0.0014 Mean 0.83 0.00042 0.83 0.0015 Median 0.80 0.00041 0.82 0.0015 LASSO 0.79 0.00038 0.82 0.0015 Ridge 0.83 0.00042 0.83 0.0016 Random forest 0.38 0.00007 0.49 0.0004 Note: Ensemble models combined and calibrated weak learner model predictions from the development set. AUPRC: area under the precision recall curve; AUROC: area under the receiver operating curve. tions was changed (Figure 2). Absolute change by partition choice The subgroup performance analysis showed that the ridge regres- was minimal as evidenced by the small absolute differences in y-axes sion models resulted in disparate performance in terms of AUPRC shown (eg, <0.0001 change in AUPRC by number of folds for the and AUROC for race and age despite small absolute AUPRC differ- fatal model). ences (Table 4). Case imbalance may be driving these differences. Correcting performance differences is necessary for accurately assessing risk in the state. When the number of training partitions Weak learner feature importances was varied, AUPRCs varied minimally if at all (Figure 2). The top 15 model features from the 10 weak learner models for fatal In the fatal overdose model, the top predictors were face valid as and nonfatal overdose were determined by ranking their mean re- known risk factors for opioid-related overdose (Figure 3). The total sponse variances (Figures 3 and 4). Twelve features were within the quantity of controlled substances prescribed was close to the top of top 15 of both the fatal and nonfatal overdose models. the list. Notably, overlapping benzodiazepine prescriptions were more important in the prediction of fatal opioid-related overdose than nonfatal. Multidrug combinations have been known to play a large role in the fatality potential of opioid-related overdoses and DISCUSSION benzodiazepines have a synergistic respiratory depressant effect This study supports the validity of combining statewide PDMP data when taken with opioids. with clinical discharge and socioeconomic data to predict fatal and Informatics implications of this study include the importance of nonfatal opioid overdose within 30 days of a controlled substance partitioning and sampling to lessen overfitting in settings with high prescription fill. Partitioning and ensembling the data allowed us to stake, but rare (at state scale), outcomes. Efforts to predict risk at an use all study data despite computational limits. We modeled risk at actionable timepoint, for example, a prescription fill event, do not the prescription level, making these models applicable to any indi- obviate aggregating risk analyses to levels relevant for public health vidual prescription with historical data. Aggregating these predic- intervention such as the community and regional levels. US states tions enables risk to be calculated at varying levels of detail for have long implemented PDMPs, but most have not disseminated better informed public health decision-making. predictive modeling approaches at this scale and none of the nearby AUROCs and AUPRCs of the fatal and nonfatal models in the states in the southern United States have done so. Characterizing development set improved in the test set after ensembling (Table 1). OPR risk in our state might inform better prevention both in TN Risk concentration analyses consistently captured half the outcomes and in neighbor states, as the overdose crisis varies considerably of interest in the top quantiles of risk (Table 2). Given the presence near and across state lines. of case imbalance, the highest risk quantiles may enable TN to focus Several attributes of this overdose modeling problem increased prevention efforts more efficiently. Both ensembles were miscali- its complexity. First, extreme case imbalance resulted from the rarity brated when predicting nonfatal overdose, but the ridge regression of fatal and nonfatal overdoses at statewide scale—prevalence less ensemble was calibrated when predicting fatal overdose (Table 3). than a fraction of 1%. Second, person disambiguation in data that Future recalibration efforts should reduce these gaps. Predicting fa- were manually entered by pharmacists into the CSMD resulted in re- tal overdose in the future may enable better prevention. Prospective liance on constructed, probabilistic patient mapping indices. Ongo- evaluation with more recent data is needed. ing work within TDH continues to refine and improve this Journal of the American Medical Informatics Association, 2022, Vol. 29, No. 1 27 Table 2. Risk concentration of the ensembled fatal and nonfatal prediction models which were validated in the test set Fatal/Nonfatal Ensembling method Quantile Prescriptions Cases Proportion of cases Inclusive lower bound Exclusive upper bound Fatal Mean 1 4 106 507 32 0.033 0.00Eþ00 1.65E08 2 210 474 4 0.004 1.65E08 6.28E08 3 1 412 596 14 0.015 6.28E08 3.33E05 4 1 429 211 33 0.034 3.33E05 3.53E04 5 1 431 757 40 0.042 3.53E04 7.07E04 6 1 432 104 66 0.069 7.07E04 1.46E03 7 1 434 688 100 0.104 1.46E03 2.92E03 8 1 428 476 171 0.178 2.92E03 6.80E03 9 1 431 756 503 0.522 6.80E03 3.34E01 Ridge regression 1 1 431 758 81 0.084 3.85E05 5.47E05 2 4 776 940 43 0.045 5.47E05 5.48E05 3 950 091 4 0.004 5.48E05 5.48E05 4 1 443 236 55 0.057 5.48E05 5.54E05 5 1 420 274 60 0.062 5.54E05 5.66E05 6 1 431 800 103 0.107 5.66E05 6.00E05 7 1 431 716 159 0.165 6.00E05 6.73E05 8 1 431 754 458 0.476 6.73E05 3.19E01 Nonfatal Mean 1 1 929 336 43 0.011 0.00Eþ00 1.93E08 2 933 421 25 0.006 1.93E08 8.68E06 3 1 437 802 67 0.017 8.68E06 2.50E04 4 1 425 055 81 0.020 2.50E04 6.91E04 5 1 432 474 123 0.031 6.91E04 1.41E03 6 1 430 183 143 0.035 1.41E03 2.55E03 7 1 431 883 290 0.072 2.55E03 4.50E03 8 1 431 048 415 0.103 4.50E03 8.04E03 9 1 431 205 835 0.207 8.04E03 1.59E02 10 1 431 377 2009 0.498 1.59E02 2.86E01 Ridge regression 1 1 431 493 106 0.026 1.33E04 2.11E04 2 2 073 804 45 0.011 2.11E04 2.11E04 3 788 932 19 0.005 2.11E04 2.11E04 4 1 431 285 96 0.024 2.11E04 2.14E04 5 1 432 043 143 0.035 2.14E04 2.21E04 6 1 430 714 172 0.042 2.21E04 2.31E04 7 1 431 378 239 0.059 2.31E04 2.49E04 8 1 431 379 478 0.119 2.49E04 2.85E04 9 1 431 378 807 0.200 2.85E04 3.85E04 10 1 431 378 1926 0.478 3.85E04 9.97E01 Table 3. Calibration statistics for the mean and ridge regression ceptably high false positives. Current actionability of these models ensembling methods for the fatal and nonfatal overdose models rests upon their ability to ascribe relative risk geographically within after application in the test set TN. Studies of their ability to predict counties and regions at highest risk in need of public health resource allocation are underway. Over- Ensembled model Brier score Intercept Slope Sz Sp dose prevention is currently directed after harm has already oc- Fatal mean 0.0001305 5.5329 0.6205 191.59 0.00 curred—for example, basing “high impact area” designations on Fatal ridge regression 0.0000673 0.3313 0.9599 1.55 0.120 deaths that have already occurred, not those we seek to prevent. Nonfatal mean 0.0004239 4.0305 0.7625 272.14 0.00 Nonfatal ridge regres- 0.0002923 1.7524 0.7942 9.34 0.00 Strengths sion The training-development-test framework in this study enriched case data in the presence of case imbalance without discarding valu- disambiguation. Third, CSMD data in TN contain human and non- able noncase comparator data. Our weak learner approach over- human controlled substance prescription data. Removing those pre- came computational constraints which may apply to other groups scriptions known to be nonhuman was straightforward but ensuring attempting similarly scaled experiments. Our academic-public part- nonhuman data are not miskeyed as human was not. nership catalyzed and made possible a modeling study at this scale Neither the fatal nor nonfatal models are suitable for direct clini- coupled with design choices to enable implementation at TDH. cal application. Given the resulting model AUPRCs, high false-posi- This study included the use of comprehensive real-world data de- tive rates are expected at virtually every cutoff. While it is possible rived from statewide operational datasets. Vital records, validated that clinically actionable subgroups may exist within the high-risk by medical examiners, and certified hospital discharge records were tiers, given the size of this study, most localized clinical interventions used in the context of our partnership with stakeholders at TDH to would likely see highly variable calculated individual risk and unac- ensure modeling decisions reflected the implementation environment 28 Journal of the American Medical Informatics Association, 2022, Vol. 29, No. 1 Table 4. AUROC and AUPRC for various subgroups in the test set for the fatal and nonfatal ridge regression ensembled models Characteristic Subgroup Fatal Nonfatal AUROC AUPRC Cases (%) Controls AUROC AUPRC Cases (%) Controls (%) (%) Age 20–29 0.83 0.00030 16 (1.74) 393 406 0.75 0.0014 141 (3.50) 381 438 (3.22) (3.12) 30–39 0.79 0.00036 126 (13.74) 1 417 833 0.75 0.0011 502 (12.48) 1 421 592 (11.60) (11.63) 40–49 0.79 0.00054 233 (25.41) 1 934 651 0.80 0.0013 590 (14.66) 1 945 431 (15.83) (15.91) 50–59 0.80 0.00062 351 (38.28) 2 583 639 0.82 0.0021 966 (24.01) 2 606 848 (21.14) (21.32) 60–69 0.84 0.00045 172 (18.76) 2 695 746 0.82 0.0019 1036 (25.75) 2 704 083 (22.06) (22.12) 70–79 0.91 0.00022 14 (1.53) 1 876 554 0.83 0.0015 546 (13.57) 1 856 381 (15.36) (15.18) 80–89 0.95 0.00020 5 (0.55) 948 381 0.78 0.0013 217 (5.39) 945 839 (7.76) (7.74) Sex F 0.84 0.00048 470 (48.81) 7 633 488 0.81 0.0016 2439 (60.61) 7 657 711 (53.42) (53.61) M 0.81 0.00044 447 (46.42) 4 556 799 0.80 0.0015 1570 (39.02) 4 542 227 (31.89) (31.80) U 0.74 0.00006 46 (4.78) 2 100 560 0.99 0.0009 15 (0.37) 2 084 944 (14.70) (14.60) Race Asian-American N/A N/A 0 (0.00) 12 253 N/A N/A 0 (0.00) 11 428 (0.090) (0.080) Black 0.86 0.00023 35 (3.63) 1 101 369 0.79 0.0010 198 (4.92) 1 105 227 (7.71) (7.74) Native American N/A N/A 0 (0.00) 1 888 N/A N/A 0 (0.00) 1947 (0.010) (0.010) Other 0.78 0.00021 2 (0.21) 32 659 0.88 0.0006 4 (0.10) 32 400 (0.23) (0.23) Unknown 0.79 0.00041 105 (10.90) 2 794 866 0.92 0.0031 413 (10.26) 2 763 247 (19.56) (19.34) White 0.83 0.00045 821 (85.25) 10 347 812 0.80 0.0015 3409 (84.72) 10 370 633 (72.41) (72.60) Ethnicity Hispanic 0.83 0.00034 2 (0.21) 25 870 0.81 0.0009 5 (0.12) 25 665 (0.18) (0.18) Non-Hispanic 0.81 0.00035 709 (73.62) 10 404 402 0.80 0.0014 3080 (76.54) 10 448 420 (72.80) (73.14) Unknown 0.86 0.00066 252 (26.17) 3 860 575 0.89 0.0021 939 (23.33) 3 810 797 (27.01) (26.68) RUCC 1, metro, >1 000 000 0.86 0.00064 357 (37.07) 4 856 038 0.84 0.0021 1593 (39.59) 4 898 834 (33.98) (34.29) 2, metro, 250 000–1 0.83 0.00042 301 (31.26) 3 852 307 0.83 0.0015 1006 (25.00) 3 829 032 000 000 (26.96) (26.8) 3, metro, <250 000 0.78 0.00021 76 (7.89) 1 491 989 0.83 0.0010 325 (8.08) 1 491 254 (10.44) (10.44) 4, urban, >20 000þ 0.80 0.00046 67 (6.96) 1 346 081 0.81 0.0016 398 (9.89) 1 330 476 metro adjacent (9.42) (9.31) 5, urban, >20 000þ N/A N/A 0 (0.00) 94 940 0.88 0.0007 13 (0.32) 101 859 (0.66) (0.71) 6, urban, 2500–19 0.82 0.00030 101 (10.49) 1 714 426 0.80 0.0013 459 (11.41) 1 709 034 999 metro adjacent (12.00) (11.96) 7, urban, 2500–19 0.71 0.00014 26 (2.70) 453 522 0.82 0.0016 108 (2.68) 442 912 999 (3.17) (3.10) 8, rural, <2500, 0.68 0.00020 21 (2.18) 320 187 0.82 0.0011 86 (2.14) 312 688 metro adjacent (2.24) (2.19) 9, rural, <2500 0.80 0.00510 14 (1.45) 161 357 0.82 0.0011 36 (0.89) 168 793 (1.13) (1.18) AUPRC: area under the precision recall curve; AUROC: area under the receiver operating curve; RUCC: Rural–Urban Continuity Codes. Journal of the American Medical Informatics Association, 2022, Vol. 29, No. 1 29 Figure 2. AUPRC of the LASSO, max, mean, median, min, and ridge regression ensembling methods for fatal and nonfatal overdose models when the number of partitions was changed. Note: compressed y-axes used to visualize minimal differences in models by number of partitions. AUPRC: area under the precision re- call curve. Figure 3. Top 15 predictive features by mean rank of importance for the fatal opioid overdose model. and were responsive to public health informatics requirements for validation with more recent data is needed. Carceral, other criminal overdose prevention. We leveraged the broad expertise among our justice data, and ambulatory clinical data were not available here, TDH and VUMC multidisciplinary partnership, working in close but have been previously used to predict opioid overdose risk. Our communication throughout. decision to predict overdose within 30 days, supported by measuring outcomes over time was chosen empirically and in discussion with TDH (Supplementary Material). Tools to identify patients at longer- Limitations term risk may be important for future prevention efforts. Statewide data used in this study were limited to a 6-year time pe- While our models did not explicitly use race as a predictor, other riod ending in 2017. Given the changing face of the opioid epidemic, variables were still likely proxies for race and health inequalities in 30 Journal of the American Medical Informatics Association, 2022, Vol. 29, No. 1 Figure 4. Top 15 predictive features by mean rank of importance for the nonfatal opioid overdose model. our predictions. Our subgroup analysis showed that race and age tors to produce ensembled opioid-related overdose risk models for vary in both AUROC and AUPRC (Table 4). Understanding the TN. Through an academic-state partnership, our models we able to cause and impact of inaccurately calculating risk for different sub- granularly predict fatal and nonfatal overdose risk within 30 days of groups may have critical policy implications. Sampling may improve receiving a controlled substance prescription. These predictions this disparity. More data are needed and a dedicated investigation in when aggregated may lead to more informed prevention efforts at collaboration with experts in health inequalities is indicated. A large the local, county, and regional levels. percentage of prescriptions had unknown race and gender given a lack of hospital discharge data for those individuals. FUNDING In addition, our outcome ascertainment strategy did not seek to determine if the patient’s last prescription was the actual cause of This work was supported by the Harold Rogers Prescription Drug Monitor- the overdose outcome nor was it used in those risk calculations. His- ing Program Grant No. 2016-PM-BX-K002 and Comprehensive Opioid Abuse Site-based Program Grant No. 2018-PM-BX-0007 awarded by the Bu- torical clinical and demographic information were also added to reau of Justice Assistance. The Bureau of Justice Assistance is a component of these models from batched HDDS data. Calculating risk in real-time the Department of Justice’s Office of Justice Programs, which also includes remains challenging given the additional steps necessary to incorpo- the Bureau of Justice Statistics, the National Institute of Justice, the Office of rate data entered close to the time of prediction. Juvenile Justice and Delinquency Prevention, the Office for Victims of Crime, and the SMART Office. Points of view or opinions in this document are those Future work of the author and do not necessarily represent the official position or policies Implementation of these models into internal state systems is cur- of the U.S. Department of Justice. rently being reviewed. The choice to do so may provide a platform The above funders had no role in any of the following: design and con- for prospective validation opportunities and public health perspec- duct of the study; collection, management, analysis, and interpretation of the tives unprecedented in TN. While a small number of proprietary data; preparation, review, or approval of the manuscript; or decision to sub- risk scores exist in this domain, none are being used at the state level mit the manuscript for publication (authors BT, AR, CC, and SL). in TN. Implementing these models would complement traditional RFS #34301-29519—Predicting opioid overdose in TN using controlled substance monitoring data and vital statistics (authors CGW, MR, DW, KR, epidemiologic methods that identify risk and guide planning for pre- QC, and CEF). vention. Future work includes a need to study the interpretability of Funding for the Research Derivative and BioVU Synthetic Derivative is these models and the need to assess for drift and apply recalibration through UL1 RR024975/RR/NCRR, PI: Gordon Bernard. prospectively. Outcome rates and prescription rates have changed since 2017. More advanced feature engineering and additional ex- ternal data sources might improve these models further. AUTHOR CONTRIBUTIONS MLM, BT, and CGW outlined the study. QC investigated study validity. KR CONCLUSION coordinated the work as performed. SCL, CEF, and AR compiled and ana- Historical statewide PDMP data, hospital discharge data, and death lyzed prior research. MR, SCL, DW, and ML processed all reference data. certificates from vital records were linked to socioeconomic indica- DW delineated study outcomes. MR transformed study data, trained the Journal of the American Medical Informatics Association, 2022, Vol. 29, No. 1 31 10. Martins SS, Ponicki W, Smith N, et al. Prescription drug monitoring pro- weak learners, and performed the subgroup analysis. CGW performed the grams operational characteristics and fatal heroin poisoning. Int J Drug weak learner ensembling. DW analyzed the modeling results and repeated the Policy 2019; 74: 174–80. analysis with varying partitions. MR, DW, and CEF produced figures. All 11. Geissert P, Hallvik S, Van Otterloo J, et al. High risk prescribing and opi- authors contributed to writing and revisions. MR and CGW take responsibil- oid overdose: prospects for prescription drug monitoring program based ity for the integrity and accuracy of the study. proactive alerts. Pain 2018; 159 (1): 150–6. 12. Chang H-Y, Krawczyk N, Schneider KE, et al. A predictive risk model for nonfatal opioid overdose in a statewide population of buprenorphine SUPPLEMENTARY MATERIAL patients. Drug Alcohol Depend 2019; 201: 127–33. Supplementary material is available at Journal of the American Medical Infor- 13. Hastings JS, Howison M, Inman SE. Predicting high-risk opioid prescrip- matics Association online. tions before they are given. Proc Natl Acad Sci U S A 2020; 117 (4): 1917–23. 14. Saloner B, Chang H-Y, Krawczyk N, et al. Predictive modeling of opioid overdose using linked statewide medical and criminal justice data. JAMA ACKNOWLEDGMENTS Psychiatry 2020; 77 (11): 1155. Thank you to the leadership of VUMC and of TDH for enabling this work 15. Nechuta SJ, Tyndall BD, Mukhopadhyay S, et al. Sociodemographic fac- and for their continued efforts toward combatting the opioid crisis in Tennes- tors, prescription history and opioid overdose deaths: a statewide analysis see. using linked PDMP and mortality data. Drug Alcohol Depend 2018; 190: 62–71. 16. Krishnaswami S, Mukhopadhyay S, McPheeters M, et al. Prescribing pat- terns before and after a non-fatal drug overdose using Tennessee’s Con- CONFLICT OF INTEREST STATEMENT trolled Substance Monitoring Database linked to hospital discharge data. None declared. Prev Med 2020; 130: 105883. 17. Kelty E, Hulse G. Fatal and non-fatal opioid overdose in opioid dependent patients treated with methadone, buprenorphine or implant naltrexone. Int J Drug Policy 2017; 46: 54–60. DATA AVAILABILITY 18. Park TW, Lin LA, Hosanagar A, et al. Understanding risk factors for opi- Study data including CSMD, HDDS, and vital statistics were provided by oid overdose in clinical populations to inform treatment and policy. J Ad- TDH under contract and a memorandum of understanding. These data are dict Med 2016; 10 (6): 369–81. not available for sharing by study authors but interested parties are welcome 19. Lo-Ciganic W-H, Huang JL, Zhang HH, et al. Evaluation of machine- to request these data from TDH. Reference data derived from online reposito- learning algorithms for predicting opioid overdose risk among medicare ries open to the public are shareable upon request. beneficiaries with opioid prescriptions. JAMA Netw Open 2019; 2 (3): e190968. 20. Mitra G, Wood E, Nguyen P, et al. Drug use patterns predict risk of non- REFERENCES fatal overdose among street-involved youth in a Canadian setting. Drug Alcohol Depend 2015; 153: 135–9. 1. Controlled Substance Monitoring Database: 2020 Report to the 111th 21. Phalen P, Ray B, Watson DP, et al. Fentanyl related overdose in Indianap- Tennessee General Assembly. Tennessee Department of Health; 2020. olis: estimating trends using multilevel Bayesian models. Addict Behav https://www.tn.gov/content/dam/tn/health/healthprofboards/csmd/2020_ 2018; 86: 4–10. CSMD_Annual_Report.pdf Accessed March 29, 2021. 22. Ellis RJ, Wang Z, Genes N, et al. Predicting opioid dependence from elec- 2. Chakravarthy B, Shah S, Lotfipour S. Prescription drug monitoring pro- tronic health records with machine learning. BioData Min 2019; 12: 3. grams and other interventions to combat prescription opioid abuse. West J 23. Glanz JM, Narwaney KJ, Mueller SR, et al. Prediction model for two-year Emerg Med 2012; 13 (5): 422–5. risk of opioid overdose among patients prescribed chronic opioid therapy. 3. Rolheiser LA, Cordes J, Subramanian SV. Opioid prescribing rates by con- J Gen Intern Med 2018; 33 (10): 1646–53. gressional districts, United States, 2016. Am J Public Health 2018; 108 24. Zedler B, Xie L, Wang L, et al. Development of a risk index for serious (9): 1214–9. prescription opioid-induced respiratory depression or overdose in Veter- 4. National Center for Injury Prevention and Control. U.S. Opioid Dispens- ans’ Health Administration Patients. Pain Med 2015; 16 (8): 1566–79. ing Rate Maps. Drug Overdose CDC Injury Center; 2020. https://www. 25. Zedler BK, Saunders WB, Joyce AR, et al. Validation of a screening risk in- cdc.gov/drugoverdose/maps/rxrate-maps.html Accessed March 29, 2021. dex for serious prescription opioid-induced respiratory depression or over- 5. Tennessee’s Annual Overdose Report 2021: Report on Epidemiologic dose in a US Commercial Health Plan Claims Database. Pain Med 2018; Data, Efforts, and Collaborations to Address the Overdose Epidemic. 19 (1): 68–78. Tennessee Department of Health; 2021. https://www.tn.gov/content/dam/ 26. Holt CT, McCall KL, Cattabriga G, et al. Using controlled substance re- tn/health/documents/pdo/2021%20TN%20Annual%20Overdose% ceipt patterns to predict prescription overdose death. Pharmacology 2018; 20Report.pdf Accessed March 29, 2021. 101 (3–4): 140–7. 6. National Institute on Drug Abuse. Tennessee: Opioid-Involved Deaths 27. Ferris LM, Saloner B, Krawczyk N, et al. Predicting opioid overdose and Related Harms. 2020. https://www.drugabuse.gov/drug-topics/ deaths using prescription drug monitoring program data. Am J Prev Med opioids/opioid-summaries-by-state/tennessee-opioid-involved-deaths-re- 2019; 57 (6): e211–7. lated-harms Accessed March 29, 2021. 28. Eisenberg MD, Saloner B, Krawczyk N, et al. Use of opioid overdose deaths 7. Patrick SW, Fry CE, Jones TF, et al. Implementation of prescription drug reported in one state’s criminal justice, hospital, and prescription databases monitoring programs associated with reductions in opioid-related death to identify risk of opioid fatalities. JAMA Intern Med 2019; 179 (7): 980–2. rates. Health Aff (Millwood) 2016; 35 (7): 1324–32. 29. Johnson C. Nearly 60 Doctors, Other Medical Workers Charged In Fede- 8. Lin DH, Lucas E, Murimi IB, et al. Physician attitudes and experiences ral Opioid Sting. NPR.org. 2019. https://www.npr.org/2019/04/17/ with Maryland’s prescription drug monitoring program (PDMP). Addic- 714014919/nearly-60-docs-other-medical-workers-face-charges-in-fede- tion 2017; 112 (2): 311–9. ral-opioid-sting Accessed March 29, 2021. 9. Strickler GK, Zhang K, Halpin JF, et al. Effects of mandatory pre- 30. Cerd a M, Ransome Y, Keyes KM, et al. Revisiting the role of the urban en- scription drug monitoring program (PDMP) use laws on prescriber vironment in substance use: the case of analgesic overdose fatalities. Am J registration and use and on risky prescribing. Drug Alcohol Depend Public Health 2013; 103 (12): 2252–60. 2019; 199: 1–9. 32 Journal of the American Medical Informatics Association, 2022, Vol. 29, No. 1 31. Kind AJ, Jencks S, Brock J, et al. Neighborhood socioeconomic disadvan- 39. Nechuta S, Mukhopadhyay S, Krishnaswami S, et al. Record linkage tage and 30 day rehospitalizations: an analysis of Medicare data. Ann In- approaches using prescription drug monitoring program and mortality tern Med 2014; 161 (11): 765–74. data for public health analyses and epidemiologic studies. Epidemiology 32. United States Census Bureau. Economic Census Tables: 2012–2017. Eco- 2020; 31 (1): 22–31. nomic Census Tables. https://www.census.gov/programs-surveys/eco- 40. Brown S, Elkin P, Rosenbloom S, et al. VA National Drug File Reference nomic-census/data/tables.html Accessed March 30, 2021. Terminology: a cross-institutional content coverage study. Stud Health 33. Rickles M, Rebeiro PF, Sizemore L, et al. Tennessee’s in-state vulnerability Technol Inform 2004; 107 (Pt 1): 477–81. assessment for a “rapid dissemination of human immunodeficiency virus 41. Clinical Classifications Software (CCS) 2015. Agency for Healthcare Re- or hepatitis C virus infection” event utilizing data about the opioid epi- search and Quality; 2016. https://www.hcup-us.ahrq.gov/toolssoftware/ demic. Clin Infect Dis 2018; 66 (11): 1722–32. ccs/CCSUsersGuide.pdf Accessed March 30, 2021. 34. United States Census Bureau. 2012–2016 ACS 5-Year Data Profile. 2016 42. American Medical Association. ICD-10-CM 2019 the Complete Offi- Data Profiles American Community Survey. US Census Bureau. https:// cial Codebook. Geneva, Switzerland: American Medical Association; www.census.gov/acs/www/data/data-tables-and-tools/data-profiles/2016/ 2018. Accessed March 30, 2021. 43. Wright MN, Ziegler A. ranger: a fast implementation of random for- 35. United States Department of Agriculture: Economic Research Service. ests for high dimensional data in Cþþ and R. J Stat Softw 2017; 77: Rural-Urban Continuum Codes. USDA ERS—Rural-Urban Continuum 1–17. Codes; 2020. https://www.ers.usda.gov/data-products/rural-urban-con- 44. Walsh CG, Sharman K, Hripcsak G. Beyond discrimination: a comparison tinuum-codes.aspx Accessed March 30, 2021. of calibration methods and clinical usefulness of predictive models of 36. Flanagan BE, Gregory EW, Hallisey EJ, et al. A social vulnerability index for readmission risk. J Biomed Inform 2017; 76: 9–18. disaster management. JHomelSecurEmerg Manag 2011; 8: Article 3, 1–24. 45. Steyerberg EW, Borsboom GJJM, Houwelingen H. V, et al. Validation 37. McPheeters M, Nechuta S, Miller S, et al. Prescription Drug Overdose Re- and updating of predictive logistic regression models: a study on sample port, 2018. TN Department of Health; 2018. https://www.tn.gov/content/ size and shrinkage. Statist Med 2004; 23 (16): 2567–86. dam/tn/health/documents/pdo/PDO_2018_Report_02.06.18.pdf 46. Tibshirani R. Regression shrinkage and selection via the lasso. J R Stat Soc Accessed March 30, 2021. Ser B Methodol 1996; 58 (1): 267–88. 38. World Health Organization. The International Statistical Classification of 47. Park TW, Saitz R, Ganoczy D, et al. Benzodiazepine prescribing patterns Diseases and Health Related Problems ICD-10: Tenth Revision. Volume 1: and deaths from drug overdose among US veterans receiving opioid anal- Tabular List. Chicago, IL: World Health Organization; 2004. gesics: case-cohort study. BMJ 2015; 350: h2698.

Journal

Journal of the American Medical Informatics Association – Oxford University Press

Published: Oct 19, 2021

Keywords: drug overdose; opioid epidemic; machine learning; prescription drug monitoring programs; vital statistics

There are no references for this article.