Get 20M+ Full-Text Papers For Less Than $1.50/day. Start a 14-Day Trial for You or Your Team.

Learn More →

Identifying surgical site infections in electronic health data using predictive models

Identifying surgical site infections in electronic health data using predictive models Abstract Objective The objective was to prospectively derive and validate a prediction rule for detecting cases warranting investigation for surgical site infections (SSI) after ambulatory surgery. Methods We analysed electronic health record (EHR) data for children who underwent ambulatory surgery at one of 4 ambulatory surgical facilities. Using regularized logistic regression and random forests, we derived SSI prediction rules using 30 months of data (derivation set) and evaluated performance with data from the subsequent 10 months (validation set). Models were developed both with and without data extracted from free text. We also evaluated the presence of an antibiotic prescription within 60 days after surgery as an independent indicator of SSI evidence. Our goal was to exceed 80% sensitivity and 10% positive predictive value (PPV). Results We identified 234 surgeries with evidence of SSI among the 7910 surgeries available for analysis. We derived and validated an optimal prediction rule that included free text data using a random forest model (sensitivity = 0.9, PPV = 0.28). Presence of an antibiotic prescription had poor sensitivity (0.65) when applied to the derivation data but performed better when applied to the validation data (sensitivity = 0.84, PPV = 0.28). Conclusions EHR data can facilitate SSI surveillance with adequate sensitivity and PPV. predictive modelling, surgical site infection, quality improvement Introduction Ambulatory surgical procedures now comprise approximately three quarters of all surgeries performed in the United States. Surgical site infections (SSI) are uncommon but can lead to significant increases in cost, morbidity, and mortality.1,2 Public reporting of SSI after specific inpatient surgical procedures is mandated by the Centers for Medicaid and Medicare Services (CMS) and some state Departments of Health.3,4 To improve patient care and to comply with public reporting mandates, healthcare systems rely on specially trained infection preventionists (IP) to perform SSI surveillance. IPs use a variety of strategies to detect patients who have developed a post-operative infection, but traditional detection strategies may be less effective to detect SSI that arise when a patient is no longer hospitalized. A review of surveillance methods in 2006 concluded that, at the time, there were no valid methods for SSI surveillance.5 SSI surveillance is challenging in part due to the complexity of the clinical definition.6 Administrative data such as diagnosis codes have been shown to have acceptable positive predictive value in selected domains,7–9 but the sensitivity of administrative data for SSI surveillance purposes is unknown. Important information relevant for SSI surveillance such as medical history and physical examination findings typically resides only in free text, and the addition of free text information to case identification algorithms has been shown to improve sensitivity in multiple domains.10 Better surveillance data regarding the incidence and epidemiology of SSI after ambulatory surgery will help improve the quality of care delivered to patients undergoing ambulatory surgery. Given the volume of surgeries performed, the relatively low rate of SSI, and the challenges described above, novel surveillance strategies are needed to identify SSI cases after ambulatory surgery. Although electronic health record (EHR) vendors increasingly provide tools to support surveillance efforts, these tools traditionally use structured data elements captured during routine care, which may impair the accuracy of these systems.11 To date, these EHR systems have not provided adequate features to extract information from free text across large panels of patients for surveillance activities. To fill this gap in the ability to perform SSI surveillance, we derived a prediction rule that uses structured as well as free text information from EHRs to identify potential SSI cases. Given the potential inconsistencies in clinician documentation as well as the complexity of the SSI definition and the role of the IP to adjudicate whether a patient meets this definition, our priority was to develop a prediction rule with high sensitivity to identify any potential SSI case, which would then prompt a thorough manual review by IPs. Our objective was to prospectively derive and validate a prediction rule that supports SSI surveillance efforts by IPs by using both codified information and free text. Our performance goal was for the prediction rule to exceed 80% sensitivity and 10% positive predictive value (PPV). Methods We performed a secondary analysis of a cohort of children who underwent ambulatory surgery. This cohort was assembled for a larger project that aimed to describe the epidemiology of SSI after ambulatory paediatric surgery and to develop to an “infection prevention workstation” that facilitated manual review efforts of IPs by applying a prediction rule to focus their surveillance on cases with a high likelihood of having an SSI. In this context we felt it was reasonable to expect IPs to manually review 10 surgical cases to identify one true case. Consequently, our performance goal was to maximize the sensitivity of the prediction rule while achieving a PPV of at least 10%. Study population and setting Our cohort was comprised of paediatric ambulatory surgery patients (< 18 years of age) who underwent surgery between 1 May 2012 and 31 Dec 2015 at one of 4 surgical centres (3 ambulatory surgical facilities and 1 hospital-based facility) affiliated with the Children’s Hospital of Philadelphia (CHOP) healthcare system. This health network also includes 31 primary care practices, 8 specialty care practices, a 521-bed acute care hospital, and an emergency department located in southeastern Pennsylvania and southern New Jersey with a common EHR. Ambulatory surgery was defined as admission, procedure, and discharge on the same calendar day. We applied 2010 NHSN criteria to define a surgical procedure, which required that the procedure was performed in an operating room and that incision and complete closure of the wound occurred during the same operating room visit.6 The primary EHR in use during the study period was Epic™ (Epic Systems, Inc., Verona WI). This EHR was used to document all aspects of healthcare delivery at any site within the healthcare network, including surgical, outpatient, emergency and inpatient care with the exception that surgical procedure data prior to 4 May 2013 was collected and stored in OR Manager (Picis Clinical Solutions, Inc., Wakefield, MA). Study design Our study was conducted in two phases. Using data from the first 30 months of the study, we derived a prediction rule to identify surgical cases with evidence of SSI during the 60-day follow-up period after eligible surgeries (derivation phase). Using data from the subsequent 10 months, we prospectively validated the performance of the prediction rule using information from eligible surgeries (validation phase). Both study phases relied on a reference standard for evidence of SSI established by two IPs through manual review (see below). This study was reviewed and approved by the Institutional Review Board at the Children’s Hospital of Philadelphia. Establishing the reference standard for evidence of SSI We prospectively established the reference standard set of cases with evidence of an SSI by telephone enrolment and parent interview 30 to 40 days after surgery, and at 60 days after surgery by review of EHR data. Manual case review was completed if any of the following conditions were present: (1) interview indicated a potential post-surgical infection, (2) data extracted from the EHR indicated that an antibiotic was prescribed, or (3) child had an emergency department visit or hospitalization. One of two certified IPs reviewed all available encounters in the EHR on post-operative days 1 to 60 related to healing of the surgical wound. Because our objective was to develop a prediction rule that supports SSI surveillance efforts by IPs, we aimed to identify all surgeries with any evidence or suggestion of SSI or atypical healing of the surgical wound as suggested directly by diagnoses or wound symptoms or as suggested indirectly by actions taken by the clinicians including prescription of antibiotics, follow-up surgical procedures, or culture orders. Table 1 includes the EHR criteria for an encounter with potential evidence of SSI. Other healthcare encounters during the 60-day post-operative period were marked as “negative for evidence of SSI.” Table 1. Evidence of possible infection related to surgery Criteria Infection diagnosis of surgical site, code or described in notes Infection diagnosis of unspecified site, code or described in notes Purulent fluid Two of: dehiscence, erythema, warmth, tenderness, oedema, and/or non-purulent fluid Culture of surgical site Culture of unspecified site Antibiotic prescription, excluding continuation of therapy, in the absence of an unrelated infectious diagnosis Drainage procedure Criteria Infection diagnosis of surgical site, code or described in notes Infection diagnosis of unspecified site, code or described in notes Purulent fluid Two of: dehiscence, erythema, warmth, tenderness, oedema, and/or non-purulent fluid Culture of surgical site Culture of unspecified site Antibiotic prescription, excluding continuation of therapy, in the absence of an unrelated infectious diagnosis Drainage procedure Table 1. Evidence of possible infection related to surgery Criteria Infection diagnosis of surgical site, code or described in notes Infection diagnosis of unspecified site, code or described in notes Purulent fluid Two of: dehiscence, erythema, warmth, tenderness, oedema, and/or non-purulent fluid Culture of surgical site Culture of unspecified site Antibiotic prescription, excluding continuation of therapy, in the absence of an unrelated infectious diagnosis Drainage procedure Criteria Infection diagnosis of surgical site, code or described in notes Infection diagnosis of unspecified site, code or described in notes Purulent fluid Two of: dehiscence, erythema, warmth, tenderness, oedema, and/or non-purulent fluid Culture of surgical site Culture of unspecified site Antibiotic prescription, excluding continuation of therapy, in the absence of an unrelated infectious diagnosis Drainage procedure Deriving a prediction model to identify evidence of SSI Our goal was to identify healthcare encounters in the 60-day period after ambulatory surgery that contained any evidence of possible SSI. Our optimization target (maximal sensitivity with PPV of at least 10%) was assessed at the level of the surgical case rather than the individual follow-up encounters. Models unable to achieve at least 80% sensitivity with a PPV of 10% or better were rejected from consideration. Model derivation proceeded in four steps: (1) transforming structured health data into analysable form, (2) handling diagnosis codes, (3) incorporating free text information, and (4) tuning model parameters. Details of these steps are provided in the following sections. All analyses were performed in R version 3.3.3 with the addition of the text mining (tm), glmnet, ranger, and pROC packages.12–16 Transformation of structured electronic health data We first focused on gathering structured data in the EHR to identify encounters containing evidence of a possible SSI. Candidate variables were determined after a review of the SSI guidelines and in consultation with IPs and other subject matter experts. The candidate variables included patient characteristics (age at surgery, gender, race, ethnicity and insurance type), characteristics of the surgery (duration of surgery in minutes, and start time of surgery), and time varying characteristics of the follow-up visit (visit type, post-operative day number, presence of a culture order, presence of a systemic antibiotic prescription, and patient temperature). ICD-9 and ICD-10 diagnosis codes Diagnosis codes represented a unique problem because the coding system changed from ICD-9 to ICD-10 during the study period. We chose to include the descriptive words associated with the diagnosis codes as input features rather than the codes themselves. This approach was motivated by the observation that certain key diagnostic descriptors (eg carbuncle, cellulitis, abscess, etc.) occurred both in the ICD-9 and ICD-10 descriptions. Consequently, we approached the text descriptors for coded diagnoses similar to a free text analysis problem. We took the stem for each word in the diagnosis descriptors, used binary weighting of the word stems (each was either present vs absent), and included these features in model development as a “bag of words.” For example, the ICD-10 diagnosis “Carbuncle, unspecified” (L02.93) was represented in the model as presence of the stems for the words “carbuncle” and “unspecified.” Narrative documentation To improve the scalability of our approach, we sought to identify the most parsimonious model that would achieve adequate accuracy. Consequently, we did not include narrative documentation in our initial derivation models. During iterative revisions we added pre-specified key words and phrases related to SSI that had been identified by searching narrative documentation. Regular expression matching patterns were used to count the occurrences of each key word or phrase within the narrative documentation on each follow-up day. Occasionally there were unexpectedly high term counts (eg due to the recurring presence of terms such as “pain” in some documentation templates). We therefore decreased the weight of high counts by taking the square root prior to including the count of each key word or phrase in the model. Model tuning We considered 2 machine learning classifier models that are appropriate for high dimensional data. The first model was logistic regression with lasso regularization (R package “glmnet”), in which a regularization term is imposed on a logistic regression model to prevent over-fitting and to aid in feature selection.13,17,18 The other model was random forests (R package “ranger”), which grows a forest of classification trees for a binary outcome using a training sample and can provide a probability estimate of membership in each class.14,19,20 For interested readers, several tutorials related to the use of these machine learning algorithms are available online.21,22 Using the ambulatory surgeries in the derivation set, optimal model configuration parameters (eg regularization constant) were selected by a grid search using each model’s cross-validation functions in R where possible (eg “cv.glmnet” for the regularized logistic regression). Due to class imbalance arising from the very small number of visits with evidence of SSI, we included class weights in the grid search for model parameters to determine if assigning lower weights to the visits without SSI evidence would improve model performance. To evaluate the utility of adding narrative documentation to the models, we constructed both the regularized logistic regression and random forest models with and without the variables derived from the key words and phrases. A detailed description of our approach to model construction and tuning is included along with annotated R code excerpts in the Supplementary Appendix. Validating the SSI prediction rule After selecting optimal features and model parameters for the two models, we performed predictions on a validation set of ambulatory surgical cases that occurred during the final 10 months of the study period. The structured and narrative EHR data during the follow-up period for these surgical cases was prepared using the same steps described previously with the exception that the word stem features from diagnosis descriptors and the key words/phrases from narrative documentation were constructed using the dictionary of terms constructed from the training documents. Consequently, novel terms in the validation set that were not present in the training set did not contribute to classification. The models were then used to predict whether each ambulatory surgery was associated with evidence of SSI on at least one follow-up day. Using the probability output from each model we plotted receiver operating characteristics (ROC) and calculated the area under the curve (AUC) to compare each model’s performance as a test to differentiate surgeries associated with evidence of SSI from those without evidence. Using the ROCs, we then determined the probability thresholds that achieved 80% and 90% sensitivity for each model. The PPV was then assessed for each model at these two levels of sensitivity. We used bootstrap sampling (10 000 iterations) to estimate 95% confidence intervals (CI) for each performance statistic. Results During the 40-month study period, children experienced 19 777 ambulatory surgeries eligible for SSI surveillance. Telephone interviews were completed and permission obtained to review the EHR for 8502 of these surgeries. Follow-up encounters within 60 days after surgery were available for 7910 surgeries (see study flow diagram, Figure 1). Figure 1. View largeDownload slide Study flow diagram. Figure 1. View largeDownload slide Study flow diagram. Development of prediction rule During the model derivation phase of the study, telephone interviews were completed and consent was obtained to perform chart review for 6871 eligible surgeries that had at least one healthcare encounter during the follow-up period. Each of these surgeries gave rise to follow-up encounters (telephone, office visit, ED or inpatient) on a median of 3 occasions (interquartile range 2-4) during the follow-up period. Multiple healthcare encounters on the same calendar day were considered only a single follow-up episode. Charts were manually reviewed for 1106 surgeries (16%) due to concerns for SSI based either on the telephone interview or the presence of an antibiotic prescription within 60 days of surgery. Among the manually reviewed surgeries there was evidence of SSI present in at least one follow-up encounter for 209 surgeries in the training period (19% of the reviewed cases, 3.0% of all cases). An antibiotic prescription was present during the follow-up period for 135 (65%) of these surgeries. Tuning model configuration parameters After excluding rare terms from diagnosis descriptions, keywords and phrases that occurred less than 5 times, we had a total of 289 candidate predictor variables consisting of 24 variables derived from the structured EHR data, 217 distinct word stems from diagnosis descriptors, and 48 distinct keywords or phrases (Table 1). The optimal lasso penalty (regularization parameter “lambda”) for our logistic regression model was 0.002 regardless of whether or not the 48 variables derived from keywords and phrases were included. For random forests with 5000 trees to ensure stabilized errors, the algorithm’s default value for “mtry” (the number of variables randomly selected at each node of the tree) was optimal (ie the square root of the number of candidate predictor variables). Measures of importance (word frequency, regression coefficients, and permutation importance) for the most frequent terms from the logistic regression and random forest models are also shown in Table 2; some variables had no association with SSI in logistic regression (eg the terms “pain,” and “warm”) yet had relatively high importance in the random forest model. These results are presumably due to interactions with one or more additional terms included in the models (eg the effect of these variables was likely modified by other variables). Table 2. Top ten most frequent keywords in the 6871 training cases. The percent of surgeries associated with at least one occurrence of each term in the follow-up period is reported separately for surgical cases with evidence of SSI and cases without for comparison. Two different measures of importance are also shown: the coefficient from regularized logistic regression model, and the permutation importance from the random forest model PERCENT OF SURGERIES MODEL IMPORTANCE Term SSI Evidence Present (N = 209) No SSI Evidence (N = 6662) Logistic Regression Coefficient Random Forest Permutation Importance Pain 174 (83.3%) 3686 (55.3%) 0 0.000566 Swelling 138 (66.0%) 1915 (28.7%) +0.157 0.000351 Infection 153 (73.2%) 1845 (27.7%) +0.357 0.000200 Warm 91 (43.5%) 1618 (24.3%) 0 0.000169 Drainage 140 (67.0%) 1497 (22.5%) +0.501 0.000559 Redness 135 (64.6%) 1344 (20.2%) +0.515 0.000260 Red 100 (47.8%) 1117 (16.8%) +0.078 0.000166 Erythema 95 (45.5%) 1077 (16.2%) +0.033 0.000276 Oedema 74 (35.4%) 943 (14.2%) 0 0.000012 Tenderness 54 (25.8%) 706 (10.6%) +0.042 0.000031 PERCENT OF SURGERIES MODEL IMPORTANCE Term SSI Evidence Present (N = 209) No SSI Evidence (N = 6662) Logistic Regression Coefficient Random Forest Permutation Importance Pain 174 (83.3%) 3686 (55.3%) 0 0.000566 Swelling 138 (66.0%) 1915 (28.7%) +0.157 0.000351 Infection 153 (73.2%) 1845 (27.7%) +0.357 0.000200 Warm 91 (43.5%) 1618 (24.3%) 0 0.000169 Drainage 140 (67.0%) 1497 (22.5%) +0.501 0.000559 Redness 135 (64.6%) 1344 (20.2%) +0.515 0.000260 Red 100 (47.8%) 1117 (16.8%) +0.078 0.000166 Erythema 95 (45.5%) 1077 (16.2%) +0.033 0.000276 Oedema 74 (35.4%) 943 (14.2%) 0 0.000012 Tenderness 54 (25.8%) 706 (10.6%) +0.042 0.000031 Table 2. Top ten most frequent keywords in the 6871 training cases. The percent of surgeries associated with at least one occurrence of each term in the follow-up period is reported separately for surgical cases with evidence of SSI and cases without for comparison. Two different measures of importance are also shown: the coefficient from regularized logistic regression model, and the permutation importance from the random forest model PERCENT OF SURGERIES MODEL IMPORTANCE Term SSI Evidence Present (N = 209) No SSI Evidence (N = 6662) Logistic Regression Coefficient Random Forest Permutation Importance Pain 174 (83.3%) 3686 (55.3%) 0 0.000566 Swelling 138 (66.0%) 1915 (28.7%) +0.157 0.000351 Infection 153 (73.2%) 1845 (27.7%) +0.357 0.000200 Warm 91 (43.5%) 1618 (24.3%) 0 0.000169 Drainage 140 (67.0%) 1497 (22.5%) +0.501 0.000559 Redness 135 (64.6%) 1344 (20.2%) +0.515 0.000260 Red 100 (47.8%) 1117 (16.8%) +0.078 0.000166 Erythema 95 (45.5%) 1077 (16.2%) +0.033 0.000276 Oedema 74 (35.4%) 943 (14.2%) 0 0.000012 Tenderness 54 (25.8%) 706 (10.6%) +0.042 0.000031 PERCENT OF SURGERIES MODEL IMPORTANCE Term SSI Evidence Present (N = 209) No SSI Evidence (N = 6662) Logistic Regression Coefficient Random Forest Permutation Importance Pain 174 (83.3%) 3686 (55.3%) 0 0.000566 Swelling 138 (66.0%) 1915 (28.7%) +0.157 0.000351 Infection 153 (73.2%) 1845 (27.7%) +0.357 0.000200 Warm 91 (43.5%) 1618 (24.3%) 0 0.000169 Drainage 140 (67.0%) 1497 (22.5%) +0.501 0.000559 Redness 135 (64.6%) 1344 (20.2%) +0.515 0.000260 Red 100 (47.8%) 1117 (16.8%) +0.078 0.000166 Erythema 95 (45.5%) 1077 (16.2%) +0.033 0.000276 Oedema 74 (35.4%) 943 (14.2%) 0 0.000012 Tenderness 54 (25.8%) 706 (10.6%) +0.042 0.000031 Validation set performance During the validation period, charts were manually reviewed for 130 surgeries with evidence of SSI from a total of 1039 total surgeries (13%). Evidence of possible SSI was present in at least one follow-up encounter for 25 of the 130 manually reviewed surgeries in the validation period (2.4% of all cases). Measures of prediction performance on the validation set (AUC and PPV at two target levels of sensitivity) for logistic regression and random forest models are shown in Table 3. To demonstrate the trade-off between sensitivity and specificity, the ROC for the 4 models measured on the validation set are shown in Figure 2. All models were able to achieve the target sensitivity levels, but the random forest model that included keywords from free text was able to achieve an estimated sensitivity of 0.9 with the highest PPV (0.28, [95% CI 0.19, 0.38]). On the validation set, presence of an antibiotic prescription also performed well (sensitivity 0.84 [95% CI 0.64, 0.96], precision 0.28 [95% CI 0.18, 0.39]), which exceeded the performance observed in the derivation set (sensitivity 0.65 [95% CI 0.58, 0.71], precision 0.30 [95% CI 0.26, 0.35]). Table 3. Performance statistics of the four prediction models assessed on the validation set. Area under the receiver-operating characteristic is reported along with positive predictive values observed at pre-specified target levels of sensitivity. Bootstrap sampling was performed to estimate 95% confidence intervals (CI). The lower confidence limits of attainable sensitivity are noted below in footnotes Positive Predictive Value (PPV) at Target Sensitivity Area Under Curve (AUC) [95% CI] PPV at 0.8 Sensitivitya [95% CI] PPV at 0.9 Sensitivityb [95% CI] With keywords  Random forest 0.97 [0.92, 0.98] 0.31 [0.21, 0.44] 0.28 [0.19, 0.38]  Logistic regression 0.97 [0.94, 0.98] 0.40 [0.27, 0.54] 0.17 [0.12, 0.25] No keywords  Random forest 0.94 [0.84, 0.97] 0.19 [0.12, 0.27] 0.12 [0.08, 0.17]  Logistic regression 0.94 [0.86, 0.97] 0.32 [0.21, 0.44] 0.11 [0.07, 0.16] Positive Predictive Value (PPV) at Target Sensitivity Area Under Curve (AUC) [95% CI] PPV at 0.8 Sensitivitya [95% CI] PPV at 0.9 Sensitivityb [95% CI] With keywords  Random forest 0.97 [0.92, 0.98] 0.31 [0.21, 0.44] 0.28 [0.19, 0.38]  Logistic regression 0.97 [0.94, 0.98] 0.40 [0.27, 0.54] 0.17 [0.12, 0.25] No keywords  Random forest 0.94 [0.84, 0.97] 0.19 [0.12, 0.27] 0.12 [0.08, 0.17]  Logistic regression 0.94 [0.86, 0.97] 0.32 [0.21, 0.44] 0.11 [0.07, 0.16] a Lower limit of sensitivity was 0.58 for all models b Lower limit of sensitivity was 0.73 for all models Table 3. Performance statistics of the four prediction models assessed on the validation set. Area under the receiver-operating characteristic is reported along with positive predictive values observed at pre-specified target levels of sensitivity. Bootstrap sampling was performed to estimate 95% confidence intervals (CI). The lower confidence limits of attainable sensitivity are noted below in footnotes Positive Predictive Value (PPV) at Target Sensitivity Area Under Curve (AUC) [95% CI] PPV at 0.8 Sensitivitya [95% CI] PPV at 0.9 Sensitivityb [95% CI] With keywords  Random forest 0.97 [0.92, 0.98] 0.31 [0.21, 0.44] 0.28 [0.19, 0.38]  Logistic regression 0.97 [0.94, 0.98] 0.40 [0.27, 0.54] 0.17 [0.12, 0.25] No keywords  Random forest 0.94 [0.84, 0.97] 0.19 [0.12, 0.27] 0.12 [0.08, 0.17]  Logistic regression 0.94 [0.86, 0.97] 0.32 [0.21, 0.44] 0.11 [0.07, 0.16] Positive Predictive Value (PPV) at Target Sensitivity Area Under Curve (AUC) [95% CI] PPV at 0.8 Sensitivitya [95% CI] PPV at 0.9 Sensitivityb [95% CI] With keywords  Random forest 0.97 [0.92, 0.98] 0.31 [0.21, 0.44] 0.28 [0.19, 0.38]  Logistic regression 0.97 [0.94, 0.98] 0.40 [0.27, 0.54] 0.17 [0.12, 0.25] No keywords  Random forest 0.94 [0.84, 0.97] 0.19 [0.12, 0.27] 0.12 [0.08, 0.17]  Logistic regression 0.94 [0.86, 0.97] 0.32 [0.21, 0.44] 0.11 [0.07, 0.16] a Lower limit of sensitivity was 0.58 for all models b Lower limit of sensitivity was 0.73 for all models Figure 2. View largeDownload slide Receiver operating characteristics for the four prediction models on the validation set, with smoothing added to improve legibility. Figure 2. View largeDownload slide Receiver operating characteristics for the four prediction models on the validation set, with smoothing added to improve legibility. Discussion In this manuscript we have described a method for screening EHR data for the rare, but clinically important outcome of SSI. We developed a random forest model using information from both structured and free text information that achieved an estimated sensitivity of 0.9 with a PPV of 0.28. Using this model as a pre-screening tool, IPs would identify the vast majority of SSI cases that met the CDC definition by reviewing charts for about 4 cases to identify one case that met the NHSN definition of SSI (estimated lower limit of sensitivity was 0.73). This is a marked improvement over the current situation where all surgical cases must be reviewed to identify the approximately 3% that have evidence of SSI (over 30 chart reviews to identify one case with evidence of SSI). Logistic regression with the lasso penalty also performed well both with and without information from free text. Notably, as shown in Table 3, at the sensitivity level of 0.8, the logistic regression models attained a higher PPV than the corresponding random forest models. We also found that the presence of an antibiotic prescription on post-operative days 1-60 might perform well as a simplified approach to identifying the majority of cases of possible SSI when applied to our validation cohort (sensitivity 0.84, PPV 0.28). This approach had poor sensitivity when applied to our derivation cohort (0.65), possibly due to less frequent use of electronic prescribing at the start of our study. Health systems interested in screening EHR data for SSI should consider this simple approach, but derivation of a more complex prediction rule may be necessary if prescription data are not consistently available. Although the increasing availability of electronic health data offers tremendous opportunities for IPs to more readily identify rare outcomes such as SSI, few tools have been developed to support their work. Central line infections have long been a target for close scrutiny as a marker of the quality and safety of inpatient care, thus numerous tools have been developed and evaluated to support the mandated reporting of these outcomes.23–25 Less effort has been put into developing similar tools for SSI, likely because few jurisdictions require reporting of SSI after ambulatory surgery. In other domains, the EHR has been used as a tool to screen for rare events or rare cohorts. This has included the use of keyword searches, rule-based algorithms, and machine learning, which have yielded results that are similar to our results in the domain of SSI surveillance.10,26,27 Other studies have evaluated natural language processing for developing patient cohorts with some limited success, consistent with our attempts with investigating this approach.28 Studies have also evaluated EHR records for post-surgical complications using natural language processing with reasonable success rates.29,30 Generally, studies of rare diseases and clinical events have found that billing diagnoses (ICD-9/10) are often insufficient, and that processing of text information increases the yield for these algorithms.31,32 Future research should continue to look creatively at the potential to use both structured and narrative EHR information to identify rare, but clinically important events. Limitations Although there were 4 ambulatory surgical facilities participating in our study, these facilities were all part of a single healthcare system, which may limit the ability to directly apply the random forest model we developed in new settings. However, it is likely that institutions seeking to adopt a similar approach could improve the performance of the model by adding examples of known SSI cases from their own health system in a model re-training process. Our approach assumes the availability of certain types of information in electronic form such as antibiotic prescriptions and progress notes information. Without this information, health systems will likely have difficulty screening charts for evidence of SSI using our approach. Fortunately, thanks in part to EHR incentive programs, at this time most large health systems are using the EHR to capture these types of information.33 Our study used text representations of ICD-9 and ICD-10 diagnoses due to the transition in coding that happened during our study. Prior research has revealed that mapping tables between ICD-9 and ICD-10 codes are inexact, and particularly suffer from differences in the granularity of terms.34 For SSI surveillance activities, accurate mappings of diagnostic codes are required both for codes related to surgical complications, and for codes describing the common conditions where antibiotics are indicated. These mapping efforts can be substantial. For example, in our own work with antimicrobial stewardship activities focused on respiratory tract infections we identified 98 ICD-9 codes representing bacterial respiratory tract infections,35 which required a manual review of 1285 mapped codes to successfully continue these stewardship activities after the transition to ICD-10. However, in the future, organizations may benefit from simpler approaches that rely exclusively on ICD-10 codes. Conclusion An automated prediction rule derived from both structured and narrative EHR data using the random forest machine learning algorithm can be used to screen for surgical site infections after ambulatory surgery with high recall (sensitivity) and acceptable precision (positive predictive value). Presence of an antibiotic prescription within 60 days after surgery may also perform well, but may have inadequate sensitivity in settings that have incomplete prescription data. Funding This project was wholly supported by grant R01HS020921 (Electronic Surveillance for Wound Infections after Ambulatory Pediatric Surgery) from the Agency for Healthcare Research and Quality. The content is solely the responsibility of the authors and does not necessarily represent the official views of the Agency for Healthcare Research and Quality. Competing interests None. Contributors Robert Grundmeier contributed to the conception and design of the study, acquisition of data, analysis and interpretation of the data, drafted the manuscript, and approved the final manuscript as submitted. Rui Xiao contributed to the conception and design of the study, analysis and interpretation of the data, critically reviewed the manuscript, and approved the final manuscript as submitted. Rachael K. Ross contributed to the conception and design of the study, critically reviewed the manuscript, and approved the final manuscript as submitted. Mark J. Ramos contributed to the acquisition of data, critically reviewed the manuscript, and approved the final manuscript as submitted. Dean Karavite, Jeremy Michel, Jeffrey Gerber, and Susan Coffin contributed to the conception and design of the study, critically reviewed the manuscript, and approved the final manuscript as submitted. SUPPLEMENTARY MATERIAL Supplementary material is available at Journal of the American Medical Informatics Association online. ACKNOWLEDGEMENTS We thank our infection preventionists, Susan L. Rettig and Eva E. Teszner, for their tireless efforts manually reviewing charts for this project. References 1 Cullen KA , Hall MJ , Golosinskiy A. Ambulatory surgery in the United States, 2006 . Natl Health Stat Report 2009 : 1 – 25 . https://www.cdc.gov/nchs/data/nhsr/nhsr011.pdf Accessed March 2, 2018. 2 Zimlichman E , Henderson D , Tamir O , et al. . Health care–associated infections: a meta-analysis of costs and financial impact on the US health care system . JAMA Intern Med 2013 ; 173 22 : 2039 – 46 . Google Scholar CrossRef Search ADS PubMed 3 West N , Eng T. Monitoring and reporting hospital-acquired conditions: a federalist approach . Medicare Medicaid Res Rev 2014 ; 4 4 : E1 – 16 . Google Scholar CrossRef Search ADS 4 Radey LA , West N , Eng T , et al. . State Government Tracking of Hospital- Acquired Conditions Final Prepared for Published Online First; 2010 . https://www.cms.gov/Research-Statistics-Data-and-Systems/Statistics-Trends-and-Reports/Reports/Downloads/West_HAC_2010.pdf Accessed March 2, 2018. 5 Petherick ES , Dalton JE , Moore PJ , et al. . Methods for identifying surgical wound infection after discharge from hospital: a systematic review . BMC Infect Dis 2006 ; 6 1 : 170 . Google Scholar CrossRef Search ADS PubMed 6 Horan TC , Andrus M , Dudeck MA. CDC/NHSN surveillance definition of health care-associated infection and criteria for specific types of infections in the acute care setting . Am J Infect Control 2008 ; 36 5 : 309 – 32 . Google Scholar CrossRef Search ADS PubMed 7 Warren DK , Nickel KB , Wallace AE , et al. . Can additional information be obtained from claims data to support surgical site infection diagnosis codes? Infect Control Hosp Epidemiol 2014 ; 35 ( S3 ): S124 – 32 . Google Scholar CrossRef Search ADS PubMed 8 Olsen MA , Ball KE , Nickel KB , et al. . Validation of ICD-9-CM diagnosis codes for surgical site infection and noninfectious wound complications after mastectomy . Infect Control Hosp Epidemiol 2017 ; 38 03 : 334 – 9 . Google Scholar CrossRef Search ADS PubMed 9 Rhee C , Huang SS , Berríos-Torres SI , et al. . Surgical site infection surveillance following ambulatory surgery . Infect Control Hosp Epidemiol 2015 ; 36 02 : 225 – 8 . Google Scholar CrossRef Search ADS PubMed 10 Ford E , Carroll JA , Smith HE , et al. . Extracting information from the text of electronic medical records to improve case detection: a systematic review . J Am Med Inform Assoc 2016 ; 23 5 : 1007 – 15 . Google Scholar CrossRef Search ADS PubMed 11 Hoffman S , Podgurski A. Big bad data: law, public health, and biomedical databases . J Law Med Ethics 2013 ; 41 (suppl 1) : 56 – 60 . Google Scholar CrossRef Search ADS PubMed 12 Feinerer I. Introduction to the tm package: text mining in R . Compr R Arch Netw ; 2017 : 1 – 8 . https://cran.r-project.org/web/packages/tm/vignettes/tm.pdf Accessed March 2, 2018. 13 Friedman J , Hastie T , Tibshirani R. Regularization paths for generalized linear models via coordinate descent . J Stat Soft 2010 ; 33 1 . doi:10.18637/jss.v033.i01. 14 Wright MN , Ziegler A. ranger: a fast implementation of random forests for high dimensional data in C++ and R . J Stat Soft 2017 ; 77 1 . doi:10.18637/jss.v077.i01. 15 Robin X , Turck N , Hainard A , et al. . pROC: an open-source package for R and S+ to analyze and compare ROC curves . BMC Bioinform 2011 ; 12 1 : 77 . Google Scholar CrossRef Search ADS 16 R Core Team . R: a language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria; 2017 . http://www.r-project.org/ Accessed March 2, 2018. 17 Tibshirani R. Regression selection and shrinkage via the Lasso . J R Stat Soc B 1996 ; 58 : 267 – 88 . 18 Tibshirani R. Regression shrinkage and selection via the lasso: A retrospective . J R Stat Soc Ser B Stat Methodol 2011 ; 73 3 : 273 – 82 . Google Scholar CrossRef Search ADS 19 Malley JD , Kruppa J , Dasgupta A , et al. . Probability machines: consistent probability estimation using nonparametric learning machines . Methods Inf Med 2011 ; 51 1 : 74 – 81 . Google Scholar CrossRef Search ADS PubMed 20 Breiman L. Random forests . Mach Learn 2001 ; 45 1 : 5 – 32 . Google Scholar CrossRef Search ADS 21 Leek J , Peng RD , Caffo B. Practical Machine Learning. Johns Hopkins University, Coursera; 2018 . https://www.coursera.org/learn/practical-machine-learning Accessed April 16, 2018. 22 Fox E , Guestrin C. Machine Learning. University of Washington, Coursera; 2018 . https://www.coursera.org/specializations/machine-learning Accessed April 16, 2018. 23 Snyders RE , Goris AJ , Gase KA , et al. . Increasing the reliability of fully automated surveillance for central line–associated bloodstream infections . Infect Control Hosp Epidemiol 2015 ; 36 12 : 1396 – 400 . Google Scholar CrossRef Search ADS PubMed 24 Woeltje KF , McMullen KM , Butler AM , et al. . Electronic surveillance for healthcare-associated central line—associated bloodstream infections outside the intensive care unit . Infect Control Hosp Epidemiol 2011 ; 32 11 : 1086 – 90 . Google Scholar CrossRef Search ADS PubMed 25 Quan KA , Cousins SM , Porter DD , et al. . Electronic health record solutions to reduce central line-associated bloodstream infections by enhancing documentation of central line insertion practices, line days, and daily line necessity . Am J Infect Control 2016 ; 44 4 : 438 – 43 . Google Scholar CrossRef Search ADS PubMed 26 Garg R , Dong S , Shah S , et al. . A bootstrap machine learning approach to identify rare disease patients from electronic health records. arXiv Prepr. Published Online First; 2016 . http://arxiv.org/abs/1609.01586. Accessed March 2, 2018. 27 Liao S , Xiao J , Xie Y , et al. . Towards use of electronic health records: cancer classification. In: MSM ’17 Proceedings of the Symposium on Modeling and Simulation in Medicine . San Diego, CA, USA : Society for Computer Simulation International ; 2017 . https://dl.acm.org/citation.cfm?id=3108764. Accessed March 2, 2018. 28 Shivade C , Raghavan P , Fosler-Lussier E , et al. . A review of approaches to identifying patient phenotype cohorts using electronic health records . J Am Med Inform Assoc 2014 ; 21 2 : 221 – 30 . Google Scholar CrossRef Search ADS PubMed 29 Henry FF , Murff HJ , Matheny ME , et al. . Exploring the frontier of electronic health record surveillance the case of postoperative complications . Med Care 2013 ; 51 6 : 509 – 16 . Google Scholar CrossRef Search ADS PubMed 30 Murff HJ , FitzHenry F , Matheny ME , et al. . Automated identification of postoperative complications within an electronic medical record using natural language processing . JAMA 2011 ; 306 8 : 848 – 55 . Google Scholar CrossRef Search ADS PubMed 31 Carrell DS , Cronkite D , Palmer RE , et al. . Using natural language processing to identify problem usage of prescription opioids . Int J Med Inform 2015 ; 84 12 : 1057 – 64 . Google Scholar CrossRef Search ADS PubMed 32 Wang M , Cyhaniuk A , Cooper DL , et al. . Identification of people with acquired hemophilia in a large electronic health record database . J Blood Med 2017 ; 8 : 89 – 97 . Google Scholar CrossRef Search ADS PubMed 33 Blumenthal D , Tavenner M. The ‘meaningful use’ regulation for electronic health records . N Engl J Med 2010 ; 363 6 : 501 – 4 . Google Scholar CrossRef Search ADS PubMed 34 Caskey R , Zaman J , Nam H , et al. . The transition to ICD-10-CM: challenges for pediatric practice . Pediatrics 2014 ; 134 1 : 31 – 6 . Google Scholar CrossRef Search ADS PubMed 35 Goyal MK , Johnson TJ , Chamberlain JM , et al. . Racial and ethnic differences in antibiotic use for viral illness in emergency departments . Pediatrics 2017 ; 140 4 : e20170203 . Google Scholar CrossRef Search ADS PubMed © The Author(s) 2018. Published by Oxford University Press on behalf of the American Medical Informatics Association. All rights reserved. For permissions, please email: journals.permissions@oup.com This article is published and distributed under the terms of the Oxford University Press, Standard Journals Publication Model (https://academic.oup.com/journals/pages/about_us/legal/notices) http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png Journal of the American Medical Informatics Association Oxford University Press

Identifying surgical site infections in electronic health data using predictive models

Loading next page...
 
/lp/oxford-university-press/identifying-surgical-site-infections-in-electronic-health-data-using-FIDUDve6UY

References (30)

Publisher
Oxford University Press
Copyright
© The Author(s) 2018. Published by Oxford University Press on behalf of the American Medical Informatics Association. All rights reserved. For permissions, please email: journals.permissions@oup.com
ISSN
1067-5027
eISSN
1527-974X
DOI
10.1093/jamia/ocy075
Publisher site
See Article on Publisher Site

Abstract

Abstract Objective The objective was to prospectively derive and validate a prediction rule for detecting cases warranting investigation for surgical site infections (SSI) after ambulatory surgery. Methods We analysed electronic health record (EHR) data for children who underwent ambulatory surgery at one of 4 ambulatory surgical facilities. Using regularized logistic regression and random forests, we derived SSI prediction rules using 30 months of data (derivation set) and evaluated performance with data from the subsequent 10 months (validation set). Models were developed both with and without data extracted from free text. We also evaluated the presence of an antibiotic prescription within 60 days after surgery as an independent indicator of SSI evidence. Our goal was to exceed 80% sensitivity and 10% positive predictive value (PPV). Results We identified 234 surgeries with evidence of SSI among the 7910 surgeries available for analysis. We derived and validated an optimal prediction rule that included free text data using a random forest model (sensitivity = 0.9, PPV = 0.28). Presence of an antibiotic prescription had poor sensitivity (0.65) when applied to the derivation data but performed better when applied to the validation data (sensitivity = 0.84, PPV = 0.28). Conclusions EHR data can facilitate SSI surveillance with adequate sensitivity and PPV. predictive modelling, surgical site infection, quality improvement Introduction Ambulatory surgical procedures now comprise approximately three quarters of all surgeries performed in the United States. Surgical site infections (SSI) are uncommon but can lead to significant increases in cost, morbidity, and mortality.1,2 Public reporting of SSI after specific inpatient surgical procedures is mandated by the Centers for Medicaid and Medicare Services (CMS) and some state Departments of Health.3,4 To improve patient care and to comply with public reporting mandates, healthcare systems rely on specially trained infection preventionists (IP) to perform SSI surveillance. IPs use a variety of strategies to detect patients who have developed a post-operative infection, but traditional detection strategies may be less effective to detect SSI that arise when a patient is no longer hospitalized. A review of surveillance methods in 2006 concluded that, at the time, there were no valid methods for SSI surveillance.5 SSI surveillance is challenging in part due to the complexity of the clinical definition.6 Administrative data such as diagnosis codes have been shown to have acceptable positive predictive value in selected domains,7–9 but the sensitivity of administrative data for SSI surveillance purposes is unknown. Important information relevant for SSI surveillance such as medical history and physical examination findings typically resides only in free text, and the addition of free text information to case identification algorithms has been shown to improve sensitivity in multiple domains.10 Better surveillance data regarding the incidence and epidemiology of SSI after ambulatory surgery will help improve the quality of care delivered to patients undergoing ambulatory surgery. Given the volume of surgeries performed, the relatively low rate of SSI, and the challenges described above, novel surveillance strategies are needed to identify SSI cases after ambulatory surgery. Although electronic health record (EHR) vendors increasingly provide tools to support surveillance efforts, these tools traditionally use structured data elements captured during routine care, which may impair the accuracy of these systems.11 To date, these EHR systems have not provided adequate features to extract information from free text across large panels of patients for surveillance activities. To fill this gap in the ability to perform SSI surveillance, we derived a prediction rule that uses structured as well as free text information from EHRs to identify potential SSI cases. Given the potential inconsistencies in clinician documentation as well as the complexity of the SSI definition and the role of the IP to adjudicate whether a patient meets this definition, our priority was to develop a prediction rule with high sensitivity to identify any potential SSI case, which would then prompt a thorough manual review by IPs. Our objective was to prospectively derive and validate a prediction rule that supports SSI surveillance efforts by IPs by using both codified information and free text. Our performance goal was for the prediction rule to exceed 80% sensitivity and 10% positive predictive value (PPV). Methods We performed a secondary analysis of a cohort of children who underwent ambulatory surgery. This cohort was assembled for a larger project that aimed to describe the epidemiology of SSI after ambulatory paediatric surgery and to develop to an “infection prevention workstation” that facilitated manual review efforts of IPs by applying a prediction rule to focus their surveillance on cases with a high likelihood of having an SSI. In this context we felt it was reasonable to expect IPs to manually review 10 surgical cases to identify one true case. Consequently, our performance goal was to maximize the sensitivity of the prediction rule while achieving a PPV of at least 10%. Study population and setting Our cohort was comprised of paediatric ambulatory surgery patients (< 18 years of age) who underwent surgery between 1 May 2012 and 31 Dec 2015 at one of 4 surgical centres (3 ambulatory surgical facilities and 1 hospital-based facility) affiliated with the Children’s Hospital of Philadelphia (CHOP) healthcare system. This health network also includes 31 primary care practices, 8 specialty care practices, a 521-bed acute care hospital, and an emergency department located in southeastern Pennsylvania and southern New Jersey with a common EHR. Ambulatory surgery was defined as admission, procedure, and discharge on the same calendar day. We applied 2010 NHSN criteria to define a surgical procedure, which required that the procedure was performed in an operating room and that incision and complete closure of the wound occurred during the same operating room visit.6 The primary EHR in use during the study period was Epic™ (Epic Systems, Inc., Verona WI). This EHR was used to document all aspects of healthcare delivery at any site within the healthcare network, including surgical, outpatient, emergency and inpatient care with the exception that surgical procedure data prior to 4 May 2013 was collected and stored in OR Manager (Picis Clinical Solutions, Inc., Wakefield, MA). Study design Our study was conducted in two phases. Using data from the first 30 months of the study, we derived a prediction rule to identify surgical cases with evidence of SSI during the 60-day follow-up period after eligible surgeries (derivation phase). Using data from the subsequent 10 months, we prospectively validated the performance of the prediction rule using information from eligible surgeries (validation phase). Both study phases relied on a reference standard for evidence of SSI established by two IPs through manual review (see below). This study was reviewed and approved by the Institutional Review Board at the Children’s Hospital of Philadelphia. Establishing the reference standard for evidence of SSI We prospectively established the reference standard set of cases with evidence of an SSI by telephone enrolment and parent interview 30 to 40 days after surgery, and at 60 days after surgery by review of EHR data. Manual case review was completed if any of the following conditions were present: (1) interview indicated a potential post-surgical infection, (2) data extracted from the EHR indicated that an antibiotic was prescribed, or (3) child had an emergency department visit or hospitalization. One of two certified IPs reviewed all available encounters in the EHR on post-operative days 1 to 60 related to healing of the surgical wound. Because our objective was to develop a prediction rule that supports SSI surveillance efforts by IPs, we aimed to identify all surgeries with any evidence or suggestion of SSI or atypical healing of the surgical wound as suggested directly by diagnoses or wound symptoms or as suggested indirectly by actions taken by the clinicians including prescription of antibiotics, follow-up surgical procedures, or culture orders. Table 1 includes the EHR criteria for an encounter with potential evidence of SSI. Other healthcare encounters during the 60-day post-operative period were marked as “negative for evidence of SSI.” Table 1. Evidence of possible infection related to surgery Criteria Infection diagnosis of surgical site, code or described in notes Infection diagnosis of unspecified site, code or described in notes Purulent fluid Two of: dehiscence, erythema, warmth, tenderness, oedema, and/or non-purulent fluid Culture of surgical site Culture of unspecified site Antibiotic prescription, excluding continuation of therapy, in the absence of an unrelated infectious diagnosis Drainage procedure Criteria Infection diagnosis of surgical site, code or described in notes Infection diagnosis of unspecified site, code or described in notes Purulent fluid Two of: dehiscence, erythema, warmth, tenderness, oedema, and/or non-purulent fluid Culture of surgical site Culture of unspecified site Antibiotic prescription, excluding continuation of therapy, in the absence of an unrelated infectious diagnosis Drainage procedure Table 1. Evidence of possible infection related to surgery Criteria Infection diagnosis of surgical site, code or described in notes Infection diagnosis of unspecified site, code or described in notes Purulent fluid Two of: dehiscence, erythema, warmth, tenderness, oedema, and/or non-purulent fluid Culture of surgical site Culture of unspecified site Antibiotic prescription, excluding continuation of therapy, in the absence of an unrelated infectious diagnosis Drainage procedure Criteria Infection diagnosis of surgical site, code or described in notes Infection diagnosis of unspecified site, code or described in notes Purulent fluid Two of: dehiscence, erythema, warmth, tenderness, oedema, and/or non-purulent fluid Culture of surgical site Culture of unspecified site Antibiotic prescription, excluding continuation of therapy, in the absence of an unrelated infectious diagnosis Drainage procedure Deriving a prediction model to identify evidence of SSI Our goal was to identify healthcare encounters in the 60-day period after ambulatory surgery that contained any evidence of possible SSI. Our optimization target (maximal sensitivity with PPV of at least 10%) was assessed at the level of the surgical case rather than the individual follow-up encounters. Models unable to achieve at least 80% sensitivity with a PPV of 10% or better were rejected from consideration. Model derivation proceeded in four steps: (1) transforming structured health data into analysable form, (2) handling diagnosis codes, (3) incorporating free text information, and (4) tuning model parameters. Details of these steps are provided in the following sections. All analyses were performed in R version 3.3.3 with the addition of the text mining (tm), glmnet, ranger, and pROC packages.12–16 Transformation of structured electronic health data We first focused on gathering structured data in the EHR to identify encounters containing evidence of a possible SSI. Candidate variables were determined after a review of the SSI guidelines and in consultation with IPs and other subject matter experts. The candidate variables included patient characteristics (age at surgery, gender, race, ethnicity and insurance type), characteristics of the surgery (duration of surgery in minutes, and start time of surgery), and time varying characteristics of the follow-up visit (visit type, post-operative day number, presence of a culture order, presence of a systemic antibiotic prescription, and patient temperature). ICD-9 and ICD-10 diagnosis codes Diagnosis codes represented a unique problem because the coding system changed from ICD-9 to ICD-10 during the study period. We chose to include the descriptive words associated with the diagnosis codes as input features rather than the codes themselves. This approach was motivated by the observation that certain key diagnostic descriptors (eg carbuncle, cellulitis, abscess, etc.) occurred both in the ICD-9 and ICD-10 descriptions. Consequently, we approached the text descriptors for coded diagnoses similar to a free text analysis problem. We took the stem for each word in the diagnosis descriptors, used binary weighting of the word stems (each was either present vs absent), and included these features in model development as a “bag of words.” For example, the ICD-10 diagnosis “Carbuncle, unspecified” (L02.93) was represented in the model as presence of the stems for the words “carbuncle” and “unspecified.” Narrative documentation To improve the scalability of our approach, we sought to identify the most parsimonious model that would achieve adequate accuracy. Consequently, we did not include narrative documentation in our initial derivation models. During iterative revisions we added pre-specified key words and phrases related to SSI that had been identified by searching narrative documentation. Regular expression matching patterns were used to count the occurrences of each key word or phrase within the narrative documentation on each follow-up day. Occasionally there were unexpectedly high term counts (eg due to the recurring presence of terms such as “pain” in some documentation templates). We therefore decreased the weight of high counts by taking the square root prior to including the count of each key word or phrase in the model. Model tuning We considered 2 machine learning classifier models that are appropriate for high dimensional data. The first model was logistic regression with lasso regularization (R package “glmnet”), in which a regularization term is imposed on a logistic regression model to prevent over-fitting and to aid in feature selection.13,17,18 The other model was random forests (R package “ranger”), which grows a forest of classification trees for a binary outcome using a training sample and can provide a probability estimate of membership in each class.14,19,20 For interested readers, several tutorials related to the use of these machine learning algorithms are available online.21,22 Using the ambulatory surgeries in the derivation set, optimal model configuration parameters (eg regularization constant) were selected by a grid search using each model’s cross-validation functions in R where possible (eg “cv.glmnet” for the regularized logistic regression). Due to class imbalance arising from the very small number of visits with evidence of SSI, we included class weights in the grid search for model parameters to determine if assigning lower weights to the visits without SSI evidence would improve model performance. To evaluate the utility of adding narrative documentation to the models, we constructed both the regularized logistic regression and random forest models with and without the variables derived from the key words and phrases. A detailed description of our approach to model construction and tuning is included along with annotated R code excerpts in the Supplementary Appendix. Validating the SSI prediction rule After selecting optimal features and model parameters for the two models, we performed predictions on a validation set of ambulatory surgical cases that occurred during the final 10 months of the study period. The structured and narrative EHR data during the follow-up period for these surgical cases was prepared using the same steps described previously with the exception that the word stem features from diagnosis descriptors and the key words/phrases from narrative documentation were constructed using the dictionary of terms constructed from the training documents. Consequently, novel terms in the validation set that were not present in the training set did not contribute to classification. The models were then used to predict whether each ambulatory surgery was associated with evidence of SSI on at least one follow-up day. Using the probability output from each model we plotted receiver operating characteristics (ROC) and calculated the area under the curve (AUC) to compare each model’s performance as a test to differentiate surgeries associated with evidence of SSI from those without evidence. Using the ROCs, we then determined the probability thresholds that achieved 80% and 90% sensitivity for each model. The PPV was then assessed for each model at these two levels of sensitivity. We used bootstrap sampling (10 000 iterations) to estimate 95% confidence intervals (CI) for each performance statistic. Results During the 40-month study period, children experienced 19 777 ambulatory surgeries eligible for SSI surveillance. Telephone interviews were completed and permission obtained to review the EHR for 8502 of these surgeries. Follow-up encounters within 60 days after surgery were available for 7910 surgeries (see study flow diagram, Figure 1). Figure 1. View largeDownload slide Study flow diagram. Figure 1. View largeDownload slide Study flow diagram. Development of prediction rule During the model derivation phase of the study, telephone interviews were completed and consent was obtained to perform chart review for 6871 eligible surgeries that had at least one healthcare encounter during the follow-up period. Each of these surgeries gave rise to follow-up encounters (telephone, office visit, ED or inpatient) on a median of 3 occasions (interquartile range 2-4) during the follow-up period. Multiple healthcare encounters on the same calendar day were considered only a single follow-up episode. Charts were manually reviewed for 1106 surgeries (16%) due to concerns for SSI based either on the telephone interview or the presence of an antibiotic prescription within 60 days of surgery. Among the manually reviewed surgeries there was evidence of SSI present in at least one follow-up encounter for 209 surgeries in the training period (19% of the reviewed cases, 3.0% of all cases). An antibiotic prescription was present during the follow-up period for 135 (65%) of these surgeries. Tuning model configuration parameters After excluding rare terms from diagnosis descriptions, keywords and phrases that occurred less than 5 times, we had a total of 289 candidate predictor variables consisting of 24 variables derived from the structured EHR data, 217 distinct word stems from diagnosis descriptors, and 48 distinct keywords or phrases (Table 1). The optimal lasso penalty (regularization parameter “lambda”) for our logistic regression model was 0.002 regardless of whether or not the 48 variables derived from keywords and phrases were included. For random forests with 5000 trees to ensure stabilized errors, the algorithm’s default value for “mtry” (the number of variables randomly selected at each node of the tree) was optimal (ie the square root of the number of candidate predictor variables). Measures of importance (word frequency, regression coefficients, and permutation importance) for the most frequent terms from the logistic regression and random forest models are also shown in Table 2; some variables had no association with SSI in logistic regression (eg the terms “pain,” and “warm”) yet had relatively high importance in the random forest model. These results are presumably due to interactions with one or more additional terms included in the models (eg the effect of these variables was likely modified by other variables). Table 2. Top ten most frequent keywords in the 6871 training cases. The percent of surgeries associated with at least one occurrence of each term in the follow-up period is reported separately for surgical cases with evidence of SSI and cases without for comparison. Two different measures of importance are also shown: the coefficient from regularized logistic regression model, and the permutation importance from the random forest model PERCENT OF SURGERIES MODEL IMPORTANCE Term SSI Evidence Present (N = 209) No SSI Evidence (N = 6662) Logistic Regression Coefficient Random Forest Permutation Importance Pain 174 (83.3%) 3686 (55.3%) 0 0.000566 Swelling 138 (66.0%) 1915 (28.7%) +0.157 0.000351 Infection 153 (73.2%) 1845 (27.7%) +0.357 0.000200 Warm 91 (43.5%) 1618 (24.3%) 0 0.000169 Drainage 140 (67.0%) 1497 (22.5%) +0.501 0.000559 Redness 135 (64.6%) 1344 (20.2%) +0.515 0.000260 Red 100 (47.8%) 1117 (16.8%) +0.078 0.000166 Erythema 95 (45.5%) 1077 (16.2%) +0.033 0.000276 Oedema 74 (35.4%) 943 (14.2%) 0 0.000012 Tenderness 54 (25.8%) 706 (10.6%) +0.042 0.000031 PERCENT OF SURGERIES MODEL IMPORTANCE Term SSI Evidence Present (N = 209) No SSI Evidence (N = 6662) Logistic Regression Coefficient Random Forest Permutation Importance Pain 174 (83.3%) 3686 (55.3%) 0 0.000566 Swelling 138 (66.0%) 1915 (28.7%) +0.157 0.000351 Infection 153 (73.2%) 1845 (27.7%) +0.357 0.000200 Warm 91 (43.5%) 1618 (24.3%) 0 0.000169 Drainage 140 (67.0%) 1497 (22.5%) +0.501 0.000559 Redness 135 (64.6%) 1344 (20.2%) +0.515 0.000260 Red 100 (47.8%) 1117 (16.8%) +0.078 0.000166 Erythema 95 (45.5%) 1077 (16.2%) +0.033 0.000276 Oedema 74 (35.4%) 943 (14.2%) 0 0.000012 Tenderness 54 (25.8%) 706 (10.6%) +0.042 0.000031 Table 2. Top ten most frequent keywords in the 6871 training cases. The percent of surgeries associated with at least one occurrence of each term in the follow-up period is reported separately for surgical cases with evidence of SSI and cases without for comparison. Two different measures of importance are also shown: the coefficient from regularized logistic regression model, and the permutation importance from the random forest model PERCENT OF SURGERIES MODEL IMPORTANCE Term SSI Evidence Present (N = 209) No SSI Evidence (N = 6662) Logistic Regression Coefficient Random Forest Permutation Importance Pain 174 (83.3%) 3686 (55.3%) 0 0.000566 Swelling 138 (66.0%) 1915 (28.7%) +0.157 0.000351 Infection 153 (73.2%) 1845 (27.7%) +0.357 0.000200 Warm 91 (43.5%) 1618 (24.3%) 0 0.000169 Drainage 140 (67.0%) 1497 (22.5%) +0.501 0.000559 Redness 135 (64.6%) 1344 (20.2%) +0.515 0.000260 Red 100 (47.8%) 1117 (16.8%) +0.078 0.000166 Erythema 95 (45.5%) 1077 (16.2%) +0.033 0.000276 Oedema 74 (35.4%) 943 (14.2%) 0 0.000012 Tenderness 54 (25.8%) 706 (10.6%) +0.042 0.000031 PERCENT OF SURGERIES MODEL IMPORTANCE Term SSI Evidence Present (N = 209) No SSI Evidence (N = 6662) Logistic Regression Coefficient Random Forest Permutation Importance Pain 174 (83.3%) 3686 (55.3%) 0 0.000566 Swelling 138 (66.0%) 1915 (28.7%) +0.157 0.000351 Infection 153 (73.2%) 1845 (27.7%) +0.357 0.000200 Warm 91 (43.5%) 1618 (24.3%) 0 0.000169 Drainage 140 (67.0%) 1497 (22.5%) +0.501 0.000559 Redness 135 (64.6%) 1344 (20.2%) +0.515 0.000260 Red 100 (47.8%) 1117 (16.8%) +0.078 0.000166 Erythema 95 (45.5%) 1077 (16.2%) +0.033 0.000276 Oedema 74 (35.4%) 943 (14.2%) 0 0.000012 Tenderness 54 (25.8%) 706 (10.6%) +0.042 0.000031 Validation set performance During the validation period, charts were manually reviewed for 130 surgeries with evidence of SSI from a total of 1039 total surgeries (13%). Evidence of possible SSI was present in at least one follow-up encounter for 25 of the 130 manually reviewed surgeries in the validation period (2.4% of all cases). Measures of prediction performance on the validation set (AUC and PPV at two target levels of sensitivity) for logistic regression and random forest models are shown in Table 3. To demonstrate the trade-off between sensitivity and specificity, the ROC for the 4 models measured on the validation set are shown in Figure 2. All models were able to achieve the target sensitivity levels, but the random forest model that included keywords from free text was able to achieve an estimated sensitivity of 0.9 with the highest PPV (0.28, [95% CI 0.19, 0.38]). On the validation set, presence of an antibiotic prescription also performed well (sensitivity 0.84 [95% CI 0.64, 0.96], precision 0.28 [95% CI 0.18, 0.39]), which exceeded the performance observed in the derivation set (sensitivity 0.65 [95% CI 0.58, 0.71], precision 0.30 [95% CI 0.26, 0.35]). Table 3. Performance statistics of the four prediction models assessed on the validation set. Area under the receiver-operating characteristic is reported along with positive predictive values observed at pre-specified target levels of sensitivity. Bootstrap sampling was performed to estimate 95% confidence intervals (CI). The lower confidence limits of attainable sensitivity are noted below in footnotes Positive Predictive Value (PPV) at Target Sensitivity Area Under Curve (AUC) [95% CI] PPV at 0.8 Sensitivitya [95% CI] PPV at 0.9 Sensitivityb [95% CI] With keywords  Random forest 0.97 [0.92, 0.98] 0.31 [0.21, 0.44] 0.28 [0.19, 0.38]  Logistic regression 0.97 [0.94, 0.98] 0.40 [0.27, 0.54] 0.17 [0.12, 0.25] No keywords  Random forest 0.94 [0.84, 0.97] 0.19 [0.12, 0.27] 0.12 [0.08, 0.17]  Logistic regression 0.94 [0.86, 0.97] 0.32 [0.21, 0.44] 0.11 [0.07, 0.16] Positive Predictive Value (PPV) at Target Sensitivity Area Under Curve (AUC) [95% CI] PPV at 0.8 Sensitivitya [95% CI] PPV at 0.9 Sensitivityb [95% CI] With keywords  Random forest 0.97 [0.92, 0.98] 0.31 [0.21, 0.44] 0.28 [0.19, 0.38]  Logistic regression 0.97 [0.94, 0.98] 0.40 [0.27, 0.54] 0.17 [0.12, 0.25] No keywords  Random forest 0.94 [0.84, 0.97] 0.19 [0.12, 0.27] 0.12 [0.08, 0.17]  Logistic regression 0.94 [0.86, 0.97] 0.32 [0.21, 0.44] 0.11 [0.07, 0.16] a Lower limit of sensitivity was 0.58 for all models b Lower limit of sensitivity was 0.73 for all models Table 3. Performance statistics of the four prediction models assessed on the validation set. Area under the receiver-operating characteristic is reported along with positive predictive values observed at pre-specified target levels of sensitivity. Bootstrap sampling was performed to estimate 95% confidence intervals (CI). The lower confidence limits of attainable sensitivity are noted below in footnotes Positive Predictive Value (PPV) at Target Sensitivity Area Under Curve (AUC) [95% CI] PPV at 0.8 Sensitivitya [95% CI] PPV at 0.9 Sensitivityb [95% CI] With keywords  Random forest 0.97 [0.92, 0.98] 0.31 [0.21, 0.44] 0.28 [0.19, 0.38]  Logistic regression 0.97 [0.94, 0.98] 0.40 [0.27, 0.54] 0.17 [0.12, 0.25] No keywords  Random forest 0.94 [0.84, 0.97] 0.19 [0.12, 0.27] 0.12 [0.08, 0.17]  Logistic regression 0.94 [0.86, 0.97] 0.32 [0.21, 0.44] 0.11 [0.07, 0.16] Positive Predictive Value (PPV) at Target Sensitivity Area Under Curve (AUC) [95% CI] PPV at 0.8 Sensitivitya [95% CI] PPV at 0.9 Sensitivityb [95% CI] With keywords  Random forest 0.97 [0.92, 0.98] 0.31 [0.21, 0.44] 0.28 [0.19, 0.38]  Logistic regression 0.97 [0.94, 0.98] 0.40 [0.27, 0.54] 0.17 [0.12, 0.25] No keywords  Random forest 0.94 [0.84, 0.97] 0.19 [0.12, 0.27] 0.12 [0.08, 0.17]  Logistic regression 0.94 [0.86, 0.97] 0.32 [0.21, 0.44] 0.11 [0.07, 0.16] a Lower limit of sensitivity was 0.58 for all models b Lower limit of sensitivity was 0.73 for all models Figure 2. View largeDownload slide Receiver operating characteristics for the four prediction models on the validation set, with smoothing added to improve legibility. Figure 2. View largeDownload slide Receiver operating characteristics for the four prediction models on the validation set, with smoothing added to improve legibility. Discussion In this manuscript we have described a method for screening EHR data for the rare, but clinically important outcome of SSI. We developed a random forest model using information from both structured and free text information that achieved an estimated sensitivity of 0.9 with a PPV of 0.28. Using this model as a pre-screening tool, IPs would identify the vast majority of SSI cases that met the CDC definition by reviewing charts for about 4 cases to identify one case that met the NHSN definition of SSI (estimated lower limit of sensitivity was 0.73). This is a marked improvement over the current situation where all surgical cases must be reviewed to identify the approximately 3% that have evidence of SSI (over 30 chart reviews to identify one case with evidence of SSI). Logistic regression with the lasso penalty also performed well both with and without information from free text. Notably, as shown in Table 3, at the sensitivity level of 0.8, the logistic regression models attained a higher PPV than the corresponding random forest models. We also found that the presence of an antibiotic prescription on post-operative days 1-60 might perform well as a simplified approach to identifying the majority of cases of possible SSI when applied to our validation cohort (sensitivity 0.84, PPV 0.28). This approach had poor sensitivity when applied to our derivation cohort (0.65), possibly due to less frequent use of electronic prescribing at the start of our study. Health systems interested in screening EHR data for SSI should consider this simple approach, but derivation of a more complex prediction rule may be necessary if prescription data are not consistently available. Although the increasing availability of electronic health data offers tremendous opportunities for IPs to more readily identify rare outcomes such as SSI, few tools have been developed to support their work. Central line infections have long been a target for close scrutiny as a marker of the quality and safety of inpatient care, thus numerous tools have been developed and evaluated to support the mandated reporting of these outcomes.23–25 Less effort has been put into developing similar tools for SSI, likely because few jurisdictions require reporting of SSI after ambulatory surgery. In other domains, the EHR has been used as a tool to screen for rare events or rare cohorts. This has included the use of keyword searches, rule-based algorithms, and machine learning, which have yielded results that are similar to our results in the domain of SSI surveillance.10,26,27 Other studies have evaluated natural language processing for developing patient cohorts with some limited success, consistent with our attempts with investigating this approach.28 Studies have also evaluated EHR records for post-surgical complications using natural language processing with reasonable success rates.29,30 Generally, studies of rare diseases and clinical events have found that billing diagnoses (ICD-9/10) are often insufficient, and that processing of text information increases the yield for these algorithms.31,32 Future research should continue to look creatively at the potential to use both structured and narrative EHR information to identify rare, but clinically important events. Limitations Although there were 4 ambulatory surgical facilities participating in our study, these facilities were all part of a single healthcare system, which may limit the ability to directly apply the random forest model we developed in new settings. However, it is likely that institutions seeking to adopt a similar approach could improve the performance of the model by adding examples of known SSI cases from their own health system in a model re-training process. Our approach assumes the availability of certain types of information in electronic form such as antibiotic prescriptions and progress notes information. Without this information, health systems will likely have difficulty screening charts for evidence of SSI using our approach. Fortunately, thanks in part to EHR incentive programs, at this time most large health systems are using the EHR to capture these types of information.33 Our study used text representations of ICD-9 and ICD-10 diagnoses due to the transition in coding that happened during our study. Prior research has revealed that mapping tables between ICD-9 and ICD-10 codes are inexact, and particularly suffer from differences in the granularity of terms.34 For SSI surveillance activities, accurate mappings of diagnostic codes are required both for codes related to surgical complications, and for codes describing the common conditions where antibiotics are indicated. These mapping efforts can be substantial. For example, in our own work with antimicrobial stewardship activities focused on respiratory tract infections we identified 98 ICD-9 codes representing bacterial respiratory tract infections,35 which required a manual review of 1285 mapped codes to successfully continue these stewardship activities after the transition to ICD-10. However, in the future, organizations may benefit from simpler approaches that rely exclusively on ICD-10 codes. Conclusion An automated prediction rule derived from both structured and narrative EHR data using the random forest machine learning algorithm can be used to screen for surgical site infections after ambulatory surgery with high recall (sensitivity) and acceptable precision (positive predictive value). Presence of an antibiotic prescription within 60 days after surgery may also perform well, but may have inadequate sensitivity in settings that have incomplete prescription data. Funding This project was wholly supported by grant R01HS020921 (Electronic Surveillance for Wound Infections after Ambulatory Pediatric Surgery) from the Agency for Healthcare Research and Quality. The content is solely the responsibility of the authors and does not necessarily represent the official views of the Agency for Healthcare Research and Quality. Competing interests None. Contributors Robert Grundmeier contributed to the conception and design of the study, acquisition of data, analysis and interpretation of the data, drafted the manuscript, and approved the final manuscript as submitted. Rui Xiao contributed to the conception and design of the study, analysis and interpretation of the data, critically reviewed the manuscript, and approved the final manuscript as submitted. Rachael K. Ross contributed to the conception and design of the study, critically reviewed the manuscript, and approved the final manuscript as submitted. Mark J. Ramos contributed to the acquisition of data, critically reviewed the manuscript, and approved the final manuscript as submitted. Dean Karavite, Jeremy Michel, Jeffrey Gerber, and Susan Coffin contributed to the conception and design of the study, critically reviewed the manuscript, and approved the final manuscript as submitted. SUPPLEMENTARY MATERIAL Supplementary material is available at Journal of the American Medical Informatics Association online. ACKNOWLEDGEMENTS We thank our infection preventionists, Susan L. Rettig and Eva E. Teszner, for their tireless efforts manually reviewing charts for this project. References 1 Cullen KA , Hall MJ , Golosinskiy A. Ambulatory surgery in the United States, 2006 . Natl Health Stat Report 2009 : 1 – 25 . https://www.cdc.gov/nchs/data/nhsr/nhsr011.pdf Accessed March 2, 2018. 2 Zimlichman E , Henderson D , Tamir O , et al. . Health care–associated infections: a meta-analysis of costs and financial impact on the US health care system . JAMA Intern Med 2013 ; 173 22 : 2039 – 46 . Google Scholar CrossRef Search ADS PubMed 3 West N , Eng T. Monitoring and reporting hospital-acquired conditions: a federalist approach . Medicare Medicaid Res Rev 2014 ; 4 4 : E1 – 16 . Google Scholar CrossRef Search ADS 4 Radey LA , West N , Eng T , et al. . State Government Tracking of Hospital- Acquired Conditions Final Prepared for Published Online First; 2010 . https://www.cms.gov/Research-Statistics-Data-and-Systems/Statistics-Trends-and-Reports/Reports/Downloads/West_HAC_2010.pdf Accessed March 2, 2018. 5 Petherick ES , Dalton JE , Moore PJ , et al. . Methods for identifying surgical wound infection after discharge from hospital: a systematic review . BMC Infect Dis 2006 ; 6 1 : 170 . Google Scholar CrossRef Search ADS PubMed 6 Horan TC , Andrus M , Dudeck MA. CDC/NHSN surveillance definition of health care-associated infection and criteria for specific types of infections in the acute care setting . Am J Infect Control 2008 ; 36 5 : 309 – 32 . Google Scholar CrossRef Search ADS PubMed 7 Warren DK , Nickel KB , Wallace AE , et al. . Can additional information be obtained from claims data to support surgical site infection diagnosis codes? Infect Control Hosp Epidemiol 2014 ; 35 ( S3 ): S124 – 32 . Google Scholar CrossRef Search ADS PubMed 8 Olsen MA , Ball KE , Nickel KB , et al. . Validation of ICD-9-CM diagnosis codes for surgical site infection and noninfectious wound complications after mastectomy . Infect Control Hosp Epidemiol 2017 ; 38 03 : 334 – 9 . Google Scholar CrossRef Search ADS PubMed 9 Rhee C , Huang SS , Berríos-Torres SI , et al. . Surgical site infection surveillance following ambulatory surgery . Infect Control Hosp Epidemiol 2015 ; 36 02 : 225 – 8 . Google Scholar CrossRef Search ADS PubMed 10 Ford E , Carroll JA , Smith HE , et al. . Extracting information from the text of electronic medical records to improve case detection: a systematic review . J Am Med Inform Assoc 2016 ; 23 5 : 1007 – 15 . Google Scholar CrossRef Search ADS PubMed 11 Hoffman S , Podgurski A. Big bad data: law, public health, and biomedical databases . J Law Med Ethics 2013 ; 41 (suppl 1) : 56 – 60 . Google Scholar CrossRef Search ADS PubMed 12 Feinerer I. Introduction to the tm package: text mining in R . Compr R Arch Netw ; 2017 : 1 – 8 . https://cran.r-project.org/web/packages/tm/vignettes/tm.pdf Accessed March 2, 2018. 13 Friedman J , Hastie T , Tibshirani R. Regularization paths for generalized linear models via coordinate descent . J Stat Soft 2010 ; 33 1 . doi:10.18637/jss.v033.i01. 14 Wright MN , Ziegler A. ranger: a fast implementation of random forests for high dimensional data in C++ and R . J Stat Soft 2017 ; 77 1 . doi:10.18637/jss.v077.i01. 15 Robin X , Turck N , Hainard A , et al. . pROC: an open-source package for R and S+ to analyze and compare ROC curves . BMC Bioinform 2011 ; 12 1 : 77 . Google Scholar CrossRef Search ADS 16 R Core Team . R: a language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria; 2017 . http://www.r-project.org/ Accessed March 2, 2018. 17 Tibshirani R. Regression selection and shrinkage via the Lasso . J R Stat Soc B 1996 ; 58 : 267 – 88 . 18 Tibshirani R. Regression shrinkage and selection via the lasso: A retrospective . J R Stat Soc Ser B Stat Methodol 2011 ; 73 3 : 273 – 82 . Google Scholar CrossRef Search ADS 19 Malley JD , Kruppa J , Dasgupta A , et al. . Probability machines: consistent probability estimation using nonparametric learning machines . Methods Inf Med 2011 ; 51 1 : 74 – 81 . Google Scholar CrossRef Search ADS PubMed 20 Breiman L. Random forests . Mach Learn 2001 ; 45 1 : 5 – 32 . Google Scholar CrossRef Search ADS 21 Leek J , Peng RD , Caffo B. Practical Machine Learning. Johns Hopkins University, Coursera; 2018 . https://www.coursera.org/learn/practical-machine-learning Accessed April 16, 2018. 22 Fox E , Guestrin C. Machine Learning. University of Washington, Coursera; 2018 . https://www.coursera.org/specializations/machine-learning Accessed April 16, 2018. 23 Snyders RE , Goris AJ , Gase KA , et al. . Increasing the reliability of fully automated surveillance for central line–associated bloodstream infections . Infect Control Hosp Epidemiol 2015 ; 36 12 : 1396 – 400 . Google Scholar CrossRef Search ADS PubMed 24 Woeltje KF , McMullen KM , Butler AM , et al. . Electronic surveillance for healthcare-associated central line—associated bloodstream infections outside the intensive care unit . Infect Control Hosp Epidemiol 2011 ; 32 11 : 1086 – 90 . Google Scholar CrossRef Search ADS PubMed 25 Quan KA , Cousins SM , Porter DD , et al. . Electronic health record solutions to reduce central line-associated bloodstream infections by enhancing documentation of central line insertion practices, line days, and daily line necessity . Am J Infect Control 2016 ; 44 4 : 438 – 43 . Google Scholar CrossRef Search ADS PubMed 26 Garg R , Dong S , Shah S , et al. . A bootstrap machine learning approach to identify rare disease patients from electronic health records. arXiv Prepr. Published Online First; 2016 . http://arxiv.org/abs/1609.01586. Accessed March 2, 2018. 27 Liao S , Xiao J , Xie Y , et al. . Towards use of electronic health records: cancer classification. In: MSM ’17 Proceedings of the Symposium on Modeling and Simulation in Medicine . San Diego, CA, USA : Society for Computer Simulation International ; 2017 . https://dl.acm.org/citation.cfm?id=3108764. Accessed March 2, 2018. 28 Shivade C , Raghavan P , Fosler-Lussier E , et al. . A review of approaches to identifying patient phenotype cohorts using electronic health records . J Am Med Inform Assoc 2014 ; 21 2 : 221 – 30 . Google Scholar CrossRef Search ADS PubMed 29 Henry FF , Murff HJ , Matheny ME , et al. . Exploring the frontier of electronic health record surveillance the case of postoperative complications . Med Care 2013 ; 51 6 : 509 – 16 . Google Scholar CrossRef Search ADS PubMed 30 Murff HJ , FitzHenry F , Matheny ME , et al. . Automated identification of postoperative complications within an electronic medical record using natural language processing . JAMA 2011 ; 306 8 : 848 – 55 . Google Scholar CrossRef Search ADS PubMed 31 Carrell DS , Cronkite D , Palmer RE , et al. . Using natural language processing to identify problem usage of prescription opioids . Int J Med Inform 2015 ; 84 12 : 1057 – 64 . Google Scholar CrossRef Search ADS PubMed 32 Wang M , Cyhaniuk A , Cooper DL , et al. . Identification of people with acquired hemophilia in a large electronic health record database . J Blood Med 2017 ; 8 : 89 – 97 . Google Scholar CrossRef Search ADS PubMed 33 Blumenthal D , Tavenner M. The ‘meaningful use’ regulation for electronic health records . N Engl J Med 2010 ; 363 6 : 501 – 4 . Google Scholar CrossRef Search ADS PubMed 34 Caskey R , Zaman J , Nam H , et al. . The transition to ICD-10-CM: challenges for pediatric practice . Pediatrics 2014 ; 134 1 : 31 – 6 . Google Scholar CrossRef Search ADS PubMed 35 Goyal MK , Johnson TJ , Chamberlain JM , et al. . Racial and ethnic differences in antibiotic use for viral illness in emergency departments . Pediatrics 2017 ; 140 4 : e20170203 . Google Scholar CrossRef Search ADS PubMed © The Author(s) 2018. Published by Oxford University Press on behalf of the American Medical Informatics Association. All rights reserved. For permissions, please email: journals.permissions@oup.com This article is published and distributed under the terms of the Oxford University Press, Standard Journals Publication Model (https://academic.oup.com/journals/pages/about_us/legal/notices)

Journal

Journal of the American Medical Informatics AssociationOxford University Press

Published: Sep 1, 2018

There are no references for this article.