Get 20M+ Full-Text Papers For Less Than $1.50/day. Start a 14-Day Trial for You or Your Team.

Learn More →

Automatic population of eMeasurements from EHR systems for inpatient falls

Automatic population of eMeasurements from EHR systems for inpatient falls Abstract Objective Representing nursing data sets in a standard way will help to facilitate sharing relevant information across settings. We aimed to populate nursing process and outcome metrics with electronic health record (EHR) data and then compare the results with event reporting systems. Methods We used the “eMeasure” development process of the National Quality Forum adopted by the American Nurses Association. We used operational definitions of quality measures from the American Nurses Association and the US Institute for Healthcare Improvement and employed concept mapping of local data elements to 2 controlled vocabularies to define a standard data dictionary: (1) Logical Observation Identifiers Names and Codes and (2) International Classification for Nursing Practice. We assessed feasibility using the nursing data set of 7829 and 8199 patients from 2 general hospitals with different EHR systems. Using inpatient falls as a use case, we compared the populated measures with results from the event reporting systems. Results We identified 17 care components and 118 unique concepts and matched them with data elements in the EHRs. Including suboptimal mapping, 98% of the assessment concepts mapped to Logical Observation Identifiers Names and Codes and 52.9% of intervention concepts mapped to International Classification for Nursing Practice. While not all process indicators were available from event reporting systems, we successfully populated 9 fall prevention process indicators and the fall rate outcome indicator from the 2 EHRs. We were unable to populate the falls with an injury rate indicator. Conclusions EHR data can populate fall prevention process measure metrics and at least one inpatient fall prevention outcome metric. Introduction The rapid implementation of electronic health records (EHRs) and the integration of EHR data into clinical data repositories (CDRs) means that large quantities of clinical data are now available. Combining these data with computational technologies shows promise in improving nursing knowledge representation and patient outcomes.1 In an effort to align quality indicators with codified clinical and billing data, the US Centers for Medicare and Medicaid Services converted 113 National Quality Forum (NQF)-endorsed Clinical Quality Measures from a paper-based format relying on chart abstraction into an electronic “eMeasure” format.2 For nursing-sensitive quality indicators, little is known about the feasibility of using nursing data in EHRs to populate the eMeasure format as well as for benchmarking across disparate EHRs. Nonstandardized data are used in most EHR systems due to the lack of knowledge regarding standardized processes and data among the nursing executive and clinician workforce.1 However, multiple years of experience with EHR systems creates an opportunity to explore ways to standardize locally developed nursing vocabularies. Our previous nationwide survey of EHR systems conducted in 2012 revealed that more than half of general hospitals in Korea implemented a local nursing vocabulary.3 We hypothesized that if nursing data in these local systems were represented in a standard way, they could be shared and compared on regional and national levels and would facilitate the identification of trends and outcomes. Here we describe the process of transforming local nursing data into a standardized format to populate fall prevention quality measures, and demonstrate the feasibility of this method within 2 disparate EHR environments. We also compared this approach with the existing manual approach using standard event reporting systems. Methods Site and Settings Two hospitals located in metropolitan areas of Korea with different locally developed EHR systems participated in this study. One was a tertiary academic hospital with 900 beds and the other was a secondary hospital with 800 beds. Both hospitals used EHR systems for >8 years that included nursing vocabularies with both local terms and nursing statements that were based on the International Classification for Nursing Practice (ICNP) and North American Nursing Diagnosis Association, Nursing Intervention Classification, and Nursing Outcomes Classification (3N) controlled vocabularies. Identification of Outcome Metrics for Inpatient Falls The eMeasure development process of the NQF adopted by the American Nurses Association (ANA) and the following National Database of Nursing Quality Indicator outcome metrics for patient falls were used to aggregate local data and to populate quality measures: patient fall rate and patient falls with injury rate.4 For process metrics, we used the Institute for Healthcare Improvement definitions.5 Process indicators measure compliance with key care components of evidence-based nursing. Improvements in the scores for individual process measures indicate that the processes underlying these care elements have improved, and improvements in patient outcomes should follow. Mant6 advised using process metrics to measure the quality of health care if linkages exist between given processes of care and patient outcomes. Where such measures are available, relevant, and practical, they are used in conjunction with outcome metrics since they are sensitive to differences in the quality of care in the short term; typically, before outcome measures are available.6,7 Identification of Standard Content of Fall Related Care We formed a project team consisting of experts in the following relevant domains: nursing informatics, terminology, quality management, and patient safety. Two College of Nursing professors, 4 graduate students with 5–10 years of practical experience at university hospitals, nursing staff, and managers of the sites were involved. We used the International Classification for Patient Safety’s (ICPS) conceptual framework to identify patient safety concepts for falls.8 The process of how standard content was identified from a literature review, and arranged into key care components under the ICPS framework, is described in detail elsewhere.9,10 The aim of the literature review was to identify evidence-based nursing content related to falls that could be mapped to nursing documentation in the EHR systems through standard terminologies. We identified 17 key care components corresponding to fall prevention and post-fall nursing care. We also searched national and regional nurse-sensitive quality databases with the aim of supporting quality assessment and demonstrating the nursing contribution to patient care using measures similar to those proposed in the ANA initiative in 199511: National Database of Nursing Quality Indicator,12 California Nursing Outcomes Coalition,13 Military Nursing Outcomes Database,14 and VA Administration Nursing Outcomes Database.15 Data Collection We performed the following 4 data collection steps: (1) review of practice guidelines and literature to identify fall prevention key care components, process, and outcome measures; (2) data mapping from local nursing vocabularies to standardized terminologies; (3) populating eMeasures with standardized data; and (4) comparing an eMeasurement system with an event reporting system in terms of its ability to generate process and outcome metrics. Figure 1 shows the relationships between these steps. Figure 1. View largeDownload slide The 4 steps of this study procedure. LOINC: Logical Observation Identifiers Names and Codes; ICNP: International Classification for Nursing Practice. Figure 1. View largeDownload slide The 4 steps of this study procedure. LOINC: Logical Observation Identifiers Names and Codes; ICNP: International Classification for Nursing Practice. Identifying fall prevention key care components and metrics We selected the 7 guidelines or tools recommended by The Joint Commission in 2015 and the Korean Hospital Nurses Association Nursing Practice Guidelines to identify key care components and fall prevention measures.16,22Figure 2 displays the list of resources. Figure 2. Inpatient fall prevention practice guidelines and tools reviewed. Agency for Healthcare Research and Quality Fall Prevention Toolkit17 ECRI Institute Falls Prevention Guidance18 Institute for Clinical Systems Improvement Prevention of Falls (Acute Care) Protocol19 Institute for Healthcare Improvement's (IHI) Transforming Care At The Bedside How-to Guide5 The Joint Commission Center for Transforming Healthcare's Targeted Solutions Tool (TST®) for Preventing Falls20 Veterans Affairs. VA National Center for Patient Safety: Falls Toolkit21 Veterans Affair (VA) National Center for Patient Safety Implementation Guide For Fall Injury Reduction22 Korean Hospital Nurses Association Nursing Practice Guidelines of Prevention of Inpatient Falls23 Figure 2. Inpatient fall prevention practice guidelines and tools reviewed. Agency for Healthcare Research and Quality Fall Prevention Toolkit17 ECRI Institute Falls Prevention Guidance18 Institute for Clinical Systems Improvement Prevention of Falls (Acute Care) Protocol19 Institute for Healthcare Improvement's (IHI) Transforming Care At The Bedside How-to Guide5 The Joint Commission Center for Transforming Healthcare's Targeted Solutions Tool (TST®) for Preventing Falls20 Veterans Affairs. VA National Center for Patient Safety: Falls Toolkit21 Veterans Affair (VA) National Center for Patient Safety Implementation Guide For Fall Injury Reduction22 Korean Hospital Nurses Association Nursing Practice Guidelines of Prevention of Inpatient Falls23 Outcome and process metrics and the corresponding care components to be extracted from each EHR system were vetted with inpatient nursing teams. The calculation of outcome indicator logic followed the ANA/National Quality Forum endorsed guidelines for data collection.24 Data mapping to standardized nursing terminologies The standard fall prevention content consisted of nursing assessment, diagnosis, and intervention activities. We refer to nursing data as the data set in EHR systems including 3 broad categories of elements: (1) nursing care; (2) patient or client demographics; and (3) service elements. Data elements are the atomic units of nursing data. For example, when evaluating cognitive function, specifically delirium, several descriptive terms exist in standard content or EHR systems including altered mental status, confusional state, and delirium. We designate delirium as one data element. We mapped data elements to standard terminology concepts from Logical Observation Identifiers Names and Codes (LOINC) for administrative, assessments, and diagnoses, and ICNP for interventions. The Health and Welfare Ministry of Korea endorses LOINC and ICNP as national healthcare standards. We used Regenstrief LOINC Mapping Assistant (RELMA, LOINC browser) and the ICNP Browser at the International Council of Nursing. Two graduate students with 5 and 10 years of clinical experience conducted the initial mapping, then the authors (IC and EHB) reviewed and discussed further to classify the mapping results into 3 categories of optimal (exact match), suboptimal (partial match), and not mapped (no match). We used the concept ontology of the Systematized Nomenclature of Medicine-Clinical Terminology (SNOMED-CT) as a sematic map indicating a type of graphic organizer that visually represented relationships among categories of concepts. We also used the 8 guidelines of preventing falls for assuring the relationships among concepts in the context of fall prevention. We classified the cases into the optimal mapping, in which a data element completely corresponding to a concept semantically or in relation to an “is a” (parent or child relation) with a concept. We classified discrepancies or disagreements on semantics between reviewers as suboptimal.25,26 The data dictionary for each hospital guided the collection and normalization of data from multiple sources: the EHR systems, reporting systems, and administrative databases. Table 1 shows the scheme of the data dictionary and data examples. The EHR data included both structured and semi-structured formats. We coded nursing statements in nursing notes in a semi-structured format. Table 1. Scheme of Inpatient Fall Data Dictionary and Data Examples Field Example of assessment Example of intervention Care component Cognitive factors Impaired mobility Care type Assessment Intervention Care activity Observe (assess or evaluate) delirium/confusional state/altered mental status/ reversible dementia/organic brain syndrome Regularly scheduled assistance with toileting Data element Delirium Aid of toileting Definition Confusional state Regularly scheduled assistance with toileting for patients with impaired mobility Terminology LOINC ICNP Intervention Concept(s) Delirium [MDSv3] Assisting with toileting Terminology code 54626-7 10023531 Local facility ID Hospital A Hospital B Local EHR ID.version eChart.2.0 SmartCare.3.0 Local screen-item name Nursing notes Nursing notes Local expression Observed delirium symptom(degreea: disorganized) Provide regularly assistance with toileting Local data type Assessment statement Intervention statement Local code S03745 A05595 Field Example of assessment Example of intervention Care component Cognitive factors Impaired mobility Care type Assessment Intervention Care activity Observe (assess or evaluate) delirium/confusional state/altered mental status/ reversible dementia/organic brain syndrome Regularly scheduled assistance with toileting Data element Delirium Aid of toileting Definition Confusional state Regularly scheduled assistance with toileting for patients with impaired mobility Terminology LOINC ICNP Intervention Concept(s) Delirium [MDSv3] Assisting with toileting Terminology code 54626-7 10023531 Local facility ID Hospital A Hospital B Local EHR ID.version eChart.2.0 SmartCare.3.0 Local screen-item name Nursing notes Nursing notes Local expression Observed delirium symptom(degreea: disorganized) Provide regularly assistance with toileting Local data type Assessment statement Intervention statement Local code S03745 A05595 aAttribute and value of the nursing statement. Table 1. Scheme of Inpatient Fall Data Dictionary and Data Examples Field Example of assessment Example of intervention Care component Cognitive factors Impaired mobility Care type Assessment Intervention Care activity Observe (assess or evaluate) delirium/confusional state/altered mental status/ reversible dementia/organic brain syndrome Regularly scheduled assistance with toileting Data element Delirium Aid of toileting Definition Confusional state Regularly scheduled assistance with toileting for patients with impaired mobility Terminology LOINC ICNP Intervention Concept(s) Delirium [MDSv3] Assisting with toileting Terminology code 54626-7 10023531 Local facility ID Hospital A Hospital B Local EHR ID.version eChart.2.0 SmartCare.3.0 Local screen-item name Nursing notes Nursing notes Local expression Observed delirium symptom(degreea: disorganized) Provide regularly assistance with toileting Local data type Assessment statement Intervention statement Local code S03745 A05595 Field Example of assessment Example of intervention Care component Cognitive factors Impaired mobility Care type Assessment Intervention Care activity Observe (assess or evaluate) delirium/confusional state/altered mental status/ reversible dementia/organic brain syndrome Regularly scheduled assistance with toileting Data element Delirium Aid of toileting Definition Confusional state Regularly scheduled assistance with toileting for patients with impaired mobility Terminology LOINC ICNP Intervention Concept(s) Delirium [MDSv3] Assisting with toileting Terminology code 54626-7 10023531 Local facility ID Hospital A Hospital B Local EHR ID.version eChart.2.0 SmartCare.3.0 Local screen-item name Nursing notes Nursing notes Local expression Observed delirium symptom(degreea: disorganized) Provide regularly assistance with toileting Local data type Assessment statement Intervention statement Local code S03745 A05595 aAttribute and value of the nursing statement. Populating falls indicators from EHR data Based on the data dictionary, we retrieved and collected data from 4 medical-surgical nursing units of each hospital over one calendar year (2014). We used the operational patient selection criteria of NQF measures #0141 (patient fall rate) and #0202 (patient falls with injury) for the targeted medical-surgical units. We included: (1) adult inpatients aged 18 years and older and (2) patients who had been admitted for at least 24 h. We excluded patients with a primary pediatric, psychiatric, or obstetrics diagnosis, or who received resuscitation treatment, or died. We included only patient stays on targeted units during the study period. Comparing the eMeasurements with event reporting systems The 2 hospitals independently submitted data from their locally developed event reporting systems. Data consisted of patient demographics, admission information, situational descriptions about falls, post-fall measures, and management information used for root-cause analysis. We compared the completeness of indicators populated with manual event reporting data with indicators populated directly from the EHR systems. We also used these data to cross-check cases recorded in the EHR system and the event reporting system. Three authors (IC, EHB, and SYL) conducted chart reviews to identify fall events hospital managers confirmed new cases. Data Analysis The Institutional Review Board at each hospital approved this research project. Each hospital provided de-identified data from the clinical data repository (CDR). To identify the target patients, we applied operational criteria that were modified from 2 ANA endorsed nursing-sensitive outcomes; total falls per 1000 patient days for fall rate, and injury falls per 1000 patient days for injury fall rate.24,27 We expressed process indicators as proportions or average frequencies, such as the number of nursing activities conducted to assess a patient’s disease-related factors, cognitive factors, behavior factors, or therapeutics during the days when the patient was at risk of falling. Results Identifying Fall Prevention Key Care Components and Measures Figure 3, the Fall Prevention Indicator Map, includes the key fall prevention care components and measures identified through literature and guideline review. The indicator map represents the structural relationships between fall prevention key care components and indicators. The left side of the figure displays the parts of the care components used to calculate the indicators that appear on the right side of the figure. The figure includes 2 outcome indicators and 9 process indicators. Figure 3. View largeDownload slide Indicator map of inpatient falls (aoutcome indicators, bprocess indicators). Figure 3. View largeDownload slide Indicator map of inpatient falls (aoutcome indicators, bprocess indicators). Data Mapping to Standardized Nursing Terminologies The concepts of nursing assessments and fall incident categories were all successfully mapped to LOINC except the concept of injury level in incident characteristics. According to its definition, fall injury level has 4 categories in its value list: none, minor, moderate, and major. However, we mapped this concept to the LOINC concept event outcome of the Medical Event Reporting System—Total HealthSystem taxonomy, for which the value list was 4 categories: no recovery with harm, no recovery with no harm, near miss with unplanned recovery, and near miss with planned recovery. The post-fall assessment concepts mapped to LOINC except the following 2 concepts: absolute bed rest state and close observation state. A total of 52.9% of nursing intervention concepts mapped to ICNP (including suboptimal mappings). For example, we successfully mapped the concepts observation and surveillance, provision of protocol, and post-fall care provided to 18 intervention statements in ICNP. However, we could not find ICNP interventions on the nursing activities, communication and education such as sharing high risk patients and use of visual fall risk indicator. The closest interventions were the collaborating with interprofessional team and fall prevention in ICNP. We also found almost all of the universal precaution nursing activities and environmental interventions were related to the ICNP interventions, fall prevention and environmental safety management. However, the ICNP interventions were too implicit and abstract. We classified those into the not mapped. Tables 2 and 3, respectively, present examples of the mapping results for LOINC and ICNP intervention statements. Each data element is used to apply the inclusion or exclusion criteria, compute denominators, or aggregate by the care component. The full version of mapping list is online in Supplementary Material S1. Table 2. Example Nursing Assessment Data Elements Whose Concepts Mapped onto LOINC Care component Data element LOINC concepts (ID) Property Time System Scale Method Relation with populating indicator Reason for encounter Primary diagnosis Primary diagnosis (18630-4) Imp Pt Patient Nom — Inclusion criteria Administrative Length of stay Hospital stay duration (78033-8) Time Pt Patient Qn — Patient days Pathophysiologic disease-related factors Use of anticoagulants/coagulopathy diagnosis/the first day of surgical procedure Bleeding risk [CCC] (81225-5) Find Pt Cardiovascular system Ord Observed.CCC Frequency of nursing assessment of risk factors Nursing diagnosis: visual impairment Visual alteration [CCC] (28242-6) Find Pt Eye Ord Observed.CCC Cognitive factors Delirium Delirium [MDSv3] (54626-7) – Pt Patient – MDSv3 Consciousness Level of consciousness (80288-4) Find Pt Patient Ord – Behavior factors Fall history Fall history on admission [MDSv3] (54849-5) – Pt Patient – MDSv3 Therapeutic factors IV line Intravenous apparatus (59457-2) Find Pt Patient Ord Morse Fall Scale Foley catheter Indwelling catheter [MDS] (45628-5) Find Pt Patient Ord MDS Incident characteristic Injury level Event outcome [MERSTH] (42545-4) Find Pt Patient Ord Observed. MERSTH Frequency of injurious falls Care component Data element LOINC concepts (ID) Property Time System Scale Method Relation with populating indicator Reason for encounter Primary diagnosis Primary diagnosis (18630-4) Imp Pt Patient Nom — Inclusion criteria Administrative Length of stay Hospital stay duration (78033-8) Time Pt Patient Qn — Patient days Pathophysiologic disease-related factors Use of anticoagulants/coagulopathy diagnosis/the first day of surgical procedure Bleeding risk [CCC] (81225-5) Find Pt Cardiovascular system Ord Observed.CCC Frequency of nursing assessment of risk factors Nursing diagnosis: visual impairment Visual alteration [CCC] (28242-6) Find Pt Eye Ord Observed.CCC Cognitive factors Delirium Delirium [MDSv3] (54626-7) – Pt Patient – MDSv3 Consciousness Level of consciousness (80288-4) Find Pt Patient Ord – Behavior factors Fall history Fall history on admission [MDSv3] (54849-5) – Pt Patient – MDSv3 Therapeutic factors IV line Intravenous apparatus (59457-2) Find Pt Patient Ord Morse Fall Scale Foley catheter Indwelling catheter [MDS] (45628-5) Find Pt Patient Ord MDS Incident characteristic Injury level Event outcome [MERSTH] (42545-4) Find Pt Patient Ord Observed. MERSTH Frequency of injurious falls CCC, clinical care classification; MDS, Minimum Data Set; MERSTH, Medical Event Reporting System Total Healthsystem. Table 2. Example Nursing Assessment Data Elements Whose Concepts Mapped onto LOINC Care component Data element LOINC concepts (ID) Property Time System Scale Method Relation with populating indicator Reason for encounter Primary diagnosis Primary diagnosis (18630-4) Imp Pt Patient Nom — Inclusion criteria Administrative Length of stay Hospital stay duration (78033-8) Time Pt Patient Qn — Patient days Pathophysiologic disease-related factors Use of anticoagulants/coagulopathy diagnosis/the first day of surgical procedure Bleeding risk [CCC] (81225-5) Find Pt Cardiovascular system Ord Observed.CCC Frequency of nursing assessment of risk factors Nursing diagnosis: visual impairment Visual alteration [CCC] (28242-6) Find Pt Eye Ord Observed.CCC Cognitive factors Delirium Delirium [MDSv3] (54626-7) – Pt Patient – MDSv3 Consciousness Level of consciousness (80288-4) Find Pt Patient Ord – Behavior factors Fall history Fall history on admission [MDSv3] (54849-5) – Pt Patient – MDSv3 Therapeutic factors IV line Intravenous apparatus (59457-2) Find Pt Patient Ord Morse Fall Scale Foley catheter Indwelling catheter [MDS] (45628-5) Find Pt Patient Ord MDS Incident characteristic Injury level Event outcome [MERSTH] (42545-4) Find Pt Patient Ord Observed. MERSTH Frequency of injurious falls Care component Data element LOINC concepts (ID) Property Time System Scale Method Relation with populating indicator Reason for encounter Primary diagnosis Primary diagnosis (18630-4) Imp Pt Patient Nom — Inclusion criteria Administrative Length of stay Hospital stay duration (78033-8) Time Pt Patient Qn — Patient days Pathophysiologic disease-related factors Use of anticoagulants/coagulopathy diagnosis/the first day of surgical procedure Bleeding risk [CCC] (81225-5) Find Pt Cardiovascular system Ord Observed.CCC Frequency of nursing assessment of risk factors Nursing diagnosis: visual impairment Visual alteration [CCC] (28242-6) Find Pt Eye Ord Observed.CCC Cognitive factors Delirium Delirium [MDSv3] (54626-7) – Pt Patient – MDSv3 Consciousness Level of consciousness (80288-4) Find Pt Patient Ord – Behavior factors Fall history Fall history on admission [MDSv3] (54849-5) – Pt Patient – MDSv3 Therapeutic factors IV line Intravenous apparatus (59457-2) Find Pt Patient Ord Morse Fall Scale Foley catheter Indwelling catheter [MDS] (45628-5) Find Pt Patient Ord MDS Incident characteristic Injury level Event outcome [MERSTH] (42545-4) Find Pt Patient Ord Observed. MERSTH Frequency of injurious falls CCC, clinical care classification; MDS, Minimum Data Set; MERSTH, Medical Event Reporting System Total Healthsystem. Table 3. Example Nursing Intervention Data Elements Whose Concepts Mapped onto ICNP Care component Data element ICNP nursing intervention statement Code Relation with populating indicator Universal precaution Keep floor surfaces clean and dry Fall preventiona 10040211 Frequency of provision of universal precaution Make sure patient’s essential needs are within easy reach Fall preventiona 10040211 Communication Use visual indicator Fall preventiona 10040211 Frequency of provision of communication and education interventions Notify or take over risk information between clinicians Collaborating with interprofessional team 10039416 Observation and surveillance Use bed or chair alarm Maintaining fall safety alarm 10041525 Frequency of provision of observation and surveillance interventions Assess fall risks (using a tool) Assessing risk of falls 10023520 Counting use of fall risk tool Provision of protocol Provide safety aids Providing safety device 10024527 Frequency of provision of protocol interventions Provide scheduled assistance with toileting Assisting with toileting 10023531 Offer help using the toilet Provision of education Educate high-risk patient/family Teaching family about fall prevention 10040269 Frequency of provision of communication and education interventions Environmental intervention Rearrange room to clear paths Environment safety managementa 10042507 Frequency of provision of environmental interventions Adjust bed into locked position Secure raised toilet seat to commode Post-fall care provided Apply ice pack Applying cold pack 10036468 Frequency of nursing assessment of risk factors Perform neurologic assessment periodically Monitoring neurologic status 10035326 Notify physician and other nurses about a fall Collaborating with interprofessional teama 10039416 Care component Data element ICNP nursing intervention statement Code Relation with populating indicator Universal precaution Keep floor surfaces clean and dry Fall preventiona 10040211 Frequency of provision of universal precaution Make sure patient’s essential needs are within easy reach Fall preventiona 10040211 Communication Use visual indicator Fall preventiona 10040211 Frequency of provision of communication and education interventions Notify or take over risk information between clinicians Collaborating with interprofessional team 10039416 Observation and surveillance Use bed or chair alarm Maintaining fall safety alarm 10041525 Frequency of provision of observation and surveillance interventions Assess fall risks (using a tool) Assessing risk of falls 10023520 Counting use of fall risk tool Provision of protocol Provide safety aids Providing safety device 10024527 Frequency of provision of protocol interventions Provide scheduled assistance with toileting Assisting with toileting 10023531 Offer help using the toilet Provision of education Educate high-risk patient/family Teaching family about fall prevention 10040269 Frequency of provision of communication and education interventions Environmental intervention Rearrange room to clear paths Environment safety managementa 10042507 Frequency of provision of environmental interventions Adjust bed into locked position Secure raised toilet seat to commode Post-fall care provided Apply ice pack Applying cold pack 10036468 Frequency of nursing assessment of risk factors Perform neurologic assessment periodically Monitoring neurologic status 10035326 Notify physician and other nurses about a fall Collaborating with interprofessional teama 10039416 aMeans the not mapped category. Table 3. Example Nursing Intervention Data Elements Whose Concepts Mapped onto ICNP Care component Data element ICNP nursing intervention statement Code Relation with populating indicator Universal precaution Keep floor surfaces clean and dry Fall preventiona 10040211 Frequency of provision of universal precaution Make sure patient’s essential needs are within easy reach Fall preventiona 10040211 Communication Use visual indicator Fall preventiona 10040211 Frequency of provision of communication and education interventions Notify or take over risk information between clinicians Collaborating with interprofessional team 10039416 Observation and surveillance Use bed or chair alarm Maintaining fall safety alarm 10041525 Frequency of provision of observation and surveillance interventions Assess fall risks (using a tool) Assessing risk of falls 10023520 Counting use of fall risk tool Provision of protocol Provide safety aids Providing safety device 10024527 Frequency of provision of protocol interventions Provide scheduled assistance with toileting Assisting with toileting 10023531 Offer help using the toilet Provision of education Educate high-risk patient/family Teaching family about fall prevention 10040269 Frequency of provision of communication and education interventions Environmental intervention Rearrange room to clear paths Environment safety managementa 10042507 Frequency of provision of environmental interventions Adjust bed into locked position Secure raised toilet seat to commode Post-fall care provided Apply ice pack Applying cold pack 10036468 Frequency of nursing assessment of risk factors Perform neurologic assessment periodically Monitoring neurologic status 10035326 Notify physician and other nurses about a fall Collaborating with interprofessional teama 10039416 Care component Data element ICNP nursing intervention statement Code Relation with populating indicator Universal precaution Keep floor surfaces clean and dry Fall preventiona 10040211 Frequency of provision of universal precaution Make sure patient’s essential needs are within easy reach Fall preventiona 10040211 Communication Use visual indicator Fall preventiona 10040211 Frequency of provision of communication and education interventions Notify or take over risk information between clinicians Collaborating with interprofessional team 10039416 Observation and surveillance Use bed or chair alarm Maintaining fall safety alarm 10041525 Frequency of provision of observation and surveillance interventions Assess fall risks (using a tool) Assessing risk of falls 10023520 Counting use of fall risk tool Provision of protocol Provide safety aids Providing safety device 10024527 Frequency of provision of protocol interventions Provide scheduled assistance with toileting Assisting with toileting 10023531 Offer help using the toilet Provision of education Educate high-risk patient/family Teaching family about fall prevention 10040269 Frequency of provision of communication and education interventions Environmental intervention Rearrange room to clear paths Environment safety managementa 10042507 Frequency of provision of environmental interventions Adjust bed into locked position Secure raised toilet seat to commode Post-fall care provided Apply ice pack Applying cold pack 10036468 Frequency of nursing assessment of risk factors Perform neurologic assessment periodically Monitoring neurologic status 10035326 Notify physician and other nurses about a fall Collaborating with interprofessional teama 10039416 aMeans the not mapped category. The data model represented in Tables 2 and 3 includes 84 LOINC nursing assessment concepts and 22 ICNP intervention concepts. We mapped 43 (51.2%) and 15 (68.2%) of those to local data in 1 EHR, and 37 (44.0%) and 15 (68.2%) of those for the other EHR. We classified 5 of the 15 intervention mappings as suboptimal. Some data elements comprised several data item-value sets, such as fall risk assessment tools and regular rounding. An array of data items comprised the fall risk assessment tool: total score, sub-items and scores, and risk status. The regular rounding as a fall prevention intervention included several universal precaution activities, such as keeping hospital bed brakes locked, educating on use of call light, keeping the patient’s personal possessions within reach, and assessing the need for 1:1 monitoring. The fall prevention education provided on admission was also a composite intervention. Patients’ personal risk factors determined the other interventions provided. Populating Falls Indicators from EHR Data Data from 7829 (73 525 patient days) and 8199 (58 530 patient days) patients admitted during the research period from 2 hospitals met the inclusion criteria. Using the 2 data sets from EHR systems, 10 of the 11 indicators were successfully populated; the exception was the rate of falls with injury (see Table 4). The fall rates were computed 0.71 and 0.41 per 1000 patient days, respectively (see Table 2 for data elements used to populate the outcome indicator). The falls with injury rate was calculated at 0.27/1000 patient days at only one hospital. It was calculated from 2 data elements of the length of stay and the injury level. Coded nursing assessments, diagnoses and interventions recorded in each EHR system populated the 9 process indicators (see examples in Tables 2 and 3). Table 4. Comparison of Outcome and Process Indicators in EHR systems, Event Reporting Systems, and the Combination of These 2 Systems Outcome and process indicators on fall prevention Hospital A Hospital B EHRs Event report EHRs + event report EHRs Event report EHRs + event report Fall ratea (falls per 1000 patient days) 0.71 0.79 1.32 0.41 0.02 0.43 Injury fall ratea (falls with injury per 1000 patient days) NA 0.27 NA NA NA NA Patients assessed using a fall risk tool within 24 h of hospital admissionb (%) 88.29 NA 88.29 96.98 NA 96.98 Average frequency of using a fall risk assessment toolb 0.44 NA 0.44 0.55 NA 0.55 Patients with a nursing assessment of risk factors within 24 hb (%) 94.44 NA 94.44 89.9 NA 89.9 Patients with a daily nursing assessment of risk factorsb (%) 1.29 NA 1.29 1.0 NA 1.0 Patient days when there was a risk of falling (%) 16.97 NA 16.97 21.0 NA 21.0 At-risk patients to which universal precautions were applied within 24 h of risk identificationb (%) 99.78 NA 99.78 0.87 NA 0.87 At-risk patients to which communication and education interventions were applied within 24 h of risk identificationb (%) 99.22 NA 99.22 10.9 NA 10.9 At-risk patients to which observation and surveillance interventions were applied within 24 h of risk identificationb (%) 78.81 NA 78.81 0 NA 0 At-risk patients to which provision of protocol interventions were applied within 24 h of risk identificationb (%) 13.42 NA 13.42 6.0 NA 6.0 At-risk patients to which environmental interventions were applied within 24 h of risk identificationb (%) 92.75 NA 92.75 89.1 NA 89.1 Outcome and process indicators on fall prevention Hospital A Hospital B EHRs Event report EHRs + event report EHRs Event report EHRs + event report Fall ratea (falls per 1000 patient days) 0.71 0.79 1.32 0.41 0.02 0.43 Injury fall ratea (falls with injury per 1000 patient days) NA 0.27 NA NA NA NA Patients assessed using a fall risk tool within 24 h of hospital admissionb (%) 88.29 NA 88.29 96.98 NA 96.98 Average frequency of using a fall risk assessment toolb 0.44 NA 0.44 0.55 NA 0.55 Patients with a nursing assessment of risk factors within 24 hb (%) 94.44 NA 94.44 89.9 NA 89.9 Patients with a daily nursing assessment of risk factorsb (%) 1.29 NA 1.29 1.0 NA 1.0 Patient days when there was a risk of falling (%) 16.97 NA 16.97 21.0 NA 21.0 At-risk patients to which universal precautions were applied within 24 h of risk identificationb (%) 99.78 NA 99.78 0.87 NA 0.87 At-risk patients to which communication and education interventions were applied within 24 h of risk identificationb (%) 99.22 NA 99.22 10.9 NA 10.9 At-risk patients to which observation and surveillance interventions were applied within 24 h of risk identificationb (%) 78.81 NA 78.81 0 NA 0 At-risk patients to which provision of protocol interventions were applied within 24 h of risk identificationb (%) 13.42 NA 13.42 6.0 NA 6.0 At-risk patients to which environmental interventions were applied within 24 h of risk identificationb (%) 92.75 NA 92.75 89.1 NA 89.1 aOutcome indicators and bItalics process indicators. Abbreviation: NA, not available. Table 4. Comparison of Outcome and Process Indicators in EHR systems, Event Reporting Systems, and the Combination of These 2 Systems Outcome and process indicators on fall prevention Hospital A Hospital B EHRs Event report EHRs + event report EHRs Event report EHRs + event report Fall ratea (falls per 1000 patient days) 0.71 0.79 1.32 0.41 0.02 0.43 Injury fall ratea (falls with injury per 1000 patient days) NA 0.27 NA NA NA NA Patients assessed using a fall risk tool within 24 h of hospital admissionb (%) 88.29 NA 88.29 96.98 NA 96.98 Average frequency of using a fall risk assessment toolb 0.44 NA 0.44 0.55 NA 0.55 Patients with a nursing assessment of risk factors within 24 hb (%) 94.44 NA 94.44 89.9 NA 89.9 Patients with a daily nursing assessment of risk factorsb (%) 1.29 NA 1.29 1.0 NA 1.0 Patient days when there was a risk of falling (%) 16.97 NA 16.97 21.0 NA 21.0 At-risk patients to which universal precautions were applied within 24 h of risk identificationb (%) 99.78 NA 99.78 0.87 NA 0.87 At-risk patients to which communication and education interventions were applied within 24 h of risk identificationb (%) 99.22 NA 99.22 10.9 NA 10.9 At-risk patients to which observation and surveillance interventions were applied within 24 h of risk identificationb (%) 78.81 NA 78.81 0 NA 0 At-risk patients to which provision of protocol interventions were applied within 24 h of risk identificationb (%) 13.42 NA 13.42 6.0 NA 6.0 At-risk patients to which environmental interventions were applied within 24 h of risk identificationb (%) 92.75 NA 92.75 89.1 NA 89.1 Outcome and process indicators on fall prevention Hospital A Hospital B EHRs Event report EHRs + event report EHRs Event report EHRs + event report Fall ratea (falls per 1000 patient days) 0.71 0.79 1.32 0.41 0.02 0.43 Injury fall ratea (falls with injury per 1000 patient days) NA 0.27 NA NA NA NA Patients assessed using a fall risk tool within 24 h of hospital admissionb (%) 88.29 NA 88.29 96.98 NA 96.98 Average frequency of using a fall risk assessment toolb 0.44 NA 0.44 0.55 NA 0.55 Patients with a nursing assessment of risk factors within 24 hb (%) 94.44 NA 94.44 89.9 NA 89.9 Patients with a daily nursing assessment of risk factorsb (%) 1.29 NA 1.29 1.0 NA 1.0 Patient days when there was a risk of falling (%) 16.97 NA 16.97 21.0 NA 21.0 At-risk patients to which universal precautions were applied within 24 h of risk identificationb (%) 99.78 NA 99.78 0.87 NA 0.87 At-risk patients to which communication and education interventions were applied within 24 h of risk identificationb (%) 99.22 NA 99.22 10.9 NA 10.9 At-risk patients to which observation and surveillance interventions were applied within 24 h of risk identificationb (%) 78.81 NA 78.81 0 NA 0 At-risk patients to which provision of protocol interventions were applied within 24 h of risk identificationb (%) 13.42 NA 13.42 6.0 NA 6.0 At-risk patients to which environmental interventions were applied within 24 h of risk identificationb (%) 92.75 NA 92.75 89.1 NA 89.1 aOutcome indicators and bItalics process indicators. Abbreviation: NA, not available. Comparing the eMeasurements with Event Reporting Systems None of the process indicators and only the fall incident outcome indicator was available from the reporting data (see Table 4). The falls with injury rate were not always specified due to incomplete data reported in incident reports. The variables of Table 4 are outcome and process indicators, and a denominator for process indicators to be applied within 24 h of risk identification. We found patient falls recorded in the EHR systems, but not reported in the incident reporting systems. These cases corresponded to 39 (40.2%) out of 97 incidents for one hospital and 24 (96%) out of 25 incidents for the other hospital. Through the chart review, we found 45 (46.4%) and 1 (4%) cases recorded in the incident reporting system, but not recorded in the EHR systems. Figure 4 illustrates the summary of these results using Venn diagrams. Figure 4. View largeDownload slide Comparison of falls detection results in EHRs and event reporting systems. Figure 4. View largeDownload slide Comparison of falls detection results in EHRs and event reporting systems. When we combined the fall incidents from EHRs and event reporting systems, the fall rates were 1.32 and 0.43, respectively. Discussion In the present study, we successfully populated nursing quality indicators from EHR systems in a format comparable with those used at other facilities. This was possible due to the availability of standard fall prevention content, standard terminologies, and a mapping process completed by experienced informaticians and nursing experts. Despite the challenges in mapping some nursing interventions to ICNP, we successfully autopopulated all the process indicators. In addition, using EHR data we detected fall events that were not available in the event reporting systems. LOINC had excellent content coverage for the 10 nursing assessment care components needed to populate the fall prevention indicators. The care components contain data elements that were measurements or observations captured in nursing assessments. This result demonstrates higher concept coverage than found in an earlier study28 which mapped 294 concepts extracted from nursing flowsheet documents of 6 organizations onto SNOMED CT and LOINC. Here, they reported mapping rates from 70% to 82%. Some of the data in our study comprised nursing judgments in diagnoses such as visual alteration, auditory alteration, and physical mobility impairment. These concepts mapped to precoordinated concepts from nursing terminology or assessment tools such as the Clinical Care Classification29 or the Morse Fall Scale30; both of which are incorporated in LOINC. Our findings imply that LOINC includes acceptable content coverage of nursing judgments (e.g., findings) related to falls prevention. LOINC is a catalog of measurements including laboratory tests, clinical measures such as vital signs, anthropomorphic measures, and standardized survey instruments. This finding suggests that LOINC can also be used as a standard terminology to represent fall prevention nursing assessment and observation concepts.31 As for intervention mapping, a relatively small number of ICNP intervention statements matched the nursing intervention data elements. Most universal precaution and environmental intervention care components were not mapped to ICNP interventions. The root cause is the low levels of granularity at which the precoordinated intervention statements in ICNP are aggregated. For example, the intervention statements of sharing/communication high risk patients between staff, use of visual indicator, keep family sitter, and communicate patient’s history, risk factor, treatment, and plan all corresponded to the single concept, collaborating with interprofessional team. Other statements imply various sets of activities that a nurse could provide to patients, such as assessing risk for falls on admission, fall prevention, and post-fall evaluation. The expressions do not provide detailed information about the kinds of activities planned and provided in practice; but rather as an abstract representation of relevant activities. The ICNP’s level of abstraction restricts the accuracy of concept mapping and secondary use to represent practice, quality reporting and research. However, ICNP has current mappings to SNOMED CT and this hierarchy could be leveraged to provide more detailed child concepts.32 Since the ICNP version 2 release, the ICNP terminology team has developed terminology subsets that are both clinically relevant and user-friendly. They encourage the development and maintenance of ICNP catalogs or subsets as part of the ICNP terminology life cycle, if such catalogs will enhance the usability of the terminology in practice. The ICN has reported that a pressure ulcer catalog is under development for the patient safety domain. However, there is no explicit fall prevention catalog to represent activities to prevent patient falls, even though inpatient falls are widespread, a common nursing-sensitive outcome, and a serious threat to patient safety. Our research team previously notified the ICNP terminology team of the importance of an inpatient fall prevention catalog and commenced its development using the conceptual ICPS framework.10 Unlike the event reporting systems, our EHR-based approach allowed us to populate process indicators as well as outcome indicators across multiple time periods. Using the existing event reporting system, we could extract outcome indicators over a relatively limited period. The incident reports did not have complete nursing process data. Considering the utility of process indicators in the short term for improving the quality of care6 and the fact that inpatient falls are relatively rare event (e.g., occurring to 0.05%–0.3% patients33,34), it is a great advantage that EHRs are able to compute process indicators. Process indicators have more power to detect real differences in the quality of nursing care provided each day. The ability to compute process indicators from EHR data supports the feasibility of using eMeasures for internal quality improvement activities, as well as for national reporting.2 The eMeasure approach has the potential to reduce the nursing management burden associated with manual reporting and increase efficiency in monitoring. One unexpected finding of the present study was the detection of new fall incidents from EHR data that had not been reported via the event reporting systems. The validation process detected several false-positive cases, but true-positive cases increased the percent of falls identified at the 2 hospitals by 40% and 96%, respectively. This finding confirmed empirically the under-reporting of safety incidents in the event reporting systems.35,36 This discovery implies that the use of EHRs in outcome measurement offers a new opportunity to address under-reporting. To achieve more accurate patient falls outcome reporting, we may need to redesign the reporting process. Use of EHR data to supplement event reporting system data is one option. However, omissions of fall event recording in EHRs is not straightforward. Some nurses may forget to document a fall. Another possibility is that nurses avoid entering the same content into 2 different systems (e.g., double documenting). A study37,38 investigating the accuracy of nursing notes in 6 EHR systems using a time-motion method, reported 85%–92% matching rates between the activities observed and recorded. This report supports the possibility of the later. Redesigning the incident reporting process to include reporting incidents solely in the EHR system, could lead to more complete data capture. One of the strengths of this study is that the participating hospitals independently implemented EHR systems that cover all inpatient nursing documentation. These systems have been in place for >8 years. Our analysis relied on the hospital CDRs, not a data mart. This made it possible to access any form of data (e.g., structured, semi-structured, and free text) without missing any data included in the live EHR systems. Another strength is the fact that we populated eMeasures using semi-structured data representing nursing process information. To our knowledge, this was not done previously. One study39 analyzed flowsheet data documented in multiple EHR systems and created 14 nursing information models based on clinical topics. It defined a flowsheet as a format for facilitating structured data capture and as a template that can be specific to a unit or discipline. They found that flowsheets contain groups of related assessments and interventions, and individual items for actual documentation. Another study40 analyzed flowsheet data of 2 million patients and developed an ontology to support secondary data use. However, the study is based on data documented at a single site in an Epic Systems Corporation EHR (single vendor). One limitation of this study is the generalizability of our findings to other measures and hospitals. However, considering the characteristics of inpatient falls, we believe that the data elements and processing methods will apply to other preventable adverse event quality measures such as pressure ulcers, pain management, and catheter-associated urinary tract infections. Based on ANA’s criteria for selecting quality indicators for conversion to eMeasures, inpatient falls is a strong candidate. The generalizability of our results to other care delivery systems and EHR systems is unknown. However, the approach to extract process and outcome indicators in a standardized and comparable format can be leveraged with any EHR system. Given that most EHR systems are not based on standards that enable semantic interoperability, the approach described could contribute to the benchmarking of nursing performance across sites and settings. Conclusions This study demonstrated the data from EHR environments can generate fall prevention process measures and at least one outcome metric. This supports sharing and comparing nursing-sensitive process and outcome metrics for multiple purposes, including the secondary use of a clinical nursing data set. Although the content and use of standards in EHR systems are incomplete and based on highly variable user preferences, standard nursing terminologies and content can be used to address existing gaps and EHR data can contribute to nursing quality improvement. Competing interests None. Funding This study was supported by a grant of the Korea Healthcare Technology R&D Project, Ministry for Health, Welfare & Family Affairs, Republic of Korea (No. HI15C1089). IC was also supported by a grant from Inha University (No. 50463-01) and the Hospital Nurses Association of Korean Nurses Association in 2015. Contributors IC conceived and designed the study, supervised, and contributed to the data analysis, interpreted results, and drafted and revised the paper. EHB and SYL contributed to study design, data acquisition, and data analysis. PCD provided substantial contribution to the interpretation of data analysis and revised the work critically for important intellectual content. Supplementary Material Supplementary material is available at Journal of the American Medical Informatics Association online. ACKNOWLEDGMENTS We thank Yeoun-Sook Lee, Jungmee Han, and Won-Hee Park of the Medical Information Departments of the 2 hospitals for helping us to retrieve the raw data efficiently. We also appreciate the clinical staffs of the Nursing Departments and graduate students involved in qualitative data processing, literature review, and discussions. References 1 Westra BL , Latimer GE , Matney SA et al. , A national action plan for sharable and comparable nursing data to support practice and translational research for transforming health care . J Am Med Inform Assoc 2015 ; 22 3 : 600 – 607 . Google Scholar PubMed 2 Amster A , Jentzsch J , Pasupuleti H , Subramanian K . Completeness, accuracy, and computability of National Quality Forum-specified eMeasures . J Am Med Inform Assoc 2015 ; 22 2 : 409 – 416 . Google Scholar PubMed 3 Cho I , Choi W , Choi WH . Adoption of electronic nursing records and standard nursing terminology in acute-care practices: findings from a Nationwide Survey . In: Saranto K , Weaver CA , Chang P , ed. Nursing Informatics 2014: Proceedings of the 12th International Congress on Nursing Informatics . Taipei, Taiwan : IOS Press ; 2014 . 4 Warren JJ , Dunton N . eMeasure Development Process Examplar . Silver Spring, MD: American Nurses Association ; 2015 . 5 Boushon B , Nielsen G , Quigley P et al. , How-to Guide: Reducing Patient Injuries From Falls . Cambridge, MA : Institute for Healthcare Improvement ; 2013 . 6 Mant J . Process versus outcome indicators in the assessment of quality of health care . Int J Qual Health Care 2001 ; 13 6 : 475 – 480 . Google Scholar CrossRef Search ADS PubMed 7 Idvall E , Rooke L , Hamrin E . Quality indicators in clinical nursing: a review of the literature . J Adv Nur 1997 ; 25 1 : 6 – 17 . Google Scholar CrossRef Search ADS 8 World Health Organization . Conceptual Framework for the International Classification for Patient Safety . Vol. 2009 . Geneva: World Health Organization ; 2009 : 1 – 149 . 9 Cho I , Chae J , Kim J , Jung M , Kim Y , Yoo A . Development of ICNP-based inpatient fall prevention catalogue . Int Nurs Rev 2017 (Under submission) . 10 Jung M , Cho I , Kim J , Chae J . Comparing inpatient falls guidelines to develop an ICNP-based nursing catalogue for ENRs . Paper presented at: Medinfo 2017 . Hangzhou, China ; 2017 . 11 American Nurses Association . Nursing Care Report Card for Acute Care . Silver Spring, MD: American Nurses Publishing ; 1995 . 12 Montalvo I . The National Database of Nursing Quality Indicators™ (NDNQI®) . Online J Issues Nurs 2007 ; 12 3 : 112–214. 13 Aydin CE , Bolton LB , Donaldson N et al. , Creating and analyzing a statewide nursing quality measurement database . Image J Nurs Sch 2004 ; 36 4 : 371 – 378 . Google Scholar CrossRef Search ADS 14 Patrician PA , Loan L , McCarthy M , Brosch LR , Davey KS . Towards evidence-based management: Creating an informative database of nursing-sensitive indicators . J Nurs Sch 2010 ; 42 4 : 358 – 366 . Google Scholar CrossRef Search ADS 15 Veteran's Health Affairs . Office of Nursing Services (ONS) Annual Report 2009 . 2009 . https://www.va.gov/NURSING/docs/OfficeofNursingServices-ONS_Annual_Report_2009-WEB.pdf. Accessed January 10, 2017. 16 The Joint Commission . Sentinel event alert 55: Preventing falls and fall-related injuries in health care facilities . Sentinel Event Alert . https://www.jointcommission.org/sea_issue_55/. Accessed March 10, 2018. 17 Ganz DA , Huang C , Saliba D , Shier V . Preventing Falls in Hospitals. Content last reviewed January 2013. Agency for Healthcare Research and Quality, Rockville, MD. http://www.ahrq.gov/professionals/systems/hospital/fallpxtoolkit/index.html. Accessed March 10, 2018. 18 ECRI Institute . Falls . 2009 . https://www.ecri.org/components/HRC/Pages/SafSec2.aspx. Accessed March, 2018. 19 Degelau J BM , Bungum L , Flavin PL et al. , Prevention of Falls (Acute Care) . 2012 . https://www.icsi.org/guidelines__more/catalog_guidelines_and_more/catalog_guidelines/catalog_patient_safetyreliability_guidelines/falls/. Accessed March, 2017. 20 The Joint Commission Center for Transforming Healthcare . New! Targeted Solutions Tool for Preventing Falls . 2016 . http://www.centerfortransforminghealthcare.org/tst_pfi.aspx. Accessed March, 2018. 21 U.S. Department of Veterans Affairs . VA National Center for Patient Safety: Falls Toolkit . 2015 . http://www.patientsafety.va.gov/professionals/onthejob/falls.asp. Accessed March, 2018. 22 Neily J , Quigley PA , Essen K . Implementation Guide for Fall Injury Reduction: VA National Center for Patient Safety Reducing Preventable Falls and Fall-Related Injuries . 2015 . http://www.patientsafety.va.gov/docs/fallstoolkit14/falls_implementation_%20guide%20_02_2015.pdf. Accessed March, 2018. 23 Korean Hospital Nurses Association . Nursing Practice Guidelines of Prevention of Inpatient Falls . 2008 . http://khna.or.kr/bbs/linkfile/resource/khna_Fcare.pdf. Accessed October 10, 2017. 24 American Nurses Association . Guidelines for data collection on the American Nurses Association's national quality forum endorsed measures: Nursing care hours per patient day; skill-mix; falls; falls with injury . KU School of Nursing, Kansas City . 2010 . 25 Cho I , Park HA . Evaluation of the expressiveness of an ICNP-based nursing data dictionary in a computerized nursing record system . J Am Med Inform Assoc 2006 ; 13 : 456 – 464 . Google Scholar CrossRef Search ADS PubMed 26 Matney S , Bakken S , Huff SM . Representing nursing assessments in clinical information systems using the logical observation identifiers, names, and codes database . J Biomed inform 2003 ; 36 4 : 287 – 293 . Google Scholar CrossRef Search ADS PubMed 27 Dykes P , Collins S . Building linkages between nursing care and improved patient outcomes: the role of health information technology . Online J Issues Nurs 2013 ; 18 3 : Manuscript 4. 28 Harris MR LL , Miller H , Hook M , Dykes PC , Matney SA . Harmonizing and extending standards from a domain-specific and bottom-up approach: an example from development through use in clinical applications . J Am Med Inform Assoc 2015 ; 22 3 : 545 – 552 . Google Scholar PubMed 29 Saba V . Clinical Care Classification (CCC) System Manual: a Guide to Nursing Documentation . New York: Springer Publishing Company ; 2006 . 30 Morse JM , Morse RM , Tylko SJ . Development of a scale to identify the fall-prone patient . Can J Aging/La Revue canadienne du vieillissement 1989 ; 8 4 : 366 – 377 . Google Scholar CrossRef Search ADS 31 Vreeman DJ , McDonald CJ , Huff SM . LOINC®: a universal catalogue of individual clinical observations and uniform representation of enumerated collections . Int J Funct Inform Personalised Med 2010 ; 3 4 : 273 – 291 . Google Scholar CrossRef Search ADS 32 Dykes PC , Wantland D , Whittenburg L , Lipsitz S , Saba VK . A pilot study to explore the feasibility of using the Clinical Care Classification System for developing a reliable costing method for nursing services . AMIA Annu Symp Proc. 2013 ; 2013 : 364 – 71 . Google Scholar PubMed 33 Staggs VS , Davidson J , Dunton N , Crosser B . Challenges in defining and categorizing falls on diverse unit types: Lessons from expansion of the NDNQI falls indicator . J Nurs Care Qual 2015 ; 30 2 : 106 – 112 . Google Scholar CrossRef Search ADS PubMed 34 Park IS , Cho I , Kim EM , Kim MK . Evaluation of a fall risk assessment tool to establish continuous quality improvement process for inpatients' falls . J Korean Acad Nurs Admin 2011 ; 17 4 : 484 – 492 . Google Scholar CrossRef Search ADS 35 Sari AB-A , Sheldon TA , Cracknell A , Turnbull A . Sensitivity of routine system for reporting patient safety incidents in an NHS hospital: retrospective patient case note review . BMJ 2007 ; 334 7584 : 79 . Google Scholar CrossRef Search ADS PubMed 36 Oliver D , Healey F , Haines TP . Preventing falls and fall-related injuries in hospitals . Clin Geri Med 2010 ; 26 4 : 645 – 692 . Google Scholar CrossRef Search ADS 37 Cho I , Kim M-K . Consistency analysis between electronic nursing records' data and observed direct nursing activities at multiple inpatient settings . 2017 Korean Society of Medical Informatics Spring Conference . Incheon, Republic of Korea : KOSMI ; 2017 . 38 Cho I . Analysis on functionality, usability, and use patterns of electronic nursing record systems: a multi-institutional time-motion approach . Seoul, Republic of Korea: Korean Hospital Nurses Association Research Report . 2014 January 17 . 39 Westra BL , Christie BA , Byrne MD et al. , Modeling flowsheet data for quality improvement and research . Paper presented at: Summer Institute in Nursing Informatics . Maryland, USA ; 2015 . 40 Johnson SG , Byrne MD , Christie B et al. , Modeling flowsheet data for clinical research . AMIA Summits Transl Sci Proc 2015 ; 2015 : 77 – 81 . Google Scholar PubMed © The Author(s) 2018. Published by Oxford University Press on behalf of the American Medical Informatics Association. All rights reserved. For permissions, please email: journals.permissions@oup.com This article is published and distributed under the terms of the Oxford University Press, Standard Journals Publication Model (https://academic.oup.com/journals/pages/about_us/legal/notices) http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png Journal of the American Medical Informatics Association Oxford University Press

Automatic population of eMeasurements from EHR systems for inpatient falls

Loading next page...
 
/lp/oxford-university-press/automatic-population-of-emeasurements-from-ehr-systems-for-inpatient-hxVgb62S00
Publisher
Oxford University Press
Copyright
© The Author(s) 2018. Published by Oxford University Press on behalf of the American Medical Informatics Association. All rights reserved. For permissions, please email: journals.permissions@oup.com
ISSN
1067-5027
eISSN
1527-974X
DOI
10.1093/jamia/ocy018
Publisher site
See Article on Publisher Site

Abstract

Abstract Objective Representing nursing data sets in a standard way will help to facilitate sharing relevant information across settings. We aimed to populate nursing process and outcome metrics with electronic health record (EHR) data and then compare the results with event reporting systems. Methods We used the “eMeasure” development process of the National Quality Forum adopted by the American Nurses Association. We used operational definitions of quality measures from the American Nurses Association and the US Institute for Healthcare Improvement and employed concept mapping of local data elements to 2 controlled vocabularies to define a standard data dictionary: (1) Logical Observation Identifiers Names and Codes and (2) International Classification for Nursing Practice. We assessed feasibility using the nursing data set of 7829 and 8199 patients from 2 general hospitals with different EHR systems. Using inpatient falls as a use case, we compared the populated measures with results from the event reporting systems. Results We identified 17 care components and 118 unique concepts and matched them with data elements in the EHRs. Including suboptimal mapping, 98% of the assessment concepts mapped to Logical Observation Identifiers Names and Codes and 52.9% of intervention concepts mapped to International Classification for Nursing Practice. While not all process indicators were available from event reporting systems, we successfully populated 9 fall prevention process indicators and the fall rate outcome indicator from the 2 EHRs. We were unable to populate the falls with an injury rate indicator. Conclusions EHR data can populate fall prevention process measure metrics and at least one inpatient fall prevention outcome metric. Introduction The rapid implementation of electronic health records (EHRs) and the integration of EHR data into clinical data repositories (CDRs) means that large quantities of clinical data are now available. Combining these data with computational technologies shows promise in improving nursing knowledge representation and patient outcomes.1 In an effort to align quality indicators with codified clinical and billing data, the US Centers for Medicare and Medicaid Services converted 113 National Quality Forum (NQF)-endorsed Clinical Quality Measures from a paper-based format relying on chart abstraction into an electronic “eMeasure” format.2 For nursing-sensitive quality indicators, little is known about the feasibility of using nursing data in EHRs to populate the eMeasure format as well as for benchmarking across disparate EHRs. Nonstandardized data are used in most EHR systems due to the lack of knowledge regarding standardized processes and data among the nursing executive and clinician workforce.1 However, multiple years of experience with EHR systems creates an opportunity to explore ways to standardize locally developed nursing vocabularies. Our previous nationwide survey of EHR systems conducted in 2012 revealed that more than half of general hospitals in Korea implemented a local nursing vocabulary.3 We hypothesized that if nursing data in these local systems were represented in a standard way, they could be shared and compared on regional and national levels and would facilitate the identification of trends and outcomes. Here we describe the process of transforming local nursing data into a standardized format to populate fall prevention quality measures, and demonstrate the feasibility of this method within 2 disparate EHR environments. We also compared this approach with the existing manual approach using standard event reporting systems. Methods Site and Settings Two hospitals located in metropolitan areas of Korea with different locally developed EHR systems participated in this study. One was a tertiary academic hospital with 900 beds and the other was a secondary hospital with 800 beds. Both hospitals used EHR systems for >8 years that included nursing vocabularies with both local terms and nursing statements that were based on the International Classification for Nursing Practice (ICNP) and North American Nursing Diagnosis Association, Nursing Intervention Classification, and Nursing Outcomes Classification (3N) controlled vocabularies. Identification of Outcome Metrics for Inpatient Falls The eMeasure development process of the NQF adopted by the American Nurses Association (ANA) and the following National Database of Nursing Quality Indicator outcome metrics for patient falls were used to aggregate local data and to populate quality measures: patient fall rate and patient falls with injury rate.4 For process metrics, we used the Institute for Healthcare Improvement definitions.5 Process indicators measure compliance with key care components of evidence-based nursing. Improvements in the scores for individual process measures indicate that the processes underlying these care elements have improved, and improvements in patient outcomes should follow. Mant6 advised using process metrics to measure the quality of health care if linkages exist between given processes of care and patient outcomes. Where such measures are available, relevant, and practical, they are used in conjunction with outcome metrics since they are sensitive to differences in the quality of care in the short term; typically, before outcome measures are available.6,7 Identification of Standard Content of Fall Related Care We formed a project team consisting of experts in the following relevant domains: nursing informatics, terminology, quality management, and patient safety. Two College of Nursing professors, 4 graduate students with 5–10 years of practical experience at university hospitals, nursing staff, and managers of the sites were involved. We used the International Classification for Patient Safety’s (ICPS) conceptual framework to identify patient safety concepts for falls.8 The process of how standard content was identified from a literature review, and arranged into key care components under the ICPS framework, is described in detail elsewhere.9,10 The aim of the literature review was to identify evidence-based nursing content related to falls that could be mapped to nursing documentation in the EHR systems through standard terminologies. We identified 17 key care components corresponding to fall prevention and post-fall nursing care. We also searched national and regional nurse-sensitive quality databases with the aim of supporting quality assessment and demonstrating the nursing contribution to patient care using measures similar to those proposed in the ANA initiative in 199511: National Database of Nursing Quality Indicator,12 California Nursing Outcomes Coalition,13 Military Nursing Outcomes Database,14 and VA Administration Nursing Outcomes Database.15 Data Collection We performed the following 4 data collection steps: (1) review of practice guidelines and literature to identify fall prevention key care components, process, and outcome measures; (2) data mapping from local nursing vocabularies to standardized terminologies; (3) populating eMeasures with standardized data; and (4) comparing an eMeasurement system with an event reporting system in terms of its ability to generate process and outcome metrics. Figure 1 shows the relationships between these steps. Figure 1. View largeDownload slide The 4 steps of this study procedure. LOINC: Logical Observation Identifiers Names and Codes; ICNP: International Classification for Nursing Practice. Figure 1. View largeDownload slide The 4 steps of this study procedure. LOINC: Logical Observation Identifiers Names and Codes; ICNP: International Classification for Nursing Practice. Identifying fall prevention key care components and metrics We selected the 7 guidelines or tools recommended by The Joint Commission in 2015 and the Korean Hospital Nurses Association Nursing Practice Guidelines to identify key care components and fall prevention measures.16,22Figure 2 displays the list of resources. Figure 2. Inpatient fall prevention practice guidelines and tools reviewed. Agency for Healthcare Research and Quality Fall Prevention Toolkit17 ECRI Institute Falls Prevention Guidance18 Institute for Clinical Systems Improvement Prevention of Falls (Acute Care) Protocol19 Institute for Healthcare Improvement's (IHI) Transforming Care At The Bedside How-to Guide5 The Joint Commission Center for Transforming Healthcare's Targeted Solutions Tool (TST®) for Preventing Falls20 Veterans Affairs. VA National Center for Patient Safety: Falls Toolkit21 Veterans Affair (VA) National Center for Patient Safety Implementation Guide For Fall Injury Reduction22 Korean Hospital Nurses Association Nursing Practice Guidelines of Prevention of Inpatient Falls23 Figure 2. Inpatient fall prevention practice guidelines and tools reviewed. Agency for Healthcare Research and Quality Fall Prevention Toolkit17 ECRI Institute Falls Prevention Guidance18 Institute for Clinical Systems Improvement Prevention of Falls (Acute Care) Protocol19 Institute for Healthcare Improvement's (IHI) Transforming Care At The Bedside How-to Guide5 The Joint Commission Center for Transforming Healthcare's Targeted Solutions Tool (TST®) for Preventing Falls20 Veterans Affairs. VA National Center for Patient Safety: Falls Toolkit21 Veterans Affair (VA) National Center for Patient Safety Implementation Guide For Fall Injury Reduction22 Korean Hospital Nurses Association Nursing Practice Guidelines of Prevention of Inpatient Falls23 Outcome and process metrics and the corresponding care components to be extracted from each EHR system were vetted with inpatient nursing teams. The calculation of outcome indicator logic followed the ANA/National Quality Forum endorsed guidelines for data collection.24 Data mapping to standardized nursing terminologies The standard fall prevention content consisted of nursing assessment, diagnosis, and intervention activities. We refer to nursing data as the data set in EHR systems including 3 broad categories of elements: (1) nursing care; (2) patient or client demographics; and (3) service elements. Data elements are the atomic units of nursing data. For example, when evaluating cognitive function, specifically delirium, several descriptive terms exist in standard content or EHR systems including altered mental status, confusional state, and delirium. We designate delirium as one data element. We mapped data elements to standard terminology concepts from Logical Observation Identifiers Names and Codes (LOINC) for administrative, assessments, and diagnoses, and ICNP for interventions. The Health and Welfare Ministry of Korea endorses LOINC and ICNP as national healthcare standards. We used Regenstrief LOINC Mapping Assistant (RELMA, LOINC browser) and the ICNP Browser at the International Council of Nursing. Two graduate students with 5 and 10 years of clinical experience conducted the initial mapping, then the authors (IC and EHB) reviewed and discussed further to classify the mapping results into 3 categories of optimal (exact match), suboptimal (partial match), and not mapped (no match). We used the concept ontology of the Systematized Nomenclature of Medicine-Clinical Terminology (SNOMED-CT) as a sematic map indicating a type of graphic organizer that visually represented relationships among categories of concepts. We also used the 8 guidelines of preventing falls for assuring the relationships among concepts in the context of fall prevention. We classified the cases into the optimal mapping, in which a data element completely corresponding to a concept semantically or in relation to an “is a” (parent or child relation) with a concept. We classified discrepancies or disagreements on semantics between reviewers as suboptimal.25,26 The data dictionary for each hospital guided the collection and normalization of data from multiple sources: the EHR systems, reporting systems, and administrative databases. Table 1 shows the scheme of the data dictionary and data examples. The EHR data included both structured and semi-structured formats. We coded nursing statements in nursing notes in a semi-structured format. Table 1. Scheme of Inpatient Fall Data Dictionary and Data Examples Field Example of assessment Example of intervention Care component Cognitive factors Impaired mobility Care type Assessment Intervention Care activity Observe (assess or evaluate) delirium/confusional state/altered mental status/ reversible dementia/organic brain syndrome Regularly scheduled assistance with toileting Data element Delirium Aid of toileting Definition Confusional state Regularly scheduled assistance with toileting for patients with impaired mobility Terminology LOINC ICNP Intervention Concept(s) Delirium [MDSv3] Assisting with toileting Terminology code 54626-7 10023531 Local facility ID Hospital A Hospital B Local EHR ID.version eChart.2.0 SmartCare.3.0 Local screen-item name Nursing notes Nursing notes Local expression Observed delirium symptom(degreea: disorganized) Provide regularly assistance with toileting Local data type Assessment statement Intervention statement Local code S03745 A05595 Field Example of assessment Example of intervention Care component Cognitive factors Impaired mobility Care type Assessment Intervention Care activity Observe (assess or evaluate) delirium/confusional state/altered mental status/ reversible dementia/organic brain syndrome Regularly scheduled assistance with toileting Data element Delirium Aid of toileting Definition Confusional state Regularly scheduled assistance with toileting for patients with impaired mobility Terminology LOINC ICNP Intervention Concept(s) Delirium [MDSv3] Assisting with toileting Terminology code 54626-7 10023531 Local facility ID Hospital A Hospital B Local EHR ID.version eChart.2.0 SmartCare.3.0 Local screen-item name Nursing notes Nursing notes Local expression Observed delirium symptom(degreea: disorganized) Provide regularly assistance with toileting Local data type Assessment statement Intervention statement Local code S03745 A05595 aAttribute and value of the nursing statement. Table 1. Scheme of Inpatient Fall Data Dictionary and Data Examples Field Example of assessment Example of intervention Care component Cognitive factors Impaired mobility Care type Assessment Intervention Care activity Observe (assess or evaluate) delirium/confusional state/altered mental status/ reversible dementia/organic brain syndrome Regularly scheduled assistance with toileting Data element Delirium Aid of toileting Definition Confusional state Regularly scheduled assistance with toileting for patients with impaired mobility Terminology LOINC ICNP Intervention Concept(s) Delirium [MDSv3] Assisting with toileting Terminology code 54626-7 10023531 Local facility ID Hospital A Hospital B Local EHR ID.version eChart.2.0 SmartCare.3.0 Local screen-item name Nursing notes Nursing notes Local expression Observed delirium symptom(degreea: disorganized) Provide regularly assistance with toileting Local data type Assessment statement Intervention statement Local code S03745 A05595 Field Example of assessment Example of intervention Care component Cognitive factors Impaired mobility Care type Assessment Intervention Care activity Observe (assess or evaluate) delirium/confusional state/altered mental status/ reversible dementia/organic brain syndrome Regularly scheduled assistance with toileting Data element Delirium Aid of toileting Definition Confusional state Regularly scheduled assistance with toileting for patients with impaired mobility Terminology LOINC ICNP Intervention Concept(s) Delirium [MDSv3] Assisting with toileting Terminology code 54626-7 10023531 Local facility ID Hospital A Hospital B Local EHR ID.version eChart.2.0 SmartCare.3.0 Local screen-item name Nursing notes Nursing notes Local expression Observed delirium symptom(degreea: disorganized) Provide regularly assistance with toileting Local data type Assessment statement Intervention statement Local code S03745 A05595 aAttribute and value of the nursing statement. Populating falls indicators from EHR data Based on the data dictionary, we retrieved and collected data from 4 medical-surgical nursing units of each hospital over one calendar year (2014). We used the operational patient selection criteria of NQF measures #0141 (patient fall rate) and #0202 (patient falls with injury) for the targeted medical-surgical units. We included: (1) adult inpatients aged 18 years and older and (2) patients who had been admitted for at least 24 h. We excluded patients with a primary pediatric, psychiatric, or obstetrics diagnosis, or who received resuscitation treatment, or died. We included only patient stays on targeted units during the study period. Comparing the eMeasurements with event reporting systems The 2 hospitals independently submitted data from their locally developed event reporting systems. Data consisted of patient demographics, admission information, situational descriptions about falls, post-fall measures, and management information used for root-cause analysis. We compared the completeness of indicators populated with manual event reporting data with indicators populated directly from the EHR systems. We also used these data to cross-check cases recorded in the EHR system and the event reporting system. Three authors (IC, EHB, and SYL) conducted chart reviews to identify fall events hospital managers confirmed new cases. Data Analysis The Institutional Review Board at each hospital approved this research project. Each hospital provided de-identified data from the clinical data repository (CDR). To identify the target patients, we applied operational criteria that were modified from 2 ANA endorsed nursing-sensitive outcomes; total falls per 1000 patient days for fall rate, and injury falls per 1000 patient days for injury fall rate.24,27 We expressed process indicators as proportions or average frequencies, such as the number of nursing activities conducted to assess a patient’s disease-related factors, cognitive factors, behavior factors, or therapeutics during the days when the patient was at risk of falling. Results Identifying Fall Prevention Key Care Components and Measures Figure 3, the Fall Prevention Indicator Map, includes the key fall prevention care components and measures identified through literature and guideline review. The indicator map represents the structural relationships between fall prevention key care components and indicators. The left side of the figure displays the parts of the care components used to calculate the indicators that appear on the right side of the figure. The figure includes 2 outcome indicators and 9 process indicators. Figure 3. View largeDownload slide Indicator map of inpatient falls (aoutcome indicators, bprocess indicators). Figure 3. View largeDownload slide Indicator map of inpatient falls (aoutcome indicators, bprocess indicators). Data Mapping to Standardized Nursing Terminologies The concepts of nursing assessments and fall incident categories were all successfully mapped to LOINC except the concept of injury level in incident characteristics. According to its definition, fall injury level has 4 categories in its value list: none, minor, moderate, and major. However, we mapped this concept to the LOINC concept event outcome of the Medical Event Reporting System—Total HealthSystem taxonomy, for which the value list was 4 categories: no recovery with harm, no recovery with no harm, near miss with unplanned recovery, and near miss with planned recovery. The post-fall assessment concepts mapped to LOINC except the following 2 concepts: absolute bed rest state and close observation state. A total of 52.9% of nursing intervention concepts mapped to ICNP (including suboptimal mappings). For example, we successfully mapped the concepts observation and surveillance, provision of protocol, and post-fall care provided to 18 intervention statements in ICNP. However, we could not find ICNP interventions on the nursing activities, communication and education such as sharing high risk patients and use of visual fall risk indicator. The closest interventions were the collaborating with interprofessional team and fall prevention in ICNP. We also found almost all of the universal precaution nursing activities and environmental interventions were related to the ICNP interventions, fall prevention and environmental safety management. However, the ICNP interventions were too implicit and abstract. We classified those into the not mapped. Tables 2 and 3, respectively, present examples of the mapping results for LOINC and ICNP intervention statements. Each data element is used to apply the inclusion or exclusion criteria, compute denominators, or aggregate by the care component. The full version of mapping list is online in Supplementary Material S1. Table 2. Example Nursing Assessment Data Elements Whose Concepts Mapped onto LOINC Care component Data element LOINC concepts (ID) Property Time System Scale Method Relation with populating indicator Reason for encounter Primary diagnosis Primary diagnosis (18630-4) Imp Pt Patient Nom — Inclusion criteria Administrative Length of stay Hospital stay duration (78033-8) Time Pt Patient Qn — Patient days Pathophysiologic disease-related factors Use of anticoagulants/coagulopathy diagnosis/the first day of surgical procedure Bleeding risk [CCC] (81225-5) Find Pt Cardiovascular system Ord Observed.CCC Frequency of nursing assessment of risk factors Nursing diagnosis: visual impairment Visual alteration [CCC] (28242-6) Find Pt Eye Ord Observed.CCC Cognitive factors Delirium Delirium [MDSv3] (54626-7) – Pt Patient – MDSv3 Consciousness Level of consciousness (80288-4) Find Pt Patient Ord – Behavior factors Fall history Fall history on admission [MDSv3] (54849-5) – Pt Patient – MDSv3 Therapeutic factors IV line Intravenous apparatus (59457-2) Find Pt Patient Ord Morse Fall Scale Foley catheter Indwelling catheter [MDS] (45628-5) Find Pt Patient Ord MDS Incident characteristic Injury level Event outcome [MERSTH] (42545-4) Find Pt Patient Ord Observed. MERSTH Frequency of injurious falls Care component Data element LOINC concepts (ID) Property Time System Scale Method Relation with populating indicator Reason for encounter Primary diagnosis Primary diagnosis (18630-4) Imp Pt Patient Nom — Inclusion criteria Administrative Length of stay Hospital stay duration (78033-8) Time Pt Patient Qn — Patient days Pathophysiologic disease-related factors Use of anticoagulants/coagulopathy diagnosis/the first day of surgical procedure Bleeding risk [CCC] (81225-5) Find Pt Cardiovascular system Ord Observed.CCC Frequency of nursing assessment of risk factors Nursing diagnosis: visual impairment Visual alteration [CCC] (28242-6) Find Pt Eye Ord Observed.CCC Cognitive factors Delirium Delirium [MDSv3] (54626-7) – Pt Patient – MDSv3 Consciousness Level of consciousness (80288-4) Find Pt Patient Ord – Behavior factors Fall history Fall history on admission [MDSv3] (54849-5) – Pt Patient – MDSv3 Therapeutic factors IV line Intravenous apparatus (59457-2) Find Pt Patient Ord Morse Fall Scale Foley catheter Indwelling catheter [MDS] (45628-5) Find Pt Patient Ord MDS Incident characteristic Injury level Event outcome [MERSTH] (42545-4) Find Pt Patient Ord Observed. MERSTH Frequency of injurious falls CCC, clinical care classification; MDS, Minimum Data Set; MERSTH, Medical Event Reporting System Total Healthsystem. Table 2. Example Nursing Assessment Data Elements Whose Concepts Mapped onto LOINC Care component Data element LOINC concepts (ID) Property Time System Scale Method Relation with populating indicator Reason for encounter Primary diagnosis Primary diagnosis (18630-4) Imp Pt Patient Nom — Inclusion criteria Administrative Length of stay Hospital stay duration (78033-8) Time Pt Patient Qn — Patient days Pathophysiologic disease-related factors Use of anticoagulants/coagulopathy diagnosis/the first day of surgical procedure Bleeding risk [CCC] (81225-5) Find Pt Cardiovascular system Ord Observed.CCC Frequency of nursing assessment of risk factors Nursing diagnosis: visual impairment Visual alteration [CCC] (28242-6) Find Pt Eye Ord Observed.CCC Cognitive factors Delirium Delirium [MDSv3] (54626-7) – Pt Patient – MDSv3 Consciousness Level of consciousness (80288-4) Find Pt Patient Ord – Behavior factors Fall history Fall history on admission [MDSv3] (54849-5) – Pt Patient – MDSv3 Therapeutic factors IV line Intravenous apparatus (59457-2) Find Pt Patient Ord Morse Fall Scale Foley catheter Indwelling catheter [MDS] (45628-5) Find Pt Patient Ord MDS Incident characteristic Injury level Event outcome [MERSTH] (42545-4) Find Pt Patient Ord Observed. MERSTH Frequency of injurious falls Care component Data element LOINC concepts (ID) Property Time System Scale Method Relation with populating indicator Reason for encounter Primary diagnosis Primary diagnosis (18630-4) Imp Pt Patient Nom — Inclusion criteria Administrative Length of stay Hospital stay duration (78033-8) Time Pt Patient Qn — Patient days Pathophysiologic disease-related factors Use of anticoagulants/coagulopathy diagnosis/the first day of surgical procedure Bleeding risk [CCC] (81225-5) Find Pt Cardiovascular system Ord Observed.CCC Frequency of nursing assessment of risk factors Nursing diagnosis: visual impairment Visual alteration [CCC] (28242-6) Find Pt Eye Ord Observed.CCC Cognitive factors Delirium Delirium [MDSv3] (54626-7) – Pt Patient – MDSv3 Consciousness Level of consciousness (80288-4) Find Pt Patient Ord – Behavior factors Fall history Fall history on admission [MDSv3] (54849-5) – Pt Patient – MDSv3 Therapeutic factors IV line Intravenous apparatus (59457-2) Find Pt Patient Ord Morse Fall Scale Foley catheter Indwelling catheter [MDS] (45628-5) Find Pt Patient Ord MDS Incident characteristic Injury level Event outcome [MERSTH] (42545-4) Find Pt Patient Ord Observed. MERSTH Frequency of injurious falls CCC, clinical care classification; MDS, Minimum Data Set; MERSTH, Medical Event Reporting System Total Healthsystem. Table 3. Example Nursing Intervention Data Elements Whose Concepts Mapped onto ICNP Care component Data element ICNP nursing intervention statement Code Relation with populating indicator Universal precaution Keep floor surfaces clean and dry Fall preventiona 10040211 Frequency of provision of universal precaution Make sure patient’s essential needs are within easy reach Fall preventiona 10040211 Communication Use visual indicator Fall preventiona 10040211 Frequency of provision of communication and education interventions Notify or take over risk information between clinicians Collaborating with interprofessional team 10039416 Observation and surveillance Use bed or chair alarm Maintaining fall safety alarm 10041525 Frequency of provision of observation and surveillance interventions Assess fall risks (using a tool) Assessing risk of falls 10023520 Counting use of fall risk tool Provision of protocol Provide safety aids Providing safety device 10024527 Frequency of provision of protocol interventions Provide scheduled assistance with toileting Assisting with toileting 10023531 Offer help using the toilet Provision of education Educate high-risk patient/family Teaching family about fall prevention 10040269 Frequency of provision of communication and education interventions Environmental intervention Rearrange room to clear paths Environment safety managementa 10042507 Frequency of provision of environmental interventions Adjust bed into locked position Secure raised toilet seat to commode Post-fall care provided Apply ice pack Applying cold pack 10036468 Frequency of nursing assessment of risk factors Perform neurologic assessment periodically Monitoring neurologic status 10035326 Notify physician and other nurses about a fall Collaborating with interprofessional teama 10039416 Care component Data element ICNP nursing intervention statement Code Relation with populating indicator Universal precaution Keep floor surfaces clean and dry Fall preventiona 10040211 Frequency of provision of universal precaution Make sure patient’s essential needs are within easy reach Fall preventiona 10040211 Communication Use visual indicator Fall preventiona 10040211 Frequency of provision of communication and education interventions Notify or take over risk information between clinicians Collaborating with interprofessional team 10039416 Observation and surveillance Use bed or chair alarm Maintaining fall safety alarm 10041525 Frequency of provision of observation and surveillance interventions Assess fall risks (using a tool) Assessing risk of falls 10023520 Counting use of fall risk tool Provision of protocol Provide safety aids Providing safety device 10024527 Frequency of provision of protocol interventions Provide scheduled assistance with toileting Assisting with toileting 10023531 Offer help using the toilet Provision of education Educate high-risk patient/family Teaching family about fall prevention 10040269 Frequency of provision of communication and education interventions Environmental intervention Rearrange room to clear paths Environment safety managementa 10042507 Frequency of provision of environmental interventions Adjust bed into locked position Secure raised toilet seat to commode Post-fall care provided Apply ice pack Applying cold pack 10036468 Frequency of nursing assessment of risk factors Perform neurologic assessment periodically Monitoring neurologic status 10035326 Notify physician and other nurses about a fall Collaborating with interprofessional teama 10039416 aMeans the not mapped category. Table 3. Example Nursing Intervention Data Elements Whose Concepts Mapped onto ICNP Care component Data element ICNP nursing intervention statement Code Relation with populating indicator Universal precaution Keep floor surfaces clean and dry Fall preventiona 10040211 Frequency of provision of universal precaution Make sure patient’s essential needs are within easy reach Fall preventiona 10040211 Communication Use visual indicator Fall preventiona 10040211 Frequency of provision of communication and education interventions Notify or take over risk information between clinicians Collaborating with interprofessional team 10039416 Observation and surveillance Use bed or chair alarm Maintaining fall safety alarm 10041525 Frequency of provision of observation and surveillance interventions Assess fall risks (using a tool) Assessing risk of falls 10023520 Counting use of fall risk tool Provision of protocol Provide safety aids Providing safety device 10024527 Frequency of provision of protocol interventions Provide scheduled assistance with toileting Assisting with toileting 10023531 Offer help using the toilet Provision of education Educate high-risk patient/family Teaching family about fall prevention 10040269 Frequency of provision of communication and education interventions Environmental intervention Rearrange room to clear paths Environment safety managementa 10042507 Frequency of provision of environmental interventions Adjust bed into locked position Secure raised toilet seat to commode Post-fall care provided Apply ice pack Applying cold pack 10036468 Frequency of nursing assessment of risk factors Perform neurologic assessment periodically Monitoring neurologic status 10035326 Notify physician and other nurses about a fall Collaborating with interprofessional teama 10039416 Care component Data element ICNP nursing intervention statement Code Relation with populating indicator Universal precaution Keep floor surfaces clean and dry Fall preventiona 10040211 Frequency of provision of universal precaution Make sure patient’s essential needs are within easy reach Fall preventiona 10040211 Communication Use visual indicator Fall preventiona 10040211 Frequency of provision of communication and education interventions Notify or take over risk information between clinicians Collaborating with interprofessional team 10039416 Observation and surveillance Use bed or chair alarm Maintaining fall safety alarm 10041525 Frequency of provision of observation and surveillance interventions Assess fall risks (using a tool) Assessing risk of falls 10023520 Counting use of fall risk tool Provision of protocol Provide safety aids Providing safety device 10024527 Frequency of provision of protocol interventions Provide scheduled assistance with toileting Assisting with toileting 10023531 Offer help using the toilet Provision of education Educate high-risk patient/family Teaching family about fall prevention 10040269 Frequency of provision of communication and education interventions Environmental intervention Rearrange room to clear paths Environment safety managementa 10042507 Frequency of provision of environmental interventions Adjust bed into locked position Secure raised toilet seat to commode Post-fall care provided Apply ice pack Applying cold pack 10036468 Frequency of nursing assessment of risk factors Perform neurologic assessment periodically Monitoring neurologic status 10035326 Notify physician and other nurses about a fall Collaborating with interprofessional teama 10039416 aMeans the not mapped category. The data model represented in Tables 2 and 3 includes 84 LOINC nursing assessment concepts and 22 ICNP intervention concepts. We mapped 43 (51.2%) and 15 (68.2%) of those to local data in 1 EHR, and 37 (44.0%) and 15 (68.2%) of those for the other EHR. We classified 5 of the 15 intervention mappings as suboptimal. Some data elements comprised several data item-value sets, such as fall risk assessment tools and regular rounding. An array of data items comprised the fall risk assessment tool: total score, sub-items and scores, and risk status. The regular rounding as a fall prevention intervention included several universal precaution activities, such as keeping hospital bed brakes locked, educating on use of call light, keeping the patient’s personal possessions within reach, and assessing the need for 1:1 monitoring. The fall prevention education provided on admission was also a composite intervention. Patients’ personal risk factors determined the other interventions provided. Populating Falls Indicators from EHR Data Data from 7829 (73 525 patient days) and 8199 (58 530 patient days) patients admitted during the research period from 2 hospitals met the inclusion criteria. Using the 2 data sets from EHR systems, 10 of the 11 indicators were successfully populated; the exception was the rate of falls with injury (see Table 4). The fall rates were computed 0.71 and 0.41 per 1000 patient days, respectively (see Table 2 for data elements used to populate the outcome indicator). The falls with injury rate was calculated at 0.27/1000 patient days at only one hospital. It was calculated from 2 data elements of the length of stay and the injury level. Coded nursing assessments, diagnoses and interventions recorded in each EHR system populated the 9 process indicators (see examples in Tables 2 and 3). Table 4. Comparison of Outcome and Process Indicators in EHR systems, Event Reporting Systems, and the Combination of These 2 Systems Outcome and process indicators on fall prevention Hospital A Hospital B EHRs Event report EHRs + event report EHRs Event report EHRs + event report Fall ratea (falls per 1000 patient days) 0.71 0.79 1.32 0.41 0.02 0.43 Injury fall ratea (falls with injury per 1000 patient days) NA 0.27 NA NA NA NA Patients assessed using a fall risk tool within 24 h of hospital admissionb (%) 88.29 NA 88.29 96.98 NA 96.98 Average frequency of using a fall risk assessment toolb 0.44 NA 0.44 0.55 NA 0.55 Patients with a nursing assessment of risk factors within 24 hb (%) 94.44 NA 94.44 89.9 NA 89.9 Patients with a daily nursing assessment of risk factorsb (%) 1.29 NA 1.29 1.0 NA 1.0 Patient days when there was a risk of falling (%) 16.97 NA 16.97 21.0 NA 21.0 At-risk patients to which universal precautions were applied within 24 h of risk identificationb (%) 99.78 NA 99.78 0.87 NA 0.87 At-risk patients to which communication and education interventions were applied within 24 h of risk identificationb (%) 99.22 NA 99.22 10.9 NA 10.9 At-risk patients to which observation and surveillance interventions were applied within 24 h of risk identificationb (%) 78.81 NA 78.81 0 NA 0 At-risk patients to which provision of protocol interventions were applied within 24 h of risk identificationb (%) 13.42 NA 13.42 6.0 NA 6.0 At-risk patients to which environmental interventions were applied within 24 h of risk identificationb (%) 92.75 NA 92.75 89.1 NA 89.1 Outcome and process indicators on fall prevention Hospital A Hospital B EHRs Event report EHRs + event report EHRs Event report EHRs + event report Fall ratea (falls per 1000 patient days) 0.71 0.79 1.32 0.41 0.02 0.43 Injury fall ratea (falls with injury per 1000 patient days) NA 0.27 NA NA NA NA Patients assessed using a fall risk tool within 24 h of hospital admissionb (%) 88.29 NA 88.29 96.98 NA 96.98 Average frequency of using a fall risk assessment toolb 0.44 NA 0.44 0.55 NA 0.55 Patients with a nursing assessment of risk factors within 24 hb (%) 94.44 NA 94.44 89.9 NA 89.9 Patients with a daily nursing assessment of risk factorsb (%) 1.29 NA 1.29 1.0 NA 1.0 Patient days when there was a risk of falling (%) 16.97 NA 16.97 21.0 NA 21.0 At-risk patients to which universal precautions were applied within 24 h of risk identificationb (%) 99.78 NA 99.78 0.87 NA 0.87 At-risk patients to which communication and education interventions were applied within 24 h of risk identificationb (%) 99.22 NA 99.22 10.9 NA 10.9 At-risk patients to which observation and surveillance interventions were applied within 24 h of risk identificationb (%) 78.81 NA 78.81 0 NA 0 At-risk patients to which provision of protocol interventions were applied within 24 h of risk identificationb (%) 13.42 NA 13.42 6.0 NA 6.0 At-risk patients to which environmental interventions were applied within 24 h of risk identificationb (%) 92.75 NA 92.75 89.1 NA 89.1 aOutcome indicators and bItalics process indicators. Abbreviation: NA, not available. Table 4. Comparison of Outcome and Process Indicators in EHR systems, Event Reporting Systems, and the Combination of These 2 Systems Outcome and process indicators on fall prevention Hospital A Hospital B EHRs Event report EHRs + event report EHRs Event report EHRs + event report Fall ratea (falls per 1000 patient days) 0.71 0.79 1.32 0.41 0.02 0.43 Injury fall ratea (falls with injury per 1000 patient days) NA 0.27 NA NA NA NA Patients assessed using a fall risk tool within 24 h of hospital admissionb (%) 88.29 NA 88.29 96.98 NA 96.98 Average frequency of using a fall risk assessment toolb 0.44 NA 0.44 0.55 NA 0.55 Patients with a nursing assessment of risk factors within 24 hb (%) 94.44 NA 94.44 89.9 NA 89.9 Patients with a daily nursing assessment of risk factorsb (%) 1.29 NA 1.29 1.0 NA 1.0 Patient days when there was a risk of falling (%) 16.97 NA 16.97 21.0 NA 21.0 At-risk patients to which universal precautions were applied within 24 h of risk identificationb (%) 99.78 NA 99.78 0.87 NA 0.87 At-risk patients to which communication and education interventions were applied within 24 h of risk identificationb (%) 99.22 NA 99.22 10.9 NA 10.9 At-risk patients to which observation and surveillance interventions were applied within 24 h of risk identificationb (%) 78.81 NA 78.81 0 NA 0 At-risk patients to which provision of protocol interventions were applied within 24 h of risk identificationb (%) 13.42 NA 13.42 6.0 NA 6.0 At-risk patients to which environmental interventions were applied within 24 h of risk identificationb (%) 92.75 NA 92.75 89.1 NA 89.1 Outcome and process indicators on fall prevention Hospital A Hospital B EHRs Event report EHRs + event report EHRs Event report EHRs + event report Fall ratea (falls per 1000 patient days) 0.71 0.79 1.32 0.41 0.02 0.43 Injury fall ratea (falls with injury per 1000 patient days) NA 0.27 NA NA NA NA Patients assessed using a fall risk tool within 24 h of hospital admissionb (%) 88.29 NA 88.29 96.98 NA 96.98 Average frequency of using a fall risk assessment toolb 0.44 NA 0.44 0.55 NA 0.55 Patients with a nursing assessment of risk factors within 24 hb (%) 94.44 NA 94.44 89.9 NA 89.9 Patients with a daily nursing assessment of risk factorsb (%) 1.29 NA 1.29 1.0 NA 1.0 Patient days when there was a risk of falling (%) 16.97 NA 16.97 21.0 NA 21.0 At-risk patients to which universal precautions were applied within 24 h of risk identificationb (%) 99.78 NA 99.78 0.87 NA 0.87 At-risk patients to which communication and education interventions were applied within 24 h of risk identificationb (%) 99.22 NA 99.22 10.9 NA 10.9 At-risk patients to which observation and surveillance interventions were applied within 24 h of risk identificationb (%) 78.81 NA 78.81 0 NA 0 At-risk patients to which provision of protocol interventions were applied within 24 h of risk identificationb (%) 13.42 NA 13.42 6.0 NA 6.0 At-risk patients to which environmental interventions were applied within 24 h of risk identificationb (%) 92.75 NA 92.75 89.1 NA 89.1 aOutcome indicators and bItalics process indicators. Abbreviation: NA, not available. Comparing the eMeasurements with Event Reporting Systems None of the process indicators and only the fall incident outcome indicator was available from the reporting data (see Table 4). The falls with injury rate were not always specified due to incomplete data reported in incident reports. The variables of Table 4 are outcome and process indicators, and a denominator for process indicators to be applied within 24 h of risk identification. We found patient falls recorded in the EHR systems, but not reported in the incident reporting systems. These cases corresponded to 39 (40.2%) out of 97 incidents for one hospital and 24 (96%) out of 25 incidents for the other hospital. Through the chart review, we found 45 (46.4%) and 1 (4%) cases recorded in the incident reporting system, but not recorded in the EHR systems. Figure 4 illustrates the summary of these results using Venn diagrams. Figure 4. View largeDownload slide Comparison of falls detection results in EHRs and event reporting systems. Figure 4. View largeDownload slide Comparison of falls detection results in EHRs and event reporting systems. When we combined the fall incidents from EHRs and event reporting systems, the fall rates were 1.32 and 0.43, respectively. Discussion In the present study, we successfully populated nursing quality indicators from EHR systems in a format comparable with those used at other facilities. This was possible due to the availability of standard fall prevention content, standard terminologies, and a mapping process completed by experienced informaticians and nursing experts. Despite the challenges in mapping some nursing interventions to ICNP, we successfully autopopulated all the process indicators. In addition, using EHR data we detected fall events that were not available in the event reporting systems. LOINC had excellent content coverage for the 10 nursing assessment care components needed to populate the fall prevention indicators. The care components contain data elements that were measurements or observations captured in nursing assessments. This result demonstrates higher concept coverage than found in an earlier study28 which mapped 294 concepts extracted from nursing flowsheet documents of 6 organizations onto SNOMED CT and LOINC. Here, they reported mapping rates from 70% to 82%. Some of the data in our study comprised nursing judgments in diagnoses such as visual alteration, auditory alteration, and physical mobility impairment. These concepts mapped to precoordinated concepts from nursing terminology or assessment tools such as the Clinical Care Classification29 or the Morse Fall Scale30; both of which are incorporated in LOINC. Our findings imply that LOINC includes acceptable content coverage of nursing judgments (e.g., findings) related to falls prevention. LOINC is a catalog of measurements including laboratory tests, clinical measures such as vital signs, anthropomorphic measures, and standardized survey instruments. This finding suggests that LOINC can also be used as a standard terminology to represent fall prevention nursing assessment and observation concepts.31 As for intervention mapping, a relatively small number of ICNP intervention statements matched the nursing intervention data elements. Most universal precaution and environmental intervention care components were not mapped to ICNP interventions. The root cause is the low levels of granularity at which the precoordinated intervention statements in ICNP are aggregated. For example, the intervention statements of sharing/communication high risk patients between staff, use of visual indicator, keep family sitter, and communicate patient’s history, risk factor, treatment, and plan all corresponded to the single concept, collaborating with interprofessional team. Other statements imply various sets of activities that a nurse could provide to patients, such as assessing risk for falls on admission, fall prevention, and post-fall evaluation. The expressions do not provide detailed information about the kinds of activities planned and provided in practice; but rather as an abstract representation of relevant activities. The ICNP’s level of abstraction restricts the accuracy of concept mapping and secondary use to represent practice, quality reporting and research. However, ICNP has current mappings to SNOMED CT and this hierarchy could be leveraged to provide more detailed child concepts.32 Since the ICNP version 2 release, the ICNP terminology team has developed terminology subsets that are both clinically relevant and user-friendly. They encourage the development and maintenance of ICNP catalogs or subsets as part of the ICNP terminology life cycle, if such catalogs will enhance the usability of the terminology in practice. The ICN has reported that a pressure ulcer catalog is under development for the patient safety domain. However, there is no explicit fall prevention catalog to represent activities to prevent patient falls, even though inpatient falls are widespread, a common nursing-sensitive outcome, and a serious threat to patient safety. Our research team previously notified the ICNP terminology team of the importance of an inpatient fall prevention catalog and commenced its development using the conceptual ICPS framework.10 Unlike the event reporting systems, our EHR-based approach allowed us to populate process indicators as well as outcome indicators across multiple time periods. Using the existing event reporting system, we could extract outcome indicators over a relatively limited period. The incident reports did not have complete nursing process data. Considering the utility of process indicators in the short term for improving the quality of care6 and the fact that inpatient falls are relatively rare event (e.g., occurring to 0.05%–0.3% patients33,34), it is a great advantage that EHRs are able to compute process indicators. Process indicators have more power to detect real differences in the quality of nursing care provided each day. The ability to compute process indicators from EHR data supports the feasibility of using eMeasures for internal quality improvement activities, as well as for national reporting.2 The eMeasure approach has the potential to reduce the nursing management burden associated with manual reporting and increase efficiency in monitoring. One unexpected finding of the present study was the detection of new fall incidents from EHR data that had not been reported via the event reporting systems. The validation process detected several false-positive cases, but true-positive cases increased the percent of falls identified at the 2 hospitals by 40% and 96%, respectively. This finding confirmed empirically the under-reporting of safety incidents in the event reporting systems.35,36 This discovery implies that the use of EHRs in outcome measurement offers a new opportunity to address under-reporting. To achieve more accurate patient falls outcome reporting, we may need to redesign the reporting process. Use of EHR data to supplement event reporting system data is one option. However, omissions of fall event recording in EHRs is not straightforward. Some nurses may forget to document a fall. Another possibility is that nurses avoid entering the same content into 2 different systems (e.g., double documenting). A study37,38 investigating the accuracy of nursing notes in 6 EHR systems using a time-motion method, reported 85%–92% matching rates between the activities observed and recorded. This report supports the possibility of the later. Redesigning the incident reporting process to include reporting incidents solely in the EHR system, could lead to more complete data capture. One of the strengths of this study is that the participating hospitals independently implemented EHR systems that cover all inpatient nursing documentation. These systems have been in place for >8 years. Our analysis relied on the hospital CDRs, not a data mart. This made it possible to access any form of data (e.g., structured, semi-structured, and free text) without missing any data included in the live EHR systems. Another strength is the fact that we populated eMeasures using semi-structured data representing nursing process information. To our knowledge, this was not done previously. One study39 analyzed flowsheet data documented in multiple EHR systems and created 14 nursing information models based on clinical topics. It defined a flowsheet as a format for facilitating structured data capture and as a template that can be specific to a unit or discipline. They found that flowsheets contain groups of related assessments and interventions, and individual items for actual documentation. Another study40 analyzed flowsheet data of 2 million patients and developed an ontology to support secondary data use. However, the study is based on data documented at a single site in an Epic Systems Corporation EHR (single vendor). One limitation of this study is the generalizability of our findings to other measures and hospitals. However, considering the characteristics of inpatient falls, we believe that the data elements and processing methods will apply to other preventable adverse event quality measures such as pressure ulcers, pain management, and catheter-associated urinary tract infections. Based on ANA’s criteria for selecting quality indicators for conversion to eMeasures, inpatient falls is a strong candidate. The generalizability of our results to other care delivery systems and EHR systems is unknown. However, the approach to extract process and outcome indicators in a standardized and comparable format can be leveraged with any EHR system. Given that most EHR systems are not based on standards that enable semantic interoperability, the approach described could contribute to the benchmarking of nursing performance across sites and settings. Conclusions This study demonstrated the data from EHR environments can generate fall prevention process measures and at least one outcome metric. This supports sharing and comparing nursing-sensitive process and outcome metrics for multiple purposes, including the secondary use of a clinical nursing data set. Although the content and use of standards in EHR systems are incomplete and based on highly variable user preferences, standard nursing terminologies and content can be used to address existing gaps and EHR data can contribute to nursing quality improvement. Competing interests None. Funding This study was supported by a grant of the Korea Healthcare Technology R&D Project, Ministry for Health, Welfare & Family Affairs, Republic of Korea (No. HI15C1089). IC was also supported by a grant from Inha University (No. 50463-01) and the Hospital Nurses Association of Korean Nurses Association in 2015. Contributors IC conceived and designed the study, supervised, and contributed to the data analysis, interpreted results, and drafted and revised the paper. EHB and SYL contributed to study design, data acquisition, and data analysis. PCD provided substantial contribution to the interpretation of data analysis and revised the work critically for important intellectual content. Supplementary Material Supplementary material is available at Journal of the American Medical Informatics Association online. ACKNOWLEDGMENTS We thank Yeoun-Sook Lee, Jungmee Han, and Won-Hee Park of the Medical Information Departments of the 2 hospitals for helping us to retrieve the raw data efficiently. We also appreciate the clinical staffs of the Nursing Departments and graduate students involved in qualitative data processing, literature review, and discussions. References 1 Westra BL , Latimer GE , Matney SA et al. , A national action plan for sharable and comparable nursing data to support practice and translational research for transforming health care . J Am Med Inform Assoc 2015 ; 22 3 : 600 – 607 . Google Scholar PubMed 2 Amster A , Jentzsch J , Pasupuleti H , Subramanian K . Completeness, accuracy, and computability of National Quality Forum-specified eMeasures . J Am Med Inform Assoc 2015 ; 22 2 : 409 – 416 . Google Scholar PubMed 3 Cho I , Choi W , Choi WH . Adoption of electronic nursing records and standard nursing terminology in acute-care practices: findings from a Nationwide Survey . In: Saranto K , Weaver CA , Chang P , ed. Nursing Informatics 2014: Proceedings of the 12th International Congress on Nursing Informatics . Taipei, Taiwan : IOS Press ; 2014 . 4 Warren JJ , Dunton N . eMeasure Development Process Examplar . Silver Spring, MD: American Nurses Association ; 2015 . 5 Boushon B , Nielsen G , Quigley P et al. , How-to Guide: Reducing Patient Injuries From Falls . Cambridge, MA : Institute for Healthcare Improvement ; 2013 . 6 Mant J . Process versus outcome indicators in the assessment of quality of health care . Int J Qual Health Care 2001 ; 13 6 : 475 – 480 . Google Scholar CrossRef Search ADS PubMed 7 Idvall E , Rooke L , Hamrin E . Quality indicators in clinical nursing: a review of the literature . J Adv Nur 1997 ; 25 1 : 6 – 17 . Google Scholar CrossRef Search ADS 8 World Health Organization . Conceptual Framework for the International Classification for Patient Safety . Vol. 2009 . Geneva: World Health Organization ; 2009 : 1 – 149 . 9 Cho I , Chae J , Kim J , Jung M , Kim Y , Yoo A . Development of ICNP-based inpatient fall prevention catalogue . Int Nurs Rev 2017 (Under submission) . 10 Jung M , Cho I , Kim J , Chae J . Comparing inpatient falls guidelines to develop an ICNP-based nursing catalogue for ENRs . Paper presented at: Medinfo 2017 . Hangzhou, China ; 2017 . 11 American Nurses Association . Nursing Care Report Card for Acute Care . Silver Spring, MD: American Nurses Publishing ; 1995 . 12 Montalvo I . The National Database of Nursing Quality Indicators™ (NDNQI®) . Online J Issues Nurs 2007 ; 12 3 : 112–214. 13 Aydin CE , Bolton LB , Donaldson N et al. , Creating and analyzing a statewide nursing quality measurement database . Image J Nurs Sch 2004 ; 36 4 : 371 – 378 . Google Scholar CrossRef Search ADS 14 Patrician PA , Loan L , McCarthy M , Brosch LR , Davey KS . Towards evidence-based management: Creating an informative database of nursing-sensitive indicators . J Nurs Sch 2010 ; 42 4 : 358 – 366 . Google Scholar CrossRef Search ADS 15 Veteran's Health Affairs . Office of Nursing Services (ONS) Annual Report 2009 . 2009 . https://www.va.gov/NURSING/docs/OfficeofNursingServices-ONS_Annual_Report_2009-WEB.pdf. Accessed January 10, 2017. 16 The Joint Commission . Sentinel event alert 55: Preventing falls and fall-related injuries in health care facilities . Sentinel Event Alert . https://www.jointcommission.org/sea_issue_55/. Accessed March 10, 2018. 17 Ganz DA , Huang C , Saliba D , Shier V . Preventing Falls in Hospitals. Content last reviewed January 2013. Agency for Healthcare Research and Quality, Rockville, MD. http://www.ahrq.gov/professionals/systems/hospital/fallpxtoolkit/index.html. Accessed March 10, 2018. 18 ECRI Institute . Falls . 2009 . https://www.ecri.org/components/HRC/Pages/SafSec2.aspx. Accessed March, 2018. 19 Degelau J BM , Bungum L , Flavin PL et al. , Prevention of Falls (Acute Care) . 2012 . https://www.icsi.org/guidelines__more/catalog_guidelines_and_more/catalog_guidelines/catalog_patient_safetyreliability_guidelines/falls/. Accessed March, 2017. 20 The Joint Commission Center for Transforming Healthcare . New! Targeted Solutions Tool for Preventing Falls . 2016 . http://www.centerfortransforminghealthcare.org/tst_pfi.aspx. Accessed March, 2018. 21 U.S. Department of Veterans Affairs . VA National Center for Patient Safety: Falls Toolkit . 2015 . http://www.patientsafety.va.gov/professionals/onthejob/falls.asp. Accessed March, 2018. 22 Neily J , Quigley PA , Essen K . Implementation Guide for Fall Injury Reduction: VA National Center for Patient Safety Reducing Preventable Falls and Fall-Related Injuries . 2015 . http://www.patientsafety.va.gov/docs/fallstoolkit14/falls_implementation_%20guide%20_02_2015.pdf. Accessed March, 2018. 23 Korean Hospital Nurses Association . Nursing Practice Guidelines of Prevention of Inpatient Falls . 2008 . http://khna.or.kr/bbs/linkfile/resource/khna_Fcare.pdf. Accessed October 10, 2017. 24 American Nurses Association . Guidelines for data collection on the American Nurses Association's national quality forum endorsed measures: Nursing care hours per patient day; skill-mix; falls; falls with injury . KU School of Nursing, Kansas City . 2010 . 25 Cho I , Park HA . Evaluation of the expressiveness of an ICNP-based nursing data dictionary in a computerized nursing record system . J Am Med Inform Assoc 2006 ; 13 : 456 – 464 . Google Scholar CrossRef Search ADS PubMed 26 Matney S , Bakken S , Huff SM . Representing nursing assessments in clinical information systems using the logical observation identifiers, names, and codes database . J Biomed inform 2003 ; 36 4 : 287 – 293 . Google Scholar CrossRef Search ADS PubMed 27 Dykes P , Collins S . Building linkages between nursing care and improved patient outcomes: the role of health information technology . Online J Issues Nurs 2013 ; 18 3 : Manuscript 4. 28 Harris MR LL , Miller H , Hook M , Dykes PC , Matney SA . Harmonizing and extending standards from a domain-specific and bottom-up approach: an example from development through use in clinical applications . J Am Med Inform Assoc 2015 ; 22 3 : 545 – 552 . Google Scholar PubMed 29 Saba V . Clinical Care Classification (CCC) System Manual: a Guide to Nursing Documentation . New York: Springer Publishing Company ; 2006 . 30 Morse JM , Morse RM , Tylko SJ . Development of a scale to identify the fall-prone patient . Can J Aging/La Revue canadienne du vieillissement 1989 ; 8 4 : 366 – 377 . Google Scholar CrossRef Search ADS 31 Vreeman DJ , McDonald CJ , Huff SM . LOINC®: a universal catalogue of individual clinical observations and uniform representation of enumerated collections . Int J Funct Inform Personalised Med 2010 ; 3 4 : 273 – 291 . Google Scholar CrossRef Search ADS 32 Dykes PC , Wantland D , Whittenburg L , Lipsitz S , Saba VK . A pilot study to explore the feasibility of using the Clinical Care Classification System for developing a reliable costing method for nursing services . AMIA Annu Symp Proc. 2013 ; 2013 : 364 – 71 . Google Scholar PubMed 33 Staggs VS , Davidson J , Dunton N , Crosser B . Challenges in defining and categorizing falls on diverse unit types: Lessons from expansion of the NDNQI falls indicator . J Nurs Care Qual 2015 ; 30 2 : 106 – 112 . Google Scholar CrossRef Search ADS PubMed 34 Park IS , Cho I , Kim EM , Kim MK . Evaluation of a fall risk assessment tool to establish continuous quality improvement process for inpatients' falls . J Korean Acad Nurs Admin 2011 ; 17 4 : 484 – 492 . Google Scholar CrossRef Search ADS 35 Sari AB-A , Sheldon TA , Cracknell A , Turnbull A . Sensitivity of routine system for reporting patient safety incidents in an NHS hospital: retrospective patient case note review . BMJ 2007 ; 334 7584 : 79 . Google Scholar CrossRef Search ADS PubMed 36 Oliver D , Healey F , Haines TP . Preventing falls and fall-related injuries in hospitals . Clin Geri Med 2010 ; 26 4 : 645 – 692 . Google Scholar CrossRef Search ADS 37 Cho I , Kim M-K . Consistency analysis between electronic nursing records' data and observed direct nursing activities at multiple inpatient settings . 2017 Korean Society of Medical Informatics Spring Conference . Incheon, Republic of Korea : KOSMI ; 2017 . 38 Cho I . Analysis on functionality, usability, and use patterns of electronic nursing record systems: a multi-institutional time-motion approach . Seoul, Republic of Korea: Korean Hospital Nurses Association Research Report . 2014 January 17 . 39 Westra BL , Christie BA , Byrne MD et al. , Modeling flowsheet data for quality improvement and research . Paper presented at: Summer Institute in Nursing Informatics . Maryland, USA ; 2015 . 40 Johnson SG , Byrne MD , Christie B et al. , Modeling flowsheet data for clinical research . AMIA Summits Transl Sci Proc 2015 ; 2015 : 77 – 81 . Google Scholar PubMed © The Author(s) 2018. Published by Oxford University Press on behalf of the American Medical Informatics Association. All rights reserved. For permissions, please email: journals.permissions@oup.com This article is published and distributed under the terms of the Oxford University Press, Standard Journals Publication Model (https://academic.oup.com/journals/pages/about_us/legal/notices)

Journal

Journal of the American Medical Informatics AssociationOxford University Press

Published: Apr 6, 2018

References