Using the RE-AIM framework to evaluate internal and external validity of mobile phone–based interventions in diabetes self-management education and support

Yoshida,, Yilin; Patil, Sonal, J; Brownson, Ross, C; Boren, Suzanne, A; Kim,, Min; Dobson,, Rosie; Waki,, Kayo; Greenwood, Deborah, A; Torbjørnsen,, Astrid; Ramachandran,, Ambady; Masi,, Christopher; Fonseca, Vivian, A; Simoes, Eduardo, J

doi:10.1093/jamia/ocaa041

Using the RE-AIM framework to evaluate internal and external validity of mobile phone–based interventions in diabetes self-management education and support

Yoshida,, Yilin;Patil, Sonal, J;Brownson, Ross, C;Boren, Suzanne, A;Kim,, Min;Dobson,, Rosie;Waki,, Kayo;Greenwood, Deborah, A;Torbjørnsen,, Astrid;Ramachandran,, Ambady;Masi,, Christopher;Fonseca, Vivian, A;Simoes, Eduardo, J 2020-06-01 00:00:00 Abstract Objective We evaluated the extent to which studies that tested short message service (SMS)– and application (app)-based interventions for diabetes self-management education and support (DSMES) report on factors that inform both internal and external validity as measured by the RE-AIM (Reach, Efficacy/Effectiveness, Adoption, Implementation, and Maintenance) framework. Materials and Methods We systematically searched PubMed, Embase, Web of Science, CINAHL (Cumulative Index of Nursing and Allied Health Literature), and IEEE Xplore Digital Library for articles from January 1, 2009, to February 28, 2019. We carried out a multistage screening process followed by email communications with study authors for missing or discrepant information. Two independent coders coded eligible articles using a 23-item validated data extraction tool based on the RE-AIM framework. Results Twenty studies (21 articles) were included in the analysis. The comprehensiveness of reporting on the RE-AIM criteria across the SMS- and app-based DSMES studies was low. With respect to internal validity, most interventions were well described and primary clinical or behavioral outcomes were measured and reported. However, gaps exist in areas of attrition, measures of potential negative outcomes, the extent to which the protocol was delivered as intended, and description on delivery agents. Likewise, we found limited information on external validity indicators across adoption, implementation, and maintenance domains. Conclusions Reporting gaps were found in internal validity but more so in external validity in the current SMS- and app-based DSMES literature. Because most studies in this review were efficacy studies, the generalizability of these interventions cannot be determined. Future research should adopt the RE-AIM dimensions to improve the quality of reporting and enhance the likelihood of translating research to practice. mobile phone–based intervention, diabetes self-management education and support, SMS, app, RE-AIM INTRODUCTION Diabetes self-management education and support (DSMES) is a necessary component in diabetes care. DSMES plays an important role in glycemic management and preventing or delaying diabetes-related complications when administered alongside medical care and management.1,2 However, many barriers such as competing priorities, transportation barriers, underperceived seriousness of diabetes, and lack of accessible services discourage patients from obtaining traditional institution-based self-management education or training.3–6 There is a need for developing alternative DMSES interventions that are accessible and low cost for populations with diabetes. Mobile phone technology can serve as an effective platform to deliver DSMES because of its ubiquitous availability and adoption by all populations.7,8 The 2 most common tools used in mobile phone–based interventions are short message service (SMS) and smartphone applications (apps). SMS and app-based interventions have been applied in multiple behavioral interventions, including promoting physical activity, tracking healthy eating, monitoring blood glucose, taking medication, monitoring complications, and problem solving, and show promising results.9–17 One recent meta-analysis and systematic review focused on multiple strategies of health technology interventions in diabetes management identified a larger effect size in mobile phone–based interventions compared with other forms of technological interventions, such as computer- or Internet-based interventions.18 However, despite the increased popularity and demonstrated efficacy of mobile phone–based interventions in diabetes management, it remains largely unknown whether these interventions can be translated beyond the research setting and be broadly adopted in clinical and other settings. Additionally, review studies in this area have concentrated on reporting issues through the lens of internal validity.10,13 To date, no systematic review has evaluated domains of external validity and identified gaps that could inform the generalizability and translatability of mobile phone–based interventions in the specific area of DSMES. To bridge efficacy and effectiveness, or research and practice, Glasgow et al developed the RE-AIM (Reach, Efficacy/Effectiveness, Adoption, Implementation, and Maintenance) framework.19,20 The RE-AIM framework consists of 5 dimensions: reach into the target population and representativeness of the study sample; efficacy or effectiveness of the intervention on the primary outcome tested under either optimal or real-world conditions, effect on quality of life, and avoidance of unintended or negative consequences; adoption by target setting, organizations, and staff; implementation on consistency and cost of delivery; and maintenance of intervention effect over time.19,20 The RE-AIM framework has been utilized in reporting internal and external validity across numerous behavioral intervention studies, including physical activity, weight loss maintenance, and health literacy.21–27 These findings provide recommendations and future directions to improve the quality of reporting and to enhance the likelihood of translating research to practice. The goal of this review is to evaluate the extent to which studies, both randomized controlled trials (RCTs) and non-RCTs, testing SMS and app-based interventions to facilitate DSMES report on factors that inform both internal and external validity of the intervention. In addition to data extracted from literature databases, we also included data directly collected from study authors via emails. Recommendations to improve the quality of reporting and the likelihood of broad dissemination of effective SMS- and app-based interventions are provided based on 2 sources of data and other relevant evaluations of behaviors interventions. MATERIALS AND METHODS Database search and study inclusion We adopted the validated PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) approach to conduct the literature review. PRISMA is an evidence-based minimum set of items for reporting in systematic reviews.28 We systematically searched PubMed for eligible articles from January 1, 2009 to February 28, 2019, using combinations of the following MeSH (Medical Subject Headings) (M) and text word (TW) search terms: (1) Diabetes Mellitus Type 2 (M), non insulin dependent diabetes (T), T2DM (T); (2) self-management (M), self care (M); and (3) mHealth (TW), mobile health (TW), cell phone (M), cell phone (TW), mobile phone (TW), short message service (TW), and text messaging (M). Similar searches were conducted in Embase, Web of Science, CINAHL (Cumulative Index of Nursing and Allied Health Literature), and the IEEE Xplore Digital Library. We also performed supplementary searches using the reference lists of eligible articles and relevant systematic review and other review articles. We decided to search from 2009, as 2009 marked a shift in technology from basic mobile phones to smartphones, and therefore the use of apps.29 Eligibility criteria Studies were included if they were peer-reviewed studies testing the effect of SMS- or app-based interventions on DSMES among patients with type 2 diabetes. We excluded studies that did not use SMS or an app as the primary intervention (ie, used only phone calls or wearables devices for remote monitoring). In addition, we excluded qualitative research (ie, exclusive evaluations on patients’ attitudes or perceptions on interventions), studies of cost-effectiveness analyses, and protocol articles. Study screening We used a multistage screening process in which search results were first pooled and transferred to EndNote software (Clarivate Analytics, Philadelphia, PA) for duplicate removal. Next, 2 reviewers (Y.Y. and S.J.P.) independently screened article titles and abstracts for relevance. Studies with “type 2 diabetes” or “non–insulin dependent diabetes” in the titles and abstracts and included a mobile phone–based intervention for DSMES were selected for full text review. Finally, the 2 reviewers conducted full text reviews of selected articles to confirm eligibility. Articles extracted from reference lists underwent an identical process (Figure 1). Figure 1. Open in new tabDownload slide Article screening process (PRISMA [Preferred Reporting Items for Systematic Reviews and Meta-Analyses] 2018 flow diagram). CINAHL: Cumulative Index of Nursing and Allied Health Literature. Figure 1. Open in new tabDownload slide Article screening process (PRISMA [Preferred Reporting Items for Systematic Reviews and Meta-Analyses] 2018 flow diagram). CINAHL: Cumulative Index of Nursing and Allied Health Literature. RE-AIM coding and scoring A 23-item validated data extraction tool based on the RE-AIM framework was used to assess the extent of reporting study elements related to internal and external validity. Table 1 presents details on components that assess the RE-AIM dimensions. “Yes” or “no” were used to code the presence or absence of the RE-AIM components outlined in Supplementary Table 2. Each article was coded by 2 researchers independently; disagreement in coding was discussed until consensus was reached (Y.Y. and S.J.P.). Table 1. Evaluation dimensions and measures of the RE-AIM framework Dimension . Componenta . Internal or external validity indicator . Description22 . Reach The number, proportion, and representativeness of individuals who are willing to participate in a given intervention. Methods to identify target population Internal Description of the process by which the target population was identified in the intervention Inclusion criteria Internal Description of characteristics of the target population that were used to determine if a potential participant was eligible in the intervention Exclusion criteria Internal Description of characteristics of the target population that prevent a potential participant from being eligible to participate Sample size Internal The number of people who agree to participate Participation rate Internal Sample size divided by the target population denominator Representativeness Internal, external Comparison of characteristics of the study participants in comparison to the target population Efficacy/effectiveness The impact of an intervention on important outcomes, including potential negative effects. Measures/results at least 1 follow-up Internal The study outcomes are measured at a time point after baseline Intent-to-treat analysis utilized Internal Analyzing participants in trials in the groups to which they were randomized, regardless of whether they received or adhered to the allocated intervention Satisfactionb or potential negative outcomes Internal Measuring acceptability and usability of the intervention in participants; evaluate unintended consequences that may result from the intervention Attrition Internal The proportion that was lost of follow-up or dropped out of the intervention Adoption The number, proportion, and characteristics of adopting settings and interventions agents Description of intervention locationc Internal, external Description of characteristics of the location of the intervention Description of staff who delivered the programc Internal, external Description of characteristics of staff who delivered the intervention Method to identify staff who delivered the intervention External Description of the process by which the staff was identified for participation in the study Level of expertise of delivery agent External Training and educational background among those who delivered the intervention Inclusion/exclusion criteria of delivery agent or setting External Description of the eligibility criteria of the setting/agent Adoption rate of delivery agent or setting External The number of participating delivery settings or agents divided by the number of eligible and approached delivery settings or agents Implementation The extent to which the intervention is delivered as intended (eg, information on duration and frequency of intervention, fidelity to the intervention protocol, and cost including time and money) Intervention duration and frequency Internal Length of the intervention (ie, days, weeks, months, and length of each intervention contact) and number of contacts with participants Extent protocol delivered as intended (%) Internal Description of the fidelity to the intervention protocol Measures of cost of implementation Internal, external The costs including both money and time of delivery across all levels of implementation Maintenance The extent to which a participant maintains the change due to intervention and an intervention becomes institutionalized or part of the routine organization practices Assessed outcomes ≥6 mo postintervention External Description of follow-up outcome measures of individuals at some duration after intervention was terminated Indicators of program level maintenance External Description of program continuation after completion of the research study Measures of cost of maintenance External The ongoing cost of maintaining delivery across all levels of the intervention Program adopted in other setting/populations External Description of the intervention being adopted beyond the original setting and population Dimension . Componenta . Internal or external validity indicator . Description22 . Reach The number, proportion, and representativeness of individuals who are willing to participate in a given intervention. Methods to identify target population Internal Description of the process by which the target population was identified in the intervention Inclusion criteria Internal Description of characteristics of the target population that were used to determine if a potential participant was eligible in the intervention Exclusion criteria Internal Description of characteristics of the target population that prevent a potential participant from being eligible to participate Sample size Internal The number of people who agree to participate Participation rate Internal Sample size divided by the target population denominator Representativeness Internal, external Comparison of characteristics of the study participants in comparison to the target population Efficacy/effectiveness The impact of an intervention on important outcomes, including potential negative effects. Measures/results at least 1 follow-up Internal The study outcomes are measured at a time point after baseline Intent-to-treat analysis utilized Internal Analyzing participants in trials in the groups to which they were randomized, regardless of whether they received or adhered to the allocated intervention Satisfactionb or potential negative outcomes Internal Measuring acceptability and usability of the intervention in participants; evaluate unintended consequences that may result from the intervention Attrition Internal The proportion that was lost of follow-up or dropped out of the intervention Adoption The number, proportion, and characteristics of adopting settings and interventions agents Description of intervention locationc Internal, external Description of characteristics of the location of the intervention Description of staff who delivered the programc Internal, external Description of characteristics of staff who delivered the intervention Method to identify staff who delivered the intervention External Description of the process by which the staff was identified for participation in the study Level of expertise of delivery agent External Training and educational background among those who delivered the intervention Inclusion/exclusion criteria of delivery agent or setting External Description of the eligibility criteria of the setting/agent Adoption rate of delivery agent or setting External The number of participating delivery settings or agents divided by the number of eligible and approached delivery settings or agents Implementation The extent to which the intervention is delivered as intended (eg, information on duration and frequency of intervention, fidelity to the intervention protocol, and cost including time and money) Intervention duration and frequency Internal Length of the intervention (ie, days, weeks, months, and length of each intervention contact) and number of contacts with participants Extent protocol delivered as intended (%) Internal Description of the fidelity to the intervention protocol Measures of cost of implementation Internal, external The costs including both money and time of delivery across all levels of implementation Maintenance The extent to which a participant maintains the change due to intervention and an intervention becomes institutionalized or part of the routine organization practices Assessed outcomes ≥6 mo postintervention External Description of follow-up outcome measures of individuals at some duration after intervention was terminated Indicators of program level maintenance External Description of program continuation after completion of the research study Measures of cost of maintenance External The ongoing cost of maintaining delivery across all levels of the intervention Program adopted in other setting/populations External Description of the intervention being adopted beyond the original setting and population RE-AIM: Reach, Efficacy/Effectiveness, Adoption, Implementation, and Maintenance. a Components were derived from a reliable extraction tool.18–20,23 b Components were included to ensure relevancy with the mobile health interventions. c Components were informed by other RE-AIM framework reviews of health behavior interventions.21,48 Open in new tab Table 1. Evaluation dimensions and measures of the RE-AIM framework Dimension . Componenta . Internal or external validity indicator . Description22 . Reach The number, proportion, and representativeness of individuals who are willing to participate in a given intervention. Methods to identify target population Internal Description of the process by which the target population was identified in the intervention Inclusion criteria Internal Description of characteristics of the target population that were used to determine if a potential participant was eligible in the intervention Exclusion criteria Internal Description of characteristics of the target population that prevent a potential participant from being eligible to participate Sample size Internal The number of people who agree to participate Participation rate Internal Sample size divided by the target population denominator Representativeness Internal, external Comparison of characteristics of the study participants in comparison to the target population Efficacy/effectiveness The impact of an intervention on important outcomes, including potential negative effects. Measures/results at least 1 follow-up Internal The study outcomes are measured at a time point after baseline Intent-to-treat analysis utilized Internal Analyzing participants in trials in the groups to which they were randomized, regardless of whether they received or adhered to the allocated intervention Satisfactionb or potential negative outcomes Internal Measuring acceptability and usability of the intervention in participants; evaluate unintended consequences that may result from the intervention Attrition Internal The proportion that was lost of follow-up or dropped out of the intervention Adoption The number, proportion, and characteristics of adopting settings and interventions agents Description of intervention locationc Internal, external Description of characteristics of the location of the intervention Description of staff who delivered the programc Internal, external Description of characteristics of staff who delivered the intervention Method to identify staff who delivered the intervention External Description of the process by which the staff was identified for participation in the study Level of expertise of delivery agent External Training and educational background among those who delivered the intervention Inclusion/exclusion criteria of delivery agent or setting External Description of the eligibility criteria of the setting/agent Adoption rate of delivery agent or setting External The number of participating delivery settings or agents divided by the number of eligible and approached delivery settings or agents Implementation The extent to which the intervention is delivered as intended (eg, information on duration and frequency of intervention, fidelity to the intervention protocol, and cost including time and money) Intervention duration and frequency Internal Length of the intervention (ie, days, weeks, months, and length of each intervention contact) and number of contacts with participants Extent protocol delivered as intended (%) Internal Description of the fidelity to the intervention protocol Measures of cost of implementation Internal, external The costs including both money and time of delivery across all levels of implementation Maintenance The extent to which a participant maintains the change due to intervention and an intervention becomes institutionalized or part of the routine organization practices Assessed outcomes ≥6 mo postintervention External Description of follow-up outcome measures of individuals at some duration after intervention was terminated Indicators of program level maintenance External Description of program continuation after completion of the research study Measures of cost of maintenance External The ongoing cost of maintaining delivery across all levels of the intervention Program adopted in other setting/populations External Description of the intervention being adopted beyond the original setting and population Dimension . Componenta . Internal or external validity indicator . Description22 . Reach The number, proportion, and representativeness of individuals who are willing to participate in a given intervention. Methods to identify target population Internal Description of the process by which the target population was identified in the intervention Inclusion criteria Internal Description of characteristics of the target population that were used to determine if a potential participant was eligible in the intervention Exclusion criteria Internal Description of characteristics of the target population that prevent a potential participant from being eligible to participate Sample size Internal The number of people who agree to participate Participation rate Internal Sample size divided by the target population denominator Representativeness Internal, external Comparison of characteristics of the study participants in comparison to the target population Efficacy/effectiveness The impact of an intervention on important outcomes, including potential negative effects. Measures/results at least 1 follow-up Internal The study outcomes are measured at a time point after baseline Intent-to-treat analysis utilized Internal Analyzing participants in trials in the groups to which they were randomized, regardless of whether they received or adhered to the allocated intervention Satisfactionb or potential negative outcomes Internal Measuring acceptability and usability of the intervention in participants; evaluate unintended consequences that may result from the intervention Attrition Internal The proportion that was lost of follow-up or dropped out of the intervention Adoption The number, proportion, and characteristics of adopting settings and interventions agents Description of intervention locationc Internal, external Description of characteristics of the location of the intervention Description of staff who delivered the programc Internal, external Description of characteristics of staff who delivered the intervention Method to identify staff who delivered the intervention External Description of the process by which the staff was identified for participation in the study Level of expertise of delivery agent External Training and educational background among those who delivered the intervention Inclusion/exclusion criteria of delivery agent or setting External Description of the eligibility criteria of the setting/agent Adoption rate of delivery agent or setting External The number of participating delivery settings or agents divided by the number of eligible and approached delivery settings or agents Implementation The extent to which the intervention is delivered as intended (eg, information on duration and frequency of intervention, fidelity to the intervention protocol, and cost including time and money) Intervention duration and frequency Internal Length of the intervention (ie, days, weeks, months, and length of each intervention contact) and number of contacts with participants Extent protocol delivered as intended (%) Internal Description of the fidelity to the intervention protocol Measures of cost of implementation Internal, external The costs including both money and time of delivery across all levels of implementation Maintenance The extent to which a participant maintains the change due to intervention and an intervention becomes institutionalized or part of the routine organization practices Assessed outcomes ≥6 mo postintervention External Description of follow-up outcome measures of individuals at some duration after intervention was terminated Indicators of program level maintenance External Description of program continuation after completion of the research study Measures of cost of maintenance External The ongoing cost of maintaining delivery across all levels of the intervention Program adopted in other setting/populations External Description of the intervention being adopted beyond the original setting and population RE-AIM: Reach, Efficacy/Effectiveness, Adoption, Implementation, and Maintenance. a Components were derived from a reliable extraction tool.18–20,23 b Components were included to ensure relevancy with the mobile health interventions. c Components were informed by other RE-AIM framework reviews of health behavior interventions.21,48 Open in new tab Missing data inquiry from study authors After the article screening and data extraction, we contacted corresponding authors by email when interventions included in the final review had missing or discrepant information. We introduced the purpose of our study and provided inquiries about missing or discrepant data according to RE-AIM criteria as well as additional questions regarding study implementation and generalizability. If there was no response to our initial email, reminders were sent at 7, 14, and 30 days. RESULTS Study characteristics Our search yielded 20 studies (Supplementary Table 2). Of these, 14 were RCTs and 2 were pragmatic trials. Four were quasi-experimental studies, among which 1 was an intention-to-treat trial. Nine studies centered on app-based interventions, and 11 focused on SMS-based or the combination of SMS and phone call–based interventions. Eight studies were conducted in urban settings and 1 included both urban and rural settings. Five studies were conducted in the United States, 3 were conducted in Europe (Italy, Spain, and England), 4 were conducted in East Asia, 2 were conducted in India, 2 were conducted in Iran, 1 had locations in both Australia and Taiwan, 1 were conducted in Canada, 1 were conducted in New Zealand, and 1 were conducted in Bahrain. A total of 12 authors from 12 studies responded to our email inquiries, with 9 addressing our inquires (Supplementary Table 2). RE-AIM analysis Reach All 20 studies reported specific inclusion and exclusion criteria and methods to identify target population. All 20 studies provided sample size (median sample size = 140; range, 32-427). Five studies did not report on number of eligible participants30–35; therefore, participation rate could not be calculated. Among 16 studies that provided the participation rates, median participation rate was 83% (range, 5.7%-100%). No studies in this review explicitly described characteristics of both participants and nonparticipants; therefore, we were unable to make conclusions about representativeness of the samples (Table 2). Table 2. Proportion of mobile health intervention on DSMES reporting RE-AIM framework dimensions and components (n = 20) Dimension and componenta . Proportion (%) . Notes . Reach Methods to identify target population 100 Inclusion criteria 100 Exclusion criteria 94.4 Sample size 100 Median sample size = 140 (range, 32-427) Participation rate 70 Median rate 83% (range, 5.7%-100%) Representativeness 0 Efficacy/effectiveness Measures/results at least 1 follow-up 100 Intention-to-treat analysis utilized 45 Satisfactionb or potential negative outcomes 25 Attrition 14.8 Median attrition rate 15% (range, 2%-35%) Qualitive methods to measure efficacy/effectiveness 50 Adoption Description of intervention locationc 90 Description of staff who delivered the programc 0 Method to identify staff who delivered the intervention 0 Level of expertise of delivery agent 45 Inclusion/exclusion criteria of delivery agent or setting 5.5 (setting); 0 (staff) Adoption rate of delivery agent or setting 0 Implementation Intervention duration and frequency 100 Median follow-up 4.5 months, (range, 3-22 months) Extent protocol delivered as intended 30 Measures of cost of implementation 15 2 studies reported monetary cost, 1 study reported time cost Maintenance Assessed outcomes ≥6 mo postintervention 5 Indicators of program level maintenance 0 Measures of cost of maintenance 0 Program adopted in other setting/populations 20 Dimension and componenta . Proportion (%) . Notes . Reach Methods to identify target population 100 Inclusion criteria 100 Exclusion criteria 94.4 Sample size 100 Median sample size = 140 (range, 32-427) Participation rate 70 Median rate 83% (range, 5.7%-100%) Representativeness 0 Efficacy/effectiveness Measures/results at least 1 follow-up 100 Intention-to-treat analysis utilized 45 Satisfactionb or potential negative outcomes 25 Attrition 14.8 Median attrition rate 15% (range, 2%-35%) Qualitive methods to measure efficacy/effectiveness 50 Adoption Description of intervention locationc 90 Description of staff who delivered the programc 0 Method to identify staff who delivered the intervention 0 Level of expertise of delivery agent 45 Inclusion/exclusion criteria of delivery agent or setting 5.5 (setting); 0 (staff) Adoption rate of delivery agent or setting 0 Implementation Intervention duration and frequency 100 Median follow-up 4.5 months, (range, 3-22 months) Extent protocol delivered as intended 30 Measures of cost of implementation 15 2 studies reported monetary cost, 1 study reported time cost Maintenance Assessed outcomes ≥6 mo postintervention 5 Indicators of program level maintenance 0 Measures of cost of maintenance 0 Program adopted in other setting/populations 20 DSMES: diabetes self-management education and support; RE-AIM: Reach, Efficacy/Effectiveness, Adoption, Implementation, and Maintenance. a Components were derived from a reliable extraction tool.19–21,24 b Components were included to ensure relevancy with the mobile health interventions. c Components were informed by other RE-AIM framework reviews of health behavior interventions.22,36 Open in new tab Table 2. Proportion of mobile health intervention on DSMES reporting RE-AIM framework dimensions and components (n = 20) Dimension and componenta . Proportion (%) . Notes . Reach Methods to identify target population 100 Inclusion criteria 100 Exclusion criteria 94.4 Sample size 100 Median sample size = 140 (range, 32-427) Participation rate 70 Median rate 83% (range, 5.7%-100%) Representativeness 0 Efficacy/effectiveness Measures/results at least 1 follow-up 100 Intention-to-treat analysis utilized 45 Satisfactionb or potential negative outcomes 25 Attrition 14.8 Median attrition rate 15% (range, 2%-35%) Qualitive methods to measure efficacy/effectiveness 50 Adoption Description of intervention locationc 90 Description of staff who delivered the programc 0 Method to identify staff who delivered the intervention 0 Level of expertise of delivery agent 45 Inclusion/exclusion criteria of delivery agent or setting 5.5 (setting); 0 (staff) Adoption rate of delivery agent or setting 0 Implementation Intervention duration and frequency 100 Median follow-up 4.5 months, (range, 3-22 months) Extent protocol delivered as intended 30 Measures of cost of implementation 15 2 studies reported monetary cost, 1 study reported time cost Maintenance Assessed outcomes ≥6 mo postintervention 5 Indicators of program level maintenance 0 Measures of cost of maintenance 0 Program adopted in other setting/populations 20 Dimension and componenta . Proportion (%) . Notes . Reach Methods to identify target population 100 Inclusion criteria 100 Exclusion criteria 94.4 Sample size 100 Median sample size = 140 (range, 32-427) Participation rate 70 Median rate 83% (range, 5.7%-100%) Representativeness 0 Efficacy/effectiveness Measures/results at least 1 follow-up 100 Intention-to-treat analysis utilized 45 Satisfactionb or potential negative outcomes 25 Attrition 14.8 Median attrition rate 15% (range, 2%-35%) Qualitive methods to measure efficacy/effectiveness 50 Adoption Description of intervention locationc 90 Description of staff who delivered the programc 0 Method to identify staff who delivered the intervention 0 Level of expertise of delivery agent 45 Inclusion/exclusion criteria of delivery agent or setting 5.5 (setting); 0 (staff) Adoption rate of delivery agent or setting 0 Implementation Intervention duration and frequency 100 Median follow-up 4.5 months, (range, 3-22 months) Extent protocol delivered as intended 30 Measures of cost of implementation 15 2 studies reported monetary cost, 1 study reported time cost Maintenance Assessed outcomes ≥6 mo postintervention 5 Indicators of program level maintenance 0 Measures of cost of maintenance 0 Program adopted in other setting/populations 20 DSMES: diabetes self-management education and support; RE-AIM: Reach, Efficacy/Effectiveness, Adoption, Implementation, and Maintenance. a Components were derived from a reliable extraction tool.19–21,24 b Components were included to ensure relevancy with the mobile health interventions. c Components were informed by other RE-AIM framework reviews of health behavior interventions.22,36 Open in new tab Effectiveness All 20 studies reported on findings for at least 1 follow-up measurement. Six (33.3%) studies had more than 1 follow-up.31,37–42 Nine (45%) studies reported on intention-to-treat analysis.32,38–47 Eight (40%) provided information on missing data or imputation measures.37,39–45,47 The median attrition rate was 15% (range, 2%-35%). Five (25%) studies reported satisfaction or potential negative outcomes.37,40–43,48 Ten (50%) studies reported including qualitative methods to measure efficacy or effectiveness of their studies. Among these, 7 used satisfaction surveys or patients’ feedback from real-time monitoring systems to understand participants’ satisfaction and acceptability of the intervention.32,33,37,38,40–45,49 One study reported intermittent technical challenges faced by few participants,37 and another study reported intervention burden or poor response to participant requests as reasons for dropout from the study.42 Information on adverse outcomes related to communication mode such as misinformation, confusion, or data errors were not reported in any studies (Table 2). Adoption Ten of 20 (50%) studies provided information on adoption items.30,37–39,43–46,48,50 Most study participants (n = 18; 90%) were recruited in primary or secondary care settings and utilized physician or self-referral. One study recruited participants online,44 and another identified potential eligible participants using automatic search from anonymous pharmacy data of 40 pharmacies.45 Four studies (20%) provided brief descriptions of characteristics of the recruitment settings.37,38,43,48 Regarding intervention delivery, 10 (50%) studies reported level of staff expertise. Three studies used diabetes educators,39,44,50 1 used nurses,48 1 used physicians,40 1 used a combination of clinicians and diabetes educators,30 1 used a combination of nurses and research staff,46 1 used a combination of nurse and nutritionist,41,42 and 2 used only research staff to facilitate intervention delivery.38,47 No study provided characteristics, method to identify, eligibility, or adoption rate of intervention staff. Even though most studies reached patients in a primary or secondary care setting, the intervention may be delivered in nonclinical settings (eg, home) remotely though app and text. Only one study39 reported on eligibility for setting selection, method to identify setting, and setting adoption rate. However, this setting was the recruitment setting rather the intervention setting39 (Table 2). Implementation All 20 studies reported study duration (median length of follow-up = 4.5 months; range, 3-22 months). One study did not report intervention frequency or dosage.38 In terms of intervention fidelity, 6 (30%) studies reported that the intervention protocol was delivered as intended.31,37,40–43,48 One study author replied by email that their intervention was not carried out as intended.50 One study was an intention-to-treat trial,44 and 10 study authors did not respond to our inquiry on adoption components. Two studies measured monetary cost of the intervention,37,43 and 1 measured time cost.50 Two studies reported providing incentives to intervention staff,41,42,44 and 2 provided incentives to participants (Table 2).43,48 Maintenance Only one study measured outcomes at 6-month follow-up after the study completed, according to study author’s response via email.37 No study reported indicators of program-level maintenance nor cost of maintenance. Authors from 4 studies informed us that their interventions have been adopted in other settings or patient populations41–43,45 or were being considered for a large scale implementation (Table 2).37 Applicability of studies conducted in the United States Of the studies conducted in the United States, app-based intervention studies either provided mobile phones with unlimited service plans39 or required patients to own an iPhone,44 whereas SMS-based interventions required patients to own cellphones capable of receiving text messages.48,50 Most studies had predominantly White non-Hispanic participants, with one study including predominantly Hispanic participants,43 and another study had 39.3% African American participants.39 One study identified that patient messages and diabetes educator responses were in the electronic health records, where primary care physicians (PCPs) could access to all messages.50 In a study by Bauer et al,48 physicians received information on diagnosis and management of diabetic neuropathy as they exclusively recruited participants with peripheral diabetes neuropathy; however, change in treatment secondary to SMS or app was not tracked. Quinn et al39 compared the control group with 3 additional groups: (1) an app-based intervention group with no PCP involvement, (2) an app-based intervention group in which PCPs received unanalyzed raw data, and (3) an app-based intervention group in which PCPs received summarized report by fax or email with treatment recommendations. Findings indicate that addition of clinical decision support did not differ from an app-based intervention group with no physician involvement.39 Two studies had no physician involvement.43,44 Overall, mobile phone–based DSMES interventions in the United States did not provide actionable summarized data in the electronic health record for PCPs to provide clinical decision support. Connecting mobile phone–based data to clinical management Four studies conducted outside of the United States reported some form of provider management of mobile phone–based patient entered data. One study conducted in China provided patients with an action plan as previsit summaries for physician visits.33 Another Chinese study also provided guidance for blood glucose monitoring, diet, and exercise.35 Another study conducted in India had a web portal and smartphone app for physicians to communicate treatment titration or recommendations.38 One study from Bahrain gave patients the mobile number of the diabetes educators’ and physicians’ to allow for texting between visits.30 A study from Korea had medical staff from a hospital provide tailored recommendations based on patients’ self-monitoring data of glucose and blood pressure.49 DISCUSSION Discussion notes for main results This study used the RE-AIM framework to systematically review the state of research on SMS- and app-based interventions targeting DSMES from both internal and external validity perspectives. Methods to target populations and selection criteria were reported across studies. The participation rate (average at 70%) is similar to other behavioral interventions (average at 76%) that include physical activity promotion or smoking cessation at individual level.19 This participation rate is encouraging and suggests that SMS- and app-based DMSE programs are appealing to patients. However, it is important to note that most studies recruited participants at academic medical centers, hospitals systems, or other institutions focused on clinical care. Previous research has demonstrated that recruitment at clinical settings may offer increased access to patients who are ready to participate in trials because of better information provided by clinicians regarding the study.34,51–53 As a result, patients with no access to clinical care, compounded by low health literacy, mistrust of the system, or lack of information or awareness of research opportunities, are likely to be excluded.34 Additionally, no study in this review described the target population or indicated representativeness of the study sample to a larger population. Many mobile phone–based interventions in behavioral change fall short on describing the target population, raising concerns of generalizability of these interventions to varying sociodemographic groups.21,22,27,36,54 Similarly, the convenience sampling that interventions employed also challenges the understanding of whether the intervention is reaching subgroups of a population and those individuals that could benefit the most.22,34 Efficacy or effectiveness of SMS- and app-based interventions on diabetes disease or behavioral outcomes was reported in all studies, while information on maintenance of the changes was absent. This finding is consistent with interventions in other areas.36,55,56,21,24 Less than half of the studies performed an intention-to-treat analysis. The rest of the studies presented results of those who completed the follow-up. Whether the SMS- or app-based interventions produce lasting effects is questionable because most studies did not examine maintenance at least 6 months past an intervention. Researchers in a previous review focusing on promoting physical activity through mobile health technology suggested that mobile phone–based interventions are a relatively new area of research that the studies still emphasized to determine whether the interventions can initiate change.22 The researchers also indicated that mobile phone–based interventions may reduce the likelihood of maintaining disease or behavioral outcome changes, as fast advancements in technology could make current interventions obsolete.22 Moreover, the potential of technical problems may reduce motivation and discourage engagement.22 These reasons could in part explain the lack of description on maintenance in SMS- and app-based intervention studies in this review. Additionally, determinants of efficacy or effectiveness remain largely unexplained. Only 5 studies reported using qualitative methods to measure efficacy or effectiveness. The degree to which patients found mobile phone–based interventions to be acceptable, feasible, and sustainable to use are not documented, which hinders the understanding of the potential long-term effects of those interventions. Organizational or delivery-level aspects of the RE-AIM framework have been historically underreported across behavioral interventions.57 This is also the case for SMS- and app-based DSMES research. We found very limited descriptions of the methods used to engage those who delivered the intervention or description of their characteristics, the extent to which the intervention was delivered as intended, and if any adaptions were made to the intervention during the study period. Furthermore, costs across the RE-AIM framework are also important to inform dissemination but are often missed in reporting.58 In mobile phone–based interventions specially, previous research has suggested other costs in addition to implementation costs need to be documented. This includes costs associated with recruitment, equipment, technology (eg, mobile phone, service plan, technical maintenance), and even future cost of continuing to use the service.22 Information on cost can help to determine the generatability or replicability of an intervention. The most obvious omission in reporting is maintenance at the organizational level. Few research projects have the resources to ensure that their interventions can be sustained at the organizational level. Moreover, readiness to adopt and implement mobile phone–based programs are also hindered by financial resources, policies, and workplace culture.59,60 Indeed, in email conversations with study authors, only one author informed us that their SMS-based diabetes self-management program is being considered for national implementation.37 Mobile phone–based programs have the potential to gather large amounts of health data that can be used to better inform interventions and clinical management plans.61 These data may also meet the interests of major government stakeholders looking for fiscal efficiencies in healthcare delivery.61 However, most mobile-based interventions fail to move beyond the pilot or efficacy trial phase, owing to barriers to implementation and sustainability. Previous research discussion indicates that compared with other industries, health care is relatively reluctant to adopt new technologies because of resistance to change within organizations, absence or inadequacy of policies, and funding issues.60,62–65 Ross et al63 reported that a frequent reason for unsuccessful implementation of a new technology is that it does not fit well with work practices or daily clinical work. They also suggested that unless a technology is adaptable to fit with roles, tasks, and workflows of clinicians and there is access to dedicated technical support staff, resistance to implementing a new technology will remain a challenge.63 More importantly, intervention researchers need to understand the standards and policies regarding data safety and privacy, professional liability, and potential pitfalls of data sharing between systems and organizations.64,65 Another major implementation barrier is the termination of funding support, as the additional costs of privacy and security testing, ongoing technology support and development, and software maintenance do not fit well with government supported funding cycles for research and development.60,66 In our review, termination of funding is also one of the most frequently reported implementation barriers by study authors. Recommendations for future research Table 3 provides recommendations for future research to improve the assessment and reporting on individual and organizational level factors that will support the internal and external validity of SMS- and app-based intervention in DSMES. We constructed this table using evidence from the current review as well as recommendations from other relevant evaluations on behavioral interventions.22,57,67–71 In addition to these recommendations, efforts from other stakeholders including funding organizations and practitioners should also be in place to promote translatability of mobile phone–based solutions in diabetes management. Such efforts include encouraging reporting of negative or unintended outcomes and positive outcomes across RE-AIM dimensions, so that the feasible and efficacious parts of an intervention can be replicated.23 Funding organizations may consider requiring researchers to develop a plan for sustainability throughout the research. Practitioners should also demand more information on external validity of an intervention with demonstrated efficacy. Information on adoption, implementation, and maintenance will help them clarify whether an efficacious intervention is a good fit for their organization.23 Table 3. Recommendations based on gaps identified from the RE-AIM framework evaluation Dimension and component . Key issues . Overall and specific recommendations . Reach Representativeness of study sample to target population Document and understand access, awareness, and appropriateness of intervention to meet target population needs67 Compare characteristics (eg, sociodemographic, economic, and behavioral) of participants with nonparticipants or the general local population to understand the representativeness of the sample22 Include inclusion criteria (eg, health/disease conditions, such as hemoglobin A1c ≥8.0%, capable of using mobile phone) and exclusion to Provide explanation on why certain individuals were not eligible for participation22 Include recruitment setting, methods, and recruitment adaptations22 Efficacy/effectiveness Suitability and credibility of study design, data collection, and evaluation Document participants’ satisfaction, negative outcomes, and subgroup effects in addition to reporting primary outcome67 Report study design (eg, randomized controlled trial) and whether a comparison group is included Report use intention-to-treat analysis22 and missing data procedure68 Report qualitative methods to acceptability and usability of the intervention; report potential negative outcomes of the intervention22 Report subgroup effects (eg, sex, race, age, or health/disease condition that influence the intervention effect)22 Adoption Diffusion of intervention program at organizational and delivery level and factors influence the adoption Document and understand contextual factors related to adoption and developing guides to help users enhance adoption67 Report characteristics of intervention location and delivery agent and their selection criteria22; if applicable, describe adoption rate of intervention location and delivery agent69 If applicable, describe the expertise of the delivery staff69 If applicable, describe participation rate of delivery setting/agent69 Provide information on the level of human involvement required for an SMS- and app-based intervention compared with the level of human involvement for a routine application70 Provide information on the prompts/reminders required for SMS- and app-based intervention compared with the level of prompts/reminders for a routine application70 Provide information on any interventions (including training sessions/support) that are implemented in addition to the targeted SMS- and app-based intervention70 Implementation Fidelity of intervention program, including intervention uptake, development, monitoring, and adaptation and factors influence the implementation Document standardized measures for capturing implementation fidelity, multilevel assessment of cost (both monetary and time) and adaption made for implementation67 Describe intervention content and use parameters (eg, frequency, optimal timing for use, heaviness of use)70, and frequency of inter-person and virtual sessions, if applicable. Provide information intervention costs including price for mobile phone, mobile phone data plan, and incentives for program development/participation given to staff or participants68 Record percent delivered as intended (eg, SMS sent/unsent/received/not received; any application functioning problems)22 Describe adaptations made to the intervention during implementation (eg, fitting strategies and methods to user culture)68 Report strategies to monitoring and gathering feedback (from both participants and staffs) during implementation69 Report time cost to participants and staff in addition to time used for intervention implementation. Maintenance Sustainability of intervention program at both the individual and organizational levels Document and understand dynamic, complex, and multilevel factors leading to sustainment67 Include an assessment of maintenance of the intervention outcomes (clinical and/or behavioral, such as hemoglobin A1c reduction, healthy eating) 6 mo after the completion of the intervention71 Report broader outcomes (eg, policy development associated with the intervention)57 Provide a context in which to evaluate the long-term outcome (eg, ongoing institutional and policy support for program maintenance and ongoing cost)57 Include a sustainability plan regarding how the intervention could be sustained at both the individual and organizational levels or, if applicable, provide data on the degree to which the intervention is sustained over time22,71 Dimension and component . Key issues . Overall and specific recommendations . Reach Representativeness of study sample to target population Document and understand access, awareness, and appropriateness of intervention to meet target population needs67 Compare characteristics (eg, sociodemographic, economic, and behavioral) of participants with nonparticipants or the general local population to understand the representativeness of the sample22 Include inclusion criteria (eg, health/disease conditions, such as hemoglobin A1c ≥8.0%, capable of using mobile phone) and exclusion to Provide explanation on why certain individuals were not eligible for participation22 Include recruitment setting, methods, and recruitment adaptations22 Efficacy/effectiveness Suitability and credibility of study design, data collection, and evaluation Document participants’ satisfaction, negative outcomes, and subgroup effects in addition to reporting primary outcome67 Report study design (eg, randomized controlled trial) and whether a comparison group is included Report use intention-to-treat analysis22 and missing data procedure68 Report qualitative methods to acceptability and usability of the intervention; report potential negative outcomes of the intervention22 Report subgroup effects (eg, sex, race, age, or health/disease condition that influence the intervention effect)22 Adoption Diffusion of intervention program at organizational and delivery level and factors influence the adoption Document and understand contextual factors related to adoption and developing guides to help users enhance adoption67 Report characteristics of intervention location and delivery agent and their selection criteria22; if applicable, describe adoption rate of intervention location and delivery agent69 If applicable, describe the expertise of the delivery staff69 If applicable, describe participation rate of delivery setting/agent69 Provide information on the level of human involvement required for an SMS- and app-based intervention compared with the level of human involvement for a routine application70 Provide information on the prompts/reminders required for SMS- and app-based intervention compared with the level of prompts/reminders for a routine application70 Provide information on any interventions (including training sessions/support) that are implemented in addition to the targeted SMS- and app-based intervention70 Implementation Fidelity of intervention program, including intervention uptake, development, monitoring, and adaptation and factors influence the implementation Document standardized measures for capturing implementation fidelity, multilevel assessment of cost (both monetary and time) and adaption made for implementation67 Describe intervention content and use parameters (eg, frequency, optimal timing for use, heaviness of use)70, and frequency of inter-person and virtual sessions, if applicable. Provide information intervention costs including price for mobile phone, mobile phone data plan, and incentives for program development/participation given to staff or participants68 Record percent delivered as intended (eg, SMS sent/unsent/received/not received; any application functioning problems)22 Describe adaptations made to the intervention during implementation (eg, fitting strategies and methods to user culture)68 Report strategies to monitoring and gathering feedback (from both participants and staffs) during implementation69 Report time cost to participants and staff in addition to time used for intervention implementation. Maintenance Sustainability of intervention program at both the individual and organizational levels Document and understand dynamic, complex, and multilevel factors leading to sustainment67 Include an assessment of maintenance of the intervention outcomes (clinical and/or behavioral, such as hemoglobin A1c reduction, healthy eating) 6 mo after the completion of the intervention71 Report broader outcomes (eg, policy development associated with the intervention)57 Provide a context in which to evaluate the long-term outcome (eg, ongoing institutional and policy support for program maintenance and ongoing cost)57 Include a sustainability plan regarding how the intervention could be sustained at both the individual and organizational levels or, if applicable, provide data on the degree to which the intervention is sustained over time22,71 RE-AIM: Reach, Efficacy/Effectiveness, Adoption, Implementation, and Maintenance; SMS: short message service. Open in new tab Table 3. Recommendations based on gaps identified from the RE-AIM framework evaluation Dimension and component . Key issues . Overall and specific recommendations . Reach Representativeness of study sample to target population Document and understand access, awareness, and appropriateness of intervention to meet target population needs67 Compare characteristics (eg, sociodemographic, economic, and behavioral) of participants with nonparticipants or the general local population to understand the representativeness of the sample22 Include inclusion criteria (eg, health/disease conditions, such as hemoglobin A1c ≥8.0%, capable of using mobile phone) and exclusion to Provide explanation on why certain individuals were not eligible for participation22 Include recruitment setting, methods, and recruitment adaptations22 Efficacy/effectiveness Suitability and credibility of study design, data collection, and evaluation Document participants’ satisfaction, negative outcomes, and subgroup effects in addition to reporting primary outcome67 Report study design (eg, randomized controlled trial) and whether a comparison group is included Report use intention-to-treat analysis22 and missing data procedure68 Report qualitative methods to acceptability and usability of the intervention; report potential negative outcomes of the intervention22 Report subgroup effects (eg, sex, race, age, or health/disease condition that influence the intervention effect)22 Adoption Diffusion of intervention program at organizational and delivery level and factors influence the adoption Document and understand contextual factors related to adoption and developing guides to help users enhance adoption67 Report characteristics of intervention location and delivery agent and their selection criteria22; if applicable, describe adoption rate of intervention location and delivery agent69 If applicable, describe the expertise of the delivery staff69 If applicable, describe participation rate of delivery setting/agent69 Provide information on the level of human involvement required for an SMS- and app-based intervention compared with the level of human involvement for a routine application70 Provide information on the prompts/reminders required for SMS- and app-based intervention compared with the level of prompts/reminders for a routine application70 Provide information on any interventions (including training sessions/support) that are implemented in addition to the targeted SMS- and app-based intervention70 Implementation Fidelity of intervention program, including intervention uptake, development, monitoring, and adaptation and factors influence the implementation Document standardized measures for capturing implementation fidelity, multilevel assessment of cost (both monetary and time) and adaption made for implementation67 Describe intervention content and use parameters (eg, frequency, optimal timing for use, heaviness of use)70, and frequency of inter-person and virtual sessions, if applicable. Provide information intervention costs including price for mobile phone, mobile phone data plan, and incentives for program development/participation given to staff or participants68 Record percent delivered as intended (eg, SMS sent/unsent/received/not received; any application functioning problems)22 Describe adaptations made to the intervention during implementation (eg, fitting strategies and methods to user culture)68 Report strategies to monitoring and gathering feedback (from both participants and staffs) during implementation69 Report time cost to participants and staff in addition to time used for intervention implementation. Maintenance Sustainability of intervention program at both the individual and organizational levels Document and understand dynamic, complex, and multilevel factors leading to sustainment67 Include an assessment of maintenance of the intervention outcomes (clinical and/or behavioral, such as hemoglobin A1c reduction, healthy eating) 6 mo after the completion of the intervention71 Report broader outcomes (eg, policy development associated with the intervention)57 Provide a context in which to evaluate the long-term outcome (eg, ongoing institutional and policy support for program maintenance and ongoing cost)57 Include a sustainability plan regarding how the intervention could be sustained at both the individual and organizational levels or, if applicable, provide data on the degree to which the intervention is sustained over time22,71 Dimension and component . Key issues . Overall and specific recommendations . Reach Representativeness of study sample to target population Document and understand access, awareness, and appropriateness of intervention to meet target population needs67 Compare characteristics (eg, sociodemographic, economic, and behavioral) of participants with nonparticipants or the general local population to understand the representativeness of the sample22 Include inclusion criteria (eg, health/disease conditions, such as hemoglobin A1c ≥8.0%, capable of using mobile phone) and exclusion to Provide explanation on why certain individuals were not eligible for participation22 Include recruitment setting, methods, and recruitment adaptations22 Efficacy/effectiveness Suitability and credibility of study design, data collection, and evaluation Document participants’ satisfaction, negative outcomes, and subgroup effects in addition to reporting primary outcome67 Report study design (eg, randomized controlled trial) and whether a comparison group is included Report use intention-to-treat analysis22 and missing data procedure68 Report qualitative methods to acceptability and usability of the intervention; report potential negative outcomes of the intervention22 Report subgroup effects (eg, sex, race, age, or health/disease condition that influence the intervention effect)22 Adoption Diffusion of intervention program at organizational and delivery level and factors influence the adoption Document and understand contextual factors related to adoption and developing guides to help users enhance adoption67 Report characteristics of intervention location and delivery agent and their selection criteria22; if applicable, describe adoption rate of intervention location and delivery agent69 If applicable, describe the expertise of the delivery staff69 If applicable, describe participation rate of delivery setting/agent69 Provide information on the level of human involvement required for an SMS- and app-based intervention compared with the level of human involvement for a routine application70 Provide information on the prompts/reminders required for SMS- and app-based intervention compared with the level of prompts/reminders for a routine application70 Provide information on any interventions (including training sessions/support) that are implemented in addition to the targeted SMS- and app-based intervention70 Implementation Fidelity of intervention program, including intervention uptake, development, monitoring, and adaptation and factors influence the implementation Document standardized measures for capturing implementation fidelity, multilevel assessment of cost (both monetary and time) and adaption made for implementation67 Describe intervention content and use parameters (eg, frequency, optimal timing for use, heaviness of use)70, and frequency of inter-person and virtual sessions, if applicable. Provide information intervention costs including price for mobile phone, mobile phone data plan, and incentives for program development/participation given to staff or participants68 Record percent delivered as intended (eg, SMS sent/unsent/received/not received; any application functioning problems)22 Describe adaptations made to the intervention during implementation (eg, fitting strategies and methods to user culture)68 Report strategies to monitoring and gathering feedback (from both participants and staffs) during implementation69 Report time cost to participants and staff in addition to time used for intervention implementation. Maintenance Sustainability of intervention program at both the individual and organizational levels Document and understand dynamic, complex, and multilevel factors leading to sustainment67 Include an assessment of maintenance of the intervention outcomes (clinical and/or behavioral, such as hemoglobin A1c reduction, healthy eating) 6 mo after the completion of the intervention71 Report broader outcomes (eg, policy development associated with the intervention)57 Provide a context in which to evaluate the long-term outcome (eg, ongoing institutional and policy support for program maintenance and ongoing cost)57 Include a sustainability plan regarding how the intervention could be sustained at both the individual and organizational levels or, if applicable, provide data on the degree to which the intervention is sustained over time22,71 RE-AIM: Reach, Efficacy/Effectiveness, Adoption, Implementation, and Maintenance; SMS: short message service. Open in new tab Limitations Several limitations concerning this review need to be considered. First, there was a small number of eligible studies available for this study. However, we used a validated PRISMA approach when we selected literature and extracted results from studies using RCTs or quasi-experimental designs. Authors’ comments were also incorporated into findings of the review. Small sample size and inadequate blinding of some studies in this review may still contribute to risk of bias. Larger and methodologically robust trails are very much needed. Second, it is possible that our search strategy could not identify all relevant articles. For example, we excluded articles that were published in non–English-language or trial registry data. However, we searched 3 main databases including PubMed, Embase, and Web of Science, in addition to 2 topic-specific databases, CINAHL Plus and the IEEE Xplore Digital Library. We used broad inclusion and exclusion criteria to increase the likelihood of capturing relevant studies to minimize publication bias. We also used a manual search of reference lists of eligible articles for relevant systematic reviews and narrative reviews. We addressed a focused area of mobile phone–based solutions, specifically interventions delivered by SMS and apps that are the most common in technology based behavioral interventions due to increased smartphone ownership. The exclusive review on SMS- and app-based interventions, rather than a broad area of mobile health interventions including various wearable devices, improves the transferability of our results to a prime area of smartphone-based interventions. Further, some authors did not report on specific adoption, implementation, and maintenance strategies that were in fact used in their studies, increasing the risk of reporting bias. To address this concern, we contacted the authors directly to inquire missing information regarding external validity. CONCLUSION SMS- and app-based interventions may have potential to promote diabetes self-management among patients with diabetes as access to smartphone increase; however, the population impact in the United States and other countries remains to be confirmed. Our systematic review based on the RE-AIM framework synthesized important findings from this emergent body of literature. This review is among the first to address issues in external validity of SMS- and app-based DSMES studies that is lacking from traditional reviews predominately focused on internal validity and intervention efficacy or effectiveness. Our review demonstrated that the comprehensiveness of reporting on RE-AIM criteria across the mobile phone–based DSMES studies was relatively low and with many gaps in internal validity reporting (ie, extent to which the protocol was delivered as intended) and, more so in external validity (across domains in adoption, implementation, and maintenance). Without this information, it is difficult to determine the internal validity and external validity of mobile phone–based interventions in DSMES. We encourage researchers to improve reporting around RE-AIM dimensions, specifically information on intention-to-treat analysis; mixed methods to understand acceptability, usability, and implementation of an intervention; costs; location, delivery agent selection; characteristics; and maintenance planning. Continued efforts in improving quality of research development and reporting will ensure mobile phone–based interventions to address diabetes will be broadly applicable across diverse settings and populations. FUNDING This study was supported in part by the National Institute of Diabetes and Digestive and Kidney Diseases of the National Institutes of Health (award numbers P30DK092950 and P30DK020579) and the Centers for Disease Control and Prevention (Cooperative Agreement number U48DP006395). SJP’s contribution was supported by the National Center for Advancing Translational Sciences of the National Institutes of Health under Award Number KL2 TR002346. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health. AUTHOR CONTRIBUTIONS YY designed the study, undertook the literature search and data extraction, contacted individual study authors, and wrote the article. SJP refined the coding book, verified the coding results, and assessed clinical applicability of the included studies. RCB, SAB, MK, VAF, and EJS provided critical appraisals and edits to the article. RD, KW, DAG, AT, AR, and CM provided important information regarding their individual studies. SUPPLEMENTARY MATERIAL Supplementary material is available at Journal of the American Medical Informatics Association online. Acknowledgments The authors would like to acknowledge Dr Eirik Arsand from Norwegian Centre for E-health Research, University Hospital of North Norway, Tromsø, Norway, for his information on an app that was extended from the app included in an article under our review. We would like to thank Dr Maryam Peimani from Tehran University of Medical Sciences in Iran, Ms Marcia Vervloet from the Netherlands Institute for Health Services Research in the Netherlands, and Dr Sanjay Arora from Keck School of Medicine at University of Southern California in the United States for their responses to our inquires. We would also like to thank Ms Lauren Hartz from University of Tokyo to reply our inquires in English on behalf of KM. Finally, we would like to thank Ms Allison Phad from Center for Diabetes Translation Research Washington University in St. Louis for providing her edits to the manuscript. CONFLICT OF INTEREST STATEMENT None declared. REFERENCES 1 Beck J , Greenwood DA, Blanton L, et al. . 2017 National Standards for Diabetes Self-Management Education and Support . Diabetes Educ 2017 ; 43 ( 5 ): 449 – 64 . Google Scholar Crossref Search ADS PubMed WorldCat 2 Haas L , Maryniuk M, Beck J, et al. . National standards for diabetes self-management education and support . Diabetes Educ 2012 ; 38 ( 5 ): 619 – 29 . Google Scholar Crossref Search ADS PubMed WorldCat 3 Benoit SR , Ji M, Fleming R, Philis-Tsimikas A. Predictors of dropouts from a San Diego diabetes program: a case control study . Prev Chronic Dis 2004 ; 1 : A10 . Google Scholar PubMed OpenURL Placeholder Text WorldCat 4 Gucciardi E , Demelo M, Offenheim A, Stewart DE. Factors contributing to attrition behavior in diabetes self-management programs: a mixed method approach . BMC Health Serv Res 2008 ; 8 ( 1 ): 33 . Google Scholar Crossref Search ADS PubMed WorldCat 5 Peyrot M , Rubin RR. Access to diabetes self-management education . Diabetes Educ 2008 ; 34 ( 1 ): 90 – 7 . Google Scholar Crossref Search ADS PubMed WorldCat 6 Li R , Shrestha SS, Lipman R, et al. . Diabetes self-management education and training among privately insured persons with newly diagnosed diabetes–United States, 2011-2012 . MMWR Morb Mortal Wkly Rep 2014 ; 63 ( 46 ): 1045 – 9 . Google Scholar PubMed OpenURL Placeholder Text WorldCat 7 Pew Research Center. Mobile Technology Fact Sheet . Washington, DC : Pew Research Center ; 2019 . Google Scholar Google Preview OpenURL Placeholder Text WorldCat COPAC 8 Greenwood DA , Gee PM, Fatkin KJ, Peeples M. A systematic review of reviews evaluating technology-enabled diabetes self-management education and support . J Diabetes Sci Technol 2017 ; 11 ( 5 ): 1015 – 27 . Google Scholar Crossref Search ADS PubMed WorldCat 9 Arambepola C , Ricci-Cabello I, Manikavasagam P, Roberts N, French DP, Farmer A. The impact of automated brief messages promoting lifestyle changes delivered via mobile devices to people with type 2 diabetes: a systematic literature review and meta-analysis of controlled trials . J Med Internet Res 2016 ; 18 ( 4 ): e86 . Google Scholar Crossref Search ADS PubMed WorldCat 10 Arnhold M , Quade M, Kirch W. Mobile applications for diabetics: a systematic review and expert-based usability evaluation considering the special requirements of diabetes patients age 50 years or older . J Med Internet Res 2014 ; 16 ( 4 ): e104 . Google Scholar Crossref Search ADS PubMed WorldCat 11 Choi J , Lee JH, Vittinghoff E, Fukuoka Y. mHealth physical activity intervention: a randomized pilot study in physically inactive pregnant women . Matern Child Health J 2016 ; 20 ( 5 ): 1091 – 101 . Google Scholar Crossref Search ADS PubMed WorldCat 12 Coughlin SS. Mobile technology for self-monitoring of blood glucose among patients with type 2 diabetes mellitus . Mhealth 2017 ; 3 : 47 . Google Scholar Crossref Search ADS PubMed WorldCat 13 Haider R , Sudini L, Chow CK, Cheung NW. Mobile phone text messaging in improving glycaemic control for patients with type 2 diabetes mellitus: a systematic review and meta-analysis . Diabetes Res Clin Pract 2019 ; 150 : 27 – 37 . Google Scholar Crossref Search ADS PubMed WorldCat 14 Hall AK , Cole-Lewis H, Bernhardt JM. Mobile text messaging for health: a systematic review of reviews . Annu Rev Public Health 2015 ; 36 ( 1 ): 393 – 415 . Google Scholar Crossref Search ADS PubMed WorldCat 15 Kankanhalli A , Shin J, Oh H. Mobile-based interventions for dietary behavior change and health outcomes: scoping review . JMIR Mhealth Uhealth 2019 ; 7 ( 1 ): e11312 . Google Scholar Crossref Search ADS PubMed WorldCat 16 Orr JA , King RJ. Mobile phone SMS messages can enhance healthy behaviour: a meta-analysis of randomised controlled trials . Health Psychol Rev 2015 ; 9 ( 4 ): 397 – 416 . Google Scholar Crossref Search ADS PubMed WorldCat 17 Park JYE , Li J, Howren A, Tsao NW, De Vera M. Mobile phone apps targeting medication adherence: quality assessment and content analysis of user reviews . JMIR Mhealth Uhealth 2019 ; 7 ( 1 ): e11919 . Google Scholar Crossref Search ADS PubMed WorldCat 18 Yoshida Y , Boren SA, Soares J, Popescu M, Nielson SD, Simoes EJ. Effect of health information technologies on glycemic control among patients with type 2 diabetes . Curr Diab Rep 2018 ; 18 ( 12 ): 130 . Google Scholar Crossref Search ADS PubMed WorldCat 19 Glasgow RE , Klesges LM, Dzewaltowski DA, Bull SS, Estabrooks P. The future of health behavior change research: what is needed to improve translation of research into health promotion practice? Ann Behav Med 2004 ; 27 ( 1 ): 3 – 12 . Google Scholar Crossref Search ADS PubMed WorldCat 20 Glasgow RE , Vogt TM, Boles SM. Evaluating the public health impact of health promotion interventions: the RE-AIM framework . Am J Public Health 1999 ; 89 ( 9 ): 1322 – 7 . Google Scholar Crossref Search ADS PubMed WorldCat 21 Akers JD , Estabrooks PA, Davy BM. Translational research: bridging the gap between long-term weight loss maintenance research and practice . J Am Diet Assoc 2010 ; 110 ( 10 ): 1511 – 22, 22 e1–3 . Google Scholar Crossref Search ADS PubMed WorldCat 22 Blackman KC , Zoellner J, Berrey LM, et al. . Assessing the internal and external validity of mobile health physical activity promotion interventions: a systematic literature review using the RE-AIM framework . J Med Internet Res 2013 ; 15 ( 10 ): e224 . Google Scholar Crossref Search ADS PubMed WorldCat 23 Bull SS , Gillette C, Glasgow RE, Estabrooks P. Work site health promotion research: to what extent can we generalize the results and what is needed to translate research to practice? Health Educ Behav 2003 ; 30 ( 5 ): 537 – 49 . Google Scholar Crossref Search ADS PubMed WorldCat 24 Dzewaltowski DA , Estabrooks PA, Klesges LM, Bull S, Glasgow RE. Behavior change intervention research in community settings: how generalizable are the results? Health Promot Int 2004 ; 19 ( 2 ): 235 – 45 . Google Scholar Crossref Search ADS PubMed WorldCat 25 Klesges LM , Dzewaltowski DA, Glasgow RE. Review of external validity reporting in childhood obesity prevention research . Am J Prev Med 2008 ; 34 ( 3 ): 216 – 23 . Google Scholar Crossref Search ADS PubMed WorldCat 26 McGoey T , Root Z, Bruner MW, Law B. Evaluation of physical activity interventions in children via the reach, efficacy/effectiveness, adoption, implementation, and maintenance (RE-AIM) framework: A systematic review of randomized and non-randomized trials . Prev Med 2016 ; 82 : 8 – 19 . Google Scholar Crossref Search ADS PubMed WorldCat 27 White SM , McAuley E, Estabrooks PA, Courneya KS. Translating physical activity interventions for breast cancer survivors into practice: an evaluation of randomized controlled trials . Ann Behav Med 2009 ; 37 ( 1 ): 10 – 9 . Google Scholar Crossref Search ADS PubMed WorldCat 28 Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA). PRISMA 2009 Checklist . Helsinki, Finland : PRISMA ; 2009 . Google Scholar Google Preview OpenURL Placeholder Text WorldCat COPAC 29 Sutter JD. The top 10 tech trends of 2009 . CNN. http://www.cnn.com/2009/TECH/12/22/top.tech.trends.2009/index.html Accessed •••. 30 Hussein WI , Hasan K, Jaradat AA. Effectiveness of mobile phone short message service on diabetes mellitus management; the SMS-DM study . Diabetes Res Clin Pract 2011 ; 94 ( 1 ): e24 – 6 . Google Scholar Crossref Search ADS PubMed WorldCat 31 Shetty AS , Chamukuttan S, Nanditha A, Raj RK, Ramachandran A. Reinforcement of adherence to prescription recommendations in Asian Indian diabetes patients using short message service (SMS)–a pilot study . J Assoc Physicians India 2011 ; 59 : 711 – 4 . Google Scholar PubMed OpenURL Placeholder Text WorldCat 32 Waki K , Fujita H, Uchimura Y, et al. . DialBetics: a novel smartphone-based self-management support system for type 2 diabetes patients . J Diabetes Sci Technol 2014 ; 8 ( 2 ): 209 – 15 . Google Scholar Crossref Search ADS PubMed WorldCat 33 Zhou W , Chen M, Yuan J, Sun Y. Welltang—A smart phone-based diabetes management application—Improves blood glucose control in Chinese people with diabetes . Diabetes Res Clin Pract 2016 ; 116 : 105 – 10 . Google Scholar Crossref Search ADS PubMed WorldCat 34 Zimmerman LP , Goel S, Sathar S, et al. . A novel patient recruitment strategy: patient selection directly from the community through linkage to clinical data . Appl Clin Inform 2018 ; 9 ( 1 ): 114 – 21 . Google Scholar Crossref Search ADS PubMed WorldCat 35 Sun C , Sun L, Xi S, et al. . Mobile phone–based telemedicine practice in older chinese patients with type 2 diabetes mellitus: randomized controlled trial . JMIR Mhealth Uhealth 2019 ; 7 ( 1 ): e10664 . Google Scholar Crossref Search ADS PubMed WorldCat 36 Allen K , Zoellner J, Motley M, Estabrooks PA. Understanding the internal and external validity of health literacy interventions: a systematic literature review using the RE-AIM framework . J Health Commun 2011 ; 16 (sup3 ): 55 – 72 . Google Scholar Crossref Search ADS WorldCat 37 Dobson R , Whittaker R, Jiang Y, et al. . Text message-based diabetes self-management support (SMS4BG): study protocol for a randomised controlled trial . Trials 2016 ; 17 ( 1 ): 179 . Google Scholar Crossref Search ADS PubMed WorldCat 38 Kleinman NJ , Shah A, Shah S, Phatak S, Viswanathan V. Improved medication adherence and frequency of blood glucose self-testing using an m-health platform versus usual care in a multisite randomized clinical trial among people with type 2 diabetes in India . Telemed J E Health 2017 ; 23 ( 9 ): 733 – 40 . Google Scholar Crossref Search ADS PubMed WorldCat 39 Quinn CC , Shardell MD, Terrin ML, Barr EA, Ballew SH, Gruber-Baldini AL. Cluster-randomized trial of a mobile phone personalized behavioral intervention for blood glucose control . Diabetes Care 2011 ; 34 ( 9 ): 1934 – 42 . Google Scholar Crossref Search ADS PubMed WorldCat 40 Rossi MC , Nicolucci A, Di Bartolo P, et al. . Diabetes Interactive Diary: a new telemedicine system enabling flexible diet and insulin therapy while improving quality of life: an open-label, international, multicenter, randomized study . Diabetes Care 2010 ; 33 ( 1 ): 109 – 15 . Google Scholar Crossref Search ADS PubMed WorldCat 41 Holmen H , Torbjornsen A, Wahl AK, et al. . A mobile health intervention for self-management and lifestyle change for persons with type 2 diabetes, part 2: one-year results from the Norwegian randomized controlled trial renewing healTH . JMIR Mhealth Uhealth 2014 ; 2 ( 4 ): e57 . Google Scholar Crossref Search ADS PubMed WorldCat 42 Torbjornsen A , Jenum AK, Smastuen MC, et al. . A low-intensity mobile health intervention with and without health counseling for persons with type 2 diabetes, part 1: baseline and short-term results from a randomized controlled trial in the Norwegian part of renewing health . JMIR Mhealth Uhealth 2014 ; 2 ( 4 ): e52 . Google Scholar Crossref Search ADS PubMed WorldCat 43 Arora S , Peters AL, Burner E, Lam CN, Menchine M. Trial to examine text message-based mHealth in emergency department patients with diabetes (TExT-MED): a randomized controlled trial . Ann Emerg Med 2014 ; 63 ( 6 ): 745 – 54.e6 . Google Scholar Crossref Search ADS PubMed WorldCat 44 Kumar S , Moseson H, Uppal J, Juusola JL. A diabetes mobile app with in-app coaching from a certified diabetes educator reduces A1C for individuals with type 2 diabetes . Diabetes Educ 2018 ; 44 ( 3 ): 226 – 36 . Google Scholar Crossref Search ADS PubMed WorldCat 45 Vervloet M , van Dijk L, de Bakker DH, et al. . Short- and long-term effects of real-time medication monitoring with short message service (SMS) reminders for missed doses on the refill adherence of people with Type 2 diabetes: evidence from a randomized controlled trial . Diabet Med 2014 ; 31 ( 7 ): 821 – 8 . Google Scholar Crossref Search ADS PubMed WorldCat 46 Wu CJ , Sung HC, Chang AM, Atherton J, Kostner K, McPhail SM. Cardiac-diabetes self-management program for Australians and Taiwanese: a randomized blocked design study . Nurs Health Sci 2017 ; 19 ( 3 ): 307 – 15 . Google Scholar Crossref Search ADS PubMed WorldCat 47 Agarwal P , Mukerji G, Desveaux L, et al. . Mobile app for improved self-management of type 2 diabetes: multicenter pragmatic randomized controlled trial . JMIR Mhealth Uhealth 2019 ; 7 ( 1 ): e10321 . Google Scholar Crossref Search ADS PubMed WorldCat 48 Bauer V , Goodman N, Lapin B, et al. . Text messaging to improve disease management in patients with painful diabetic peripheral neuropathy . Diabetes Educ 2018 ; 44 ( 3 ): 237 – 48 . Google Scholar Crossref Search ADS PubMed WorldCat 49 Kim HS , Choi W, Baek EK, et al. . Efficacy of the smartphone-based glucose management application stratified by user satisfaction . Diabetes Metab J 2014 ; 38 ( 3 ): 204 – 10 . Google Scholar Crossref Search ADS PubMed WorldCat 50 Greenwood DA , Hankins AI, Parise CA, Spier V, Olveda J, Buss KA. A comparison of in-person, telephone, and secure messaging for type 2 diabetes self-management support . Diabetes Educ 2014 ; 40 ( 4 ): 516 – 25 . Google Scholar Crossref Search ADS PubMed WorldCat 51 Brandon DT , Isaac LA, LaVeist TA. The legacy of Tuskegee and trust in medical care: is Tuskegee responsible for race differences in mistrust of medical care? J Natl Med Assoc 2005 ; 97 ( 7 ): 951 – 6 . Google Scholar PubMed OpenURL Placeholder Text WorldCat 52 Carpenter WR , Tyree S, Wu Y, et al. . A surveillance system for monitoring, public reporting, and improving minority access to cancer clinical trials . Clin Trials 2012 ; 9 ( 4 ): 426 – 35 . Google Scholar Crossref Search ADS PubMed WorldCat 53 Greiner KA , Friedman DB, Adams SA, et al. . Effective recruitment strategies and community-based participatory research: community networks program centers’ recruitment in cancer prevention studies . Cancer Epidemiol Biomarkers Prev 2014 ; 23 ( 3 ): 416 – 23 . Google Scholar Crossref Search ADS PubMed WorldCat 54 Glasgow RE , Bull SS, Gillette C, Klesges LM, Dzewaltowski DA. Behavior change intervention research in healthcare settings: a review of recent reports with emphasis on external validity . Am J Prev Med 2002 ; 23 ( 1 ): 62 – 9 . Google Scholar Crossref Search ADS PubMed WorldCat 55 Antikainen I , Ellis R. A RE-AIM evaluation of theory-based physical activity interventions . J Sport Exerc Psychol 2011 ; 33 ( 2 ): 198 – 214 . Google Scholar Crossref Search ADS PubMed WorldCat 56 Estabrooks P , Dzewaltowski DA, Glasgow RE, Klesges LM. Reporting of validity from school health promotion studies published in 12 leading journals, 1996-2000 . J Sch Health 2003 ; 73 ( 1 ): 21 – 8 . Google Scholar Crossref Search ADS PubMed WorldCat 57 Gaglio B , Shoup JA, Glasgow RE. The RE-AIM framework: a systematic review of use over time . Am J Public Health 2013 ; 103 ( 6 ): e38 – 46 . Google Scholar Crossref Search ADS PubMed WorldCat 58 Estabrooks PA , Allen KC. Updating, employing, and adapting: a commentary on What does it mean to “employ” the RE-AIM model . Eval Health Prof 2013 ; 36 ( 1 ): 67 – 72 . Google Scholar Crossref Search ADS PubMed WorldCat 59 Joe J , Demiris G. Older adults and mobile phones for health: a review . J Biomed Inform 2013 ; 46 ( 5 ): 947 – 54 . Google Scholar Crossref Search ADS PubMed WorldCat 60 May CR , Finch TL, Cornford J, et al. . Integrating telecare for chronic disease management in the community: what needs to be done? BMC Health Serv Res 2011 ; 11 ( 1 ): 131 . Google Scholar Crossref Search ADS PubMed WorldCat 61 Nielsen JA , Mengiste SA. Analysing the diffusion and adoption of mobile IT across social worlds . Health Informatics J 2014 ; 20 ( 2 ): 87 – 103 . Google Scholar Crossref Search ADS PubMed WorldCat 62 Hebert MA , Korabek B, Scott RE. Moving research into practice: a decision framework for integrating home telehealth into chronic illness care . Int J Med Inform 2006 ; 75 ( 12 ): 786 – 94 . Google Scholar Crossref Search ADS PubMed WorldCat 63 Ross J , Stevenson F, Lau R, Murray E. Factors that influence the implementation of e-health: a systematic review of systematic reviews (an update) . Implement Sci 2016 ; 11 ( 1 ): 146 . Google Scholar Crossref Search ADS PubMed WorldCat 64 Benavides-Vaello S , Strode A, Sheeran BC. Using technology in the delivery of mental health and substance abuse treatment in rural communities: a review . J Behav Health Serv Res 2013 ; 40 ( 1 ): 111 – 20 . Google Scholar Crossref Search ADS PubMed WorldCat 65 Jennett PA , Scott RE, Affleck Hall L, et al. . Policy implications associated with the socioeconomic and health system impact of telehealth: a case study from Canada . Telemed J E Health 2004 ; 10 ( 1 ): 77 – 83 . Google Scholar Crossref Search ADS PubMed WorldCat 66 Matthew-Maich N , Harris L, Ploeg J, et al. . Designing, implementing, and evaluating mobile health technologies for managing chronic conditions in older adults: a scoping review . JMIR Mhealth Uhealth 2016 ; 4 ( 2 ): e29 . Google Scholar Crossref Search ADS PubMed WorldCat 67 Glasgow RE , Harden SM, Gaglio B, et al. . RE-AIM planning and evaluation framework: adapting to new science and practice with a 20-year review . Front Public Health 2019 ; 7 : 64 . Google Scholar Crossref Search ADS PubMed WorldCat 68 Stellefson M , Chaney B, Barry AE, et al. . Web 2.0 chronic disease self-management for older adults: a systematic review . J Med Internet Res 2013 ; 15 ( 2 ): e35 . Google Scholar Crossref Search ADS PubMed WorldCat 69 Molleman GR , Ploeg MA, Hosman CM, Peters LH. Preffi 2.0—a quality assessment tool . Promot Educ 2006 ; 13 ( 1 ): 9 – 14 . Google Scholar Crossref Search ADS PubMed WorldCat 70 Eysenbach G , Group C. E-CONSORT-E-HEALTH: improving and standardizing evaluation reports of Web-based and mobile health interventions . J Med Internet Res 2011 ; 13 ( 4 ): e126 . Google Scholar Crossref Search ADS PubMed WorldCat 71 Kessler RS , Purcell EP, Glasgow RE, Klesges LM, Benkeser RM, Peek CJ. What does it mean to “employ” the RE-AIM model? Eval Health Prof 2013 ; 36 ( 1 ): 44 – 66 . Google Scholar Crossref Search ADS PubMed WorldCat © The Author(s) 2020. Published by Oxford University Press on behalf of the American Medical Informatics Association. All rights reserved. For permissions, please email: journals.permissions@oup.com This article is published and distributed under the terms of the Oxford University Press, Standard Journals Publication Model (https://academic.oup.com/journals/pages/open_access/funder_policies/chorus/standard_publication_model) http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png Journal of the American Medical Informatics Association Oxford University Press http://www.deepdyve.com/lp/oxford-university-press/using-the-re-aim-framework-to-evaluate-internal-and-external-validity-kNKWyVorVH

Loading next page...

References (77)

Jayne Orr, R. King (2015)
Mobile phone SMS messages can enhance healthy behaviour: a meta-analysis of randomised controlled trials
Health Psychology Review, 9
P. Agarwal, MD Mukerji, MD MSc, PhD Ivers, PhD Bhattacharyya, P. Md, MSc Md, PhD Bouck, T. Jamieson, MD Onabajo, MSc Cooper, H. Marani, MSc Jeffs, F. PhD (2019)
Mobile App for Improved Self-Management of Type 2 Diabetes: Multicenter Pragmatic Randomized Controlled Trial
JMIR mHealth and uHealth, 7
(2013)
Using Technology in the Delivery of Mental Health and Substance Abuse Treatment in Rural Communities : A Review
(2019)
citation_publisher=Pew Research Center, Washington, DC; Mobile Technology Fact Sheet
Haas (2012)
National standards for diabetes self-management education and support
Diabetes Educ, 38
R. Glasgow, S. Harden, B. Gaglio, Borsika Rabin, Matthew Smith, G. Porter, M. Ory, P. Estabrooks (2019)
RE-AIM Planning and Evaluation Framework: Adapting to New Science and Practice With a 20-Year Review
Frontiers in Public Health, 7
P. Estabrooks, D. Dzewaltowski, R. Glasgow, L. Klesges (2003)
Reporting of validity from school health promotion studies published in 12 leading journals, 1996-2000.
The Journal of school health, 73 1
Charlene Quinn, Michelle Shardell, Michael Terrin, Erik Barr, S. Ballew, Ann Gruber-Baldini (2011)
Cluster-Randomized Trial of a Mobile Phone Personalized Behavioral Intervention for Blood Glucose Control
Diabetes Care, 34
(2019)
Mobile Technology Fact Sheet
R. Glasgow, S. Bull, C. Gillette, L. Klesges, D. Dzewaltowski (2002)
Behavior change intervention research in healthcare settings: a review of recent reports with emphasis on external validity.
American journal of preventive medicine, 23 1
Li (2014)
Diabetes self-management education and training among privately insured persons with newly diagnosed diabetes–United States, 2011-2012
MMWR Morb Mortal Wkly Rep, 63
J. Joe, G. Demiris (2013)
Older adults and mobile phones for health: A review
Journal of biomedical informatics, 46 5
M. Stellefson, B. Chaney, A. Barry, E. Chavarria, Bethany Tennant, K. Walsh-Childers, P. Sriram, Justin Zagora (2013)
Web 2.0 Chronic Disease Self-Management for Older Adults: A Systematic Review
Journal of Medical Internet Research, 15
E. Gucciardi, Margaret Demelo, Ana Offenheim, D. Stewart (2008)
Factors contributing to attrition behavior in diabetes self-management programs: A mixed method approach
BMC Health Services Research, 8
D. Dzewaltowski, P. Estabrooks, L. Klesges, S. Bull, R. Glasgow (2004)
Behavior change intervention research in community settings: how generalizable are the results?
Health promotion international, 19 2
R. Kessler, E. Purcell, R. Glasgow, L. Klesges, R. Benkeser, C. Peek (2013)
What Does It Mean to “Employ” the RE-AIM Model?
Evaluation & the Health Professions, 36
M. Rossi, A. Nicolucci, P. Bartolo, D. Bruttomesso, A. Girelli, F. Ampudia, D. Kerr, A. Ceriello, Carmen Mayor, F. Pellegrini, D. Horwitz, G. Vespasiani (2009)
Diabetes Interactive Diary: A New Telemedicine System Enabling Flexible Diet and Insulin Therapy While Improving Quality of Life
Diabetes Care, 33
Beck (2017)
2017 National Standards for Diabetes Self-Management Education and Support
Diabetes Educ, 43
Kacie Blackman, J. Zoellner, Leanna Berrey, Ramine Alexander, Jason Fanning, J. Hill, P. Estabrooks (2013)
Assessing the Internal and External Validity of Mobile Health Physical Activity Promotion Interventions: A Systematic Literature Review Using the RE-AIM Framework
Journal of Medical Internet Research, 15
Lindsay Zimmerman, Satyender Goel, Shazia Sathar, Charon Gladfelter, A. Onate, Lindsey Kane, S. Sital, Jasmin Phua, Paris Davis, Helen Margellos-Anast, D. Meltzer, Tamar Polonsky, R. Shah, W. Trick, F. Ahmad, A. Kho (2018)
A Novel Patient Recruitment Strategy: Patient Selection Directly from the Community through Linkage to Clinical Data
Applied Clinical Informatics, 09
G. Eysenbach (2011)
CONSORT-EHEALTH: Improving and Standardizing Evaluation Reports of Web-based and Mobile Health Interventions
Journal of Medical Internet Research, 13
C. May, T. Finch, J. Cornford, C. Exley, C. Gately, S. Kirk, K. Jenkings, Janice Osbourne, A. Robinson, A. Rogers, Robert Wilson, F. Mair (2011)
Integrating telecare for chronic disease management in the community: What needs to be done?
BMC Health Services Research, 11
M. Arnhold, M. Quade, W. Kirch (2014)
Mobile Applications for Diabetics: A Systematic Review and Expert-Based Usability Evaluation Considering the Special Requirements of Diabetes Patients Age 50 Years or Older
Journal of Medical Internet Research, 16
G. Molleman, Machteld Ploeg, C. Hosman, Louk Peters (2006)
Preffi 2.0- a Quality Assessment Tool
Global Health Promotion, 13
Hun‐Sung Kim, Wona Choi, Eun Baek, Y. Kim, So Yang, I. Choi, K. Yoon, Jae-Hyoung Cho (2014)
Efficacy of the Smartphone-Based Glucose Management Application Stratified by User Satisfaction
Diabetes & Metabolism Journal, 38
Rabbia Haider, Rabbia Haider, Likhitha Sudini, C. Chow, N. Cheung (2019)
Mobile phone text messaging in improving glycaemic control for patients with type 2 diabetes mellitus: A systematic review and meta-analysis.
Diabetes research and clinical practice, 150
JiWon Choi, Ji Lee, E. Vittinghoff, Yoshimi Fukuoka (2016)
mHealth Physical Activity Intervention: A Randomized Pilot Study in Physically Inactive Pregnant Women
Maternal and Child Health Journal, 20
N. Kleinman, Avani Shah, Sanjiv Shah, S. Phatak, V. Viswanathan (2017)
Improved Medication Adherence and Frequency of Blood Glucose Self-Testing Using an m-Health Platform Versus Usual Care in a Multisite Randomized Clinical Trial Among People with Type 2 Diabetes in India.
Telemedicine journal and e-health : the official journal of the American Telemedicine Association, 23 9
Nancy Matthew-Maich, L. Harris, J. Ploeg, M. Markle-Reid, R. Valaitis, S. Ibrahim, A. Gafni, S. Isaacs (2016)
Designing, Implementing, and Evaluating Mobile Health Technologies for Managing Chronic Conditions in Older Adults: A Scoping Review
JMIR mHealth and uHealth, 4
Weibin Zhou, Min Chen, Jingyun Yuan, Yan Sun (2016)
Welltang - A smart phone-based diabetes management application - Improves blood glucose control in Chinese people with diabetes.
Diabetes research and clinical practice, 116
S. White, E. McAuley, P. Estabrooks, K. Courneya (2009)
Translating Physical Activity Interventions for Breast Cancer Survivors into Practice: An Evaluation of Randomized Controlled Trials
Annals of Behavioral Medicine, 37
Joni Beck, Deborah Greenwood, Lori Blanton, Sandra Bollinger, M. Butcher, J. Condon, M. Cypress, Priscilla Faulkner, Amy Fischl, Theresa Francis, L. Kolb, J. Lavin-Tompkins, J. MacLeod, M. Maryniuk, C. Mensing, E. Orzeck, David Pope, Jodi Pulizzi, Ardis Reed, A. Rhinehart, L. Siminerio, Jing Wang (2017)
2017 National Standards for Diabetes Self-Management Education and Support
Diabetes Care, 40
Deborah Greenwood, Andrea Hankins, C. Parise, Valerie Spier, Joan Olveda, Kimberly Buss (2014)
A Comparison of In-person, Telephone, and Secure Messaging for Type 2 Diabetes Self-Management Support
The Diabetes Educator, 40
(2009)
citation_publisher=PRISMA, Helsinki, Finland; PRISMA 2009 Checklist
P. Jennett, R. Scott, L. Hall, D. Hailey, A. Ohinmaa, Christian Anderson, R. Thomas, B. Young, D. Lorenzetti (2004)
Policy implications associated with the socioeconomic and health system impact of telehealth: a case study from Canada.
Telemedicine journal and e-health : the official journal of the American Telemedicine Association, 10 1
W. Hussein, Khadija Hasan, Ahmed Jaradat (2011)
Effectiveness of mobile phone short message service on diabetes mellitus management; the SMS-DM study.
Diabetes research and clinical practice, 94 1
S. Bull, C. Gillette, R. Glasgow, P. Estabrooks (2003)
Work Site Health Promotion Research: To what Extent can we Generalize the Results and what is Needed to Translate Research to Practice?
Health Education & Behavior, 30
R. Glasgow, T. Vogt, S. Boles (1999)
Evaluating the public health impact of health promotion interventions: the RE-AIM framework.
American journal of public health, 89 9
W. Carpenter, S. Tyree, Yang Wu, A. Meyer, L. DiMartino, L. Zullig, P. Godley (2012)
A surveillance system for monitoring, public reporting, and improving minority access to cancer clinical trials
Clinical Trials, 9
R. Dobson, R. Whittaker, Yannan Jiang, M. Shepherd, R. Maddison, Karen Carter, R. Cutfield, C. McNamara, M. Khanolkar, R. Murphy (2016)
Text message-based diabetes self-management support (SMS4BG): study protocol for a randomised controlled trial
Trials, 17
A. Kankanhalli, Jieun Shin, Hyelim Oh (2018)
Mobile-Based Interventions for Dietary Behavior Change and Health Outcomes: Scoping Review
JMIR mHealth and uHealth, 7
S. Benoit, M. Ji, Regina Fleming, A. Philis-Tsimikas (2004)
Predictors of Dropouts From a San Diego Diabetes Program: A Case Control Study
Preventing Chronic Disease, 1
J. Ross, F. Stevenson, R. Lau, E. Murray (2016)
Factors that influence the implementation of e-health: a systematic review of systematic reviews (an update)
Implementation Science : IS, 11
A. Shetty, S. Chamukuttan, A. Nanditha, R. Raj, Ambady Ramachandran (2011)
Reinforcement of adherence to prescription recommendations in Asian Indian diabetes patients using short message service (SMS)--a pilot study.
The Journal of the Association of Physicians of India, 59
Victoria Bauer, N. Goodman, B. Lapin, Camille Cooley, E. Wang, Terri Craig, S. Glosner, M. Juhn, J. Cappelleri, A. Sadosky, Christopher Masi (2018)
Text Messaging to Improve Disease Management in Patients With Painful Diabetic Peripheral Neuropathy
The Diabetes Educator, 44
M. Peyrot, R. Rubin (2008)
Access to Diabetes Self-management Education
The Diabetes Educator, 34
M. Hebert, Barbara Korabek, Richard Scott (2006)
Moving research into practice: A decision framework for integrating home telehealth into chronic illness care
International journal of medical informatics, 75 12
J. Nielsen, S. Mengiste (2014)
Analysing the diffusion and adoption of mobile IT across social worlds
Health Informatics Journal, 20
M. Vervloet, L. Dijk, D. Bakker, P. Souverein, J. Santen-Reestman, B. Vlijmen, M. Aarle, L. Hoek, Marcel Bouvy (2014)
Short‐ and long‐term effects of real‐time medication monitoring with short message service (SMS) reminders for missed doses on the refill adherence of people with Type 2 diabetes: evidence from a randomized controlled trial
Diabetic Medicine, 31
C. Wu, H. Sung, A. Chang, J. Atherton, K. Kostner, S. McPhail (2017)
Cardiac‐diabetes self‐management program for Australians and Taiwanese: A randomized blocked design study
Nursing & Health Sciences, 19
L. Klesges, D. Dzewaltowski, R. Glasgow (2008)
Review of external validity reporting in childhood obesity prevention research.
American journal of preventive medicine, 34 3
Kacie Allen, J. Zoellner, Monica Motley, P. Estabrooks (2011)
Understanding the Internal and External Validity of Health Literacy Interventions: A Systematic Literature Review Using the RE-AIM Framework
Journal of Health Communication, 16
D. Brandon, L. Isaac, T. Laveist (2005)
The legacy of Tuskegee and trust in medical care: is Tuskegee responsible for race differences in mistrust of medical care?
Journal of the National Medical Association, 97 7
The top 10 tech trends of 2009 . CNN
K. Greiner, D. Friedman, S. Adams, C. Gwede, P. Cupertino, K. Engelman, C. Meade, J. Hébert (2014)
Effective Recruitment Strategies and Community-Based Participatory Research: Community Networks Program Centers' Recruitment in Cancer Prevention Studies
Cancer Epidemiology, Biomarkers & Prevention, 23
Shefali Kumar, H. Moseson, Jas Uppal, J. Juusola (2018)
A Diabetes Mobile App With In-App Coaching From a Certified Diabetes Educator Reduces A1C for Individuals With Type 2 Diabetes
The Diabetes Educator, 44
J. Park, Jenny Li, A. Howren, N. Tsao, M. Vera (2019)
Mobile Phone Apps Targeting Medication Adherence: Quality Assessment and Content Analysis of User Reviews
JMIR mHealth and uHealth, 7
Amanda Hall, H. Cole-Lewis, J. Bernhardt (2015)
Mobile text messaging for health: a systematic review of reviews.
Annual review of public health, 36
Joni Beck, Deborah Greenwood, Lori Blanton, Sandra Bollinger, M. Butcher, J. Condon, M. Cypress, Priscilla Faulkner, Amy Fischl, Theresa Francis, L. Kolb, J. Lavin-Tompkins, Janice MacLeod, M. Maryniuk, C. Mensing, E. Orzeck, David Pope, Jodi Pulizzi, Ardis Reed, A. Rhinehart, Linda Siminerio, Jing Wang (2012)
2017 National Standards for Diabetes Self-Management Education and Support
Diabetes Spectrum : A Publication of the American Diabetes Association, 30
Tara McGoey, Zachary Root, M. Bruner, Barbi Law (2015)
Evaluation of physical activity interventions in children via the reach, efficacy/effectiveness, adoption, implementation, and maintenance (RE-AIM) framework: A systematic review of randomized and non-randomized trials.
Preventive medicine, 82
Rui Li, S. Shrestha, R.D. Lipman, N. Burrows, L. Kolb, Stephanie Rutledge (2014)
Diabetes Self-Management Education and Training Among Privately Insured Persons with Newly Diagnosed Diabetes — United States, 2011–2012
MMWR. Morbidity and Mortality Weekly Report, 63
Yilin Yoshida, S. Boren, J. Soares, M. Popescu, Stephen Nielson, E. Simões (2018)
Effect of Health Information Technologies on Glycemic Control Among Patients with Type 2 Diabetes
Current Diabetes Reports, 18
S. Coughlin (2017)
Mobile technology for self-monitoring of blood glucose among patients with type 2 diabetes mellitus.
mHealth, 3
B. Gaglio, J. Shoup, R. Glasgow (2013)
The RE-AIM framework: a systematic review of use over time.
American journal of public health, 103 6
P. Estabrooks, Kacie Allen (2013)
Updating, Employing, and Adapting
Evaluation & the Health Professions, 36
Iina Antikainen, R. Ellis (2011)
A RE-AIM evaluation of theory-based physical activity interventions.
Journal of sport & exercise psychology, 33 2
A. Iris, Kon Moshe, H. Arnold, L. PaulvanMinnen (2015)
Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) flow chart.
PLOS ONE
L. Haas, M. Maryniuk, Joni Beck, C. Cox, Paulina Duker, Laura Edwards, E. Fisher, Lenita Hanson, Daniel Kent, L. Kolb, S. Mclaughlin, E. Orzeck, J. Piette, A. Rhinehart, R. Rothman, Sara Sklaroff, D. Tomky, Gretchen Youssef (2012)
National Standards for Diabetes Self-Management Education and Support
Diabetes Care, 35
R. Glasgow, L. Klesges, D. Dzewaltowski, S. Bull, P. Estabrooks (2004)
The future of health behavior change research: What is needed to improve translation of research into health promotion practice?
Annals of Behavioral Medicine, 27
Deborah Greenwood, P. Gee, Kathy Fatkin, M. Peeples (2017)
A Systematic Review of Reviews Evaluating Technology-Enabled Diabetes Self-Management Education and Support
Journal of Diabetes Science and Technology, 11
Heidi Holmen, A. Torbjørnsen, A. Wahl, A. Jenum, M. Småstuen, E. Årsand, L. Ribu (2014)
A Mobile Health Intervention for Self-Management and Lifestyle Change for Persons With Type 2 Diabetes, Part 2: One-Year Results From the Norwegian Randomized Controlled Trial RENEWING HEALTH
JMIR mHealth and uHealth, 2
S. Arora, A. Peters, E. Burner, C. Lam, M. Menchine (2014)
Trial to examine text message-based mHealth in emergency department patients with diabetes (TExT-MED): a randomized controlled trial.
Annals of emergency medicine, 63 6
A. Torbjørnsen, A. Jenum, M. Småstuen, E. Årsand, Heidi Holmen, A. Wahl, L. Ribu (2014)
A Low-Intensity Mobile Health Intervention With and Without Health Counseling for Persons With Type 2 Diabetes, Part 1: Baseline and Short-Term Results From a Randomized Controlled Trial in the Norwegian Part of RENEWING HEALTH
JMIR mHealth and uHealth, 2
Chenglin Sun, Lin Sun, Shugang Xi, Hong Zhang, Huan Wang, Yakun Feng, Yufeng Deng, Haimin Wang, Xianchao Xiao, Gang Wang, Yuan Gao, Guixia Wang (2019)
Mobile Phone–Based Telemedicine Practice in Older Chinese Patients with Type 2 Diabetes Mellitus: Randomized Controlled Trial
JMIR mHealth and uHealth, 7
C. Arambepola, I. Ricci-Cabello, P. Manikavasagam, N. Roberts, D. French, A. Farmer (2016)
The Impact of Automated Brief Messages Promoting Lifestyle Changes Delivered Via Mobile Devices to People with Type 2 Diabetes: A Systematic Literature Review and Meta-Analysis of Controlled Trials
Journal of Medical Internet Research, 18
K. Waki, H. Fujita, Y. Uchimura, K. Omae, E. Aramaki, Shigeko Kato, Hanae Lee, Haruka Kobayashi, T. Kadowaki, K. Ohe (2014)
DialBetics: A Novel Smartphone-based Self-management Support System for Type 2 Diabetes Patients.
Journal of diabetes science and technology, 8 2
Jeremy Akers, P. Estabrooks, B. Davy (2010)
Translational research: bridging the gap between long-term weight loss maintenance research and practice.
Journal of the American Dietetic Association, 110 10

Publisher: Oxford University Press
Copyright: © The Author(s) 2020. Published by Oxford University Press on behalf of the American Medical Informatics Association. All rights reserved. For permissions, please email: journals.permissions@oup.com
ISSN: 1067-5027
eISSN: 1527-974X
DOI: 10.1093/jamia/ocaa041
Publisher site: See Article on Publisher Site

Abstract

Abstract Objective We evaluated the extent to which studies that tested short message service (SMS)– and application (app)-based interventions for diabetes self-management education and support (DSMES) report on factors that inform both internal and external validity as measured by the RE-AIM (Reach, Efficacy/Effectiveness, Adoption, Implementation, and Maintenance) framework. Materials and Methods We systematically searched PubMed, Embase, Web of Science, CINAHL (Cumulative Index of Nursing and Allied Health Literature), and IEEE Xplore Digital Library for articles from January 1, 2009, to February 28, 2019. We carried out a multistage screening process followed by email communications with study authors for missing or discrepant information. Two independent coders coded eligible articles using a 23-item validated data extraction tool based on the RE-AIM framework. Results Twenty studies (21 articles) were included in the analysis. The comprehensiveness of reporting on the RE-AIM criteria across the SMS- and app-based DSMES studies was low. With respect to internal validity, most interventions were well described and primary clinical or behavioral outcomes were measured and reported. However, gaps exist in areas of attrition, measures of potential negative outcomes, the extent to which the protocol was delivered as intended, and description on delivery agents. Likewise, we found limited information on external validity indicators across adoption, implementation, and maintenance domains. Conclusions Reporting gaps were found in internal validity but more so in external validity in the current SMS- and app-based DSMES literature. Because most studies in this review were efficacy studies, the generalizability of these interventions cannot be determined. Future research should adopt the RE-AIM dimensions to improve the quality of reporting and enhance the likelihood of translating research to practice. mobile phone–based intervention, diabetes self-management education and support, SMS, app, RE-AIM INTRODUCTION Diabetes self-management education and support (DSMES) is a necessary component in diabetes care. DSMES plays an important role in glycemic management and preventing or delaying diabetes-related complications when administered alongside medical care and management.1,2 However, many barriers such as competing priorities, transportation barriers, underperceived seriousness of diabetes, and lack of accessible services discourage patients from obtaining traditional institution-based self-management education or training.3–6 There is a need for developing alternative DMSES interventions that are accessible and low cost for populations with diabetes. Mobile phone technology can serve as an effective platform to deliver DSMES because of its ubiquitous availability and adoption by all populations.7,8 The 2 most common tools used in mobile phone–based interventions are short message service (SMS) and smartphone applications (apps). SMS and app-based interventions have been applied in multiple behavioral interventions, including promoting physical activity, tracking healthy eating, monitoring blood glucose, taking medication, monitoring complications, and problem solving, and show promising results.9–17 One recent meta-analysis and systematic review focused on multiple strategies of health technology interventions in diabetes management identified a larger effect size in mobile phone–based interventions compared with other forms of technological interventions, such as computer- or Internet-based interventions.18 However, despite the increased popularity and demonstrated efficacy of mobile phone–based interventions in diabetes management, it remains largely unknown whether these interventions can be translated beyond the research setting and be broadly adopted in clinical and other settings. Additionally, review studies in this area have concentrated on reporting issues through the lens of internal validity.10,13 To date, no systematic review has evaluated domains of external validity and identified gaps that could inform the generalizability and translatability of mobile phone–based interventions in the specific area of DSMES. To bridge efficacy and effectiveness, or research and practice, Glasgow et al developed the RE-AIM (Reach, Efficacy/Effectiveness, Adoption, Implementation, and Maintenance) framework.19,20 The RE-AIM framework consists of 5 dimensions: reach into the target population and representativeness of the study sample; efficacy or effectiveness of the intervention on the primary outcome tested under either optimal or real-world conditions, effect on quality of life, and avoidance of unintended or negative consequences; adoption by target setting, organizations, and staff; implementation on consistency and cost of delivery; and maintenance of intervention effect over time.19,20 The RE-AIM framework has been utilized in reporting internal and external validity across numerous behavioral intervention studies, including physical activity, weight loss maintenance, and health literacy.21–27 These findings provide recommendations and future directions to improve the quality of reporting and to enhance the likelihood of translating research to practice. The goal of this review is to evaluate the extent to which studies, both randomized controlled trials (RCTs) and non-RCTs, testing SMS and app-based interventions to facilitate DSMES report on factors that inform both internal and external validity of the intervention. In addition to data extracted from literature databases, we also included data directly collected from study authors via emails. Recommendations to improve the quality of reporting and the likelihood of broad dissemination of effective SMS- and app-based interventions are provided based on 2 sources of data and other relevant evaluations of behaviors interventions. MATERIALS AND METHODS Database search and study inclusion We adopted the validated PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) approach to conduct the literature review. PRISMA is an evidence-based minimum set of items for reporting in systematic reviews.28 We systematically searched PubMed for eligible articles from January 1, 2009 to February 28, 2019, using combinations of the following MeSH (Medical Subject Headings) (M) and text word (TW) search terms: (1) Diabetes Mellitus Type 2 (M), non insulin dependent diabetes (T), T2DM (T); (2) self-management (M), self care (M); and (3) mHealth (TW), mobile health (TW), cell phone (M), cell phone (TW), mobile phone (TW), short message service (TW), and text messaging (M). Similar searches were conducted in Embase, Web of Science, CINAHL (Cumulative Index of Nursing and Allied Health Literature), and the IEEE Xplore Digital Library. We also performed supplementary searches using the reference lists of eligible articles and relevant systematic review and other review articles. We decided to search from 2009, as 2009 marked a shift in technology from basic mobile phones to smartphones, and therefore the use of apps.29 Eligibility criteria Studies were included if they were peer-reviewed studies testing the effect of SMS- or app-based interventions on DSMES among patients with type 2 diabetes. We excluded studies that did not use SMS or an app as the primary intervention (ie, used only phone calls or wearables devices for remote monitoring). In addition, we excluded qualitative research (ie, exclusive evaluations on patients’ attitudes or perceptions on interventions), studies of cost-effectiveness analyses, and protocol articles. Study screening We used a multistage screening process in which search results were first pooled and transferred to EndNote software (Clarivate Analytics, Philadelphia, PA) for duplicate removal. Next, 2 reviewers (Y.Y. and S.J.P.) independently screened article titles and abstracts for relevance. Studies with “type 2 diabetes” or “non–insulin dependent diabetes” in the titles and abstracts and included a mobile phone–based intervention for DSMES were selected for full text review. Finally, the 2 reviewers conducted full text reviews of selected articles to confirm eligibility. Articles extracted from reference lists underwent an identical process (Figure 1). Figure 1. Open in new tabDownload slide Article screening process (PRISMA [Preferred Reporting Items for Systematic Reviews and Meta-Analyses] 2018 flow diagram). CINAHL: Cumulative Index of Nursing and Allied Health Literature. Figure 1. Open in new tabDownload slide Article screening process (PRISMA [Preferred Reporting Items for Systematic Reviews and Meta-Analyses] 2018 flow diagram). CINAHL: Cumulative Index of Nursing and Allied Health Literature. RE-AIM coding and scoring A 23-item validated data extraction tool based on the RE-AIM framework was used to assess the extent of reporting study elements related to internal and external validity. Table 1 presents details on components that assess the RE-AIM dimensions. “Yes” or “no” were used to code the presence or absence of the RE-AIM components outlined in Supplementary Table 2. Each article was coded by 2 researchers independently; disagreement in coding was discussed until consensus was reached (Y.Y. and S.J.P.). Table 1. Evaluation dimensions and measures of the RE-AIM framework Dimension . Componenta . Internal or external validity indicator . Description22 . Reach The number, proportion, and representativeness of individuals who are willing to participate in a given intervention. Methods to identify target population Internal Description of the process by which the target population was identified in the intervention Inclusion criteria Internal Description of characteristics of the target population that were used to determine if a potential participant was eligible in the intervention Exclusion criteria Internal Description of characteristics of the target population that prevent a potential participant from being eligible to participate Sample size Internal The number of people who agree to participate Participation rate Internal Sample size divided by the target population denominator Representativeness Internal, external Comparison of characteristics of the study participants in comparison to the target population Efficacy/effectiveness The impact of an intervention on important outcomes, including potential negative effects. Measures/results at least 1 follow-up Internal The study outcomes are measured at a time point after baseline Intent-to-treat analysis utilized Internal Analyzing participants in trials in the groups to which they were randomized, regardless of whether they received or adhered to the allocated intervention Satisfactionb or potential negative outcomes Internal Measuring acceptability and usability of the intervention in participants; evaluate unintended consequences that may result from the intervention Attrition Internal The proportion that was lost of follow-up or dropped out of the intervention Adoption The number, proportion, and characteristics of adopting settings and interventions agents Description of intervention locationc Internal, external Description of characteristics of the location of the intervention Description of staff who delivered the programc Internal, external Description of characteristics of staff who delivered the intervention Method to identify staff who delivered the intervention External Description of the process by which the staff was identified for participation in the study Level of expertise of delivery agent External Training and educational background among those who delivered the intervention Inclusion/exclusion criteria of delivery agent or setting External Description of the eligibility criteria of the setting/agent Adoption rate of delivery agent or setting External The number of participating delivery settings or agents divided by the number of eligible and approached delivery settings or agents Implementation The extent to which the intervention is delivered as intended (eg, information on duration and frequency of intervention, fidelity to the intervention protocol, and cost including time and money) Intervention duration and frequency Internal Length of the intervention (ie, days, weeks, months, and length of each intervention contact) and number of contacts with participants Extent protocol delivered as intended (%) Internal Description of the fidelity to the intervention protocol Measures of cost of implementation Internal, external The costs including both money and time of delivery across all levels of implementation Maintenance The extent to which a participant maintains the change due to intervention and an intervention becomes institutionalized or part of the routine organization practices Assessed outcomes ≥6 mo postintervention External Description of follow-up outcome measures of individuals at some duration after intervention was terminated Indicators of program level maintenance External Description of program continuation after completion of the research study Measures of cost of maintenance External The ongoing cost of maintaining delivery across all levels of the intervention Program adopted in other setting/populations External Description of the intervention being adopted beyond the original setting and population Dimension . Componenta . Internal or external validity indicator . Description22 . Reach The number, proportion, and representativeness of individuals who are willing to participate in a given intervention. Methods to identify target population Internal Description of the process by which the target population was identified in the intervention Inclusion criteria Internal Description of characteristics of the target population that were used to determine if a potential participant was eligible in the intervention Exclusion criteria Internal Description of characteristics of the target population that prevent a potential participant from being eligible to participate Sample size Internal The number of people who agree to participate Participation rate Internal Sample size divided by the target population denominator Representativeness Internal, external Comparison of characteristics of the study participants in comparison to the target population Efficacy/effectiveness The impact of an intervention on important outcomes, including potential negative effects. Measures/results at least 1 follow-up Internal The study outcomes are measured at a time point after baseline Intent-to-treat analysis utilized Internal Analyzing participants in trials in the groups to which they were randomized, regardless of whether they received or adhered to the allocated intervention Satisfactionb or potential negative outcomes Internal Measuring acceptability and usability of the intervention in participants; evaluate unintended consequences that may result from the intervention Attrition Internal The proportion that was lost of follow-up or dropped out of the intervention Adoption The number, proportion, and characteristics of adopting settings and interventions agents Description of intervention locationc Internal, external Description of characteristics of the location of the intervention Description of staff who delivered the programc Internal, external Description of characteristics of staff who delivered the intervention Method to identify staff who delivered the intervention External Description of the process by which the staff was identified for participation in the study Level of expertise of delivery agent External Training and educational background among those who delivered the intervention Inclusion/exclusion criteria of delivery agent or setting External Description of the eligibility criteria of the setting/agent Adoption rate of delivery agent or setting External The number of participating delivery settings or agents divided by the number of eligible and approached delivery settings or agents Implementation The extent to which the intervention is delivered as intended (eg, information on duration and frequency of intervention, fidelity to the intervention protocol, and cost including time and money) Intervention duration and frequency Internal Length of the intervention (ie, days, weeks, months, and length of each intervention contact) and number of contacts with participants Extent protocol delivered as intended (%) Internal Description of the fidelity to the intervention protocol Measures of cost of implementation Internal, external The costs including both money and time of delivery across all levels of implementation Maintenance The extent to which a participant maintains the change due to intervention and an intervention becomes institutionalized or part of the routine organization practices Assessed outcomes ≥6 mo postintervention External Description of follow-up outcome measures of individuals at some duration after intervention was terminated Indicators of program level maintenance External Description of program continuation after completion of the research study Measures of cost of maintenance External The ongoing cost of maintaining delivery across all levels of the intervention Program adopted in other setting/populations External Description of the intervention being adopted beyond the original setting and population RE-AIM: Reach, Efficacy/Effectiveness, Adoption, Implementation, and Maintenance. a Components were derived from a reliable extraction tool.18–20,23 b Components were included to ensure relevancy with the mobile health interventions. c Components were informed by other RE-AIM framework reviews of health behavior interventions.21,48 Open in new tab Table 1. Evaluation dimensions and measures of the RE-AIM framework Dimension . Componenta . Internal or external validity indicator . Description22 . Reach The number, proportion, and representativeness of individuals who are willing to participate in a given intervention. Methods to identify target population Internal Description of the process by which the target population was identified in the intervention Inclusion criteria Internal Description of characteristics of the target population that were used to determine if a potential participant was eligible in the intervention Exclusion criteria Internal Description of characteristics of the target population that prevent a potential participant from being eligible to participate Sample size Internal The number of people who agree to participate Participation rate Internal Sample size divided by the target population denominator Representativeness Internal, external Comparison of characteristics of the study participants in comparison to the target population Efficacy/effectiveness The impact of an intervention on important outcomes, including potential negative effects. Measures/results at least 1 follow-up Internal The study outcomes are measured at a time point after baseline Intent-to-treat analysis utilized Internal Analyzing participants in trials in the groups to which they were randomized, regardless of whether they received or adhered to the allocated intervention Satisfactionb or potential negative outcomes Internal Measuring acceptability and usability of the intervention in participants; evaluate unintended consequences that may result from the intervention Attrition Internal The proportion that was lost of follow-up or dropped out of the intervention Adoption The number, proportion, and characteristics of adopting settings and interventions agents Description of intervention locationc Internal, external Description of characteristics of the location of the intervention Description of staff who delivered the programc Internal, external Description of characteristics of staff who delivered the intervention Method to identify staff who delivered the intervention External Description of the process by which the staff was identified for participation in the study Level of expertise of delivery agent External Training and educational background among those who delivered the intervention Inclusion/exclusion criteria of delivery agent or setting External Description of the eligibility criteria of the setting/agent Adoption rate of delivery agent or setting External The number of participating delivery settings or agents divided by the number of eligible and approached delivery settings or agents Implementation The extent to which the intervention is delivered as intended (eg, information on duration and frequency of intervention, fidelity to the intervention protocol, and cost including time and money) Intervention duration and frequency Internal Length of the intervention (ie, days, weeks, months, and length of each intervention contact) and number of contacts with participants Extent protocol delivered as intended (%) Internal Description of the fidelity to the intervention protocol Measures of cost of implementation Internal, external The costs including both money and time of delivery across all levels of implementation Maintenance The extent to which a participant maintains the change due to intervention and an intervention becomes institutionalized or part of the routine organization practices Assessed outcomes ≥6 mo postintervention External Description of follow-up outcome measures of individuals at some duration after intervention was terminated Indicators of program level maintenance External Description of program continuation after completion of the research study Measures of cost of maintenance External The ongoing cost of maintaining delivery across all levels of the intervention Program adopted in other setting/populations External Description of the intervention being adopted beyond the original setting and population Dimension . Componenta . Internal or external validity indicator . Description22 . Reach The number, proportion, and representativeness of individuals who are willing to participate in a given intervention. Methods to identify target population Internal Description of the process by which the target population was identified in the intervention Inclusion criteria Internal Description of characteristics of the target population that were used to determine if a potential participant was eligible in the intervention Exclusion criteria Internal Description of characteristics of the target population that prevent a potential participant from being eligible to participate Sample size Internal The number of people who agree to participate Participation rate Internal Sample size divided by the target population denominator Representativeness Internal, external Comparison of characteristics of the study participants in comparison to the target population Efficacy/effectiveness The impact of an intervention on important outcomes, including potential negative effects. Measures/results at least 1 follow-up Internal The study outcomes are measured at a time point after baseline Intent-to-treat analysis utilized Internal Analyzing participants in trials in the groups to which they were randomized, regardless of whether they received or adhered to the allocated intervention Satisfactionb or potential negative outcomes Internal Measuring acceptability and usability of the intervention in participants; evaluate unintended consequences that may result from the intervention Attrition Internal The proportion that was lost of follow-up or dropped out of the intervention Adoption The number, proportion, and characteristics of adopting settings and interventions agents Description of intervention locationc Internal, external Description of characteristics of the location of the intervention Description of staff who delivered the programc Internal, external Description of characteristics of staff who delivered the intervention Method to identify staff who delivered the intervention External Description of the process by which the staff was identified for participation in the study Level of expertise of delivery agent External Training and educational background among those who delivered the intervention Inclusion/exclusion criteria of delivery agent or setting External Description of the eligibility criteria of the setting/agent Adoption rate of delivery agent or setting External The number of participating delivery settings or agents divided by the number of eligible and approached delivery settings or agents Implementation The extent to which the intervention is delivered as intended (eg, information on duration and frequency of intervention, fidelity to the intervention protocol, and cost including time and money) Intervention duration and frequency Internal Length of the intervention (ie, days, weeks, months, and length of each intervention contact) and number of contacts with participants Extent protocol delivered as intended (%) Internal Description of the fidelity to the intervention protocol Measures of cost of implementation Internal, external The costs including both money and time of delivery across all levels of implementation Maintenance The extent to which a participant maintains the change due to intervention and an intervention becomes institutionalized or part of the routine organization practices Assessed outcomes ≥6 mo postintervention External Description of follow-up outcome measures of individuals at some duration after intervention was terminated Indicators of program level maintenance External Description of program continuation after completion of the research study Measures of cost of maintenance External The ongoing cost of maintaining delivery across all levels of the intervention Program adopted in other setting/populations External Description of the intervention being adopted beyond the original setting and population RE-AIM: Reach, Efficacy/Effectiveness, Adoption, Implementation, and Maintenance. a Components were derived from a reliable extraction tool.18–20,23 b Components were included to ensure relevancy with the mobile health interventions. c Components were informed by other RE-AIM framework reviews of health behavior interventions.21,48 Open in new tab Missing data inquiry from study authors After the article screening and data extraction, we contacted corresponding authors by email when interventions included in the final review had missing or discrepant information. We introduced the purpose of our study and provided inquiries about missing or discrepant data according to RE-AIM criteria as well as additional questions regarding study implementation and generalizability. If there was no response to our initial email, reminders were sent at 7, 14, and 30 days. RESULTS Study characteristics Our search yielded 20 studies (Supplementary Table 2). Of these, 14 were RCTs and 2 were pragmatic trials. Four were quasi-experimental studies, among which 1 was an intention-to-treat trial. Nine studies centered on app-based interventions, and 11 focused on SMS-based or the combination of SMS and phone call–based interventions. Eight studies were conducted in urban settings and 1 included both urban and rural settings. Five studies were conducted in the United States, 3 were conducted in Europe (Italy, Spain, and England), 4 were conducted in East Asia, 2 were conducted in India, 2 were conducted in Iran, 1 had locations in both Australia and Taiwan, 1 were conducted in Canada, 1 were conducted in New Zealand, and 1 were conducted in Bahrain. A total of 12 authors from 12 studies responded to our email inquiries, with 9 addressing our inquires (Supplementary Table 2). RE-AIM analysis Reach All 20 studies reported specific inclusion and exclusion criteria and methods to identify target population. All 20 studies provided sample size (median sample size = 140; range, 32-427). Five studies did not report on number of eligible participants30–35; therefore, participation rate could not be calculated. Among 16 studies that provided the participation rates, median participation rate was 83% (range, 5.7%-100%). No studies in this review explicitly described characteristics of both participants and nonparticipants; therefore, we were unable to make conclusions about representativeness of the samples (Table 2). Table 2. Proportion of mobile health intervention on DSMES reporting RE-AIM framework dimensions and components (n = 20) Dimension and componenta . Proportion (%) . Notes . Reach Methods to identify target population 100 Inclusion criteria 100 Exclusion criteria 94.4 Sample size 100 Median sample size = 140 (range, 32-427) Participation rate 70 Median rate 83% (range, 5.7%-100%) Representativeness 0 Efficacy/effectiveness Measures/results at least 1 follow-up 100 Intention-to-treat analysis utilized 45 Satisfactionb or potential negative outcomes 25 Attrition 14.8 Median attrition rate 15% (range, 2%-35%) Qualitive methods to measure efficacy/effectiveness 50 Adoption Description of intervention locationc 90 Description of staff who delivered the programc 0 Method to identify staff who delivered the intervention 0 Level of expertise of delivery agent 45 Inclusion/exclusion criteria of delivery agent or setting 5.5 (setting); 0 (staff) Adoption rate of delivery agent or setting 0 Implementation Intervention duration and frequency 100 Median follow-up 4.5 months, (range, 3-22 months) Extent protocol delivered as intended 30 Measures of cost of implementation 15 2 studies reported monetary cost, 1 study reported time cost Maintenance Assessed outcomes ≥6 mo postintervention 5 Indicators of program level maintenance 0 Measures of cost of maintenance 0 Program adopted in other setting/populations 20 Dimension and componenta . Proportion (%) . Notes . Reach Methods to identify target population 100 Inclusion criteria 100 Exclusion criteria 94.4 Sample size 100 Median sample size = 140 (range, 32-427) Participation rate 70 Median rate 83% (range, 5.7%-100%) Representativeness 0 Efficacy/effectiveness Measures/results at least 1 follow-up 100 Intention-to-treat analysis utilized 45 Satisfactionb or potential negative outcomes 25 Attrition 14.8 Median attrition rate 15% (range, 2%-35%) Qualitive methods to measure efficacy/effectiveness 50 Adoption Description of intervention locationc 90 Description of staff who delivered the programc 0 Method to identify staff who delivered the intervention 0 Level of expertise of delivery agent 45 Inclusion/exclusion criteria of delivery agent or setting 5.5 (setting); 0 (staff) Adoption rate of delivery agent or setting 0 Implementation Intervention duration and frequency 100 Median follow-up 4.5 months, (range, 3-22 months) Extent protocol delivered as intended 30 Measures of cost of implementation 15 2 studies reported monetary cost, 1 study reported time cost Maintenance Assessed outcomes ≥6 mo postintervention 5 Indicators of program level maintenance 0 Measures of cost of maintenance 0 Program adopted in other setting/populations 20 DSMES: diabetes self-management education and support; RE-AIM: Reach, Efficacy/Effectiveness, Adoption, Implementation, and Maintenance. a Components were derived from a reliable extraction tool.19–21,24 b Components were included to ensure relevancy with the mobile health interventions. c Components were informed by other RE-AIM framework reviews of health behavior interventions.22,36 Open in new tab Table 2. Proportion of mobile health intervention on DSMES reporting RE-AIM framework dimensions and components (n = 20) Dimension and componenta . Proportion (%) . Notes . Reach Methods to identify target population 100 Inclusion criteria 100 Exclusion criteria 94.4 Sample size 100 Median sample size = 140 (range, 32-427) Participation rate 70 Median rate 83% (range, 5.7%-100%) Representativeness 0 Efficacy/effectiveness Measures/results at least 1 follow-up 100 Intention-to-treat analysis utilized 45 Satisfactionb or potential negative outcomes 25 Attrition 14.8 Median attrition rate 15% (range, 2%-35%) Qualitive methods to measure efficacy/effectiveness 50 Adoption Description of intervention locationc 90 Description of staff who delivered the programc 0 Method to identify staff who delivered the intervention 0 Level of expertise of delivery agent 45 Inclusion/exclusion criteria of delivery agent or setting 5.5 (setting); 0 (staff) Adoption rate of delivery agent or setting 0 Implementation Intervention duration and frequency 100 Median follow-up 4.5 months, (range, 3-22 months) Extent protocol delivered as intended 30 Measures of cost of implementation 15 2 studies reported monetary cost, 1 study reported time cost Maintenance Assessed outcomes ≥6 mo postintervention 5 Indicators of program level maintenance 0 Measures of cost of maintenance 0 Program adopted in other setting/populations 20 Dimension and componenta . Proportion (%) . Notes . Reach Methods to identify target population 100 Inclusion criteria 100 Exclusion criteria 94.4 Sample size 100 Median sample size = 140 (range, 32-427) Participation rate 70 Median rate 83% (range, 5.7%-100%) Representativeness 0 Efficacy/effectiveness Measures/results at least 1 follow-up 100 Intention-to-treat analysis utilized 45 Satisfactionb or potential negative outcomes 25 Attrition 14.8 Median attrition rate 15% (range, 2%-35%) Qualitive methods to measure efficacy/effectiveness 50 Adoption Description of intervention locationc 90 Description of staff who delivered the programc 0 Method to identify staff who delivered the intervention 0 Level of expertise of delivery agent 45 Inclusion/exclusion criteria of delivery agent or setting 5.5 (setting); 0 (staff) Adoption rate of delivery agent or setting 0 Implementation Intervention duration and frequency 100 Median follow-up 4.5 months, (range, 3-22 months) Extent protocol delivered as intended 30 Measures of cost of implementation 15 2 studies reported monetary cost, 1 study reported time cost Maintenance Assessed outcomes ≥6 mo postintervention 5 Indicators of program level maintenance 0 Measures of cost of maintenance 0 Program adopted in other setting/populations 20 DSMES: diabetes self-management education and support; RE-AIM: Reach, Efficacy/Effectiveness, Adoption, Implementation, and Maintenance. a Components were derived from a reliable extraction tool.19–21,24 b Components were included to ensure relevancy with the mobile health interventions. c Components were informed by other RE-AIM framework reviews of health behavior interventions.22,36 Open in new tab Effectiveness All 20 studies reported on findings for at least 1 follow-up measurement. Six (33.3%) studies had more than 1 follow-up.31,37–42 Nine (45%) studies reported on intention-to-treat analysis.32,38–47 Eight (40%) provided information on missing data or imputation measures.37,39–45,47 The median attrition rate was 15% (range, 2%-35%). Five (25%) studies reported satisfaction or potential negative outcomes.37,40–43,48 Ten (50%) studies reported including qualitative methods to measure efficacy or effectiveness of their studies. Among these, 7 used satisfaction surveys or patients’ feedback from real-time monitoring systems to understand participants’ satisfaction and acceptability of the intervention.32,33,37,38,40–45,49 One study reported intermittent technical challenges faced by few participants,37 and another study reported intervention burden or poor response to participant requests as reasons for dropout from the study.42 Information on adverse outcomes related to communication mode such as misinformation, confusion, or data errors were not reported in any studies (Table 2). Adoption Ten of 20 (50%) studies provided information on adoption items.30,37–39,43–46,48,50 Most study participants (n = 18; 90%) were recruited in primary or secondary care settings and utilized physician or self-referral. One study recruited participants online,44 and another identified potential eligible participants using automatic search from anonymous pharmacy data of 40 pharmacies.45 Four studies (20%) provided brief descriptions of characteristics of the recruitment settings.37,38,43,48 Regarding intervention delivery, 10 (50%) studies reported level of staff expertise. Three studies used diabetes educators,39,44,50 1 used nurses,48 1 used physicians,40 1 used a combination of clinicians and diabetes educators,30 1 used a combination of nurses and research staff,46 1 used a combination of nurse and nutritionist,41,42 and 2 used only research staff to facilitate intervention delivery.38,47 No study provided characteristics, method to identify, eligibility, or adoption rate of intervention staff. Even though most studies reached patients in a primary or secondary care setting, the intervention may be delivered in nonclinical settings (eg, home) remotely though app and text. Only one study39 reported on eligibility for setting selection, method to identify setting, and setting adoption rate. However, this setting was the recruitment setting rather the intervention setting39 (Table 2). Implementation All 20 studies reported study duration (median length of follow-up = 4.5 months; range, 3-22 months). One study did not report intervention frequency or dosage.38 In terms of intervention fidelity, 6 (30%) studies reported that the intervention protocol was delivered as intended.31,37,40–43,48 One study author replied by email that their intervention was not carried out as intended.50 One study was an intention-to-treat trial,44 and 10 study authors did not respond to our inquiry on adoption components. Two studies measured monetary cost of the intervention,37,43 and 1 measured time cost.50 Two studies reported providing incentives to intervention staff,41,42,44 and 2 provided incentives to participants (Table 2).43,48 Maintenance Only one study measured outcomes at 6-month follow-up after the study completed, according to study author’s response via email.37 No study reported indicators of program-level maintenance nor cost of maintenance. Authors from 4 studies informed us that their interventions have been adopted in other settings or patient populations41–43,45 or were being considered for a large scale implementation (Table 2).37 Applicability of studies conducted in the United States Of the studies conducted in the United States, app-based intervention studies either provided mobile phones with unlimited service plans39 or required patients to own an iPhone,44 whereas SMS-based interventions required patients to own cellphones capable of receiving text messages.48,50 Most studies had predominantly White non-Hispanic participants, with one study including predominantly Hispanic participants,43 and another study had 39.3% African American participants.39 One study identified that patient messages and diabetes educator responses were in the electronic health records, where primary care physicians (PCPs) could access to all messages.50 In a study by Bauer et al,48 physicians received information on diagnosis and management of diabetic neuropathy as they exclusively recruited participants with peripheral diabetes neuropathy; however, change in treatment secondary to SMS or app was not tracked. Quinn et al39 compared the control group with 3 additional groups: (1) an app-based intervention group with no PCP involvement, (2) an app-based intervention group in which PCPs received unanalyzed raw data, and (3) an app-based intervention group in which PCPs received summarized report by fax or email with treatment recommendations. Findings indicate that addition of clinical decision support did not differ from an app-based intervention group with no physician involvement.39 Two studies had no physician involvement.43,44 Overall, mobile phone–based DSMES interventions in the United States did not provide actionable summarized data in the electronic health record for PCPs to provide clinical decision support. Connecting mobile phone–based data to clinical management Four studies conducted outside of the United States reported some form of provider management of mobile phone–based patient entered data. One study conducted in China provided patients with an action plan as previsit summaries for physician visits.33 Another Chinese study also provided guidance for blood glucose monitoring, diet, and exercise.35 Another study conducted in India had a web portal and smartphone app for physicians to communicate treatment titration or recommendations.38 One study from Bahrain gave patients the mobile number of the diabetes educators’ and physicians’ to allow for texting between visits.30 A study from Korea had medical staff from a hospital provide tailored recommendations based on patients’ self-monitoring data of glucose and blood pressure.49 DISCUSSION Discussion notes for main results This study used the RE-AIM framework to systematically review the state of research on SMS- and app-based interventions targeting DSMES from both internal and external validity perspectives. Methods to target populations and selection criteria were reported across studies. The participation rate (average at 70%) is similar to other behavioral interventions (average at 76%) that include physical activity promotion or smoking cessation at individual level.19 This participation rate is encouraging and suggests that SMS- and app-based DMSE programs are appealing to patients. However, it is important to note that most studies recruited participants at academic medical centers, hospitals systems, or other institutions focused on clinical care. Previous research has demonstrated that recruitment at clinical settings may offer increased access to patients who are ready to participate in trials because of better information provided by clinicians regarding the study.34,51–53 As a result, patients with no access to clinical care, compounded by low health literacy, mistrust of the system, or lack of information or awareness of research opportunities, are likely to be excluded.34 Additionally, no study in this review described the target population or indicated representativeness of the study sample to a larger population. Many mobile phone–based interventions in behavioral change fall short on describing the target population, raising concerns of generalizability of these interventions to varying sociodemographic groups.21,22,27,36,54 Similarly, the convenience sampling that interventions employed also challenges the understanding of whether the intervention is reaching subgroups of a population and those individuals that could benefit the most.22,34 Efficacy or effectiveness of SMS- and app-based interventions on diabetes disease or behavioral outcomes was reported in all studies, while information on maintenance of the changes was absent. This finding is consistent with interventions in other areas.36,55,56,21,24 Less than half of the studies performed an intention-to-treat analysis. The rest of the studies presented results of those who completed the follow-up. Whether the SMS- or app-based interventions produce lasting effects is questionable because most studies did not examine maintenance at least 6 months past an intervention. Researchers in a previous review focusing on promoting physical activity through mobile health technology suggested that mobile phone–based interventions are a relatively new area of research that the studies still emphasized to determine whether the interventions can initiate change.22 The researchers also indicated that mobile phone–based interventions may reduce the likelihood of maintaining disease or behavioral outcome changes, as fast advancements in technology could make current interventions obsolete.22 Moreover, the potential of technical problems may reduce motivation and discourage engagement.22 These reasons could in part explain the lack of description on maintenance in SMS- and app-based intervention studies in this review. Additionally, determinants of efficacy or effectiveness remain largely unexplained. Only 5 studies reported using qualitative methods to measure efficacy or effectiveness. The degree to which patients found mobile phone–based interventions to be acceptable, feasible, and sustainable to use are not documented, which hinders the understanding of the potential long-term effects of those interventions. Organizational or delivery-level aspects of the RE-AIM framework have been historically underreported across behavioral interventions.57 This is also the case for SMS- and app-based DSMES research. We found very limited descriptions of the methods used to engage those who delivered the intervention or description of their characteristics, the extent to which the intervention was delivered as intended, and if any adaptions were made to the intervention during the study period. Furthermore, costs across the RE-AIM framework are also important to inform dissemination but are often missed in reporting.58 In mobile phone–based interventions specially, previous research has suggested other costs in addition to implementation costs need to be documented. This includes costs associated with recruitment, equipment, technology (eg, mobile phone, service plan, technical maintenance), and even future cost of continuing to use the service.22 Information on cost can help to determine the generatability or replicability of an intervention. The most obvious omission in reporting is maintenance at the organizational level. Few research projects have the resources to ensure that their interventions can be sustained at the organizational level. Moreover, readiness to adopt and implement mobile phone–based programs are also hindered by financial resources, policies, and workplace culture.59,60 Indeed, in email conversations with study authors, only one author informed us that their SMS-based diabetes self-management program is being considered for national implementation.37 Mobile phone–based programs have the potential to gather large amounts of health data that can be used to better inform interventions and clinical management plans.61 These data may also meet the interests of major government stakeholders looking for fiscal efficiencies in healthcare delivery.61 However, most mobile-based interventions fail to move beyond the pilot or efficacy trial phase, owing to barriers to implementation and sustainability. Previous research discussion indicates that compared with other industries, health care is relatively reluctant to adopt new technologies because of resistance to change within organizations, absence or inadequacy of policies, and funding issues.60,62–65 Ross et al63 reported that a frequent reason for unsuccessful implementation of a new technology is that it does not fit well with work practices or daily clinical work. They also suggested that unless a technology is adaptable to fit with roles, tasks, and workflows of clinicians and there is access to dedicated technical support staff, resistance to implementing a new technology will remain a challenge.63 More importantly, intervention researchers need to understand the standards and policies regarding data safety and privacy, professional liability, and potential pitfalls of data sharing between systems and organizations.64,65 Another major implementation barrier is the termination of funding support, as the additional costs of privacy and security testing, ongoing technology support and development, and software maintenance do not fit well with government supported funding cycles for research and development.60,66 In our review, termination of funding is also one of the most frequently reported implementation barriers by study authors. Recommendations for future research Table 3 provides recommendations for future research to improve the assessment and reporting on individual and organizational level factors that will support the internal and external validity of SMS- and app-based intervention in DSMES. We constructed this table using evidence from the current review as well as recommendations from other relevant evaluations on behavioral interventions.22,57,67–71 In addition to these recommendations, efforts from other stakeholders including funding organizations and practitioners should also be in place to promote translatability of mobile phone–based solutions in diabetes management. Such efforts include encouraging reporting of negative or unintended outcomes and positive outcomes across RE-AIM dimensions, so that the feasible and efficacious parts of an intervention can be replicated.23 Funding organizations may consider requiring researchers to develop a plan for sustainability throughout the research. Practitioners should also demand more information on external validity of an intervention with demonstrated efficacy. Information on adoption, implementation, and maintenance will help them clarify whether an efficacious intervention is a good fit for their organization.23 Table 3. Recommendations based on gaps identified from the RE-AIM framework evaluation Dimension and component . Key issues . Overall and specific recommendations . Reach Representativeness of study sample to target population Document and understand access, awareness, and appropriateness of intervention to meet target population needs67 Compare characteristics (eg, sociodemographic, economic, and behavioral) of participants with nonparticipants or the general local population to understand the representativeness of the sample22 Include inclusion criteria (eg, health/disease conditions, such as hemoglobin A1c ≥8.0%, capable of using mobile phone) and exclusion to Provide explanation on why certain individuals were not eligible for participation22 Include recruitment setting, methods, and recruitment adaptations22 Efficacy/effectiveness Suitability and credibility of study design, data collection, and evaluation Document participants’ satisfaction, negative outcomes, and subgroup effects in addition to reporting primary outcome67 Report study design (eg, randomized controlled trial) and whether a comparison group is included Report use intention-to-treat analysis22 and missing data procedure68 Report qualitative methods to acceptability and usability of the intervention; report potential negative outcomes of the intervention22 Report subgroup effects (eg, sex, race, age, or health/disease condition that influence the intervention effect)22 Adoption Diffusion of intervention program at organizational and delivery level and factors influence the adoption Document and understand contextual factors related to adoption and developing guides to help users enhance adoption67 Report characteristics of intervention location and delivery agent and their selection criteria22; if applicable, describe adoption rate of intervention location and delivery agent69 If applicable, describe the expertise of the delivery staff69 If applicable, describe participation rate of delivery setting/agent69 Provide information on the level of human involvement required for an SMS- and app-based intervention compared with the level of human involvement for a routine application70 Provide information on the prompts/reminders required for SMS- and app-based intervention compared with the level of prompts/reminders for a routine application70 Provide information on any interventions (including training sessions/support) that are implemented in addition to the targeted SMS- and app-based intervention70 Implementation Fidelity of intervention program, including intervention uptake, development, monitoring, and adaptation and factors influence the implementation Document standardized measures for capturing implementation fidelity, multilevel assessment of cost (both monetary and time) and adaption made for implementation67 Describe intervention content and use parameters (eg, frequency, optimal timing for use, heaviness of use)70, and frequency of inter-person and virtual sessions, if applicable. Provide information intervention costs including price for mobile phone, mobile phone data plan, and incentives for program development/participation given to staff or participants68 Record percent delivered as intended (eg, SMS sent/unsent/received/not received; any application functioning problems)22 Describe adaptations made to the intervention during implementation (eg, fitting strategies and methods to user culture)68 Report strategies to monitoring and gathering feedback (from both participants and staffs) during implementation69 Report time cost to participants and staff in addition to time used for intervention implementation. Maintenance Sustainability of intervention program at both the individual and organizational levels Document and understand dynamic, complex, and multilevel factors leading to sustainment67 Include an assessment of maintenance of the intervention outcomes (clinical and/or behavioral, such as hemoglobin A1c reduction, healthy eating) 6 mo after the completion of the intervention71 Report broader outcomes (eg, policy development associated with the intervention)57 Provide a context in which to evaluate the long-term outcome (eg, ongoing institutional and policy support for program maintenance and ongoing cost)57 Include a sustainability plan regarding how the intervention could be sustained at both the individual and organizational levels or, if applicable, provide data on the degree to which the intervention is sustained over time22,71 Dimension and component . Key issues . Overall and specific recommendations . Reach Representativeness of study sample to target population Document and understand access, awareness, and appropriateness of intervention to meet target population needs67 Compare characteristics (eg, sociodemographic, economic, and behavioral) of participants with nonparticipants or the general local population to understand the representativeness of the sample22 Include inclusion criteria (eg, health/disease conditions, such as hemoglobin A1c ≥8.0%, capable of using mobile phone) and exclusion to Provide explanation on why certain individuals were not eligible for participation22 Include recruitment setting, methods, and recruitment adaptations22 Efficacy/effectiveness Suitability and credibility of study design, data collection, and evaluation Document participants’ satisfaction, negative outcomes, and subgroup effects in addition to reporting primary outcome67 Report study design (eg, randomized controlled trial) and whether a comparison group is included Report use intention-to-treat analysis22 and missing data procedure68 Report qualitative methods to acceptability and usability of the intervention; report potential negative outcomes of the intervention22 Report subgroup effects (eg, sex, race, age, or health/disease condition that influence the intervention effect)22 Adoption Diffusion of intervention program at organizational and delivery level and factors influence the adoption Document and understand contextual factors related to adoption and developing guides to help users enhance adoption67 Report characteristics of intervention location and delivery agent and their selection criteria22; if applicable, describe adoption rate of intervention location and delivery agent69 If applicable, describe the expertise of the delivery staff69 If applicable, describe participation rate of delivery setting/agent69 Provide information on the level of human involvement required for an SMS- and app-based intervention compared with the level of human involvement for a routine application70 Provide information on the prompts/reminders required for SMS- and app-based intervention compared with the level of prompts/reminders for a routine application70 Provide information on any interventions (including training sessions/support) that are implemented in addition to the targeted SMS- and app-based intervention70 Implementation Fidelity of intervention program, including intervention uptake, development, monitoring, and adaptation and factors influence the implementation Document standardized measures for capturing implementation fidelity, multilevel assessment of cost (both monetary and time) and adaption made for implementation67 Describe intervention content and use parameters (eg, frequency, optimal timing for use, heaviness of use)70, and frequency of inter-person and virtual sessions, if applicable. Provide information intervention costs including price for mobile phone, mobile phone data plan, and incentives for program development/participation given to staff or participants68 Record percent delivered as intended (eg, SMS sent/unsent/received/not received; any application functioning problems)22 Describe adaptations made to the intervention during implementation (eg, fitting strategies and methods to user culture)68 Report strategies to monitoring and gathering feedback (from both participants and staffs) during implementation69 Report time cost to participants and staff in addition to time used for intervention implementation. Maintenance Sustainability of intervention program at both the individual and organizational levels Document and understand dynamic, complex, and multilevel factors leading to sustainment67 Include an assessment of maintenance of the intervention outcomes (clinical and/or behavioral, such as hemoglobin A1c reduction, healthy eating) 6 mo after the completion of the intervention71 Report broader outcomes (eg, policy development associated with the intervention)57 Provide a context in which to evaluate the long-term outcome (eg, ongoing institutional and policy support for program maintenance and ongoing cost)57 Include a sustainability plan regarding how the intervention could be sustained at both the individual and organizational levels or, if applicable, provide data on the degree to which the intervention is sustained over time22,71 RE-AIM: Reach, Efficacy/Effectiveness, Adoption, Implementation, and Maintenance; SMS: short message service. Open in new tab Table 3. Recommendations based on gaps identified from the RE-AIM framework evaluation Dimension and component . Key issues . Overall and specific recommendations . Reach Representativeness of study sample to target population Document and understand access, awareness, and appropriateness of intervention to meet target population needs67 Compare characteristics (eg, sociodemographic, economic, and behavioral) of participants with nonparticipants or the general local population to understand the representativeness of the sample22 Include inclusion criteria (eg, health/disease conditions, such as hemoglobin A1c ≥8.0%, capable of using mobile phone) and exclusion to Provide explanation on why certain individuals were not eligible for participation22 Include recruitment setting, methods, and recruitment adaptations22 Efficacy/effectiveness Suitability and credibility of study design, data collection, and evaluation Document participants’ satisfaction, negative outcomes, and subgroup effects in addition to reporting primary outcome67 Report study design (eg, randomized controlled trial) and whether a comparison group is included Report use intention-to-treat analysis22 and missing data procedure68 Report qualitative methods to acceptability and usability of the intervention; report potential negative outcomes of the intervention22 Report subgroup effects (eg, sex, race, age, or health/disease condition that influence the intervention effect)22 Adoption Diffusion of intervention program at organizational and delivery level and factors influence the adoption Document and understand contextual factors related to adoption and developing guides to help users enhance adoption67 Report characteristics of intervention location and delivery agent and their selection criteria22; if applicable, describe adoption rate of intervention location and delivery agent69 If applicable, describe the expertise of the delivery staff69 If applicable, describe participation rate of delivery setting/agent69 Provide information on the level of human involvement required for an SMS- and app-based intervention compared with the level of human involvement for a routine application70 Provide information on the prompts/reminders required for SMS- and app-based intervention compared with the level of prompts/reminders for a routine application70 Provide information on any interventions (including training sessions/support) that are implemented in addition to the targeted SMS- and app-based intervention70 Implementation Fidelity of intervention program, including intervention uptake, development, monitoring, and adaptation and factors influence the implementation Document standardized measures for capturing implementation fidelity, multilevel assessment of cost (both monetary and time) and adaption made for implementation67 Describe intervention content and use parameters (eg, frequency, optimal timing for use, heaviness of use)70, and frequency of inter-person and virtual sessions, if applicable. Provide information intervention costs including price for mobile phone, mobile phone data plan, and incentives for program development/participation given to staff or participants68 Record percent delivered as intended (eg, SMS sent/unsent/received/not received; any application functioning problems)22 Describe adaptations made to the intervention during implementation (eg, fitting strategies and methods to user culture)68 Report strategies to monitoring and gathering feedback (from both participants and staffs) during implementation69 Report time cost to participants and staff in addition to time used for intervention implementation. Maintenance Sustainability of intervention program at both the individual and organizational levels Document and understand dynamic, complex, and multilevel factors leading to sustainment67 Include an assessment of maintenance of the intervention outcomes (clinical and/or behavioral, such as hemoglobin A1c reduction, healthy eating) 6 mo after the completion of the intervention71 Report broader outcomes (eg, policy development associated with the intervention)57 Provide a context in which to evaluate the long-term outcome (eg, ongoing institutional and policy support for program maintenance and ongoing cost)57 Include a sustainability plan regarding how the intervention could be sustained at both the individual and organizational levels or, if applicable, provide data on the degree to which the intervention is sustained over time22,71 Dimension and component . Key issues . Overall and specific recommendations . Reach Representativeness of study sample to target population Document and understand access, awareness, and appropriateness of intervention to meet target population needs67 Compare characteristics (eg, sociodemographic, economic, and behavioral) of participants with nonparticipants or the general local population to understand the representativeness of the sample22 Include inclusion criteria (eg, health/disease conditions, such as hemoglobin A1c ≥8.0%, capable of using mobile phone) and exclusion to Provide explanation on why certain individuals were not eligible for participation22 Include recruitment setting, methods, and recruitment adaptations22 Efficacy/effectiveness Suitability and credibility of study design, data collection, and evaluation Document participants’ satisfaction, negative outcomes, and subgroup effects in addition to reporting primary outcome67 Report study design (eg, randomized controlled trial) and whether a comparison group is included Report use intention-to-treat analysis22 and missing data procedure68 Report qualitative methods to acceptability and usability of the intervention; report potential negative outcomes of the intervention22 Report subgroup effects (eg, sex, race, age, or health/disease condition that influence the intervention effect)22 Adoption Diffusion of intervention program at organizational and delivery level and factors influence the adoption Document and understand contextual factors related to adoption and developing guides to help users enhance adoption67 Report characteristics of intervention location and delivery agent and their selection criteria22; if applicable, describe adoption rate of intervention location and delivery agent69 If applicable, describe the expertise of the delivery staff69 If applicable, describe participation rate of delivery setting/agent69 Provide information on the level of human involvement required for an SMS- and app-based intervention compared with the level of human involvement for a routine application70 Provide information on the prompts/reminders required for SMS- and app-based intervention compared with the level of prompts/reminders for a routine application70 Provide information on any interventions (including training sessions/support) that are implemented in addition to the targeted SMS- and app-based intervention70 Implementation Fidelity of intervention program, including intervention uptake, development, monitoring, and adaptation and factors influence the implementation Document standardized measures for capturing implementation fidelity, multilevel assessment of cost (both monetary and time) and adaption made for implementation67 Describe intervention content and use parameters (eg, frequency, optimal timing for use, heaviness of use)70, and frequency of inter-person and virtual sessions, if applicable. Provide information intervention costs including price for mobile phone, mobile phone data plan, and incentives for program development/participation given to staff or participants68 Record percent delivered as intended (eg, SMS sent/unsent/received/not received; any application functioning problems)22 Describe adaptations made to the intervention during implementation (eg, fitting strategies and methods to user culture)68 Report strategies to monitoring and gathering feedback (from both participants and staffs) during implementation69 Report time cost to participants and staff in addition to time used for intervention implementation. Maintenance Sustainability of intervention program at both the individual and organizational levels Document and understand dynamic, complex, and multilevel factors leading to sustainment67 Include an assessment of maintenance of the intervention outcomes (clinical and/or behavioral, such as hemoglobin A1c reduction, healthy eating) 6 mo after the completion of the intervention71 Report broader outcomes (eg, policy development associated with the intervention)57 Provide a context in which to evaluate the long-term outcome (eg, ongoing institutional and policy support for program maintenance and ongoing cost)57 Include a sustainability plan regarding how the intervention could be sustained at both the individual and organizational levels or, if applicable, provide data on the degree to which the intervention is sustained over time22,71 RE-AIM: Reach, Efficacy/Effectiveness, Adoption, Implementation, and Maintenance; SMS: short message service. Open in new tab Limitations Several limitations concerning this review need to be considered. First, there was a small number of eligible studies available for this study. However, we used a validated PRISMA approach when we selected literature and extracted results from studies using RCTs or quasi-experimental designs. Authors’ comments were also incorporated into findings of the review. Small sample size and inadequate blinding of some studies in this review may still contribute to risk of bias. Larger and methodologically robust trails are very much needed. Second, it is possible that our search strategy could not identify all relevant articles. For example, we excluded articles that were published in non–English-language or trial registry data. However, we searched 3 main databases including PubMed, Embase, and Web of Science, in addition to 2 topic-specific databases, CINAHL Plus and the IEEE Xplore Digital Library. We used broad inclusion and exclusion criteria to increase the likelihood of capturing relevant studies to minimize publication bias. We also used a manual search of reference lists of eligible articles for relevant systematic reviews and narrative reviews. We addressed a focused area of mobile phone–based solutions, specifically interventions delivered by SMS and apps that are the most common in technology based behavioral interventions due to increased smartphone ownership. The exclusive review on SMS- and app-based interventions, rather than a broad area of mobile health interventions including various wearable devices, improves the transferability of our results to a prime area of smartphone-based interventions. Further, some authors did not report on specific adoption, implementation, and maintenance strategies that were in fact used in their studies, increasing the risk of reporting bias. To address this concern, we contacted the authors directly to inquire missing information regarding external validity. CONCLUSION SMS- and app-based interventions may have potential to promote diabetes self-management among patients with diabetes as access to smartphone increase; however, the population impact in the United States and other countries remains to be confirmed. Our systematic review based on the RE-AIM framework synthesized important findings from this emergent body of literature. This review is among the first to address issues in external validity of SMS- and app-based DSMES studies that is lacking from traditional reviews predominately focused on internal validity and intervention efficacy or effectiveness. Our review demonstrated that the comprehensiveness of reporting on RE-AIM criteria across the mobile phone–based DSMES studies was relatively low and with many gaps in internal validity reporting (ie, extent to which the protocol was delivered as intended) and, more so in external validity (across domains in adoption, implementation, and maintenance). Without this information, it is difficult to determine the internal validity and external validity of mobile phone–based interventions in DSMES. We encourage researchers to improve reporting around RE-AIM dimensions, specifically information on intention-to-treat analysis; mixed methods to understand acceptability, usability, and implementation of an intervention; costs; location, delivery agent selection; characteristics; and maintenance planning. Continued efforts in improving quality of research development and reporting will ensure mobile phone–based interventions to address diabetes will be broadly applicable across diverse settings and populations. FUNDING This study was supported in part by the National Institute of Diabetes and Digestive and Kidney Diseases of the National Institutes of Health (award numbers P30DK092950 and P30DK020579) and the Centers for Disease Control and Prevention (Cooperative Agreement number U48DP006395). SJP’s contribution was supported by the National Center for Advancing Translational Sciences of the National Institutes of Health under Award Number KL2 TR002346. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health. AUTHOR CONTRIBUTIONS YY designed the study, undertook the literature search and data extraction, contacted individual study authors, and wrote the article. SJP refined the coding book, verified the coding results, and assessed clinical applicability of the included studies. RCB, SAB, MK, VAF, and EJS provided critical appraisals and edits to the article. RD, KW, DAG, AT, AR, and CM provided important information regarding their individual studies. SUPPLEMENTARY MATERIAL Supplementary material is available at Journal of the American Medical Informatics Association online. Acknowledgments The authors would like to acknowledge Dr Eirik Arsand from Norwegian Centre for E-health Research, University Hospital of North Norway, Tromsø, Norway, for his information on an app that was extended from the app included in an article under our review. We would like to thank Dr Maryam Peimani from Tehran University of Medical Sciences in Iran, Ms Marcia Vervloet from the Netherlands Institute for Health Services Research in the Netherlands, and Dr Sanjay Arora from Keck School of Medicine at University of Southern California in the United States for their responses to our inquires. We would also like to thank Ms Lauren Hartz from University of Tokyo to reply our inquires in English on behalf of KM. Finally, we would like to thank Ms Allison Phad from Center for Diabetes Translation Research Washington University in St. Louis for providing her edits to the manuscript. CONFLICT OF INTEREST STATEMENT None declared. REFERENCES 1 Beck J , Greenwood DA, Blanton L, et al. . 2017 National Standards for Diabetes Self-Management Education and Support . Diabetes Educ 2017 ; 43 ( 5 ): 449 – 64 . Google Scholar Crossref Search ADS PubMed WorldCat 2 Haas L , Maryniuk M, Beck J, et al. . National standards for diabetes self-management education and support . Diabetes Educ 2012 ; 38 ( 5 ): 619 – 29 . Google Scholar Crossref Search ADS PubMed WorldCat 3 Benoit SR , Ji M, Fleming R, Philis-Tsimikas A. Predictors of dropouts from a San Diego diabetes program: a case control study . Prev Chronic Dis 2004 ; 1 : A10 . Google Scholar PubMed OpenURL Placeholder Text WorldCat 4 Gucciardi E , Demelo M, Offenheim A, Stewart DE. Factors contributing to attrition behavior in diabetes self-management programs: a mixed method approach . BMC Health Serv Res 2008 ; 8 ( 1 ): 33 . Google Scholar Crossref Search ADS PubMed WorldCat 5 Peyrot M , Rubin RR. Access to diabetes self-management education . Diabetes Educ 2008 ; 34 ( 1 ): 90 – 7 . Google Scholar Crossref Search ADS PubMed WorldCat 6 Li R , Shrestha SS, Lipman R, et al. . Diabetes self-management education and training among privately insured persons with newly diagnosed diabetes–United States, 2011-2012 . MMWR Morb Mortal Wkly Rep 2014 ; 63 ( 46 ): 1045 – 9 . Google Scholar PubMed OpenURL Placeholder Text WorldCat 7 Pew Research Center. Mobile Technology Fact Sheet . Washington, DC : Pew Research Center ; 2019 . Google Scholar Google Preview OpenURL Placeholder Text WorldCat COPAC 8 Greenwood DA , Gee PM, Fatkin KJ, Peeples M. A systematic review of reviews evaluating technology-enabled diabetes self-management education and support . J Diabetes Sci Technol 2017 ; 11 ( 5 ): 1015 – 27 . Google Scholar Crossref Search ADS PubMed WorldCat 9 Arambepola C , Ricci-Cabello I, Manikavasagam P, Roberts N, French DP, Farmer A. The impact of automated brief messages promoting lifestyle changes delivered via mobile devices to people with type 2 diabetes: a systematic literature review and meta-analysis of controlled trials . J Med Internet Res 2016 ; 18 ( 4 ): e86 . Google Scholar Crossref Search ADS PubMed WorldCat 10 Arnhold M , Quade M, Kirch W. Mobile applications for diabetics: a systematic review and expert-based usability evaluation considering the special requirements of diabetes patients age 50 years or older . J Med Internet Res 2014 ; 16 ( 4 ): e104 . Google Scholar Crossref Search ADS PubMed WorldCat 11 Choi J , Lee JH, Vittinghoff E, Fukuoka Y. mHealth physical activity intervention: a randomized pilot study in physically inactive pregnant women . Matern Child Health J 2016 ; 20 ( 5 ): 1091 – 101 . Google Scholar Crossref Search ADS PubMed WorldCat 12 Coughlin SS. Mobile technology for self-monitoring of blood glucose among patients with type 2 diabetes mellitus . Mhealth 2017 ; 3 : 47 . Google Scholar Crossref Search ADS PubMed WorldCat 13 Haider R , Sudini L, Chow CK, Cheung NW. Mobile phone text messaging in improving glycaemic control for patients with type 2 diabetes mellitus: a systematic review and meta-analysis . Diabetes Res Clin Pract 2019 ; 150 : 27 – 37 . Google Scholar Crossref Search ADS PubMed WorldCat 14 Hall AK , Cole-Lewis H, Bernhardt JM. Mobile text messaging for health: a systematic review of reviews . Annu Rev Public Health 2015 ; 36 ( 1 ): 393 – 415 . Google Scholar Crossref Search ADS PubMed WorldCat 15 Kankanhalli A , Shin J, Oh H. Mobile-based interventions for dietary behavior change and health outcomes: scoping review . JMIR Mhealth Uhealth 2019 ; 7 ( 1 ): e11312 . Google Scholar Crossref Search ADS PubMed WorldCat 16 Orr JA , King RJ. Mobile phone SMS messages can enhance healthy behaviour: a meta-analysis of randomised controlled trials . Health Psychol Rev 2015 ; 9 ( 4 ): 397 – 416 . Google Scholar Crossref Search ADS PubMed WorldCat 17 Park JYE , Li J, Howren A, Tsao NW, De Vera M. Mobile phone apps targeting medication adherence: quality assessment and content analysis of user reviews . JMIR Mhealth Uhealth 2019 ; 7 ( 1 ): e11919 . Google Scholar Crossref Search ADS PubMed WorldCat 18 Yoshida Y , Boren SA, Soares J, Popescu M, Nielson SD, Simoes EJ. Effect of health information technologies on glycemic control among patients with type 2 diabetes . Curr Diab Rep 2018 ; 18 ( 12 ): 130 . Google Scholar Crossref Search ADS PubMed WorldCat 19 Glasgow RE , Klesges LM, Dzewaltowski DA, Bull SS, Estabrooks P. The future of health behavior change research: what is needed to improve translation of research into health promotion practice? Ann Behav Med 2004 ; 27 ( 1 ): 3 – 12 . Google Scholar Crossref Search ADS PubMed WorldCat 20 Glasgow RE , Vogt TM, Boles SM. Evaluating the public health impact of health promotion interventions: the RE-AIM framework . Am J Public Health 1999 ; 89 ( 9 ): 1322 – 7 . Google Scholar Crossref Search ADS PubMed WorldCat 21 Akers JD , Estabrooks PA, Davy BM. Translational research: bridging the gap between long-term weight loss maintenance research and practice . J Am Diet Assoc 2010 ; 110 ( 10 ): 1511 – 22, 22 e1–3 . Google Scholar Crossref Search ADS PubMed WorldCat 22 Blackman KC , Zoellner J, Berrey LM, et al. . Assessing the internal and external validity of mobile health physical activity promotion interventions: a systematic literature review using the RE-AIM framework . J Med Internet Res 2013 ; 15 ( 10 ): e224 . Google Scholar Crossref Search ADS PubMed WorldCat 23 Bull SS , Gillette C, Glasgow RE, Estabrooks P. Work site health promotion research: to what extent can we generalize the results and what is needed to translate research to practice? Health Educ Behav 2003 ; 30 ( 5 ): 537 – 49 . Google Scholar Crossref Search ADS PubMed WorldCat 24 Dzewaltowski DA , Estabrooks PA, Klesges LM, Bull S, Glasgow RE. Behavior change intervention research in community settings: how generalizable are the results? Health Promot Int 2004 ; 19 ( 2 ): 235 – 45 . Google Scholar Crossref Search ADS PubMed WorldCat 25 Klesges LM , Dzewaltowski DA, Glasgow RE. Review of external validity reporting in childhood obesity prevention research . Am J Prev Med 2008 ; 34 ( 3 ): 216 – 23 . Google Scholar Crossref Search ADS PubMed WorldCat 26 McGoey T , Root Z, Bruner MW, Law B. Evaluation of physical activity interventions in children via the reach, efficacy/effectiveness, adoption, implementation, and maintenance (RE-AIM) framework: A systematic review of randomized and non-randomized trials . Prev Med 2016 ; 82 : 8 – 19 . Google Scholar Crossref Search ADS PubMed WorldCat 27 White SM , McAuley E, Estabrooks PA, Courneya KS. Translating physical activity interventions for breast cancer survivors into practice: an evaluation of randomized controlled trials . Ann Behav Med 2009 ; 37 ( 1 ): 10 – 9 . Google Scholar Crossref Search ADS PubMed WorldCat 28 Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA). PRISMA 2009 Checklist . Helsinki, Finland : PRISMA ; 2009 . Google Scholar Google Preview OpenURL Placeholder Text WorldCat COPAC 29 Sutter JD. The top 10 tech trends of 2009 . CNN. http://www.cnn.com/2009/TECH/12/22/top.tech.trends.2009/index.html Accessed •••. 30 Hussein WI , Hasan K, Jaradat AA. Effectiveness of mobile phone short message service on diabetes mellitus management; the SMS-DM study . Diabetes Res Clin Pract 2011 ; 94 ( 1 ): e24 – 6 . Google Scholar Crossref Search ADS PubMed WorldCat 31 Shetty AS , Chamukuttan S, Nanditha A, Raj RK, Ramachandran A. Reinforcement of adherence to prescription recommendations in Asian Indian diabetes patients using short message service (SMS)–a pilot study . J Assoc Physicians India 2011 ; 59 : 711 – 4 . Google Scholar PubMed OpenURL Placeholder Text WorldCat 32 Waki K , Fujita H, Uchimura Y, et al. . DialBetics: a novel smartphone-based self-management support system for type 2 diabetes patients . J Diabetes Sci Technol 2014 ; 8 ( 2 ): 209 – 15 . Google Scholar Crossref Search ADS PubMed WorldCat 33 Zhou W , Chen M, Yuan J, Sun Y. Welltang—A smart phone-based diabetes management application—Improves blood glucose control in Chinese people with diabetes . Diabetes Res Clin Pract 2016 ; 116 : 105 – 10 . Google Scholar Crossref Search ADS PubMed WorldCat 34 Zimmerman LP , Goel S, Sathar S, et al. . A novel patient recruitment strategy: patient selection directly from the community through linkage to clinical data . Appl Clin Inform 2018 ; 9 ( 1 ): 114 – 21 . Google Scholar Crossref Search ADS PubMed WorldCat 35 Sun C , Sun L, Xi S, et al. . Mobile phone–based telemedicine practice in older chinese patients with type 2 diabetes mellitus: randomized controlled trial . JMIR Mhealth Uhealth 2019 ; 7 ( 1 ): e10664 . Google Scholar Crossref Search ADS PubMed WorldCat 36 Allen K , Zoellner J, Motley M, Estabrooks PA. Understanding the internal and external validity of health literacy interventions: a systematic literature review using the RE-AIM framework . J Health Commun 2011 ; 16 (sup3 ): 55 – 72 . Google Scholar Crossref Search ADS WorldCat 37 Dobson R , Whittaker R, Jiang Y, et al. . Text message-based diabetes self-management support (SMS4BG): study protocol for a randomised controlled trial . Trials 2016 ; 17 ( 1 ): 179 . Google Scholar Crossref Search ADS PubMed WorldCat 38 Kleinman NJ , Shah A, Shah S, Phatak S, Viswanathan V. Improved medication adherence and frequency of blood glucose self-testing using an m-health platform versus usual care in a multisite randomized clinical trial among people with type 2 diabetes in India . Telemed J E Health 2017 ; 23 ( 9 ): 733 – 40 . Google Scholar Crossref Search ADS PubMed WorldCat 39 Quinn CC , Shardell MD, Terrin ML, Barr EA, Ballew SH, Gruber-Baldini AL. Cluster-randomized trial of a mobile phone personalized behavioral intervention for blood glucose control . Diabetes Care 2011 ; 34 ( 9 ): 1934 – 42 . Google Scholar Crossref Search ADS PubMed WorldCat 40 Rossi MC , Nicolucci A, Di Bartolo P, et al. . Diabetes Interactive Diary: a new telemedicine system enabling flexible diet and insulin therapy while improving quality of life: an open-label, international, multicenter, randomized study . Diabetes Care 2010 ; 33 ( 1 ): 109 – 15 . Google Scholar Crossref Search ADS PubMed WorldCat 41 Holmen H , Torbjornsen A, Wahl AK, et al. . A mobile health intervention for self-management and lifestyle change for persons with type 2 diabetes, part 2: one-year results from the Norwegian randomized controlled trial renewing healTH . JMIR Mhealth Uhealth 2014 ; 2 ( 4 ): e57 . Google Scholar Crossref Search ADS PubMed WorldCat 42 Torbjornsen A , Jenum AK, Smastuen MC, et al. . A low-intensity mobile health intervention with and without health counseling for persons with type 2 diabetes, part 1: baseline and short-term results from a randomized controlled trial in the Norwegian part of renewing health . JMIR Mhealth Uhealth 2014 ; 2 ( 4 ): e52 . Google Scholar Crossref Search ADS PubMed WorldCat 43 Arora S , Peters AL, Burner E, Lam CN, Menchine M. Trial to examine text message-based mHealth in emergency department patients with diabetes (TExT-MED): a randomized controlled trial . Ann Emerg Med 2014 ; 63 ( 6 ): 745 – 54.e6 . Google Scholar Crossref Search ADS PubMed WorldCat 44 Kumar S , Moseson H, Uppal J, Juusola JL. A diabetes mobile app with in-app coaching from a certified diabetes educator reduces A1C for individuals with type 2 diabetes . Diabetes Educ 2018 ; 44 ( 3 ): 226 – 36 . Google Scholar Crossref Search ADS PubMed WorldCat 45 Vervloet M , van Dijk L, de Bakker DH, et al. . Short- and long-term effects of real-time medication monitoring with short message service (SMS) reminders for missed doses on the refill adherence of people with Type 2 diabetes: evidence from a randomized controlled trial . Diabet Med 2014 ; 31 ( 7 ): 821 – 8 . Google Scholar Crossref Search ADS PubMed WorldCat 46 Wu CJ , Sung HC, Chang AM, Atherton J, Kostner K, McPhail SM. Cardiac-diabetes self-management program for Australians and Taiwanese: a randomized blocked design study . Nurs Health Sci 2017 ; 19 ( 3 ): 307 – 15 . Google Scholar Crossref Search ADS PubMed WorldCat 47 Agarwal P , Mukerji G, Desveaux L, et al. . Mobile app for improved self-management of type 2 diabetes: multicenter pragmatic randomized controlled trial . JMIR Mhealth Uhealth 2019 ; 7 ( 1 ): e10321 . Google Scholar Crossref Search ADS PubMed WorldCat 48 Bauer V , Goodman N, Lapin B, et al. . Text messaging to improve disease management in patients with painful diabetic peripheral neuropathy . Diabetes Educ 2018 ; 44 ( 3 ): 237 – 48 . Google Scholar Crossref Search ADS PubMed WorldCat 49 Kim HS , Choi W, Baek EK, et al. . Efficacy of the smartphone-based glucose management application stratified by user satisfaction . Diabetes Metab J 2014 ; 38 ( 3 ): 204 – 10 . Google Scholar Crossref Search ADS PubMed WorldCat 50 Greenwood DA , Hankins AI, Parise CA, Spier V, Olveda J, Buss KA. A comparison of in-person, telephone, and secure messaging for type 2 diabetes self-management support . Diabetes Educ 2014 ; 40 ( 4 ): 516 – 25 . Google Scholar Crossref Search ADS PubMed WorldCat 51 Brandon DT , Isaac LA, LaVeist TA. The legacy of Tuskegee and trust in medical care: is Tuskegee responsible for race differences in mistrust of medical care? J Natl Med Assoc 2005 ; 97 ( 7 ): 951 – 6 . Google Scholar PubMed OpenURL Placeholder Text WorldCat 52 Carpenter WR , Tyree S, Wu Y, et al. . A surveillance system for monitoring, public reporting, and improving minority access to cancer clinical trials . Clin Trials 2012 ; 9 ( 4 ): 426 – 35 . Google Scholar Crossref Search ADS PubMed WorldCat 53 Greiner KA , Friedman DB, Adams SA, et al. . Effective recruitment strategies and community-based participatory research: community networks program centers’ recruitment in cancer prevention studies . Cancer Epidemiol Biomarkers Prev 2014 ; 23 ( 3 ): 416 – 23 . Google Scholar Crossref Search ADS PubMed WorldCat 54 Glasgow RE , Bull SS, Gillette C, Klesges LM, Dzewaltowski DA. Behavior change intervention research in healthcare settings: a review of recent reports with emphasis on external validity . Am J Prev Med 2002 ; 23 ( 1 ): 62 – 9 . Google Scholar Crossref Search ADS PubMed WorldCat 55 Antikainen I , Ellis R. A RE-AIM evaluation of theory-based physical activity interventions . J Sport Exerc Psychol 2011 ; 33 ( 2 ): 198 – 214 . Google Scholar Crossref Search ADS PubMed WorldCat 56 Estabrooks P , Dzewaltowski DA, Glasgow RE, Klesges LM. Reporting of validity from school health promotion studies published in 12 leading journals, 1996-2000 . J Sch Health 2003 ; 73 ( 1 ): 21 – 8 . Google Scholar Crossref Search ADS PubMed WorldCat 57 Gaglio B , Shoup JA, Glasgow RE. The RE-AIM framework: a systematic review of use over time . Am J Public Health 2013 ; 103 ( 6 ): e38 – 46 . Google Scholar Crossref Search ADS PubMed WorldCat 58 Estabrooks PA , Allen KC. Updating, employing, and adapting: a commentary on What does it mean to “employ” the RE-AIM model . Eval Health Prof 2013 ; 36 ( 1 ): 67 – 72 . Google Scholar Crossref Search ADS PubMed WorldCat 59 Joe J , Demiris G. Older adults and mobile phones for health: a review . J Biomed Inform 2013 ; 46 ( 5 ): 947 – 54 . Google Scholar Crossref Search ADS PubMed WorldCat 60 May CR , Finch TL, Cornford J, et al. . Integrating telecare for chronic disease management in the community: what needs to be done? BMC Health Serv Res 2011 ; 11 ( 1 ): 131 . Google Scholar Crossref Search ADS PubMed WorldCat 61 Nielsen JA , Mengiste SA. Analysing the diffusion and adoption of mobile IT across social worlds . Health Informatics J 2014 ; 20 ( 2 ): 87 – 103 . Google Scholar Crossref Search ADS PubMed WorldCat 62 Hebert MA , Korabek B, Scott RE. Moving research into practice: a decision framework for integrating home telehealth into chronic illness care . Int J Med Inform 2006 ; 75 ( 12 ): 786 – 94 . Google Scholar Crossref Search ADS PubMed WorldCat 63 Ross J , Stevenson F, Lau R, Murray E. Factors that influence the implementation of e-health: a systematic review of systematic reviews (an update) . Implement Sci 2016 ; 11 ( 1 ): 146 . Google Scholar Crossref Search ADS PubMed WorldCat 64 Benavides-Vaello S , Strode A, Sheeran BC. Using technology in the delivery of mental health and substance abuse treatment in rural communities: a review . J Behav Health Serv Res 2013 ; 40 ( 1 ): 111 – 20 . Google Scholar Crossref Search ADS PubMed WorldCat 65 Jennett PA , Scott RE, Affleck Hall L, et al. . Policy implications associated with the socioeconomic and health system impact of telehealth: a case study from Canada . Telemed J E Health 2004 ; 10 ( 1 ): 77 – 83 . Google Scholar Crossref Search ADS PubMed WorldCat 66 Matthew-Maich N , Harris L, Ploeg J, et al. . Designing, implementing, and evaluating mobile health technologies for managing chronic conditions in older adults: a scoping review . JMIR Mhealth Uhealth 2016 ; 4 ( 2 ): e29 . Google Scholar Crossref Search ADS PubMed WorldCat 67 Glasgow RE , Harden SM, Gaglio B, et al. . RE-AIM planning and evaluation framework: adapting to new science and practice with a 20-year review . Front Public Health 2019 ; 7 : 64 . Google Scholar Crossref Search ADS PubMed WorldCat 68 Stellefson M , Chaney B, Barry AE, et al. . Web 2.0 chronic disease self-management for older adults: a systematic review . J Med Internet Res 2013 ; 15 ( 2 ): e35 . Google Scholar Crossref Search ADS PubMed WorldCat 69 Molleman GR , Ploeg MA, Hosman CM, Peters LH. Preffi 2.0—a quality assessment tool . Promot Educ 2006 ; 13 ( 1 ): 9 – 14 . Google Scholar Crossref Search ADS PubMed WorldCat 70 Eysenbach G , Group C. E-CONSORT-E-HEALTH: improving and standardizing evaluation reports of Web-based and mobile health interventions . J Med Internet Res 2011 ; 13 ( 4 ): e126 . Google Scholar Crossref Search ADS PubMed WorldCat 71 Kessler RS , Purcell EP, Glasgow RE, Klesges LM, Benkeser RM, Peek CJ. What does it mean to “employ” the RE-AIM model? Eval Health Prof 2013 ; 36 ( 1 ): 44 – 66 . Google Scholar Crossref Search ADS PubMed WorldCat © The Author(s) 2020. Published by Oxford University Press on behalf of the American Medical Informatics Association. All rights reserved. For permissions, please email: journals.permissions@oup.com This article is published and distributed under the terms of the Oxford University Press, Standard Journals Publication Model (https://academic.oup.com/journals/pages/open_access/funder_policies/chorus/standard_publication_model)

Journal

Journal of the American Medical Informatics Association – Oxford University Press

Published: Jun 1, 2020

Get 20M+ Full-Text Papers For Less Than $1.50/day. Start a 14-Day Trial for You or Your Team.

Learn More →

Using the RE-AIM framework to evaluate internal and external validity of mobile phone–based interventions in diabetes self-management education and support

Using the RE-AIM framework to evaluate internal and external validity of mobile phone–based interventions in diabetes self-management education and support

Get 20M+ Full-Text Papers For Less Than $1.50/day. Start a 14-Day Trial for You or Your Team.

Learn More →

Using the RE-AIM framework to evaluate internal and external validity of mobile phone–based interventions in diabetes self-management education and support

Using the RE-AIM framework to evaluate internal and external validity of mobile phone–based interventions in diabetes self-management education and support

References (77)

Abstract

Journal

Recommended Articles

There are no references for this article.

Our policy towards the use of cookies