Get 20M+ Full-Text Papers For Less Than $1.50/day. Start a 14-Day Trial for You or Your Team.

Learn More →

Spanish adaptation and validation of the User Version of the Mobile Application Rating Scale (uMARS)

Spanish adaptation and validation of the User Version of the Mobile Application Rating Scale (uMARS) Abstract Objective While the professional version of the Mobile App Rating Scale (MARS) has already been translated, and validated into the Spanish language, its user-centered counterpart has not yet been adapted. Furthermore, no other similar tools exist in the Spanish language. The aim of this paper is to adapt and validate User Version of the MARS (uMARS) into the Spanish language. Materials and Methods Cross-cultural adaptation, translation, and metric evaluation. The internal consistency and test-retest reliability of the Spanish version of the uMARS were evaluated using the RadarCovid app. Two hundred and sixteen participants rated the app using the translated scale. The app was then rated again 2 weeks later by 21 of these participants to measure test-retest reliability. Results No major differences were observed between the uMARS original and the Spanish version. Discrimination indices (item-scale correlation) obtained appropriate results for both raters. The Spanish uMARS presented with excellent internal consistency, α = .89 and .67 for objective and subjective quality, respectively, and temporal stability (r > 0.82 for all items and subscales). Discussion The Spanish uMARS is a useful tool for health professionals to recommend high-quality mobile apps to their patients based on the user’s perspective and for researchers and app developers to use end-user feedback and evaluation, to help them identify highly appraised and valued components, as well as areas for further development, to continue ensuring the increasing quality and prominence of the area of mHealth. Conclusion uMARS Spanish version is an instrument with adequate metric properties to assess the quality of health apps from the user perspective. mobile health, mHealth, mobile applications, patient safety, clinical decisions INTRODUCTION The area of mHealth continues to grow exponentially, with over 400 000 health apps available to consumers globally.1,2 Along with the proliferation of health apps, a need to assess their quality, efficacy and establish regulations with the aim of making the most of mHealth emerges.3–5 Apps are generally well accepted by target users6 and there is an increasing interest in their integration in standard healthcare services.7 Furthermore, the domain of health apps continues to draw researchers’ interests due to the paradigm shift toward patient empowerment and patient-centered models of healthcare delivery.8 Without appropriate evaluation and regulation, health app use bears the risk of multiple potential harms to users (eg, excessive charges, negative health effects, or privacy breaches).9 Little public information on the quality of apps is available to advise consumers of appropriate app selection. App stores provide star ratings that may be artificially inflated and user reviews that are subjective by nature.10 Thus, there is a need for the continuous quality review and evaluation of health apps to help consumers and researchers navigate the mHealth space.11–13 In recent years, scientific publications reviewing the quality of health apps have proliferated14,15; however, those reviews are addressed predominantly to the scientific community and offer limited accessibility to the public. Popular health app blog posts offer a more detailed insight into app quality, but do not rely on systematic quality evaluation and scrutiny, and authors may lack the expertise to assess apps objectively. Among the most reliable hubs of information include the app libraries of reputable organizations which use expert raters and provide detailed information to readers.16 To facilitate research into this area, the Mobile App Rating Scale (MARS), developed by Stoyanov et al,17 offers health professionals and researchers an objective tool for exploring and rating the Engagement, Functionality, Esthetics, and Information quality of health apps. Today the MARS is one of the most widely used tools for health app quality evaluation. The scale was originally developed in English17 and subsequently adapted into Italian,18 German,19 Arabic,20 and Spanish languages.21 Currently, it is being adapted for the evaluation of other e-tools.22 With the increasing focus of healthcare on patient-centered models, in which apps are already present,23,24 it is essential that nonexpert app users are also able to provide their own quality evaluations. A recent review shows that there are practically no tools that allow evaluation from the user’s perspective.25 Thus, it is necessary to note that an adaptation of the original MARS for end-users, the User Version of the MARS (uMARS) questionnaire, offers a solution for this problem.26 The use of uMARS has allowed researchers to determine the user-rated positive and negative quality characteristics of health apps and to highlight areas requiring further development.27 The scale has also facilitated the improvement of the engagement scores,28 the better usability, functionality, and perceived impact29 of health apps or benefits of interventions in which apps are used.30 The growing number of health apps in Spanish requires the translation and adaptation of health app quality scales to facilitate research and evaluation in this area. Spanish is the second most spoken language worldwide. A Spanish version of the professional MARS21 already exists, but the uMARS has not been translated and published to date. Thus, the aim of this study was to adapt uMARS to the Spanish language and to validate the resultant version. MATERIALS AND METHODS Study design The English version of uMARS was cross-culturally adapted for the Spanish language using the same process used in the MARS adaptation.21 To this end, a cross-cultural adaptation, translation, and measure of statistical reliability were developed. The corresponding STROBE checklist was used in this study. Description of uMARS uMARS consists of 20 items assessing objective and subjective app quality,26 rated on a 5-point Likert scale ranging from 1 (“poor”) to 5 (“excellent”). In addition, items 13–16 include the additional option of “not applicable.” The objective quality score is calculated as the mean of the scores of 4 dimensions: engagement (items 1–5), functionality (items 6–9), esthetics (items 10–12), and information (items 13–16). A subjective quality score is obtained as the mean of 4 subjective items (17–20). The final uMARS subscale includes 6 items, designed to assess the perceived impact of the app on the user’s awareness, knowledge, attitudes, intention to change, help-seeking, and the probability of changing the target health behavior, also rated on a 5-point Likert scale ranging from 1 (“strongly disagree”) to 5 (“strongly agree”). The original version of uMARS demonstrated good levels of Interclass correlation coefficients (ICCs) (0.66 and 0.70 in 2 different periods).26 Cross-cultural adaptation and translation process The adaptation and translation processes were developed using several methods. Item and conceptual equivalences were assessed by 2 health professionals with expertise in mHealth. Conceptual equivalence included assessing the relevance, meaning, and importance of the scales and items, while the evaluation of the items included the level of acceptance and relevance for the Spanish culture.31 After that, a translation into Spanish was independently conducted by 2 native Spanish speakers proficient in the English language. Subsequently, a draft version was prepared by consensus and pilot-tested with 8 adults to ensure they understood the item content and response scales. The consensus version was finally developed and blind back-translated by a professional translator, expert in translating health-specific content and finally, to assess the semantic equivalence, the consensus version was reviewed independently by a third researcher, an author of the MARS Spanish version.32 App assessment The test-retest reliability of the uMARS Spanish version was calculated for the subscales and total scores of the scale after 2 weeks of app use, followed by a second assessment 2 weeks later. The research team agreed that the RadarCovid app was an appropriate choice for the reliability trial, as it is available for free from both the Apple and Android stores it is highly popular amongst the public and does not require the use of any additional devices. RadarCovid is an app designed to help to prevent COVID spread by anonymously reporting possible contacts that the user has had in the last 14 days with an infected person. It also contains a section with information about the measures to be adopted to prevent contagion and links to specialized COVID-19-related pages. Convenience sampling methodology was utilized. Eligible participants were Spanish residents aged over 18 years, who had access to an Apple or Android phone. Participants were contacted online using social media (Twitter and Facebook) or message services (WhatsApp). Those who accepted to participate digitally signed an informed consent and were asked to download RadarCovid app and use it for 2 weeks. Thereafter, app ratings were collected through an online survey. Participants were asked about the number of apps and their usage in order to ascertain whether they knew how to use apps, and the number of times they accessed the apps. Those participants who answered 0 were excluded from the analysis. Data analysis The discrimination indexes were calculated for individual items using corrected correlation item-subscale. The internal consistencies of the uMARS subscales and total score were calculated using Cronbach’s alpha. Test-retest reliabilities were calculated for the subscales and total scores of the uMARS. ICCs were calculated using a random-effects average measures model with absolute agreement. Data were analyzed with SPSS version 24. RESULTS Cross-cultural adaptation and translation process Following the methodology described in the “Materials and Methods” section, both the conceptual analysis and the translation were considered relevant and appropriate for the Spanish culture. No major differences were observed between the 2 independent translations of the questionnaire and the final version of uMARS Spanish version was created upon reaching consensus on any discrepancies. Finally, the lead author of this article reviewed and approved the blind reverse translation. Metric properties assessment of the apps Two hundred and sixteen participants assessed the app and 21 participants performed the retest. Participants’ basic characteristics are presented in Table 1. Table 1. Participant characteristics . n = 216 . n = 21 . Age, mean (SD) 39.1 (13.46) 37.90 (13.55) Education level (%)  Primary (elementary school) 21.7 28.6  Secondary (middle and high school) 63.0 66.7  University 15.3 4.8 Mean number of apps on users’ phones (SD) 8.29 (6.96) 6.38 (2.91) Apps frequency of use (%)  Daily 97.7 95.2  Monthly 2.3 4.8 Mean number of times RadarCovid was used each week (SD) 2.62 (2.90) 2.43 (2.52) . n = 216 . n = 21 . Age, mean (SD) 39.1 (13.46) 37.90 (13.55) Education level (%)  Primary (elementary school) 21.7 28.6  Secondary (middle and high school) 63.0 66.7  University 15.3 4.8 Mean number of apps on users’ phones (SD) 8.29 (6.96) 6.38 (2.91) Apps frequency of use (%)  Daily 97.7 95.2  Monthly 2.3 4.8 Mean number of times RadarCovid was used each week (SD) 2.62 (2.90) 2.43 (2.52) Open in new tab Table 1. Participant characteristics . n = 216 . n = 21 . Age, mean (SD) 39.1 (13.46) 37.90 (13.55) Education level (%)  Primary (elementary school) 21.7 28.6  Secondary (middle and high school) 63.0 66.7  University 15.3 4.8 Mean number of apps on users’ phones (SD) 8.29 (6.96) 6.38 (2.91) Apps frequency of use (%)  Daily 97.7 95.2  Monthly 2.3 4.8 Mean number of times RadarCovid was used each week (SD) 2.62 (2.90) 2.43 (2.52) . n = 216 . n = 21 . Age, mean (SD) 39.1 (13.46) 37.90 (13.55) Education level (%)  Primary (elementary school) 21.7 28.6  Secondary (middle and high school) 63.0 66.7  University 15.3 4.8 Mean number of apps on users’ phones (SD) 8.29 (6.96) 6.38 (2.91) Apps frequency of use (%)  Daily 97.7 95.2  Monthly 2.3 4.8 Mean number of times RadarCovid was used each week (SD) 2.62 (2.90) 2.43 (2.52) Open in new tab The internal consistency values (Cronbach’s alpha) were high, as shown in Table 2. Table 2. Internal consistency values (Cronbach’s alpha) Scale . Cronbach’s alpha . Objective quality .89  Section A (Engagement) .69  Section B (Functionality) .79  Section C (Aesthetics) .83  Section D (Information) .81 Subjective quality .67 Total .90 Scale . Cronbach’s alpha . Objective quality .89  Section A (Engagement) .69  Section B (Functionality) .79  Section C (Aesthetics) .83  Section D (Information) .81 Subjective quality .67 Total .90 Open in new tab Table 2. Internal consistency values (Cronbach’s alpha) Scale . Cronbach’s alpha . Objective quality .89  Section A (Engagement) .69  Section B (Functionality) .79  Section C (Aesthetics) .83  Section D (Information) .81 Subjective quality .67 Total .90 Scale . Cronbach’s alpha . Objective quality .89  Section A (Engagement) .69  Section B (Functionality) .79  Section C (Aesthetics) .83  Section D (Information) .81 Subjective quality .67 Total .90 Open in new tab The discrimination indexes (item-scale correlation) were calculated for each of the 6 scales and independent items, and all the items attained adequacy (>0.20), except item 19 (0.15) (Table 3). The temporal stability of the scores assigned by the raters was also examined. To assess the test-retest reliability, the scores assigned at 2 time points were used, with high intracorrelations (ranging from 0.82 to 1) (Table 3). Table 3. Intraclass and item-test correlations Subscale/item . Intraclass correlation (95% CI) . Item-test correlation . Objective quality 0.91 (0.76–0.96)  Section A (Engagement) 0.97 (0.92–0.99)   1 Entertainment 1.00 0.43   2 Interest 0.99 (0.97–0.99) 0.56   3 Customization 0.98 (0.95–0.99) 0.49   4 Interactivity 0.85 (0.63–0.94) 0.45   5 Target group 0.91 (0.79–0.96) 0.28  Section B (Functionality) 0.97 (0.93–0.99)   6 Performance 0.98 (0.95–0.99) 0.42   7 Ease of use 0.95 (0.87–0.98) 0.69   8 Navigation 0.96 (0.90–0.98) 0.65   9 Gestural design 0.98 (0.95–0.99) 0.74  Section C (Aesthetics) 0.95 (0.89–0.98)   10 Layout 0.98 (0.95–0.99) 0.66   11 Graphics 0.98 (0.95–0.99) 0.75   12 Visual appeal 0.92 (0.80–0.97) 0.69  Section D (Information) 0.87 (0.69–0.95)   13 Quality of information 0.85 (0.63–0.94) 0.70   14 Quantity of information 0.91 (0.77–0.96) 0.63   15 Visual information 0.95 (0.87–0.98) 0.58   16 Credibility of source 0.95 (0.88–0.98) 0.60 Subjective quality 0.83 (0.59–0.93)   17 Would you recommend 0.93 (0.82–0.97) 0.69   18 How many times 0.82 (0.56–0.93) 0.47   19 Would you pay 1.00 0.15   20 Overall (star) rating 0.88 (0.70–0.95) 0.55 Awareness 0.92 (0.80–0.97) Knowledge 0.90 (0.77–0.96) Attitudes 0.97 (0.91–0.99) Intention to change 0.95 (0.87–0.98) Help-seeking 0.99 (0.98–1) Behavior change 0.97 (0.94–0.99) Subscale/item . Intraclass correlation (95% CI) . Item-test correlation . Objective quality 0.91 (0.76–0.96)  Section A (Engagement) 0.97 (0.92–0.99)   1 Entertainment 1.00 0.43   2 Interest 0.99 (0.97–0.99) 0.56   3 Customization 0.98 (0.95–0.99) 0.49   4 Interactivity 0.85 (0.63–0.94) 0.45   5 Target group 0.91 (0.79–0.96) 0.28  Section B (Functionality) 0.97 (0.93–0.99)   6 Performance 0.98 (0.95–0.99) 0.42   7 Ease of use 0.95 (0.87–0.98) 0.69   8 Navigation 0.96 (0.90–0.98) 0.65   9 Gestural design 0.98 (0.95–0.99) 0.74  Section C (Aesthetics) 0.95 (0.89–0.98)   10 Layout 0.98 (0.95–0.99) 0.66   11 Graphics 0.98 (0.95–0.99) 0.75   12 Visual appeal 0.92 (0.80–0.97) 0.69  Section D (Information) 0.87 (0.69–0.95)   13 Quality of information 0.85 (0.63–0.94) 0.70   14 Quantity of information 0.91 (0.77–0.96) 0.63   15 Visual information 0.95 (0.87–0.98) 0.58   16 Credibility of source 0.95 (0.88–0.98) 0.60 Subjective quality 0.83 (0.59–0.93)   17 Would you recommend 0.93 (0.82–0.97) 0.69   18 How many times 0.82 (0.56–0.93) 0.47   19 Would you pay 1.00 0.15   20 Overall (star) rating 0.88 (0.70–0.95) 0.55 Awareness 0.92 (0.80–0.97) Knowledge 0.90 (0.77–0.96) Attitudes 0.97 (0.91–0.99) Intention to change 0.95 (0.87–0.98) Help-seeking 0.99 (0.98–1) Behavior change 0.97 (0.94–0.99) Open in new tab Table 3. Intraclass and item-test correlations Subscale/item . Intraclass correlation (95% CI) . Item-test correlation . Objective quality 0.91 (0.76–0.96)  Section A (Engagement) 0.97 (0.92–0.99)   1 Entertainment 1.00 0.43   2 Interest 0.99 (0.97–0.99) 0.56   3 Customization 0.98 (0.95–0.99) 0.49   4 Interactivity 0.85 (0.63–0.94) 0.45   5 Target group 0.91 (0.79–0.96) 0.28  Section B (Functionality) 0.97 (0.93–0.99)   6 Performance 0.98 (0.95–0.99) 0.42   7 Ease of use 0.95 (0.87–0.98) 0.69   8 Navigation 0.96 (0.90–0.98) 0.65   9 Gestural design 0.98 (0.95–0.99) 0.74  Section C (Aesthetics) 0.95 (0.89–0.98)   10 Layout 0.98 (0.95–0.99) 0.66   11 Graphics 0.98 (0.95–0.99) 0.75   12 Visual appeal 0.92 (0.80–0.97) 0.69  Section D (Information) 0.87 (0.69–0.95)   13 Quality of information 0.85 (0.63–0.94) 0.70   14 Quantity of information 0.91 (0.77–0.96) 0.63   15 Visual information 0.95 (0.87–0.98) 0.58   16 Credibility of source 0.95 (0.88–0.98) 0.60 Subjective quality 0.83 (0.59–0.93)   17 Would you recommend 0.93 (0.82–0.97) 0.69   18 How many times 0.82 (0.56–0.93) 0.47   19 Would you pay 1.00 0.15   20 Overall (star) rating 0.88 (0.70–0.95) 0.55 Awareness 0.92 (0.80–0.97) Knowledge 0.90 (0.77–0.96) Attitudes 0.97 (0.91–0.99) Intention to change 0.95 (0.87–0.98) Help-seeking 0.99 (0.98–1) Behavior change 0.97 (0.94–0.99) Subscale/item . Intraclass correlation (95% CI) . Item-test correlation . Objective quality 0.91 (0.76–0.96)  Section A (Engagement) 0.97 (0.92–0.99)   1 Entertainment 1.00 0.43   2 Interest 0.99 (0.97–0.99) 0.56   3 Customization 0.98 (0.95–0.99) 0.49   4 Interactivity 0.85 (0.63–0.94) 0.45   5 Target group 0.91 (0.79–0.96) 0.28  Section B (Functionality) 0.97 (0.93–0.99)   6 Performance 0.98 (0.95–0.99) 0.42   7 Ease of use 0.95 (0.87–0.98) 0.69   8 Navigation 0.96 (0.90–0.98) 0.65   9 Gestural design 0.98 (0.95–0.99) 0.74  Section C (Aesthetics) 0.95 (0.89–0.98)   10 Layout 0.98 (0.95–0.99) 0.66   11 Graphics 0.98 (0.95–0.99) 0.75   12 Visual appeal 0.92 (0.80–0.97) 0.69  Section D (Information) 0.87 (0.69–0.95)   13 Quality of information 0.85 (0.63–0.94) 0.70   14 Quantity of information 0.91 (0.77–0.96) 0.63   15 Visual information 0.95 (0.87–0.98) 0.58   16 Credibility of source 0.95 (0.88–0.98) 0.60 Subjective quality 0.83 (0.59–0.93)   17 Would you recommend 0.93 (0.82–0.97) 0.69   18 How many times 0.82 (0.56–0.93) 0.47   19 Would you pay 1.00 0.15   20 Overall (star) rating 0.88 (0.70–0.95) 0.55 Awareness 0.92 (0.80–0.97) Knowledge 0.90 (0.77–0.96) Attitudes 0.97 (0.91–0.99) Intention to change 0.95 (0.87–0.98) Help-seeking 0.99 (0.98–1) Behavior change 0.97 (0.94–0.99) Open in new tab DISCUSSION This study confirms the metric properties of the Spanish version of uMARS, a new instrument that provides a measurement of app quality from the user’s perspective. In addition, it will also benefit health professionals since in combination with the quality assessment that they can perform with the Spanish version of MARS,21 it will allow them to recommend high-quality mobile apps to their patients. The Spanish version of MARS has been used by developers and health workers in both healthcare and research projects. uMARS could possibly be used in both contexts and be administered and recommended by health workers with clinical and research aims. The sum of the conceptual, item, semantic, operational, and measurement equivalences suggests that the Spanish version of uMARS is reliable in fulfilling the purpose of the original scale and that it is functionally equivalent to the original version.31 Similar metric properties were obtained by the present study as those presented in the English version of the scale.26 Both instruments have excellent internal consistency and inter-rater reliability. The Spanish uMARS presented with slightly higher Cronbach’s alpha values compared to its English counterpart on all, except the engagement subscale, which could be related to the characteristics of the RadarCovid app. Further studies should be developed to ascertain the score. Item 19 showed low inter-item correlations. This could be a result of the Spanish health system being public and universal. And it points to a false belief of gratuity, which leads to people to assume that anything related to health should be free and therefore they do not support paying for the use of a health app. Most people who took the uMARS had secondary or superior education. It would be convenient to develop further research assessing the feasibility of using MARS in a population with a lower educational level. However, the literature already addresses the use of other versions of uMARS in populations with primary education levels,30,32,33 which suggests that the Spanish version of uMARS can be used independently of the population distribution of that variable. Despite the continuously increasing number and potential of health-related apps and the likelihood that they will gradually be integrated into public health services,34,35 to date, there are multiple quality-related issues that hinder their wide applicability.36,37 The uMARS offers a validated heuristic for the objective evaluation and identification of strengths and limitations of existing health apps with the help of end-users. The European Commission38 states that with the increased focus on user-centricity, modern digital health tools provide the opportunity to cross the boundaries between traditional provision and the self-administration of clinical care—a shift that is driven by collaboration and increased user participation. Recent studies highlight the importance and the benefits of engaging patients and prospective users in the design and development of interventions, health tools, and measures.39,40 While the professional MARS contains complex terminology for the population, such as “Evidence base: Has the app been trialled/tested; must be verified by evidence (in published scientific literature)?,”21 and thus does not allow quality assessment from the end-user perspective, which can detract from the rigor of app quality assessment,41 the uMARS addresses this limitation and offers an opportunity for conducting user-centered research. Thus, the current study offers an excellent opportunity for similar studies to be conducted in Spanish-speaking countries globally. Both the MARS and uMARS are now available in the Spanish language, which will make it possible to make a more reliable appraisal of the quality of health apps, from the perspective of end-users or laypersons. The current research showcases that the uMARS Spanish version offers an objective, valid and reliable measure of health app quality. However, it should be noted that there are other essential criteria linked to the evaluation of eHealth and mHealth. For this reason, observational studies should be conducted to determine app effectiveness. Future research should determine the relationship between app quality and app effectiveness, particularly in view of the user perspective. To the best of our knowledge, no research to date reports whether higher user-perceived quality is associated with better app effectiveness. This area of research could pose and answer some very intriguing questions. In addition, as stated earlier, further research should investigate the validity of the uMARS alongside other health app quality assessment measures. Limitations One possible limitation of the study is that the validation was based solely on RadarCovid app. RadarCovid itself offers relatively limited information content and it is likely that other health apps can better showcase the validity and reliability of some of the uMARS items. However, one point should be noted that the measurements observed are well-aligned with the original version of uMARS. Another limitation is that the Spanish uMARS cannot be checked alongside another Spanish scale for validity purposes, as no similar scales exist yet. We hope that with the continuous development of research in this area, other app quality scales will become available and the uMARS can be used alongside those, to assess its validity. Finally, the metric properties assessed were for Castilian Spanish (Spain), which must be taken into consideration if the version is to be used with other Spanish-speaking populations with a different variety of Spanish, such as Latin American. CONCLUSION The uMARS Spanish version is an instrument with adequate metric properties which can be used to assess the quality of health apps from the user perspective. The results of this study show great promise for the scale adaptation. This tool will be very useful to obtain feedback on the quality of health apps from the user perspective, to help health professionals identify and consider for recommendation high-quality apps, based on an empirical evaluation. Researchers and app developers can use the translated tool to utilize end-user feedback and evaluation, to help them identify highly appraised and valued components, as well as areas for further development, to continue ensuring the increasing quality and prominence of the area of mHealth. AUTHOR CONTRIBUTIONS All authors have made a substantial, direct, intellectual contribution to this research. RM-P designed the research. MdMF-A and SC-S conducted the data collection. MC conducted the data analyses. RM-P, MdMF-A, and SS were responsible for data interpretation of results and drafting of the manuscript. All authors critically revised the draft and approved the final version. CONFLICT OF INTEREST STATEMENT None declared. DATA AVAILABILITY The data underlying this article will be shared on reasonable request to the corresponding author. REFERENCES 1 Pohl M. 325,000 Mobile Health Apps Available in 2017—Android Now the Leading mHealth Platform. 2017 . https://research2guidance.com/325000-mobile-health-apps-available-in-2017/ Accessed April 19, 2021. 2 Williams L . Top 10 mHealth Apps That Are Revolutionizing Healthcare. 2020 . https://www.kolabtree.com/blog/top-10-mhealth-apps-that-are-revolutionizing-healthcare/ Accessed April 19, 2021. 3 Ferguson C , Jackson D. Selecting, appraising, recommending and using mobile applications (apps) in nursing . J Clin Nurs 2017 ; 26 ( 21–22 ): 3253 – 5 . Google Scholar PubMed OpenURL Placeholder Text WorldCat 4 Grainger R , Devan H, Sangelaji B, et al. Issues in reporting of systematic review methods in health app-focused reviews: a scoping review . Health Informatics J 2020 ; 26 ( 4 ): 2930 – 45 . Google Scholar Crossref Search ADS PubMed WorldCat 5 Iribarren SJ , Akande TO, Kamp KJ, et al. Effectiveness of mobile apps to promote health and manage disease: systematic review and meta-analysis of randomized controlled trials . JMIR Mhealth Uhealth 2021 ; 9 ( 1 ): e21563 . Google Scholar Crossref Search ADS PubMed WorldCat 6 Boulos MN , Wheeler S, Tavares C, et al. How smartphones are changing the face of mobile and participatory healthcare: an overview, with example from eCAALYX . Biomed Eng Online 2011 ; 10 : 24 . Google Scholar Crossref Search ADS PubMed WorldCat 7 Yang CH , Maher JP, Conroy DE. Acceptability of mobile health interventions to reduce inactivity-related health risk in central Pennsylvania adults . Prev Med Rep 2015 ; 2 : 669 – 72 . Google Scholar Crossref Search ADS PubMed WorldCat 8 Pérez-Jover V , Sala-González M, Guilabert M, et al. Mobile apps for increasing treatment adherence: systematic review . J Med Internet Res 2019 ; 21 ( 6 ): e12505 . Google Scholar Crossref Search ADS PubMed WorldCat 9 Parker L , Bero L, Gillies D, et al. The “Hot Potato” of mental health app regulation: a critical case study of the Australian policy arena . Int J Health Policy Manag 2019 ; 8 ( 3 ): 168 – 76 . Google Scholar Crossref Search ADS PubMed WorldCat 10 Kuehnhausen M , Frost VS. Trusting smartphone apps? To install or not to install, that is the question. In: IEEE International Multi-Disciplinary Conference on Cognitive Methods in Situation Awareness and Decision Support (CogSIMA); 2013 : 30–7. doi: 10.1109/CogSIMA.2013.6523820 . 11 Akbar S , Coiera E, Magrabi F. Safety concerns with consumer-facing mobile health applications and their consequences: a scoping review . J Am Med Inform Assoc 2020 ; 27 ( 2 ): 330 – 40 . Google Scholar Crossref Search ADS PubMed WorldCat 12 Hensher M , Cooper P, Dona S, et al. Scoping review: development and assessment of evaluation frameworks of mobile health apps for recommendations to consumers . J Am Med Inform Assoc 2021 ; 28 ( 6 ): 1318 – 29 . Google Scholar Crossref Search ADS PubMed WorldCat 13 Llorens-Vernet P , Miró J. Standards for mobile health-related apps: systematic review and development of a guide . JMIR Mhealth Uhealth 2020 ; 8 ( 3 ): e13057 . Google Scholar Crossref Search ADS PubMed WorldCat 14 Rincon E , Monteiro-Guerra F, Rivera-Romero O, et al. Mobile phone apps for quality of life and well-being assessment in breast and prostate cancer patients: systematic review . JMIR Mhealth Uhealth 2017 ; 5 ( 12 ): e187 . Google Scholar Crossref Search ADS PubMed WorldCat 15 Wilson H , Stoyanov SR, Gandabhai S, et al. The quality and accuracy of mobile apps to prevent driving after drinking alcohol . JMIR Mhealth Uhealth 2016 ; 4 ( 3 ): e98 . Google Scholar Crossref Search ADS PubMed WorldCat 16 One Mind PsyberGuide . Apps and Digital Health Resources Reviewed by Experts. 2021 . https://onemindpsyberguide.org/ Accessed April 19, 2021. 17 Stoyanov SR , Hides L, Kavanagh DJ, et al. Mobile App Rating Scale: a new tool for assessing the quality of health mobile apps . JMIR Mhealth Uhealth 2015 ; 3 ( 1 ): e27 . Google Scholar Crossref Search ADS PubMed WorldCat 18 Domnich A , Arata L, Amicizia D, et al. Development and validation of the Italian version of the Mobile Application Rating Scale and its generalisability to apps targeting primary prevention . BMC Med Inform Decis Mak 2016 ; 16 : 83 . Google Scholar Crossref Search ADS PubMed WorldCat 19 Messner EM , Terhorst Y, Barke A, et al. The German version of the Mobile App Rating Scale (MARS-G): development and validation study . JMIR Mhealth Uhealth 2020 ; 8 ( 3 ): e14479 . Google Scholar Crossref Search ADS PubMed WorldCat 20 Bardus M , Awada N, Ghandour LA, et al. The Arabic version of the Mobile App Rating Scale: development and validation study . JMIR Mhealth Uhealth 2020 ; 8 ( 3 ): e16956 . Google Scholar Crossref Search ADS PubMed WorldCat 21 Martin-Payo R , Fernandez-Álvarez MM, Blanco-Díaz M, et al. Spanish adaptation and validation of the Mobile Application Rating Scale questionnaire . Int J Med Inform 2019 ; 129 : 95 – 9 . Google Scholar Crossref Search ADS PubMed WorldCat 22 Roberts AE , Davenport TA, Wong T, et al. Evaluating the quality and safety of health-related apps and e-tools: adapting the Mobile App Rating Scale and developing a quality assurance protocol . Internet Interv 2021 ; 24 : 100379 . Google Scholar Crossref Search ADS PubMed WorldCat 23 Bakogiannis C , Tsarouchas A, Mouselimis D, et al. A Patient-Oriented App (ThessHF) to improve self-care quality in heart failure: from evidence-based design to pilot study . JMIR Mhealth Uhealth 2021 ; 9 ( 4 ): e24271 . Google Scholar Crossref Search ADS PubMed WorldCat 24 Petersen CL , Weeks WB, Norin O, et al. Development and implementation of a person-centered, technology-enhanced care model for managing chronic conditions: cohort study . JMIR Mhealth Uhealth 2019 ; 7 ( 3 ): e11082 . Google Scholar Crossref Search ADS PubMed WorldCat 25 Azad-Khaneghah P , Neubauer N, Miguel Cruz A, et al. Mobile health app usability and quality rating scales: a systematic review . Disabil Rehabil Assist Technol 2021 ; 16 : 712 – 21 . Google Scholar Crossref Search ADS PubMed WorldCat 26 Stoyanov SR , Hides L, Kavanagh DJ, et al. Development and validation of the User Version of the Mobile Application Rating Scale (uMARS) . JMIR Mhealth Uhealth 2016 ; 4 ( 2 ): e72 . Google Scholar Crossref Search ADS PubMed WorldCat 27 Ferguson MA , Maidment DW, Gomez R, et al. The feasibility of an m-health educational programme (m2Hear) to improve outcomes in first-time hearing aid users . Int J Audiol 2021 ; 60 ( Suppl. 1 ): S30 – 41 . Google Scholar PubMed OpenURL Placeholder Text WorldCat 28 Serlachius A , Schache K, Kieser A, et al. Association between user engagement of a mobile health app for gout and improvements in self-care behaviors: randomized controlled trial . JMIR Mhealth Uhealth 2019 ; 7 ( 8 ): e15021 . Google Scholar Crossref Search ADS PubMed WorldCat 29 O’Reilly MA , Slevin P, Ward T, et al. A wearable sensor-based exercise biofeedback system: mixed methods evaluation of formulift . JMIR Mhealth Uhealth 2018 ; 6 ( 1 ): e33 . Google Scholar Crossref Search ADS PubMed WorldCat 30 Davidson S , Fletcher S, Wadley G, et al. A mobile phone app to improve the mental health of taxi drivers: single-arm feasibility trial . JMIR Mhealth Uhealth 2020 ; 8 ( 1 ): e13133 . Google Scholar Crossref Search ADS PubMed WorldCat 31 Herdman M , Fox-Rushby J, Badia X. ‘Equivalence’ and the translation and adaptation of health-related quality of life questionnaires . Qual Life Res 1997 ; 6 ( 3 ): 237 – 47 . Google Scholar Crossref Search ADS PubMed WorldCat 32 Peek J , Hay K, Hughes P, et al. Feasibility and acceptability of a smoking cessation smartphone app (My QuitBuddy) in older persons: pilot randomized controlled trial . JMIR Form Res 2021 ; 5 ( 4 ): e24976 . Google Scholar Crossref Search ADS PubMed WorldCat 33 Fijačko N , Gosak L, Cilar L, et al. The effects of gamification and oral self-care on oral hygiene in children: systematic search in app stores and evaluation of apps . JMIR Mhealth Uhealth 2020 ; 8 ( 7 ): e16365 . Google Scholar Crossref Search ADS PubMed WorldCat 34 Kao CK , Liebovitz DM. Consumer mobile health apps: current state, barriers, and future directions . PM R 2017 ; 9 ( 5S ): S106 – S15 . Google Scholar PubMed OpenURL Placeholder Text WorldCat 35 LeBeau K , Huey LG, Hart M. Assessing the quality of mobile apps used by occupational therapists: evaluation using the User Version of the Mobile Application Rating Scale . JMIR Mhealth Uhealth 2019 ; 7 ( 5 ): e13019 . Google Scholar Crossref Search ADS PubMed WorldCat 36 Binyamin SS , Zafar BA. Proposing a mobile apps acceptance model for users in the health area: a systematic literature review and meta-analysis . Health Informatics J 2021 ; 27 ( 1 ): 1460458220976737 . Google Scholar Crossref Search ADS PubMed WorldCat 37 Dou K , Yu P, Deng N, et al. Patients’ acceptance of smartphone health technology for chronic disease management: a theoretical model and empirical test . JMIR Mhealth Uhealth 2017 ; 5 ( 12 ): e177 . Google Scholar Crossref Search ADS PubMed WorldCat 38 European Commission . Shaping Europe’s Digital Future. 2021 . https://digital-strategy.ec.europa.eu/en/policies/ehealth Accessed April 19, 2021. 39 Scholl I , Zill JM, Härter M, et al. An integrative model of patient-centeredness—a systematic review and concept analysis . PLoS One 2014 ; 9 ( 9 ): e107828 . Google Scholar Crossref Search ADS PubMed WorldCat 40 Vaisson G , Provencher T, Dugas M, et al. User involvement in the design and development of patient decision aids and other personal health tools: a systematic review . Med Decis Making 2021 ; 41 ( 3 ): 261 – 74 . Google Scholar Crossref Search ADS PubMed WorldCat 41 O’Neil A , Cocker F, Rarau P, et al. Using digital interventions to improve the cardiometabolic health of populations: a meta-review of reporting quality . J Am Med Inform Assoc 2017 ; 24 ( 4 ): 867 – 79 . Google Scholar Crossref Search ADS PubMed WorldCat © The Author(s) 2021. Published by Oxford University Press on behalf of the American Medical Informatics Association. All rights reserved. For permissions, please email: journals.permissions@oup.com This article is published and distributed under the terms of the Oxford University Press, Standard Journals Publication Model (https://academic.oup.com/journals/pages/open_access/funder_policies/chorus/standard_publication_model) http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png Journal of the American Medical Informatics Association Oxford University Press

Spanish adaptation and validation of the User Version of the Mobile Application Rating Scale (uMARS)

Loading next page...
 
/lp/oxford-university-press/spanish-adaptation-and-validation-of-the-user-version-of-the-mobile-80yAsuxVlP
Publisher
Oxford University Press
Copyright
Copyright © 2021 American Medical Informatics Association
ISSN
1067-5027
eISSN
1527-974X
DOI
10.1093/jamia/ocab216
Publisher site
See Article on Publisher Site

Abstract

Abstract Objective While the professional version of the Mobile App Rating Scale (MARS) has already been translated, and validated into the Spanish language, its user-centered counterpart has not yet been adapted. Furthermore, no other similar tools exist in the Spanish language. The aim of this paper is to adapt and validate User Version of the MARS (uMARS) into the Spanish language. Materials and Methods Cross-cultural adaptation, translation, and metric evaluation. The internal consistency and test-retest reliability of the Spanish version of the uMARS were evaluated using the RadarCovid app. Two hundred and sixteen participants rated the app using the translated scale. The app was then rated again 2 weeks later by 21 of these participants to measure test-retest reliability. Results No major differences were observed between the uMARS original and the Spanish version. Discrimination indices (item-scale correlation) obtained appropriate results for both raters. The Spanish uMARS presented with excellent internal consistency, α = .89 and .67 for objective and subjective quality, respectively, and temporal stability (r > 0.82 for all items and subscales). Discussion The Spanish uMARS is a useful tool for health professionals to recommend high-quality mobile apps to their patients based on the user’s perspective and for researchers and app developers to use end-user feedback and evaluation, to help them identify highly appraised and valued components, as well as areas for further development, to continue ensuring the increasing quality and prominence of the area of mHealth. Conclusion uMARS Spanish version is an instrument with adequate metric properties to assess the quality of health apps from the user perspective. mobile health, mHealth, mobile applications, patient safety, clinical decisions INTRODUCTION The area of mHealth continues to grow exponentially, with over 400 000 health apps available to consumers globally.1,2 Along with the proliferation of health apps, a need to assess their quality, efficacy and establish regulations with the aim of making the most of mHealth emerges.3–5 Apps are generally well accepted by target users6 and there is an increasing interest in their integration in standard healthcare services.7 Furthermore, the domain of health apps continues to draw researchers’ interests due to the paradigm shift toward patient empowerment and patient-centered models of healthcare delivery.8 Without appropriate evaluation and regulation, health app use bears the risk of multiple potential harms to users (eg, excessive charges, negative health effects, or privacy breaches).9 Little public information on the quality of apps is available to advise consumers of appropriate app selection. App stores provide star ratings that may be artificially inflated and user reviews that are subjective by nature.10 Thus, there is a need for the continuous quality review and evaluation of health apps to help consumers and researchers navigate the mHealth space.11–13 In recent years, scientific publications reviewing the quality of health apps have proliferated14,15; however, those reviews are addressed predominantly to the scientific community and offer limited accessibility to the public. Popular health app blog posts offer a more detailed insight into app quality, but do not rely on systematic quality evaluation and scrutiny, and authors may lack the expertise to assess apps objectively. Among the most reliable hubs of information include the app libraries of reputable organizations which use expert raters and provide detailed information to readers.16 To facilitate research into this area, the Mobile App Rating Scale (MARS), developed by Stoyanov et al,17 offers health professionals and researchers an objective tool for exploring and rating the Engagement, Functionality, Esthetics, and Information quality of health apps. Today the MARS is one of the most widely used tools for health app quality evaluation. The scale was originally developed in English17 and subsequently adapted into Italian,18 German,19 Arabic,20 and Spanish languages.21 Currently, it is being adapted for the evaluation of other e-tools.22 With the increasing focus of healthcare on patient-centered models, in which apps are already present,23,24 it is essential that nonexpert app users are also able to provide their own quality evaluations. A recent review shows that there are practically no tools that allow evaluation from the user’s perspective.25 Thus, it is necessary to note that an adaptation of the original MARS for end-users, the User Version of the MARS (uMARS) questionnaire, offers a solution for this problem.26 The use of uMARS has allowed researchers to determine the user-rated positive and negative quality characteristics of health apps and to highlight areas requiring further development.27 The scale has also facilitated the improvement of the engagement scores,28 the better usability, functionality, and perceived impact29 of health apps or benefits of interventions in which apps are used.30 The growing number of health apps in Spanish requires the translation and adaptation of health app quality scales to facilitate research and evaluation in this area. Spanish is the second most spoken language worldwide. A Spanish version of the professional MARS21 already exists, but the uMARS has not been translated and published to date. Thus, the aim of this study was to adapt uMARS to the Spanish language and to validate the resultant version. MATERIALS AND METHODS Study design The English version of uMARS was cross-culturally adapted for the Spanish language using the same process used in the MARS adaptation.21 To this end, a cross-cultural adaptation, translation, and measure of statistical reliability were developed. The corresponding STROBE checklist was used in this study. Description of uMARS uMARS consists of 20 items assessing objective and subjective app quality,26 rated on a 5-point Likert scale ranging from 1 (“poor”) to 5 (“excellent”). In addition, items 13–16 include the additional option of “not applicable.” The objective quality score is calculated as the mean of the scores of 4 dimensions: engagement (items 1–5), functionality (items 6–9), esthetics (items 10–12), and information (items 13–16). A subjective quality score is obtained as the mean of 4 subjective items (17–20). The final uMARS subscale includes 6 items, designed to assess the perceived impact of the app on the user’s awareness, knowledge, attitudes, intention to change, help-seeking, and the probability of changing the target health behavior, also rated on a 5-point Likert scale ranging from 1 (“strongly disagree”) to 5 (“strongly agree”). The original version of uMARS demonstrated good levels of Interclass correlation coefficients (ICCs) (0.66 and 0.70 in 2 different periods).26 Cross-cultural adaptation and translation process The adaptation and translation processes were developed using several methods. Item and conceptual equivalences were assessed by 2 health professionals with expertise in mHealth. Conceptual equivalence included assessing the relevance, meaning, and importance of the scales and items, while the evaluation of the items included the level of acceptance and relevance for the Spanish culture.31 After that, a translation into Spanish was independently conducted by 2 native Spanish speakers proficient in the English language. Subsequently, a draft version was prepared by consensus and pilot-tested with 8 adults to ensure they understood the item content and response scales. The consensus version was finally developed and blind back-translated by a professional translator, expert in translating health-specific content and finally, to assess the semantic equivalence, the consensus version was reviewed independently by a third researcher, an author of the MARS Spanish version.32 App assessment The test-retest reliability of the uMARS Spanish version was calculated for the subscales and total scores of the scale after 2 weeks of app use, followed by a second assessment 2 weeks later. The research team agreed that the RadarCovid app was an appropriate choice for the reliability trial, as it is available for free from both the Apple and Android stores it is highly popular amongst the public and does not require the use of any additional devices. RadarCovid is an app designed to help to prevent COVID spread by anonymously reporting possible contacts that the user has had in the last 14 days with an infected person. It also contains a section with information about the measures to be adopted to prevent contagion and links to specialized COVID-19-related pages. Convenience sampling methodology was utilized. Eligible participants were Spanish residents aged over 18 years, who had access to an Apple or Android phone. Participants were contacted online using social media (Twitter and Facebook) or message services (WhatsApp). Those who accepted to participate digitally signed an informed consent and were asked to download RadarCovid app and use it for 2 weeks. Thereafter, app ratings were collected through an online survey. Participants were asked about the number of apps and their usage in order to ascertain whether they knew how to use apps, and the number of times they accessed the apps. Those participants who answered 0 were excluded from the analysis. Data analysis The discrimination indexes were calculated for individual items using corrected correlation item-subscale. The internal consistencies of the uMARS subscales and total score were calculated using Cronbach’s alpha. Test-retest reliabilities were calculated for the subscales and total scores of the uMARS. ICCs were calculated using a random-effects average measures model with absolute agreement. Data were analyzed with SPSS version 24. RESULTS Cross-cultural adaptation and translation process Following the methodology described in the “Materials and Methods” section, both the conceptual analysis and the translation were considered relevant and appropriate for the Spanish culture. No major differences were observed between the 2 independent translations of the questionnaire and the final version of uMARS Spanish version was created upon reaching consensus on any discrepancies. Finally, the lead author of this article reviewed and approved the blind reverse translation. Metric properties assessment of the apps Two hundred and sixteen participants assessed the app and 21 participants performed the retest. Participants’ basic characteristics are presented in Table 1. Table 1. Participant characteristics . n = 216 . n = 21 . Age, mean (SD) 39.1 (13.46) 37.90 (13.55) Education level (%)  Primary (elementary school) 21.7 28.6  Secondary (middle and high school) 63.0 66.7  University 15.3 4.8 Mean number of apps on users’ phones (SD) 8.29 (6.96) 6.38 (2.91) Apps frequency of use (%)  Daily 97.7 95.2  Monthly 2.3 4.8 Mean number of times RadarCovid was used each week (SD) 2.62 (2.90) 2.43 (2.52) . n = 216 . n = 21 . Age, mean (SD) 39.1 (13.46) 37.90 (13.55) Education level (%)  Primary (elementary school) 21.7 28.6  Secondary (middle and high school) 63.0 66.7  University 15.3 4.8 Mean number of apps on users’ phones (SD) 8.29 (6.96) 6.38 (2.91) Apps frequency of use (%)  Daily 97.7 95.2  Monthly 2.3 4.8 Mean number of times RadarCovid was used each week (SD) 2.62 (2.90) 2.43 (2.52) Open in new tab Table 1. Participant characteristics . n = 216 . n = 21 . Age, mean (SD) 39.1 (13.46) 37.90 (13.55) Education level (%)  Primary (elementary school) 21.7 28.6  Secondary (middle and high school) 63.0 66.7  University 15.3 4.8 Mean number of apps on users’ phones (SD) 8.29 (6.96) 6.38 (2.91) Apps frequency of use (%)  Daily 97.7 95.2  Monthly 2.3 4.8 Mean number of times RadarCovid was used each week (SD) 2.62 (2.90) 2.43 (2.52) . n = 216 . n = 21 . Age, mean (SD) 39.1 (13.46) 37.90 (13.55) Education level (%)  Primary (elementary school) 21.7 28.6  Secondary (middle and high school) 63.0 66.7  University 15.3 4.8 Mean number of apps on users’ phones (SD) 8.29 (6.96) 6.38 (2.91) Apps frequency of use (%)  Daily 97.7 95.2  Monthly 2.3 4.8 Mean number of times RadarCovid was used each week (SD) 2.62 (2.90) 2.43 (2.52) Open in new tab The internal consistency values (Cronbach’s alpha) were high, as shown in Table 2. Table 2. Internal consistency values (Cronbach’s alpha) Scale . Cronbach’s alpha . Objective quality .89  Section A (Engagement) .69  Section B (Functionality) .79  Section C (Aesthetics) .83  Section D (Information) .81 Subjective quality .67 Total .90 Scale . Cronbach’s alpha . Objective quality .89  Section A (Engagement) .69  Section B (Functionality) .79  Section C (Aesthetics) .83  Section D (Information) .81 Subjective quality .67 Total .90 Open in new tab Table 2. Internal consistency values (Cronbach’s alpha) Scale . Cronbach’s alpha . Objective quality .89  Section A (Engagement) .69  Section B (Functionality) .79  Section C (Aesthetics) .83  Section D (Information) .81 Subjective quality .67 Total .90 Scale . Cronbach’s alpha . Objective quality .89  Section A (Engagement) .69  Section B (Functionality) .79  Section C (Aesthetics) .83  Section D (Information) .81 Subjective quality .67 Total .90 Open in new tab The discrimination indexes (item-scale correlation) were calculated for each of the 6 scales and independent items, and all the items attained adequacy (>0.20), except item 19 (0.15) (Table 3). The temporal stability of the scores assigned by the raters was also examined. To assess the test-retest reliability, the scores assigned at 2 time points were used, with high intracorrelations (ranging from 0.82 to 1) (Table 3). Table 3. Intraclass and item-test correlations Subscale/item . Intraclass correlation (95% CI) . Item-test correlation . Objective quality 0.91 (0.76–0.96)  Section A (Engagement) 0.97 (0.92–0.99)   1 Entertainment 1.00 0.43   2 Interest 0.99 (0.97–0.99) 0.56   3 Customization 0.98 (0.95–0.99) 0.49   4 Interactivity 0.85 (0.63–0.94) 0.45   5 Target group 0.91 (0.79–0.96) 0.28  Section B (Functionality) 0.97 (0.93–0.99)   6 Performance 0.98 (0.95–0.99) 0.42   7 Ease of use 0.95 (0.87–0.98) 0.69   8 Navigation 0.96 (0.90–0.98) 0.65   9 Gestural design 0.98 (0.95–0.99) 0.74  Section C (Aesthetics) 0.95 (0.89–0.98)   10 Layout 0.98 (0.95–0.99) 0.66   11 Graphics 0.98 (0.95–0.99) 0.75   12 Visual appeal 0.92 (0.80–0.97) 0.69  Section D (Information) 0.87 (0.69–0.95)   13 Quality of information 0.85 (0.63–0.94) 0.70   14 Quantity of information 0.91 (0.77–0.96) 0.63   15 Visual information 0.95 (0.87–0.98) 0.58   16 Credibility of source 0.95 (0.88–0.98) 0.60 Subjective quality 0.83 (0.59–0.93)   17 Would you recommend 0.93 (0.82–0.97) 0.69   18 How many times 0.82 (0.56–0.93) 0.47   19 Would you pay 1.00 0.15   20 Overall (star) rating 0.88 (0.70–0.95) 0.55 Awareness 0.92 (0.80–0.97) Knowledge 0.90 (0.77–0.96) Attitudes 0.97 (0.91–0.99) Intention to change 0.95 (0.87–0.98) Help-seeking 0.99 (0.98–1) Behavior change 0.97 (0.94–0.99) Subscale/item . Intraclass correlation (95% CI) . Item-test correlation . Objective quality 0.91 (0.76–0.96)  Section A (Engagement) 0.97 (0.92–0.99)   1 Entertainment 1.00 0.43   2 Interest 0.99 (0.97–0.99) 0.56   3 Customization 0.98 (0.95–0.99) 0.49   4 Interactivity 0.85 (0.63–0.94) 0.45   5 Target group 0.91 (0.79–0.96) 0.28  Section B (Functionality) 0.97 (0.93–0.99)   6 Performance 0.98 (0.95–0.99) 0.42   7 Ease of use 0.95 (0.87–0.98) 0.69   8 Navigation 0.96 (0.90–0.98) 0.65   9 Gestural design 0.98 (0.95–0.99) 0.74  Section C (Aesthetics) 0.95 (0.89–0.98)   10 Layout 0.98 (0.95–0.99) 0.66   11 Graphics 0.98 (0.95–0.99) 0.75   12 Visual appeal 0.92 (0.80–0.97) 0.69  Section D (Information) 0.87 (0.69–0.95)   13 Quality of information 0.85 (0.63–0.94) 0.70   14 Quantity of information 0.91 (0.77–0.96) 0.63   15 Visual information 0.95 (0.87–0.98) 0.58   16 Credibility of source 0.95 (0.88–0.98) 0.60 Subjective quality 0.83 (0.59–0.93)   17 Would you recommend 0.93 (0.82–0.97) 0.69   18 How many times 0.82 (0.56–0.93) 0.47   19 Would you pay 1.00 0.15   20 Overall (star) rating 0.88 (0.70–0.95) 0.55 Awareness 0.92 (0.80–0.97) Knowledge 0.90 (0.77–0.96) Attitudes 0.97 (0.91–0.99) Intention to change 0.95 (0.87–0.98) Help-seeking 0.99 (0.98–1) Behavior change 0.97 (0.94–0.99) Open in new tab Table 3. Intraclass and item-test correlations Subscale/item . Intraclass correlation (95% CI) . Item-test correlation . Objective quality 0.91 (0.76–0.96)  Section A (Engagement) 0.97 (0.92–0.99)   1 Entertainment 1.00 0.43   2 Interest 0.99 (0.97–0.99) 0.56   3 Customization 0.98 (0.95–0.99) 0.49   4 Interactivity 0.85 (0.63–0.94) 0.45   5 Target group 0.91 (0.79–0.96) 0.28  Section B (Functionality) 0.97 (0.93–0.99)   6 Performance 0.98 (0.95–0.99) 0.42   7 Ease of use 0.95 (0.87–0.98) 0.69   8 Navigation 0.96 (0.90–0.98) 0.65   9 Gestural design 0.98 (0.95–0.99) 0.74  Section C (Aesthetics) 0.95 (0.89–0.98)   10 Layout 0.98 (0.95–0.99) 0.66   11 Graphics 0.98 (0.95–0.99) 0.75   12 Visual appeal 0.92 (0.80–0.97) 0.69  Section D (Information) 0.87 (0.69–0.95)   13 Quality of information 0.85 (0.63–0.94) 0.70   14 Quantity of information 0.91 (0.77–0.96) 0.63   15 Visual information 0.95 (0.87–0.98) 0.58   16 Credibility of source 0.95 (0.88–0.98) 0.60 Subjective quality 0.83 (0.59–0.93)   17 Would you recommend 0.93 (0.82–0.97) 0.69   18 How many times 0.82 (0.56–0.93) 0.47   19 Would you pay 1.00 0.15   20 Overall (star) rating 0.88 (0.70–0.95) 0.55 Awareness 0.92 (0.80–0.97) Knowledge 0.90 (0.77–0.96) Attitudes 0.97 (0.91–0.99) Intention to change 0.95 (0.87–0.98) Help-seeking 0.99 (0.98–1) Behavior change 0.97 (0.94–0.99) Subscale/item . Intraclass correlation (95% CI) . Item-test correlation . Objective quality 0.91 (0.76–0.96)  Section A (Engagement) 0.97 (0.92–0.99)   1 Entertainment 1.00 0.43   2 Interest 0.99 (0.97–0.99) 0.56   3 Customization 0.98 (0.95–0.99) 0.49   4 Interactivity 0.85 (0.63–0.94) 0.45   5 Target group 0.91 (0.79–0.96) 0.28  Section B (Functionality) 0.97 (0.93–0.99)   6 Performance 0.98 (0.95–0.99) 0.42   7 Ease of use 0.95 (0.87–0.98) 0.69   8 Navigation 0.96 (0.90–0.98) 0.65   9 Gestural design 0.98 (0.95–0.99) 0.74  Section C (Aesthetics) 0.95 (0.89–0.98)   10 Layout 0.98 (0.95–0.99) 0.66   11 Graphics 0.98 (0.95–0.99) 0.75   12 Visual appeal 0.92 (0.80–0.97) 0.69  Section D (Information) 0.87 (0.69–0.95)   13 Quality of information 0.85 (0.63–0.94) 0.70   14 Quantity of information 0.91 (0.77–0.96) 0.63   15 Visual information 0.95 (0.87–0.98) 0.58   16 Credibility of source 0.95 (0.88–0.98) 0.60 Subjective quality 0.83 (0.59–0.93)   17 Would you recommend 0.93 (0.82–0.97) 0.69   18 How many times 0.82 (0.56–0.93) 0.47   19 Would you pay 1.00 0.15   20 Overall (star) rating 0.88 (0.70–0.95) 0.55 Awareness 0.92 (0.80–0.97) Knowledge 0.90 (0.77–0.96) Attitudes 0.97 (0.91–0.99) Intention to change 0.95 (0.87–0.98) Help-seeking 0.99 (0.98–1) Behavior change 0.97 (0.94–0.99) Open in new tab DISCUSSION This study confirms the metric properties of the Spanish version of uMARS, a new instrument that provides a measurement of app quality from the user’s perspective. In addition, it will also benefit health professionals since in combination with the quality assessment that they can perform with the Spanish version of MARS,21 it will allow them to recommend high-quality mobile apps to their patients. The Spanish version of MARS has been used by developers and health workers in both healthcare and research projects. uMARS could possibly be used in both contexts and be administered and recommended by health workers with clinical and research aims. The sum of the conceptual, item, semantic, operational, and measurement equivalences suggests that the Spanish version of uMARS is reliable in fulfilling the purpose of the original scale and that it is functionally equivalent to the original version.31 Similar metric properties were obtained by the present study as those presented in the English version of the scale.26 Both instruments have excellent internal consistency and inter-rater reliability. The Spanish uMARS presented with slightly higher Cronbach’s alpha values compared to its English counterpart on all, except the engagement subscale, which could be related to the characteristics of the RadarCovid app. Further studies should be developed to ascertain the score. Item 19 showed low inter-item correlations. This could be a result of the Spanish health system being public and universal. And it points to a false belief of gratuity, which leads to people to assume that anything related to health should be free and therefore they do not support paying for the use of a health app. Most people who took the uMARS had secondary or superior education. It would be convenient to develop further research assessing the feasibility of using MARS in a population with a lower educational level. However, the literature already addresses the use of other versions of uMARS in populations with primary education levels,30,32,33 which suggests that the Spanish version of uMARS can be used independently of the population distribution of that variable. Despite the continuously increasing number and potential of health-related apps and the likelihood that they will gradually be integrated into public health services,34,35 to date, there are multiple quality-related issues that hinder their wide applicability.36,37 The uMARS offers a validated heuristic for the objective evaluation and identification of strengths and limitations of existing health apps with the help of end-users. The European Commission38 states that with the increased focus on user-centricity, modern digital health tools provide the opportunity to cross the boundaries between traditional provision and the self-administration of clinical care—a shift that is driven by collaboration and increased user participation. Recent studies highlight the importance and the benefits of engaging patients and prospective users in the design and development of interventions, health tools, and measures.39,40 While the professional MARS contains complex terminology for the population, such as “Evidence base: Has the app been trialled/tested; must be verified by evidence (in published scientific literature)?,”21 and thus does not allow quality assessment from the end-user perspective, which can detract from the rigor of app quality assessment,41 the uMARS addresses this limitation and offers an opportunity for conducting user-centered research. Thus, the current study offers an excellent opportunity for similar studies to be conducted in Spanish-speaking countries globally. Both the MARS and uMARS are now available in the Spanish language, which will make it possible to make a more reliable appraisal of the quality of health apps, from the perspective of end-users or laypersons. The current research showcases that the uMARS Spanish version offers an objective, valid and reliable measure of health app quality. However, it should be noted that there are other essential criteria linked to the evaluation of eHealth and mHealth. For this reason, observational studies should be conducted to determine app effectiveness. Future research should determine the relationship between app quality and app effectiveness, particularly in view of the user perspective. To the best of our knowledge, no research to date reports whether higher user-perceived quality is associated with better app effectiveness. This area of research could pose and answer some very intriguing questions. In addition, as stated earlier, further research should investigate the validity of the uMARS alongside other health app quality assessment measures. Limitations One possible limitation of the study is that the validation was based solely on RadarCovid app. RadarCovid itself offers relatively limited information content and it is likely that other health apps can better showcase the validity and reliability of some of the uMARS items. However, one point should be noted that the measurements observed are well-aligned with the original version of uMARS. Another limitation is that the Spanish uMARS cannot be checked alongside another Spanish scale for validity purposes, as no similar scales exist yet. We hope that with the continuous development of research in this area, other app quality scales will become available and the uMARS can be used alongside those, to assess its validity. Finally, the metric properties assessed were for Castilian Spanish (Spain), which must be taken into consideration if the version is to be used with other Spanish-speaking populations with a different variety of Spanish, such as Latin American. CONCLUSION The uMARS Spanish version is an instrument with adequate metric properties which can be used to assess the quality of health apps from the user perspective. The results of this study show great promise for the scale adaptation. This tool will be very useful to obtain feedback on the quality of health apps from the user perspective, to help health professionals identify and consider for recommendation high-quality apps, based on an empirical evaluation. Researchers and app developers can use the translated tool to utilize end-user feedback and evaluation, to help them identify highly appraised and valued components, as well as areas for further development, to continue ensuring the increasing quality and prominence of the area of mHealth. AUTHOR CONTRIBUTIONS All authors have made a substantial, direct, intellectual contribution to this research. RM-P designed the research. MdMF-A and SC-S conducted the data collection. MC conducted the data analyses. RM-P, MdMF-A, and SS were responsible for data interpretation of results and drafting of the manuscript. All authors critically revised the draft and approved the final version. CONFLICT OF INTEREST STATEMENT None declared. DATA AVAILABILITY The data underlying this article will be shared on reasonable request to the corresponding author. REFERENCES 1 Pohl M. 325,000 Mobile Health Apps Available in 2017—Android Now the Leading mHealth Platform. 2017 . https://research2guidance.com/325000-mobile-health-apps-available-in-2017/ Accessed April 19, 2021. 2 Williams L . Top 10 mHealth Apps That Are Revolutionizing Healthcare. 2020 . https://www.kolabtree.com/blog/top-10-mhealth-apps-that-are-revolutionizing-healthcare/ Accessed April 19, 2021. 3 Ferguson C , Jackson D. Selecting, appraising, recommending and using mobile applications (apps) in nursing . J Clin Nurs 2017 ; 26 ( 21–22 ): 3253 – 5 . Google Scholar PubMed OpenURL Placeholder Text WorldCat 4 Grainger R , Devan H, Sangelaji B, et al. Issues in reporting of systematic review methods in health app-focused reviews: a scoping review . Health Informatics J 2020 ; 26 ( 4 ): 2930 – 45 . Google Scholar Crossref Search ADS PubMed WorldCat 5 Iribarren SJ , Akande TO, Kamp KJ, et al. Effectiveness of mobile apps to promote health and manage disease: systematic review and meta-analysis of randomized controlled trials . JMIR Mhealth Uhealth 2021 ; 9 ( 1 ): e21563 . Google Scholar Crossref Search ADS PubMed WorldCat 6 Boulos MN , Wheeler S, Tavares C, et al. How smartphones are changing the face of mobile and participatory healthcare: an overview, with example from eCAALYX . Biomed Eng Online 2011 ; 10 : 24 . Google Scholar Crossref Search ADS PubMed WorldCat 7 Yang CH , Maher JP, Conroy DE. Acceptability of mobile health interventions to reduce inactivity-related health risk in central Pennsylvania adults . Prev Med Rep 2015 ; 2 : 669 – 72 . Google Scholar Crossref Search ADS PubMed WorldCat 8 Pérez-Jover V , Sala-González M, Guilabert M, et al. Mobile apps for increasing treatment adherence: systematic review . J Med Internet Res 2019 ; 21 ( 6 ): e12505 . Google Scholar Crossref Search ADS PubMed WorldCat 9 Parker L , Bero L, Gillies D, et al. The “Hot Potato” of mental health app regulation: a critical case study of the Australian policy arena . Int J Health Policy Manag 2019 ; 8 ( 3 ): 168 – 76 . Google Scholar Crossref Search ADS PubMed WorldCat 10 Kuehnhausen M , Frost VS. Trusting smartphone apps? To install or not to install, that is the question. In: IEEE International Multi-Disciplinary Conference on Cognitive Methods in Situation Awareness and Decision Support (CogSIMA); 2013 : 30–7. doi: 10.1109/CogSIMA.2013.6523820 . 11 Akbar S , Coiera E, Magrabi F. Safety concerns with consumer-facing mobile health applications and their consequences: a scoping review . J Am Med Inform Assoc 2020 ; 27 ( 2 ): 330 – 40 . Google Scholar Crossref Search ADS PubMed WorldCat 12 Hensher M , Cooper P, Dona S, et al. Scoping review: development and assessment of evaluation frameworks of mobile health apps for recommendations to consumers . J Am Med Inform Assoc 2021 ; 28 ( 6 ): 1318 – 29 . Google Scholar Crossref Search ADS PubMed WorldCat 13 Llorens-Vernet P , Miró J. Standards for mobile health-related apps: systematic review and development of a guide . JMIR Mhealth Uhealth 2020 ; 8 ( 3 ): e13057 . Google Scholar Crossref Search ADS PubMed WorldCat 14 Rincon E , Monteiro-Guerra F, Rivera-Romero O, et al. Mobile phone apps for quality of life and well-being assessment in breast and prostate cancer patients: systematic review . JMIR Mhealth Uhealth 2017 ; 5 ( 12 ): e187 . Google Scholar Crossref Search ADS PubMed WorldCat 15 Wilson H , Stoyanov SR, Gandabhai S, et al. The quality and accuracy of mobile apps to prevent driving after drinking alcohol . JMIR Mhealth Uhealth 2016 ; 4 ( 3 ): e98 . Google Scholar Crossref Search ADS PubMed WorldCat 16 One Mind PsyberGuide . Apps and Digital Health Resources Reviewed by Experts. 2021 . https://onemindpsyberguide.org/ Accessed April 19, 2021. 17 Stoyanov SR , Hides L, Kavanagh DJ, et al. Mobile App Rating Scale: a new tool for assessing the quality of health mobile apps . JMIR Mhealth Uhealth 2015 ; 3 ( 1 ): e27 . Google Scholar Crossref Search ADS PubMed WorldCat 18 Domnich A , Arata L, Amicizia D, et al. Development and validation of the Italian version of the Mobile Application Rating Scale and its generalisability to apps targeting primary prevention . BMC Med Inform Decis Mak 2016 ; 16 : 83 . Google Scholar Crossref Search ADS PubMed WorldCat 19 Messner EM , Terhorst Y, Barke A, et al. The German version of the Mobile App Rating Scale (MARS-G): development and validation study . JMIR Mhealth Uhealth 2020 ; 8 ( 3 ): e14479 . Google Scholar Crossref Search ADS PubMed WorldCat 20 Bardus M , Awada N, Ghandour LA, et al. The Arabic version of the Mobile App Rating Scale: development and validation study . JMIR Mhealth Uhealth 2020 ; 8 ( 3 ): e16956 . Google Scholar Crossref Search ADS PubMed WorldCat 21 Martin-Payo R , Fernandez-Álvarez MM, Blanco-Díaz M, et al. Spanish adaptation and validation of the Mobile Application Rating Scale questionnaire . Int J Med Inform 2019 ; 129 : 95 – 9 . Google Scholar Crossref Search ADS PubMed WorldCat 22 Roberts AE , Davenport TA, Wong T, et al. Evaluating the quality and safety of health-related apps and e-tools: adapting the Mobile App Rating Scale and developing a quality assurance protocol . Internet Interv 2021 ; 24 : 100379 . Google Scholar Crossref Search ADS PubMed WorldCat 23 Bakogiannis C , Tsarouchas A, Mouselimis D, et al. A Patient-Oriented App (ThessHF) to improve self-care quality in heart failure: from evidence-based design to pilot study . JMIR Mhealth Uhealth 2021 ; 9 ( 4 ): e24271 . Google Scholar Crossref Search ADS PubMed WorldCat 24 Petersen CL , Weeks WB, Norin O, et al. Development and implementation of a person-centered, technology-enhanced care model for managing chronic conditions: cohort study . JMIR Mhealth Uhealth 2019 ; 7 ( 3 ): e11082 . Google Scholar Crossref Search ADS PubMed WorldCat 25 Azad-Khaneghah P , Neubauer N, Miguel Cruz A, et al. Mobile health app usability and quality rating scales: a systematic review . Disabil Rehabil Assist Technol 2021 ; 16 : 712 – 21 . Google Scholar Crossref Search ADS PubMed WorldCat 26 Stoyanov SR , Hides L, Kavanagh DJ, et al. Development and validation of the User Version of the Mobile Application Rating Scale (uMARS) . JMIR Mhealth Uhealth 2016 ; 4 ( 2 ): e72 . Google Scholar Crossref Search ADS PubMed WorldCat 27 Ferguson MA , Maidment DW, Gomez R, et al. The feasibility of an m-health educational programme (m2Hear) to improve outcomes in first-time hearing aid users . Int J Audiol 2021 ; 60 ( Suppl. 1 ): S30 – 41 . Google Scholar PubMed OpenURL Placeholder Text WorldCat 28 Serlachius A , Schache K, Kieser A, et al. Association between user engagement of a mobile health app for gout and improvements in self-care behaviors: randomized controlled trial . JMIR Mhealth Uhealth 2019 ; 7 ( 8 ): e15021 . Google Scholar Crossref Search ADS PubMed WorldCat 29 O’Reilly MA , Slevin P, Ward T, et al. A wearable sensor-based exercise biofeedback system: mixed methods evaluation of formulift . JMIR Mhealth Uhealth 2018 ; 6 ( 1 ): e33 . Google Scholar Crossref Search ADS PubMed WorldCat 30 Davidson S , Fletcher S, Wadley G, et al. A mobile phone app to improve the mental health of taxi drivers: single-arm feasibility trial . JMIR Mhealth Uhealth 2020 ; 8 ( 1 ): e13133 . Google Scholar Crossref Search ADS PubMed WorldCat 31 Herdman M , Fox-Rushby J, Badia X. ‘Equivalence’ and the translation and adaptation of health-related quality of life questionnaires . Qual Life Res 1997 ; 6 ( 3 ): 237 – 47 . Google Scholar Crossref Search ADS PubMed WorldCat 32 Peek J , Hay K, Hughes P, et al. Feasibility and acceptability of a smoking cessation smartphone app (My QuitBuddy) in older persons: pilot randomized controlled trial . JMIR Form Res 2021 ; 5 ( 4 ): e24976 . Google Scholar Crossref Search ADS PubMed WorldCat 33 Fijačko N , Gosak L, Cilar L, et al. The effects of gamification and oral self-care on oral hygiene in children: systematic search in app stores and evaluation of apps . JMIR Mhealth Uhealth 2020 ; 8 ( 7 ): e16365 . Google Scholar Crossref Search ADS PubMed WorldCat 34 Kao CK , Liebovitz DM. Consumer mobile health apps: current state, barriers, and future directions . PM R 2017 ; 9 ( 5S ): S106 – S15 . Google Scholar PubMed OpenURL Placeholder Text WorldCat 35 LeBeau K , Huey LG, Hart M. Assessing the quality of mobile apps used by occupational therapists: evaluation using the User Version of the Mobile Application Rating Scale . JMIR Mhealth Uhealth 2019 ; 7 ( 5 ): e13019 . Google Scholar Crossref Search ADS PubMed WorldCat 36 Binyamin SS , Zafar BA. Proposing a mobile apps acceptance model for users in the health area: a systematic literature review and meta-analysis . Health Informatics J 2021 ; 27 ( 1 ): 1460458220976737 . Google Scholar Crossref Search ADS PubMed WorldCat 37 Dou K , Yu P, Deng N, et al. Patients’ acceptance of smartphone health technology for chronic disease management: a theoretical model and empirical test . JMIR Mhealth Uhealth 2017 ; 5 ( 12 ): e177 . Google Scholar Crossref Search ADS PubMed WorldCat 38 European Commission . Shaping Europe’s Digital Future. 2021 . https://digital-strategy.ec.europa.eu/en/policies/ehealth Accessed April 19, 2021. 39 Scholl I , Zill JM, Härter M, et al. An integrative model of patient-centeredness—a systematic review and concept analysis . PLoS One 2014 ; 9 ( 9 ): e107828 . Google Scholar Crossref Search ADS PubMed WorldCat 40 Vaisson G , Provencher T, Dugas M, et al. User involvement in the design and development of patient decision aids and other personal health tools: a systematic review . Med Decis Making 2021 ; 41 ( 3 ): 261 – 74 . Google Scholar Crossref Search ADS PubMed WorldCat 41 O’Neil A , Cocker F, Rarau P, et al. Using digital interventions to improve the cardiometabolic health of populations: a meta-review of reporting quality . J Am Med Inform Assoc 2017 ; 24 ( 4 ): 867 – 79 . Google Scholar Crossref Search ADS PubMed WorldCat © The Author(s) 2021. Published by Oxford University Press on behalf of the American Medical Informatics Association. All rights reserved. For permissions, please email: journals.permissions@oup.com This article is published and distributed under the terms of the Oxford University Press, Standard Journals Publication Model (https://academic.oup.com/journals/pages/open_access/funder_policies/chorus/standard_publication_model)

Journal

Journal of the American Medical Informatics AssociationOxford University Press

Published: Oct 6, 2021

References