Access the full text.
Sign up today, get DeepDyve free for 14 days.
Loading next page...
/lp/de-gruyter/evaluation-of-three-educational-use-cases-for-using-virtual-patients-irSb0zV4c1
References
References for this paper are not available at this time. We will be adding them shortly, thank you for your patience.
- Publisher
- de Gruyter
- Copyright
- Copyright © 2015 by the
- ISSN
- 1895-9091
- eISSN
- 1896-530X
- DOI
- 10.1515/bams-2015-0007
- Publisher site
- See Article on Publisher Site
Abstract
Background: Massive Open Online Courses (MOOCs) extended with Virtual Patients (VPs) may foster specific medical skills. In particular, three educational use cases have been proposed to enable interactivity and foster clinical reasoning skills training: collective evaluation of decision making in the context of uncertainty, collective repurposing of cases with division of discussion into subgroups, and computational models in short cases for flexible selection and adaptive learning with VPs. The aim of this study was to evaluate the educational strengths and weaknesses of the proposed use cases. Methods: We went through a two-round modified Delphi process. A panel of experts was formed and asked with open-ended questions to identify the strengths and weaknesses of each use case. The obtained responses were categorized thematically; four specific aspects of the use cases were isolated. In the second phase, the panel was asked to read the collected, categorized responses and prioritize the use cases focusing on each of the four identified aspects. Results: Six experts participated in the process. According to their opinion, decision making in uncertain context was the most feasible in implementation and in fostering clinical reasoning skills training; cultural repurposing was judged to leverage the MOOC potential the most; and computational models in short cases were considered the most interesting use case for the learners. *Corresponding author: Natalia Stathakarou, Karolinska Institutet, Department of Learning Informatics Management and Ethics, 17177 Stockholm, Sweden, Phone: +46 (0) 700979154, E-mail: natalia.stathakarou@ki.se Nabil Zary: Karolinska Institutet, Department of Learning Informatics Management and Ethics, Stockholm, Sweden Andrzej A. Kononowicz: Karolinska Institutet, Department of Learning Informatics Management and Ethics, Stockholm, Sweden; Department of Bioinformatics and Telemedicine, Jagiellonian University Medical College, Kraków, Poland Background Massive Open Online Courses (MOOCs) have been gaining widespread attention by promising open access to highquality education. Currently in different stages of development, MOOCs vary in respect to the followed pedagogies and openness of the content. At present, MOOCs are mainly classified into cMOOCs and xMOOCs; while cMOOCs foster peer learning, xMOOCs are mainly based on videos and automated feedback and have been criticized for enabling passive transmission of knowledge [1, 2]. Besides the criticism of the MOOCs' instructional design and pedagogical approaches, concerns are expressed over the participants' low completion rates [3]. In the field of health-care education, MOOCs have been introduced as the innovation that has the potential to reframe and enhance medical education [4]. A recent study reveals that 98 MOOCs were offered during 2013 related to health-care and medical education and suggests that MOOCs may be used as a way to reinforce continuing medical education [5]. Although online videos may support health-care education [5, 6], they can address just part of the learning process [7]. As a new trend in online education, MOOCs also provide opportunities for large-scale experiments on the instructional design that may allow research on the participants' learning process [8]; recent studies measure the learners' engagement by analyzing large sets of data [9]. Few ones, however, make robust claims about the participants' learning, and even fewer about the potential of particular instructional designs to improve the learning process. As a consequence, there is a need "to adopt a wider range of research designs with greater attention to causal factors promoting student learning" [8]. 114Stathakarou et al.: Virtual Patients in MOOCs: a Delphi study To support health-care education and enable the training of specific medical skills, we have discussed in previous studies the concept of integrating Virtual Patients (VPs) in MOOCs from both educational and technical perspectives [10, 11]. The term "virtual patients" has various meanings in the literature; in this paper, we understand VPs as "interactive computer simulations of real-life clinical scenario for the purpose of health-care and medical training, education, or assessment" [12]. The web-based interactive patient use case is the most common type for clinical reasoning skills and decisionmaking training [13]. There are indications that integrating VPs with MOOCs may enable interactivity and foster clinical reasoning skills training on a massive scale. In particular, three use cases have been suggested [10]: collective evaluation of decision making in the context of uncertainty, collective repurposing of cases with division of discussion into subgroups, and computational models in short cases for flexible selection and adaptive learning with VPs. We hope that following these use cases, the MOOC participants may leverage the distributed nature of intelligence and mass customization the main features of the MOOC environment [3]. As the implementation of the use cases is still in early stages, we aimed to evaluate the use cases with an experts group for the purpose of prioritizing their future development. The Delphi technique was selected in order to obtain feedback on the benefits, threats, and challenges of the proposed use cases. The Delphi technique can be defined as a "method used for the systematic collection and aggregation of informed judgments from a group of experts on specific questions or issues" [14]. The remote character of this process allows the participants to provide uninfluenced answers; the process is recommended to be anonymous [14]. The individuals' answers are combined in order to gain consensus on the defined issues [15] and identify a lack of agreement or an incomplete state of knowledge. In particular, the Delphi technique is characterized and distinguished from other group decision-making processes by the features of allowing adequate time for reflection and a problem-focused approach, providing anonymous uninfluenced expert input, iterations with controlled feedback, and statistical group responses [16, 17]. stakeholders, and it is not required that the expert panel is a representative sample for statistical purposes; representativeness is more important for the quality of the expert panel rather than its number [18]. Three stakeholder groups constituting the experts' panel were identified: health-care providers/educators, VP system developers, and learners/users of the system. By the time that this study was conducted, health-care MOOCs were still in early stages. Therefore, it was decided not to involve MOOC experts that might not be aware of the health-care education needs. In order to form the panel of experts, we identified and targeted those experts willing to participate in the study out of the predefined stakeholder groups; the criterion for inviting the participants was that they should have experience with VPs from the viewpoint of the different stakeholder groups. Ten potential participants were identified within Karolinska Institutet and cooperating universities and were invited by e-mail. Two of the invited participants were health-care professionals, educators, and VP system developers; one of the participants was a health-care professional, educator, and VP researcher; one of the participants was a health-care professional and VP researcher; two of the participants were VP system developers, while the last two were medical students. Consents of the invited participants to use their responses in an anonymous format for the study were requested. The participants were informed about the validation process and the study's purpose. The Delphi method was followed in two stages: in the first phase, we presented the suggested use cases to the participants who were asked with open-ended questions to identify the strengths and weaknesses of each one. The participants were also asked to provide information regarding their current profession and/or activity. The proposed educational use cases were described in a simplified manner, including an introduction about VPs and MOOCs. The survey was validated by a VP system developer and a health-care professional who did not participate in the survey. The Lime Survey [19] tool was used for collecting data from the first phase of the survey. The survey can be found in Supplementary Material, Appendix A that accompanies the article at http://www.degruyter. com/view/j/bams.2015.11.issue-2/bams-2015-0007/bams2015-0007.xml?format=INT. Next, the responses were categorized thematically by focusing the analysis on the purpose of validation and identifying connections between the participants' responses [20]. This led to acquiring different categories of identified issues for the use cases. The purpose of the Materials and methods The Delphi approach recommends that the number of participants exceeds 10, including all the possible Stathakarou et al.: Virtual Patients in MOOCs: a Delphi study115 second validation round was to prioritize the use cases. In particular, the members of the Delphi panel who responded to the first round were first asked to read the collected and categorized responses; next, they were asked to order each of the three use cases for each of the extracted aspects in terms of their potential. The questionnaire can be found in Appendix B that accompanies the article at http://www.degruyter.com/view/j/bams.2015.11.issue-2/ bams-2015-0007/bams-2015-0007.xml?format=INT. The VP system developer and the health-care professional who validated the first phase of the survey also provided recommendations that were considered during the implementation of the second phase. The data collected by the second validation phase were categorized in tables for each of the aspects where the weight of each use case was estimated; the weight of the first ranked use case was considered to be 3 (maximum weight), the second ranked use case 2, while the last ranked use case 1 (minimum weight). That finally led to the relative prioritization of each use case. A. Use case: Collective evaluation of decision making in the context of uncertainty. 1. Advantages for the learners The learners may identify strengths and weaknesses in their clinical reasoning process. The learners have the opportunity to observe the clinical reasoning process of other learners/experts and by that identify alternatives that may deepen the reasoning. Appropriate method for learning treating methods. The method is similar to real clinical decision making, where information is often incomplete. The method can create metacognitive skills. Results First phase of the Delphi study The results were obtained from 6 out of 10 participants who responded to the survey (60% participation). Table 1 presents the identifiers of the participants and their declared profession/activity. The collected responses of the first round of the modified Delphi technique were put into four different categories where possible: advantages/disadvantages for the learners and for the instructors/developers. In particular, while the responses of the participants stated the advantages of the use case A for the instructors and developers, in the use cases B and C no responses were identified to fit into this category. As a result, use cases B and C include the categories advantages/disadvantages for the learners and disadvantages for the instructors/developers. Table 1:Profession of each participant. Participant 1 Participant 2 Participant 3 Participant 4 Participant 5 Participant 6 Health-care professional, educator, and VP system developer Health-care professional, educator, and VP system developer VP system developer Health-care professional, educator, and VP researcher Health-care professional and VP researcher Medical student 2. Advantages for the instructors/developers A good solution to teach how to deal with uncertainty, which is frequent in medicine. The method provides flexibility in the designing of the questions. The question type is appropriate for VPs. The method provides the possibility to add the learners' score to the expert opinions when it is high. 3. Disadvantages for the learners Responses depend on the expertise of the participants and thus might not provide appropriate feedback. By providing feedback for a question extracted from a VP case, one of the advantages of the VPs is diluted: that the feedback is provided at a later stage, bringing the VP close to a real patient. Experts and novices argue in a different way; the novice might not be able to understand the experts' reasoning results. The case can be simplified and by that diminish the reality of the real patient cases. The method may not be appropriate to examine novices. 4. Disadvantages for the developers/instructors Complex technical implementation. Need to be designed based on guidelines and approaches that exist. Possibility of collecting experts' answers that vary considerably, in case that the provided answers' options are only subtly different and thus lack a reliable consensus to gauge learners' answers against. The method requires a sizable pool of experts. The score of a learner, having a high score in clinical reasoning while not enough experience in treating and decision making, can be added to the experts' one. 116Stathakarou et al.: Virtual Patients in MOOCs: a Delphi study B. Use case: Cultural repurposing. 1. Advantages for the learners Possibility to learn facts, different perspectives, approaches, and cultural differences. The method allows the students to construct their own knowledge through peer discussion. Localization hierarchy adds flexibility in navigating in the discussion board. Flexibility in choosing which discussion the learner would like to participate in. The method is beneficial for the decision-making process. The method can differentiate the local guidelines. The method allows learners to identify problems and issues. The method provides opportunity to the learner to experiment with the content, which can be a powerful way of learning. Learners can try individually to experiment with the cases without being based on the MOOC community. Experimenting with the cases can enhance clinical reasoning skills. The method might provide the opportunity to the learners to learn from inappropriate options for treating or decision making. The method may particularly benefit learning for physiology and biochemistry. The method may particularly benefit learning for management of chronic diseases. 2. Disadvantages for the learners Difficulty in triggering a discussion because of potential low participation. A particularly engaged learner is required to browse and monitor the discussion forums. The quality and depth of the discussion can be diminished by unmotivated learners. The hierarchical form of the discussion board can limit the depth of the discussion and the learning opportunities the learners are exposed to. Simplified discussions can mislead the learners about the real cultural variations in the processes. The method might not be suitable for common cases. This method might not be beneficial for learners in particularly advanced VP cases unless an expert is involved. 3. Disadvantages for the developers/instructors Participants need to be provided with clear benefits in order to participate in/return to the discussion. Tasks should be included to motivate the learners to participate. Discussion boards usually are not easy to navigate, discouraging the learners not to participate. If learners do not receive notification e-mails about responses in the forum they participate in, they might not return to the conversation. 2. Disadvantages for the learners The usefulness of this approach is dependent on the expertise of users. Learners might insert values or experiment with the content without seriously considering their actions. The parameters should be appropriate to the learning level of the MOOC participants as they might lack experience to apply them. The method might not be appropriate for novices. The cases might seem unrealistic. 3. Disadvantage for the instructors/developers The method requires good designing for the cases. Technically difficult implementation considering all the variants. The authenticity of the case can be lost if clinicians are not involved in the development of the cases. Requires a collection of validated and maybe real data. Second phase of the Delphi study From the themes identified in the first validation phase, four of the categories were extracted for use in the second validation phase. The selection of the issues was informed by their relevance in meeting the aims of the study. Two of the issues were selected as they meet the aim of the integration: to enhance the clinical reasoning skills by taking advantage of the MOOC opportunities. The next issue was selected because it highlights a problem of the current form of MOOCs: the low completion rates; while the last one was selected to rank the technical feasibility of the use cases, as an important aspect to be considered before moving to their actual implementation. In this phase of the survey, five out of the six participants from the first phase of the Delphi process responded C. Use case: Computational models in short cases. 1. Advantages for the learners The case can be enriched by the generated values of the students while experimenting with the learning discovering tasks. Stathakarou et al.: Virtual Patients in MOOCs: a Delphi study117 Table 2:Use cases' prioritization for the first aspect: enhancing learners' clinical reasoning skills. Rank/participant ID Participant 1 Participant 2 Participant 3 Participant 4 Participant 5 3 C A A A A 2 A C B C B 1 B B C B C Table 4:Use cases' prioritization for the third aspect: taking advantage of the MOOCs' potential. Rank/participant ID Participant 1 Participant 2 Participant 3 Participant 4 Participant 5 3 B B A B B 2 A A B A A 1 C C C C C in prioritizing the proposed use cases (83.3% participation: participants 15 listed in Table 1). The responses were categorized according to each aspect and are presented in Tables 25. For each aspect, the collected answers are ranked; the relative prioritization of each use case is visualized in Figure 1. In each table and in the figure, we mark the use cases as follows: (A) collective evaluation of decision making in the context of uncertainty, (B) cultural repurposing, and (C) computational models in short cases. Discussion Interpretation of the results We evaluated three use cases for integrating VPs in MOOCs by applying a modified Delphi technique. By that, we aimed to get insights into the potential benefits, threats, and challenges of each use case and prioritize their implementation. Yet, the educational impact of the proposed use cases on the participants is still uncertain and should be investigated in future studies. The results revealed that, in the opinion of experts, the first use case collective evaluation of decision making in the context of uncertainty is of highest priority for enhancing the clinical reasoning skills and of feasibility in implementation. While considering the aspect of raising the learners' interest, the computational models in short cases were prioritized; the cultural repurposing use case was anticipated to provide the greatest advantage of the MOOCs' potential. Table 3:Use cases' prioritization for the second aspect: potential of raising learners' interest. Rank/participant ID Participant 1 Participant 2 Participant 3 Participant 4 Participant 5 3 C C C B C 2 B A A C B 1 A B B A A The obtained preferences form an interesting pattern where each use case is rated highest in one of the established aspects. This encourages us to continue our work on all three suggested ideas. It was surprising for us to see that the participants regarded use case A as the easiest to implement. We did not expect that considering the complexity of taking into account the various levels of expertise [21] and cultural diversity of the MOOC users [22]. Only one of the participants noticed the technical complexity of the required scoring algorithm. It was also unanticipated by us that the experts will not prioritize use case A as a method for taking advantage of the MOOC potential; in particular, we expected that use case A has the potential to bring insights in regard to uncertainty and reasoning differentiation driven by the mass collection of the results. Adding discussion boards to individual screen cards to discuss the diversity of diagnosis and treatment process (use case B) was regarded as least feasible; the experts were worried that despite the massive participation in the course, the learners will not be motivated enough to discuss in regional subgroups, which might also lead to lack of quality in discussion forums. Relevant studies report mixed MOOC participants' experiences with the discussion forums, where they were either disappointed by poor quality or overwhelmed by the wealth of comments when including mass unstructured threads [23, 24]. Being aware of this comment, we still believe, however, that dividing the discussion forum into local/regional levels may have the potential to address part of the reported issues, such as lack of structure and the anonymity. Also encouraging was the experts' opinion that Table 5:Use cases' prioritization for the fourth aspect: feasibility of their implementation. Rank/participant ID Participant 1 Participant 2 Participant 3 Participant 4 Participant 5 3 A A C A A 2 C B B C C 1 B C A B B 118Stathakarou et al.: Virtual Patients in MOOCs: a Delphi study Enhancing learners' clinical reasoning skills Taking advantage of the MOOCs' potential A 17% 37% B C A 30% 46% 24% B C 46 % Potential of raising learners' interest Feasibility of their implementation 24% 46% 30% A B C 23% 33% 44% A B C Figure 1:Use cases' relative prioritization. (A) Collective evaluation of decision making in the context of uncertainty; (B) cultural repurposing; (C) computational models in short cases. the divided discussion boards may take advantage of the MOOCs' potential. Regarding use case C, the concept of integrating computational models within VP cases has received positive feedback in previous studies [25] and according to the experts has the greatest potential in raising the learners' interest. Even though case C was considered by the experts to take the least advantage of the MOOC potential, we see the possibility that computational models integrated into short cases enable adaptive learning and provide insights of the participants' learning experience. During the prioritization process, a comment was made by one of the participants, emphasizing that different levels of the learners require different training methods; the response of this participant prioritized the use cases for the first two aspects in two different ways for senior and junior medical students. As in the first phase of the survey, one of the identified issues considered the use cases to be appropriate for an advanced level of learners; the prioritizing of the use cases for senior medical students were selected to represent the response of this participant. This observation indicates that the rating of use cases is complex as it might depend on several external factors like training level, specific topic of the MOOC, or quality of implementation of use cases; while, on the other hand, the MOOC is open to a wide audience and includes different learning groups. Limitations of the study The Delphi process involved a panel of experts of three different stakeholder groups: health-care providers/ educators, VP system developers, and learners/users of the system. Ten potential participants were identified to meet the defined stakeholder groups and were invited by e-mail; however, only five responses were collected and analyzed to illustrate the relative prioritization of the cases by the end of the study. Although the participants were representing all the stakeholders that were identified and the data acquired by the Delphi approach validation were not intended to be used for statistical purposes but rather to acquire a qualitative view of the strengths and weaknesses of the use cases, the participants' number should exceed 10, as recommended by the Delphi approach. This requirement could not be fulfilled in this study. Conclusions The widespread interest in MOOCs motivates research on new methods that may benefit the learning process. In this study, we evaluated three previously suggested educational use cases for integrating VPs in MOOCs in order to identify their potential strengths and weaknesses. We hope by that to have provided insights into the educational perspective of the integration and invited further Stathakarou et al.: Virtual Patients in MOOCs: a Delphi study119 research to reflect on learning in this type of educational environments. Acknowledgments: The authors would like to acknowledge their gratitude to the experts participating in this study for their substantial contributions. In addition, we would like to thank the KiX MOOC team and the Open Labyrinth community for many inspiring discussions. Author contributions: All the authors have accepted responsibility for the entire content of this submitted manuscript and approved submission. Research funding: None declared. Employment or leadership: None declared. Honorarium: None declared. Competing interests: The funding organization(s) played no role in the study design; in the collection, analysis, and interpretation of data; in the writing of the report; or in the decision to submit the report for publication.
Journal
Bio-Algorithms and Med-Systems – de Gruyter
Published: Jun 15, 2015