Get 20M+ Full-Text Papers For Less Than $1.50/day. Start a 14-Day Trial for You or Your Team.

Learn More →

Learner-generated versus author-provided computer-based flow diagrams in medical education

Learner-generated versus author-provided computer-based flow diagrams in medical education principles), whereas flow diagrams help in mastering procedural knowledge (knowledge how to solve concrete problems). The relation between these two knowledge types is complex and still the subject of *Corresponding author: Andrzej A. Kononowicz, Department of Bioinformatics and Telemedicine, Jagiellonian University Medical College, Lazarza 16, 31-034, Kraków, Poland, E-mail: andrzej.kononowicz@uj.edu.pl Jakub Kenig: Third Department of General Surgery, Jagiellonian University Medical College, Kraków, Poland Introduction Graphic organizers are `two-dimensional visual knowledge representations' including flow diagrams, knowledge and concept maps, timelines and tables [1]. Provision of graphic organizers which illustrate the content of learning material is a common method of supporting meaningful learning of complex text passages. Alternatively, learners are asked to construct their own graphic organizers based on what they have read to facilitate comprehension and 38Kononowicz and Kenig: Computer-based flow diagrams in medical education intensive research [8]. Davidowitz and Rollnick used flow diagrams as a method of preparation for laboratories in chemistry observing positive responses from students [9]. DeMeo reported on the construction of flow diagrams as an alternative to lectures for introducing students to the solving of aqueous acid base equilibria problems [10]. An important research question centers on whether learners should create their own graphic organizers or use those provided by their instructors. It does not seem to be easy to give an a priori judgment on which option is to be preferred because there are theories supporting both approaches. When constructing graphic organizers, students organize the text into coherent structures and connect them by relations [2]. It engages students in a productive learning activity through the construction of their own mental representations of the content to be learned, and this may be regarded as a sign of `deeper learning'. Craik and Lockhart's [11] theory of levels of processing posits that deeper learning activities result in more durable knowledge. By contrast, in the process of learning-by-viewing students are prompted to become involved in learning by comparing how a linear text was converted into a spatial structure prepared by the learning activity instructor [2]. Cognitive load theory prefers learningby-viewing over learning-by-doing scenario because the construction of graphic organizers by students induces extraneous processing which limits the available mental resources for generative processing and thus makes the learning process less efficient [12]. A study by Stull and Mayer [2] compared the learning outcomes of learner-generated with author-provided graphical organizers illustrating a reading material from a popular biology textbook. The study involved a group of college students from a psychology participant pool. It was concluded that increased activity on the part of the learner caused by producing graphic organizes does not lead to deeper understanding. Students who used author-generated graphic organizers learned faster and with better results than the students preparing their own graphic organizers. computer-based. This provided the students with additional opportunities that were not available in `pen and paper' drawings such as unlimited space, easy corrections and interactive elements hiding the complexity of descriptions. Students did not work individually but in pairs, which was expected to decrease the extraneous processing related to constructing graphic organizers. Students could discuss their ideas on how to draw the flow diagrams and work in parallel on the text. In their study, Stull and Mayer applied different types of graphic organizers (flow diagrams, hierarchies, small concept maps), whereas in this study only computer-based flow diagrams were used. This paper also enhanced Stull and Mayer's study with new analyses. A knowledge test was carried out not only immediately before and after the learning activity but also 1 month later to determine the durability of acquired knowledge. Student satisfaction was also analyzed. The quality of graphic organizers was assessed to check whether it influenced the learning outcome measured by the knowledge test. This study also aimed to test the application of flow diagrams in medical teaching. The papers referenced in the introductory part reported on the application of flow diagrams in chemistry teaching [9, 10]. The authors of this paper are not aware of any similar experiments in medical education. In addition, the reports from the application of flow diagrams in chemistry concentrated merely on student satisfaction and did not involve any forms of knowledge testing. Materials and methods Study protocol The study group consisted of 36, third to fifth year, voluntary students recruited from an extracurricular group of students at Jagiellonian University Medical College interested in general surgery. Students organized themselves into pairs and were randomly assigned to groups A or B. Student identities were encoded to make the result analysis anonymous. Pairs of students were randomly assigned to two groups: group A ­ author-provided graphic organizer (learning-by-viewing scenario); group B ­ learnergenerated graphic organizer (learning-by-doing scenario). All students took the same pre-test. After the pre-test both groups were put in two separate rooms. Students received the same review paper on Boerhaave's syndrome [13] and were instructed to read it through carefully. Purpose of the study The aim of this paper was to shed more light on the proper application of learning-by-doing and learning-by-viewing scenarios involving graphic organizers in a computerbased environment. In contrast to the paper by Stull and Mayer [2], the authoring of graphic organizers in this study was Kononowicz and Kenig: Computer-based flow diagrams in medical education39 Boerhaave's syndrome is an often misdiagnosed spontaneous rupture of the esophagus which leads to death if untreated. Rapid decisions regarding proper management are crucial for good chance of recovery. The paper was slightly modified with the exclusion of a graphic organizer and references to it, and a few sentences, to ensure that the removed graphic organizer could be constructed solely based on the text of the paper, were added. The length of the paper was approximately 1750 words. Group A also received a link to an electronic version of the author-provided graphic organizer on Boerhaave's syndrome (Figure 1) and a sheet with a printout of the graphic organizer with digits assigned to some of the nodes in the diagram. The printout did not contain text descriptions which were displayed when the diagram elements in the electronic version were clicked. The task of the students in group A was, while working in pairs, to go through the diagram elements with assigned digits and mark (with these digits) fragments of the article corresponding to the content of the preselected flow diagram elements. Students in group B were asked to create, in pairs, a flow diagram presenting the content of the paper on Boerhaave's syndrome using the `Bit Pathways' program [14]. Students had access to a short tutorial on how to use the `Bit Pathways' application, a short description of all flow diagram elements available in `Bit Pathways' for this study (i.e., start/end elements, task element, decision element and connectors) and a printout of a sample graphic organizer from a different topic (in-hospital resuscitation) also created with that application. All students had already experience with the `Bit Pathways' program after a class on clinical pathways in the `Basics of Computer Science' course in their third year of study. No time limit was imposed on either group. Students worked as long as they wished and were allowed to communicate within their pairs, but not with other pairs participating in the study. Students sat individually for the post-test and answered the satisfaction survey immediately after completing their tasks. They were not informed about the correct answers in the tests or given any feedback regarding their performance in the tasks in order to not influence study results. One month after the study students from both experimental groups were asked to write a follow-up knowledge Figure 1The author-provided schema on the management of Boerhaave's syndrome. 40Kononowicz and Kenig: Computer-based flow diagrams in medical education test without prior notice. In addition, students were given the task of reconstructing the author-provided graphic organizers showing the optimal procedure for the treatment of Boerhaave's syndrome having at their disposal a list of permitted action and decision elements. The problem of finding a universal and objective evaluation of graphic organizers is very complex and is beyond the focus of this paper. Research reports on developing rubrics for learner-generated graphic organizers may be found, for example, in [15, 16]. For the purpose of this paper, a simple, preliminary scale for grading the graphic organizers created in group B was proposed. The scale consisted of three measures evaluating: (i) content of the pathway (max. 10 points), (ii) its legibility (max. 5 points) and (iii) notation (max. 3 points). For evaluation of the content, the presence (1 point each) and correct location (1 point each) of three crucial action elements (`endoscopy treatment', `surgery' and `conservative treatment') was checked. In addition, the presence (1 point each) and placement (1 point each) of two decision elements was examined (`rupture < 48h ago?' and `signs of sepsis?'). The presence (or absence) of other elements was not graded. Legibility was graded by a 5-point Likert scale question checking the reviewer's attitude towards the sentence `the pathway presented the content of the paper in an easily understandable way' (5 ­ strongly agree, 1 ­ strongly disagree). The `notation' measure checked the standard-conformant use of flow diagram elements: 3 ­ no errors, 2 ­ minor errors, 1 ­ lack of understanding of the notation used in constructing flow diagrams. Grading was carried out by the authors of this paper (J.K. and A.K.). A preliminary observation of the results of learnergenerated graphic organizers suggested that the students in group B had problems selecting the right actions and decision elements for the graphic organizers. We investigated whether students from group B could construct a correct pathway if they had preselected flow diagram elements at their disposal. Thus, an additional assignment was given to check the students' success ratio in arranging the recommended actions in the right order. This exercise was conducted immediately after the follow-up test, that is, 1 month past the learning activity. All action and decision elements from the author-provided graphic organizer were presented in random order on the margin of a blank sheet of paper. The students' task was to create from them (using the `paper and pen' method due to the lack of sufficient computers in the examination room) a correct flow diagram presenting the right treatment procedure in the case of Boerhaave's syndrome. The graphic organizers created by students were divided into two classes: correct and incorrect. The correct class included graphic organizers showing the tasks and decision elements in the right clinical order ­ minor deviations not influencing the clinical outcome of the patients were permitted. The incorrect flow diagrams showed inadequate order of the tasks and decision elements that were not recommended by the paper and were potentially harmful for the patients. Students from group A were asked to perform the same task in order to form a reference group. A survey into students' satisfaction after the learning activity comprises four questions: three of them in a 5-point Likert scale (5 ­ definitely agree, ..., 1 ­ definitely disagree) asking about students' attitudes towards the following statements: (S1) I have learned useful things in the meeting today; (S2) I liked the learning method applied in my experimental group; (S3) I have the feeling I have learned a lot today. The fourth survey item was a free text question asking about general opinions and impressions after the learning activity. Materials Three knowledge tests were carried out: a pre-test (immediately before the learning activity), post-test (immediately after the learning activity) and a follow-up test (1 month after the learning activity). The tests were developed by the second author of the paper (J.K.). All tests contained the same 12 multiple choice question items regarding proper treatment of Boerhaave's syndrome. All questions were equally rated: 1 point for a correct answer, 0 points for a wrong or no answer, resulting in a maximum of 12 points. To prevent the students from memorizing the questions by rote, the order of the questions/answers in subsequent tests was altered and new `distractor' questions regarding Boerhaave's syndrome were added to the test. Outcomes of the `distractor' questions were not analyzed. The author-provided and all learner-generated graphic organizers were constructed using the `Bit Pathways' application, developed by Kononowicz and Holler [14]. The system was developed to give subject matter experts, teachers and students an easy way of presenting procedural knowledge in a web-based environment. It enables the authoring of flow diagram-like graphic organizers by dragging and dropping a set of elements (action, decision, start/ end elements) from the toolbar onto the canvas and connecting them by arrows showing the chronological order of tasks. All elements may have a structured text description (without space limitations) edited via a property panel. The text-intensive descriptions of the element are displayed in pop-up windows upon activation of a diagram element by Kononowicz and Kenig: Computer-based flow diagrams in medical education41 the mouse cursor. The graphic organizers are stored on a central server and can be viewed by a conventional web browser such as Internet Explorer or Firefox. Figure 1 presents the author-provided schema on Boerhaave's syndrome created in the `Bit Pathways' program. The content of the graphic organizer is based on a paper by de Schipper etal. [13] and was slightly changed (by adding a diagnosis element) to address the learning objectives of the class. The diagram consists of four action elements, three decision elements, eleven connectors, one start and three end elements. The additional descriptions displayed by clicking the elements of the diagram contained, in total, 420 words copied directly from the paper by de Schipper etal. [13]. All graphic organizers were available or constructed in English and not in the students' native Polish language. However, all students had previously declared knowledge of the English language and the inconvenience was the same for both groups and therefore it should have affected the final results of all students in a similar way. Table 1Demographic structure of the study groups. All n Female Female Male Male Year (mean) Year (3) Year (4) Year (5) 36 21 58.3% 15 41.7% 3.9 8 22 6 Group A 20 12 60.0% 8 40.0% 4.0 4 12 4 Group B 16 9 56.3% 7 43.8% 3.9 4 10 2 students of different years in similar proportions in both study groups (by having age as one of the parameters in the random group assignment procedure), we do not consider the year of study as a major factor because Boerhaave's syndrome is not formally part of the medical curriculum at Jagiellonian University Medical College. Time Data analysis Statistical data analysis was done in Statsoft Statistica 9.0 (Statsoft Inc., Tulsa, OK, USA). Validity of the assumption of normal distribution was tested with Shapiro-Wilk's test. The equality of variances was confirmed with Levene's test. Knowledge gain for repeated measures was calculated using Friedman's test. Differences between two sample means (if assumptions were met) were tested using Student's t-test. In the case of non-normally distributed samples, non-parametric Mann-Whitney's U-test was applied. Dichotomous variables were compared using Fisher's exact test. Effect size was calculated using Cohen's d with pooled standard deviations (SDs). Correlations between data were checked with Spearman's rank correlation coefficient. Significance level was set at 0.05. The duration of the learning activity was measured from the start of the pre-test until the start of the post-test. The time taken by two students in group A was not available due to a technical problem. A summary of all time results is presented in Table 2 (time scale: hh:mm:ss, hh ­ hours, mm ­ minutes, ss ­ seconds). The exercise took statistically significant less time for group A than group B (p < 0.001, Student's t-test). Knowledge acquisition Four students of the initial group (two from group A and two from group B) were missing at the follow-up test conducted 1 month after the pre- and post-tests and thus their results have been omitted from knowledge acquisition analysis. Table 3 presents a summary of all knowledge test results. There was no difference between group A and group B in the pre-test score (p = 0.99, Student's t-test). The normality assumption was not met for the follow-up test in group Results The sample Table 2Time spent on the tasks in groups A and B. Table 1 presents the age and gender profiles of the study groups. The majority of students who participated in the study were from fourth year. There were slightly more women than men in the study. The division of students into study groups retained the age and gender proportions. Even though particular attention was paid to having Group A 18 Group B 16 01:48:10 00:15:41 Time hh:mm:ss Mean SD 01:17:16 00:14:49 42Kononowicz and Kenig: Computer-based flow diagrams in medical education Table 3Results of knowledge tests. n Group A 18 PrePostFollow-up Group B 14 4.21/1.19 7.64/1.78 5.29/1.77 p-Value organizer grade and score on the follow-up test could was not statistically significant (n=7, Spearman's R=0.54, p=0.21), even though the trend suggests that for a larger group of study subjects a relationship could be demonstrated. Mean (pts)/SD Mean (pts)/SD Mean (pts)/SD 4.22/1.66 8.17/1.72 6.94/2.15 0.99a 0.41a 0.03b Graphic organizer reconstruction The results of the graphic organizer reconstruction assignment are presented in Table 5. The task was performed significantly better by group A (Fisher's exact test, p = 0.01), which is not surprising considering the fact that group A had seen the correct graphic organizer 1 month previously. More surprising was the large percentage of students from group B (62%) making significant mistakes in their flow diagrams even when provided with the correct selection of crucial steps in the algorithm. p-Value according to Student's t-test; bp-value according to MannWhitney's U-test. B (p = 0.02, Shapiro-Wilk's test), and for that reason no ANOVA test for repeated measurements could be applied. The learning activity produced significant knowledge gains in both groups (pA < 0.001; pB < 0.001, Friedman's test). The mean value of the post-test for the given sample was higher in group A than in group B, but the difference did not reach a statistically significant level (p = 0.41, Student's t-test). In the follow-up test, the difference between the groups was significant (p = 0.03, Mann-Whitney's U-test, = 0.05, Cohen's d = 0.84). Students' satisfaction The results of the satisfaction survey are presented in Table 6. Students of both groups liked the learning activity. The satisfaction level in group A was extraordinarily high (e.g., S1 = 4.95 and S2 = 4.80 in a max. 5-point scale). The results in group B were statistically significantly worse (p-Value in Mann-Whitney's U-test), but still fairly good (mean value of all questions above 4 in a 5-point Likert scale). Some of the students complained in the free text comments that the article was in English rather than in their native language, as this hindered their comprehension. Evaluation of learner-generated flow diagrams A total of eight graphic organizers were created by students from group B who had teamed up into groups of two. Table 4 presents a quantitative summary of the size of author-provided versus learner-generated graphic organizers. Graphic organizers presented by students were on average twice as large as those provided by the authors for group A. Many of the additionally added elements contained data irrelevant for the treatment process or even statements not present in the paper. There was a strong positive correlation between the graphic organizer score of the group and the sum of scores obtained by group members on the post-test (n=8, Spearman's R=0.75, p=0.03). This suggests that the quality of the graphic organizer might be a good measure of students' knowledge acquisition. The correlation of the graphic Table 4Quantitative comparison of the author-provided and learner-generated graphic organizers. Authorprovided Number of action elements Number of decision elements Number of connectors 4 3 11 Learner-generated Mean 10.75 6.13 24.50 SD 4.10 2.90 8.59 Min. Max. 5 2 16 19 12 43 Discussion The outcome of the reported experiment is consistent with results from the paper by Stull and Mayer [2], even though it was conducted in a different experimental environment. Neither the computerized environment nor the collaboration in groups improved the performance in the learningby-doing group in comparison to the learning-by-viewing scenario (Table 3). It is worth emphasizing that the students Table 5Success ratio in reconstructing a graphic organizer. Group A (n = 18) n Correct Incorrect 15 3 % 83 17 n 5 8 Group B (n = 13) % 38 62 Kononowicz and Kenig: Computer-based flow diagrams in medical education43 Table 6Results of students' satisfaction survey. Group A Likert scale (1­5) S1 S2 S3 Group B Mean/SD n=20 4.95/0.22 4.80/0.41 4.65/0.41 n=16 4.50/0.63 4.19/1.05 4.06/1.05 0.01 0.04 0.02 p-Value S1) I have learned useful things in the meeting today; S2) I liked the learning method applied in my experimental group; S3) I have the feeling I have learned a lot today. Looking at the results from a different perspective, it is worth noting that computer-based author-provided graphic organizers turned out to be highly popular among students. The subjective impression of the students regarding their learning gain after this scenario reached the level of 4.95, in a 5-point Likert scale which is astonishingly good. It needs to be stressed that learning by comparing the content of an interactive flow diagram-like web-based graphic organizer with the content of a scientific paper is not a common type of learning-by-viewing scenario at medical faculties, and it is worth being verified more closely in comparison to other computer-aided instructional methods. in the learner-generated graphic organizer group required significantly more time to study and performed worse than the other group. Most surprising (considering, e.g., the results of Farrand etal. [6], and not checked in the paper by Stull and Mayer) was the fact that the more active learning scenario did not lead to more durable knowledge than in the learning-by-viewing group measured 1 month after the learning activity. Also not checked in the former paper was students' satisfaction after the learning activity, showing in this study a clear preference for the learning-by-viewing scenario. This result is in line with the observations of Farrand etal., suggesting that lack of motivation to construct their own graphic organizers was a major hindrance in their usage. A potential explanation for worse performance of the learning-by-doing group given by Stull and Mayer was the extraneous cognitive load connected with mastering a new learning method which consumed too much of the students' attention. This study seems to confirm this view. Even though students were deeply engaged in collaborative work in groups for more than 100 min, on average the results of their work were disappointing. It should be noted that the diagram was created by a subject matter expert, whereas the student-generated diagrams were created by novices. The understanding of an expert is far superior to that of the students and this was clearly visible in the obtained results. However, it should be noted that especially in the case of recently published data, or for postgraduate, life-long learners, it is rather likely that no subject expert pre-prepared organizers for the text to be learned will be available. For such situations the learning-by-doing scenario can still be very attractive. Based on the analyses of diagrams created by the students, we came to the conclusion that the current scenario gives the students too little guidance on the efficient use of flow diagrams in learning. Lack of sufficient scaffolding in learning activities is a recognized problem of instructional design [17, 18] and should be dealt with by providing better guidance and support within learning activities. Limitations of the study A limitation of the study was the small sample size. Given the often expected effect size of 0.5 and study power of 0.8, the sample size should be twice as large as was available (n = 64). The number of test items in the examination was rather small and heterogeneous, thus making the test less internally consistent. No standardized, well-verified, high-quality tool for knowledge assessment was available for this topic, which was clearly a drawback of the study. Another issue worth mentioning is that a multiple choice question might not be optimal to test the learnergenerated diagram group which is governed by the active learning principle. The learners were not trained to memorize the material but actively create the diagram based on their understanding of the content. Perhaps, a written examination to test their understanding would be more appropriate. However, the active reconstructing a graphic organizer test did not give contradictory results, which seems to confirm conclusions from the multiple choice question test. Future work Even though the results of this study discourage the application of computer-based graphic organizers generated by students in learning new content in comparison to learning-by-viewing author-provided graphic organizers, this does not rule out the application of this method in general. Future efforts should be channeled into finding methods for providing scaffolding for students in the learning-bydoing scenario to decrease the extraneous cognitive load. The strong correlation between the scoring of the graphic organizers and the results in the knowledge tests suggests that this method could potentially be applied as a sophisticated assessment tool. More studies on the checking 44Kononowicz and Kenig: Computer-based flow diagrams in medical education of the reliability of this assessment method are needed. It would be interesting to conduct a comparative study of flow diagrams with other graphic organizers in the learning-by-doing scenario. Finally, the learning-by-doing scenario presented here might be an interesting tool for teaching students the basics of medical informatics by illustrating the process of authoring rule sets for decision support systems by asking the students to construct their own decision trees. This scenario, which has already been tentatively verified in another publication [14] still needs to be tested in greater detail. The problem of too much cognitive load could be remedied by asking the students to select topics in which they feel confident enough to construct flow diagrams. The learning-by-viewing scenario can also be further investigated by testing it against other, less cognitively demanding instructional designs such as a set of worked examples [19] (interactive patient case stories, virtual patients), which illustrate the same topic. Another scenario might be to ask students to find errors or steps without evidence in an author-generated graphic organizer taking a text-based review paper as the gold standard. Conclusions This study reports concerns about the usage of learner-generated graphic organizers. The application of a computerbased environment and collaboration in groups did not improve the knowledge acquisition outcomes in comparison with author-provided graphic organizers. Better guided instructional designs for the learning-by-doing scenario are definitely needed. The computer-based author-provided graphic organizers turned out to be highly popular among students and led to durable knowledge gains. Because this type of learning-by-viewing is rarely in use at medical universities its wider usage should be considered. Received October 18, 2012; revised January 22, 2013; accepted January 24, 2013; previously published online February 23, 2013 http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png Bio-Algorithms and Med-Systems de Gruyter

Learner-generated versus author-provided computer-based flow diagrams in medical education

Download PDF
8 pages

Loading next page...
 
/lp/de-gruyter/learner-generated-versus-author-provided-computer-based-flow-diagrams-N04QIkmrgj

References

References for this paper are not available at this time. We will be adding them shortly, thank you for your patience.

Publisher
de Gruyter
Copyright
Copyright © 2013 by the
ISSN
1895-9091
eISSN
1896-530X
DOI
10.1515/bams-2013-0005
Publisher site
See Article on Publisher Site

Abstract

principles), whereas flow diagrams help in mastering procedural knowledge (knowledge how to solve concrete problems). The relation between these two knowledge types is complex and still the subject of *Corresponding author: Andrzej A. Kononowicz, Department of Bioinformatics and Telemedicine, Jagiellonian University Medical College, Lazarza 16, 31-034, Kraków, Poland, E-mail: andrzej.kononowicz@uj.edu.pl Jakub Kenig: Third Department of General Surgery, Jagiellonian University Medical College, Kraków, Poland Introduction Graphic organizers are `two-dimensional visual knowledge representations' including flow diagrams, knowledge and concept maps, timelines and tables [1]. Provision of graphic organizers which illustrate the content of learning material is a common method of supporting meaningful learning of complex text passages. Alternatively, learners are asked to construct their own graphic organizers based on what they have read to facilitate comprehension and 38Kononowicz and Kenig: Computer-based flow diagrams in medical education intensive research [8]. Davidowitz and Rollnick used flow diagrams as a method of preparation for laboratories in chemistry observing positive responses from students [9]. DeMeo reported on the construction of flow diagrams as an alternative to lectures for introducing students to the solving of aqueous acid base equilibria problems [10]. An important research question centers on whether learners should create their own graphic organizers or use those provided by their instructors. It does not seem to be easy to give an a priori judgment on which option is to be preferred because there are theories supporting both approaches. When constructing graphic organizers, students organize the text into coherent structures and connect them by relations [2]. It engages students in a productive learning activity through the construction of their own mental representations of the content to be learned, and this may be regarded as a sign of `deeper learning'. Craik and Lockhart's [11] theory of levels of processing posits that deeper learning activities result in more durable knowledge. By contrast, in the process of learning-by-viewing students are prompted to become involved in learning by comparing how a linear text was converted into a spatial structure prepared by the learning activity instructor [2]. Cognitive load theory prefers learningby-viewing over learning-by-doing scenario because the construction of graphic organizers by students induces extraneous processing which limits the available mental resources for generative processing and thus makes the learning process less efficient [12]. A study by Stull and Mayer [2] compared the learning outcomes of learner-generated with author-provided graphical organizers illustrating a reading material from a popular biology textbook. The study involved a group of college students from a psychology participant pool. It was concluded that increased activity on the part of the learner caused by producing graphic organizes does not lead to deeper understanding. Students who used author-generated graphic organizers learned faster and with better results than the students preparing their own graphic organizers. computer-based. This provided the students with additional opportunities that were not available in `pen and paper' drawings such as unlimited space, easy corrections and interactive elements hiding the complexity of descriptions. Students did not work individually but in pairs, which was expected to decrease the extraneous processing related to constructing graphic organizers. Students could discuss their ideas on how to draw the flow diagrams and work in parallel on the text. In their study, Stull and Mayer applied different types of graphic organizers (flow diagrams, hierarchies, small concept maps), whereas in this study only computer-based flow diagrams were used. This paper also enhanced Stull and Mayer's study with new analyses. A knowledge test was carried out not only immediately before and after the learning activity but also 1 month later to determine the durability of acquired knowledge. Student satisfaction was also analyzed. The quality of graphic organizers was assessed to check whether it influenced the learning outcome measured by the knowledge test. This study also aimed to test the application of flow diagrams in medical teaching. The papers referenced in the introductory part reported on the application of flow diagrams in chemistry teaching [9, 10]. The authors of this paper are not aware of any similar experiments in medical education. In addition, the reports from the application of flow diagrams in chemistry concentrated merely on student satisfaction and did not involve any forms of knowledge testing. Materials and methods Study protocol The study group consisted of 36, third to fifth year, voluntary students recruited from an extracurricular group of students at Jagiellonian University Medical College interested in general surgery. Students organized themselves into pairs and were randomly assigned to groups A or B. Student identities were encoded to make the result analysis anonymous. Pairs of students were randomly assigned to two groups: group A ­ author-provided graphic organizer (learning-by-viewing scenario); group B ­ learnergenerated graphic organizer (learning-by-doing scenario). All students took the same pre-test. After the pre-test both groups were put in two separate rooms. Students received the same review paper on Boerhaave's syndrome [13] and were instructed to read it through carefully. Purpose of the study The aim of this paper was to shed more light on the proper application of learning-by-doing and learning-by-viewing scenarios involving graphic organizers in a computerbased environment. In contrast to the paper by Stull and Mayer [2], the authoring of graphic organizers in this study was Kononowicz and Kenig: Computer-based flow diagrams in medical education39 Boerhaave's syndrome is an often misdiagnosed spontaneous rupture of the esophagus which leads to death if untreated. Rapid decisions regarding proper management are crucial for good chance of recovery. The paper was slightly modified with the exclusion of a graphic organizer and references to it, and a few sentences, to ensure that the removed graphic organizer could be constructed solely based on the text of the paper, were added. The length of the paper was approximately 1750 words. Group A also received a link to an electronic version of the author-provided graphic organizer on Boerhaave's syndrome (Figure 1) and a sheet with a printout of the graphic organizer with digits assigned to some of the nodes in the diagram. The printout did not contain text descriptions which were displayed when the diagram elements in the electronic version were clicked. The task of the students in group A was, while working in pairs, to go through the diagram elements with assigned digits and mark (with these digits) fragments of the article corresponding to the content of the preselected flow diagram elements. Students in group B were asked to create, in pairs, a flow diagram presenting the content of the paper on Boerhaave's syndrome using the `Bit Pathways' program [14]. Students had access to a short tutorial on how to use the `Bit Pathways' application, a short description of all flow diagram elements available in `Bit Pathways' for this study (i.e., start/end elements, task element, decision element and connectors) and a printout of a sample graphic organizer from a different topic (in-hospital resuscitation) also created with that application. All students had already experience with the `Bit Pathways' program after a class on clinical pathways in the `Basics of Computer Science' course in their third year of study. No time limit was imposed on either group. Students worked as long as they wished and were allowed to communicate within their pairs, but not with other pairs participating in the study. Students sat individually for the post-test and answered the satisfaction survey immediately after completing their tasks. They were not informed about the correct answers in the tests or given any feedback regarding their performance in the tasks in order to not influence study results. One month after the study students from both experimental groups were asked to write a follow-up knowledge Figure 1The author-provided schema on the management of Boerhaave's syndrome. 40Kononowicz and Kenig: Computer-based flow diagrams in medical education test without prior notice. In addition, students were given the task of reconstructing the author-provided graphic organizers showing the optimal procedure for the treatment of Boerhaave's syndrome having at their disposal a list of permitted action and decision elements. The problem of finding a universal and objective evaluation of graphic organizers is very complex and is beyond the focus of this paper. Research reports on developing rubrics for learner-generated graphic organizers may be found, for example, in [15, 16]. For the purpose of this paper, a simple, preliminary scale for grading the graphic organizers created in group B was proposed. The scale consisted of three measures evaluating: (i) content of the pathway (max. 10 points), (ii) its legibility (max. 5 points) and (iii) notation (max. 3 points). For evaluation of the content, the presence (1 point each) and correct location (1 point each) of three crucial action elements (`endoscopy treatment', `surgery' and `conservative treatment') was checked. In addition, the presence (1 point each) and placement (1 point each) of two decision elements was examined (`rupture < 48h ago?' and `signs of sepsis?'). The presence (or absence) of other elements was not graded. Legibility was graded by a 5-point Likert scale question checking the reviewer's attitude towards the sentence `the pathway presented the content of the paper in an easily understandable way' (5 ­ strongly agree, 1 ­ strongly disagree). The `notation' measure checked the standard-conformant use of flow diagram elements: 3 ­ no errors, 2 ­ minor errors, 1 ­ lack of understanding of the notation used in constructing flow diagrams. Grading was carried out by the authors of this paper (J.K. and A.K.). A preliminary observation of the results of learnergenerated graphic organizers suggested that the students in group B had problems selecting the right actions and decision elements for the graphic organizers. We investigated whether students from group B could construct a correct pathway if they had preselected flow diagram elements at their disposal. Thus, an additional assignment was given to check the students' success ratio in arranging the recommended actions in the right order. This exercise was conducted immediately after the follow-up test, that is, 1 month past the learning activity. All action and decision elements from the author-provided graphic organizer were presented in random order on the margin of a blank sheet of paper. The students' task was to create from them (using the `paper and pen' method due to the lack of sufficient computers in the examination room) a correct flow diagram presenting the right treatment procedure in the case of Boerhaave's syndrome. The graphic organizers created by students were divided into two classes: correct and incorrect. The correct class included graphic organizers showing the tasks and decision elements in the right clinical order ­ minor deviations not influencing the clinical outcome of the patients were permitted. The incorrect flow diagrams showed inadequate order of the tasks and decision elements that were not recommended by the paper and were potentially harmful for the patients. Students from group A were asked to perform the same task in order to form a reference group. A survey into students' satisfaction after the learning activity comprises four questions: three of them in a 5-point Likert scale (5 ­ definitely agree, ..., 1 ­ definitely disagree) asking about students' attitudes towards the following statements: (S1) I have learned useful things in the meeting today; (S2) I liked the learning method applied in my experimental group; (S3) I have the feeling I have learned a lot today. The fourth survey item was a free text question asking about general opinions and impressions after the learning activity. Materials Three knowledge tests were carried out: a pre-test (immediately before the learning activity), post-test (immediately after the learning activity) and a follow-up test (1 month after the learning activity). The tests were developed by the second author of the paper (J.K.). All tests contained the same 12 multiple choice question items regarding proper treatment of Boerhaave's syndrome. All questions were equally rated: 1 point for a correct answer, 0 points for a wrong or no answer, resulting in a maximum of 12 points. To prevent the students from memorizing the questions by rote, the order of the questions/answers in subsequent tests was altered and new `distractor' questions regarding Boerhaave's syndrome were added to the test. Outcomes of the `distractor' questions were not analyzed. The author-provided and all learner-generated graphic organizers were constructed using the `Bit Pathways' application, developed by Kononowicz and Holler [14]. The system was developed to give subject matter experts, teachers and students an easy way of presenting procedural knowledge in a web-based environment. It enables the authoring of flow diagram-like graphic organizers by dragging and dropping a set of elements (action, decision, start/ end elements) from the toolbar onto the canvas and connecting them by arrows showing the chronological order of tasks. All elements may have a structured text description (without space limitations) edited via a property panel. The text-intensive descriptions of the element are displayed in pop-up windows upon activation of a diagram element by Kononowicz and Kenig: Computer-based flow diagrams in medical education41 the mouse cursor. The graphic organizers are stored on a central server and can be viewed by a conventional web browser such as Internet Explorer or Firefox. Figure 1 presents the author-provided schema on Boerhaave's syndrome created in the `Bit Pathways' program. The content of the graphic organizer is based on a paper by de Schipper etal. [13] and was slightly changed (by adding a diagnosis element) to address the learning objectives of the class. The diagram consists of four action elements, three decision elements, eleven connectors, one start and three end elements. The additional descriptions displayed by clicking the elements of the diagram contained, in total, 420 words copied directly from the paper by de Schipper etal. [13]. All graphic organizers were available or constructed in English and not in the students' native Polish language. However, all students had previously declared knowledge of the English language and the inconvenience was the same for both groups and therefore it should have affected the final results of all students in a similar way. Table 1Demographic structure of the study groups. All n Female Female Male Male Year (mean) Year (3) Year (4) Year (5) 36 21 58.3% 15 41.7% 3.9 8 22 6 Group A 20 12 60.0% 8 40.0% 4.0 4 12 4 Group B 16 9 56.3% 7 43.8% 3.9 4 10 2 students of different years in similar proportions in both study groups (by having age as one of the parameters in the random group assignment procedure), we do not consider the year of study as a major factor because Boerhaave's syndrome is not formally part of the medical curriculum at Jagiellonian University Medical College. Time Data analysis Statistical data analysis was done in Statsoft Statistica 9.0 (Statsoft Inc., Tulsa, OK, USA). Validity of the assumption of normal distribution was tested with Shapiro-Wilk's test. The equality of variances was confirmed with Levene's test. Knowledge gain for repeated measures was calculated using Friedman's test. Differences between two sample means (if assumptions were met) were tested using Student's t-test. In the case of non-normally distributed samples, non-parametric Mann-Whitney's U-test was applied. Dichotomous variables were compared using Fisher's exact test. Effect size was calculated using Cohen's d with pooled standard deviations (SDs). Correlations between data were checked with Spearman's rank correlation coefficient. Significance level was set at 0.05. The duration of the learning activity was measured from the start of the pre-test until the start of the post-test. The time taken by two students in group A was not available due to a technical problem. A summary of all time results is presented in Table 2 (time scale: hh:mm:ss, hh ­ hours, mm ­ minutes, ss ­ seconds). The exercise took statistically significant less time for group A than group B (p < 0.001, Student's t-test). Knowledge acquisition Four students of the initial group (two from group A and two from group B) were missing at the follow-up test conducted 1 month after the pre- and post-tests and thus their results have been omitted from knowledge acquisition analysis. Table 3 presents a summary of all knowledge test results. There was no difference between group A and group B in the pre-test score (p = 0.99, Student's t-test). The normality assumption was not met for the follow-up test in group Results The sample Table 2Time spent on the tasks in groups A and B. Table 1 presents the age and gender profiles of the study groups. The majority of students who participated in the study were from fourth year. There were slightly more women than men in the study. The division of students into study groups retained the age and gender proportions. Even though particular attention was paid to having Group A 18 Group B 16 01:48:10 00:15:41 Time hh:mm:ss Mean SD 01:17:16 00:14:49 42Kononowicz and Kenig: Computer-based flow diagrams in medical education Table 3Results of knowledge tests. n Group A 18 PrePostFollow-up Group B 14 4.21/1.19 7.64/1.78 5.29/1.77 p-Value organizer grade and score on the follow-up test could was not statistically significant (n=7, Spearman's R=0.54, p=0.21), even though the trend suggests that for a larger group of study subjects a relationship could be demonstrated. Mean (pts)/SD Mean (pts)/SD Mean (pts)/SD 4.22/1.66 8.17/1.72 6.94/2.15 0.99a 0.41a 0.03b Graphic organizer reconstruction The results of the graphic organizer reconstruction assignment are presented in Table 5. The task was performed significantly better by group A (Fisher's exact test, p = 0.01), which is not surprising considering the fact that group A had seen the correct graphic organizer 1 month previously. More surprising was the large percentage of students from group B (62%) making significant mistakes in their flow diagrams even when provided with the correct selection of crucial steps in the algorithm. p-Value according to Student's t-test; bp-value according to MannWhitney's U-test. B (p = 0.02, Shapiro-Wilk's test), and for that reason no ANOVA test for repeated measurements could be applied. The learning activity produced significant knowledge gains in both groups (pA < 0.001; pB < 0.001, Friedman's test). The mean value of the post-test for the given sample was higher in group A than in group B, but the difference did not reach a statistically significant level (p = 0.41, Student's t-test). In the follow-up test, the difference between the groups was significant (p = 0.03, Mann-Whitney's U-test, = 0.05, Cohen's d = 0.84). Students' satisfaction The results of the satisfaction survey are presented in Table 6. Students of both groups liked the learning activity. The satisfaction level in group A was extraordinarily high (e.g., S1 = 4.95 and S2 = 4.80 in a max. 5-point scale). The results in group B were statistically significantly worse (p-Value in Mann-Whitney's U-test), but still fairly good (mean value of all questions above 4 in a 5-point Likert scale). Some of the students complained in the free text comments that the article was in English rather than in their native language, as this hindered their comprehension. Evaluation of learner-generated flow diagrams A total of eight graphic organizers were created by students from group B who had teamed up into groups of two. Table 4 presents a quantitative summary of the size of author-provided versus learner-generated graphic organizers. Graphic organizers presented by students were on average twice as large as those provided by the authors for group A. Many of the additionally added elements contained data irrelevant for the treatment process or even statements not present in the paper. There was a strong positive correlation between the graphic organizer score of the group and the sum of scores obtained by group members on the post-test (n=8, Spearman's R=0.75, p=0.03). This suggests that the quality of the graphic organizer might be a good measure of students' knowledge acquisition. The correlation of the graphic Table 4Quantitative comparison of the author-provided and learner-generated graphic organizers. Authorprovided Number of action elements Number of decision elements Number of connectors 4 3 11 Learner-generated Mean 10.75 6.13 24.50 SD 4.10 2.90 8.59 Min. Max. 5 2 16 19 12 43 Discussion The outcome of the reported experiment is consistent with results from the paper by Stull and Mayer [2], even though it was conducted in a different experimental environment. Neither the computerized environment nor the collaboration in groups improved the performance in the learningby-doing group in comparison to the learning-by-viewing scenario (Table 3). It is worth emphasizing that the students Table 5Success ratio in reconstructing a graphic organizer. Group A (n = 18) n Correct Incorrect 15 3 % 83 17 n 5 8 Group B (n = 13) % 38 62 Kononowicz and Kenig: Computer-based flow diagrams in medical education43 Table 6Results of students' satisfaction survey. Group A Likert scale (1­5) S1 S2 S3 Group B Mean/SD n=20 4.95/0.22 4.80/0.41 4.65/0.41 n=16 4.50/0.63 4.19/1.05 4.06/1.05 0.01 0.04 0.02 p-Value S1) I have learned useful things in the meeting today; S2) I liked the learning method applied in my experimental group; S3) I have the feeling I have learned a lot today. Looking at the results from a different perspective, it is worth noting that computer-based author-provided graphic organizers turned out to be highly popular among students. The subjective impression of the students regarding their learning gain after this scenario reached the level of 4.95, in a 5-point Likert scale which is astonishingly good. It needs to be stressed that learning by comparing the content of an interactive flow diagram-like web-based graphic organizer with the content of a scientific paper is not a common type of learning-by-viewing scenario at medical faculties, and it is worth being verified more closely in comparison to other computer-aided instructional methods. in the learner-generated graphic organizer group required significantly more time to study and performed worse than the other group. Most surprising (considering, e.g., the results of Farrand etal. [6], and not checked in the paper by Stull and Mayer) was the fact that the more active learning scenario did not lead to more durable knowledge than in the learning-by-viewing group measured 1 month after the learning activity. Also not checked in the former paper was students' satisfaction after the learning activity, showing in this study a clear preference for the learning-by-viewing scenario. This result is in line with the observations of Farrand etal., suggesting that lack of motivation to construct their own graphic organizers was a major hindrance in their usage. A potential explanation for worse performance of the learning-by-doing group given by Stull and Mayer was the extraneous cognitive load connected with mastering a new learning method which consumed too much of the students' attention. This study seems to confirm this view. Even though students were deeply engaged in collaborative work in groups for more than 100 min, on average the results of their work were disappointing. It should be noted that the diagram was created by a subject matter expert, whereas the student-generated diagrams were created by novices. The understanding of an expert is far superior to that of the students and this was clearly visible in the obtained results. However, it should be noted that especially in the case of recently published data, or for postgraduate, life-long learners, it is rather likely that no subject expert pre-prepared organizers for the text to be learned will be available. For such situations the learning-by-doing scenario can still be very attractive. Based on the analyses of diagrams created by the students, we came to the conclusion that the current scenario gives the students too little guidance on the efficient use of flow diagrams in learning. Lack of sufficient scaffolding in learning activities is a recognized problem of instructional design [17, 18] and should be dealt with by providing better guidance and support within learning activities. Limitations of the study A limitation of the study was the small sample size. Given the often expected effect size of 0.5 and study power of 0.8, the sample size should be twice as large as was available (n = 64). The number of test items in the examination was rather small and heterogeneous, thus making the test less internally consistent. No standardized, well-verified, high-quality tool for knowledge assessment was available for this topic, which was clearly a drawback of the study. Another issue worth mentioning is that a multiple choice question might not be optimal to test the learnergenerated diagram group which is governed by the active learning principle. The learners were not trained to memorize the material but actively create the diagram based on their understanding of the content. Perhaps, a written examination to test their understanding would be more appropriate. However, the active reconstructing a graphic organizer test did not give contradictory results, which seems to confirm conclusions from the multiple choice question test. Future work Even though the results of this study discourage the application of computer-based graphic organizers generated by students in learning new content in comparison to learning-by-viewing author-provided graphic organizers, this does not rule out the application of this method in general. Future efforts should be channeled into finding methods for providing scaffolding for students in the learning-bydoing scenario to decrease the extraneous cognitive load. The strong correlation between the scoring of the graphic organizers and the results in the knowledge tests suggests that this method could potentially be applied as a sophisticated assessment tool. More studies on the checking 44Kononowicz and Kenig: Computer-based flow diagrams in medical education of the reliability of this assessment method are needed. It would be interesting to conduct a comparative study of flow diagrams with other graphic organizers in the learning-by-doing scenario. Finally, the learning-by-doing scenario presented here might be an interesting tool for teaching students the basics of medical informatics by illustrating the process of authoring rule sets for decision support systems by asking the students to construct their own decision trees. This scenario, which has already been tentatively verified in another publication [14] still needs to be tested in greater detail. The problem of too much cognitive load could be remedied by asking the students to select topics in which they feel confident enough to construct flow diagrams. The learning-by-viewing scenario can also be further investigated by testing it against other, less cognitively demanding instructional designs such as a set of worked examples [19] (interactive patient case stories, virtual patients), which illustrate the same topic. Another scenario might be to ask students to find errors or steps without evidence in an author-generated graphic organizer taking a text-based review paper as the gold standard. Conclusions This study reports concerns about the usage of learner-generated graphic organizers. The application of a computerbased environment and collaboration in groups did not improve the knowledge acquisition outcomes in comparison with author-provided graphic organizers. Better guided instructional designs for the learning-by-doing scenario are definitely needed. The computer-based author-provided graphic organizers turned out to be highly popular among students and led to durable knowledge gains. Because this type of learning-by-viewing is rarely in use at medical universities its wider usage should be considered. Received October 18, 2012; revised January 22, 2013; accepted January 24, 2013; previously published online February 23, 2013

Journal

Bio-Algorithms and Med-Systems – de Gruyter

Published: Mar 1, 2013

There are no references for this article.