Access the full text.
Sign up today, get DeepDyve free for 14 days.
Evaluating knowledge and learning in psychotherapy is a growing field of research. Studies that develop and evaluate valid tests are lacking, however. Here, in the context of internet-based cognitive behavioral therapy (ICBT) for adolescents, a new test was developed using subject matter experts, consensus among researchers, self-reports by youths, and a literature review. An explorative factor analysis was performed on 93 adolescents between 15 and 19 years old, resulting in a three- factor solution with 20 items, accounting for 41% of the total variance. The factors were Act in aversive states, Using positive reinforcement, and Shifting attention. The internal consistency for the whole instrument was good (α = .84). Hopefully, the procedure presented here can contribute to the field by illustrating one way of evaluating knowledge in ICBT. The clinical use of the knowledge test needs further investigation. Keywords psychology, social sciences, cognitive behavioral therapy, psychotherapy, clinical psychology, measurement and scaling methods, research methods, educational psychology, applied psychology, approaches behavior (Coles et al., 2016). Various measures of knowl- Introduction edge have been used and studies are published on how to Evaluating Knowledge in General Health Care quantify knowledge acquisition in robust ways, to make more valid conclusions about interventions’ effects on The role of knowledge and learning in health care interven- knowledge gain (Wei et al., 2013). tions is a growing field of research. Researchers in the areas of psychoeducation and patient education stress the impor- tance of assessing knowledge when evaluating interventions Evaluating Knowledge in Cognitive Behavioral in relation to mental illness (Fox, 2009; Lukens & Therapy McFarlande, 2004; Wei et al., 2013). Overall, these areas aim An important area of research when evaluating knowledge to teach individuals about their conditions and their treat- and learning in psychotherapy is to assess what clients ments to improve their capacity to recognize, prevent, and actually learn and remember during and after their treatment. manage their mental illness. They are often based on the The literature on evaluation of knowledge acquisition and notion that psychoeducation and explicit knowledge can enhance processes such as cognitive mastery and experi- ences of empowerment, and thereby help patients engage in Linköping University, Sweden healthier behaviors in the presence of their problems (Brewin, Karolinska Institute, Stockholm, Sweden 1996; Ryhänen et al., 2010; Sajatovic et al., 2007). For University College London, UK instance, the importance of knowledge is particularly evident in research on mental health literacy. This is a research area Corresponding Author: which aims to teach the general public about risks, symp- Matilda Berg, Department of Behavioural Sciences and Learning, Linköping toms, and treatments, to affect other desirable outcomes such University, SE-58183 Linköping, Sweden. as less stigmatized attitudes and increased help-seeking Email: email@example.com Creative Commons CC BY: This article is distributed under the terms of the Creative Commons Attribution 4.0 License (https://creativecommons.org/licenses/by/4.0/) which permits any use, reproduction and distribution of the work without further permission provided the original work is attributed as specified on the SAGE and Open Access pages (https://us.sagepub.com/en-us/nam/open-access-at-sage). 2 SAGE Open learning in psychotherapy is, however, surprisingly scarce. that their instrument was not psychometrically robust and One relevant form of psychotherapy with regard to knowl- called for further research to establish its validity and edge and learning is cognitive behavioral therapy (CBT; reliability. Andersson, 2016; Harvey et al., 2014). The overall goal of CBT is to teach patients how to recognize unwanted symp- Evaluating Knowledge in Internet-Based CBT toms and to let these symptoms serve as cues for the use of more adaptive management strategies (Albano & Kendall, One suitable research field for evaluating knowledge acqui- 2002; Arch & Craske, 2008; Brewin, 1996). CBT empha- sition and learning during therapy is when CBT is provided sizes educational components and most treatment manuals over the internet (ICBT [internet-based cognitive behavioral involve thorough rationales and instructions to help clients therapy]; Harvey et al., 2014), as ICBT is mainly based on learn new behavioral and cognitive strategies. One major educational texts containing rationales, psychoeducation, purpose of CBT is to involve clients in more adaptive learn- and explicit instructions for learning new skills (Andersson, ing experiences, targeting both implicit knowledge gained 2016). Thus, the contents and principles of CBT are the same through exercises such as behavioral activation, and expo- but in ICBT they are administered in a more structured and sure, as well as explicit knowledge through interventions mainly text-based format. ICBT makes it possible for clients such as psychoeducation, self-monitoring strategies, or iden- to receive the same information and instructions on a weekly tifying and challenging patient’s maladaptive cognitive basis which enables control over the type and amount of assumptions through logic or alternative, more adaptive ones information provided. (Brewin, 1996). Thus, there is a clinical value in assessing To our knowledge, three published controlled treatment the role of knowledge and learning in CBT. studies have evaluated knowledge gain in ICBT. Inspired by Researchers have begun to assess and report data on earlier research on knowledge acquisition in bibliotherapy patient learning, memory, and knowledge acquisition. For (Friedberg et al., 1998; Scogin et al., 1998), two studies have instance, by integrating learning theories and findings from evaluated knowledge gain in ICBT for adults (Andersson cognitive science and educational science, Harvey et al. et al., 2012; Strandskov et al., 2017) and one similar gain in (2014) highlighted the potential importance of remembering adolescents with depression (Berg et al., 2019). In these stud- treatment content to enhance treatment outcome in CBT. ies, knowledge was measured via multiple-choice tests con- Poor memory of treatment information has been associated taining items covering the specific diagnosis/problem and with poor treatment adherence and outcome (Gumport et al., CBT for the condition. Generally, the use of multiple-choice 2015; Harvey et al., 2016). These researchers have mainly tests has several advantages. It is important to develop and measured memory and learning through free-recall tasks, use objective scales as a complement to open-ended ques- open-ended questions, and vignettes. For instance, by using tions. Open-ended questions have a tendency of inducing a free-recall task, patients were asked to write down recalled bias (Keulers et al., 2007) and recall tasks tend to result in information from their treatment during a 10-min period less information recalled compared with recognition and (Dong et al., 2017; Harvey et al., 2016; Lee & Harvey, 2015). cued-recall tasks (Gumport et al., 2018). Andersson et al. Each response was compared with a predetermined list of (2012) constructed a test of 11 multiple-choice items related important treatment points in cognitive therapy (CT), and to social anxiety and its treatment, whereas Strandskov et al. answers were scored by independent coders. Results to date (2017) designed a multiple-choice test of 16 items to mea- have revealed mixed effects and the studies have often been sure knowledge about symptoms and CBT treatment in eat- underpowered, but overall, remembering treatment content ing disorders. Berg et al. (2019) developed a test of 17 items seems to be of relevance for treatment outcome in terms of measuring knowledge about depression and comorbid anxi- depressive symptoms and individual functioning. Moreover, ety and how to treat it with CBT. these researchers assessed learning during treatment by ask- Furthermore, these studies incorporated certainty ratings ing clients via telephone about their thoughts concerning the in the tests, asking the clients to rate level of certainty for last therapy session and whether they lately had applied any each response on a 3- or 4-point Likert-type scale (i.e., Are skills learned in therapy (Gumport et al., 2015, 2018). you guessing, uncertain, or certain about your answer?). Learning is also defined and measured as clients’ capability Incorporating level of certainty is one way to evaluate to generalize treatment content, by presenting participants whether clients are certain about the wrong answer or guess- with two fictive vignettes, asking them what they would ing the right answer (Andersson et al., 2012). The distinction think (cognitive generalization) and how they would respond between accuracy and certainty is commonly used in studies in a similar situation (behavioral generalization). Data are in cognitive psychology (Tiedens & Linton, 2001), and in then coded by independent raters in terms of number of accu- line with the need to differentiate between actual knowledge rately recalled therapy points. Taken together, the studies and beliefs when measuring knowledge about mental health show mixed findings, but there are some promising findings (O’Connor et al., 2014). Moreover, there is a “common showing that learning treatment outcome is related to depres- sense” aspect of CBT, in that clients may already know and sion levels after treatment. Gumport et al. (2018) highlighted understand many of the psychoeducational points and Berg et al. 3 principles, but feel unsure about actually trying to change evaluation when delivered through an internet format (ICBT) their behavior and if the knowledge applies to them. Certainty as researchers can have control over which information the is a potentially important aspect in motivating clients to participants have received through the texts in the treatment engage in the behavioral components of CBT (Friedberg program. Knowledge evaluation could easily be based on the et al., 1998). For example, trusting the rationale about expo- content of the modules and its intended learning outcome. sure might be of importance in actually challenging ones’ Little is known what clients actually remember from the con- fears in real-life situations. tent in the modules and in which degree they can explicitly In conclusion, we believe this topic warrants further retrieve CBT principles learned through internet. One impor- research. Overall, as in research in general health care, stud- tant issue, however, concerns the type and depth of knowl- ies lack thorough descriptions of their test development pro- edge measured in earlier ICBT trials (Andersson et al., 2012; cedures and psychometrically robust measures. For instance, Berg et al., 2019; Strandskov et al., 2017). In contrast to Andersson et al. (2012) reported an internal consistency of more general knowledge about symptoms and treatment, as Cronbach’s α = .56, Strandskov et al., (2017) α = .62, and measured in studies of patient education and mental health Berg et al. (2019) α =.64, all being less than satisfactory. No literacy, tests in ICBT could preferably measure application further analyses on psychometrics were described in these or generalization of knowledge from the treatment modules, studies. Thus, there is a need to develop tests with higher focusing on strategies and skills that aim to aid clients in reliability and construct validity to improve how explicit their everyday life. For instance, it can be feasible to assess knowledge acquisition and learning in CBT and ICBT are clients’ ability to apply principles or skills learned during captured and conceptualized. It is also important to validate treatment in hypothetical scenarios (Dong et al., 2017). subscales and make distinctions between different knowl- Finally, evaluating knowledge and learning could be of edge constructs (O’Connor et al., 2014). extra relevance for certain target groups. Youths with depres- sion and anxiety symptoms are such a group, given the high relapse rate of depression and its comorbidity with anxiety Creating a Test of Knowledge Acquisition in ICBT (Costello et al., 2003; Ebert et al., 2015; Thapar et al., 2012), A valid test needs a valid theory. Overall, a clear definition of and the potential protective effect knowledge could have on learning and knowledge in CBT is an important issue in all of these conditions. For instance, in a study on adults, these above-mentioned studies. As mentioned, knowing the Kronmüller et al. (2007) found that knowledge about treat- central components in CBT could be regarded as a contin- ment could predict level of depressive symptoms 2 years uum, ranging from implicit skills (procedural knowledge) to later. Research in mental health literacy highlights the impor- explicit facts, concepts, and principles (declarative knowl- tance of early knowledge intervention, commonly focusing edge; Brewin, 1996). On one hand, CBT focuses on behavior on the adolescent population (Burns & Rapee, 2006; Coles in critical situations, learning new ways to respond to exter- et al., 2016). If clients learn and gain knowledge early on, nal and internal stimuli, such as staying in a crucial situation risk of relapse might decrease. in the presence of intrusive thoughts or fear (Arch & Craske, On the basis of these previous findings and to address 2008). Terides et al. (2016) aimed to quantitatively capture some of the shortcomings in the evaluation procedures of procedural knowledge in CBT and constructed a test evaluat- knowledge acquisition and learning, a new instrument for ing practical skills usage, called the frequency of actions and assessing what adolescents learn during therapy was devel- thoughts questionnaire (FATS). It contained 12 items cover- oped, focusing on explicit knowledge of general and applied ing four factors labeled cognitive restructuring, rewarding CBT principles. We will present the test construction proce- behaviors, social interactions, and activity scheduling. dure that was based on both theoretical considerations and Preliminary evidence confirms its validity and that higher empirical findings. Hopefully, the procedure presented here self-rated skill usage is associated with greater symptom can contribute to the field by illustrating one way to evaluate reduction of anxiety and depression. knowledge gain in ICBT. In addition, however, CBT is also about learning explicit concepts and principles, such as understanding the rationale Materials and Methods for exposure or how to manage negative thoughts through cognitive restructuring. Harvey et al. (2014) used the term Item Design “treatment points” to capture these concepts and principles. A primary goal when developing an instrument or a test is to More specifically, a treatment point was defined as an insight, create a valid measure in which the items reflect the underly- skill, or strategy from therapy deemed important for patients ing latent construct(s). Thus, the theoretical concepts used in to remember or implement in their everyday lives. In CBT, a knowledge test should preferably be conceptualized before there is a handful of treatment points believed to be helpful testing it empirically (Clark & Watson, 1995). If an instru- and meaningful for clients to understand explicitly (to cope ment lacks a conceptualized theory of what it is supposed to with life and applying it in new situations later on). These target, the instrument might also lack construct validity. explicit CBT concepts and principles could be well suited for 4 SAGE Open Here, the conceptualization of the construct(s) was briefly Table 1. Brief Characteristics of Participants. formulated as “explicit knowledge about general and applied Participants core CBT-principles” in the context of adolescent anxiety Variable n = 93 and depression. Age, M (SD) 15.59 (1.2) Furthermore, a valid theory does not only articulate what Gender, n (%) a construct is, but also what it is not. Therefore, when creat- Girls 47 (50.5) ing the initial item pool, an overinclusive number of items Boys 46 (49.5) were generated. An overinclusive initial item pool enables Previous knowledge about CBT psychometric analyses to detect and separate weak or unre- Yes, n (%) 12 (12.9) lated items from items more strongly related to the underly- Slightly, n (%) 21 (22.6) ing construct(s) (Clark & Watson, 1995). No, n (%) 60 (64.5) Items were generated based on three main sources. One source was the treatment content and findings from previous Note. CBT = cognitive behavioral therapy. Range 15 to 19. One participant was 13 and one 14 years old. studies on youths, to see what can be addressed in the mod- Question: Do you know anything about CBT, since before? ules and thus preferably be a part of a knowledge test (Silfvernagel et al., 2015, 2017; Topooco et al., 2018). Moreover, items were created based on a thorough literature clinical experts, assuring that the revised items were in line review covering research about knowledge acquisition, with CBT theory and that the test covered relevant treatment memory, and learning in general health care, CBT, and ICBT. points. Third, items were generated by consulting three clinical psy- In addition, certainty ratings were added in the question- chologists with approximately 10 years of experience in naire, that is, respondents were asked about how certain they treating youths with anxiety and depression. The experts felt with each response (“Guessing,” “Pretty certain,” or stated and summarized the core CBT principles for these “Totally certain”). As mentioned, incorporating certainty is conditions. one way to evaluate whether respondents actually have Furthermore, the phrasing of the items was carefully gained the correct knowledge rather than guessing the right attended to. Some items were phrased as general questions answer. Furthermore, it is important given the potential but most of them as mini-vignettes. The aim was to tap appli- “common sense” aspect of CBT and the clinical relevance cation and generalization of CBT principles, rather than only certainty could have on clients’ motivation to actually apply assessing recognition and recall of basic facts only. Preferably, what they learn. a knowledge test in ICBT should tap a form of explicit knowl- edge closer to the concept of procedural knowledge ICBT, as Participants in explicit application of CBT principles on relevant everyday life situations. The items were constructed in a multiple- A total of 93 participants were included in the study. They choice format (three response options), as it is expected to were mainly recruited from three high schools in Sweden, give more reliable and stable results (Clark & Watson, 1995; where teachers asked their students to fill out the test during Haladyna, 1994). Initially, 46 items were generated, targeting a class. See Table 1 for a brief description of their theoretical facts, concepts, and treatment points about expo- demographics. sure, behavioral activation, management of intrusive negative thoughts, and applied behavior analysis. Data Collection Next, to evaluate whether the items were appropriate and readable for the target population, the instrument was ini- The instrument was distributed via Limesurvey (https:// tially tested on nine adolescents aged about 16 years old. www.limesurvey.org), an online interface for administering Eight of them were girls and one was a boy. They were surveys. Respondents were mainly students from two high recruited via teachers from three different classes from two schools in Sweden who were between 15 and 19 years old. different high schools. The pilot test was conducted using A total of 192 individuals initiated the test and a total of 93 cognitive interviews, a method where the individuals are individuals submitted their answers. Among the 93 indi- asked to read and reflect about items out loud (Krosnick, viduals who submitted their answers, a total of 86 individu- 1999). By observing and listening to the respondents’ reac- als answered all of the questions while seven of the tions during the interview, the observer can revise or remove respondents left some of the questions unanswered. No dif- ambiguous or irrelevant items and hence increase the con- ferences between those who completed all the questions struct validity of the instrument. Several revisions and clari- and those who did not, in terms of age, gender, or previous fications had to be made. The items were phrased more knowledge about CBT were found (t(91) = −.045, p = concrete, straightforward, and at a more adequate level of .964, χ (1, 2) = 1.74–3.75, p = .429–.111). The low num- difficulty. A total of 13 items were removed, with 33 remain- ber of completers could be due to lack of any compensation ing. The revised version went through a final review with the for completing the test. Berg et al. 5 The determinant analysis indicated a low level of inter- Statistical Analyses item correlations, suggesting risk for multicollinearity and All statistical analyses were performed on IBM SPSS statis- thus implying that it would be inappropriate to perform an tics, Version 24. To assess internal consistency, we used alpha. EFA. However, the diagnostic assessments also revealed a An exploratory factor analysis (EFA) was performed. EFA KMO of 0.77, that is, over diagnostic threshold, and Bartlett’s is a suitable method when the purpose is to investigate theo- 2 test of sphericity was significant, χ (528) = 1,278.68, p < retically interesting latent constructs and establish the psy- .01. Taken together, the diagnostic tests supported the suit- chometric properties of a new instrument (Fabrigar et al., ability of proceeding with an EFA. 1999). EFA enables the detection of other possible latent The rotated EFA indicated six factors with an eigenvalue dimensions in the material and can reveal how each item con- above 1, explaining 45% of the variance. However, inspect- tributes to these potential underlying factors, beyond internal ing the scree plot, it suggested a factor solution of three fac- consistency. Using EFA, all individual inter-item correlations tors. In addition, the parallel analysis also supported a can be examined. Preferably, all items should correlate mod- three-factor solution (see Figure 1). erately to each other in the range of .30 to .90 (Clark & Thus, a three-factor solution was evaluated further by Watson, 1995). Furthermore, if items are clustered together in running the EFA analysis again. The three-factor solution meaningful underlying factors, each item should correlate explained 32% of the total variance. Items not loading above with the underlying factor somewhere in the range of .30 to .4 on any of the three factors were removed. Thus, 13 items .50 (Fabrigar et al., 1999; DeVellis, 2016). EFA accounts for were removed. Finally, the EFA analysis was performed on measurements errors and require data on interval or quasi- the 20 remaining items. This three-factor explained 41% of interval level, as assumed for the data collected here. the total variance. A full overview of the specific items, the First, it is important to determine whether data are suit- three-factor solution, and the correlations between each item able for performing an EFA, that is, if there is a reasonable and their respective factor can be found in Table 2. A review level of inter-item correlations to further evaluate possible of the variance explained by the three-factor solution can be subfactors. The diagnostic analyses used were the Kaiser– obtained in Table 3. The final knowledge test can be found in Meyer–Olkin (KMO) measure of sampling adequacy, Supplemental Appendix A. The inter-correlations between assessing the potential of finding reliable and distinct con- the three factors were as follows: Factors 1 and 2: r = .07, structs. A KMO of 0.60 is generally considered as the cutoff p = .494; Factors 1 and 3: r = .02, p = .839; Factors 2 and value for continuing with an EFA. Furthermore, Bartlett’s 3: r = .02, p = .878. test of sphericity was implemented to assess whether corre- lations are significantly different from zero. Finally, the Final Factor Solution determinant analysis examined whether there were the inter- item correlations were at a suitable level. The results of the factor solution revealed one factor labeled Next, using principal axis factoring, the numbers of “Act in aversive states.” It contained 12 items relating to the potential factors in the material were estimated. Principal core CBT principle of approaching and exposing yourself to axis factoring is an EFA method that aims to determine the situations and negative internal experiences, rather than number of factors by creating an extracted factor solution avoiding them. This sometimes includes the ability to toler- that maximizes the amount of explained variance (De Winter ate fear in the present moment, for example, “Marcus is just & Dodou, 2012). Moreover, an orthogonal rotation was about to attend an important class, but becomes nervous, gets implemented (varimax), to clarify the suggested factor solu- palpitations and chills. According to CBT, what could he tion. The purpose was to find the factor solution that was the try?” (Item 7), or engage in behavioral activation in the pres- easiest one to interpret (simple solution). ence of depressive thoughts, for example, “Peter is tired even To explore the validity of the extracted factor solution, though he sleeps a lot. He does not have the energy to do eigenvalues were used, establishing factors that are greater things he used to. What would be most helpful for him to try, than 1, according to the Kaiser criterion. To graphically inspect according to CBT?” (Item 9). The goal is to replace strate- the number of factors before the last major drop in eigenval- gies of short-term aversion reduction with strategies more in ues, the scree test was also used. As a final step in validating line with valued long-term consequences. the extracted factor solution, a parallel analysis was performed, The second factor was labeled “Using positive reinforce- using the syntax by O’Connor (2000), comparing the retained ment,” with five items reflecting the CBT principle of using, factor solution with randomly generated data sets. focusing, and planning for positive reinforcement, such as engaging in behaviors in line with positive goals, for exam- Results ple, “Lisa is afraid to use the subway to school. According to CBT, what would make her feel better?” (Item 19) or sched- General Results uling rewards contingent upon valued behavior, for example “Oscar wants to be more helpful at home but has trouble get- The reliability analysis revealed a strong internal consistency ting started. How can he help himself, according to CBT?” of α = .85 for the 33 initial items and an excellent alpha rat- (Item 20). ing for the certainty ratings α = .92. 6 SAGE Open Figure 1. Parallel analysis of the factor solution. The third factor relates to “Shifting attention,” containing Via an exploration of theoretically relevant constructs three items reflecting CBT strategies how to shift your focus using EFA, a three-factor solution with 20 items was found and attention in the presence of unhelpful cognitions or reac- with a high internal consistency for the whole instrument. The tions, for example, “Sofia is at a party and feels socially first factor was defined as “Act in aversive states.” This factor excluded. She is thinking that nobody likes her and that her seems to capture the central CBT-principle of approaching friends find her boring. What is most helpful for her to do, aversive situations, that is, how to enhance exposure to inter- according to CBT?” (Item 14). nal negative responses and reactions and reduce tendencies of The internal consistency for the final factor solution was avoiding them (Abramowitz, 2013; Arch & Craske, 2008). α = .84. Factor 1 had an α of .82, Factor 2 had an α of .73, The factor reflects the core CBT approach to act with nega- and Factor 3 had an α of .58. See Table 4 for the means, stan- tive and aversive thoughts, feelings, and sensations, in line dard deviations, and internal consistencies among the with adaptive long-term consequences. The ability to accept factors. and challenge negative emotional states is associated with reduced long-term distress, whereas experiential avoidance strategies tend to be associated with more severe symptoms Discussion of anxiety (Abramowitz, 2013). Some of these items are in line with the learning-based exposure paradigms of CBT, General Discussion where a major goal is to teach clients that they can tolerate This aim of this study was to develop and evaluate a new fear and aversive states (Arch & Craske, 2008). It is a shift knowledge test, with the aim to assess explicit knowledge from teaching clients to stay in a situation until fear has about general and applied CBT principles in the context of declined and instead help them to stay in the situation in pres- adolescents suffering from anxiety and depression. Our goal ence of aversive states, while getting in contact with more was to address some of the issues raised in relation to previ- neutral or positive consequences of the situation. Factor 2, ous research on knowledge gain and learning in general “Using positive reinforcers,” seems to capture the CBT prin- health care, CBT, and ICBT, in particular, exemplifying one ciple about using positive reinforces contingent upon valued way to construct a knowledge test for adolescents. Items action (Albano & Kendall, 2002; Brewin, 1996). This can be were mainly created using a literature review, subject matter illustrated using rewards or focusing on the positive aspects experts, and content from earlier studies on ICBT for adoles- of a previously avoided behavior. Increasing contact with cents with anxiety and depression. positive reinforcement is a generic CBT principle, but more Berg et al. 7 Table 2. Final Three-Factor Solution. Items Item loading on each factor Item scores M (SD) 1 2 3 Raw Certainty Weighted 1. According to CBT. What happens if you escape from a .505 0.52 (0.05) 1.60 (0.08) .67 (0.08) harmless situation that raises anxiety? 2. When trying to manage depressive feelings and tiredness, .527 0.36 (0.05) 1.54 (0.07) 0.39 (0.07) which strategy is most effective, in the long run? 3. Which are the two most helpful strategies to deal with .475 0.63 (0.05) 1.67 (0.08) 0.66 (0.08) stress, in the long run? 4. Michael often feels depressed and dull. What is the best .513 0.60 (0.05) 1.57 (0.08) 0.59 (0.08) thing he can do to feel better, according to CBT? 5. Emma suffers from low self-esteem, which prevents her from .773 0.37 (0.05) 1.53 (0.08) 0.41 (0.08) joining a theater class. What advice would CBT give her? 6. Molly is afraid of going downtown when the city is .480 0.51 (0.05) 1.57 (0.07) 0.48 (0.07) crowded with people. According to CBT, what could she try to do? 7. Marcus is just about to attend an important class, but .627 0.56 (0.05) 1.59 (0.08) 0.61 (0.08) becomes nervous, gets palpitations and chills. According to CBT, what could he try? 8. Alice finds it scary to eat lunch with her best friends at .680 0.59 (0.05) 1.60 (0.08) 0.63 (0.08) school, so she ends up eating lunch by herself. What could she try doing, according to CBT? 9. Peter is tired even though he sleeps a lot. He does not .662 0.51 (0.05) 1.40 (0.07) 0.56 (0.07) have the energy to do things he used to. What would be most helpful for him to try, according to CBT? 10. Eric has a hectic schedule and plans everything in detail to −.473 0.29 (0.05) 1.53 (0.08) 0.20 (0.07) do everything he wants to do. Yet, he does not feel happy. What would be the main reason, according to CBT? 11. Jenny is afraid that nobody will like her. She is only doing .593 0.42 (0.05) 1.46 (0.07) 0.39 (0.07) what others want her to do, in fear of being left out by her friends. According to CBT, what would be good to try? 12. Ebba is about to hang out with a new friend she likes, .605 0.61 (0.05) 1.57 (0.08) 0.72 (0.08) but worries about how it will turn out. She is afraid it will be awkward or boring. What is helpful for her to do, according to CBT? 13. Edvin is afraid of embarrassing himself in front of his .604 0.46 (0.05) 1.58 (0.08) 0.54 (0.07) friends and rarely joins them when they are hanging out, even though he wants to. What could he try to do, according to CBT? 14. Sofia is at a party and feels socially excluded. She is .463 0.40 (0.05) 1.46 (0.07) 0.37 (0.07) thinking that nobody likes her and that her friends find her boring. What is most helpful for her to do, according to CBT? 15. Johanna is suddenly overwhelmed by her emotions, she .553 0.28 (0.05) 1.47 (0.07) 0.19 (0.06) experiences difficulties in breathing and thinking normally. She is angry and upset since she is not invited to a party a girl in her class is hosting. What is most helpful to do in this moment, according to CBT? 16. Anna is about to hold a lecture to her class, but she feels .566 0.55 (0.05) 1.55 (0.08) 0.56 (0.07) very nervous and plans to pretend to have a cold to avoid it. How could one perceive this situation, according to CBT? 17. Sara worries about her future, she often thinks about what .707 0.39 (0.05) 1.49 (0.07) 0.35 (0.07) she should do and gets stuck while searching the web for various options on education and travels. What would CBT recommend her to do? 18. Gustav is just about to start studying, but feels an urge to .717 0.64 (0.05) 1.56 (0.08) 0.69 (0.07) check out Instagram and often gets stuck while doing it. What would CBT advise him to do? 19. Lisa is afraid to use the subway to school. According to .429 0.42 (0.05) 1.53 (0.07) 0.38 (0.07) CBT, what would make her feel better? 20. Oscar wants to be more helpful at home but has trouble .663 0.78 (0.04) 1.64 (0.08) 0.86 (0.07) getting started. How can he help himself, according to CBT? Note. Principal axis factoring using orthogonal rotation. CBT = cognitive behavioral therapy. 8 SAGE Open Table 3. Total Variance Explained With the Three Factor Solution. Factor M SD α 1 Act with aversion 6.37 4.95 .82 2 Using positive reinforces 2.97 2.38 .73 3 Shifting attention 0.91 1.40 .58 Full scale 10.25 6.82 .84 Table 4. Means, Standard Deviations, and Internal Consistency of the Three Factors. Original sample (N = 93) Number Factor Eigen value Variance % Cumulative % 1 Act with aversion 4.42 22.11 22.11 2 Using positive reinforcement 2.33 11.66 33.78 3 Shifting your attention 1.44 7.21 40.99 explicitly and evidently used when treating depressive states adjusting for the fact that clients can be certain but incorrect (Dimidjian et al., 2011). Factor 3, “Shifting attention,” con- or correct but unsure about their answer. Especially, there is tained three items and reflected the central role of managing a commonsense aspect in CBT where the right alternative mental processes of negative beliefs, impulses, and atten- could be more or less obvious (as commented by the subject tional biases in CBT for anxiety-related problems (Arch & matter experts and the participants in the cognitive inter- Craske, 2008; Waters & Craske, 2016). This factor seems to view). Incorporating ratings of certainty is one way to mea- capture strategies of shifting your attention, for example, sure this aspect, but there might also be other more useful strategies in line with the CBT notion that thoughts are ways to tap and control for accuracy in contrast to certainty. hypotheses, rather than objective facts, not needed to be acted Furthermore, the clinical importance of certainty warrants upon. Attention shifting seems to be connected to the ability future research. Does certainty increase clients’ motivation to inhibit and interrupt pre-potent impulses and to engage in in trying out strategies and is higher certainty a desirable out- more adaptive behaviors (Bögels et al., 2008; Donald et al., come in psychotherapy? 2014). This, finally, relates to the clinical use of the instrument, To our knowledge, this is the first study in which a knowl- which needs further evaluation. Studies measuring knowl- edge test to be used in ICBT was developed using EFA, edge acquisition through multiple-choice tests have found showing three distinct factors in the context of adolescent unclear or no associations with treatment outcome depression and anxiety. The procedure resulted in a knowl- (Andersson et al., 2012; Berg et al., 2019; Scogin et al., edge test with higher reliability and validity than the previ- 1998; Strandskov et al., 2017). In comparison with free- ous tests used in ICBT (e.g., Andersson et al., 2012; Berg recall tasks, multiple-choice test rather measures recogni- et al., 2019; Strandskov et al., 2017). This is in line with the tion or cued recall (Gumport et al., 2018). Even if clients need of developing more robust measures and validated sub- can recognize the correct answers about CBTsstrategies, it scales when assessing knowledge in mental health interven- does not necessarily imply that they also recall and apply tions (O’Connor et al., 2014; Wei et al., 2013). The third them in real-life situations. This kind of acquired knowl- factor, however, had poorer internal consistency, which edge might be better captured through open-ended ques- could be due to the number of items or reflecting a lack of tions assessing free-recall and clients’ own words (Gumport reliability for the subconstruct. Importantly, the factor solu- et al., 2018). tion explained 41% and it is possible that other interesting Other main important future research questions regarding dimensions in the knowledge of CBT are missing. Also, the clinical use of the instrument are whether acquired overall, the scales’ convergent and discriminant validity knowledge about these core aspects during therapy can serve needs to be evaluated further, for instance, by correlating the as a protective function in the long run for youths with anxi- scale against other scales of knowledge, symptoms, or ety and depression. In addition, if measured and evaluated in behavioral outcomes in clinical samples. a valid and reliable way, knowledge and learning could be One part of the instrument warranting further investiga- studied as a potential mediator of treatment outcome. This is tion is the role of certainty ratings. As argued above, it is of in line with recent trends in clinical research attempting to importance to separate knowing from guessing or believing understand the active components in successful treatments, (O’Connor et al., 2014; Tiedens & Linton, 2001), and thus rather than only evaluating its general effects (Kazdin, 2007). Berg et al. 9 evaluation of knowledge about CBT strategies is how to Limitations evaluate a pragmatic application of the strategies in question. This study has several important limitations. For instance, as Knowing declarative knowledge does not necessarily equal revealed by the diagnostic test a priori EFA, the determinant using it in real life, and the clinical relevance of the construct indicated that the overall inter-item correlations were too measured here needs to be examined further. low. Thus, there is a risk of multicollinearity. Furthermore, to Despite these limitations, this study has several important perform an EFA, 100 participants are recommended (Fabrigar contributions. Hopefully, by describing one way to perform a & Wegner, 2012). In our study, 93 individuals submitted the test construction procedure, the field can keep developing test, and 86 completed all of the items. A larger sample size psychometrically robust tests that enable more useful con- is preferred and the results should thus be interpreted with clusions regarding what clients know, learn, and remember caution. Together, these two issues decrease the possibility to from ICBT. make stable conclusions from the EFA. Future studies should test the scale in larger samples. Furthermore, there is a need Acknowledgments of replicating the factor solution in the context of adolescent The authors of the current study would like to acknowledge the cli- depression and anxiety and also for other target groups. We nicians Maria Zetterqvist, Hana Jamali, and Helena Klemetz for do not know whether the results obtained here can be gener- their valuable contributions as subject matter experts during the test alized to other populations. The present test was developed construction procedure. George Vlaescu is thanked for his excellent in a school setting and not in a clinical group and the next technical support. Furthermore, all the teachers and adolescents step is to evaluate the test in a clinical setting. who participated are thanked for responding and providing feed- Importantly, EFA is partly a subjective method, as the back on the test. interpretation of results is based on subjective decisions. Using the Kaiser criterion or scree test can lead to over-or- Declaration of Conflicting Interests under factoring, or missing factors that are theoretically rel- The author(s) declared no potential conflicts of interest with respect evant. Theoretically, a test covering general and applied CBT to the research, authorship, and/or publication of this article. principles captures more than three underlying dimensions, as there are several important theoretical underpinnings, psy- Funding choeducative models, and managing strategies in CBT for The author(s) disclosed receipt of the following financial support anxiety and depression (sleeping strategies, breathing and for the research, authorship, and/or publication of this article: This relaxation techniques, etc.). It is important, however, to high- study was supported by a grant from the Swedish Foundation for light that this knowledge test aimed to target core, transdiag- Humanities and Social Sciences, Grant P16-0883:1 (from the nostic aspects of CBT strategies in the context of adolescent Swedish Central Bank). The funding source was not involved in the anxiety and depression, rather than being a generic test appli- execution or content of the study. cable in diverse populations. It is also reasonable that the content in knowledge tests varies in relevance, depending on ORCID iD where and for whom an ICBT treatment is provided. When Matilda Berg https://orcid.org/0000-0003-3375-0556 evaluating CBT for other problems, such as schizophrenia, other types of items and knowledge content would be of rel- Supplemental Material evance. Indeed, it is difficult to develop a test that covers all Supplemental material for this article is available online. content deemed necessary. Hopefully, this test construction procedure can exemplify one way to capture knowledge gain References about anxiety and depression when receiving CBT for youths in an internet-based format. Abramowitz, J. S. (2013). The practice of exposure therapy: Another important issue when measuring knowledge and Relevance of cognitive-behavioral theory and extinction the- ory. Behavior Therapy, 44(4), 548–558. learning about CBT principles, as highlighted by the subject Albano, A. M., & Kendall, P. C. (2002). Cognitive behavioural matter experts during the test construction procedure, is how therapy for children and adolescents with anxiety disorders: to frame and measure flexibility in the use and application of Clinical research advances. International Review of Psychiatry, explicit CBT strategies when constructing a multiple-choice 14, 129–134. test. The items in the knowledge test might capture the core Andersson, G. (2016). Internet-delivered psychological treatments. point about tolerating and challenging one’s fears and inter- Annual Review of Clinical Psychology, 12, 157–179. nal negative reactions, rather than avoiding them by thinking Andersson, G., Carlbring, P., Furmark, T., & SOFIE Research positively or using safety behaviors. However, various Group. (2012). Therapist experience and knowledge acquisi- avoidance strategies are also helpful for clients in their tion in internet-delivered CBT for social anxiety disorder: A everyday life. For instance, research has shown the potential randomized controlled trial. PloS One, 7(5), e37411. functionality of safety behaviors in certain contexts Arch, J. J., & Craske, M. G. (2008). Acceptance and commit- ment therapy and cognitive behavioral therapy for anxiety (Rachman et al., 2008). Thus, one challenge regarding the 10 SAGE Open disorders: Different treatments, similar mechanisms? Clinical patient education programs. Patient Education and Counseling, Psychology: Science and Practice, 15(4), 263–279. 77(1), 6–13. Berg, M., Rozental, A., Johansson, S., Liljethörn, L., Radvogin, E., Friedberg, R. D., Viglione, D. J., Fidaleo, R. A., Celeste, B. L., Topooco, N., & Andersson, G. (2019). The role of knowledge Lovette, J., Street, G., Yerka, E., Bieraugel, M., Dumas, M., in Internet-based cognitive behavioural therapy for adoles- & Beal, K. G. (1998). Measuring how we preach what we cent depression: Results from a randomised controlled study. practice: Psychoeducational change in depressed inpatients. Internet Interventions, 15, 10–17. Journal of Rational-Emotive & Cognitive-Behavior Therapy, Bögels, S., Hoogstad, B., van Dun, L., de Schutter, S., & Restifo, 16(1), 45–59. K. (2008). Mindfulness training for adolescents with external- Gumport, N. B., Dong, L., Lee, J. Y., & Harvey, A. G. (2018). izing disorders and their parents. Behavioural and Cognitive Patient learning of treatment contents in cognitive therapy. Psychotherapy, 36(2), 193–209. Journal of Behavior Therapy and Experimental Psychiatry, 58, Brewin, C. R. (1996). Theoretical foundations of cognitive-behav- 51–59. ioral therapy for anxiety and depression. Annual Review of Gumport, N. B., Williams, J. J., & Harvey, A. G. (2015). Learning Psychology, 47, 33–57. cognitive behavior therapy. Journal of Behavior Therapy and Burns, J. R., & Rapee, R. M. (2006). Adolescent mental health lit- Experimental Psychiatry, 48, 164–169. eracy: Young people’s knowledge of depression and help seek- Haladyna, T. M. (1994). Developing and validating multiple-choice ing. Journal of Adolescence, 29(2), 225–239. test items. Lawrence Erlbaum. Clark, L. A., & Watson, D. (1995). Constructing validity: Basic issues Harvey, A., Lee, J., Williams Hollon, S., Walker, M., Thompson, in objective scale development. Psychological Assessment, 7(3), M., & Smith, R. (2014). Improving outcome of psychoso- 309–319. cial treatments by memory and learning. Perspectives on Coles, M. E., Ravid, A., Gibb, B., George-Denn, D., Bronstein, L. Psychosocial Science, 6, 161–179. R., & McLeod, S. (2016). Adolescent mental health literacy: Harvey, A. G., Lee, J., Smith, R. L., Gumport, N. B., Hollon, S. D., Young people’s knowledge of depression and social anxiety Rabe-Hesketh, S., Hein, K., Dolsen, M., Haman, K., Kanady, disorder. Journal of Adolescent Health, 58(1), 57–62. J., Thompson, M., & Abrons, D. (2016). Improving outcome Costello, E. J., Mustillo, S., Erkanli, A., Keeler, G., & Angold, A. for mental disorders by enhancing memory for treatment. (2003). Prevalence and development of psychiatric disorders Behaviour Research and Therapy, 81, 35–46. in childhood and adolescence. Archives of General Psychiatry, Kazdin, A. E. (2007). Mediators and mechanisms of change in psy- 60(8), 837–844. chotherapy research. Annual Review of Clinical Psychology, DeVellis, R. F. (2016). Scale development: Theory and applica- 3(1), 1–27. tions (Vol. 26). Sage. Keulers, B. J., Welters, C. F., Spauwen, P. H., & Houpt, P. (2007). De Winter, J. C., & Dodou, D. (2012). Factor recovery by princi- Can face-to-face patient education be replaced by computer- pal axis factoring and maximum likelihood factor analysis as a based patient education? A randomised trial. Patient Education function of factor pattern and sample size. Journal of Applied and Counseling, 67(1), 176–182. Statistics, 39(4), 695–710. Kronmüller, K.-T., Victor, D., Schenkenbach, C., Postelnicu, I., Dimidjian, S., Barrera, M., Martell, C., Muñoz, R. F., & Lewinsohn, Backenstrass, M., Schröder, J., & Mundt, C. (2007). Knowledge P. M. (2011). The origins and current status of behavioral acti- about affective disorders and outcome of depression. Journal vation treatments for depression. Annual Review of Clinical of Affective Disorders, 104(1–3), 155–160. Psychology, 7, 1–38. Krosnick, J. A. (1999). Survey research. Annual Review of Donald, J., Abbott, M. J., & Smith, E. (2014). Comparison of Psychology, 50(1), 537–567. attention training and cognitive therapy in the treatment of Lee, J. Y., & Harvey, A. G. (2015). Memory for therapy in bipolar social phobia: A preliminary investigation. Behavioural and disorder and comorbid insomnia. Journal of Consulting and Cognitive Psychotherapy, 42(1), 74–91. Clinical Psychology, 83(1), 92–102. Dong, L., Lee, J. Y., & Harvey, A. G. (2017). Memory support Lukens, E. P., & McFarlande, W. R. (2004). Psychoeducation as strategies and bundles: A pathway to improving cognitive evidence-based practice: Considerations for practice, research, therapy for depression? Journal of Consulting and Clinical and policy. Brief Treatment and Crisis Intervention, 4(3), 205– Psychology, 85(3), 187–199. 225. Ebert, D. D., Zarski, A. C., Christensen, H., Stikkelbroek, Y., O’Connor, B. P. (2000). SPSS and SAS programs for determin- Cuijpers, P., Berking, M., & Riper, H. (2015). Internet and ing the number of components using parallel analysis and computer-based cognitive behavioral therapy for anxiety and Velicer’s MAP test. Behavior Research Methods, Instruments, depression in youth: A meta-analysis of randomized controlled & Computers, 32(3), 396–402. outcome trials. PLoS ONE, 10(3), e0119895. O’Connor, M., Casey, L., & Clough, B. (2014). Measuring mental Fabrigar, L. R., & Wegener, D. T. (2012). Exploratory factor analy- health literacy: A review of scale-based measures. Journal of sis. Oxford University Press. Mental Health, 23(4), 197–204. Fabrigar, L. R., Wegener, D. T., MacCallum, R. C., & Strahan, Rachman, S., Radomsky, A. S., & Shafran, R. (2008). Safety behav- E. J. (1999). Evaluating the use of exploratory factor analy- iour: A reconsideration. Behaviour Research and Therapy, sis in psychological research. Psychological Methods, 4(3), 46(2), 163–173. 272–299. Ryhänen, A. M., Siekkinen, M., Rankinen, S., Korvenranta, H., & Fox, M. P. (2009). A systematic review of the literature reporting on Leino-Kilpi, H. (2010). The effects of Internet or interactive studies that examined the impact of interactive, computer-based computer-based patient education in the field of breast cancer: A Berg et al. 11 systematic literature review. Patient Education and Counselling, Tiedens, L. Z., & Linton, S. (2001). Judgment under emotional 79, 5–13. certainty and uncertainty: The effects of specific emotions Sajatovic, M., Chen, P., Dines, P., & Shirley, E. R. (2007). on information processing. Journal of Personality and Social Psychoeducational approaches to medication adherence in Psychology, 81(6), 973–988. patients with bipolar disorder. Disease Management & Health Topooco, N., Berg, M., Johansson, S., Liljethörn, L., Radvogin, R., Outcomes, 15(3), 181–192. Nordgren, L., Zetterqvist, M., & Andersson, G. (2018). The Scogin, F., Jamison, C., Floyd, M., & Chaplin, W. (1998). efficacy of chat- and internet-based cognitive behaviour ther- Measuring learning in depression treatment: A cognitive bib- apy for adolescent depression: A randomised controlled trial. liotherapy test. Cognitive Therapy Research, 22(5), 475–482. British Journal of Psychiatry Open, 4, 199–207. Silfvernagel, K., Gren Landell, M., Emanuelsson, M., Carlbring, P., Waters, A. M., & Craske, M. G. (2016). Towards a cognitive- & Andersson, G. (2015). Individually tailored internet-based learning formulation of youth anxiety: A narrative review of cognitive behavior therapy for adolescents with anxiety disor- theory and evidence and implications for treatment. Clinical ders: A pilot effectiveness study. Internet Interventions, 23(3), Psychology Review, 50, 50–66. 297–302. Wei, Y., Hayden, J. A., Kutcher, S., Zygmunt, A., & McGrath, P. Silfvernagel, K., Wasserman, C., & Andersson, G. (2017). (2013). The effectiveness of school mental health literacy pro- Individually tailored internet-based cognitive behavioural ther- grams to address knowledge, attitudes and help seeking among apy for young adults with anxiety disorders: A pilot effective- youth. Early Intervention in Psychiatry, 7(2), 109–121. ness study. Internet Interventions, 8, 48–52. Strandskov, S. W., Ghaderi, A., Andersson, H., Parmskog, N., Hjort, Author Biographies E., Wärn, A., Jannert, M., & Andersson, G. (2017). Effects of Matilda Berg is a PhD student of Psychology at Linköping tailored and ACT-influenced Internet-based CBT for eating University. Her research focuses on how to measure and manipulate disorders and the relation between knowledge acquisition and knowledge in internet-based Cognitive Behavioral Therapy for outcome: A randomized controlled trial. Behavior Therapy, adolescents with anxiety and depression. 485(5), 624–637. Terides, M. D., Dear, B. F., Karin, E., Jones, M. P., Gandy, M., Gerhard Andersson is a professor at Linköping University. He has Fogliati, V. J., . . . Titov, N. (2016). The Frequency of Actions a PhD in both Clinical Psychology and in Otorhinolaryngology. and Thoughts Scale: Development and psychometric validation of a measure of adaptive behaviours and cognitions. Cognitive Alexander Rozental is a post doctoral researcher at the Department Behaviour Therapy, 45(3), 196–216. of Clinical Neuroscience at Karolinska Institutet. His research Thapar, A., Collishaw, S., Pine, D. S., & Thapar, A. K. (2012). involves the treatment of psychiatric disorders via the Internet, neg- Depression in adolescence. The Lancet, 379(9820), 1056– ative effects of psychological treatments, as well as the topics of 1067. procrastination and perfectionism.
SAGE Open – SAGE
Published: Jan 9, 2020
Keywords: psychology; social sciences; cognitive behavioral therapy; psychotherapy; clinical psychology; measurement and scaling methods; research methods; educational psychology; applied psychology; approaches
Access the full text.
Sign up today, get DeepDyve free for 14 days.