Get 20M+ Full-Text Papers For Less Than $1.50/day. Subscribe now for You or Your Team.

Learn More →

Item response analysis of the Positive and Negative Syndrome Scale

Item response analysis of the Positive and Negative Syndrome Scale Background: Statistical models based on item response theory were used to examine (a) the performance of individual Positive and Negative Syndrome Scale (PANSS) items and their options, (b) the effectiveness of various subscales to discriminate among individual differences in symptom severity, and (c) the appropriateness of cutoff scores recently recommended by Andreasen and her colleagues (2005) to establish symptom remission. Methods: Option characteristic curves were estimated using a nonparametric item response model to examine the probability of endorsing each of 7 options within each of 30 PANSS items as a function of standardized, overall symptom severity. Our data were baseline PANSS scores from 9205 patients with schizophrenia or schizoaffective disorder who were enrolled between 1995 and 2003 in either a large, naturalistic, observational study or else in 1 of 12 randomized, double-blind, clinical trials comparing olanzapine to other antipsychotic drugs. Results: Our analyses show that the majority of items forming the Positive and Negative subscales of the PANSS perform very well. We also identified key areas for improvement or revision in items and options within the General Psychopathology subscale. The Positive and Negative subscale scores are not only more discriminating of individual differences in symptom severity than the General Psychopathology subscale score, but are also more efficient on average than the 30-item total score. Of the 8 items recently recommended to establish symptom remission, 1 performed markedly different from the 7 others and should either be deleted or rescored requiring that patients achieve a lower score of 2 (rather than 3) to signal remission. Conclusion: This first item response analysis of the PANSS supports its sound psychometric properties; most PANSS items were either very good or good at assessing overall severity of illness. These analyses did identify some items which might be further improved for measuring individual severity differences or for defining remission thresholds. Findings also suggest that the Positive and Negative subscales are more sensitive to change than the PANSS total score and, thus, may constitute a "mini PANSS" that may be more reliable, require shorter administration and training time, and possibly reduce sample sizes needed for future research. Page 1 of 10 (page number not for citation purposes) BMC Psychiatry 2007, 7:66 http://www.biomedcentral.com/1471-244X/7/66 mining whether or not remission has been met. Alter- Background The Positive and Negative Syndrome Scale (PANSS) is the nately, if the threshold of "3 or less" corresponds to higher most widely used measure of symptom severity in schizo- severity for an item (i.e., is more easily achieved as symp- phrenia [1-3]. This 30-item scale is typically administered tom severity improves), then either that item is redundant by trained clinicians who evaluate patients' current sever- (since it is more likely to be reached first) or else its thresh- ity level on each symptom (item) by endorsing 1 of 7 old should be revised downward. For example, the remis- options (weights) numbered 1 through 7. The PANSS has sion threshold for that item could be set at 2 rather than demonstrated high internal reliability [4,5], good con- at 3. IRT analyses examine the manner in which individ- struct validity [4], and excellent sensitivity to change in ual item options (and cutoff scores) are related to overall both short term [6] and long term trials [7]. However, symptom severity which is central to evaluating the despite extensive psychometric research, it is unclear how appropriateness of proposed cut-point scores (thresholds) individual PANSS items differ in their usefulness in assess- determining illness remission. ing the severity of schizophrenia. The 3 primary purposes of our study are (a) to examine Indeed, studies examining the psychometric properties of and characterize the performance of individual items the PANSS have focused, without exception, on estimates from the PANSS at both the option (severity) and item of scale reliability, validity, and factor analysis using (symptom) levels with the goal of identifying areas for methods from classical test theory [8] and have typically improvement, (b) to examine the ability of various sub- identified 5 underlying factors [9-14]. Approaches based scales to discriminate among individual difference in ill- on classical test theory rely primarily on omnibus statistics ness severity, which might then identify a better measure that average over levels of individual variation and offer of change, and (c) to evaluate the appropriateness of items no means to gauge the quality of individual items or and options proposed for determining when symptoms options across different levels of symptom severity. In remission has been achieved. contrast, methods based on item response theory (IRT) [15] provide significant improvements over classical tech- Methods niques, as they model the relation between item responses Participants Data included baseline PANSS item scores from 9205 and symptom severity directly, quantifying how the per- formance of individual items and options (severity levels patients with schizophrenia, schizoaffective, or schizo- 1 to 7) change as a function of overall, standardized, phreniform disorders who were enrolled between 1995 symptom severity. IRT analyses provide unique and rele- and 2003 in either a large, naturalistic, observational vant information concerning (a) how well a set of item study or else in 1 of 12 randomized, double-blind clinical options assess the entire continuum of symptom severity, trials comparing olanzapine to other antipsychotic drugs. (b) whether weights assigned to individual item options Participants' diagnosis was based on DSM-IV criteria per are appropriate for measuring a particular trait or symp- treating physician and patient's medical record. Partici- tom, and (c) how well individual items or subscales are pants were primarily male (65%), could be either inpa- connected to the underlying construct and discriminate tients or outpatients, and had an average age of 39.0 (SD among individual differences in symptom severity (see = 11.5). Their initial PANSS total score averaged 81.9 (SD the publication by Santor and Ramsay [16] for an over- = 22.0) but ranged broadly from 30 to 177. view). Analytic models IRT methods are ideal for examining the performance of IRT consists of a broad class of statistical procedures [18] options within items that are to be used to define remis- used to model the association between responses to sur- sion of psychopathology. Andreasen and her colleagues vey items (in probabilistic terms) and an underlying [17] published guidelines recommending that schizo- latent trait, characteristic, or condition, such as overall phrenia remission be defined as achieving option scores symptom severity. We employed the same nonparametric less than or equal to 3 on each of only 8 PANSS items: kernel-smoothing techniques implemented in the soft- Delusions, Unusual Thought Content, Hallucinatory ware (TESTGRAF) developed by Ramsay [19,20] to esti- Behavior, Conceptual Disorganization, Mannerisms and mate option characteristic curves, which serve as a Posturing, Blunted Affect, Social Withdrawal, and Lack of measure of item and option effectiveness. These tech- Spontaneity. Setting equal remission thresholds (≤3) for niques have been used previously to examine the psycho- all 8 items suggests that the level of symptom severity cor- metric properties of self-report scales and to evaluate item responding to "3 or less" is generally equal for all 8 items. bias [21-23] and a detailed description of the algorithm If the region of symptom severity at which Options 1, 2, used to estimate response curves has been published else- and 3 are most likely to be endorsed differs across items, where [16,19]. then some items are more influential than others in deter- Page 2 of 10 (page number not for citation purposes) BMC Psychiatry 2007, 7:66 http://www.biomedcentral.com/1471-244X/7/66 To illustrate how item response models can be used to score of 37.3 in our sample, and a standard normal score evaluate item and option performance, we present a set of of +2 corresponds to a total scale score of 139.8. hypothetical, "ideal" option characteristic curves in Figure 1, where severity has been expressed in both expected A number of important features of item performance are total scores (upper x-axis) and standard normal scores illustrated in Figure 1. First, the region in which each (lower x-axis). Expected total scores show the level of option tends to be endorsed most frequently is clearly symptom severity at which different options are endorsed. indicated. For example, the curve for Option 2 suggests Standard normal quantiles, which are analogous to z- that the probability of being endorsed is essentially 0 at scores, show the proportion of the sample at different lev- standardized severity -3, increases to about 0.52 at -1, and els of expected total scores. then returns to roughly 0 at severity +1. Specifically, Option 2 is more likely to be endorsed than any other Expressing severity this way is useful in that standard nor- option within the overall severity range from -1.5 to -0.5. mal scores contain rather widely appreciated information about the proportion of a population above or below Second, each of the option characteristic curves ideally integer values of -3, -2, -1, 0, +1, +2, and +3. Extreme val- increases rather rapidly with small increases in severity. ues on our curves still need to be interpreted with caution, For example, the probability of Option 2 being endorsed because, in spite of the very large number of PANSS scores doubles from 0.2 to 0.4 when severity increases by just evaluated here, sample sizes are small in the tails of the half of a standard unit from -2.0 to -1.5. Third, the severity overall severity distribution but still express information regions over which each option is most likely to be about relative levels of symptom severity. We have also endorsed are ordered, left to right, in the same way as the superimposed corresponding total symptom scores to option scores (weights) themselves. That is, the region in facilitate the interpretation of standard normal scores. A which Option 2 is most likely to be endorsed (near -1), standard normal score of -2 corresponds to a total scale falls between the regions in which Option 1 (near -2) and Option 3 (near 0) are most likely to be endorsed. Fourth, together, the options for an item span the full con- tinuum of severity from -3 to +3. Some options are only endorsed at high levels of severity (e.g., Options 5 and 6), whereas others are endorsed at low levels of severity (e.g., Options 0 and 1). If the majority of options on an item are endorsed at low levels of severity, one might characterize that item as too "easy"; a low severity patient might receive a high score on such an item. In contrast, if the majority of options on an item are endorsed only at high levels of severity, that item might be characterized as too "hard"; a high severity patient could receive a low score on such an item. Scales comprised primarily of "hard" items will be largely insensitive to individual differences in the lower or moderate range of symptom severity and pro- duce floor effects. Scales comprised primarily of "easy" items will be largely insensitive to individual differences in the high range of symptom severity and produce ceiling effects. Option c Figure 1haracteristic curves for an ideal item Once option characteristic curves have been estimated, Option characteristic curves for an ideal item. The expected item scores and expected total and subscale probability of endorsing an individual option (y-axis) from scores are easily computed. One simply multiplies the this idealized item is plotted as a function of expected total smoothed estimate of probability of endorsing each scores (upper x-axis) and standard normal quantiles (lower option by its option weight and then sums these products x-axis). Expected total scores show the level of symptom across all options within each item. Due to variation in severity at which different options are endorsed. Standard endorsement of options within items, these expected normal quantiles, which are analogous to z-scores, show the scores are arguably better estimates of the true distribu- proportion of the sample at different levels of expected total tion of schizophrenia severity measured by the PANSS scores. Key features of this idealized item are described in than are the corresponding distributions of observed the main text. scores and sums. After all, expected item scores can be Page 3 of 10 (page number not for citation purposes) BMC Psychiatry 2007, 7:66 http://www.biomedcentral.com/1471-244X/7/66 computed at each point along the severity continuum independent raters who were able to reproduce a similar from -3 to +3. pattern of results. Using the idealized item portrayed in Figure 1, we judge Defining a continuum of symptom severity items as "very good" (a) if there was some range of severity Given that option characteristic curves depend on how in which the majority of options were more likely to be symptom severity is defined, we confirmed the appropri- endorsed than any other, (b) if option characteristic ateness of modeling of items via their subscale scores by curves increased rapidly with changes in overall severity, conducting principal components analyses on our sample (c) if the region in which each option was most likely to of 9205 responses on all 30 PANSS items using varimax be endorsed were ordered, left to right, in accordance with rotation and Kaiser Normalization prior to modeling. their option scores (weights), and (d) if options for an Although the PANSS was originally designed with 3 sub- item spanned the full continuum of severity from -3 to +3. scales (Positive, Negative, and General Psychopathology), Items were judged as "good" if they had many, but not all, recent studies examining the internal structure of the scale of these features and "poor" if they showed few, if any, of [9-14] have typically identified 5 underlying factors: (a) these criteria. We recognize that these ratings are global positive symptoms, (b) negative symptoms, (c) hostile assessments of 4 very different criteria. Global ratings of symptoms, (d) disorganized symptoms, and (e) symp- "very good", "good", and "poor" along with ratings for toms of anxiety and depression. Our results suggest a 6- each of these 4 criteria for each of the 30 PANSS items are factor solution, with the first 5 factors corresponding to recorded in Table 1. We provided the criteria to 2 blinded the solution identified by Davis and Chen [9]. Using both the "proportion equal to 1" and the "eigenvalue greater Table 1: Discrimination Effectiveness for the PANSS Items Item Content Summary Evaluation 3 Factor Subscales Criterion A Criterion B Criterion C Criterion D 1. Delusions Very good Positive 6 Yes Yes Yes 2. Conceptual Disorganization Very good Positive 6 Yes Yes Yes 3. Hallucinatory Behavior Very good Positive 4 Some Yes Yes 4. Excitement Good Positive 6 Yes Yes No 5. Grandiosity Good Positive 4 Some Yes Yes 6. Suspiciousness/Persecution Very good Positive 6 Yes Yes Yes 7. Hostility Poor Positive 6 Yes Yes No 8. Blunted Affect Very good Negative 6 Yes Yes Yes 9. Emotional Withdrawal Very good Negative 7 Yes Yes Yes 10. Poor Rapport Good Negative 6 Yes Yes Yes 11. Passive Apathetic Social Withdrawal Very good Negative 7 Yes Yes Yes 12. Difficulty in Abstract Thinking Poor Negative 6 Some Some Yes 13. Lack of Spontaneity Conversation Very good Negative 6 Yes Yes Yes 14. Stereotyped Thinking Poor Negative 6 No Some Yes 15. Somatic Concern Poor General 4 No Some No 16. Anxiety Good General 5 Yes Yes No 17. Guilt Feelings Good General 5 Some Yes No 18. Tension Good General 5 No Yes Yes 19. Mannerisms and Posturing Poor General 6 Some Yes Yes 20. Depression Poor General 5 Yes Yes No 21. Motor Retardation Good General 5 Yes Yes No 22. Uncooperative Good General 6 Yes Yes No 23. Unusual Thought Content Poor General 6 Yes Yes No 24. Disorientation Poor General 4 No No No 25. Poor Attention Poor General 5 Yes Yes Yes 26. Lack of Judgment and Insight Poor General 5 Some Some Yes 27. Disturbance of Volition Poor General 6 Yes Yes Yes 28. Poor Impulse Control Good General 6 Yes Yes No 29. Preoccupation Poor General 5 Some No Yes 30. Active Social Avoidance Very good General 6 Yes Yes Yes Criterion A: Number of options for which there was some range of symptom severity in which the question was more likely to be endorsed than all other options. Criterion B: Extent to which option characteristic curves increased rapidly with changes in overall severity. Criterion C: Does the severity region in which each option was most likely to be endorsed correspond to option weights? Criterion D: Do options for an item span the full continuum of severity from -3 to +3? Page 4 of 10 (page number not for citation purposes) BMC Psychiatry 2007, 7:66 http://www.biomedcentral.com/1471-244X/7/66 than or equal to 1" criteria, 2 of the 10 items assessing dis- organized symptoms, namely Items 12 (Difficulty in Abstract Thinking) and 24 (Disorientation), loaded greater than 0.30 on a sixth factor. Eigenvalues for the 6 factors were 4.56 (Negative Symptoms), 3.41 (Disorgan- ized Symptoms), 3.16 (Positive Symptoms), 2.83 (Hostil- ity Symptoms), 2.48 (Anxiety and Depression Symptoms), and 1.65 (Items 12 and 24). Given that the internal consistency coefficient for the 10 disorganized symptoms (including Items 12 and 24) was still good (Cronbach α = 0.83), we elected to retain a 5-factor solu- tion and to include Items 12 and 24 as part of the Disor- ganized Symptoms subscale, which was originally identified by Davis and Chen [9] and independently vali- dated by others [11,13,14,24]. Cronbach α's for the 5 subscales were (a) positive factor, 0.85, (b) negative factor, 0.86, (c) hostile factor, 0.78, (d) disorganized factor, 0.83, and (e) anxiety and depression factor, 0.70. However, we also examined the performance Opt fo Figure 2 r Iiton characteristic curves em 1: Delusions and expected item total score of items that comprise the original General Psychopathol- Option characteristic curves and expected item total ogy subscale when comparing the ability of subscales to score for Item 1: Delusions. Option characteristic curves discriminate among individual differences in illness sever- (solid lines) and expected item total score (dashed line) are ity. Internal consistency for the original General Psycho- plotted as a function of scores on the Positive Symptomatol- pathology subscale was also adequate, 0.88, suggesting ogy Subscale form the PANNS, expressed as standard nor- that any of the original or newly recommended subscales mal scores (lower x-axis) and expected total scores (upper x-axis). The probability of endorsing an option characteristic would be meaningful indicators of underlying symptom curve (solid lines) is scaled on the left y-axis and the severity against which individual items and options could expected item score (dashed line) is scaled on the right y- be modeled. axis. Many features of an ideal item are evident in this plot. Results Our results are grouped in 3 categories: (a) item response modeling of individual items and options from the more severe levels of Delusions begins to increase rapidly PANSS as a function of symptom severity, (b) item with increases in the severity of positive symptoms. response modeling of various subscales, and (c) item response modeling of cutoff scores used to establish Option 2 assessing a "minimal level of suspiciousness" is symptom remission. most frequently endorsed when severity reaches a stand- ard normal score of -1, which corresponds to an expected Examining individual items and options from the PANSS total score of about 19.0 on the Positive Factor subscale See Additional file 1 which presents option characteristic and a PANSS total of 60 and then quickly decreases with curves for all items. A select number of option characteris- further increases in severity. Other option characteristic tic curves for items from the various subscales are pre- curves on this item perform equally well. Note that 6 out sented here and discussed in detail, including overall of the 7 response options (all but 2) have a region where performance as well as proposed remission thresholds. it is more likely to be endorsed than any other option, and all 7 regions occur in the same order as their weights. For Delusions example, the region in which Option 3 of mild symp- Item 1, from the PANSS Positive Factor subscale, assesses tomatology (i.e., "Presents a guarded or even openly dis- suspiciousness, defined as "Beliefs which are unfounded, trustful attitude, but thoughts, interactions, and behavior unrealistic, and idiosyncratic." Option characteristic are minimally affected.") is most likely to be endorsed lies curves for this item are presented in the second panel of between the region in which Option 2, "Questionable Figure 2 and show that the probability of rating Option 1, pathology", and Option 4, "Moderate" levels of psychopa- assessing an "absence of symptoms," decreases rapidly as thology, are most likely to be endorsed. the severity of positive psychotic symptoms begins to increase. Meanwhile, the probability of endorsing the Examining the shape of the curves also provides useful information regarding the precision with which individu- Page 5 of 10 (page number not for citation purposes) BMC Psychiatry 2007, 7:66 http://www.biomedcentral.com/1471-244X/7/66 als can be designated as showing mild or moderate psy- Grandiosity chopathology. Results show that, despite well- Grandiosity, Item 5, is defined as "Exaggerated self-opin- differentiated peaks for Options 3 (<0) and 4 (>0) on ion and unrealistic convictions of superiority, including Item 1 (Delusions), the response characteristic curves for delusions of extraordinary abilities, wealth, knowledge, these options overlap considerably suggesting that desig- fame, power, and moral righteousness" [1]. Figure 4 nations made solely on the basis of this question could be depicts the option characteristic curves for this item as imprecise and unreliable. being generally flatter and less peaked than those of other items. Indeed, the probability of endorsing Options 2, 3, Hallucinatory behavior and 5 from Item 5 increase much slower than the corre- It is instructive to compare the option characteristic curves sponding options for Items 1 or 3. for Item 1, Delusions, in Figure 2 with those for Item 3, Hallucinatory Behavior, defined as "Verbal report [s] or Other important differences between Item 5 (Grandiosity, behavior indicating perceptions which are not generated Figure 4) and Items 1 (Delusions, Figure 2) or 3 (Halluci- by external stimuli" in Figure 3. Options characteristic natory Behavior, Figure 3) are evident when comparing curves for Item 3 are far less differentiated than those for expected item scores. The expected score is depicted in Item 1. Note that Option 4 of Item 3 entirely overlaps Options 2 and 3. Indeed, at all levels of severity, the prob- ability of endorsing Option 4 is equal to or greater than the probability of endorsing Options 2 or 3. This level of analysis allows scale developers to identify items and options that might be improved either through training of raters or revision of options/items. Opt for Item 5: Figure 4 ion characteristic curves Grandiosity and expected item total score Option characteristic curves and expected item total score for Item 5: Grandiosity. Option characteristic curves (solid lines) and expected item total score (dashed line) are plotted as a function of scores on the Positive Symp- tomatology Subscale form the PANNS, expressed as stand- ard normal scores (lower x-axis) and expected total scores (upper x-axis). Option characteristic curves (solid lines) indi- cate that a number of opportunities for improvement. Option c for It Figure 3 em 3 h:arac Hallucinatory behavior teristic curves and expected item total score Options 2, 3 and 4 overlap substantially as do Options 6 and Option characteristic curves and expected item total 7 suggesting rating these options is inherently difficult for score for Item 3: Hallucinatory behavior. Option char- raters. Option 1 in this item was most frequently endorsed acteristic curves (solid lines) and expected item total score by over half the sample, indicating that these options are only (dashed line) are plotted as a function of scores on the Posi- endorsed as a higher level of symptoms severity than in other tive Symptomatology Subscale form the PANNS, expressed options. The item characteristic curve (broken line) as standard normal scores (lower x-axis) and expected total increases smoothly in all but the lowest region of symptom scores (upper x-axis). Option characteristic curves (solid severity, but increases more slowly than the item character- lines) indicate that a number of opportunities for improve- istic curve in Item 1 or 3. Expected item scores for this item ment. Options 2, 3 and 4 overlap substantially as do Options are generally lower at comparable regions of symptom sever- 6 and 7 suggesting rating these options is inherently difficult ity than for Items 1 or 3. for raters. Page 6 of 10 (page number not for citation purposes) BMC Psychiatry 2007, 7:66 http://www.biomedcentral.com/1471-244X/7/66 each of our figures by a broken or dashed line that gener- ally increases smoothly within the severity range from -2 to +2. However, results show that the expected item score for Item 1 (Delusions) and Item 3 (Hallucinatory Behav- ior) increase much more between -2 and +2 than the expected score for Item 5 (Grandiosity). Indeed, the value of the expected item scores at the midpoint of the sample (0 standard normal quantiles) is 3.5 (right hand abscissa) for Item 1 (Delusions), 3.8 for Item 3 (Hallucinatory Behavior), and only 2.2 for Item 5 (Grandiosity). In this sense, Item 5 assessing Grandiosity is "harder" (much less discriminating) than Items 1 or 3. That is, a score of 3.5 on Item 5 (Grandiosity) is expected only when symptom severity exceeds +1 standard units, which translates to 103 on the Positive Symptom subscale or 36.1 on the entire 30-item PANSS. Mannerisms and posturing Item 19 from the PANSS assesses Mannerisms and Postur- ing, which is defined as "Unnatural movements or pos- Opt for Item 19: Ma Figure 5 ion characteristic curves nnerisms and postur and expec ing ted item total score ture characterized as awkward, stilted, disorganized, or Option characteristic curves and expected item total bizarre". Option characteristic curves for this item, Figure score for Item 19: Mannerisms and posturing. Option 5, tend to increase less quickly than those for Item 1 characteristic curves (solid lines) and expected item total (Delusions.) As a result, Options 2 through 6 of Item 19 score (dashed line) are plotted as a function of scores on the are likely to be endorsed at much higher severity levels General Psychopathology Subscale from the PANNS, than the corresponding options of Item 1. Item 19 (Man- expressed as standard normal scores (lower x-axis) and expected total scores (upper x-axis). Option characteristic nerisms and Posturing) could, therefore, be termed a curves (solid lines) overlap between Options 1 and 2 but "hard" item. generally well differentiated. However, options are generally only endorsed at more severe levels of symptomatology. Evaluating total and subscale performance Option 1 was most likely to be endorsed than any other The ability of various subscales to discriminate among lev- option, in over half the sample, namely with individuals scor- els of severity was examined by computing and plotting ing less than 81 on the 30-item PANSS. expected subscale totals for the 5 PANSS factor scales of Davis and Chen [9], as well as for the 30-item PANSS scale in total (TOT) and the original General Psychopathology subscale in Figure 6. Results suggest that the Positive and ity. Specifically, the response characteristic curves for Negative subscales are more discriminating than the Gen- endorsing Option 1, 2, or 3 were computed by summing eral Psychopathology subscale score or the PANSS total across individual option characteristic curves. Results score. That is, a 1-unit change in underlying symptom show that the probability of obtaining a score of 3 or less severity corresponds to a greater expected change in per does decrease for 7 of the 8 core symptoms. For the "easy" item score (second panel) for the Positive and Negative Item 19 (Mannerisms and Posturing), the probability of subscales than for the Full Scale or General Psychopathol- obtaining a score of 3 or less decreases more slowly over a ogy subscale. slightly wider range of symptom severity. This means that a person with a severity score of +2 has a probability of 0.5 Evaluating the appropriateness of symptom remission of scoring 3 or less on Item 19 (Mannerisms and Postur- cutoff scores ing). Meanwhile, the probability of scoring 3 or less on Andreasen and her colleagues [17] recently recommended any other item drops to less than 0.2 at a severity score of that "symptom remission" be defined as achieving scores +2 having declined from a probability of 0.5 achieved at a of 3 or less on each of 8 core symptoms: Delusions, Unu- much lower severity score (Figure 7). sual Thought Content, Hallucinatory Behavior, Concep- tual Disorganization, Mannerisms and Posturing, Blunted As a result, reaching remission criteria for Item 19 will be Affect, Social Withdrawal, and Lack of Spontaneity. To easier than for the other 7 symptoms and suggests that a examine the appropriateness of this common cut-point, threshold of 1 or 2 for Item 19 might be more appropriate. we modeled the probability of being rated 3 or less for each of these 8 symptoms as a function of symptom sever- Page 7 of 10 (page number not for citation purposes) BMC Psychiatry 2007, 7:66 http://www.biomedcentral.com/1471-244X/7/66 E Figure 6 xpected total scores for subscales from the PANSS Opt posturing Figure 7 ion characteristic curves for Item 19: Mannerisms and Expected total scores for subscales from the PANSS. Option characteristic curves for Item 19: Manner- Results show that the Positive and Negative subscales are isms and posturing. Option characteristic curves describ- more discriminating (i.e., have steeper slopes) than other ing the probability with ratings of 3 or less are made as a subscales demonstrating stronger discrimination on a per function of symptom severity for items recommended to item basis (top right panel) than the total PANSS Scale. define symptom remission in patients. Results show that the option characteristic curves for this item will be endorsed at a much more severe level of symptomatology than other items and is therefore redundant or should be revised. The Discussion broken line (---) shows the option characteristic curves for a Option and item performance revised Item 19, where remission is defined on the basis of a Results of our analyses, summarized in Table 1, confirm score less than or equal to 2 (rather than 3). that most PANSS items are either very good or good at assessing the overall severity, particularly items within the Negative Symptom subscale. However, a number of items, largely those from the original General Psychopathology Subscale performance subscale, performed far less well due to difficulties with Our results show that items from the Positive and Nega- some options. One of the few, consistent difficulties for a tive subscales are generally more discriminating of indi- large number of items is the overlap in response character- vidual differences in severity than any other of the istic curves for Options 1 and 2. In fact, there may be no subscales. In particular, items from the Positive and Neg- region of severity in which Option 2, indicating minimal ative subscales are more discriminating on average than symptoms, is most likely to be endorsed than Option 1, the full 30-item PANSS total and, thus, may be more sen- indicating an absence of symptoms. This result is not sitive to change than the PANSS total. These 2 subscales unexpected, because the definition of Option 2 in all such might possibly constitute a "mini PANSS" that would be items includes "can be at the extreme of normal." This more reliable, require shorter administration and training phrasing appears to create greater overlap between time, and might even reduce the sample size needed for Options 1 and 2 than between other adjacent options. future schizophrenia research. Defining remission criteria Second, our results also demonstrate overlap between a number of adjacent option characteristic curves. In partic- Our detailed analyses of the 8 items currently recom- ular, the Difficulty in Abstract Thinking, Hostility, Unco- mended as markers of remission reveals a problem with operativeness, and Hallucinatory Behavior items display using a common threshold of "3 or less." Specifically, overlap between all items suggesting that they are poorly Option 3 of Item 19 (Mannerisms and Posturing) may be differentiated. Third, results showed that many options endorsed at slightly higher levels of severity than any of within several "hard" items are endorsed only at much the other 7 items being considered. As a result, the thresh- higher levels of severity. For example, Option 1 of Grandi- old for remission on Item 19 might be more appropriately osity was endorsed for more than half of the patients. set at 1 or 2. Examining the detailed plots of all option Page 8 of 10 (page number not for citation purposes) BMC Psychiatry 2007, 7:66 http://www.biomedcentral.com/1471-244X/7/66 characteristic curves for items used to define remission Acknowledgements This work was sponsored by Eli Lilly and Company. The authors thank (see Additional file 1) shows that Option 3 on Item 19 is Noreen Pierle (Eli Lilly and Company, Indianapolis, Indiana) for editing this the only option most likely to be endorsed in range of manuscript. symptom severity >0 (or >81 on the total PANSS). Options 1, 2, and 3 for all other items are most likely to References be endorsed in a range of severity less than 81. 1. Kay SR, Fiszbein A, Opler LA: The positive and negative syn- drome scale (PANSS) for schizophrenia. Schizophr Bull 1987, 13:261-276. Conclusion 2. Van den Oord EJ, Rujescu D, Robles JR, Giegling I, Birrell C, Bukszár This first item response analysis (IRA) of the PANSS sup- J, Murrelle L, Möller HJ, Middleton L, Muglia P: Factor structure ports the overall sound psychometric properties of the and external validity of the PANSS revisited. Schizophr Res 2006, 82:213-223. Epub 2005 Oct 17. PANSS and demonstrates that most of its items are very 3. Lader M: Rating Scales in Schizophrenia: A Review of Their good or good at assessing the overall schizophrenia sever- Usefulness for Assessing Atypical Antipsychotics. CNS Drugs 2000, 14(1):23-32(10). ity. There were a number of items, primarily on the Gen- 4. Kay SR, Opler LA, Lindenmayer JP: Reliability and validity of the eral Psychopathology subscale, that might best be positive and negative syndrome scale for schizophrenics. Psy- modified and/or scored with fewer levels. The Positive chiatry Res 1988, 23:99-110. 5. Peralta V, Cuesta MJ: Psychometric properties of the positive and Negative subscales may also be more sensitive to and negative syndrome scale (PANSS) in schizophrenia. Psy- change than the PANSS total score. On the basis of these chiatry Res 1994, 53:31-40. results, one might consider rewriting item options so that 6. Lindenmayer JP, Kay SR, Friedman C: Negative and positive schiz- ophrenic syndromes after the acute phase: a prospective fol- (a) they are more sensitive to differences at low severity low-up. Compr Psychiatry 1986, 27:276-286. and/or (b) the higher level options are endorsed at lower 7. Kay SR, Fiszbein A, Lindenmayer JP, Opler L: Positive and negative syndromes in schizophrenia as a function of chronicity. Acta severity levels. On the other hand, the observed failure to Psychiatr Scand 1986, 74:507-518. discriminate among individuals in the very lowest regions 8. Nunnally JC: Psychometric Theory 2nd edition. New York: McGraw of severity, such as -4 to -3, might be due to relatively high Hill; 1978. 9. Davis JM, Chen N: The effects of olanzapine on the 5 dimen- severity in our sample. Our curves may not be clinically sions of schizophrenia derived by factor analysis: combined useful with low severity patients. Alternately, one might results of the North American and international trials. J Clin Psychiatry 2001, 62:757-71. modify rater training to change option endorsement 10. Lancon C, Aghababian V, Llorca PM, Auquier P: Factorial structure probabilities. Our results suggest that certain options are of the Positive and Negative Syndrome Scale (PANSS): a not currently being adequately differentiated. Interest- forced five-dimensional factor analysis. Acta Psychiatr Scand 1998, 98:369-376. ingly, one might estimate option characteristic curves at 11. Lindenmayer JP, Grochowski S, Hyman RB: Five factor model of different phases or types of training to evaluate the effec- schizophrenia: replication across samples. Schizophr Res 1995, tiveness of different types and lengths of training pro- 14:229-234. 12. Lykouras L, Oulis P, Psarros K, Daskalopoulou E, Botsis A, Christo- grams. doulou GN, Stefanis C: Five-factor model of schizophrenic psy- chopathology: how valid is it? Eur Arch Psychiatry Clin Neurosci 2000, 250:93-100. Competing interests 13. Marder SR, Davis JM, Chouinard G: The effects of risperidone on The author(s) declare that they have no competing inter- the five dimensions of schizophrenia derived by factor analy- ests. sis: combined results of the North American trials. J Clin Psy- chiatry 1997, 58:538-546. Erratum in: J Clin Psychiatry 1998, 59:200. 14. White L, Harvey PD, Opler L, Lindenmayer JP: Empirical assess- Authors' contributions ment of the factorial structure of clinical symptoms in schiz- DS performed the analyses, led the interpretation of the ophrenia. A multisite, multimodel evaluation of the factorial structure of the Positive and Negative Syndrome Scale. The results, and drafted the manuscript. HA-S conceived of the PANSS Study Group. Psychopathology 1997, 30:263-274. study, participated in its design, the analytical plan, the 15. Lord FM: Applications of Item Response Theory to Practical Testing Prob- lems Hillsdale, NJ: Erlbaum; 1980. interpretation of the results, and helped draft the manu- 16. Santor DA, Ramsay JO: Progress in the technology of measure- script. JPL and RO participated in the design of the study, ment: Applications of Item Response models. Psych Assess the analytical plan, the interpretation of the results, and 1998, 10:345-359. 17. Andreasen NC, Carpenter WT Jr, Kane JM, Lasser RA, Marder SR, assisted in drafting the manuscript. All authors have read Weinberger DR: Remission in schizophrenia: proposed criteria and approved the final manuscript. and rationale for consensus. Am J Psychiatry 2005, 162:441-449. 18. van der Linden WJ, Hambleton RK: Handbook of Modern Item Response Theory New York: Springer; 1997. Additional material 19. Ramsay JO: Kernel smoothing approaches to non parametric item curve estimation. Psychometrika 1991, 56:611-630. 20. Ramsay JO: TestGraf: A Program for the Graphical Analysis of Multiple Choice Test and Questionnaire Data. Montreal, Can- Additional file 1 ada: McGill University; 2001. Appendix. Option characteristic curves for all PANSS items. 21. Santor DA, Ramsay JO, Zuroff DC: Non-parametric item analy- Click here for file sis of the Beck Depression Inventory: Evaluating gender item [http://www.biomedcentral.com/content/supplementary/1471- bias and response option weights. Psychol Assess 1994, 244X-7-66-S1.ppt] 6:255-270. Page 9 of 10 (page number not for citation purposes) BMC Psychiatry 2007, 7:66 http://www.biomedcentral.com/1471-244X/7/66 22. Santor DA, Ramsay JO, Zuroff DC, Cervantes P, Palacios J: Examin- ing scale discriminability in the BDI and CES-D as a function of depressive severity. Psychol Assess 1995, 7:131-139. 23. Santor DA, Coyne JC: Examining symptom expression as a function of symptom severity: item performance on the Hamilton Rating Scale for Depression. Psychol Assess 2001, 13:127-139. 24. Bryson G, Bell M, Greig T, Kaplan E: Internal consistency, tempo- ral stability and neuropsychological correlates of three cog- nitive components of the Positive and Negative Syndrome Scale (PANSS). Schizophr Res 1999, 38:27-35. Pre-publication history The pre-publication history for this paper can be accessed here: http://www.biomedcentral.com/1471-244X/7/66/pre pub Publish with Bio Med Central and every scientist can read your work free of charge "BioMed Central will be the most significant development for disseminating the results of biomedical researc h in our lifetime." Sir Paul Nurse, Cancer Research UK Your research papers will be: available free of charge to the entire biomedical community peer reviewed and published immediately upon acceptance cited in PubMed and archived on PubMed Central yours — you keep the copyright BioMedcentral Submit your manuscript here: http://www.biomedcentral.com/info/publishing_adv.asp Page 10 of 10 (page number not for citation purposes) http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png BMC Psychiatry Springer Journals

Item response analysis of the Positive and Negative Syndrome Scale

Loading next page...
 
/lp/springer-journals/item-response-analysis-of-the-positive-and-negative-syndrome-scale-aNifc8IfCH

References (27)

Publisher
Springer Journals
Copyright
Copyright © 2007 by Santor et al; licensee BioMed Central Ltd.
Subject
Medicine & Public Health; Psychiatry; Psychotherapy
eISSN
1471-244X
DOI
10.1186/1471-244X-7-66
pmid
18005449
Publisher site
See Article on Publisher Site

Abstract

Background: Statistical models based on item response theory were used to examine (a) the performance of individual Positive and Negative Syndrome Scale (PANSS) items and their options, (b) the effectiveness of various subscales to discriminate among individual differences in symptom severity, and (c) the appropriateness of cutoff scores recently recommended by Andreasen and her colleagues (2005) to establish symptom remission. Methods: Option characteristic curves were estimated using a nonparametric item response model to examine the probability of endorsing each of 7 options within each of 30 PANSS items as a function of standardized, overall symptom severity. Our data were baseline PANSS scores from 9205 patients with schizophrenia or schizoaffective disorder who were enrolled between 1995 and 2003 in either a large, naturalistic, observational study or else in 1 of 12 randomized, double-blind, clinical trials comparing olanzapine to other antipsychotic drugs. Results: Our analyses show that the majority of items forming the Positive and Negative subscales of the PANSS perform very well. We also identified key areas for improvement or revision in items and options within the General Psychopathology subscale. The Positive and Negative subscale scores are not only more discriminating of individual differences in symptom severity than the General Psychopathology subscale score, but are also more efficient on average than the 30-item total score. Of the 8 items recently recommended to establish symptom remission, 1 performed markedly different from the 7 others and should either be deleted or rescored requiring that patients achieve a lower score of 2 (rather than 3) to signal remission. Conclusion: This first item response analysis of the PANSS supports its sound psychometric properties; most PANSS items were either very good or good at assessing overall severity of illness. These analyses did identify some items which might be further improved for measuring individual severity differences or for defining remission thresholds. Findings also suggest that the Positive and Negative subscales are more sensitive to change than the PANSS total score and, thus, may constitute a "mini PANSS" that may be more reliable, require shorter administration and training time, and possibly reduce sample sizes needed for future research. Page 1 of 10 (page number not for citation purposes) BMC Psychiatry 2007, 7:66 http://www.biomedcentral.com/1471-244X/7/66 mining whether or not remission has been met. Alter- Background The Positive and Negative Syndrome Scale (PANSS) is the nately, if the threshold of "3 or less" corresponds to higher most widely used measure of symptom severity in schizo- severity for an item (i.e., is more easily achieved as symp- phrenia [1-3]. This 30-item scale is typically administered tom severity improves), then either that item is redundant by trained clinicians who evaluate patients' current sever- (since it is more likely to be reached first) or else its thresh- ity level on each symptom (item) by endorsing 1 of 7 old should be revised downward. For example, the remis- options (weights) numbered 1 through 7. The PANSS has sion threshold for that item could be set at 2 rather than demonstrated high internal reliability [4,5], good con- at 3. IRT analyses examine the manner in which individ- struct validity [4], and excellent sensitivity to change in ual item options (and cutoff scores) are related to overall both short term [6] and long term trials [7]. However, symptom severity which is central to evaluating the despite extensive psychometric research, it is unclear how appropriateness of proposed cut-point scores (thresholds) individual PANSS items differ in their usefulness in assess- determining illness remission. ing the severity of schizophrenia. The 3 primary purposes of our study are (a) to examine Indeed, studies examining the psychometric properties of and characterize the performance of individual items the PANSS have focused, without exception, on estimates from the PANSS at both the option (severity) and item of scale reliability, validity, and factor analysis using (symptom) levels with the goal of identifying areas for methods from classical test theory [8] and have typically improvement, (b) to examine the ability of various sub- identified 5 underlying factors [9-14]. Approaches based scales to discriminate among individual difference in ill- on classical test theory rely primarily on omnibus statistics ness severity, which might then identify a better measure that average over levels of individual variation and offer of change, and (c) to evaluate the appropriateness of items no means to gauge the quality of individual items or and options proposed for determining when symptoms options across different levels of symptom severity. In remission has been achieved. contrast, methods based on item response theory (IRT) [15] provide significant improvements over classical tech- Methods niques, as they model the relation between item responses Participants Data included baseline PANSS item scores from 9205 and symptom severity directly, quantifying how the per- formance of individual items and options (severity levels patients with schizophrenia, schizoaffective, or schizo- 1 to 7) change as a function of overall, standardized, phreniform disorders who were enrolled between 1995 symptom severity. IRT analyses provide unique and rele- and 2003 in either a large, naturalistic, observational vant information concerning (a) how well a set of item study or else in 1 of 12 randomized, double-blind clinical options assess the entire continuum of symptom severity, trials comparing olanzapine to other antipsychotic drugs. (b) whether weights assigned to individual item options Participants' diagnosis was based on DSM-IV criteria per are appropriate for measuring a particular trait or symp- treating physician and patient's medical record. Partici- tom, and (c) how well individual items or subscales are pants were primarily male (65%), could be either inpa- connected to the underlying construct and discriminate tients or outpatients, and had an average age of 39.0 (SD among individual differences in symptom severity (see = 11.5). Their initial PANSS total score averaged 81.9 (SD the publication by Santor and Ramsay [16] for an over- = 22.0) but ranged broadly from 30 to 177. view). Analytic models IRT methods are ideal for examining the performance of IRT consists of a broad class of statistical procedures [18] options within items that are to be used to define remis- used to model the association between responses to sur- sion of psychopathology. Andreasen and her colleagues vey items (in probabilistic terms) and an underlying [17] published guidelines recommending that schizo- latent trait, characteristic, or condition, such as overall phrenia remission be defined as achieving option scores symptom severity. We employed the same nonparametric less than or equal to 3 on each of only 8 PANSS items: kernel-smoothing techniques implemented in the soft- Delusions, Unusual Thought Content, Hallucinatory ware (TESTGRAF) developed by Ramsay [19,20] to esti- Behavior, Conceptual Disorganization, Mannerisms and mate option characteristic curves, which serve as a Posturing, Blunted Affect, Social Withdrawal, and Lack of measure of item and option effectiveness. These tech- Spontaneity. Setting equal remission thresholds (≤3) for niques have been used previously to examine the psycho- all 8 items suggests that the level of symptom severity cor- metric properties of self-report scales and to evaluate item responding to "3 or less" is generally equal for all 8 items. bias [21-23] and a detailed description of the algorithm If the region of symptom severity at which Options 1, 2, used to estimate response curves has been published else- and 3 are most likely to be endorsed differs across items, where [16,19]. then some items are more influential than others in deter- Page 2 of 10 (page number not for citation purposes) BMC Psychiatry 2007, 7:66 http://www.biomedcentral.com/1471-244X/7/66 To illustrate how item response models can be used to score of 37.3 in our sample, and a standard normal score evaluate item and option performance, we present a set of of +2 corresponds to a total scale score of 139.8. hypothetical, "ideal" option characteristic curves in Figure 1, where severity has been expressed in both expected A number of important features of item performance are total scores (upper x-axis) and standard normal scores illustrated in Figure 1. First, the region in which each (lower x-axis). Expected total scores show the level of option tends to be endorsed most frequently is clearly symptom severity at which different options are endorsed. indicated. For example, the curve for Option 2 suggests Standard normal quantiles, which are analogous to z- that the probability of being endorsed is essentially 0 at scores, show the proportion of the sample at different lev- standardized severity -3, increases to about 0.52 at -1, and els of expected total scores. then returns to roughly 0 at severity +1. Specifically, Option 2 is more likely to be endorsed than any other Expressing severity this way is useful in that standard nor- option within the overall severity range from -1.5 to -0.5. mal scores contain rather widely appreciated information about the proportion of a population above or below Second, each of the option characteristic curves ideally integer values of -3, -2, -1, 0, +1, +2, and +3. Extreme val- increases rather rapidly with small increases in severity. ues on our curves still need to be interpreted with caution, For example, the probability of Option 2 being endorsed because, in spite of the very large number of PANSS scores doubles from 0.2 to 0.4 when severity increases by just evaluated here, sample sizes are small in the tails of the half of a standard unit from -2.0 to -1.5. Third, the severity overall severity distribution but still express information regions over which each option is most likely to be about relative levels of symptom severity. We have also endorsed are ordered, left to right, in the same way as the superimposed corresponding total symptom scores to option scores (weights) themselves. That is, the region in facilitate the interpretation of standard normal scores. A which Option 2 is most likely to be endorsed (near -1), standard normal score of -2 corresponds to a total scale falls between the regions in which Option 1 (near -2) and Option 3 (near 0) are most likely to be endorsed. Fourth, together, the options for an item span the full con- tinuum of severity from -3 to +3. Some options are only endorsed at high levels of severity (e.g., Options 5 and 6), whereas others are endorsed at low levels of severity (e.g., Options 0 and 1). If the majority of options on an item are endorsed at low levels of severity, one might characterize that item as too "easy"; a low severity patient might receive a high score on such an item. In contrast, if the majority of options on an item are endorsed only at high levels of severity, that item might be characterized as too "hard"; a high severity patient could receive a low score on such an item. Scales comprised primarily of "hard" items will be largely insensitive to individual differences in the lower or moderate range of symptom severity and pro- duce floor effects. Scales comprised primarily of "easy" items will be largely insensitive to individual differences in the high range of symptom severity and produce ceiling effects. Option c Figure 1haracteristic curves for an ideal item Once option characteristic curves have been estimated, Option characteristic curves for an ideal item. The expected item scores and expected total and subscale probability of endorsing an individual option (y-axis) from scores are easily computed. One simply multiplies the this idealized item is plotted as a function of expected total smoothed estimate of probability of endorsing each scores (upper x-axis) and standard normal quantiles (lower option by its option weight and then sums these products x-axis). Expected total scores show the level of symptom across all options within each item. Due to variation in severity at which different options are endorsed. Standard endorsement of options within items, these expected normal quantiles, which are analogous to z-scores, show the scores are arguably better estimates of the true distribu- proportion of the sample at different levels of expected total tion of schizophrenia severity measured by the PANSS scores. Key features of this idealized item are described in than are the corresponding distributions of observed the main text. scores and sums. After all, expected item scores can be Page 3 of 10 (page number not for citation purposes) BMC Psychiatry 2007, 7:66 http://www.biomedcentral.com/1471-244X/7/66 computed at each point along the severity continuum independent raters who were able to reproduce a similar from -3 to +3. pattern of results. Using the idealized item portrayed in Figure 1, we judge Defining a continuum of symptom severity items as "very good" (a) if there was some range of severity Given that option characteristic curves depend on how in which the majority of options were more likely to be symptom severity is defined, we confirmed the appropri- endorsed than any other, (b) if option characteristic ateness of modeling of items via their subscale scores by curves increased rapidly with changes in overall severity, conducting principal components analyses on our sample (c) if the region in which each option was most likely to of 9205 responses on all 30 PANSS items using varimax be endorsed were ordered, left to right, in accordance with rotation and Kaiser Normalization prior to modeling. their option scores (weights), and (d) if options for an Although the PANSS was originally designed with 3 sub- item spanned the full continuum of severity from -3 to +3. scales (Positive, Negative, and General Psychopathology), Items were judged as "good" if they had many, but not all, recent studies examining the internal structure of the scale of these features and "poor" if they showed few, if any, of [9-14] have typically identified 5 underlying factors: (a) these criteria. We recognize that these ratings are global positive symptoms, (b) negative symptoms, (c) hostile assessments of 4 very different criteria. Global ratings of symptoms, (d) disorganized symptoms, and (e) symp- "very good", "good", and "poor" along with ratings for toms of anxiety and depression. Our results suggest a 6- each of these 4 criteria for each of the 30 PANSS items are factor solution, with the first 5 factors corresponding to recorded in Table 1. We provided the criteria to 2 blinded the solution identified by Davis and Chen [9]. Using both the "proportion equal to 1" and the "eigenvalue greater Table 1: Discrimination Effectiveness for the PANSS Items Item Content Summary Evaluation 3 Factor Subscales Criterion A Criterion B Criterion C Criterion D 1. Delusions Very good Positive 6 Yes Yes Yes 2. Conceptual Disorganization Very good Positive 6 Yes Yes Yes 3. Hallucinatory Behavior Very good Positive 4 Some Yes Yes 4. Excitement Good Positive 6 Yes Yes No 5. Grandiosity Good Positive 4 Some Yes Yes 6. Suspiciousness/Persecution Very good Positive 6 Yes Yes Yes 7. Hostility Poor Positive 6 Yes Yes No 8. Blunted Affect Very good Negative 6 Yes Yes Yes 9. Emotional Withdrawal Very good Negative 7 Yes Yes Yes 10. Poor Rapport Good Negative 6 Yes Yes Yes 11. Passive Apathetic Social Withdrawal Very good Negative 7 Yes Yes Yes 12. Difficulty in Abstract Thinking Poor Negative 6 Some Some Yes 13. Lack of Spontaneity Conversation Very good Negative 6 Yes Yes Yes 14. Stereotyped Thinking Poor Negative 6 No Some Yes 15. Somatic Concern Poor General 4 No Some No 16. Anxiety Good General 5 Yes Yes No 17. Guilt Feelings Good General 5 Some Yes No 18. Tension Good General 5 No Yes Yes 19. Mannerisms and Posturing Poor General 6 Some Yes Yes 20. Depression Poor General 5 Yes Yes No 21. Motor Retardation Good General 5 Yes Yes No 22. Uncooperative Good General 6 Yes Yes No 23. Unusual Thought Content Poor General 6 Yes Yes No 24. Disorientation Poor General 4 No No No 25. Poor Attention Poor General 5 Yes Yes Yes 26. Lack of Judgment and Insight Poor General 5 Some Some Yes 27. Disturbance of Volition Poor General 6 Yes Yes Yes 28. Poor Impulse Control Good General 6 Yes Yes No 29. Preoccupation Poor General 5 Some No Yes 30. Active Social Avoidance Very good General 6 Yes Yes Yes Criterion A: Number of options for which there was some range of symptom severity in which the question was more likely to be endorsed than all other options. Criterion B: Extent to which option characteristic curves increased rapidly with changes in overall severity. Criterion C: Does the severity region in which each option was most likely to be endorsed correspond to option weights? Criterion D: Do options for an item span the full continuum of severity from -3 to +3? Page 4 of 10 (page number not for citation purposes) BMC Psychiatry 2007, 7:66 http://www.biomedcentral.com/1471-244X/7/66 than or equal to 1" criteria, 2 of the 10 items assessing dis- organized symptoms, namely Items 12 (Difficulty in Abstract Thinking) and 24 (Disorientation), loaded greater than 0.30 on a sixth factor. Eigenvalues for the 6 factors were 4.56 (Negative Symptoms), 3.41 (Disorgan- ized Symptoms), 3.16 (Positive Symptoms), 2.83 (Hostil- ity Symptoms), 2.48 (Anxiety and Depression Symptoms), and 1.65 (Items 12 and 24). Given that the internal consistency coefficient for the 10 disorganized symptoms (including Items 12 and 24) was still good (Cronbach α = 0.83), we elected to retain a 5-factor solu- tion and to include Items 12 and 24 as part of the Disor- ganized Symptoms subscale, which was originally identified by Davis and Chen [9] and independently vali- dated by others [11,13,14,24]. Cronbach α's for the 5 subscales were (a) positive factor, 0.85, (b) negative factor, 0.86, (c) hostile factor, 0.78, (d) disorganized factor, 0.83, and (e) anxiety and depression factor, 0.70. However, we also examined the performance Opt fo Figure 2 r Iiton characteristic curves em 1: Delusions and expected item total score of items that comprise the original General Psychopathol- Option characteristic curves and expected item total ogy subscale when comparing the ability of subscales to score for Item 1: Delusions. Option characteristic curves discriminate among individual differences in illness sever- (solid lines) and expected item total score (dashed line) are ity. Internal consistency for the original General Psycho- plotted as a function of scores on the Positive Symptomatol- pathology subscale was also adequate, 0.88, suggesting ogy Subscale form the PANNS, expressed as standard nor- that any of the original or newly recommended subscales mal scores (lower x-axis) and expected total scores (upper x-axis). The probability of endorsing an option characteristic would be meaningful indicators of underlying symptom curve (solid lines) is scaled on the left y-axis and the severity against which individual items and options could expected item score (dashed line) is scaled on the right y- be modeled. axis. Many features of an ideal item are evident in this plot. Results Our results are grouped in 3 categories: (a) item response modeling of individual items and options from the more severe levels of Delusions begins to increase rapidly PANSS as a function of symptom severity, (b) item with increases in the severity of positive symptoms. response modeling of various subscales, and (c) item response modeling of cutoff scores used to establish Option 2 assessing a "minimal level of suspiciousness" is symptom remission. most frequently endorsed when severity reaches a stand- ard normal score of -1, which corresponds to an expected Examining individual items and options from the PANSS total score of about 19.0 on the Positive Factor subscale See Additional file 1 which presents option characteristic and a PANSS total of 60 and then quickly decreases with curves for all items. A select number of option characteris- further increases in severity. Other option characteristic tic curves for items from the various subscales are pre- curves on this item perform equally well. Note that 6 out sented here and discussed in detail, including overall of the 7 response options (all but 2) have a region where performance as well as proposed remission thresholds. it is more likely to be endorsed than any other option, and all 7 regions occur in the same order as their weights. For Delusions example, the region in which Option 3 of mild symp- Item 1, from the PANSS Positive Factor subscale, assesses tomatology (i.e., "Presents a guarded or even openly dis- suspiciousness, defined as "Beliefs which are unfounded, trustful attitude, but thoughts, interactions, and behavior unrealistic, and idiosyncratic." Option characteristic are minimally affected.") is most likely to be endorsed lies curves for this item are presented in the second panel of between the region in which Option 2, "Questionable Figure 2 and show that the probability of rating Option 1, pathology", and Option 4, "Moderate" levels of psychopa- assessing an "absence of symptoms," decreases rapidly as thology, are most likely to be endorsed. the severity of positive psychotic symptoms begins to increase. Meanwhile, the probability of endorsing the Examining the shape of the curves also provides useful information regarding the precision with which individu- Page 5 of 10 (page number not for citation purposes) BMC Psychiatry 2007, 7:66 http://www.biomedcentral.com/1471-244X/7/66 als can be designated as showing mild or moderate psy- Grandiosity chopathology. Results show that, despite well- Grandiosity, Item 5, is defined as "Exaggerated self-opin- differentiated peaks for Options 3 (<0) and 4 (>0) on ion and unrealistic convictions of superiority, including Item 1 (Delusions), the response characteristic curves for delusions of extraordinary abilities, wealth, knowledge, these options overlap considerably suggesting that desig- fame, power, and moral righteousness" [1]. Figure 4 nations made solely on the basis of this question could be depicts the option characteristic curves for this item as imprecise and unreliable. being generally flatter and less peaked than those of other items. Indeed, the probability of endorsing Options 2, 3, Hallucinatory behavior and 5 from Item 5 increase much slower than the corre- It is instructive to compare the option characteristic curves sponding options for Items 1 or 3. for Item 1, Delusions, in Figure 2 with those for Item 3, Hallucinatory Behavior, defined as "Verbal report [s] or Other important differences between Item 5 (Grandiosity, behavior indicating perceptions which are not generated Figure 4) and Items 1 (Delusions, Figure 2) or 3 (Halluci- by external stimuli" in Figure 3. Options characteristic natory Behavior, Figure 3) are evident when comparing curves for Item 3 are far less differentiated than those for expected item scores. The expected score is depicted in Item 1. Note that Option 4 of Item 3 entirely overlaps Options 2 and 3. Indeed, at all levels of severity, the prob- ability of endorsing Option 4 is equal to or greater than the probability of endorsing Options 2 or 3. This level of analysis allows scale developers to identify items and options that might be improved either through training of raters or revision of options/items. Opt for Item 5: Figure 4 ion characteristic curves Grandiosity and expected item total score Option characteristic curves and expected item total score for Item 5: Grandiosity. Option characteristic curves (solid lines) and expected item total score (dashed line) are plotted as a function of scores on the Positive Symp- tomatology Subscale form the PANNS, expressed as stand- ard normal scores (lower x-axis) and expected total scores (upper x-axis). Option characteristic curves (solid lines) indi- cate that a number of opportunities for improvement. Option c for It Figure 3 em 3 h:arac Hallucinatory behavior teristic curves and expected item total score Options 2, 3 and 4 overlap substantially as do Options 6 and Option characteristic curves and expected item total 7 suggesting rating these options is inherently difficult for score for Item 3: Hallucinatory behavior. Option char- raters. Option 1 in this item was most frequently endorsed acteristic curves (solid lines) and expected item total score by over half the sample, indicating that these options are only (dashed line) are plotted as a function of scores on the Posi- endorsed as a higher level of symptoms severity than in other tive Symptomatology Subscale form the PANNS, expressed options. The item characteristic curve (broken line) as standard normal scores (lower x-axis) and expected total increases smoothly in all but the lowest region of symptom scores (upper x-axis). Option characteristic curves (solid severity, but increases more slowly than the item character- lines) indicate that a number of opportunities for improve- istic curve in Item 1 or 3. Expected item scores for this item ment. Options 2, 3 and 4 overlap substantially as do Options are generally lower at comparable regions of symptom sever- 6 and 7 suggesting rating these options is inherently difficult ity than for Items 1 or 3. for raters. Page 6 of 10 (page number not for citation purposes) BMC Psychiatry 2007, 7:66 http://www.biomedcentral.com/1471-244X/7/66 each of our figures by a broken or dashed line that gener- ally increases smoothly within the severity range from -2 to +2. However, results show that the expected item score for Item 1 (Delusions) and Item 3 (Hallucinatory Behav- ior) increase much more between -2 and +2 than the expected score for Item 5 (Grandiosity). Indeed, the value of the expected item scores at the midpoint of the sample (0 standard normal quantiles) is 3.5 (right hand abscissa) for Item 1 (Delusions), 3.8 for Item 3 (Hallucinatory Behavior), and only 2.2 for Item 5 (Grandiosity). In this sense, Item 5 assessing Grandiosity is "harder" (much less discriminating) than Items 1 or 3. That is, a score of 3.5 on Item 5 (Grandiosity) is expected only when symptom severity exceeds +1 standard units, which translates to 103 on the Positive Symptom subscale or 36.1 on the entire 30-item PANSS. Mannerisms and posturing Item 19 from the PANSS assesses Mannerisms and Postur- ing, which is defined as "Unnatural movements or pos- Opt for Item 19: Ma Figure 5 ion characteristic curves nnerisms and postur and expec ing ted item total score ture characterized as awkward, stilted, disorganized, or Option characteristic curves and expected item total bizarre". Option characteristic curves for this item, Figure score for Item 19: Mannerisms and posturing. Option 5, tend to increase less quickly than those for Item 1 characteristic curves (solid lines) and expected item total (Delusions.) As a result, Options 2 through 6 of Item 19 score (dashed line) are plotted as a function of scores on the are likely to be endorsed at much higher severity levels General Psychopathology Subscale from the PANNS, than the corresponding options of Item 1. Item 19 (Man- expressed as standard normal scores (lower x-axis) and expected total scores (upper x-axis). Option characteristic nerisms and Posturing) could, therefore, be termed a curves (solid lines) overlap between Options 1 and 2 but "hard" item. generally well differentiated. However, options are generally only endorsed at more severe levels of symptomatology. Evaluating total and subscale performance Option 1 was most likely to be endorsed than any other The ability of various subscales to discriminate among lev- option, in over half the sample, namely with individuals scor- els of severity was examined by computing and plotting ing less than 81 on the 30-item PANSS. expected subscale totals for the 5 PANSS factor scales of Davis and Chen [9], as well as for the 30-item PANSS scale in total (TOT) and the original General Psychopathology subscale in Figure 6. Results suggest that the Positive and ity. Specifically, the response characteristic curves for Negative subscales are more discriminating than the Gen- endorsing Option 1, 2, or 3 were computed by summing eral Psychopathology subscale score or the PANSS total across individual option characteristic curves. Results score. That is, a 1-unit change in underlying symptom show that the probability of obtaining a score of 3 or less severity corresponds to a greater expected change in per does decrease for 7 of the 8 core symptoms. For the "easy" item score (second panel) for the Positive and Negative Item 19 (Mannerisms and Posturing), the probability of subscales than for the Full Scale or General Psychopathol- obtaining a score of 3 or less decreases more slowly over a ogy subscale. slightly wider range of symptom severity. This means that a person with a severity score of +2 has a probability of 0.5 Evaluating the appropriateness of symptom remission of scoring 3 or less on Item 19 (Mannerisms and Postur- cutoff scores ing). Meanwhile, the probability of scoring 3 or less on Andreasen and her colleagues [17] recently recommended any other item drops to less than 0.2 at a severity score of that "symptom remission" be defined as achieving scores +2 having declined from a probability of 0.5 achieved at a of 3 or less on each of 8 core symptoms: Delusions, Unu- much lower severity score (Figure 7). sual Thought Content, Hallucinatory Behavior, Concep- tual Disorganization, Mannerisms and Posturing, Blunted As a result, reaching remission criteria for Item 19 will be Affect, Social Withdrawal, and Lack of Spontaneity. To easier than for the other 7 symptoms and suggests that a examine the appropriateness of this common cut-point, threshold of 1 or 2 for Item 19 might be more appropriate. we modeled the probability of being rated 3 or less for each of these 8 symptoms as a function of symptom sever- Page 7 of 10 (page number not for citation purposes) BMC Psychiatry 2007, 7:66 http://www.biomedcentral.com/1471-244X/7/66 E Figure 6 xpected total scores for subscales from the PANSS Opt posturing Figure 7 ion characteristic curves for Item 19: Mannerisms and Expected total scores for subscales from the PANSS. Option characteristic curves for Item 19: Manner- Results show that the Positive and Negative subscales are isms and posturing. Option characteristic curves describ- more discriminating (i.e., have steeper slopes) than other ing the probability with ratings of 3 or less are made as a subscales demonstrating stronger discrimination on a per function of symptom severity for items recommended to item basis (top right panel) than the total PANSS Scale. define symptom remission in patients. Results show that the option characteristic curves for this item will be endorsed at a much more severe level of symptomatology than other items and is therefore redundant or should be revised. The Discussion broken line (---) shows the option characteristic curves for a Option and item performance revised Item 19, where remission is defined on the basis of a Results of our analyses, summarized in Table 1, confirm score less than or equal to 2 (rather than 3). that most PANSS items are either very good or good at assessing the overall severity, particularly items within the Negative Symptom subscale. However, a number of items, largely those from the original General Psychopathology Subscale performance subscale, performed far less well due to difficulties with Our results show that items from the Positive and Nega- some options. One of the few, consistent difficulties for a tive subscales are generally more discriminating of indi- large number of items is the overlap in response character- vidual differences in severity than any other of the istic curves for Options 1 and 2. In fact, there may be no subscales. In particular, items from the Positive and Neg- region of severity in which Option 2, indicating minimal ative subscales are more discriminating on average than symptoms, is most likely to be endorsed than Option 1, the full 30-item PANSS total and, thus, may be more sen- indicating an absence of symptoms. This result is not sitive to change than the PANSS total. These 2 subscales unexpected, because the definition of Option 2 in all such might possibly constitute a "mini PANSS" that would be items includes "can be at the extreme of normal." This more reliable, require shorter administration and training phrasing appears to create greater overlap between time, and might even reduce the sample size needed for Options 1 and 2 than between other adjacent options. future schizophrenia research. Defining remission criteria Second, our results also demonstrate overlap between a number of adjacent option characteristic curves. In partic- Our detailed analyses of the 8 items currently recom- ular, the Difficulty in Abstract Thinking, Hostility, Unco- mended as markers of remission reveals a problem with operativeness, and Hallucinatory Behavior items display using a common threshold of "3 or less." Specifically, overlap between all items suggesting that they are poorly Option 3 of Item 19 (Mannerisms and Posturing) may be differentiated. Third, results showed that many options endorsed at slightly higher levels of severity than any of within several "hard" items are endorsed only at much the other 7 items being considered. As a result, the thresh- higher levels of severity. For example, Option 1 of Grandi- old for remission on Item 19 might be more appropriately osity was endorsed for more than half of the patients. set at 1 or 2. Examining the detailed plots of all option Page 8 of 10 (page number not for citation purposes) BMC Psychiatry 2007, 7:66 http://www.biomedcentral.com/1471-244X/7/66 characteristic curves for items used to define remission Acknowledgements This work was sponsored by Eli Lilly and Company. The authors thank (see Additional file 1) shows that Option 3 on Item 19 is Noreen Pierle (Eli Lilly and Company, Indianapolis, Indiana) for editing this the only option most likely to be endorsed in range of manuscript. symptom severity >0 (or >81 on the total PANSS). Options 1, 2, and 3 for all other items are most likely to References be endorsed in a range of severity less than 81. 1. Kay SR, Fiszbein A, Opler LA: The positive and negative syn- drome scale (PANSS) for schizophrenia. Schizophr Bull 1987, 13:261-276. Conclusion 2. Van den Oord EJ, Rujescu D, Robles JR, Giegling I, Birrell C, Bukszár This first item response analysis (IRA) of the PANSS sup- J, Murrelle L, Möller HJ, Middleton L, Muglia P: Factor structure ports the overall sound psychometric properties of the and external validity of the PANSS revisited. Schizophr Res 2006, 82:213-223. Epub 2005 Oct 17. PANSS and demonstrates that most of its items are very 3. Lader M: Rating Scales in Schizophrenia: A Review of Their good or good at assessing the overall schizophrenia sever- Usefulness for Assessing Atypical Antipsychotics. CNS Drugs 2000, 14(1):23-32(10). ity. There were a number of items, primarily on the Gen- 4. Kay SR, Opler LA, Lindenmayer JP: Reliability and validity of the eral Psychopathology subscale, that might best be positive and negative syndrome scale for schizophrenics. Psy- modified and/or scored with fewer levels. The Positive chiatry Res 1988, 23:99-110. 5. Peralta V, Cuesta MJ: Psychometric properties of the positive and Negative subscales may also be more sensitive to and negative syndrome scale (PANSS) in schizophrenia. Psy- change than the PANSS total score. On the basis of these chiatry Res 1994, 53:31-40. results, one might consider rewriting item options so that 6. Lindenmayer JP, Kay SR, Friedman C: Negative and positive schiz- ophrenic syndromes after the acute phase: a prospective fol- (a) they are more sensitive to differences at low severity low-up. Compr Psychiatry 1986, 27:276-286. and/or (b) the higher level options are endorsed at lower 7. Kay SR, Fiszbein A, Lindenmayer JP, Opler L: Positive and negative syndromes in schizophrenia as a function of chronicity. Acta severity levels. On the other hand, the observed failure to Psychiatr Scand 1986, 74:507-518. discriminate among individuals in the very lowest regions 8. Nunnally JC: Psychometric Theory 2nd edition. New York: McGraw of severity, such as -4 to -3, might be due to relatively high Hill; 1978. 9. Davis JM, Chen N: The effects of olanzapine on the 5 dimen- severity in our sample. Our curves may not be clinically sions of schizophrenia derived by factor analysis: combined useful with low severity patients. Alternately, one might results of the North American and international trials. J Clin Psychiatry 2001, 62:757-71. modify rater training to change option endorsement 10. Lancon C, Aghababian V, Llorca PM, Auquier P: Factorial structure probabilities. Our results suggest that certain options are of the Positive and Negative Syndrome Scale (PANSS): a not currently being adequately differentiated. Interest- forced five-dimensional factor analysis. Acta Psychiatr Scand 1998, 98:369-376. ingly, one might estimate option characteristic curves at 11. Lindenmayer JP, Grochowski S, Hyman RB: Five factor model of different phases or types of training to evaluate the effec- schizophrenia: replication across samples. Schizophr Res 1995, tiveness of different types and lengths of training pro- 14:229-234. 12. Lykouras L, Oulis P, Psarros K, Daskalopoulou E, Botsis A, Christo- grams. doulou GN, Stefanis C: Five-factor model of schizophrenic psy- chopathology: how valid is it? Eur Arch Psychiatry Clin Neurosci 2000, 250:93-100. Competing interests 13. Marder SR, Davis JM, Chouinard G: The effects of risperidone on The author(s) declare that they have no competing inter- the five dimensions of schizophrenia derived by factor analy- ests. sis: combined results of the North American trials. J Clin Psy- chiatry 1997, 58:538-546. Erratum in: J Clin Psychiatry 1998, 59:200. 14. White L, Harvey PD, Opler L, Lindenmayer JP: Empirical assess- Authors' contributions ment of the factorial structure of clinical symptoms in schiz- DS performed the analyses, led the interpretation of the ophrenia. A multisite, multimodel evaluation of the factorial structure of the Positive and Negative Syndrome Scale. The results, and drafted the manuscript. HA-S conceived of the PANSS Study Group. Psychopathology 1997, 30:263-274. study, participated in its design, the analytical plan, the 15. Lord FM: Applications of Item Response Theory to Practical Testing Prob- lems Hillsdale, NJ: Erlbaum; 1980. interpretation of the results, and helped draft the manu- 16. Santor DA, Ramsay JO: Progress in the technology of measure- script. JPL and RO participated in the design of the study, ment: Applications of Item Response models. Psych Assess the analytical plan, the interpretation of the results, and 1998, 10:345-359. 17. Andreasen NC, Carpenter WT Jr, Kane JM, Lasser RA, Marder SR, assisted in drafting the manuscript. All authors have read Weinberger DR: Remission in schizophrenia: proposed criteria and approved the final manuscript. and rationale for consensus. Am J Psychiatry 2005, 162:441-449. 18. van der Linden WJ, Hambleton RK: Handbook of Modern Item Response Theory New York: Springer; 1997. Additional material 19. Ramsay JO: Kernel smoothing approaches to non parametric item curve estimation. Psychometrika 1991, 56:611-630. 20. Ramsay JO: TestGraf: A Program for the Graphical Analysis of Multiple Choice Test and Questionnaire Data. Montreal, Can- Additional file 1 ada: McGill University; 2001. Appendix. Option characteristic curves for all PANSS items. 21. Santor DA, Ramsay JO, Zuroff DC: Non-parametric item analy- Click here for file sis of the Beck Depression Inventory: Evaluating gender item [http://www.biomedcentral.com/content/supplementary/1471- bias and response option weights. Psychol Assess 1994, 244X-7-66-S1.ppt] 6:255-270. Page 9 of 10 (page number not for citation purposes) BMC Psychiatry 2007, 7:66 http://www.biomedcentral.com/1471-244X/7/66 22. Santor DA, Ramsay JO, Zuroff DC, Cervantes P, Palacios J: Examin- ing scale discriminability in the BDI and CES-D as a function of depressive severity. Psychol Assess 1995, 7:131-139. 23. Santor DA, Coyne JC: Examining symptom expression as a function of symptom severity: item performance on the Hamilton Rating Scale for Depression. Psychol Assess 2001, 13:127-139. 24. Bryson G, Bell M, Greig T, Kaplan E: Internal consistency, tempo- ral stability and neuropsychological correlates of three cog- nitive components of the Positive and Negative Syndrome Scale (PANSS). Schizophr Res 1999, 38:27-35. Pre-publication history The pre-publication history for this paper can be accessed here: http://www.biomedcentral.com/1471-244X/7/66/pre pub Publish with Bio Med Central and every scientist can read your work free of charge "BioMed Central will be the most significant development for disseminating the results of biomedical researc h in our lifetime." Sir Paul Nurse, Cancer Research UK Your research papers will be: available free of charge to the entire biomedical community peer reviewed and published immediately upon acceptance cited in PubMed and archived on PubMed Central yours — you keep the copyright BioMedcentral Submit your manuscript here: http://www.biomedcentral.com/info/publishing_adv.asp Page 10 of 10 (page number not for citation purposes)

Journal

BMC PsychiatrySpringer Journals

Published: Nov 15, 2007

There are no references for this article.