Get 20M+ Full-Text Papers For Less Than $1.50/day. Start a 14-Day Trial for You or Your Team.

Learn More →

Overcoming the Straw Man Effect in Oncology: Visualization and Ranking of Chemotherapy Regimens Using an Information Theoretic Approach

Overcoming the Straw Man Effect in Oncology: Visualization and Ranking of Chemotherapy Regimens... abstract original report Overcoming the Straw Man Effect in Oncology: Visualization and Ranking of Chemotherapy Regimens Using an Information Theoretic Approach Purpose Despite the plethora of randomized controlled trial (RCT) data, most cancer treatment recom- mendations are formulated by experts. Alternatively, network meta-analysis (NMA) is one method of analyzing multiple indirect treatment comparisons. However, NMA does not account for mixed end points or temporality. Previously, we described a prototype information theoretical approach for the construction of ranked chemotherapy treatment regimen networks. Here, we propose modifications to overcome an apparent straw man effect, where the most studied regimens were the most negatively valued. Methods RCTs from two scenarios—upfront treatment of chronic myelogenous leukemia and relapsed/ refractory multiple myeloma—were assembled into ranked networks using an automated algorithm based on effect sizes, statistical significance, surrogacy of end points, and time since RCT publication. Vertex and edge color, transparency, and size were used to visually analyze the network. This analysis led to the additional incorporation of value propagation. Results A total of 18 regimens with 42 connections (chronic myelogenous leukemia) and 28 regimens with 25 connections (relapsed/refractory multiple myeloma) were analyzed. An initial negative correlation between vertex value and size was ameliorated after value propagation, although not eliminated. Updated rankings were in close agreement with published guidelines and NMAs. Conclusion Straw man effects can distort the comparative efficacy of newer regimens at the expense of older regimens, which are often cheaper or less toxic. Using an automated method, we ameliorated this effect and producedrankingsconsistent with commonpractice and publishedguidelines in two distinct cancer settings. These findings are likely to be generalizable and suggest a new means of ranking efficacy in cancer trials. Clin Cancer Inform. © 2017 by American Society of Clinical Oncology Licensed under the Creative Commons Attribution 4.0 License Jeremy L. Warner Peter C. Yang INTRODUCTION shown that network meta-analysis applied to RCTs 2-7 Gil Alterovitz can yield powerful insights ; however, the networks Health care data can be highly convoluted, given in these studies have been relatively simple, do not the significant dimensionality, nonlinearity, and allow for mixed end points (eg, overall survival and Author affiliations and temporality present in most clinical contexts. In support information (if response rate), and do not account for temporal oncology, knowledge has been painstakingly built applicable) appear at the factors. In complex networks, layout, animation, over decades, primarily through carefully designed end of this article. and visual parameters such as size and color take randomized controlled trials (RCTs). RCT data, Corresponding author: on increasing importance. For example, visual Jeremy L. Warner, MD, which evolve longitudinally over years and usually MS, Vanderbilt University, analytics have been successfully applied to tem- involve many indirect comparisons, are known to be 2220 Pierce Ave Preston poral associations of laboratory results, phenotype Research Building 777, subject to many potential biases, ones that can be 1 relationship networks, and patterns of publication Nashville, TN 37232; difficult to discern. As a likely result of this complex- 9-11 e-mail: jeremy.warner@ by biomedical specialty and primary degree. ity, the conventional approach to the ranking and vanderbilt.edu. Visual analysis of networked RCT data may help recommendation of cancer treatments studied in Licensed under the Crea- uncover previously underappreciated biases. tive Commons Attribution RCTs has been expert consensus–driven guidelines 4.0 License. (eg, the National Comprehensive Cancer Network In previous work, we described a prototype ap- [NCCN]guidelines).Alternatively,workbyothershas proach for the automated construction of a ranked © 2017 by American Society of Clinical Oncology ascopubs.org/journal/cci JCO™ Clinical Cancer Informatics 1 chemotherapy treatment regimen network using Conversely, if the primary end point was not met information-theoretical techniques, which were but secondary end points were met, we still used the applied to the first-line treatment of chronic mye- primary end point for the valuation algorithm. We logenous leukemia (CML-1). Here, we demon- assigned a relative value (RV) as follows: 1.0 for strate how extension of the approach through strong, 0.8 for intermediate, and 0.7 for weak end additional information theoretical measures help points (Table 1; Equation 1). To determine the overcome the apparent presence of a straw man stability of the rankings, we varied RV by 65%, phenomenon. The straw man effect is a bias that 610%, and 620% in a sensitivity analysis. causes new studies to appear more promising Strength of evidence. In our pilot work, we used a because they are compared with regimens that simple win-lose-draw framework with win and lose are comparatively ineffective. Although this bias defined as a superior or inferior finding with a P has been described, the degree to which it per- value < .05, and draw defined as statistical non- vades clinical trial design is unknown. The objec- significance or formal noninferiority. Here, we tive of this paper is to present a new algorithm, built introduce a weighted entropy measure: the neg- on prior foundations, as well as to visually ana- ative logarithm of the P value. Because very small lyze this putative straw man phenomenon in the P values are difficult to interpret, this coefficient CML-1 scenario and a second scenario, the treat- is allowedtotakeamaximum valueof3(ie, ment of relapsed/refractory multiple myeloma P values ,.001 were truncated to .001). (RRMM). METHODS Effect size. We replaced the win-lose-draw frame- work with a coefficient representing the effect size Context-Specific Regimen Identification reported in the trial. For time-based outcomes (eg, The RCTs that were previously identified in the overall survival), we ideally used the hazard ratio context of CML-1 were also used in this study, 16 (HR) as the effect size. When HR was not along with several newly published RCTs. Briefly, reported, we defined the effect size either as the RCTs were identified through a PubMed query and ratio of the median survival times or as the point by hand searches of the literature and published estimate reported as significant in the publication guidelines. There were 27 RCTs identified between (eg, the 3-year event-free survival). For nontem- 1968 and 2016, with 18 distinct regimens, repre- poral measures (eg, response rate), we used the senting10,282patientsstudied(DataSupplement). calculated odds ratio as the effect size. In all cases, To identify RCTs for the context of RRMM, we used a the effect size . 1 was transformed into a co- combination of an established knowledge base of efficient E, which is positive for the winning side chemotherapy regimens, HemOnc.org, along and negative for the losing side (eg, if a publication with RCTs identified by two recent network meta- reports HR = 0.5, E = 2 for the winning side and 6,7 analyses in this setting. This yielded a total of E = 22 for the losing side). 25 RCTs published between 2004 and 2016 containing outcome information for 28 distinct Aging effects. To incorporate outdating of scientific regimens, representing 9,737 patients studied evidence, we introduced an exponential decay (Data Supplement). coefficient as a function of the time since publi- cation of trial results; additional details are in the Algorithm Modifications Data Supplement. The previous valuation algorithm, which was Vertex valuation algorithm. After incorporation of used for ranking as well as for coloration of verti- strength of evidence, effect size, and aging effects, ces, was revised to include strength of evidence, the empirical vertex valuation formula is as follows: effect sizes, aging effects, propagation, and re- fresh as explained in the following paragraphs. Table 1. End Points Used in the Examined CML-1 or RRMM Efficacy measure. For all trials, we selected the Trials, With Relative Value trial-defined primary end point, as described in the End Point Relative Value published manuscript, as the main efficacy mea- Overall survival 1.0 sure for the valuation algorithm. For trials with Progression-free survival 0.8 more than one predefined primary end point, we used the least-surrogate end point. If the primary Time to progression 0.8 end point was met, we used less-surrogate second- Overall response rate 0.7 ary end points in the algorithm if they had marginal Response rate 0.7 or better statistical significance (ie, P < .10). 2 ascopubs.org/journal/cci JCO™ Clinical Cancer Informatics Equation 1 Treatment Network Visualization We used multiple visual variables to display the v^ ¼ 2 log ðP Þ3 RV n 10 y y y¼1 treatment regimen networks: size, color, trans- 3 E 3 log ðN Þ3 f ða Þ parency, and position. See the Data Supplement y 10 y y for details. where for the n th vertex v, there are m incident edges, E is the effect size coefficient of the y th Statistical and General Methods edge, N is the total number of patients in each R version 3.4.0 and RStudio version 1.0.143 pairwise comparison, P is the P value of the y th (https://www.r-project.org/) were used for the cal- outcome, and f ða Þ is the aging coefficient de- culations. Graphs were created and displayed scribed in the previous paragraph. A positively using the igraph package version 1.0.1 (http:// valued vertex is considered recommendable, and igraph.org/r/); coloration was by the RColorBrewer a negatively valued vertex is considered contra- package. Correlations between vertex value and indicated. A vertex with value near zero is con- size before and after value propagation were cal- sideredtohavelackingevidence,conflictingresults, culated using the Pearson product-moment cor- or poor study quality such that there is insufficient relation;unadjustedPvalues, .05wereconsidered evidence on whether to recommend. Although the statistically significant. Animations of all networks valuation coefficient is unitless, the magnitude in- and the R code used to develop them are available formsthe powerofthe valuationand,thus,itisnot upon request. normalized. RESULTS Propagation and refresh. To overcome the appar- Visualization of the Treatment Regimen Networks ent straw man effect (discussed in Results), we The resultant networks for CML-1 and RRMM in investigated the introduction of indirect evidence the most recent year of analysis (2016), after in- propagation. In our pilot work, we did not assign corporation of evidence strength, effect size, and any node valuation on the basis of indirect evi- aging into the valuation algorithm, are shown in dence, such that the calculated network is akin Figures 1A and 2A; a complete list of regimens and to a single-layer perceptron (aka, pairwise network the number of patients studied for each are shown analysis). We augmented this model with informa- in the Data Supplement. On initial visualization tion propagation, which has been studied in the 17-19 of the CML-1 regimen network, a few things are context of social networks. Specifically, we immediately evident: (1) there are severe aging allow nodes that were calculated to lose value as a effects on regimens 1 through 8, with most of these result of newly introduced evidence to pass some being valued somewhere near zero; (2) the quality of their value loss to regimens to which they had of the outcome measure degrades over time, with previously been superior (ie, single-generation the newer regimens almost exclusively evaluated value propagation). Conversely, nodes that were with weak surrogate end points (blue edges); and calculated to gain value as a result of newly in- (3) the largest vertex, regimen 9 (imatinib), is also troduced evidence pass some of their value gain to the lowest ranked. Visualization of the RRMM regimens to which they had previously been in- network reveals that (1) aging effects only seem ferior. For example, in CML-1, dasatinib was dem- prominent for regimens 1 through 5; (2) outcome onstrated to be superior to imatinib ; however, measures are mostly intermediate surrogates (eg, imatinib had been shown 7 years earlier to be progression-free survival); and (3) the largest and superior to interferon a and low-dose cytarabine most connected regimens are the lowest ranked. (IFNA/LoDAC). Therefore, a portion of the value The visually apparent link among connectedness, loss assigned to the imatinib node is propagated size, and low valuation on the initial visual in- to the IFNA/LoDAC node. This has the effect of spection led us to suspect that straw man effects restoring some value to imatinib. When value is were present in both networks, but potentially propagated under these constraints, we also re- overstated. fresh the age-related devaluing coefficient by one half-life. This has the simultaneous effect of allow- Uncovering and Countering the Straw Man Effect ing more value to be propagated while also re- storing some relevance to the older regimen; this is When the vertices are plotted by vertex value analogous to decreasing impedance in an elec- versus size (ie, the total number of patients studied trical circuit, and is one possible solution to the under the regimen), the apparent tendency for problem of hindsight bias. See the Data Supple- large vertices to be negatively valued becomes ment for a more detailed, graphical description. more evident, as shown in Figure 3. In both ascopubs.org/journal/cci JCO™ Clinical Cancer Informatics 3 A B 2 2 3 3 Better Better 1 1 13. Nilotinib (42.6) 13. Nilotinib (42.6) 42.6 42.6 12. Imatinib-HD (18) 7 12. Imatinib-HD (18) 34.1 34.1 14. Dasatinib (6.5) 9. Imatinib (10.6) 25.5 25.5 10 10 14. Dasatinib (6.5) 5. IFNA (2.6) 17 8 5. IFNA (5.7) 11 7. Hydrea/lFNA (1.6) 8.5 8.5 7. Hydrea/lFNA (2.4) (tie) 6. Busulfan/lFNA (1.6) 0 6. Busulfan/lFNA (2.3) 18. Ponatinib (1) –14.3 –9.3 14 14 18. Ponatinib (1) 17. Bosutinib (0.4) –28.6 –18.6 12 9 12 17. Bosutinib (0.4) 18 3. DBM (0) –42.9 18 –28 9 3. DBM (0) (tie) 10. MRD allo-SCT (0) –57.2 –37.3 13 13 (tie) 10. MRD allo-SCT (0) (tie) 16. Imatinib/LoDAC (0) –71.5 –46.6 (tie) 16. Imatinib/LoDAC (0) (tie) 2. Radiation (0) 17 17 Worse Worse 16 16 11. IFNA/HiDAC (–1.4) 11. IFNA/HiDAC (–1.4) 2. Radiation (–2.4) 4. Hydrea (–1.6) 4. Hydrea (–3.9) 15. Imatinib/lFNA (–3.9) (tie) 15. Imatinib/lFNA (–3.9) 1. Busulfan (–4.9) 1. Busulfan (–6.3) 8. IFNA/LoDAC (–15.7) 8. IFNA/LoDAC (–71.5) 9. Imatinib (–46.6) Fig 1. Chemotherapy regimen network for first- contexts, this negative correlation was initially DISCUSSION line treatment of chronic statistically significant: For CML-1, the value of r The interpretation of complex networked data myelogenous leukemia-1 for the correlation of value and size was 20.52 (CML-1), through 2016. (A) benefits from computational approaches and vi- (95% CI, 20.80 to 20.07; P = .026). For RRMM, Initial valuations, before sualization of the results. In the examples dis- application of propagation the value of r for the correlation of value and size cussed here, multiple visual channels (ie, color, and refresh. The current was20.65 (95% CI, 20.82 to 20.37; P = .0002). transparency, size, position) provided an inte- standard of care for CML is Upon inclusion of propagation and refresh, the grated picture of context-specific treatment sce- the use of tyrosine kinase inhibitors (TKIs) in the valuation of some vertices changes dramatically, narios that evolved over many years (CML-1, 49 upfront setting. Consistent as shown in Figures 1B and 2B. In the CML-1 years; RRMM, 13 years). We were able to lever- with this, TKIs are highly network, imatinib moves from the lowest-ranked age human color perception through the use ranked, with the exception regimen to the regimen ranked third highest; of a divergent color scale, as compared with of imatinib, which is the IFNA/LoDAC inherits most of the negative value the rainbow color map often used in scientific lowest-ranked regimen. (B) Applying propagation and from imatinib and becomes the lowest-ranked visualizations. The human visual system is par- refresh to the network regimen. In the RRMM network, almost all aging ticularly well adapted for anomaly detection, ow- changes several valuations, effects disappear due to refresh, bortezomib and ing to enhanced perception of color, edges, and most notably imatinib. 28,29 lenalidomide -dexamethasone become more pos- outliers. As such, we were able to immediately DBM, dibromomannitol; itively valued, dexamethasone (Dex) becomes even recognize a potential anomaly in that the largest HD, high dose; HiDAC, high-dose cytarabine; more negatively valued, and pomalidomide- nodes in the CML-1 and RRMM networks seemed IFNA, interferon a; LoDAC, to be both highly connected (ie, compared with dexamethasone (Pom-Dex) moves from the fourth- low-dose cytarabine; MRD many other regimens) and negatively valued. This highest ranked regimen to the second-highest allo-SCT, matched related- ranked. With this adjustment, the correlation be- evidence from visual inspection led to further in- donor allogeneic stem-cell tween value and size changes and is no longer vestigation into a possible straw man effect, which transplant. significant; for CML-1, the value ofr for the correlation was initially supported by the existence of a sta- between value and size becomes 20.07 (95% tistically significant negative correlation between CI, 20.52 to 0.41; P = .78). For RRMM, the value vertex value and size for both contexts. Through of r for the correlation of value and size becomes the introduction of propagation and refresh into 20.33 (95% CI, 20.62 to 0.05; P = .09). our algorithm, we were able to ameliorate the straw man effect, although it was not eliminated entirely. Sensitivity Analysis Generally, the straw man effect is most evident With systematic variation in RV, the magnitudes of when new interventions are compared with clearly 1,30 the vertex values changed slightly, but the rank inferior regimens. A subtler version is the ten- order did not change for CML-1 or RRMM. Pos- dency to compare a new regimen with a compar- itively valued regimens remained positively valued atively effective regimen using a weaker surrogate 31-33 and vice versa. See the Data Supplement. end point, such as progression-free survival. It 4 ascopubs.org/journal/cci JCO™ Clinical Cancer Informatics 11 A B 4 5 4 5 24. Dara-Len-Dex (17.9) 24. Dara-Len-Dex (17.9) 23. Dara-Bort-Dex (16.6) 10 13. Pom-Dex (17.4) 20. Elo-Len-Dex (12.3) 2 23. Dara-Bort-Dex (16.6) 12 19. Car-Dex (11.8) 12 9 20. Elo-Len-Dex (12.3) 19 19 15 (tie) 13. Pom-Dex (11.8) Better Better 15 19. Car-Dex (11.8) 15. Bort-Dex-Panob (11) 15. Bort-Dex-Panob (11) 17.9 17.9 16. Car-Len-Dex (10.4) 16. Car-Len-Dex (10.4) 14.3 6. Bort-Doxil (7.9) 6 14.3 1 11 6. Bort-Doxil (7.9) 1 11 25. lxaz-Len-Dex (6.2) 10.7 10.7 25. lxaz-Len-Dex (6.2) 10. Bort-Thal-Dex (6) 10. Bort-Thal-Dex (6) 7.1 7.1 27. Pom-Cy-Dex (5.3) 27. Pom-Cy-Dex (5.3) 23 23 3.6 12. Bort-Vorin (4) 3.6 1. Bort (4 .7) 26. lxaz-Dex-HD (3.5) 0 14 12. Bort-Vorin (4) 22. Elo-Bort-Dex (2.5) 3 26. lxaz-Dex-HD (3.5) –8.4 –14.6 18. Bort-Siltux (0.9) 22. Elo-Bort-Dex (2.5) –16.8 27 4. Len (twice-daily) (0.5) –29.2 27 18. Bort-Siltux (0.9) 13 25 8. Bort-SC (0) –25.2 –43.8 4. Len (twice-daily) (0.5) (tie) 14. Pom-Dex (28/28) (0) 14. Pom-Dex (28/28) (–0.3) –33.6 7 –58.4 2. Bort-LD (–0.3) 7 5. Len (–0.5) –42 5. Len (–0.5) –72.9 8. Bort-SC (–0.6) 28. lxaz-Dex (–3.5) (tie) 7. Len-Dex (–0.6) Worse Worse 16 21. Elo-Len-Dex-HD (–3.8) 20 20 2. Bort-LD (–1.1) 17. Pom (–5.4) 28. lxaz-Dex (–3.5) 9. Thal-Dex (–6) 21. Elo-Len-Dex-HD (–3.8) 1. Bort (–8.6) 9. Thal-Dex (–6) 3. Dex (–25.3) 17. Pom (–7.7) 7. Len-Dex (–33.4) 11. Bort-Dex (–42) 11. Bort-Dex (–42) 3. Dex (–72.9) 26 28 26 28 Fig 2. Chemotherapy has been suggested that pharmaceutical industry many reasons, including (1) all regimens prospec- regimen network for support, along with reluctance to sponsor head-to- tively evaluated against imatinib subsequent to the treatment of relapsed/ refractory multiple head comparisons of drugs manufactured by per- IRIS (International Randomized Study of Interferon 34,35 21 myeloma, through 2016. and STI571) trial have either been neutral or ceivedcompetitors,mayexacerbatesucheffects. (A) Initial valuations, before superior to imatinib; (2) surrogate end points have Two examples where the straw man effect may be application of propagation been substituted extensively; and (3) imatinib is operational in cancer domains other than those and refresh. The current now generic and cost may be lower than treatments standard of care is doublet examined here are dacarbazine in melanoma, or triplet therapy using an still under patent protection. Given these chal- where at least nine RCTs demonstrating inferiority immunomodulatory drug or 36 lenges, it is notable that our updated algorithm still have been published between 2000 and 2017 ; proteasome inhibitor ranks imatinib highly. In the final RRMM network and docetaxelinnon–small-cell lung cancer,where backbone. This is reflected (Fig 2B), the algorithm ranks DRd as the highest at least eight RCTs demonstrating inferiority have in the ranking, although and Dex as the lowest ranked regimen, in agree- Bort-Dex and Len-Dex are been published between 2014 and 2017. The the lowest-ranked ment with the analyses by Botta et al and van straw man effect is particularly hard to identify regimens. (B) Applying Beurden-Tan et al. Interestingly, our algorithm directly from the medical literature because the propagation and refresh to ranks a doublet, Pom-Dex, as second-highest, just design and execution of RCTs may precede their the network changes behind Dara-Len-Dex. This is primarily due to the several valuations; most publication by years. Also, many contexts have the notably, Dex is much more refresh of Dex, which allows for substantial negative fortunate situation in which prognosis is improving negatively valued. Bort, 38,39 value to be propagated from Pom-Dex to Dex. The as a result of treatment, including both CML-1 bortezomib; Car, 40 validity of this finding is supported by the fact that and RRMM ; this will naturally lead to the need to carfilzomib; Cy, Pom-Dex has a category 1 recommendation by cyclophosphamide; Dara, substitute surrogate end points so that new RCTs the NCCN, although it is noted by the NCCN daratumumab; Dex, can be completed within a reasonable time. An dexamethasone; Elo, that triplet regimens are preferred to doublet intriguing possible way to mitigate the biases of RCT elotuzumab; HD, high regimens except in frail or elderly patients. design is the use of a treatment of physician’s dose; Ixaz, ixazomib; Notably, the only triplet compared with Pom-Dex, LD, low dose; Len, choice control arm, which was used in the recent pomalidomide-cyclophosphamide-dexamethasone, 41,42 lenalidomide; Panob, (negative) CheckMate 026 trial. was in a randomized phase II trial with a weak panobinostat; Pom, Any algorithmic ranking algorithm must be judged pomalidomide; SC, surrogate primary end point (overall response rate) subcutaneous; Siltux, on face validity. In the final CML-1 network (Fig 1B), and borderline significance (P = .035). Given that siltuximab; Thal, 47,48 the algorithm ranks nilotinib, imatinib (standard Pom-Dex is relatively well tolerated, and that thalidomide; Vorin, and high dose), and dasatinib the highest, in close toxicities are a serious concern for modern mye- vorinostat. 43 49 concordance with NCCN guidelines. Imatinib, in loma drugs, our findings may have real-world particular, is the subject of ongoing contention for significance. ascopubs.org/journal/cci JCO™ Clinical Cancer Informatics 5 Fig 3. Scatterplots of A B vertex value versus size for CML - 1 CML - 1 CML-1 and RRMM. Linear regression lines using the r = –0.07, P = .782 r = –0.52, P = .026 least squares approach are overlaid. For all panels, the results show that value is inversely correlated with size, although the –20 correlation becomes nonsignificant once –20 –40 propagation and refresh are instituted. Proportionate –60 –40 size and transparency are preserved from the –80 treatment network 0 500 1,000 1,500 2,000 2,500 0 500 1,000 1,500 2,000 2,500 visualization; numbering of regimens is omitted to Size Size improve clarity. CML-1, (No. of patients studied on the regimen) (No. of patients studied on the regimen) chronic myelogenous leukemia; RRMM, relapsed/refractory multiple myeloma. C D RRMM RRMM r = –0.33, P = .09 r = –0.65, P ≤ .01 –20 –10 –20 –40 –30 –60 –40 –80 –50 0 500 1,000 1,500 0 500 1,000 1,500 Size Size (No. of patients studied on the regimen) (No. of patients studied on the regimen) There are several limitations to our current ap- as SAMPL to encourage uniform reporting of proach. First and foremost, significant methodo- HRs. Current work is focused (as future work logical challenges remain in the field of dynamic will be) on the use of ensemble methods to eval- network analysis, especially with respect to in- uate how rankings change with perturbation of the 50-52 study and between-study effect modifiers. weighting coefficients, with a goal of choosing a The vertex valuation algorithm contains several consensus model that represents best fit. Be- empirically derived coefficients and, therefore, yond this methodological limitation, the measured could be subject to unmeasured bias. However, valuations may also be subject to positive publi- it is notable that the ASCO Value Framework has cation bias (ie, RCTs that demonstrate a statis- adopted similar weighting metrics to those we use tically significant result are more likely to be 53,54 59 for the surrogacy of end points. The ASCO published). We have tried to ameliorate the Value Framework and other approaches to com- known tendency for positive publication bias by parative valuation, such at the NCCN Evidence including so-called gray literature when possible. Blocks, are also empirically derived. We have Future work will investigate incorporating informa- used several definitions for effect sizes, including tion from clinical trial registries (eg, ClinicalTrials. using point estimates when HRs are not reported. gov) so that unpublished trials might be incorpo- Older publications are more likely to report point rated into the model. It is also possible that sup- estimates rather than HRs, so this could plementation with comparative effectiveness data introduce a time-based systematic bias. In rec- may change the valuations and subsequent rank- ognition of this limitation, we support efforts such ings.Wewillalsoinvestigatemoregranulardefinitions 6 ascopubs.org/journal/cci JCO™ Clinical Cancer Informatics Value Value Value Value of time of publication and incorporate evidence appeared to be the manifestation of the straw man updates into the valuation algorithm, as opposed effect.Additionalmodifications tothealgorithmledto to the current method of using interim updates hierarchical rankings with face validity for both sce- only to modify the aging coefficient. narios.Thesefindingsare likely tobe generalizable to any cancer setting with multiple indirect compari- In conclusion, we have described how the creation sons, and they suggest a new means of ranking and visual inspection of chemotherapy treatment efficacy in cancer trials. regimen networks can rapidly lead to new insights. In both of the described use cases, the visual vari- DOI: https://doi.org/10.1200/CCI.17.00079 ables of color, transparency, size, and position led to Published online on ascopubs.org/journal/cci on November 15, an almost immediate recognition of an anomaly that 2017. AUTHOR CONTRIBUTIONS Relationships may not relate to the subject matter of this manuscript. For more information about ASCO’s conflict of Conception and design: Jeremy L. Warner, Gil Alterovitz interest policy, please refer to www.asco.org/rwc or ascopubs. Collection and assembly of data: Jeremy L. Warner, Peter C. org/jco/site/ifc. Yang Data analysis and interpretation: Jeremy L. Warner, Peter C. Jeremy L. Warner Yang Stock and Other Ownership Interests: HemOnc.org LLC Manuscript writing: All authors Peter C. Yang Final approval of manuscript: All authors Stock and Other Ownership Interests: Merck, Pfizer, Cyclacel, Accountable for all aspects of the work: All authors HemOnc.org LLC Gil Alterovitz AUTHORS’ DISCLOSURES OF POTENTIAL CONFLICTS OF INTEREST No relationship to disclose The following represents disclosure information provided by authors of this manuscript. All relationships are considered ACKNOWLEDGMENT compensated. Relationships are self-held unless noted. I = We give thanks to Thomas A. Lasko, MD, PhD, for his advice on Immediate Family Member, Inst = My Institution. network meta-analysis. Affiliations Jeremy L. Warner, Vanderbilt University, Nashville, TN; Peter C. Yang, Massachusetts General Hospital; and Gil Alterovitz, Harvard Medical School and Harvard-Massachusetts Institute of Technology Division of Health Science, Boston; and Massachusetts Institute of Technology, Cambridge, MA. Support Supported in part by Grants No. R21DA025168 from the National Institute of Drug Abuse, R01HG004836 from the National Human Genome Research Institute, R00LM009826 from the National Library of Medicine, and L30CA171123 from the National Cancer Institute. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. REFERENCES 1. Ioannidis JP: Perfect study, poor evidence: Interpretation of biases preceding study design. Semin Hematol 45:160- 166, 2008 2. Baldwin D, Woods R, Lawson R, et al: Efficacy of drug treatments for generalised anxiety disorder: Systematic review and meta-analysis. BMJ 342:d1199, 2011 3. Liao WC, Chien KL, Lin YL, et al: Adjuvant treatments for resected pancreatic adenocarcinoma: A systematic review and network meta-analysis. Lancet Oncol 14:1095-1103, 2013 4. Elliott WJ, Meyer PM: Incident diabetes in clinical trials of antihypertensive drugs: A network meta-analysis. Lancet 369:201-207, 2007 5. Stettler C, Wandel S, Allemann S, et al: Outcomes associated with drug-eluting and bare-metal stents: A collaborative network meta-analysis. Lancet 370:937-948, 2007 6. Botta C, Ciliberto D, Rossi M, et al: Network meta-analysis of randomized trials in multiple myeloma: Efficacy and safety in relapsed/refractory patients. Blood Adv 1:455-466, 2017 7. van Beurden-Tan CHY, Franken MG, Blommestein HM, et al: Systematic literature review and network meta-analysis of treatment outcomes in relapsed and/or refractory multiple myeloma. J Clin Oncol 35:1312-1319, 2017 ascopubs.org/journal/cci JCO™ Clinical Cancer Informatics 7 8. Chen C: An information-theoretic view of visual analytics. IEEE Comput Graph Appl 28:18-23, 2008 9. Klimov D, Shahar Y, Taieb-Maimon M: Intelligent visualization and exploration of time-oriented data of multiple patients. Artif Intell Med 49:11-31, 2010 10. Hidalgo CA, Blumm N, Barabasi AL, et al: A dynamic network approach for the study of human phenotypes. PLOS Comput Biol 5:e1000353, 2009 11. Weber GM: Identifying translational science within the triangle of biomedicine. J Transl Med 11:126, 2013 12. Warner J, Yang P, Alterovitz G: Automated synthesis and visualization of a chemotherapy treatment regimen network. Stud Health Technol Inform 192:62-66, 2013 13. Warner JL, Cowan AJ, Hall AC, et al: HemOnc.org: A collaborative online knowledge platform for oncology pro- fessionals. J Oncol Pract 11:e336-e350, 2015 14. Ellis LM, Bernstein DS, Voest EE, et al: American Society of Clinical Oncology perspective: Raising the bar for clinical trials by defining clinically meaningful outcomes. J Clin Oncol 32:1277-1280, 2014 15. Wangge G, Klungel OH, Roes KC, et al: Interpretation and inference in noninferiority randomized controlled trials in drug research. Clin Pharmacol Ther 88:420-423, 2010 16. Spruance SL, Reid JE, Grace M, et al: Hazard ratio in clinical trials. Antimicrob Agents Chemother 48:2787-2792, 17. Bakshy E, Rosenn I, Marlow C, et al: The role of social networks in information diffusion. Presented at the 21st International Conference on World Wide Web, Lyon, France, April 16-20, 2012. 18. Cha M: Propagation phenomena in large social networks. Presented at the 23rd International Conference on World Wide Web, Seoul, Korea, April 7-11, 2014. 19. Han S, Zhuang FZ, He Q, et al: Energy model for rumor propagation on social networks. PHYSICA A 394:99-109, 2014 20. Kantarjian H, Shah NP, Hochhaus A, et al: Dasatinib versus imatinib in newly diagnosed chronic-phase chronic myeloid leukemia. N Engl J Med 362:2260-2270, 2010 21. O’Brien SG, Guilhot F, Larson RA, et al: Imatinib compared with interferon and low-dose cytarabine for newly di- agnosed chronic-phase chronic myeloid leukemia. N Engl J Med 348:994-1004, 2003 22. Hoffrage U, Hertwig R, Gigerenzer G: Hindsight bias: A by-product of knowledge updating? J Exp Psychol Learn Mem Cogn 26:566-581, 2000 23. Bertin J: La graphique. Communications 15:169-185, 1970 24. Csardi G, Nepusz T: The igraph software package for complex network research. InterJournal Complex Systems 1695, 25. Neuwirth E: RColorBrewer: ColorBrewer palettes, 2011. R Package Version 1.0-5. https://cran.r-project.org/web/ packages/RColorBrewer/index.html 26. Moreland K: Diverging color maps for scientific visualization. Presented at the 5th International Symposium on Advances in Visual Computing: Part II, Las Vegas, Nevada, November 30-December 2, 2009. 27. Borland D, Taylor MR II: Rainbow color map (still) considered harmful. IEEE Comput Graph Appl 27:14-17, 2007 28. Marr D, Hildreth E: Theory of edge detection. Proc R Soc Lond B Biol Sci 207:187-217, 1980 29. McCollough C: Color adaptation of edge-detectors in the human visual system. Science 149:1115-1116, 1965 30. Prasad V, Berger VW: Hard-wired bias: How even double-blind, randomized controlled trials can be skewed from the start. Mayo Clin Proc 90:1171-1175, 2015 31. Fleming TR, DeMets DL: Surrogate end points in clinical trials: Are we being misled? Ann Intern Med 125:605-613, 32. Booth CM, Eisenhauer EA: Progression-free survival: Meaningful or simply measurable? J Clin Oncol 30:1030-1033, 2012 33. Prasad V, Kim C, Burotto M, et al: The strength of association between surrogate end points and survival in oncology: A systematic review of trial-level meta-analyses. JAMA Intern Med 175:1389-1398, 2015 34. Peppercorn J, Blood E, Winer E, et al: Association between pharmaceutical involvement and outcomes in breast cancer clinical trials. Cancer 109:1239-1246, 2007 35. Lathyris DN, Patsopoulos NA, Salanti G, et al: Industry sponsorship and selection of comparators in randomized clinical trials. Eur J Clin Invest 40:172-182, 2010 36. Dacarbazine monotherapy. HemOnc.org. https://hemonc.org/wiki/Melanoma#Dacarbazine_monotherapy 37. Carrboplatin & paclitaxel. HemOnc.org. https://hemonc.org/wiki/Non-small_cell_lung_cancer#Carboplatin_.26_Paclitaxel_3 38. Kantarjian H, O’Brien S, Jabbour E, et al: Improved survival in chronic myeloid leukemia since the introduction of imatinib therapy: A single-institution historical experience. Blood 119:1981-1987, 2012 39. Bower H, Bjorkholm ¨ M, Dickman PW, et al: Life expectancy of patients with chronic myeloid leukemia approaches the life expectancy of the general population. J Clin Oncol 34:2851-2857, 2016 8 ascopubs.org/journal/cci JCO™ Clinical Cancer Informatics 40. Rajkumar SV, Kumar S: Multiple myeloma: Diagnosis and treatment. Mayo Clin Proc 91:101-119, 2016 41. Twelves C, Jove M, Cortes J: Recognizing the place of trials with treatment of physician’s choice as the control arm. J Clin Oncol 33:1300-1301, 2015 42. Carbone DP, Reck M, Paz-Ares L, et al: First-line nivolumab in stage IV or recurrent non-small-cell lung cancer. N Engl J Med 376:2415-2426, 2017 43. National Comprehensive Cancer Network: NCCN Clinical Practice Guidelines in Oncology: Chronic Myelogenous Leukemia, Version 1.2017, 2017. https://www.nccn.org/professionals/physician_gls/pdf/cml.pdf 44. Agrawal M, Garg RJ, Kantarjian H, et al: Chronic myeloid leukemia in the tyrosine kinase inhibitor era: What is the “best” therapy? Curr Oncol Rep 12:302-313, 2010 45. National Comprehensive Cancer Network: NCCN Clinical Practice Guidelines in Oncology: Multiple Myeloma Version 3.2017, 2017. https://www.nccn.org/professionals/physician_gls/pdf/myeloma.pdf 46. Baz RC, Martin TG III, Lin H-Y, et al: Randomized multicenter phase 2 study of pomalidomide, cyclophosphamide, and dexamethasone in relapsed refractory myeloma. Blood 127:2561-2568, 2016 47. Dimopoulos MA, Palumbo A, Corradini P, et al: Safety and efficacy of pomalidomide plus low-dose dexamethasone in STRATUS (MM-010): A phase 3b study in refractory multiple myeloma. Blood 128:497-503, 2016 48. Conticello C, Parisi M, Romano A, et al: Pomalidomide experience: An effective therapeutic approach with immu- nomodulatory drugs in a patient with relapsed-refractory multiple myeloma. Future Oncol 13(5s):3-6, 2017 49. Delforge M, Ludwig H: How I manage the toxicities of myeloma drugs. Blood 129:2359-2367, 2017 50. Li T, Puhan MA, Vedula SS, et al: Network meta-analysis-highly attractive but more methodological research is needed. BMC Med 9:79, 2011 51. Mills EJ, Ioannidis JP, Thorlund K, et al: How to use an article reporting a multiple treatment comparison meta-analysis. JAMA 308:1246-1253, 2012 52. Jansen JP, Naci H: Is network meta-analysis as valid as standard pairwise meta-analysis? It all depends on the distribution of effect modifiers. BMC Med 11:159, 2013 53. Schnipper LE, Davidson NE, Wollins DS, et al: American Society of Clinical Oncology statement: A conceptual framework to assess the value of cancer treatment options. J Clin Oncol 33:2563-2577, 2015 54. Schnipper LE, Davidson NE, Wollins DS, et al: Updating the American Society of Clinical Oncology Value Framework: Revisions and reflections in response to comments received. J Clin Oncol 34:2925-2934, 2016. 55. National Comprehensive Cancer Network: NCCN clinical practice guidelines in oncology with NCCN evidence blocks. https://www.nccn.org/evidenceblocks/. 56. Tierney JF, Stewart LA, Ghersi D, et al: Practical methods for incorporating summary time-to-event data into meta- analysis. Trials 8:16, 2007 57. Equator Network: SAMPL guidelines for statistical reporting. http://www.equator-network.org/2013/02/11/sampl- guidelines-for-statistical-reporting/ 58. Dietterich TG: Ensemble methods in machine learning, in Kittler J, Roli, F (eds.): Multiple Classifier Systems. First International Workshop, MCS 2000 Cagliari, Italy, June 21-23, 2000 Proceedings. Berlin, Germany, Springer 2000: 1-15. 59. Chan AW, Hrobjartsson A, Haahr MT, et al: Empirical evidence for selective reporting of outcomes in randomized trials: Comparison of protocols to published articles. JAMA 291:2457-2465, 2004 ascopubs.org/journal/cci JCO™ Clinical Cancer Informatics 9 http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png JCO Clinical Cancer Informatics Wolters Kluwer Health

Overcoming the Straw Man Effect in Oncology: Visualization and Ranking of Chemotherapy Regimens Using an Information Theoretic Approach

Loading next page...
 
/lp/wolters-kluwer-health/overcoming-the-straw-man-effect-in-oncology-visualization-and-ranking-q5b42pRds7

References

References for this paper are not available at this time. We will be adding them shortly, thank you for your patience.

Publisher
Wolters Kluwer Health
Copyright
(C) 2017 by Lippincott Williams & Wilkins, Inc.
ISSN
2473-4276
DOI
10.1200/CCI.17.00079
Publisher site
See Article on Publisher Site

Abstract

abstract original report Overcoming the Straw Man Effect in Oncology: Visualization and Ranking of Chemotherapy Regimens Using an Information Theoretic Approach Purpose Despite the plethora of randomized controlled trial (RCT) data, most cancer treatment recom- mendations are formulated by experts. Alternatively, network meta-analysis (NMA) is one method of analyzing multiple indirect treatment comparisons. However, NMA does not account for mixed end points or temporality. Previously, we described a prototype information theoretical approach for the construction of ranked chemotherapy treatment regimen networks. Here, we propose modifications to overcome an apparent straw man effect, where the most studied regimens were the most negatively valued. Methods RCTs from two scenarios—upfront treatment of chronic myelogenous leukemia and relapsed/ refractory multiple myeloma—were assembled into ranked networks using an automated algorithm based on effect sizes, statistical significance, surrogacy of end points, and time since RCT publication. Vertex and edge color, transparency, and size were used to visually analyze the network. This analysis led to the additional incorporation of value propagation. Results A total of 18 regimens with 42 connections (chronic myelogenous leukemia) and 28 regimens with 25 connections (relapsed/refractory multiple myeloma) were analyzed. An initial negative correlation between vertex value and size was ameliorated after value propagation, although not eliminated. Updated rankings were in close agreement with published guidelines and NMAs. Conclusion Straw man effects can distort the comparative efficacy of newer regimens at the expense of older regimens, which are often cheaper or less toxic. Using an automated method, we ameliorated this effect and producedrankingsconsistent with commonpractice and publishedguidelines in two distinct cancer settings. These findings are likely to be generalizable and suggest a new means of ranking efficacy in cancer trials. Clin Cancer Inform. © 2017 by American Society of Clinical Oncology Licensed under the Creative Commons Attribution 4.0 License Jeremy L. Warner Peter C. Yang INTRODUCTION shown that network meta-analysis applied to RCTs 2-7 Gil Alterovitz can yield powerful insights ; however, the networks Health care data can be highly convoluted, given in these studies have been relatively simple, do not the significant dimensionality, nonlinearity, and allow for mixed end points (eg, overall survival and Author affiliations and temporality present in most clinical contexts. In support information (if response rate), and do not account for temporal oncology, knowledge has been painstakingly built applicable) appear at the factors. In complex networks, layout, animation, over decades, primarily through carefully designed end of this article. and visual parameters such as size and color take randomized controlled trials (RCTs). RCT data, Corresponding author: on increasing importance. For example, visual Jeremy L. Warner, MD, which evolve longitudinally over years and usually MS, Vanderbilt University, analytics have been successfully applied to tem- involve many indirect comparisons, are known to be 2220 Pierce Ave Preston poral associations of laboratory results, phenotype Research Building 777, subject to many potential biases, ones that can be 1 relationship networks, and patterns of publication Nashville, TN 37232; difficult to discern. As a likely result of this complex- 9-11 e-mail: jeremy.warner@ by biomedical specialty and primary degree. ity, the conventional approach to the ranking and vanderbilt.edu. Visual analysis of networked RCT data may help recommendation of cancer treatments studied in Licensed under the Crea- uncover previously underappreciated biases. tive Commons Attribution RCTs has been expert consensus–driven guidelines 4.0 License. (eg, the National Comprehensive Cancer Network In previous work, we described a prototype ap- [NCCN]guidelines).Alternatively,workbyothershas proach for the automated construction of a ranked © 2017 by American Society of Clinical Oncology ascopubs.org/journal/cci JCO™ Clinical Cancer Informatics 1 chemotherapy treatment regimen network using Conversely, if the primary end point was not met information-theoretical techniques, which were but secondary end points were met, we still used the applied to the first-line treatment of chronic mye- primary end point for the valuation algorithm. We logenous leukemia (CML-1). Here, we demon- assigned a relative value (RV) as follows: 1.0 for strate how extension of the approach through strong, 0.8 for intermediate, and 0.7 for weak end additional information theoretical measures help points (Table 1; Equation 1). To determine the overcome the apparent presence of a straw man stability of the rankings, we varied RV by 65%, phenomenon. The straw man effect is a bias that 610%, and 620% in a sensitivity analysis. causes new studies to appear more promising Strength of evidence. In our pilot work, we used a because they are compared with regimens that simple win-lose-draw framework with win and lose are comparatively ineffective. Although this bias defined as a superior or inferior finding with a P has been described, the degree to which it per- value < .05, and draw defined as statistical non- vades clinical trial design is unknown. The objec- significance or formal noninferiority. Here, we tive of this paper is to present a new algorithm, built introduce a weighted entropy measure: the neg- on prior foundations, as well as to visually ana- ative logarithm of the P value. Because very small lyze this putative straw man phenomenon in the P values are difficult to interpret, this coefficient CML-1 scenario and a second scenario, the treat- is allowedtotakeamaximum valueof3(ie, ment of relapsed/refractory multiple myeloma P values ,.001 were truncated to .001). (RRMM). METHODS Effect size. We replaced the win-lose-draw frame- work with a coefficient representing the effect size Context-Specific Regimen Identification reported in the trial. For time-based outcomes (eg, The RCTs that were previously identified in the overall survival), we ideally used the hazard ratio context of CML-1 were also used in this study, 16 (HR) as the effect size. When HR was not along with several newly published RCTs. Briefly, reported, we defined the effect size either as the RCTs were identified through a PubMed query and ratio of the median survival times or as the point by hand searches of the literature and published estimate reported as significant in the publication guidelines. There were 27 RCTs identified between (eg, the 3-year event-free survival). For nontem- 1968 and 2016, with 18 distinct regimens, repre- poral measures (eg, response rate), we used the senting10,282patientsstudied(DataSupplement). calculated odds ratio as the effect size. In all cases, To identify RCTs for the context of RRMM, we used a the effect size . 1 was transformed into a co- combination of an established knowledge base of efficient E, which is positive for the winning side chemotherapy regimens, HemOnc.org, along and negative for the losing side (eg, if a publication with RCTs identified by two recent network meta- reports HR = 0.5, E = 2 for the winning side and 6,7 analyses in this setting. This yielded a total of E = 22 for the losing side). 25 RCTs published between 2004 and 2016 containing outcome information for 28 distinct Aging effects. To incorporate outdating of scientific regimens, representing 9,737 patients studied evidence, we introduced an exponential decay (Data Supplement). coefficient as a function of the time since publi- cation of trial results; additional details are in the Algorithm Modifications Data Supplement. The previous valuation algorithm, which was Vertex valuation algorithm. After incorporation of used for ranking as well as for coloration of verti- strength of evidence, effect size, and aging effects, ces, was revised to include strength of evidence, the empirical vertex valuation formula is as follows: effect sizes, aging effects, propagation, and re- fresh as explained in the following paragraphs. Table 1. End Points Used in the Examined CML-1 or RRMM Efficacy measure. For all trials, we selected the Trials, With Relative Value trial-defined primary end point, as described in the End Point Relative Value published manuscript, as the main efficacy mea- Overall survival 1.0 sure for the valuation algorithm. For trials with Progression-free survival 0.8 more than one predefined primary end point, we used the least-surrogate end point. If the primary Time to progression 0.8 end point was met, we used less-surrogate second- Overall response rate 0.7 ary end points in the algorithm if they had marginal Response rate 0.7 or better statistical significance (ie, P < .10). 2 ascopubs.org/journal/cci JCO™ Clinical Cancer Informatics Equation 1 Treatment Network Visualization We used multiple visual variables to display the v^ ¼ 2 log ðP Þ3 RV n 10 y y y¼1 treatment regimen networks: size, color, trans- 3 E 3 log ðN Þ3 f ða Þ parency, and position. See the Data Supplement y 10 y y for details. where for the n th vertex v, there are m incident edges, E is the effect size coefficient of the y th Statistical and General Methods edge, N is the total number of patients in each R version 3.4.0 and RStudio version 1.0.143 pairwise comparison, P is the P value of the y th (https://www.r-project.org/) were used for the cal- outcome, and f ða Þ is the aging coefficient de- culations. Graphs were created and displayed scribed in the previous paragraph. A positively using the igraph package version 1.0.1 (http:// valued vertex is considered recommendable, and igraph.org/r/); coloration was by the RColorBrewer a negatively valued vertex is considered contra- package. Correlations between vertex value and indicated. A vertex with value near zero is con- size before and after value propagation were cal- sideredtohavelackingevidence,conflictingresults, culated using the Pearson product-moment cor- or poor study quality such that there is insufficient relation;unadjustedPvalues, .05wereconsidered evidence on whether to recommend. Although the statistically significant. Animations of all networks valuation coefficient is unitless, the magnitude in- and the R code used to develop them are available formsthe powerofthe valuationand,thus,itisnot upon request. normalized. RESULTS Propagation and refresh. To overcome the appar- Visualization of the Treatment Regimen Networks ent straw man effect (discussed in Results), we The resultant networks for CML-1 and RRMM in investigated the introduction of indirect evidence the most recent year of analysis (2016), after in- propagation. In our pilot work, we did not assign corporation of evidence strength, effect size, and any node valuation on the basis of indirect evi- aging into the valuation algorithm, are shown in dence, such that the calculated network is akin Figures 1A and 2A; a complete list of regimens and to a single-layer perceptron (aka, pairwise network the number of patients studied for each are shown analysis). We augmented this model with informa- in the Data Supplement. On initial visualization tion propagation, which has been studied in the 17-19 of the CML-1 regimen network, a few things are context of social networks. Specifically, we immediately evident: (1) there are severe aging allow nodes that were calculated to lose value as a effects on regimens 1 through 8, with most of these result of newly introduced evidence to pass some being valued somewhere near zero; (2) the quality of their value loss to regimens to which they had of the outcome measure degrades over time, with previously been superior (ie, single-generation the newer regimens almost exclusively evaluated value propagation). Conversely, nodes that were with weak surrogate end points (blue edges); and calculated to gain value as a result of newly in- (3) the largest vertex, regimen 9 (imatinib), is also troduced evidence pass some of their value gain to the lowest ranked. Visualization of the RRMM regimens to which they had previously been in- network reveals that (1) aging effects only seem ferior. For example, in CML-1, dasatinib was dem- prominent for regimens 1 through 5; (2) outcome onstrated to be superior to imatinib ; however, measures are mostly intermediate surrogates (eg, imatinib had been shown 7 years earlier to be progression-free survival); and (3) the largest and superior to interferon a and low-dose cytarabine most connected regimens are the lowest ranked. (IFNA/LoDAC). Therefore, a portion of the value The visually apparent link among connectedness, loss assigned to the imatinib node is propagated size, and low valuation on the initial visual in- to the IFNA/LoDAC node. This has the effect of spection led us to suspect that straw man effects restoring some value to imatinib. When value is were present in both networks, but potentially propagated under these constraints, we also re- overstated. fresh the age-related devaluing coefficient by one half-life. This has the simultaneous effect of allow- Uncovering and Countering the Straw Man Effect ing more value to be propagated while also re- storing some relevance to the older regimen; this is When the vertices are plotted by vertex value analogous to decreasing impedance in an elec- versus size (ie, the total number of patients studied trical circuit, and is one possible solution to the under the regimen), the apparent tendency for problem of hindsight bias. See the Data Supple- large vertices to be negatively valued becomes ment for a more detailed, graphical description. more evident, as shown in Figure 3. In both ascopubs.org/journal/cci JCO™ Clinical Cancer Informatics 3 A B 2 2 3 3 Better Better 1 1 13. Nilotinib (42.6) 13. Nilotinib (42.6) 42.6 42.6 12. Imatinib-HD (18) 7 12. Imatinib-HD (18) 34.1 34.1 14. Dasatinib (6.5) 9. Imatinib (10.6) 25.5 25.5 10 10 14. Dasatinib (6.5) 5. IFNA (2.6) 17 8 5. IFNA (5.7) 11 7. Hydrea/lFNA (1.6) 8.5 8.5 7. Hydrea/lFNA (2.4) (tie) 6. Busulfan/lFNA (1.6) 0 6. Busulfan/lFNA (2.3) 18. Ponatinib (1) –14.3 –9.3 14 14 18. Ponatinib (1) 17. Bosutinib (0.4) –28.6 –18.6 12 9 12 17. Bosutinib (0.4) 18 3. DBM (0) –42.9 18 –28 9 3. DBM (0) (tie) 10. MRD allo-SCT (0) –57.2 –37.3 13 13 (tie) 10. MRD allo-SCT (0) (tie) 16. Imatinib/LoDAC (0) –71.5 –46.6 (tie) 16. Imatinib/LoDAC (0) (tie) 2. Radiation (0) 17 17 Worse Worse 16 16 11. IFNA/HiDAC (–1.4) 11. IFNA/HiDAC (–1.4) 2. Radiation (–2.4) 4. Hydrea (–1.6) 4. Hydrea (–3.9) 15. Imatinib/lFNA (–3.9) (tie) 15. Imatinib/lFNA (–3.9) 1. Busulfan (–4.9) 1. Busulfan (–6.3) 8. IFNA/LoDAC (–15.7) 8. IFNA/LoDAC (–71.5) 9. Imatinib (–46.6) Fig 1. Chemotherapy regimen network for first- contexts, this negative correlation was initially DISCUSSION line treatment of chronic statistically significant: For CML-1, the value of r The interpretation of complex networked data myelogenous leukemia-1 for the correlation of value and size was 20.52 (CML-1), through 2016. (A) benefits from computational approaches and vi- (95% CI, 20.80 to 20.07; P = .026). For RRMM, Initial valuations, before sualization of the results. In the examples dis- application of propagation the value of r for the correlation of value and size cussed here, multiple visual channels (ie, color, and refresh. The current was20.65 (95% CI, 20.82 to 20.37; P = .0002). transparency, size, position) provided an inte- standard of care for CML is Upon inclusion of propagation and refresh, the grated picture of context-specific treatment sce- the use of tyrosine kinase inhibitors (TKIs) in the valuation of some vertices changes dramatically, narios that evolved over many years (CML-1, 49 upfront setting. Consistent as shown in Figures 1B and 2B. In the CML-1 years; RRMM, 13 years). We were able to lever- with this, TKIs are highly network, imatinib moves from the lowest-ranked age human color perception through the use ranked, with the exception regimen to the regimen ranked third highest; of a divergent color scale, as compared with of imatinib, which is the IFNA/LoDAC inherits most of the negative value the rainbow color map often used in scientific lowest-ranked regimen. (B) Applying propagation and from imatinib and becomes the lowest-ranked visualizations. The human visual system is par- refresh to the network regimen. In the RRMM network, almost all aging ticularly well adapted for anomaly detection, ow- changes several valuations, effects disappear due to refresh, bortezomib and ing to enhanced perception of color, edges, and most notably imatinib. 28,29 lenalidomide -dexamethasone become more pos- outliers. As such, we were able to immediately DBM, dibromomannitol; itively valued, dexamethasone (Dex) becomes even recognize a potential anomaly in that the largest HD, high dose; HiDAC, high-dose cytarabine; more negatively valued, and pomalidomide- nodes in the CML-1 and RRMM networks seemed IFNA, interferon a; LoDAC, to be both highly connected (ie, compared with dexamethasone (Pom-Dex) moves from the fourth- low-dose cytarabine; MRD many other regimens) and negatively valued. This highest ranked regimen to the second-highest allo-SCT, matched related- ranked. With this adjustment, the correlation be- evidence from visual inspection led to further in- donor allogeneic stem-cell tween value and size changes and is no longer vestigation into a possible straw man effect, which transplant. significant; for CML-1, the value ofr for the correlation was initially supported by the existence of a sta- between value and size becomes 20.07 (95% tistically significant negative correlation between CI, 20.52 to 0.41; P = .78). For RRMM, the value vertex value and size for both contexts. Through of r for the correlation of value and size becomes the introduction of propagation and refresh into 20.33 (95% CI, 20.62 to 0.05; P = .09). our algorithm, we were able to ameliorate the straw man effect, although it was not eliminated entirely. Sensitivity Analysis Generally, the straw man effect is most evident With systematic variation in RV, the magnitudes of when new interventions are compared with clearly 1,30 the vertex values changed slightly, but the rank inferior regimens. A subtler version is the ten- order did not change for CML-1 or RRMM. Pos- dency to compare a new regimen with a compar- itively valued regimens remained positively valued atively effective regimen using a weaker surrogate 31-33 and vice versa. See the Data Supplement. end point, such as progression-free survival. It 4 ascopubs.org/journal/cci JCO™ Clinical Cancer Informatics 11 A B 4 5 4 5 24. Dara-Len-Dex (17.9) 24. Dara-Len-Dex (17.9) 23. Dara-Bort-Dex (16.6) 10 13. Pom-Dex (17.4) 20. Elo-Len-Dex (12.3) 2 23. Dara-Bort-Dex (16.6) 12 19. Car-Dex (11.8) 12 9 20. Elo-Len-Dex (12.3) 19 19 15 (tie) 13. Pom-Dex (11.8) Better Better 15 19. Car-Dex (11.8) 15. Bort-Dex-Panob (11) 15. Bort-Dex-Panob (11) 17.9 17.9 16. Car-Len-Dex (10.4) 16. Car-Len-Dex (10.4) 14.3 6. Bort-Doxil (7.9) 6 14.3 1 11 6. Bort-Doxil (7.9) 1 11 25. lxaz-Len-Dex (6.2) 10.7 10.7 25. lxaz-Len-Dex (6.2) 10. Bort-Thal-Dex (6) 10. Bort-Thal-Dex (6) 7.1 7.1 27. Pom-Cy-Dex (5.3) 27. Pom-Cy-Dex (5.3) 23 23 3.6 12. Bort-Vorin (4) 3.6 1. Bort (4 .7) 26. lxaz-Dex-HD (3.5) 0 14 12. Bort-Vorin (4) 22. Elo-Bort-Dex (2.5) 3 26. lxaz-Dex-HD (3.5) –8.4 –14.6 18. Bort-Siltux (0.9) 22. Elo-Bort-Dex (2.5) –16.8 27 4. Len (twice-daily) (0.5) –29.2 27 18. Bort-Siltux (0.9) 13 25 8. Bort-SC (0) –25.2 –43.8 4. Len (twice-daily) (0.5) (tie) 14. Pom-Dex (28/28) (0) 14. Pom-Dex (28/28) (–0.3) –33.6 7 –58.4 2. Bort-LD (–0.3) 7 5. Len (–0.5) –42 5. Len (–0.5) –72.9 8. Bort-SC (–0.6) 28. lxaz-Dex (–3.5) (tie) 7. Len-Dex (–0.6) Worse Worse 16 21. Elo-Len-Dex-HD (–3.8) 20 20 2. Bort-LD (–1.1) 17. Pom (–5.4) 28. lxaz-Dex (–3.5) 9. Thal-Dex (–6) 21. Elo-Len-Dex-HD (–3.8) 1. Bort (–8.6) 9. Thal-Dex (–6) 3. Dex (–25.3) 17. Pom (–7.7) 7. Len-Dex (–33.4) 11. Bort-Dex (–42) 11. Bort-Dex (–42) 3. Dex (–72.9) 26 28 26 28 Fig 2. Chemotherapy has been suggested that pharmaceutical industry many reasons, including (1) all regimens prospec- regimen network for support, along with reluctance to sponsor head-to- tively evaluated against imatinib subsequent to the treatment of relapsed/ refractory multiple head comparisons of drugs manufactured by per- IRIS (International Randomized Study of Interferon 34,35 21 myeloma, through 2016. and STI571) trial have either been neutral or ceivedcompetitors,mayexacerbatesucheffects. (A) Initial valuations, before superior to imatinib; (2) surrogate end points have Two examples where the straw man effect may be application of propagation been substituted extensively; and (3) imatinib is operational in cancer domains other than those and refresh. The current now generic and cost may be lower than treatments standard of care is doublet examined here are dacarbazine in melanoma, or triplet therapy using an still under patent protection. Given these chal- where at least nine RCTs demonstrating inferiority immunomodulatory drug or 36 lenges, it is notable that our updated algorithm still have been published between 2000 and 2017 ; proteasome inhibitor ranks imatinib highly. In the final RRMM network and docetaxelinnon–small-cell lung cancer,where backbone. This is reflected (Fig 2B), the algorithm ranks DRd as the highest at least eight RCTs demonstrating inferiority have in the ranking, although and Dex as the lowest ranked regimen, in agree- Bort-Dex and Len-Dex are been published between 2014 and 2017. The the lowest-ranked ment with the analyses by Botta et al and van straw man effect is particularly hard to identify regimens. (B) Applying Beurden-Tan et al. Interestingly, our algorithm directly from the medical literature because the propagation and refresh to ranks a doublet, Pom-Dex, as second-highest, just design and execution of RCTs may precede their the network changes behind Dara-Len-Dex. This is primarily due to the several valuations; most publication by years. Also, many contexts have the notably, Dex is much more refresh of Dex, which allows for substantial negative fortunate situation in which prognosis is improving negatively valued. Bort, 38,39 value to be propagated from Pom-Dex to Dex. The as a result of treatment, including both CML-1 bortezomib; Car, 40 validity of this finding is supported by the fact that and RRMM ; this will naturally lead to the need to carfilzomib; Cy, Pom-Dex has a category 1 recommendation by cyclophosphamide; Dara, substitute surrogate end points so that new RCTs the NCCN, although it is noted by the NCCN daratumumab; Dex, can be completed within a reasonable time. An dexamethasone; Elo, that triplet regimens are preferred to doublet intriguing possible way to mitigate the biases of RCT elotuzumab; HD, high regimens except in frail or elderly patients. design is the use of a treatment of physician’s dose; Ixaz, ixazomib; Notably, the only triplet compared with Pom-Dex, LD, low dose; Len, choice control arm, which was used in the recent pomalidomide-cyclophosphamide-dexamethasone, 41,42 lenalidomide; Panob, (negative) CheckMate 026 trial. was in a randomized phase II trial with a weak panobinostat; Pom, Any algorithmic ranking algorithm must be judged pomalidomide; SC, surrogate primary end point (overall response rate) subcutaneous; Siltux, on face validity. In the final CML-1 network (Fig 1B), and borderline significance (P = .035). Given that siltuximab; Thal, 47,48 the algorithm ranks nilotinib, imatinib (standard Pom-Dex is relatively well tolerated, and that thalidomide; Vorin, and high dose), and dasatinib the highest, in close toxicities are a serious concern for modern mye- vorinostat. 43 49 concordance with NCCN guidelines. Imatinib, in loma drugs, our findings may have real-world particular, is the subject of ongoing contention for significance. ascopubs.org/journal/cci JCO™ Clinical Cancer Informatics 5 Fig 3. Scatterplots of A B vertex value versus size for CML - 1 CML - 1 CML-1 and RRMM. Linear regression lines using the r = –0.07, P = .782 r = –0.52, P = .026 least squares approach are overlaid. For all panels, the results show that value is inversely correlated with size, although the –20 correlation becomes nonsignificant once –20 –40 propagation and refresh are instituted. Proportionate –60 –40 size and transparency are preserved from the –80 treatment network 0 500 1,000 1,500 2,000 2,500 0 500 1,000 1,500 2,000 2,500 visualization; numbering of regimens is omitted to Size Size improve clarity. CML-1, (No. of patients studied on the regimen) (No. of patients studied on the regimen) chronic myelogenous leukemia; RRMM, relapsed/refractory multiple myeloma. C D RRMM RRMM r = –0.33, P = .09 r = –0.65, P ≤ .01 –20 –10 –20 –40 –30 –60 –40 –80 –50 0 500 1,000 1,500 0 500 1,000 1,500 Size Size (No. of patients studied on the regimen) (No. of patients studied on the regimen) There are several limitations to our current ap- as SAMPL to encourage uniform reporting of proach. First and foremost, significant methodo- HRs. Current work is focused (as future work logical challenges remain in the field of dynamic will be) on the use of ensemble methods to eval- network analysis, especially with respect to in- uate how rankings change with perturbation of the 50-52 study and between-study effect modifiers. weighting coefficients, with a goal of choosing a The vertex valuation algorithm contains several consensus model that represents best fit. Be- empirically derived coefficients and, therefore, yond this methodological limitation, the measured could be subject to unmeasured bias. However, valuations may also be subject to positive publi- it is notable that the ASCO Value Framework has cation bias (ie, RCTs that demonstrate a statis- adopted similar weighting metrics to those we use tically significant result are more likely to be 53,54 59 for the surrogacy of end points. The ASCO published). We have tried to ameliorate the Value Framework and other approaches to com- known tendency for positive publication bias by parative valuation, such at the NCCN Evidence including so-called gray literature when possible. Blocks, are also empirically derived. We have Future work will investigate incorporating informa- used several definitions for effect sizes, including tion from clinical trial registries (eg, ClinicalTrials. using point estimates when HRs are not reported. gov) so that unpublished trials might be incorpo- Older publications are more likely to report point rated into the model. It is also possible that sup- estimates rather than HRs, so this could plementation with comparative effectiveness data introduce a time-based systematic bias. In rec- may change the valuations and subsequent rank- ognition of this limitation, we support efforts such ings.Wewillalsoinvestigatemoregranulardefinitions 6 ascopubs.org/journal/cci JCO™ Clinical Cancer Informatics Value Value Value Value of time of publication and incorporate evidence appeared to be the manifestation of the straw man updates into the valuation algorithm, as opposed effect.Additionalmodifications tothealgorithmledto to the current method of using interim updates hierarchical rankings with face validity for both sce- only to modify the aging coefficient. narios.Thesefindingsare likely tobe generalizable to any cancer setting with multiple indirect compari- In conclusion, we have described how the creation sons, and they suggest a new means of ranking and visual inspection of chemotherapy treatment efficacy in cancer trials. regimen networks can rapidly lead to new insights. In both of the described use cases, the visual vari- DOI: https://doi.org/10.1200/CCI.17.00079 ables of color, transparency, size, and position led to Published online on ascopubs.org/journal/cci on November 15, an almost immediate recognition of an anomaly that 2017. AUTHOR CONTRIBUTIONS Relationships may not relate to the subject matter of this manuscript. For more information about ASCO’s conflict of Conception and design: Jeremy L. Warner, Gil Alterovitz interest policy, please refer to www.asco.org/rwc or ascopubs. Collection and assembly of data: Jeremy L. Warner, Peter C. org/jco/site/ifc. Yang Data analysis and interpretation: Jeremy L. Warner, Peter C. Jeremy L. Warner Yang Stock and Other Ownership Interests: HemOnc.org LLC Manuscript writing: All authors Peter C. Yang Final approval of manuscript: All authors Stock and Other Ownership Interests: Merck, Pfizer, Cyclacel, Accountable for all aspects of the work: All authors HemOnc.org LLC Gil Alterovitz AUTHORS’ DISCLOSURES OF POTENTIAL CONFLICTS OF INTEREST No relationship to disclose The following represents disclosure information provided by authors of this manuscript. All relationships are considered ACKNOWLEDGMENT compensated. Relationships are self-held unless noted. I = We give thanks to Thomas A. Lasko, MD, PhD, for his advice on Immediate Family Member, Inst = My Institution. network meta-analysis. Affiliations Jeremy L. Warner, Vanderbilt University, Nashville, TN; Peter C. Yang, Massachusetts General Hospital; and Gil Alterovitz, Harvard Medical School and Harvard-Massachusetts Institute of Technology Division of Health Science, Boston; and Massachusetts Institute of Technology, Cambridge, MA. Support Supported in part by Grants No. R21DA025168 from the National Institute of Drug Abuse, R01HG004836 from the National Human Genome Research Institute, R00LM009826 from the National Library of Medicine, and L30CA171123 from the National Cancer Institute. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. REFERENCES 1. Ioannidis JP: Perfect study, poor evidence: Interpretation of biases preceding study design. Semin Hematol 45:160- 166, 2008 2. Baldwin D, Woods R, Lawson R, et al: Efficacy of drug treatments for generalised anxiety disorder: Systematic review and meta-analysis. BMJ 342:d1199, 2011 3. Liao WC, Chien KL, Lin YL, et al: Adjuvant treatments for resected pancreatic adenocarcinoma: A systematic review and network meta-analysis. Lancet Oncol 14:1095-1103, 2013 4. Elliott WJ, Meyer PM: Incident diabetes in clinical trials of antihypertensive drugs: A network meta-analysis. Lancet 369:201-207, 2007 5. Stettler C, Wandel S, Allemann S, et al: Outcomes associated with drug-eluting and bare-metal stents: A collaborative network meta-analysis. Lancet 370:937-948, 2007 6. Botta C, Ciliberto D, Rossi M, et al: Network meta-analysis of randomized trials in multiple myeloma: Efficacy and safety in relapsed/refractory patients. Blood Adv 1:455-466, 2017 7. van Beurden-Tan CHY, Franken MG, Blommestein HM, et al: Systematic literature review and network meta-analysis of treatment outcomes in relapsed and/or refractory multiple myeloma. J Clin Oncol 35:1312-1319, 2017 ascopubs.org/journal/cci JCO™ Clinical Cancer Informatics 7 8. Chen C: An information-theoretic view of visual analytics. IEEE Comput Graph Appl 28:18-23, 2008 9. Klimov D, Shahar Y, Taieb-Maimon M: Intelligent visualization and exploration of time-oriented data of multiple patients. Artif Intell Med 49:11-31, 2010 10. Hidalgo CA, Blumm N, Barabasi AL, et al: A dynamic network approach for the study of human phenotypes. PLOS Comput Biol 5:e1000353, 2009 11. Weber GM: Identifying translational science within the triangle of biomedicine. J Transl Med 11:126, 2013 12. Warner J, Yang P, Alterovitz G: Automated synthesis and visualization of a chemotherapy treatment regimen network. Stud Health Technol Inform 192:62-66, 2013 13. Warner JL, Cowan AJ, Hall AC, et al: HemOnc.org: A collaborative online knowledge platform for oncology pro- fessionals. J Oncol Pract 11:e336-e350, 2015 14. Ellis LM, Bernstein DS, Voest EE, et al: American Society of Clinical Oncology perspective: Raising the bar for clinical trials by defining clinically meaningful outcomes. J Clin Oncol 32:1277-1280, 2014 15. Wangge G, Klungel OH, Roes KC, et al: Interpretation and inference in noninferiority randomized controlled trials in drug research. Clin Pharmacol Ther 88:420-423, 2010 16. Spruance SL, Reid JE, Grace M, et al: Hazard ratio in clinical trials. Antimicrob Agents Chemother 48:2787-2792, 17. Bakshy E, Rosenn I, Marlow C, et al: The role of social networks in information diffusion. Presented at the 21st International Conference on World Wide Web, Lyon, France, April 16-20, 2012. 18. Cha M: Propagation phenomena in large social networks. Presented at the 23rd International Conference on World Wide Web, Seoul, Korea, April 7-11, 2014. 19. Han S, Zhuang FZ, He Q, et al: Energy model for rumor propagation on social networks. PHYSICA A 394:99-109, 2014 20. Kantarjian H, Shah NP, Hochhaus A, et al: Dasatinib versus imatinib in newly diagnosed chronic-phase chronic myeloid leukemia. N Engl J Med 362:2260-2270, 2010 21. O’Brien SG, Guilhot F, Larson RA, et al: Imatinib compared with interferon and low-dose cytarabine for newly di- agnosed chronic-phase chronic myeloid leukemia. N Engl J Med 348:994-1004, 2003 22. Hoffrage U, Hertwig R, Gigerenzer G: Hindsight bias: A by-product of knowledge updating? J Exp Psychol Learn Mem Cogn 26:566-581, 2000 23. Bertin J: La graphique. Communications 15:169-185, 1970 24. Csardi G, Nepusz T: The igraph software package for complex network research. InterJournal Complex Systems 1695, 25. Neuwirth E: RColorBrewer: ColorBrewer palettes, 2011. R Package Version 1.0-5. https://cran.r-project.org/web/ packages/RColorBrewer/index.html 26. Moreland K: Diverging color maps for scientific visualization. Presented at the 5th International Symposium on Advances in Visual Computing: Part II, Las Vegas, Nevada, November 30-December 2, 2009. 27. Borland D, Taylor MR II: Rainbow color map (still) considered harmful. IEEE Comput Graph Appl 27:14-17, 2007 28. Marr D, Hildreth E: Theory of edge detection. Proc R Soc Lond B Biol Sci 207:187-217, 1980 29. McCollough C: Color adaptation of edge-detectors in the human visual system. Science 149:1115-1116, 1965 30. Prasad V, Berger VW: Hard-wired bias: How even double-blind, randomized controlled trials can be skewed from the start. Mayo Clin Proc 90:1171-1175, 2015 31. Fleming TR, DeMets DL: Surrogate end points in clinical trials: Are we being misled? Ann Intern Med 125:605-613, 32. Booth CM, Eisenhauer EA: Progression-free survival: Meaningful or simply measurable? J Clin Oncol 30:1030-1033, 2012 33. Prasad V, Kim C, Burotto M, et al: The strength of association between surrogate end points and survival in oncology: A systematic review of trial-level meta-analyses. JAMA Intern Med 175:1389-1398, 2015 34. Peppercorn J, Blood E, Winer E, et al: Association between pharmaceutical involvement and outcomes in breast cancer clinical trials. Cancer 109:1239-1246, 2007 35. Lathyris DN, Patsopoulos NA, Salanti G, et al: Industry sponsorship and selection of comparators in randomized clinical trials. Eur J Clin Invest 40:172-182, 2010 36. Dacarbazine monotherapy. HemOnc.org. https://hemonc.org/wiki/Melanoma#Dacarbazine_monotherapy 37. Carrboplatin & paclitaxel. HemOnc.org. https://hemonc.org/wiki/Non-small_cell_lung_cancer#Carboplatin_.26_Paclitaxel_3 38. Kantarjian H, O’Brien S, Jabbour E, et al: Improved survival in chronic myeloid leukemia since the introduction of imatinib therapy: A single-institution historical experience. Blood 119:1981-1987, 2012 39. Bower H, Bjorkholm ¨ M, Dickman PW, et al: Life expectancy of patients with chronic myeloid leukemia approaches the life expectancy of the general population. J Clin Oncol 34:2851-2857, 2016 8 ascopubs.org/journal/cci JCO™ Clinical Cancer Informatics 40. Rajkumar SV, Kumar S: Multiple myeloma: Diagnosis and treatment. Mayo Clin Proc 91:101-119, 2016 41. Twelves C, Jove M, Cortes J: Recognizing the place of trials with treatment of physician’s choice as the control arm. J Clin Oncol 33:1300-1301, 2015 42. Carbone DP, Reck M, Paz-Ares L, et al: First-line nivolumab in stage IV or recurrent non-small-cell lung cancer. N Engl J Med 376:2415-2426, 2017 43. National Comprehensive Cancer Network: NCCN Clinical Practice Guidelines in Oncology: Chronic Myelogenous Leukemia, Version 1.2017, 2017. https://www.nccn.org/professionals/physician_gls/pdf/cml.pdf 44. Agrawal M, Garg RJ, Kantarjian H, et al: Chronic myeloid leukemia in the tyrosine kinase inhibitor era: What is the “best” therapy? Curr Oncol Rep 12:302-313, 2010 45. National Comprehensive Cancer Network: NCCN Clinical Practice Guidelines in Oncology: Multiple Myeloma Version 3.2017, 2017. https://www.nccn.org/professionals/physician_gls/pdf/myeloma.pdf 46. Baz RC, Martin TG III, Lin H-Y, et al: Randomized multicenter phase 2 study of pomalidomide, cyclophosphamide, and dexamethasone in relapsed refractory myeloma. Blood 127:2561-2568, 2016 47. Dimopoulos MA, Palumbo A, Corradini P, et al: Safety and efficacy of pomalidomide plus low-dose dexamethasone in STRATUS (MM-010): A phase 3b study in refractory multiple myeloma. Blood 128:497-503, 2016 48. Conticello C, Parisi M, Romano A, et al: Pomalidomide experience: An effective therapeutic approach with immu- nomodulatory drugs in a patient with relapsed-refractory multiple myeloma. Future Oncol 13(5s):3-6, 2017 49. Delforge M, Ludwig H: How I manage the toxicities of myeloma drugs. Blood 129:2359-2367, 2017 50. Li T, Puhan MA, Vedula SS, et al: Network meta-analysis-highly attractive but more methodological research is needed. BMC Med 9:79, 2011 51. Mills EJ, Ioannidis JP, Thorlund K, et al: How to use an article reporting a multiple treatment comparison meta-analysis. JAMA 308:1246-1253, 2012 52. Jansen JP, Naci H: Is network meta-analysis as valid as standard pairwise meta-analysis? It all depends on the distribution of effect modifiers. BMC Med 11:159, 2013 53. Schnipper LE, Davidson NE, Wollins DS, et al: American Society of Clinical Oncology statement: A conceptual framework to assess the value of cancer treatment options. J Clin Oncol 33:2563-2577, 2015 54. Schnipper LE, Davidson NE, Wollins DS, et al: Updating the American Society of Clinical Oncology Value Framework: Revisions and reflections in response to comments received. J Clin Oncol 34:2925-2934, 2016. 55. National Comprehensive Cancer Network: NCCN clinical practice guidelines in oncology with NCCN evidence blocks. https://www.nccn.org/evidenceblocks/. 56. Tierney JF, Stewart LA, Ghersi D, et al: Practical methods for incorporating summary time-to-event data into meta- analysis. Trials 8:16, 2007 57. Equator Network: SAMPL guidelines for statistical reporting. http://www.equator-network.org/2013/02/11/sampl- guidelines-for-statistical-reporting/ 58. Dietterich TG: Ensemble methods in machine learning, in Kittler J, Roli, F (eds.): Multiple Classifier Systems. First International Workshop, MCS 2000 Cagliari, Italy, June 21-23, 2000 Proceedings. Berlin, Germany, Springer 2000: 1-15. 59. Chan AW, Hrobjartsson A, Haahr MT, et al: Empirical evidence for selective reporting of outcomes in randomized trials: Comparison of protocols to published articles. JAMA 291:2457-2465, 2004 ascopubs.org/journal/cci JCO™ Clinical Cancer Informatics 9

Journal

JCO Clinical Cancer InformaticsWolters Kluwer Health

Published: Nov 15, 2017

References