Get 20M+ Full-Text Papers For Less Than $1.50/day. Start a 14-Day Trial for You or Your Team.

Learn More →

The reliability of aggregated probability judgments obtained through Cooke's classical model

The reliability of aggregated probability judgments obtained through Cooke's classical model Purpose – The purpose of this paper is to compare various linear opinion pooling models for aggregating probability judgments and to determine whether Cooke's performance weighting model can sift out better calibrated experts and produce better aggregated distribution. Design/methodology/approach – The leave‐one‐out cross‐validation technique is adopted to perform an out‐of‐sample comparison of Cooke's classical model, the equal weight linear pooling method, and the best expert approach. Findings – Both aggregation models significantly outperform the best expert approach, indicating the need for inputs from multiple experts. The performance score for Cooke's classical model drops considerably in out‐of‐sample analysis, indicating that Cooke's performance weight approach might have been slightly overrated before, and the performance weight aggregation method no longer dominantly outperforms the equal weight linear opinion pool. Research limitations/implications – The results show that using seed questions to sift out better calibrated experts may still be a feasible approach. However, because the superiority of Cooke's model as discussed in previous studies can no longer be claimed, whether the cost of extra efforts used in generating and evaluating seed questions is justifiable remains a question. Originality/value – Understanding the performance of various models for aggregating experts' probability judgments is critical for decision and risk analysis. Furthermore, the leave‐one‐out cross‐validation technique used in this study achieves more objective evaluations than previous studies. http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png Journal of Modelling in Management Emerald Publishing

The reliability of aggregated probability judgments obtained through Cooke's classical model

Download PDF
13 pages

Loading next page...
 
/lp/emerald-publishing/the-reliability-of-aggregated-probability-judgments-obtained-through-5f6O95vxdd

References (10)

Publisher
Emerald Publishing
Copyright
Copyright © 2009 Emerald Group Publishing Limited. All rights reserved.
ISSN
1746-5664
DOI
10.1108/17465660910973961
Publisher site
See Article on Publisher Site

Abstract

Purpose – The purpose of this paper is to compare various linear opinion pooling models for aggregating probability judgments and to determine whether Cooke's performance weighting model can sift out better calibrated experts and produce better aggregated distribution. Design/methodology/approach – The leave‐one‐out cross‐validation technique is adopted to perform an out‐of‐sample comparison of Cooke's classical model, the equal weight linear pooling method, and the best expert approach. Findings – Both aggregation models significantly outperform the best expert approach, indicating the need for inputs from multiple experts. The performance score for Cooke's classical model drops considerably in out‐of‐sample analysis, indicating that Cooke's performance weight approach might have been slightly overrated before, and the performance weight aggregation method no longer dominantly outperforms the equal weight linear opinion pool. Research limitations/implications – The results show that using seed questions to sift out better calibrated experts may still be a feasible approach. However, because the superiority of Cooke's model as discussed in previous studies can no longer be claimed, whether the cost of extra efforts used in generating and evaluating seed questions is justifiable remains a question. Originality/value – Understanding the performance of various models for aggregating experts' probability judgments is critical for decision and risk analysis. Furthermore, the leave‐one‐out cross‐validation technique used in this study achieves more objective evaluations than previous studies.

Journal

Journal of Modelling in Management – Emerald Publishing

Published: Jan 1, 2009

Keywords: Modelling; Probability theory; Uncertainty management; Decision making

There are no references for this article.