Access the full text.
Sign up today, get DeepDyve free for 14 days.
Loading next page...
/lp/association-for-computing-machinery/crowdsourcing-ground-truth-for-medical-relation-extraction-Io0DX3xCMQ
References (43)
-
Haijun Zhai, T. Lingren, Louise Deléger, Qi Li, M. Kaiser, Laura Stoutenborough, I. Solti (2013)
Web 2.0-Based Crowdsourcing for High-Quality Gold Standard Development in Clinical Natural Language ProcessingJournal of Medical Internet Research, 15
-
P. Bayerl, K. Paul (2011)
What Determines Inter-Coder Agreement in Manual Annotations? A Meta-Analytic InvestigationComputational Linguistics, 37
-
W. Chapman, P. Nadkarni, L. Hirschman, Leonard D'Avolio, G. Savova, Özlem Uzuner (2011)
Overcoming barriers to NLP for clinical text: the role of shared tasks and the need for additional creative solutionsJournal of the American Medical Informatics Association : JAMIA, 18 5
-
Timothy Finin, W. Murnane, A. Karandikar, Nicholas Keller, Justin Martineau, Mark Dredze (2010)
Annotating Named Entities in Twitter Data with Crowdsourcing -
P. Welinder, Steve Branson, Serge Belongie, P. Perona (2010)
The Multidimensional Wisdom of Crowds -
M. Schaekermann (2016)
Resolvable vs. Irresolvable Ambiguity: A New Hybrid Framework for Dealing with Uncertain Ground Truth -
O. Inel, Khalid Khamkham, T. Cristea, Anca Dumitrache, Arne Rutjes, Jelle Ploeg, Lukasz Romaszko, Lora Aroyo, Robert-Jan Sips (2014)
CrowdTruth: Machine-Human Computation Framework for Harnessing Disagreement in Gathering Annotated Data -
Q. Mcnemar (1947)
Note on the sampling error of the difference between correlated proportions or percentagesPsychometrika, 12
-
B. Knott, J. Gratch, A. Cangelosi, James Caverlee (2018)
ACM Transactions on Interactive Intelligent Systems (TiiS) Special Issue on Trust and Influence in Intelligent Human-Machine InteractionACM Transactions on Interactive Intelligent Systems (TiiS), 8
-
Jonathan M. Mortensen, Mark A. Musen, Natalya F. Noy (2013)
Crowdsourcing the verification of relationships in biomedical ontologiesProceedings of the AMIA Annual Symposium. American Medical Informatics Association
-
Jey Lau, Alexander Clark, Shalom Lappin (2014)
Measuring Gradience in Speakers' Grammaticality JudgementsCognitive Science, 36
-
Lora Aroyo, Chris Welty (2013)
Measuring Crowd Truth for Medical Relation Extraction -
a comparison between using annotations from crowd and from medical experts to train a relation extraction classifier, showing that, with the processing of ambiguity -
Barbara Plank, Dirk Hovy, Anders Søgaard (2014)
Linguistically debatable or just plain wrong? -
Lora Aroyo, Chris Welty (2014)
The Three Sides of CrowdTruthHum. Comput., 1
-
Anca Dumitrache, Lora Aroyo, Chris Welty (2015)
Achieving Expert-Level Annotation Quality with CrowdTruth: The Case of Medical Relation Extraction -
Dirk Hovy, Barbara Plank, Anders Søgaard (2014)
Experiments with crowdsourced re-annotation of a POS tagging data set -
Lora Aroyo, Chris Welty (2013)
Crowd Truth: Harnessing disagreement in crowdsourcing a relation extraction gold standard -
Kairit Sirts, Jacob Eisenstein, M. Elsner, S. Goldwater (2014)
Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)The Association for Computational Linguistics
-
O. Bodenreider (2004)
The Unified Medical Language System (UMLS): integrating biomedical terminologyNucleic acids research, 32 Database issue
-
Chris Welty, James Fan, David Gondek, A. Schlaikjer (2010)
Large Scale Relation Detection -
David Dolan (2011)
Building a Persistent Workforce on Mechanical Turk for Multilingual Data Collection -
Mike Mintz, Steven Bills, R. Snow, Dan Jurafsky (2009)
Distant supervision for relation extraction without labeled data -
E. Platz, P. Jhund, N. Girerd, E. Pivetta, J. McMurray, W. Peacock, J. Masip, F. Martín-Sánchez, Ò. Miró, S. Price, L. Cullen, A. Maisel, C. Vrints, M. Cowie, S. Disomma, Héctor Bueno, A. Mebazaa, D. Gualandro, Múcio Tavares, M. Metra, A. Coats, F. Ruschitzka, P. Seferovic, C. Mueller (2019)
Expert consensus document: Reporting checklist for quantification of pulmonary congestion by lung ultrasound in heart failureEuropean Journal of Heart Failure, 21
-
R. Snow, Brendan O'Connor, Dan Jurafsky, A. Ng (2008)
Cheap and Fast – But is it Good? Evaluating Non-Expert Annotations for Natural Language Tasks -
Anca Dumitrache, Lora Aroyo, Chris Welty (2015)
CrowdTruth Measures for Language Ambiguity: The Case of Medical Relation Extraction -
A. Aronson (2001)
Effective mapping of biomedical text to the UMLS Metathesaurus: the MetaMap programProceedings. AMIA Symposium
-
Chang Wang, James Fan (2014)
Medical Relation Extraction with Manifold Models -
Sarath Kondreddi, P. Triantafillou, G. Weikum (2014)
Combining information extraction and human computing for crowdsourced knowledge acquisition2014 IEEE 30th International Conference on Data Engineering
-
Wayne Leigh, Anne Kriete (2020)
BaselineDefinitions
-
T. Li, Benjamin Good, A. Su (2015)
Exposing ambiguities in a relation-extraction gold standard with crowdsourcingArXiv, abs/1505.06256
-
M. Cheatham, P. Hitzler (2014)
Conference v2.0: An Uncertain Version of the OAEI Conference Benchmark -
J. Mortensen, M. Musen, Natasha Noy (2013)
Crowdsourcing the Verification of Relationships in Biomedical OntologiesAMIA ... Annual Symposium proceedings. AMIA Symposium, 2013
-
Lora Aroyo, Chris Welty (2014)
The three sides of crowdtruthJ. Hum. Comput, 1
-
Mike Schaekermann, Edith Law, Alex C. Williams, William Callaghan (2016)
Resolvable vsirresolvable ambiguity: A new hybrid framework for dealing with uncertain ground truth. In Proceedings of the 1st Workshop on Human-Centered Machine Learning at SIGCHI.
-
a similar comparison between crowd annotations and distant supervision, showing that classifiers trained on crowd annotations perform better than those trained on distant supervision -
Lydia Chilton, Greg Little, D. Edge, Daniel Weld, J. Landay (2013)
Cascade: crowdsourcing taxonomy creationProceedings of the SIGCHI Conference on Human Factors in Computing Systems
-
Jongwuk Lee, Hyunsouk Cho, Jin-Woo Park, Young-rok Cha, Seung-won Hwang, Zaiqing Nie, Ji-Rong Wen (2013)
Hybrid entity clustering using crowds and dataThe VLDB Journal, 22
-
Jey Han Lau, Alexander Clark, Shalom Lappin (2014)
Measuring gradience in speakers’ grammaticality judgementsProceedings of the 36th Annual Conference of the Cognitive Science Society
-
J. Burger, E. Doughty, Samuel Bayer, David Tresner-Kirsch, Ben Wellner, J. Aberdeen, Kyungjoon Lee, M. Kann, L. Hirschman (2012)
Validating Candidate Gene-Mutation Relations in MEDLINE Abstracts via Crowdsourcing -
Jordan Peterson (2007)
The Meaning of Meaning, 1
-
Haijun Zhai, Todd Lingren, Louise Deleger, Qi Li, Megan Kaiser, Laura Stoutenborough, Imre Solti (2013)
Web 2J. Med. Internet Res., 15
-
J. Whitehill, P. Ruvolo, Tingfan Wu, J. Bergsma, J. Movellan (2009)
Whose Vote Should Count More: Optimal Integration of Labels from Labelers of Unknown Expertise
- Publisher
- Association for Computing Machinery
- Copyright
- Copyright © 2018 Owner/Author
- ISSN
- 2160-6455
- eISSN
- 2160-6463
- DOI
- 10.1145/3152889
- Publisher site
- See Article on Publisher Site
Abstract
Cognitive computing systems require human labeled data for evaluation and often for training. The standard practice used in gathering this data minimizes disagreement between annotators, and we have found this results in data that fails to account for the ambiguity inherent in language. We have proposed the CrowdTruth method for collecting ground truth through crowdsourcing, which reconsiders the role of people in machine learning based on the observation that disagreement between annotators provides a useful signal for phenomena such as ambiguity in the text. We report on using this method to build an annotated data set for medical relation extraction for the cause and treat relations, and how this data performed in a supervised training experiment. We demonstrate that by modeling ambiguity, labeled data gathered from crowd workers can (1) reach the level of quality of domain experts for this task while reducing the cost, and (2) provide better training data at scale than distant supervision. We further propose and validate new weighted measures for precision, recall, and F-measure, which account for ambiguity in both human and machine performance on this task.
Journal
ACM Transactions on Interactive Intelligent Systems (TiiS) – Association for Computing Machinery
Published: Jul 6, 2018
Keywords: Ground truth