Crowd learning with candidate labeling: An EM-based solution

Iker Beñaran-Muñoz; Jerónimo Hernández-González; Aritz Pérez

doi:10.1007/978-3-030-00374-6_2

Crowd learning with candidate labeling: An EM-based solution

Iker Beñaran-Muñoz
, Jerónimo Hernández-González
, Aritz Pérez

Basque Center for Applied Mathematics
University of the Basque Country

Research output: Chapter in Book/Conference proceeding › Conference proceeding › peer-review

1 Citation (Scopus)

Abstract

Crowdsourcing is widely used nowadays in machine learning for data labeling. Although in the traditional case annotators are asked to provide a single label for each instance, novel approaches allow annotators, in case of doubt, to choose a subset of labels as a way to extract more information from them. In both the traditional and these novel approaches, the reliability of the labelers can be modeled based on the collections of labels that they provide. In this paper, we propose an Expectation-Maximization-based method for crowdsourced data with candidate sets. Iteratively the likelihood of the parameters that model the reliability of the labelers is maximized, while the ground truth is estimated. The experimental results suggest that the proposed method performs better than the baseline aggregation schemes in terms of estimated accuracy.

Original language	English
Title of host publication	Advances in Artificial Intelligence - 18th Conference of the Spanish Association for Artificial Intelligence, CAEPIA 2018, Proceedings
Editors	Antonio Gonzalez, Alicia Troncoso, Francisco Herrera, Sergio Damas, Rosana Montes, Sergio Alonso, Oscar Cordon
Publisher	Springer Verlag
Pages	13-23
Number of pages	11
ISBN (Print)	9783030003739
DOIs	https://doi.org/10.1007/978-3-030-00374-6_2
Publication status	Published - 2018
Externally published	Yes
Event	18th Conference of the Spanish Association for Artificial Intelligence, CAEPIA 2018 - Granada, Spain Duration: 23 Oct 2018 → 26 Oct 2018

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	11160 LNAI
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	18th Conference of the Spanish Association for Artificial Intelligence, CAEPIA 2018
Country/Territory	Spain
City	Granada
Period	23/10/18 → 26/10/18

Keywords

Candidate labeling
Crowdsourced labels
Expectation-maximization based method
Supervised classification
Weak supervision

Access to Document

10.1007/978-3-030-00374-6_2

Cite this

Beñaran-Muñoz, I., Hernández-González, J., & Pérez, A. (2018). Crowd learning with candidate labeling: An EM-based solution. In A. Gonzalez, A. Troncoso, F. Herrera, S. Damas, R. Montes, S. Alonso, & O. Cordon (Eds.), Advances in Artificial Intelligence - 18th Conference of the Spanish Association for Artificial Intelligence, CAEPIA 2018, Proceedings (pp. 13-23). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 11160 LNAI). Springer Verlag. https://doi.org/10.1007/978-3-030-00374-6_2

Beñaran-Muñoz, Iker ; Hernández-González, Jerónimo ; Pérez, Aritz. / Crowd learning with candidate labeling : An EM-based solution. Advances in Artificial Intelligence - 18th Conference of the Spanish Association for Artificial Intelligence, CAEPIA 2018, Proceedings. editor / Antonio Gonzalez ; Alicia Troncoso ; Francisco Herrera ; Sergio Damas ; Rosana Montes ; Sergio Alonso ; Oscar Cordon. Springer Verlag, 2018. pp. 13-23 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{08e0cbc84f6144d3836ebd6f30c19235,

title = "Crowd learning with candidate labeling: An EM-based solution",

abstract = "Crowdsourcing is widely used nowadays in machine learning for data labeling. Although in the traditional case annotators are asked to provide a single label for each instance, novel approaches allow annotators, in case of doubt, to choose a subset of labels as a way to extract more information from them. In both the traditional and these novel approaches, the reliability of the labelers can be modeled based on the collections of labels that they provide. In this paper, we propose an Expectation-Maximization-based method for crowdsourced data with candidate sets. Iteratively the likelihood of the parameters that model the reliability of the labelers is maximized, while the ground truth is estimated. The experimental results suggest that the proposed method performs better than the baseline aggregation schemes in terms of estimated accuracy.",

keywords = "Candidate labeling, Crowdsourced labels, Expectation-maximization based method, Supervised classification, Weak supervision",

author = "Iker Be{\~n}aran-Mu{\~n}oz and Jer{\'o}nimo Hern{\'a}ndez-Gonz{\'a}lez and Aritz P{\'e}rez",

note = "Publisher Copyright: {\textcopyright} Springer Nature Switzerland AG 2018.; 18th Conference of the Spanish Association for Artificial Intelligence, CAEPIA 2018 ; Conference date: 23-10-2018 Through 26-10-2018",

year = "2018",

doi = "10.1007/978-3-030-00374-6\_2",

language = "English",

isbn = "9783030003739",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer Verlag",

pages = "13--23",

editor = "Antonio Gonzalez and Alicia Troncoso and Francisco Herrera and Sergio Damas and Rosana Montes and Sergio Alonso and Oscar Cordon",

booktitle = "Advances in Artificial Intelligence - 18th Conference of the Spanish Association for Artificial Intelligence, CAEPIA 2018, Proceedings",

address = "Germany",

}

Beñaran-Muñoz, I, Hernández-González, J & Pérez, A 2018, Crowd learning with candidate labeling: An EM-based solution. in A Gonzalez, A Troncoso, F Herrera, S Damas, R Montes, S Alonso & O Cordon (eds), Advances in Artificial Intelligence - 18th Conference of the Spanish Association for Artificial Intelligence, CAEPIA 2018, Proceedings. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 11160 LNAI, Springer Verlag, pp. 13-23, 18th Conference of the Spanish Association for Artificial Intelligence, CAEPIA 2018, Granada, Spain, 23/10/18. https://doi.org/10.1007/978-3-030-00374-6_2

Crowd learning with candidate labeling: An EM-based solution. / Beñaran-Muñoz, Iker; Hernández-González, Jerónimo; Pérez, Aritz.
Advances in Artificial Intelligence - 18th Conference of the Spanish Association for Artificial Intelligence, CAEPIA 2018, Proceedings. ed. / Antonio Gonzalez; Alicia Troncoso; Francisco Herrera; Sergio Damas; Rosana Montes; Sergio Alonso; Oscar Cordon. Springer Verlag, 2018. p. 13-23 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 11160 LNAI).

Research output: Chapter in Book/Conference proceeding › Conference proceeding › peer-review

TY - GEN

T1 - Crowd learning with candidate labeling

T2 - 18th Conference of the Spanish Association for Artificial Intelligence, CAEPIA 2018

AU - Beñaran-Muñoz, Iker

AU - Hernández-González, Jerónimo

AU - Pérez, Aritz

N1 - Publisher Copyright: © Springer Nature Switzerland AG 2018.

PY - 2018

Y1 - 2018

N2 - Crowdsourcing is widely used nowadays in machine learning for data labeling. Although in the traditional case annotators are asked to provide a single label for each instance, novel approaches allow annotators, in case of doubt, to choose a subset of labels as a way to extract more information from them. In both the traditional and these novel approaches, the reliability of the labelers can be modeled based on the collections of labels that they provide. In this paper, we propose an Expectation-Maximization-based method for crowdsourced data with candidate sets. Iteratively the likelihood of the parameters that model the reliability of the labelers is maximized, while the ground truth is estimated. The experimental results suggest that the proposed method performs better than the baseline aggregation schemes in terms of estimated accuracy.

AB - Crowdsourcing is widely used nowadays in machine learning for data labeling. Although in the traditional case annotators are asked to provide a single label for each instance, novel approaches allow annotators, in case of doubt, to choose a subset of labels as a way to extract more information from them. In both the traditional and these novel approaches, the reliability of the labelers can be modeled based on the collections of labels that they provide. In this paper, we propose an Expectation-Maximization-based method for crowdsourced data with candidate sets. Iteratively the likelihood of the parameters that model the reliability of the labelers is maximized, while the ground truth is estimated. The experimental results suggest that the proposed method performs better than the baseline aggregation schemes in terms of estimated accuracy.

KW - Candidate labeling

KW - Crowdsourced labels

KW - Expectation-maximization based method

KW - Supervised classification

KW - Weak supervision

UR - https://www.scopus.com/pages/publications/85055688269

U2 - 10.1007/978-3-030-00374-6_2

DO - 10.1007/978-3-030-00374-6_2

M3 - Conference proceeding

AN - SCOPUS:85055688269

SN - 9783030003739

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 13

EP - 23

BT - Advances in Artificial Intelligence - 18th Conference of the Spanish Association for Artificial Intelligence, CAEPIA 2018, Proceedings

A2 - Gonzalez, Antonio

A2 - Troncoso, Alicia

A2 - Herrera, Francisco

A2 - Damas, Sergio

A2 - Montes, Rosana

A2 - Alonso, Sergio

A2 - Cordon, Oscar

PB - Springer Verlag

Y2 - 23 October 2018 through 26 October 2018

ER -

Beñaran-Muñoz I, Hernández-González J, Pérez A. Crowd learning with candidate labeling: An EM-based solution. In Gonzalez A, Troncoso A, Herrera F, Damas S, Montes R, Alonso S, Cordon O, editors, Advances in Artificial Intelligence - 18th Conference of the Spanish Association for Artificial Intelligence, CAEPIA 2018, Proceedings. Springer Verlag. 2018. p. 13-23. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). doi: 10.1007/978-3-030-00374-6_2

Crowd learning with candidate labeling: An EM-based solution

Abstract

Publication series

Conference

Keywords

Access to Document

Other files and links

Fingerprint

Cite this