Learning by aggregating experts and filtering novices: a solution to crowdsourcing problems in bioinformatics

Table 1 Properties of multi-annotator algorithms.

Algorithms	Unsupervised?	Integrate labels globally?	Data dependent?	Filter novice annotation?
MV	Y	N	N	N
MAP-ML	Y	Y	N	N
GMM-MAPML	Y	Y	Y	N
AEFN	Y	Y	Y	Y

The comparisons of properties of multi-annotator algorithms are shown. 'Y' denotes that the algorithm has the property; 'N' denotes that the algorithm doesn't have the property.