Figure 4

ACT consensus analysis. (A) The two black lines represent the number of relevant articles found while traversing the dataset. The diagonal line represents a random traversal, the parabolic line above represents a traversal following the ranking proposed by the consensus predictions. The green lines are bootstrap estimates for standard deviation. The horizontal red line represents half of the 910 articles. (B) Instead of showing the traversal in articles read, this shows the time spent reading them. The diagonal line represents a random traversal, the parabolic line above represents a traversal following the ranking proposed by the consensus predictions. The green lines represent traversals using the scores provided by the systems. Note that some runs seen below the random traversal seemed to have provided the opposite ranking for the negative documents than required, based on the submission format. TC: Team consensus prediction.