Skip to main content

Table 3 Inter-annotator agreement during training

From: Construction of an annotated corpus to support biomedical information extraction

Agreement Type C1 C2 C3 C4 C5
Event identification 58.35 56.01 68.26 77.07 71.94
Argument identification (relaxed span match) 80.45 85.05 91.45 89.39 91.09
Argument identification (exact span match) 61.92 63.98 73.96 79.84 79.17
Semantic role assignment 67.27 75.21 93.91 84.89 86.59
Bio-concept identification 71.35 78.65 78.29 88.55 82.36
Bio-concept category assignment (exact category) 72.34 72.05 71.61 68.84 59.76
Bio-concept category assignment (including parent) 77.53 76.74 75.11 71.58 63.65
Bio-concept supercategory assignment 89.21 89.32 93.45 90.57 84.09
  1. Each numbered column (C1 to C5) displays the IAA results calculated after a particular cycle of training, for a number of separate annotation subtasks. Agreement was calculated between each pair of annotators, and the figures shown in the table are averages amongst all pairs of annotators. Training cycles C1 to C4 were concerned with E. coli abstracts, whilst cycle C5 concerned human abstracts