Table 4 Statistics and IAA scores of annotated corpus units. The number of agreements was counted when annotators agree on both entities and relation labels. The IAA score in this case was calculated using the Cohen’s kappa statistic
From: A corpus for plant-chemical relationships in the biomedical domain
Phases | # of corpus units | # of agreements for both entities and relations | IAA score, Kappa |
|---|---|---|---|
Phase 1 | 642 | 566 | 76.5 % |
Phase 2 | 401 | 368 | 82.0 % |
Overall | 1043 | 934 | 78.9 % |