Table 3 Estimation of consistency in cases of initial agreement. We collected 200-200 randomly chosen examples from each type of corpus text to assess the level of consistency in cases when the two students provided identical annotation for the sentence (identical means here that all cues and scope boundaries were exactly the same) and they were compared to the annotation provided by the chief annotator. The agreement rates are given here.
From: The BioScope corpus: biomedical texts annotated for uncertainty, negation and their scopes
NEGATION | |
|---|---|
keyword: | 98.65% |
left scope: | 97.27% |
right scope: | 98.64% |
full scope: | 95.91% |
SPECULATION | |
keyword: | 99.63% |
left scope: | 99.25% |
right scope: | 99.63% |
full scope: | 98.88% |