Skip to main content

Table 7 Performance metrics

From: NOBLE – Flexible concept recognition for large-scale biomedical natural language processing

Corpus Concept Recognition System TP PP FP FN Precision Recall F1 Median runtime over 10 runs (ms)** IQR (ms)
CRAFT MMTx 35,140 659 45,791 51,875 0.43 0.40 0.42 640,450 3,937
CRAFT MGrep 9,955 292 10,666 77,427 0.48 0.12 0.19 27,448* 747
CRAFT Concept Mapper 29,353 713 32,122 57,608 0.48 0.34 0.40 5,329 113
CRAFT cTAKES Dictionary Lookup 37,736 742 36,951 49,196 0.51 0.43 0.47 4,082,685 3,459
CRAFT cTAKES Fast Lookup 35,078 784 51,383 51,812 0.41 0.4 0.41 9,812 1,160
CRAFT NOBLE Coder 36,568 1,637 46,344 49,469 0.44 0.43 0.43 17,431 44
ShARe MMTx 2,375 101 1,675 1,735 0.58 0.58 0.58 52,016 2,678
ShARe MGrep 2,340 35 1,075 1,836 0.68 0.56 0.62 7,103* 148
ShARe Concept Mapper 2,302 34 2,483 1,875 0.48 0.55 0.51 1,543 57
ShARe cTAKES Dictionary Lookup 2,417 39 2,587 1,755 0.48 0.58 0.53 263,336 2,316
ShARe cTAKES Fast Lookup 2,374 36 1,101 1,801 0.68 0.57 0.62 2,754 81
ShARe NOBLE Coder 2,315 99 1,413 1,797 0.62 0.56 0.59 6,466 94
  1. *MGrep runtime is a sum of the runtime of a harness and a stand-alone MGrep invocation on a corpus
  2. **all measurements were performed on a UIMA platform and a Linux Workstation, 32GB RAM, Intel® Core™ i7-3770 CPU @ 3.40GHz