NOBLE – Flexible concept recognition for large-scale biomedical natural language processing

BMC Bioinformatics

Table 7 Performance metrics

Corpus	Concept Recognition System	TP	PP	FP	FN	Precision	Recall	F1	Median runtime over 10 runs (ms)**	IQR (ms)
CRAFT	MMTx	35,140	659	45,791	51,875	0.43	0.40	0.42	640,450	3,937
CRAFT	MGrep	9,955	292	10,666	77,427	0.48	0.12	0.19	27,448*	747
CRAFT	Concept Mapper	29,353	713	32,122	57,608	0.48	0.34	0.40	5,329	113
CRAFT	cTAKES Dictionary Lookup	37,736	742	36,951	49,196	0.51	0.43	0.47	4,082,685	3,459
CRAFT	cTAKES Fast Lookup	35,078	784	51,383	51,812	0.41	0.4	0.41	9,812	1,160
CRAFT	NOBLE Coder	36,568	1,637	46,344	49,469	0.44	0.43	0.43	17,431	44
ShARe	MMTx	2,375	101	1,675	1,735	0.58	0.58	0.58	52,016	2,678
ShARe	MGrep	2,340	35	1,075	1,836	0.68	0.56	0.62	7,103*	148
ShARe	Concept Mapper	2,302	34	2,483	1,875	0.48	0.55	0.51	1,543	57
ShARe	cTAKES Dictionary Lookup	2,417	39	2,587	1,755	0.48	0.58	0.53	263,336	2,316
ShARe	cTAKES Fast Lookup	2,374	36	1,101	1,801	0.68	0.57	0.62	2,754	81
ShARe	NOBLE Coder	2,315	99	1,413	1,797	0.62	0.56	0.59	6,466	94

^*MGrep runtime is a sum of the runtime of a harness and a stand-alone MGrep invocation on a corpus
**all measurements were performed on a UIMA platform and a Linux Workstation, 32GB RAM, Intel® Core™ i7-3770 CPU @ 3.40GHz

ISSN: 1471-2105