Cost sensitive hierarchical document classification to triage PubMed abstracts for manual curation

BMC Bioinformatics

Table 1 Hierarchical SVM classifier performance on training dataset for Level 1 predictions.

Level 1 category	Curatable abstracts	AUC individual category (SVM)	Category prediction accuracy (%)(MLP)
Allergy	1146	0.994	91.6
Autoimmunity	4350	0.988	88.9
Infectious Disease	7525	0.989	92.7
Transplantation	888	0.985	76.4
HIV	2369	0.989	92.6
Cancer	2650	0.988	89.8
Other	3905	0.985	85.4
Total	22833		89.7

Performance was evaluated with 10-fold cross-validation. An AUC value was calculated for a given category, where documents assigned by the expert to be of that category are considered "positive" while documents assigned to any other category are negative. To evaluate the performance of the document classification by the Multilayer Perceptron (MLP) into specific categories, we calculated the percent agreement of categories. The AUC and category prediction accuracy values are entered for each Level 1 category in addition to the total prediction accuracy.

ISSN: 1471-2105