Skip to main content

Table 3 Tokenizer results.

From: Building a biomedical tokenizer using the token lattice design pattern and the adapted Viterbi algorithm

Tokenizer Accuracy (%) Confidence Interval, 95%
Whitespace 53.9 52.0, 55.8
Specialist 47.7 45.8, 49.6
Medpost 92.9 91.9, 93.9
Adapted Viterbi, 0-order HMM 70.8 69.1, 72.5
Adapted Viterbi, 1st-order HMM (AV-1) 84.6 83.3, 85.9
AV-1 + random 10% of MedPost corpus 92.4 (5 run avg) 91.4, 93.4
  1. Tokenizer results.