Skip to main content

Table 3 Tokenizer results.

From: Building a biomedical tokenizer using the token lattice design pattern and the adapted Viterbi algorithm

Tokenizer

Accuracy (%)

Confidence Interval, 95%

Whitespace

53.9

52.0, 55.8

Specialist

47.7

45.8, 49.6

Medpost

92.9

91.9, 93.9

Adapted Viterbi, 0-order HMM

70.8

69.1, 72.5

Adapted Viterbi, 1st-order HMM (AV-1)

84.6

83.3, 85.9

AV-1 + random 10% of MedPost corpus

92.4 (5 run avg)

91.4, 93.4

  1. Tokenizer results.