Table 3 The knowledge base in a nutshell (I)

IF the sum of the lengths (i.e. number of characters) of all tokens s _i belonging to s is smaller than L_min, THEN discard s.

∃i | i = affix_in_sequence_tails(s) → discard(s) ∧ add(s')

s' = {s₁,...,s_i-1}

IF an affix from the list of problem affixes is matched at the tail of s, THEN discard s AND add to the facts base a modified copy of s that does not include the tokens corresponding to the matched affix.

∃i | i = affix_in_sequence_head(s) → discard(s) ∧ add(s')

s' = {s_i+1,...,s _n }

IF an affix from the list of problem affixes is matched at the head of s, THEN discard s AND add to the facts base a modified copy of s that does not include the tokens corresponding to the matched affix.

IF all tokens s _i from s belong to the custom dictionary of English words, THEN discard s.

∃(i, j) | (i, j) = affix_within_sequence(s) → discard(s) ∧ add(s') ∧ add(s'')

s' = {s₁,...,s_i-1},

s'' = {s_i+j,...,s _n }

IF an affix from the list of problem affixes is matched within s, THEN discard s AND add to the facts base the sub-lists of tokens s' and s'' that include all the tokens in s that occur before and after the matched affix respectively.

in_dictionary(s₁) ∧ length(s₁) ≥ 3 → discard(s) ∧ add(s')

s' = {s₂,...,s_n}

IF the first token of s belongs to the custom dictionary of English words AND its length (i.e. number of characters) is greater or equal to 3 THEN discard s AND add to the facts base the sub-list of tokens s', that includes all but the first token of s.

in_dictionary(s _n ) ∧ length(s _n ) ≥ 3 → discard(s) ∧ add(s')

s' = {s₁,...,s_n-1}

IF the last token of s belongs to the custom dictionary of English words AND its length (i.e. number of characters) is greater or equal to 3 THEN discard s AND add to the facts base the sub-list of tokens s', that includes all but the last token of s.

size(s) ≥ 2 → merge(s)

IF s contains 2 or more tokens, THEN convert s into a singleton by concatenating all its tokens.