Skip to main content

Table 6 BERT performance after combining sub-domain adaptation and the refined fine-tuning mechanism

From: Investigation of improving the pre-training and fine-tuning of BERT model for biomedical relation extraction

Model

PPI

DDI

ChemProt

P

R

F

P

R

F

P

R

F

BioBERT

9.0

83.3

81.0

9.9

8.1

9.0

4.3

6.3

5.3

BioBERT_SLL_Att

80.7

84.4

82.5

81.3

80.1

80.7

76.5

77.1

76.8

BioBERT_SLL_Att (+P/G)

\({\mathbf{83.1 }}\)

\( {\mathbf{84.7 }}\)

\( {\mathbf{83.8 }}\)

80.4

79.7

80.0

78.4

75.1

76.7

BioBERT_SLL_Att (+D)

81.5

84.5

82.9

\( {\mathbf{82.6 }}\)

\( {\mathbf{81.2 }}\)

\( {\mathbf{81.9 }}\)

76.8

74.7

75.7

BioBERT_SLL_Att (+CP)

82.5

84.2

83.3

81.7

77.0

79.3

\( {\mathbf{78.9 }}\)

\( {\hbox {75.2}}\)

\( {\mathbf{77.0 }}\)

PubMedBERT

80.1

84.3

82.1

82.6

81.9

82.3

78.8

75.9

77.3

PubMedBERT_SLL_Att

81.3

85.0

83.1

84.3

82.7

83.5

78.3

77.6

77.9

PubMedBERT_SLL_Att (+P/G)

\( {\hbox {81.1}}\)

\( {\mathbf{87.1 }}\)

\( {\mathbf{84.0 }}\)

83.6

80.6

82.1

79.8

77.0

78.4

PubMedBERT_SLL_Att (+D)

81.4

84.5

82.9

\( {\hbox {84.9}}\)

\( {\mathbf{83.2 }}\)

\( {\mathbf{84.0 }}\)

79.5

75.9

77.7

PubMedBERT_SLL_Att (+CP)

81.4

85.7

83.4

85.0

81.4

83.2

\( {\hbox {79.7}}\)

\( {\mathbf{77.7 }}\)

\( {\mathbf{78.7 }}\)

  1. Bold values indicate better results
  2. P: Precision; R: Recall; F: F1 Score; BioBERT/PubMedBERT: original BERT model; BioBERT/PubMedBERT_SLL_Att: model of summarizing the outputs of the last layer using attention mechanism. +P/G: add Protein/Gene-related PubMed abstracts as sub-domain data; +D: add Drug-related PubMed abstracts as sub-domain data; +CP: add protein-related and chemical-related PubMed abstracts as sub-domain data