Skip to main content

Table 6 BERT performance after combining sub-domain adaptation and the refined fine-tuning mechanism

From: Investigation of improving the pre-training and fine-tuning of BERT model for biomedical relation extraction

Model PPI DDI ChemProt
P R F P R F P R F
BioBERT 9.0 83.3 81.0 9.9 8.1 9.0 4.3 6.3 5.3
BioBERT_SLL_Att 80.7 84.4 82.5 81.3 80.1 80.7 76.5 77.1 76.8
BioBERT_SLL_Att (+P/G) \({\mathbf{83.1 }}\) \( {\mathbf{84.7 }}\) \( {\mathbf{83.8 }}\) 80.4 79.7 80.0 78.4 75.1 76.7
BioBERT_SLL_Att (+D) 81.5 84.5 82.9 \( {\mathbf{82.6 }}\) \( {\mathbf{81.2 }}\) \( {\mathbf{81.9 }}\) 76.8 74.7 75.7
BioBERT_SLL_Att (+CP) 82.5 84.2 83.3 81.7 77.0 79.3 \( {\mathbf{78.9 }}\) \( {\hbox {75.2}}\) \( {\mathbf{77.0 }}\)
PubMedBERT 80.1 84.3 82.1 82.6 81.9 82.3 78.8 75.9 77.3
PubMedBERT_SLL_Att 81.3 85.0 83.1 84.3 82.7 83.5 78.3 77.6 77.9
PubMedBERT_SLL_Att (+P/G) \( {\hbox {81.1}}\) \( {\mathbf{87.1 }}\) \( {\mathbf{84.0 }}\) 83.6 80.6 82.1 79.8 77.0 78.4
PubMedBERT_SLL_Att (+D) 81.4 84.5 82.9 \( {\hbox {84.9}}\) \( {\mathbf{83.2 }}\) \( {\mathbf{84.0 }}\) 79.5 75.9 77.7
PubMedBERT_SLL_Att (+CP) 81.4 85.7 83.4 85.0 81.4 83.2 \( {\hbox {79.7}}\) \( {\mathbf{77.7 }}\) \( {\mathbf{78.7 }}\)
  1. Bold values indicate better results
  2. P: Precision; R: Recall; F: F1 Score; BioBERT/PubMedBERT: original BERT model; BioBERT/PubMedBERT_SLL_Att: model of summarizing the outputs of the last layer using attention mechanism. +P/G: add Protein/Gene-related PubMed abstracts as sub-domain data; +D: add Drug-related PubMed abstracts as sub-domain data; +CP: add protein-related and chemical-related PubMed abstracts as sub-domain data