Chinese Text Sentiment Analysis Based on BERT-BiGRU Fusion Gated Attention

Huang Shufen; Liu Changhui; Zhang Yinglin

doi:doi:10.11648/j.ajcst.20230602.11

| Peer-Reviewed

Chinese Text Sentiment Analysis Based on BERT-BiGRU Fusion Gated Attention

Huang Shufen, Liu Changhui, Zhang Yinglin

Published in American Journal of Computer Science and Technology (Volume 6, Issue 2)

Received: 22 March 2023 Accepted: 18 April 2023 Published: 24 April 2023

Views: Downloads:

Download PDF

Share This Article

Twitter
Linked In
Facebook

Abstract

To address the problem that Word2vec static encoding cannot give accurate word vectors about contextual semantics and cannot solve the problem of multiple meanings of words, we propose to use the BERT pre-training model as a word embedding layer to obtain word vectors dynamically; we introduce the gating idea to improve on the traditional attention mechanism and propose BERT-BiGRU-GANet model. The model firstly uses the BERT pre-training model as the word vector layer to vectorize the input text by dynamic encoding; secondly, uses the bi-directional gated recursive unit model (BiGRU) to capture the dependencies between long discourse and further analyze the contextual semantics; finally, before output classification, adds the attention mechanism of fusion gating to ignore the features with little relevance and highlight the key features with weight ratio features. We conducted several comparison experiments on the Jingdong public product review dataset, and the model achieved an F1 value of 93.06%, which is 3.41%, 2.55%, and 1.12% more accurate than the BiLSTM, BiLSTM-Att, and BERT-BiGRU models, respectively. It indicates that the use of the BERT-BiGRU-GANet model has some improvement on Chinese text sentiment analysis, which is helpful in the analysis of goods and service reviews, for consumers to select goods, and for merchants to improve their goods or service reviews.

Published in	American Journal of Computer Science and Technology (Volume 6, Issue 2)
DOI	10.11648/j.ajcst.20230602.11
Page(s)	50-56
Creative Commons	This is an Open Access article, distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution and reproduction in any medium or format, provided the original work is properly cited.
Copyright	Copyright © The Author(s), 2024. Published by Science Publishing Group

Keywords

Sentiment Analysis, BERT Pre-training Model, BiGRU, Gated Attention

References

[1]	Zhong Jiawa, Liu Wei, Wang Sili, Yang Heng. A review of text sentiment analysis methods and applications [J]. Data Analysis and Knowledge Discovery, 2021, 5 (06): 1-13.
[2]	Hong Wei, Li Min. A review of research on text sentiment analysis methods [J]. Computer Engineering and Science, 2019, 41 (04): 750-757.
[3]	Sabour S, Frosst N, Hinton G E. Dynamic routing between capsules [J]. Advances in neural information processing systems, 2017, 30.
[4]	Liu, L. W., Yu, S.. Recurrent neural network (RNN) and application research [J]. Science and Technology Perspectives, 2019 (32): 54-55. DOI: 10.19694/j.cnki.issn2095-2457.2019.32.022.2.
[5]	S. Xiong, H. Lv, W. Zhao, and D. Ji, "Towards Twitter sentiment classification by multi-level sentiment-enriched word embeddings," Neurocomputing, vol. 275, pp. 2459-2466, 2018, doi: 10.1016/j.neucom.2017.11.023.
[6]	Liu H L, He Y F. Seasonal attention in LSTM and its application to text sentiment classification [J/OL]. Systems Science and Mathematics: 1-19 [2023-03-14].
[7]	Wang W, Sun YX, Qi QJ, Meng XF. A text sentiment classification model based on BiGRU-attention neural network [J]. Computer Application Research, 2019, 36 (12): 3558-3564. DOI: 10.19734/j.issn.1001-3695.2018.07.0413.
[8]	Devlin J, Chang M W, Lee K, et al. Bert: Pre-training of deep bidirectional transformers for language understanding [J]. arXiv preprint arXiv: 1810.04805, 2018.
[9]	Minqing Hu, Bing Liu. Mining and summarizing customer reviews [P]. Knowledge discovery and data mining, 2004.
[10]	Tomas Mikolov, Ilya Sutskever, Kai Chen 0010, Greg Corrado, Jeffrey Dean. Distributed Representations of Words and Phrases and their Compositionality. [J]. CoRR, 2013, abs/1310.4546.
[11]	Xu Yizhou, Lin Xiao, Lu Li-Ming. A long text sentiment classification model based on hierarchical CNN [J]. Computer Engineering and Design, 2022, 43 (04): 1121-1126. DOI: 10.16208/j.issn1000-7024.2022.04.030.
[12]	Lifu Wang, Bo Shen, Bo Hu, Xing Cao. Can Gradient Descent Provably Learn Linear Dynamic Systems? [J]. arXiv: 2211. 10582, 2022.
[13]	Cheng Y, Sun H, Chen H, et al. Sentiment analysis using multi-head attention capsules with multi-channel CNN and bidirectional GRU [J]. IEEE Access, 2021, 9: 60383-60395.
[14]	Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron C. Courville, Ruslan Salakhutdinov, Richard S. Zemel, Yoshua Bengio. Show, Attend and Tell: Neural Image Caption Generation with Visual Attention. [J]. CoRR, 2015, abs/1502. 03044.
[15]	Lima Luiz Renato, Godeiro Lucas Lúcio. Equity-premium prediction: Attention is all you need [J]. Journal of Applied Econometrics, 2022, 38 (1).
[16]	X. Sun and C. Zhang, "Detecting anomalous emotion through big data from social networks based on a deep learning method," Multimedia Tools and Applications, vol. 79, no. 13-14, pp. 9687-9687, 2020, doi: 10.1007/s11042-018-5665-6.
[17]	Peters M, Neumann M, Iyyer M, et al. Deep Contextualized Word Representations [J]. 2018.
[18]	Phan, Huyen Trang, Ngoc Thanh Nguyen, et al. AspectLevel Sentiment Analysis Using CNN Over BERTGCN [J]. IEEE Access, 2022, 10: 110402-110409.
[19]	Shao Nan. Research on discrete recommendation algorithm based on Gumbel-Softmax distribution [D]. University of Electronic Science and Technology, 2020. DOI: 10.27005/d.cnki.gdzku.2020.004485.
[20]	Yang Xiuzhang, Wu Shuai, Ren Tianshu, Liu Jianyi, Song Jiwen, Liao Wenjing. Research on sentiment analysis of e-commerce reviews by integrating multi-headed attention mechanism and BiLSTM [J]. Information Technology and Informatization, 2022 (10): 5-9.
[21]	Cui Jia-Bin. Research on text sentiment analysis based on BERT-BiGRU model [D]. Shanxi University, 2021. DOI: 10.27284/d.cnki.gsxiu.2021.000092.

Cite This Article

Plain Text BibTeX RIS

APA Style

Huang Shufen, Liu Changhui, Zhang Yinglin. (2023). Chinese Text Sentiment Analysis Based on BERT-BiGRU Fusion Gated Attention. American Journal of Computer Science and Technology, 6(2), 50-56. https://doi.org/10.11648/j.ajcst.20230602.11

Copy | Download

ACS Style

Huang Shufen; Liu Changhui; Zhang Yinglin. Chinese Text Sentiment Analysis Based on BERT-BiGRU Fusion Gated Attention. Am. J. Comput. Sci. Technol. 2023, 6(2), 50-56. doi: 10.11648/j.ajcst.20230602.11

Copy | Download

AMA Style

Huang Shufen, Liu Changhui, Zhang Yinglin. Chinese Text Sentiment Analysis Based on BERT-BiGRU Fusion Gated Attention. Am J Comput Sci Technol. 2023;6(2):50-56. doi: 10.11648/j.ajcst.20230602.11

Copy | Download

@article{10.11648/j.ajcst.20230602.11,
  author = {Huang Shufen and Liu Changhui and Zhang Yinglin},
  title = {Chinese Text Sentiment Analysis Based on BERT-BiGRU Fusion Gated Attention},
  journal = {American Journal of Computer Science and Technology},
  volume = {6},
  number = {2},
  pages = {50-56},
  doi = {10.11648/j.ajcst.20230602.11},
  url = {https://doi.org/10.11648/j.ajcst.20230602.11},
  eprint = {https://article.sciencepublishinggroup.com/pdf/10.11648.j.ajcst.20230602.11},
  abstract = {To address the problem that Word2vec static encoding cannot give accurate word vectors about contextual semantics and cannot solve the problem of multiple meanings of words, we propose to use the BERT pre-training model as a word embedding layer to obtain word vectors dynamically; we introduce the gating idea to improve on the traditional attention mechanism and propose BERT-BiGRU-GANet model. The model firstly uses the BERT pre-training model as the word vector layer to vectorize the input text by dynamic encoding; secondly, uses the bi-directional gated recursive unit model (BiGRU) to capture the dependencies between long discourse and further analyze the contextual semantics; finally, before output classification, adds the attention mechanism of fusion gating to ignore the features with little relevance and highlight the key features with weight ratio features. We conducted several comparison experiments on the Jingdong public product review dataset, and the model achieved an F1 value of 93.06%, which is 3.41%, 2.55%, and 1.12% more accurate than the BiLSTM, BiLSTM-Att, and BERT-BiGRU models, respectively. It indicates that the use of the BERT-BiGRU-GANet model has some improvement on Chinese text sentiment analysis, which is helpful in the analysis of goods and service reviews, for consumers to select goods, and for merchants to improve their goods or service reviews.},
 year = {2023}
}

Copy | Download

TY  - JOUR
T1  - Chinese Text Sentiment Analysis Based on BERT-BiGRU Fusion Gated Attention
AU  - Huang Shufen
AU  - Liu Changhui
AU  - Zhang Yinglin
Y1  - 2023/04/24
PY  - 2023
N1  - https://doi.org/10.11648/j.ajcst.20230602.11
DO  - 10.11648/j.ajcst.20230602.11
T2  - American Journal of Computer Science and Technology
JF  - American Journal of Computer Science and Technology
JO  - American Journal of Computer Science and Technology
SP  - 50
EP  - 56
PB  - Science Publishing Group
SN  - 2640-012X
UR  - https://doi.org/10.11648/j.ajcst.20230602.11
AB  - To address the problem that Word2vec static encoding cannot give accurate word vectors about contextual semantics and cannot solve the problem of multiple meanings of words, we propose to use the BERT pre-training model as a word embedding layer to obtain word vectors dynamically; we introduce the gating idea to improve on the traditional attention mechanism and propose BERT-BiGRU-GANet model. The model firstly uses the BERT pre-training model as the word vector layer to vectorize the input text by dynamic encoding; secondly, uses the bi-directional gated recursive unit model (BiGRU) to capture the dependencies between long discourse and further analyze the contextual semantics; finally, before output classification, adds the attention mechanism of fusion gating to ignore the features with little relevance and highlight the key features with weight ratio features. We conducted several comparison experiments on the Jingdong public product review dataset, and the model achieved an F1 value of 93.06%, which is 3.41%, 2.55%, and 1.12% more accurate than the BiLSTM, BiLSTM-Att, and BERT-BiGRU models, respectively. It indicates that the use of the BERT-BiGRU-GANet model has some improvement on Chinese text sentiment analysis, which is helpful in the analysis of goods and service reviews, for consumers to select goods, and for merchants to improve their goods or service reviews.
VL  - 6
IS  - 2
ER  -

Copy | Download

Author Information

Huang Shufen

College of Computer Science and Engineering, Wuhan Institute of Technology, Wuhan, China
Liu Changhui

College of Computer Science and Engineering, Wuhan Institute of Technology, Wuhan, China
Zhang Yinglin

College of Computer Science and Engineering, Wuhan Institute of Technology, Wuhan, China

Download PDF

Sections

Plain Text BibTeX RIS

APA Style

Huang Shufen, Liu Changhui, Zhang Yinglin. (2023). Chinese Text Sentiment Analysis Based on BERT-BiGRU Fusion Gated Attention. American Journal of Computer Science and Technology, 6(2), 50-56. https://doi.org/10.11648/j.ajcst.20230602.11

Copy | Download

ACS Style

Huang Shufen; Liu Changhui; Zhang Yinglin. Chinese Text Sentiment Analysis Based on BERT-BiGRU Fusion Gated Attention. Am. J. Comput. Sci. Technol. 2023, 6(2), 50-56. doi: 10.11648/j.ajcst.20230602.11

Copy | Download

AMA Style

Huang Shufen, Liu Changhui, Zhang Yinglin. Chinese Text Sentiment Analysis Based on BERT-BiGRU Fusion Gated Attention. Am J Comput Sci Technol. 2023;6(2):50-56. doi: 10.11648/j.ajcst.20230602.11

Copy | Download

@article{10.11648/j.ajcst.20230602.11,
  author = {Huang Shufen and Liu Changhui and Zhang Yinglin},
  title = {Chinese Text Sentiment Analysis Based on BERT-BiGRU Fusion Gated Attention},
  journal = {American Journal of Computer Science and Technology},
  volume = {6},
  number = {2},
  pages = {50-56},
  doi = {10.11648/j.ajcst.20230602.11},
  url = {https://doi.org/10.11648/j.ajcst.20230602.11},
  eprint = {https://article.sciencepublishinggroup.com/pdf/10.11648.j.ajcst.20230602.11},
  abstract = {To address the problem that Word2vec static encoding cannot give accurate word vectors about contextual semantics and cannot solve the problem of multiple meanings of words, we propose to use the BERT pre-training model as a word embedding layer to obtain word vectors dynamically; we introduce the gating idea to improve on the traditional attention mechanism and propose BERT-BiGRU-GANet model. The model firstly uses the BERT pre-training model as the word vector layer to vectorize the input text by dynamic encoding; secondly, uses the bi-directional gated recursive unit model (BiGRU) to capture the dependencies between long discourse and further analyze the contextual semantics; finally, before output classification, adds the attention mechanism of fusion gating to ignore the features with little relevance and highlight the key features with weight ratio features. We conducted several comparison experiments on the Jingdong public product review dataset, and the model achieved an F1 value of 93.06%, which is 3.41%, 2.55%, and 1.12% more accurate than the BiLSTM, BiLSTM-Att, and BERT-BiGRU models, respectively. It indicates that the use of the BERT-BiGRU-GANet model has some improvement on Chinese text sentiment analysis, which is helpful in the analysis of goods and service reviews, for consumers to select goods, and for merchants to improve their goods or service reviews.},
 year = {2023}
}

Copy | Download

TY  - JOUR
T1  - Chinese Text Sentiment Analysis Based on BERT-BiGRU Fusion Gated Attention
AU  - Huang Shufen
AU  - Liu Changhui
AU  - Zhang Yinglin
Y1  - 2023/04/24
PY  - 2023
N1  - https://doi.org/10.11648/j.ajcst.20230602.11
DO  - 10.11648/j.ajcst.20230602.11
T2  - American Journal of Computer Science and Technology
JF  - American Journal of Computer Science and Technology
JO  - American Journal of Computer Science and Technology
SP  - 50
EP  - 56
PB  - Science Publishing Group
SN  - 2640-012X
UR  - https://doi.org/10.11648/j.ajcst.20230602.11
AB  - To address the problem that Word2vec static encoding cannot give accurate word vectors about contextual semantics and cannot solve the problem of multiple meanings of words, we propose to use the BERT pre-training model as a word embedding layer to obtain word vectors dynamically; we introduce the gating idea to improve on the traditional attention mechanism and propose BERT-BiGRU-GANet model. The model firstly uses the BERT pre-training model as the word vector layer to vectorize the input text by dynamic encoding; secondly, uses the bi-directional gated recursive unit model (BiGRU) to capture the dependencies between long discourse and further analyze the contextual semantics; finally, before output classification, adds the attention mechanism of fusion gating to ignore the features with little relevance and highlight the key features with weight ratio features. We conducted several comparison experiments on the Jingdong public product review dataset, and the model achieved an F1 value of 93.06%, which is 3.41%, 2.55%, and 1.12% more accurate than the BiLSTM, BiLSTM-Att, and BERT-BiGRU models, respectively. It indicates that the use of the BERT-BiGRU-GANet model has some improvement on Chinese text sentiment analysis, which is helpful in the analysis of goods and service reviews, for consumers to select goods, and for merchants to improve their goods or service reviews.
VL  - 6
IS  - 2
ER  -

Copy | Download