مرکز منطقه ای اطلاع رسانی علوم و فناوری Journal of Information Systems and Telecommunication (JIST) 2322-1437 11 43 2023 8 20 Deep Transformer-based Representation for Text Chunking Deep Transformer-based Representation for Text Chunking 176 184 10.61186/jist.19894.11.43.176 en Parsa Kavehzadeh Amirkabir University of Technology Mohammad Mahdi Abdollah Pour Amirkabir University of Technology Saeedeh Momtazi Amirkabir University of Technology 2021 7 27 Text chunking is one of the basic tasks in natural language processing. Most proposed models in recent years were employed on chunking and other sequence labeling tasks simultaneously and they were mostly based on Recurrent Neural Networks (RNN) and Conditional Random Field (CRF). In this article, we use state-of-the-art transformer-based models in combination with CRF, Long Short-Term Memory (LSTM)-CRF as well as a simple dense layer to study the impact of different pre-trained models on the overall performance in text chunking. To this aim, we evaluate BERT, RoBERTa, Funnel Transformer, XLM, XLM-RoBERTa, BART, and GPT2 as candidates of contextualized models. Our experiments exhibit that all transformer-based models except GPT2 achieved close and high scores on text chunking. Due to the unique unidirectional architecture of GPT2, it shows a relatively poor performance on text chunking in comparison to other bidirectional transformer-based architectures. Our experiments also revealed that adding a LSTM layer to transformer-based models does not significantly improve the results since LSTM does not add additional features to assist the model to achieve more information from the input compared to the deep contextualized models. http://jist.ir/fa/Article/Download/19894