Revolutionary text clustering: Investigating transfer learning capacity of SBERT models through pooling techniques

Basit Öğe Kaydı
dc.authoridORTAKCI, Yasin/0000-0002-0683-2049
dc.contributor.authorOrtakci, Yasin
dc.date.accessioned2024-09-29T15:57:40Z
dc.date.available2024-09-29T15:57:40Z
dc.date.issued2024
dc.departmentKarabük Üniversitesien_US
dc.description.abstractLarge Language Models (LLMs), one of the most advanced representatives of neural networks, have revolutionized the field of natural language processing. Among the many applications of these models, text clustering is gaining increasing interest. In particular, the fact that LLMs digitize text more semantically and contextually than existing methods in the literature has led LLMs to produce more successful results with clustering algorithms. However, since these models are not specifically designed for text clustering, they can lead to processing times that exceed acceptable runtime thresholds. To address this challenge, the Sentence-BERT (SBERT) model has been proposed as a solution, offering the ability to accurately measure text similarity by transforming entire texts into dense, fixed-size vectors. SBERT has been integrated into various LLMs, resulting in the creation of diverse SBERT model variants. This study aims to assess the transfer learning capabilities of SBERT models in the context of text clustering. Furthermore, it investigates the influence of CLS (classification token), mean, and max pooling techniques on the performance of these models. In this direction, we applied these pooling techniques to DistilBERT, DistilRoBERTa, ALBERT, and MPNET based SBERT models and compared their performance on different corpora. The results show that there is no clear superiority among the SBERT models. However, the mean pooling emerged as the most effective method in 13 out of 16 text clustering tasks. This finding underscores the high compatibility of the mean pooling technique with SBERT models.en_US
dc.identifier.doi10.1016/j.jestch.2024.101730
dc.identifier.issn2215-0986
dc.identifier.scopus2-s2.0-85195462434en_US
dc.identifier.scopusqualityQ1en_US
dc.identifier.urihttps://doi.org/10.1016/j.jestch.2024.101730
dc.identifier.urihttps://hdl.handle.net/20.500.14619/4923
dc.identifier.volume55en_US
dc.identifier.wosWOS:001252306300001en_US
dc.identifier.wosqualityN/Aen_US
dc.indekslendigikaynakWeb of Scienceen_US
dc.indekslendigikaynakScopusen_US
dc.language.isoenen_US
dc.publisherElsevier - Division Reed Elsevier India Pvt Ltden_US
dc.relation.ispartofEngineering Science and Technology-An International Journal-Jestechen_US
dc.relation.publicationcategoryMakale - Uluslararası Hakemli Dergi - Kurum Öğretim Elemanıen_US
dc.rightsinfo:eu-repo/semantics/openAccessen_US
dc.subjectSBERTen_US
dc.subjectLarge language modelsen_US
dc.subjectSentence embeddingsen_US
dc.subjectText clusteringen_US
dc.subjectPooling techniquesen_US
dc.titleRevolutionary text clustering: Investigating transfer learning capacity of SBERT models through pooling techniquesen_US
dc.typeArticleen_US

Dosyalar

Koleksiyon

WoS İndeksli Yayınlar Koleksiyonu
Scopus İndeksli Yayınlar Koleksiyonu