Daily, massive number of pieces of textual information is gathered into Social Media. They comprise a challenging style as they are formed with both slang and formal words. This has become an obstacle for processing texts in Social Media. In this paper we address this issue by introducing a pre-processing pipeline for social media text. For the solution we have proposed, we are focused on English texts from famous micro blogging site, Twitter. Our major source is a set of common slang words which we gathered by incorporating various other sources. Apart from that we are resolving derivations of slang words by following spell correction based approach.
Keywords: Natural Language Processing | Machine Learning / Deep Learning | Text Normalization | Social Media | Spell Checking | Spell Correction |
@article{mapatext,
title={{Text Normalization in Social Media by using Spell Correction and Dictionary Based Approach}},
author={Mapa, Eranga and Wattaladeniya, Lasitha and Chathuranga, Chiran and Dassanayake, Samith and de Silva, Nisansa and Kohomban, Upali and Maldeniya, Danaja},
abstract={Daily, massive number of pieces of textual information is gathered into Social Media. They comprise a challenging style as they are formed with both slang and formal words. This has become an obstacle for processing texts in Social Media. In this paper we address this issue by introducing a pre-processing pipeline for social media text. For the solution we have proposed, we are focused on English texts from famous micro blogging site, Twitter. Our major source is a set of common slang words which we gathered by incorporating various other sources. Apart from that we are resolving derivations of slang words by following spell correction based approach. },
journal={Systems Learning},
volume={1},
pages={1--6},
year={2012},
misc={https://goo.gl/Mu6P4R,NLP:ML,https://i1.rgstatic.net/publication/331812257_Text_Normalization_in_Social_Media_by_using_Spell_Correction_and_Dictionary_Based_Approach/links/5c8d32b992851c1df944805d/largepreview.png,https://goo.gl/NB6Mq6}
}
ActivePI: Nisansa de Silva
We propose to advance social media text analysis by combining linguistic study, preprocessing, and deep learning models to better capture sentiment, structure, and meaning in informal, large-scale user-generated text.