Text Normalization in Social Media by using Spell Correction and Dictionary Based Approach

Eranga Mapa, Lasitha Wattaladeniya, Chiran Chathuranga, Samith Dassanayake, Nisansa de Silva, Upali Kohomban, Danaja Maldeniya
Systems Learning

Daily, massive number of pieces of textual information is gathered into Social Media. They comprise a challenging style as they are formed with both slang and formal words. This has become an obstacle for processing texts in Social Media. In this paper we address this issue by introducing a pre-processing pipeline for social media text. For the solution we have proposed, we are focused on English texts from famous micro blogging site, Twitter. Our major source is a set of common slang words which we gathered by incorporating various other sources. Apart from that we are resolving derivations of slang words by following spell correction based approach.

Keywords: Natural Language Processing | Machine Learning / Deep Learning | Text Normalization | Social Media | Spell Checking | Spell Correction |