![\"Medium](\"https:\/\/www.artefact.com\/\/wp-content\/uploads\/2021\/03\/1s986xIGqhfsN8U-09_AdA.png\")
<\/a><\/span><\/div><\/div><\/div><\/div><\/div><\/div>\n<\/div>\nNLPretext overview<\/h2><\/div>
NLPretext is composed of 4 modules: basic, social, token and augmentation.<\/p>\n
Each of them includes different functions to handle the most important text preprocessing tasks.<\/p>\n<\/div>
Basic preprocessing<\/h2><\/div>The basic module is a catalogue of transversal functions that can be used in any use case. They allow you to handle:<\/p>\n<\/div>
- <\/i><\/span>Bad whitespaces in a text, end of line characters<\/div><\/li>
- <\/i><\/span>Encoding issues<\/div><\/li>
- <\/i><\/span>\n
Special characters such as currency symbols, numbers, punctuation marks, latin and non-latin characters<\/p>\n<\/div><\/li>
- <\/i><\/span>Emails and phone numbers<\/div><\/li><\/ul>from nlpretext.basic.preprocess import replace_emails
\nexample = “I have forwarded this email to obama@whitehouse.gov”
\nexample = replace_emails(example, replace_with=”*EMAIL*”)
\nprint(example)
\n# “I have forwarded this email to *EMAIL*”<\/div>\n<\/div>Social preprocessing<\/h2><\/div>The social<\/strong> module is a catalogue of handy functions that can be useful when processing social data, such as:<\/p>\n<\/div>- <\/i><\/span>\n
hashtags extraction\/removal<\/p>\n<\/div><\/li>
- <\/i><\/span>\n
emojis extraction\/removal<\/p>\n<\/div><\/li>
- <\/i><\/span>\n
mentions extraction\/removal<\/p>\n<\/div><\/li>
- <\/i><\/span>\n
html tags cleaning<\/p>\n<\/div><\/li><\/ul>
from nlpretext.social.preprocess import extract_emojis
\nexample = “I take care of my skin \ud83d\ude00”
\nexample = extract_emojis(example)
\nprint(example) #[‘:grinning_face:’]<\/div>\n<\/div>Text augmentation<\/h2><\/div>The augmentation module helps you to generate new texts based on your given examples by modifying some words in the initial ones and to keep associated entities unchanged, if any, in the case of NER tasks. If you want words other than entities to remain unchanged, you can specify it within the stopwords argument. Modifications depend on the chosen method, the ones currently supported by the module are substitutions with synonyms using Wordnet or BERT from the nlpaug library.<\/p>\n<\/div>
Create your end to end pipeline<\/h2><\/div>Default pipeline<\/h3><\/div>Our library provides a Preprocessor object to efficiently pipe all preprocessing operations.
\nIf you need to keep all elements of your text and perform minimum cleaning, use the default pipeline. It normalizes whitespaces and removes newlines characters, fixes unicode problems and removes recurrent artifacts from social data such as mentions, hashtags and HTML tags.<\/p>\n<\/div>
from nlpretext import Preprocessor<\/div>\ntext = “I just got the best dinner in my life @latourdargent !!! I recommend \ud83d\ude00 #food #paris n”<\/div>\npreprocessor = Preprocessor()<\/div>\ntext = preprocessor.run(text) print(text)<\/div>\n# “I just got the best dinner in my life !!! I recommend”<\/div>\n<\/div>Custom pipeline<\/h3><\/div>If you have a clear idea of what preprocessing functions you want to pipe in your preprocessing pipeline, you can add them in your own Preprocessor.<\/p>\n<\/div>
from nlpretext import Preprocessor<\/div>\nfrom nlpretext.basic.preprocess import (normalize_whitespace, remove_punct, remove_eol_characters, remove_stopwords, lower_text)<\/div>\nfrom nlpretext.social.preprocess import remove_mentions, remove_hashtag, remove_emoji<\/div>\ntext = \"I just got the best dinner in my life @latourdargent !!! I recommend \ud83d\ude00 #food #paris n\"<\/div>\npreprocessor = Preprocessor()<\/div>\npreprocessor.pipe(lower_text)<\/div>\npreprocessor.pipe(remove_mentions)<\/div>\npreprocessor.pipe(remove_hashtag)<\/div>\npreprocessor.pipe(remove_emoji)<\/div>\npreprocessor.pipe(remove_eol_characters)<\/div>\npreprocessor.pipe(remove_stopwords, args=)<\/div>\npreprocessor.pipe(remove_punct)<\/div>\npreprocessor.pipe(normalize_whitespace)<\/div>\ntext = preprocessor.run(text) print(text) # \"dinner life recommend\"<\/div>\n<\/div>NLPretext installation<\/h2><\/div>To install the library please run<\/p>\n<\/div>
pip install nlpretext<\/div>\n<\/div>
NLPretext overview<\/h2><\/div>
NLPretext is composed of 4 modules: basic, social, token and augmentation.<\/p>\n
Each of them includes different functions to handle the most important text preprocessing tasks.<\/p>\n<\/div>
Basic preprocessing<\/h2><\/div>The basic module is a catalogue of transversal functions that can be used in any use case. They allow you to handle:<\/p>\n<\/div>
- <\/i><\/span>Bad whitespaces in a text, end of line characters<\/div><\/li>
- <\/i><\/span>Encoding issues<\/div><\/li>
- <\/i><\/span>\n
Special characters such as currency symbols, numbers, punctuation marks, latin and non-latin characters<\/p>\n<\/div><\/li>
- <\/i><\/span>Emails and phone numbers<\/div><\/li><\/ul>from nlpretext.basic.preprocess import replace_emails
\nexample = “I have forwarded this email to obama@whitehouse.gov”
\nexample = replace_emails(example, replace_with=”*EMAIL*”)
\nprint(example)
\n# “I have forwarded this email to *EMAIL*”<\/div>\n<\/div>Social preprocessing<\/h2><\/div>The social<\/strong> module is a catalogue of handy functions that can be useful when processing social data, such as:<\/p>\n<\/div>- <\/i><\/span>\n
hashtags extraction\/removal<\/p>\n<\/div><\/li>
- <\/i><\/span>\n
emojis extraction\/removal<\/p>\n<\/div><\/li>
- <\/i><\/span>\n
mentions extraction\/removal<\/p>\n<\/div><\/li>
- <\/i><\/span>\n
html tags cleaning<\/p>\n<\/div><\/li><\/ul>
from nlpretext.social.preprocess import extract_emojis
\nexample = “I take care of my skin \ud83d\ude00”
\nexample = extract_emojis(example)
\nprint(example) #[‘:grinning_face:’]<\/div>\n<\/div>Text augmentation<\/h2><\/div>The augmentation module helps you to generate new texts based on your given examples by modifying some words in the initial ones and to keep associated entities unchanged, if any, in the case of NER tasks. If you want words other than entities to remain unchanged, you can specify it within the stopwords argument. Modifications depend on the chosen method, the ones currently supported by the module are substitutions with synonyms using Wordnet or BERT from the nlpaug library.<\/p>\n<\/div>
Create your end to end pipeline<\/h2><\/div>Default pipeline<\/h3><\/div>Our library provides a Preprocessor object to efficiently pipe all preprocessing operations.
\nIf you need to keep all elements of your text and perform minimum cleaning, use the default pipeline. It normalizes whitespaces and removes newlines characters, fixes unicode problems and removes recurrent artifacts from social data such as mentions, hashtags and HTML tags.<\/p>\n<\/div>
from nlpretext import Preprocessor<\/div>\ntext = “I just got the best dinner in my life @latourdargent !!! I recommend \ud83d\ude00 #food #paris n”<\/div>\npreprocessor = Preprocessor()<\/div>\ntext = preprocessor.run(text) print(text)<\/div>\n# “I just got the best dinner in my life !!! I recommend”<\/div>\n<\/div>Custom pipeline<\/h3><\/div>If you have a clear idea of what preprocessing functions you want to pipe in your preprocessing pipeline, you can add them in your own Preprocessor.<\/p>\n<\/div>
from nlpretext import Preprocessor<\/div>\nfrom nlpretext.basic.preprocess import (normalize_whitespace, remove_punct, remove_eol_characters, remove_stopwords, lower_text)<\/div>\nfrom nlpretext.social.preprocess import remove_mentions, remove_hashtag, remove_emoji<\/div>\ntext = \"I just got the best dinner in my life @latourdargent !!! I recommend \ud83d\ude00 #food #paris n\"<\/div>\npreprocessor = Preprocessor()<\/div>\npreprocessor.pipe(lower_text)<\/div>\npreprocessor.pipe(remove_mentions)<\/div>\npreprocessor.pipe(remove_hashtag)<\/div>\npreprocessor.pipe(remove_emoji)<\/div>\npreprocessor.pipe(remove_eol_characters)<\/div>\npreprocessor.pipe(remove_stopwords, args=)<\/div>\npreprocessor.pipe(remove_punct)<\/div>\npreprocessor.pipe(normalize_whitespace)<\/div>\ntext = preprocessor.run(text) print(text) # \"dinner life recommend\"<\/div>\n<\/div>NLPretext installation<\/h2><\/div>To install the library please run<\/p>\n<\/div>
pip install nlpretext<\/div>\n<\/div>
The basic module is a catalogue of transversal functions that can be used in any use case. They allow you to handle:<\/p>\n<\/div>
- <\/i><\/span>Bad whitespaces in a text, end of line characters<\/div><\/li>
- <\/i><\/span>
Encoding issues<\/div><\/li>- <\/i><\/span>
\nSpecial characters such as currency symbols, numbers, punctuation marks, latin and non-latin characters<\/p>\n<\/div><\/li>
- <\/i><\/span>
Emails and phone numbers<\/div><\/li><\/ul>from nlpretext.basic.preprocess import replace_emails
\nexample = “I have forwarded this email to obama@whitehouse.gov”
\nexample = replace_emails(example, replace_with=”*EMAIL*”)
\nprint(example)
\n# “I have forwarded this email to *EMAIL*”<\/div>\n<\/div>Social preprocessing<\/h2><\/div>
The social<\/strong> module is a catalogue of handy functions that can be useful when processing social data, such as:<\/p>\n<\/div>
- <\/i><\/span>\n
hashtags extraction\/removal<\/p>\n<\/div><\/li>
- <\/i><\/span>
\nemojis extraction\/removal<\/p>\n<\/div><\/li>
- <\/i><\/span>
\nmentions extraction\/removal<\/p>\n<\/div><\/li>
- <\/i><\/span>
\nhtml tags cleaning<\/p>\n<\/div><\/li><\/ul>
from nlpretext.social.preprocess import extract_emojis
\nexample = “I take care of my skin \ud83d\ude00”
\nexample = extract_emojis(example)
\nprint(example) #[‘:grinning_face:’]<\/div>\n<\/div>Text augmentation<\/h2><\/div>
The augmentation module helps you to generate new texts based on your given examples by modifying some words in the initial ones and to keep associated entities unchanged, if any, in the case of NER tasks. If you want words other than entities to remain unchanged, you can specify it within the stopwords argument. Modifications depend on the chosen method, the ones currently supported by the module are substitutions with synonyms using Wordnet or BERT from the nlpaug library.<\/p>\n<\/div>
Create your end to end pipeline<\/h2><\/div>
Default pipeline<\/h3><\/div>
Our library provides a Preprocessor object to efficiently pipe all preprocessing operations.
\nIf you need to keep all elements of your text and perform minimum cleaning, use the default pipeline. It normalizes whitespaces and removes newlines characters, fixes unicode problems and removes recurrent artifacts from social data such as mentions, hashtags and HTML tags.<\/p>\n<\/div>from nlpretext import Preprocessor<\/div>\ntext = “I just got the best dinner in my life @latourdargent !!! I recommend \ud83d\ude00 #food #paris n”<\/div>\npreprocessor = Preprocessor()<\/div>\ntext = preprocessor.run(text) print(text)<\/div>\n# “I just got the best dinner in my life !!! I recommend”<\/div>\n<\/div>Custom pipeline<\/h3><\/div>
If you have a clear idea of what preprocessing functions you want to pipe in your preprocessing pipeline, you can add them in your own Preprocessor.<\/p>\n<\/div>
from nlpretext import Preprocessor<\/div>\nfrom nlpretext.basic.preprocess import (normalize_whitespace, remove_punct, remove_eol_characters, remove_stopwords, lower_text)<\/div>\nfrom nlpretext.social.preprocess import remove_mentions, remove_hashtag, remove_emoji<\/div>\ntext = \"I just got the best dinner in my life @latourdargent !!! I recommend \ud83d\ude00 #food #paris n\"<\/div>\npreprocessor = Preprocessor()<\/div>\npreprocessor.pipe(lower_text)<\/div>\npreprocessor.pipe(remove_mentions)<\/div>\npreprocessor.pipe(remove_hashtag)<\/div>\npreprocessor.pipe(remove_emoji)<\/div>\npreprocessor.pipe(remove_eol_characters)<\/div>\npreprocessor.pipe(remove_stopwords, args=)<\/div>\npreprocessor.pipe(remove_punct)<\/div>\npreprocessor.pipe(normalize_whitespace)<\/div>\ntext = preprocessor.run(text) print(text) # \"dinner life recommend\"<\/div>\n<\/div>NLPretext installation<\/h2><\/div>
To install the library please run<\/p>\n<\/div>
pip install nlpretext<\/div>\n<\/div> - <\/i><\/span>
- <\/i><\/span>