Text Preprocessing

Tokenization

Split a document into sentences, then each sentence into tokens. Every decision here (keep km/h together? split can't?) changes which terms end up in the index.

Sentences: Punkt

PunktSentenceTokenizer splits the example document into its 4 sentences. Punkt is unsupervised: it learns abbreviations, collocations and typical sentence starters from a corpus of text.

• PunktSentenceTokenizer(): the notebook's call, with no training text, so it knows no abbreviations.
• sent_tokenize(text): loads the Punkt model pre-trained on English (punkt_tab). Lab 01's preprocess_document uses it.

Try it: split into sentences

The outputs were computed with NLTK 3.9. Compare the two models on Abbreviations.

Words: four NLTK tokenizers

• WhitespaceTokenizer: splits only on spaces, tabs and newlines, so punctuation stays attached (Portugal,).
• WordPunctTokenizer: the regex \w+|[^\w\s]+: runs of letters and digits, and runs of punctuation.
• TreebankWordTokenizer: follows the Penn Treebank conventions. It splits contractions (ca n't), separates most punctuation, and keeps 50,000 and 366.88 whole. It expects one sentence at a time, so only a final period is split off.
• TweetTokenizer: uses regexes for URLs, emoticons, @handles, #hashtags, e-mails and phone numbers, and then for words.

Try it: compare the tokenizers

The first row has the notebook's examples, with its comments, and the second row has more cases to try. You can also type your own text. Punctuation-only tokens are red.

💡 The widgets run JavaScript ports of the NLTK 3.9 tokenizers and stemmers. We checked them against NLTK on all 33,228 sentences of mini_newsgroups and all 56,530 distinct words in them, and the output was identical.

from nltk.tokenize import PunktSentenceTokenizer
from nltk.tokenize import TweetTokenizer, WhitespaceTokenizer, WordPunctTokenizer, TreebankWordTokenizer

example_document = """
Hurricane-force winds have struck central and northern Portugal, leaving 300,000 homes without power.

The remnants of Hurricane Leslie swept in overnight on Saturday, with winds gusting up to 176km/h (109mph).

Civil defence officials said 27 people suffered minor injuries, with localised flooding, hundreds of trees uprooted and a number of flights cancelled.

The storm, one of the most powerful to ever hit the country, is now passing over northern Spain.
"""
sent_tokenizer = PunktSentenceTokenizer()
sentences = sent_tokenizer.tokenize(example_document)   # 4 sentences

tweet_tok, whsp_tok, wordpunct_tok, treebank_tok = TweetTokenizer(), WhitespaceTokenizer(), WordPunctTokenizer(), TreebankWordTokenizer()

print(whsp_tok.tokenize(sentences[0]))
# ['Hurricane-force', 'winds', 'have', 'struck', 'central', 'and', 'northern', 'Portugal,', 'leaving', '300,000', 'homes', 'without', 'power.']
print(wordpunct_tok.tokenize('On a $50,000 mortgage of 30 years at 8 percent, the monthly payment would be $366.88.'))
# ['On', 'a', '$', '50', ',', '000', 'mortgage', ..., 'be', '$', '366', '.', '88', '.']
print(treebank_tok.tokenize('On a $50,000 mortgage of 30 years at 8 percent, the monthly payment would be $366.88.'))
# ['On', 'a', '$', '50,000', 'mortgage', ..., 'be', '$', '366.88', '.']
print(treebank_tok.tokenize('My email is admin@sth.com'))    # ['My', 'email', 'is', 'admin', '@', 'sth.com']
print(tweet_tok.tokenize('My email is admin@sth.com'))       # ['My', 'email', 'is', 'admin@sth.com']
print(tweet_tok.tokenize('Last sunny day before winter :) #sunisenergy'))
# ['Last', 'sunny', 'day', 'before', 'winter', ':)', '#sunisenergy']
print(tweet_tok.tokenize('Here is my ip address 192.33.24.45'))   # we might want to fix this
# ['Here', 'is', 'my', 'ip', 'address', '192.33', '.', '24.45']
print(tweet_tok.tokenize("I can't stop singing that song!"))     # we might want to split can't in can, 't instead
# ['I', "can't", 'stop', 'singing', 'that', 'song', '!']

Querying with the same preprocessing

Your query methods must apply the same steps to the query words. A query term matches only if it went through the same tokenizer, case folding, stopword list and stemmer as the documents.

• Positive: stemming and lemmatization match other forms of a word (higher recall).
• Negative: unrelated words merge, and removed words (stopwords, numbers) make parts of the query disappear (lower precision, or no answer).

Try it: query the Lab 01 index and your pipeline

An AND query against two indexes of sci.crypt: the Lab 01 index (Tweet, lower-case, no punctuation) and the one from the pipeline above. Each scenario sets the pipeline and a query, and then you can change either.

💡 Green documents are found only by your pipeline, red ones only by the Lab 01 index. Whether a gain is good depends on the user: keys → key helps, data → every Date: header does not.