神刀安全网

Stemming and Spell Checking in R

(This article was first published on OpenCPU , and kindly contributed toR-bloggers)

Stemming and Spell Checking in R

Last week we introduced the new hunspell R package. This week a new version was released which adds support for additional languages and text analysis features.

Additional languages

By default hunspell uses the US English dictionary en_US but the new version allows for

checking and analyzing in other languages as well. The ?hunspell help page has detailed

instructions on how to install additional dictionaries.

> library(hunspell) > hunspell_info("ru_RU") $dict [1] "/Users/jeroen/workspace/hunspell/tests/testdict/ru_RU.dic"  $encoding [1] "UTF-8"  $wordchars [1] NA 
> hunspell("чёртова карова", dict = "ru_RU")[[1]] [1] "карова" 

It turned out this feature was much more difficult to implement than I expected. Much of the Hunspell

library dates from before UTF-8 became popular and therefore many dictionaries use local 8 bit character encodings such as ISO-8859-1 for English and KOI8-R for Russian. To spell check in these languages, the character encoding of the document text has to match that of the dictionary. However R only supports latin and UTF-8

so we need to convert strings in C with iconv , which opens up a new can of worms. Anyway it should

all work now.

@opencpu hunspell_stem could be very useful in interpretation issues of e.g. #wordclouds .

— Jelle Geertsma (@rdatasculptor) March 14, 2016

Text analysis and wordclouds

In last weeks post we showed how to parse and spell

check a latex file:

# Check an entire latex document library(hunspell) setwd(tempdir()) download.file("http://arxiv.org/e-print/1406.4806v1", "1406.4806v1.tar.gz",  mode = "wb") untar("1406.4806v1.tar.gz") text <- readLines("content.tex", warn = FALSE) bad_words <- hunspell(text, format = "latex") sort(unique(unlist(bad_words))) 

The new version also exposes the parser directly, so you can easily extract words and derive the stems to summarize some text, for example to display in a wordcloud.

# Summarize text by stems (e.g. for wordcloud) allwords <- hunspell_parse(text, format = "latex") stems <- unlist(hunspell_stem(unlist(allwords))) words <- head(sort(table(stems), decreasing = TRUE), 200) 

转载本站任何文章请注明:转载至神刀安全网,谢谢神刀安全网 » Stemming and Spell Checking in R

分享到:更多 ()

评论 抢沙发

  • 昵称 (必填)
  • 邮箱 (必填)
  • 网址
分享按钮