ros_tokens. rosette

Divides the input into tokens. The response contains a list of tokens.

ros_tokens(content, genre = NULL, language = NULL)

Arguments

content: either a URI or character vector of content to process
genre: document genre (optional)
language: Language: ISO 639-3 code (optional)

Examples

ros_tokens("北京大学生物系主任办公室内部会议")
#> $tokens
#> [1] "北京大学" "生物系"   "主任"     "办公室"   "内部"     "会议"    
#> 
#>