Package com.kohlschutter.boilerpipe.util

Class UnicodeTokenizer

java.lang.Object

com.kohlschutter.boilerpipe.util.UnicodeTokenizer

public class UnicodeTokenizer
extends java.lang.Object

Tokenizes text according to Unicode word boundaries and strips off non-word characters.

Field Summary

Fields

Modifier and Type	Field	Description
private static java.util.regex.Pattern	PAT_NOT_WORD_BOUNDARY
private static java.util.regex.Pattern	PAT_WORD_BOUNDARY

Constructor Summary

Constructors

Constructor	Description
UnicodeTokenizer()

Method Summary

Modifier and Type	Method	Description
static java.lang.String[]	tokenize(java.lang.CharSequence text)	Tokenizes the text and returns an array of tokens.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

PAT_WORD_BOUNDARY

private static final java.util.regex.Pattern PAT_WORD_BOUNDARY

PAT_NOT_WORD_BOUNDARY

private static final java.util.regex.Pattern PAT_NOT_WORD_BOUNDARY

Constructor Detail

UnicodeTokenizer

public UnicodeTokenizer()

Method Detail

tokenize

public static java.lang.String[] tokenize(java.lang.CharSequence text)

Tokenizes the text and returns an array of tokens.

Parameters:

text - The text

Returns:

The tokens