Tokenizer

All Known Subinterfaces:

CompoundWordTokenizer, SentenceTokenizer

All Known Implementing Classes:

SimpleSentenceTokenizer, SRXSentenceTokenizer, WordTokenizer
```
public interface Tokenizer
```
Interface for classes that tokenize text into smaller units.

- Method Summary
  
  All Methods Instance Methods Abstract Methods
  Modifier and Type Method Description
  
  java.util.List<java.lang.String> tokenize(java.lang.String text)

Method Detail

tokenize

java.util.List<java.lang.String> tokenize(java.lang.String text)