Package org.languagetool.tagging.disambiguation

Class MultiWordChunker2

java.lang.Object

org.languagetool.tagging.disambiguation.AbstractDisambiguator

org.languagetool.tagging.disambiguation.MultiWordChunker2

All Implemented Interfaces:

Disambiguator

public class MultiWordChunker2
extends AbstractDisambiguator

Multiword tagger-chunker. Note: currently does not support:

• overlapping tagging (first matching multiword entry wins)

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
private static final class	MultiWordChunker2.MultiWordEntry

Field Summary

Fields

Modifier and Type	Field	Description
private final boolean	allowFirstCapitalized
private final String	filename
private boolean	removeOtherReadings
private String	tagFormat
private Map<String,List<MultiWordChunker2.MultiWordEntry>>	tokenToPosTagMap
private static final String	WRAP_TAG

Constructor Summary

Constructors

Constructor	Description
MultiWordChunker2(String filename)
MultiWordChunker2(String filename, boolean allowFirstCapitalized)

Method Summary

Modifier and Type	Method	Description
AnalyzedSentence	disambiguate(AnalyzedSentence input)	Implements multiword POS tags, e.g., <ELLIPSIS> for ellipsis (...) start, and </ELLIPSIS> for ellipsis end.
private MultiWordChunker2.MultiWordEntry	findMultiwordEntry(AnalyzedTokenReadings[] inputTokens, int startingPosition, List<MultiWordChunker2.MultiWordEntry> multiwordItems)
protected String	formatPosTag(String posTag, int position, int multiwordLength)	Override this method if you want format POS tag differently
private boolean	isMatching(AnalyzedTokenReadings[] inputTokens, int startingPosition, MultiWordChunker2.MultiWordEntry multiWordEntry)
private void	lazyInit()
private List<String>	loadWords(InputStream stream)
protected boolean	matches(String matchText, AnalyzedTokenReadings inputTokens)
protected AnalyzedTokenReadings	prepareNewReading(String tokens, String tok, AnalyzedTokenReadings token, String tag)
private AnalyzedTokenReadings	setAndAnnotate(AnalyzedTokenReadings oldReading, AnalyzedToken newReading)
void	setRemoveOtherReadings(boolean removeOtherReadings)
void	setWrapTag(boolean wrapTag)

Methods inherited from class org.languagetool.tagging.disambiguation.AbstractDisambiguator

preDisambiguate

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

WRAP_TAG

private static final String WRAP_TAG

See Also:

• Constant Field Values

filename

private final String filename

allowFirstCapitalized

private final boolean allowFirstCapitalized

removeOtherReadings

private boolean removeOtherReadings

tagFormat

private String tagFormat

tokenToPosTagMap

private Map<String,List<MultiWordChunker2.MultiWordEntry>> tokenToPosTagMap

Constructor Details

MultiWordChunker2

public MultiWordChunker2(String filename)

Parameters:

filename - file text with multiwords and tags

MultiWordChunker2

public MultiWordChunker2(String filename,
 boolean allowFirstCapitalized)

Parameters:

filename - file text with multiwords and tags

allowFirstCapitalized - if set to true, first word of the multiword can be capitalized

Method Details

setRemoveOtherReadings

public void setRemoveOtherReadings(boolean removeOtherReadings)

Parameters:

removeOtherReadings - If true and multiword matches other readings will be removed

setWrapTag

public void setWrapTag(boolean wrapTag)

Parameters:

wrapTag - If true the tag will be wrapped with < and >

formatPosTag

protected String formatPosTag(String posTag,
 int position,
 int multiwordLength)

Override this method if you want format POS tag differently

Parameters:

posTag - POS tag for the multiword

position - Position of the token in the multiword

Returns:

Returns formatted POS tag for the multiword

lazyInit

private void lazyInit()

disambiguate

public AnalyzedSentence disambiguate(AnalyzedSentence input)

Implements multiword POS tags, e.g., <ELLIPSIS> for ellipsis (...) start, and </ELLIPSIS> for ellipsis end.

Parameters:

input - The tokens to be chunked.

Returns:

AnalyzedSentence with additional markers.

findMultiwordEntry

private MultiWordChunker2.MultiWordEntry findMultiwordEntry(AnalyzedTokenReadings[] inputTokens,
 int startingPosition,
 List<MultiWordChunker2.MultiWordEntry> multiwordItems)

isMatching

private boolean isMatching(AnalyzedTokenReadings[] inputTokens,
 int startingPosition,
 MultiWordChunker2.MultiWordEntry multiWordEntry)

matches

protected boolean matches(String matchText,
 AnalyzedTokenReadings inputTokens)

prepareNewReading

protected AnalyzedTokenReadings prepareNewReading(String tokens,
 String tok,
 AnalyzedTokenReadings token,
 String tag)

setAndAnnotate

private AnalyzedTokenReadings setAndAnnotate(AnalyzedTokenReadings oldReading,
 AnalyzedToken newReading)

loadWords

private List<String> loadWords(InputStream stream)