Package org.languagetool.languagemodel

Class LuceneSingleIndexLanguageModel

java.lang.Object

org.languagetool.languagemodel.BaseLanguageModel

org.languagetool.languagemodel.LuceneSingleIndexLanguageModel

All Implemented Interfaces:

AutoCloseable, LanguageModel

public class LuceneSingleIndexLanguageModel
extends BaseLanguageModel

Information about ngram occurrences, taken from Lucene indexes (one index per ngram level). This is not a real language model as it only returns information about occurrence counts but has no probability calculation, especially not for the case with 0 occurrences.

Since:

3.2

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
protected static class	LuceneSingleIndexLanguageModel.LuceneSearcher

Field Summary

Fields

Modifier and Type	Field	Description
private static final Map<File,LuceneSingleIndexLanguageModel.LuceneSearcher>	dirToSearcherMap
private final List<File>	indexes
private final Map<Integer,LuceneSingleIndexLanguageModel.LuceneSearcher>	luceneSearcherMap
private final long	maxNgram
private final File	topIndexDir

Fields inherited from interface org.languagetool.languagemodel.LanguageModel

GOOGLE_SENTENCE_END, GOOGLE_SENTENCE_START

Constructor Summary

Constructors

Constructor	Description
LuceneSingleIndexLanguageModel(int maxNgram)
LuceneSingleIndexLanguageModel(File topIndexDir)

Method Summary

Modifier and Type	Method	Description
private void	addIndex(File topIndexDir, int ngramSize)
static void	clearCaches()	Only used internally.
void	close()
protected void	doValidateDirectory(File topIndexDir)
private LuceneSingleIndexLanguageModel.LuceneSearcher	getCachedLuceneSearcher(File indexDir)
long	getCount(String token1)	Get the occurrence count for token.
long	getCount(List<String> tokens)	Get the occurrence count for the given token sequence.
private long	getCount(org.apache.lucene.index.Term term, LuceneSingleIndexLanguageModel.LuceneSearcher luceneSearcher)
protected LuceneSingleIndexLanguageModel.LuceneSearcher	getLuceneSearcher(int ngramSize)
long	getTotalTokenCount()
String	toString()
static void	validateDirectory(File topIndexDir)	Throw RuntimeException is the given directory does not seem to be a valid ngram top directory with sub directories 1grams etc.

Methods inherited from class org.languagetool.languagemodel.BaseLanguageModel

getPseudoProbability, getPseudoProbabilityStupidBackoff

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Details

dirToSearcherMap

private static final Map<File,LuceneSingleIndexLanguageModel.LuceneSearcher> dirToSearcherMap

indexes

private final List<File> indexes

luceneSearcherMap

private final Map<Integer,LuceneSingleIndexLanguageModel.LuceneSearcher> luceneSearcherMap

topIndexDir

private final File topIndexDir

maxNgram

private final long maxNgram

Constructor Details

LuceneSingleIndexLanguageModel

public LuceneSingleIndexLanguageModel(File topIndexDir)

Parameters:

topIndexDir - a directory which contains at least another sub directory called 3grams, which is a Lucene index with ngram occurrences as created by org.languagetool.dev.FrequencyIndexCreator.

LuceneSingleIndexLanguageModel

@Experimental
public LuceneSingleIndexLanguageModel(int maxNgram)

Method Details

validateDirectory

public static void validateDirectory(File topIndexDir)

Throw RuntimeException is the given directory does not seem to be a valid ngram top directory with sub directories 1grams etc.

Since:

3.0

clearCaches

@Experimental
public static void clearCaches()

Only used internally.

Since:

3.2

doValidateDirectory

protected void doValidateDirectory(File topIndexDir)

addIndex

private void addIndex(File topIndexDir,
 int ngramSize)

getCount

public long getCount(List<String> tokens)

Description copied from class: BaseLanguageModel

Get the occurrence count for the given token sequence.

Specified by:

getCount in class BaseLanguageModel

getCount

public long getCount(String token1)

Description copied from class: BaseLanguageModel

Get the occurrence count for token.

Specified by:

getCount in class BaseLanguageModel

getTotalTokenCount

public long getTotalTokenCount()

Specified by:

getTotalTokenCount in class BaseLanguageModel

getLuceneSearcher

protected LuceneSingleIndexLanguageModel.LuceneSearcher getLuceneSearcher(int ngramSize)

getCachedLuceneSearcher

private LuceneSingleIndexLanguageModel.LuceneSearcher getCachedLuceneSearcher(File indexDir)

getCount

private long getCount(org.apache.lucene.index.Term term,
 LuceneSingleIndexLanguageModel.LuceneSearcher luceneSearcher)

close

public void close()

toString

public String toString()

Overrides:

toString in class Object