Package edu.berkeley.nlp.lm.values

Class CompressibleProbBackoffValueContainer

java.lang.Object

edu.berkeley.nlp.lm.values.CompressibleProbBackoffValueContainer

All Implemented Interfaces:

CompressibleValueContainer<ProbBackoffPair>, ProbBackoffValueContainer, ValueContainer<ProbBackoffPair>, java.io.Serializable

public final class CompressibleProbBackoffValueContainer
extends java.lang.Object
implements ProbBackoffValueContainer

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field	Description
protected int	defaultValRank
protected long[]	numNgramsForEachOrder
protected boolean	storeSuffixIndexes
protected int[]	suffixBitsForOrder
protected boolean	useMapValueArray
protected VariableLengthBitCompressor	valueCoder
protected int	valueRadix
protected CustomWidthArray[]	valueRanks
protected int	valueWidth

Constructor Summary

Constructors

Constructor	Description
CompressibleProbBackoffValueContainer(int valueRadix, boolean storePrefixIndexes, long[] numNgramsForEachOrder, float[] probsForRank, float[] backoffsForRank, Indexer<java.lang.Float> probIndexer, int wordWidth, Indexer<java.lang.Float> backoffIndexer, int backoffWidth)
CompressibleProbBackoffValueContainer(LongToIntHashMap countCounter, int valueRadix, boolean storePrefixes, long[] numNgramsForEachOrder)

Method Summary

Modifier and Type	Method	Description
boolean	add(int[] ngram, int startPos, int endPos, int ngramOrder, long offset, long prefixOffset, int word, V val_, long suffixOffset, boolean ngramIsNew)	Adds a new value at the specified offset.
void	clearStorageAfterCompression(int ngramOrder)
void	clearStorageForOrder(int ngramOrder)
CompressibleProbBackoffValueContainer	createFreshValues(long[] numNgramsForEachOrder_)	Creates a fresh value container for copying purposes.
void	decompress(BitStream bits, int ngramOrder, boolean justConsume, ProbBackoffPair outputVal)	Reads and decompresses from the bit stream bits.
float	getBackoff(int ngramOrder, long index)
float	getBackoff(CustomWidthArray valueRanksForNgramOrder, long index)
BitList	getCompressed(long offset, int ngramOrder)	Compresses the value at the given offset into a list of bits.
protected long	getCountRank(long val)
protected ProbBackoffPair	getDefaultVal()
void	getFromOffset(long index, int ngramOrder, ProbBackoffPair outputVal)	Gets the value living at a particular offset.
protected void	getFromRank(long rank, ProbBackoffPair outputVal)
long	getInternalVal(int ngramOrder, long index)
float	getProb(int ngramOrder, long index)
float	getProb(CustomWidthArray valueRanksForOrder, long index)
protected long	getRank(int ngramOrder, long offset)
ProbBackoffPair	getScratchValue()	Creates a fresh value of object (useful for passing as an output parameter)
long	getSuffixOffset(long index, int ngramOrder)
protected int	getSuffixOffsetFromInternalVal(long internalVal)
int	numValueBits(int ngramOrder)
void	setFromOtherValues(ValueContainer<ProbBackoffPair> o)	Destructively sets internal storage from another object.
void	setMap(NgramMap<V> map)	Initializes a value container with the map that contains it
void	setSizeAtLeast(long size, int ngramOrder)	Sets internal storage for size for a particular n-gram order
boolean	storeSuffixoffsets()
void	swap(long a, long b, int ngramOrder)	Swaps values at offsets a and b.
void	trim()	Final clean up of storage.
void	trimAfterNgram(int ngramOrder, long size)	Clear storage after an n-gram order is complete
protected boolean	useValueStoringArray()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface edu.berkeley.nlp.lm.values.ProbBackoffValueContainer

getSuffixOffset

Methods inherited from interface edu.berkeley.nlp.lm.values.ValueContainer

add, clearStorageForOrder, numValueBits, setMap, setSizeAtLeast, storeSuffixoffsets, trimAfterNgram

Field Detail

valueRanks

protected final CustomWidthArray[] valueRanks

storeSuffixIndexes

protected final boolean storeSuffixIndexes

valueCoder

protected final VariableLengthBitCompressor valueCoder

valueRadix

protected final int valueRadix

valueWidth

protected int valueWidth

defaultValRank

protected final int defaultValRank

See Also:

Constant Field Values

numNgramsForEachOrder

protected final long[] numNgramsForEachOrder

suffixBitsForOrder

protected final int[] suffixBitsForOrder

useMapValueArray

protected boolean useMapValueArray

Constructor Detail

CompressibleProbBackoffValueContainer

public CompressibleProbBackoffValueContainer(LongToIntHashMap countCounter,
                                             int valueRadix,
                                             boolean storePrefixes,
                                             long[] numNgramsForEachOrder)

CompressibleProbBackoffValueContainer

public CompressibleProbBackoffValueContainer(int valueRadix,
                                             boolean storePrefixIndexes,
                                             long[] numNgramsForEachOrder,
                                             float[] probsForRank,
                                             float[] backoffsForRank,
                                             Indexer<java.lang.Float> probIndexer,
                                             int wordWidth,
                                             Indexer<java.lang.Float> backoffIndexer,
                                             int backoffWidth)

Parameters:

valueRadix -

storePrefixIndexes -

maxNgramOrder -

hasBackoffValIndexer -

noBackoffValIndexer -

probsAndBackoffsForRank -

probsForRank -

hasBackoffValIndexer -

Method Detail

createFreshValues

public CompressibleProbBackoffValueContainer createFreshValues(long[] numNgramsForEachOrder_)

Description copied from interface: ValueContainer

Creates a fresh value container for copying purposes.

Specified by:

createFreshValues in interface ValueContainer<ProbBackoffPair>

Returns:

getProb

public final float getProb(int ngramOrder,
                           long index)

Specified by:

getProb in interface ProbBackoffValueContainer

getInternalVal

public final long getInternalVal(int ngramOrder,
                                 long index)

getProb

public final float getProb(CustomWidthArray valueRanksForOrder,
                           long index)

getFromOffset

public void getFromOffset(long index,
                          int ngramOrder,
                          ProbBackoffPair outputVal)

Description copied from interface: ValueContainer

Gets the value living at a particular offset.

Specified by:

getFromOffset in interface ValueContainer<ProbBackoffPair>

getBackoff

public final float getBackoff(int ngramOrder,
                              long index)

Specified by:

getBackoff in interface ProbBackoffValueContainer

getBackoff

public final float getBackoff(CustomWidthArray valueRanksForNgramOrder,
                              long index)

getDefaultVal

protected ProbBackoffPair getDefaultVal()

getFromRank

protected void getFromRank(long rank,
                           ProbBackoffPair outputVal)

getScratchValue

public ProbBackoffPair getScratchValue()

Description copied from interface: ValueContainer

Creates a fresh value of object (useful for passing as an output parameter)

Specified by:

getScratchValue in interface ProbBackoffValueContainer

Specified by:

getScratchValue in interface ValueContainer<ProbBackoffPair>

Returns:

setFromOtherValues

public void setFromOtherValues(ValueContainer<ProbBackoffPair> o)

Description copied from interface: ValueContainer

Destructively sets internal storage from another object.

Specified by:

setFromOtherValues in interface ValueContainer<ProbBackoffPair>

trim

public void trim()

Description copied from interface: ValueContainer

Final clean up of storage.

Specified by:

trim in interface ValueContainer<ProbBackoffPair>

getCountRank

protected long getCountRank(long val)

getCompressed

public BitList getCompressed(long offset,
                             int ngramOrder)

Description copied from interface: CompressibleValueContainer

Compresses the value at the given offset into a list of bits.

Specified by:

getCompressed in interface CompressibleValueContainer<ProbBackoffPair>

Returns:

decompress

public final void decompress(BitStream bits,
                             int ngramOrder,
                             boolean justConsume,
                             ProbBackoffPair outputVal)

Description copied from interface: CompressibleValueContainer

Reads and decompresses from the bit stream bits.

Specified by:

decompress in interface CompressibleValueContainer<ProbBackoffPair>

justConsume - If true, nothing is returned, and the function simply consumes the appropriate number of bits from the BitStream.

setMap

public void setMap(NgramMap<V> map)

Description copied from interface: ValueContainer

Initializes a value container with the map that contains it

Specified by:

setMap in interface ValueContainer<V extends LongRepresentable<V>>

useValueStoringArray

protected boolean useValueStoringArray()

swap

public void swap(long a,
                 long b,
                 int ngramOrder)

Description copied from interface: CompressibleValueContainer

Swaps values at offsets a and b.

Specified by:

swap in interface CompressibleValueContainer<V extends LongRepresentable<V>>

add

public boolean add(int[] ngram,
                   int startPos,
                   int endPos,
                   int ngramOrder,
                   long offset,
                   long prefixOffset,
                   int word,
                   V val_,
                   long suffixOffset,
                   boolean ngramIsNew)

Description copied from interface: ValueContainer

Adds a new value at the specified offset.

Specified by:

add in interface ValueContainer<V extends LongRepresentable<V>>

ngramOrder - As always, ngramOrder is 0-based (0=unigram)

Returns:

Whether or not the add was successful

setSizeAtLeast

public void setSizeAtLeast(long size,
                           int ngramOrder)

Description copied from interface: ValueContainer

Sets internal storage for size for a particular n-gram order

Specified by:

setSizeAtLeast in interface ValueContainer<V extends LongRepresentable<V>>

getSuffixOffset

public long getSuffixOffset(long index,
                            int ngramOrder)

getSuffixOffsetFromInternalVal

protected int getSuffixOffsetFromInternalVal(long internalVal)

Parameters:

internalVal -

Returns:

getRank

protected long getRank(int ngramOrder,
                       long offset)

clearStorageAfterCompression

public void clearStorageAfterCompression(int ngramOrder)

Specified by:

clearStorageAfterCompression in interface CompressibleValueContainer<V extends LongRepresentable<V>>

trimAfterNgram

public void trimAfterNgram(int ngramOrder,
                           long size)

Description copied from interface: ValueContainer

Clear storage after an n-gram order is complete

Specified by:

trimAfterNgram in interface ValueContainer<V extends LongRepresentable<V>>

clearStorageForOrder

public void clearStorageForOrder(int ngramOrder)

Specified by:

clearStorageForOrder in interface ValueContainer<V extends LongRepresentable<V>>

storeSuffixoffsets

public boolean storeSuffixoffsets()

Specified by:

storeSuffixoffsets in interface ValueContainer<V extends LongRepresentable<V>>

numValueBits

public int numValueBits(int ngramOrder)

Specified by:

numValueBits in interface ValueContainer<V extends LongRepresentable<V>>