Class SnappyOutputStream

java.lang.Object

java.io.OutputStream

org.xerial.snappy.SnappyOutputStream

All Implemented Interfaces:

Closeable, Flushable, AutoCloseable

Direct Known Subclasses:

SnappyHadoopCompatibleOutputStream

public class SnappyOutputStream
extends OutputStream

This class implements a stream filter for writing compressed data using Snappy.

The input data is blocked into 32kb size (in default), and each block is compressed and then passed to the given OutputStream.

The output data format is:

1. snappy codec header defined in SnappyCodec (8 bytes)
2. codec version (4 bytes integer, at least 1) and compatible version (4 bytes integer)
3. compressed block 1 : a pair of (compressed data size [4 byte integer. Big-endian], compressed data...)
4. compressed block 2
5. ...

Note that the compressed data created by SnappyOutputStream cannot be uncompressed by Snappy.uncompress(byte[]) since the output formats of Snappy.compress(byte[]) and SnappyOutputStream are different. Use SnappyInputStream for uncompress the data generated by SnappyOutputStream.

Field Summary

Fields

Modifier and Type	Field	Description
private final int	blockSize
private boolean	closed
(package private) static final int	DEFAULT_BLOCK_SIZE
private boolean	headerWritten
protected byte[]	inputBuffer
private final BufferAllocator	inputBufferAllocator
private int	inputCursor
static final int	MAX_BLOCK_SIZE
(package private) static final int	MIN_BLOCK_SIZE
protected final OutputStream	out
protected byte[]	outputBuffer
private final BufferAllocator	outputBufferAllocator
private int	outputCursor

Constructor Summary

Constructors

Constructor	Description
SnappyOutputStream(OutputStream out)
SnappyOutputStream(OutputStream out, int blockSize)
SnappyOutputStream(OutputStream out, int blockSize, BufferAllocatorFactory bufferAllocatorFactory)

Method Summary

Modifier and Type	Method	Description
void	close()	close the stream
protected void	compressInput()
protected void	dumpOutput()
void	flush()
private boolean	hasSufficientOutputBufferFor(int inputSize)
void	rawWrite(Object array, int byteOffset, int byteLength)	Compress the raw byte array data.
(package private) static int	readInt(byte[] buffer, int pos)
void	write(byte[] b, int byteOffset, int byteLength)
void	write(double[] f)	Compress the input array data
void	write(double[] f, int off, int len)	Compress the input double array data
void	write(float[] f)	Compress the input array data
void	write(float[] f, int off, int len)	Compress the input float array data
void	write(int b)	Writes the specified byte to this output stream.
void	write(int[] f)	Compress the input array data
void	write(int[] f, int off, int len)	Compress the input int array data
void	write(long[] d)	Compress the input array data
void	write(long[] d, int off, int len)	Compress the input long array data
void	write(short[] f)	Compress the input array data
void	write(short[] f, int off, int len)	Compress the input short array data
protected void	writeBlockPreemble()	Optionally write a preemble before a block.
protected void	writeCurrentDataSize()
protected int	writeHeader()
(package private) static void	writeInt(byte[] dst, int offset, int v)

Methods inherited from class OutputStream

nullOutputStream, write

Methods inherited from class Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

MAX_BLOCK_SIZE

public static final int MAX_BLOCK_SIZE

See Also:

• Constant Field Values

MIN_BLOCK_SIZE

static final int MIN_BLOCK_SIZE

See Also:

• Constant Field Values

DEFAULT_BLOCK_SIZE

static final int DEFAULT_BLOCK_SIZE

See Also:

• Constant Field Values

out

protected final OutputStream out

blockSize

private final int blockSize

inputBufferAllocator

private final BufferAllocator inputBufferAllocator

outputBufferAllocator

private final BufferAllocator outputBufferAllocator

inputBuffer

protected byte[] inputBuffer

outputBuffer

protected byte[] outputBuffer

inputCursor

private int inputCursor

outputCursor

private int outputCursor

headerWritten

private boolean headerWritten

closed

private boolean closed

Constructor Details

SnappyOutputStream

public SnappyOutputStream(OutputStream out)

SnappyOutputStream

public SnappyOutputStream(OutputStream out,
 int blockSize)

Parameters:

out -

blockSize - byte size of the internal buffer size

Throws:

IllegalArgumentException - when blockSize is larger than 512 MiB

SnappyOutputStream

public SnappyOutputStream(OutputStream out,
 int blockSize,
 BufferAllocatorFactory bufferAllocatorFactory)

Method Details

write

public void write(byte[] b,
 int byteOffset,
 int byteLength)
           throws IOException

Overrides:

write in class OutputStream

Throws:

IOException

write

public void write(long[] d,
 int off,
 int len)
           throws IOException

Compress the input long array data

Parameters:

d - input array

off - offset in the array

len - the number of elements in the array to copy

Throws:

IOException

write

public void write(double[] f,
 int off,
 int len)
           throws IOException

Compress the input double array data

Parameters:

f - input array

off - offset in the array

len - the number of elements in the array to copy

Throws:

IOException

write

public void write(float[] f,
 int off,
 int len)
           throws IOException

Compress the input float array data

Parameters:

f - input array

off - offset in the array

len - the number of elements in the array to copy

Throws:

IOException

write

public void write(int[] f,
 int off,
 int len)
           throws IOException

Compress the input int array data

Parameters:

f - input array

off - offset in the array

len - the number of elements in the array to copy

Throws:

IOException

write

public void write(short[] f,
 int off,
 int len)
           throws IOException

Compress the input short array data

Parameters:

f - input array

off - offset in the array

len - the number of elements in the array to copy

Throws:

IOException

write

public void write(long[] d)
           throws IOException

Compress the input array data

Parameters:

d -

Throws:

IOException

write

public void write(double[] f)
           throws IOException

Compress the input array data

Parameters:

f -

Throws:

IOException

write

public void write(float[] f)
           throws IOException

Compress the input array data

Parameters:

f -

Throws:

IOException

write

public void write(int[] f)
           throws IOException

Compress the input array data

Parameters:

f -

Throws:

IOException

write

public void write(short[] f)
           throws IOException

Compress the input array data

Parameters:

f -

Throws:

IOException

hasSufficientOutputBufferFor

private boolean hasSufficientOutputBufferFor(int inputSize)

rawWrite

public void rawWrite(Object array,
 int byteOffset,
 int byteLength)
              throws IOException

Compress the raw byte array data.

Parameters:

array - array data of any type (e.g., byte[], float[], long[], ...)

byteOffset -

byteLength -

Throws:

IOException

write

public void write(int b)
           throws IOException

Writes the specified byte to this output stream. The general contract for write is that one byte is written to the output stream. The byte to be written is the eight low-order bits of the argument b. The 24 high-order bits of b are ignored.

Specified by:

write in class OutputStream

Throws:

IOException

flush

public void flush()
           throws IOException

Specified by:

flush in interface Flushable

Overrides:

flush in class OutputStream

Throws:

IOException

writeInt

static void writeInt(byte[] dst,
 int offset,
 int v)

readInt

static int readInt(byte[] buffer,
 int pos)

dumpOutput

protected void dumpOutput()
                   throws IOException

Throws:

IOException

compressInput

protected void compressInput()
                      throws IOException

Throws:

IOException

writeHeader

protected int writeHeader()

writeBlockPreemble

protected void writeBlockPreemble()

Optionally write a preemble before a block. Hadoop requires the actual block data size being written. This base implementation does nothing. Derive implementation can call writeCurrentDataSize().

writeCurrentDataSize

protected void writeCurrentDataSize()

close

public void close()
           throws IOException

close the stream

Specified by:

close in interface AutoCloseable

Specified by:

close in interface Closeable

Overrides:

close in class OutputStream

Throws:

IOException