Class SegmentedFile

java.lang.Object

org.ojalgo.netio.SegmentedFile

All Implemented Interfaces:

AutoCloseable

public class SegmentedFile
extends Object
implements AutoCloseable

Divides a large file in segments and then reads those (in parallel) using memory mapped files (memory mapped file segments). Basic usage:

1. Call newBuilder(File) to get a SegmentedFile.Builder or of(File) to get a SegmentedFile directly with the default settings.
2. Then for each of the segments obtained from segments(): Call newTextLineReader(Segment) and using that read all the lines.

Multiple threads can be used to read the segments in parallel. Each thread just works on one segment at a time, and there's a new TextLineReader for each segment.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static final class	SegmentedFile.Builder
static final class	SegmentedFile.Segment

Field Summary

Fields

Modifier and Type	Field	Description
private final FileChannel	myFileChannel
private final RandomAccessFile	myRandomAccessFile
private final SegmentedFile.Segment[]	mySegments

Constructor Summary

Constructors

Constructor	Description
SegmentedFile(RandomAccessFile file, SegmentedFile.Segment[] segments)

Method Summary

Modifier and Type	Method	Description
void	close()
(package private) SegmentedFile.Segment[]	getSegments()
static SegmentedFile.Builder	newBuilder(File file)
<T> FromFileReader<T>	newDataReader(DataReader.Deserializer<T> deserializer)
<T> DataReader<T>	newDataReader(SegmentedFile.Segment segment, DataReader.Deserializer<T> deserializer)
<T> Supplier<FromFileReader<T>>	newSequencedFactory(Function<SegmentedFile.Segment, FromFileReader<T>> factory)	Each reader instantiated by this factory will read from the segments in sequence, until all of them are done.
FromFileReader<String>	newTextLineReader()
TextLineReader	newTextLineReader(SegmentedFile.Segment segment)	Call this once for each file segment, and use the returned TextLineReader to read the file segment.
<T> FromFileReader<T>	newTextLineReader(TextLineReader.Parser<T> parser)
static SegmentedFile	of(File file)
List<SegmentedFile.Segment>	segments()

Methods inherited from class Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

myFileChannel

private final FileChannel myFileChannel

myRandomAccessFile

private final RandomAccessFile myRandomAccessFile

mySegments

private final SegmentedFile.Segment[] mySegments

Constructor Details

SegmentedFile

SegmentedFile(RandomAccessFile file,
 SegmentedFile.Segment[] segments)

Method Details

newBuilder

public static SegmentedFile.Builder newBuilder(File file)

of

public static SegmentedFile of(File file)

close

public void close()

Specified by:

close in interface AutoCloseable

newDataReader

public <T> FromFileReader<T> newDataReader(DataReader.Deserializer<T> deserializer)

newDataReader

public <T> DataReader<T> newDataReader(SegmentedFile.Segment segment,
 DataReader.Deserializer<T> deserializer)

newSequencedFactory

public <T> Supplier<FromFileReader<T>> newSequencedFactory(Function<SegmentedFile.Segment, FromFileReader<T>> factory)

Each reader instantiated by this factory will read from the segments in sequence, until all of them are done. The idea is that you can create multiple readers and have them work in parallel (each segment will only be read once by one of the readers). If you only instantiate 1 reader, then maybe you shouldn't have created file segments in the first place.

newTextLineReader

public FromFileReader<String> newTextLineReader()

newTextLineReader

public TextLineReader newTextLineReader(SegmentedFile.Segment segment)

Call this once for each file segment, and use the returned TextLineReader to read the file segment. The TextLineReader is not thread safe, and should only be used by a single thread.

The segment is a range of bytes in the file. The TextLineReader will read the bytes in the range [offset, offset + size). The segments are obtained from segments(). The segments are sorted in ascending order by offset.

newTextLineReader

public <T> FromFileReader<T> newTextLineReader(TextLineReader.Parser<T> parser)

segments

public List<SegmentedFile.Segment> segments()

getSegments

SegmentedFile.Segment[] getSegments()