Class BatchNode<T>

java.lang.Object

org.ojalgo.data.batch.BatchNode<T>

public final class BatchNode<T>
extends Object

A batch processing data node for when there's no way to fit the data in memory.

Data is stored in sharded files, and data is written/consumed and processed concurrently.

The data is processed in batches. Each batch is processed in a single thread. The number of threads is controlled by

invalid reference

#parallelism(IntSupplier)

.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static final class	BatchNode.Builder<T>
private static final class	BatchNode.TwoStepWrapper<T>

Field Summary

Fields

Modifier and Type	Field	Description
private static final Consumer<Boolean>	DUMMY
private final ToIntFunction<T>	myDistributor
private final DataInterpreter<T>	myInterpreter
private final IntSupplier	myParallelism
private final ProcessingService	myProcessor
private final int	myQueueCapacity
private Function<File, FromFileReader<T>>	myReaderFactory
private final Throughput	myReaderManager
private final ShardedFile	myShards
private final Throughput	myWriterManger

Constructor Summary

Constructors

Constructor	Description
BatchNode(BatchNode.Builder<T> builder)

Method Summary

Modifier and Type	Method	Description
void	dispose()	Dispose of this node and explicitly delete all files.
private Function<File, FromFileReader<T>>	getReaderFactory()
static <T> BatchNode.Builder<T>	newBuilder(File directory, DataInterpreter<T> interpreter)
static <T> BatchNode<T>	newInstance(File directory, DataInterpreter<T> interpreter)
(package private) FromFileReader<T>	newReader(File file)
ToFileWriter<T>	newWriter()
private void	process(File shard, Consumer<T> consumer)
private <R, A extends TwoStepMapper<T,R>> void	processAggregators(File shard, Supplier<A> aggregatorFactory, Consumer<A> processor)
void	processAll(Consumer<T> processor)	Process each and every item individually
void	processAll(Supplier<Consumer<T>> processorFactory)	Similar to processAll(Consumer) but you provide a consumer constructor/factory rather than a specific consumer.
<R, A extends TwoStepMapper<T,R>> void	processCombineable(Supplier<A> aggregatorFactory, Consumer<A> processor)	Similar to processMergeable(Supplier, Consumer) but the processor is called with the aggregator instance itself rather than its extracted results.
<R> void	processMapped(Supplier<? extends TwoStepMapper<T,R>> aggregatorFactory, Consumer<R> processor)	Deprecated. v54 Use invalid reference #processMergeable(Supplier<? extends TwoStepMapper<T, H>>,Consumer<H>) instead
<R, A extends TwoStepMapper<T,R>> void	processMergeable(Supplier<A> aggregatorFactory, Consumer<R> processor)	Each shard is processed/aggregated separately by a TwoStepMapper instance.
private <R, A extends TwoStepMapper<T,R>> void	processResults(File shard, Supplier<A> aggregatorFactory, Consumer<R> processor)
<R, A extends TwoStepMapper.Combineable<T,R,A>> R	reduceByCombining(Supplier<A> aggregatorFactory)	Calls processCombineable(Supplier, Consumer) with the invalid reference TwoStepMapper.Combineable#merge(Object) method of a global TwoStepMapper.Combineable instance as the consumer.
<R, A extends TwoStepMapper.Mergeable<T,R>> R	reduceByMerging(Supplier<A> aggregatorFactory)	Calls processMergeable(Supplier, Consumer) with the TwoStepMapper.Mergeable.merge(Object) method of a global TwoStepMapper.Mergeable instance as the consumer.
<R, A extends TwoStepMapper.Mergeable<T,R>> R	reduceMapped(Supplier<A> aggregatorFactory)	Deprecated. v54 Use invalid reference #reduceByMerging(Supplier<A>) instead

Methods inherited from class Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

DUMMY

private static final Consumer<Boolean> DUMMY

myDistributor

private final ToIntFunction<T> myDistributor

myInterpreter

private final DataInterpreter<T> myInterpreter

myParallelism

private final IntSupplier myParallelism

myProcessor

private final ProcessingService myProcessor

myQueueCapacity

private final int myQueueCapacity

myReaderFactory

private transient Function<File, FromFileReader<T>> myReaderFactory

myReaderManager

private final Throughput myReaderManager

myShards

private final ShardedFile myShards

myWriterManger

private final Throughput myWriterManger

Constructor Details

BatchNode

BatchNode(BatchNode.Builder<T> builder)

Method Details

newBuilder

public static <T> BatchNode.Builder<T> newBuilder(File directory,
 DataInterpreter<T> interpreter)

newInstance

public static <T> BatchNode<T> newInstance(File directory,
 DataInterpreter<T> interpreter)

dispose

public void dispose()

Dispose of this node and explicitly delete all files.

newWriter

public ToFileWriter<T> newWriter()

processAll

public void processAll(Consumer<T> processor)

Process each and every item individually

Parameters:

processor - Must be able to consume concurrently

processAll

public void processAll(Supplier<Consumer<T>> processorFactory)

Similar to processAll(Consumer) but you provide a consumer constructor/factory rather than a specific consumer. Internally there will be 1 consumer per worker thread instantiated. This variant is for when the consumer(s) are stateful.

processCombineable

public <R, A extends TwoStepMapper<T,R>> void processCombineable(Supplier<A> aggregatorFactory,
 Consumer<A> processor)

Similar to processMergeable(Supplier, Consumer) but the processor is called with the aggregator instance itself rather than its extracted results. This corresponds to

invalid reference

TwoStepMapper#Combineable

rather than

invalid reference

TwoStepMapper#Mergeable

.

See Also:

• processMergeable(Supplier, Consumer)

processMapped

@Deprecated
public <R> void processMapped(Supplier<? extends TwoStepMapper<T,R>> aggregatorFactory,
 Consumer<R> processor)

Deprecated.

v54 Use

invalid reference

#processMergeable(Supplier<? extends TwoStepMapper<T, H>>,Consumer<H>)

instead

Process mapped/derived data in batches.

There will be one TwoStepMapper instance per underlying file/shard – that's a batch. Those instances are likely to contain some sort of Collection or Map that hold mapped/derived data.

You must make sure that all data items that need to be in the same TwoStepMapper instance (in the same batch) are in the same file/shard. You control the number of shards via BatchNode.Builder.fragmentation(int) and which item goes in which shard via BatchNode.Builder.distributor(ToIntFunction).

Type Parameters:

R - The mapped/derived data holding type

Parameters:

aggregatorFactory - Produces the TwoStepMapper mapping instances

processor - Consumes the mapped/derived data - the results of one whole TwoStepMapper instance at the time

processMergeable

public <R, A extends TwoStepMapper<T,R>> void processMergeable(Supplier<A> aggregatorFactory,
 Consumer<R> processor)

Each shard is processed/aggregated separately by a TwoStepMapper instance. The results are then processed/merged by the provided processor.

There is one TwoStepMapper instance per underlying worker thread. Those instances are reset and reused for each shard.

The processor is called concurrently from multiple threads.

You must make sure that all data items that need to be in the same aggregator instance are in the same file/shard. You control the number of shards via BatchNode.Builder.fragmentation(int) and which item goes in which shard via BatchNode.Builder.distributor(ToIntFunction).

Parameters:

aggregatorFactory - Produces the TwoStepMapper aggregator instances

processor - Consumes the aggregated/derived data - the results of one whole TwoStepMapper instance at the time

reduceByCombining

public <R, A extends TwoStepMapper.Combineable<T,R,A>>
R reduceByCombining(Supplier<A> aggregatorFactory)

Calls processCombineable(Supplier, Consumer) with the

invalid reference

TwoStepMapper.Combineable#merge(Object)

method of a global TwoStepMapper.Combineable instance as the consumer.

reduceByMerging

public <R, A extends TwoStepMapper.Mergeable<T,R>>
R reduceByMerging(Supplier<A> aggregatorFactory)

Calls processMergeable(Supplier, Consumer) with the TwoStepMapper.Mergeable.merge(Object) method of a global TwoStepMapper.Mergeable instance as the consumer.

reduceMapped

@Deprecated
public <R, A extends TwoStepMapper.Mergeable<T,R>>
R reduceMapped(Supplier<A> aggregatorFactory)

Deprecated.

v54 Use

invalid reference

#reduceByMerging(Supplier<A>)

instead

Same as processMergeable(Supplier, Consumer), but then also reduce/merge the total results using

invalid reference

TwoStepMapper#merge(Object)

.

Create a class that implements TwoStepMapper and make sure to also implement

invalid reference

TwoStepMapper#merge(Object)

- you can only use this if merging partial (sub)results is possible. Use a constructor or factory method that produce instances of that type as the argument to this method.

getReaderFactory

private Function<File, FromFileReader<T>> getReaderFactory()

process

private void process(File shard,
 Consumer<T> consumer)

processAggregators

private <R, A extends TwoStepMapper<T,R>> void processAggregators(File shard,
 Supplier<A> aggregatorFactory,
 Consumer<A> processor)

processResults

private <R, A extends TwoStepMapper<T,R>> void processResults(File shard,
 Supplier<A> aggregatorFactory,
 Consumer<R> processor)

newReader

FromFileReader<T> newReader(File file)