etl_lib.core.ClosedLoopBatchProcessor module

class ClosedLoopBatchProcessor(context, task, predecessor, expected_rows=None)

Bases: BatchProcessor

Reporting implementation of a BatchProcessor.

Meant to be the last entry in the list of etl_lib.core.BatchProcessor driving the processing and reporting updates of the processed batches using the etl_lib.core.ProgressReporter from the context.

Parameters:

• context (ETLContext)

• task (Task)

• predecessor (BatchProcessor)

• expected_rows (int)

__init__(context, task, predecessor, expected_rows=None)

Constructs a new etl_lib.core.BatchProcessor instance.

Parameters:

• context (ETLContext) – etl_lib.core.ETLContext.ETLContext instance. It Will be available to subclasses.

• task (Task) – etl_lib.core.Task.Task this processor is part of. Needed for status reporting only.

• predecessor (BatchProcessor) – Source of batches for this processor. Can be None if no predecessor is needed (such as when this processor is the start of the queue).

• expected_rows (int)

get_batch(max_batch__size)

Provides a batch of data to the caller.

The batch itself could be called and processed from the provided predecessor or generated from other sources.

Parameters:

max_batch__size (int) – The max size of the batch the caller expects to receive.

Return type:

Generator[BatchResults, None, None]

Returns

A generator that yields batches.