etl_lib.data_source.CypherBatchSource module

class CypherBatchSource(context, task, query, record_transformer=None, **kwargs)

Bases: BatchProcessor

Parameters:

• context (ETLContext)

• task (Task)

• query (str)

• record_transformer (Callable[[Record], dict] | None)

__init__(context, task, query, record_transformer=None, **kwargs)

Constructs a new CypherBatchSource.

Parameters:

• context (ETLContext) – etl_lib.core.ETLContext.ETLContext instance.

• task (Task) – etl_lib.core.Task.Task instance owning this batchProcessor.

• query (str) – Cypher query to execute.

• record_transformer (Optional[Callable[[Record], dict]]) – Optional function to transform each record. See Neo4j API documentation on result_transformer_

• kwargs – Arguments passed as parameters with the query.

get_batch(max_batch_size)

Provides a batch of data to the caller.

The batch itself could be called and processed from the provided predecessor or generated from other sources.

Parameters:

• max_batch__size – The max size of the batch the caller expects to receive.

• max_batch_size (int)

Return type:

Generator[BatchResults, None, None]

Returns

A generator that yields batches.