etl_lib.data_source.SQLBatchSource module

class SQLBatchSource(context, task, query, record_transformer=None, **kwargs)

Bases: BatchProcessor

Parameters:

• context (ETLContext)

• task (Task)

• query (str)

• record_transformer (Callable[[dict], dict] | None)

__init__(context, task, query, record_transformer=None, **kwargs)

Constructs a new SQLBatchSource.

Parameters:

• context (ETLContext) – etl_lib.core.ETLContext.ETLContext instance.

• task (Task) – etl_lib.core.Task.Task instance owning this batchProcessor.

• query (str) – SQL query to execute.

• record_transformer (Optional[Callable[[dict], dict]]) – Optional function to transform each row (dict format).

• kwargs – Arguments passed as parameters with the query.

get_batch(max_batch_size)

Yield successive batches until the query is exhausted.

Calls _fetch_page() repeatedly, advancing the offset by the number of rows returned. Stops when no more rows are returned.

Parameters:

max_batch_size (int) – upper limit on rows per batch.

Yields:

BatchResults for each non-empty page.

Return type:

Generator[BatchResults, None, None]