etl_lib.data_source.SQLBatchSource module

class SQLBatchSource(context, task, query, record_transformer=None, **kwargs)[source]

Bases: BatchProcessor

Parameters:

context (ETLContext)
task (Task)
query (str)
record_transformer (Callable[[dict], dict] | None)

__init__(context, task, query, record_transformer=None, **kwargs)[source]

Constructs a new SQLBatchSource that streams results instead of paging them.

Parameters:

context (ETLContext)
task (Task)
query (str)
record_transformer (Callable[[dict], dict] | None)

get_batch(max_batch_size)[source]

Yield successive batches using a Server-Side Cursor (Streaming).

This avoids ‘LIMIT/OFFSET’ pagination, which causes performance degradation on large tables. Instead, it holds a cursor open and fetches rows incrementally.

Return type:: Generator[BatchResults, None, None]
Parameters:: max_batch_size (int)