etl_lib.task.data_loading.ParallelParquetLoad2Neo4jTask module

class ParallelParquetLoad2Neo4jTask(context, file, model=None, error_file=None, table_size=10, batch_size=5000, max_workers=None, prefetch=4, **parquet_reader_kwargs)

Bases: Task

Parallel Parquet → Neo4j load using the mix-and-batch strategy.

Parameters:

• context (ETLContext)

• file (Path)

• model (Type[BaseModel] | None)

• error_file (Path | None)

• table_size (int)

• batch_size (int)

• max_workers (int | None)

• prefetch (int)

__init__(context, file, model=None, error_file=None, table_size=10, batch_size=5000, max_workers=None, prefetch=4, **parquet_reader_kwargs)

Construct a Task object.

Parameters:

• context (ETLContext) – ETLContext instance. Will be available to subclasses.

• file (Path)

• model (Type[BaseModel] | None)

• error_file (Path | None)

• table_size (int)

• batch_size (int)

• max_workers (int | None)

• prefetch (int)

run_internal()

Place to provide the logic to be performed.

This base class provides all the housekeeping and reporting, so that implementation must/should not need to care about them. Exceptions should not be captured by implementations. They are handled by this base class.

Parameters:

kwargs – will be passed to run_internal

Return type:

TaskReturn

Returns:

An instance of TaskReturn.