flow_preprocessing.preprocessing_logic.preprocess.HuggingFacePreprocessor

class flow_preprocessing.preprocessing_logic.preprocess.HuggingFacePreprocessor(input_path, config, converter_factory=None)

Preprocessor for HuggingFace datasets.

Supports processing existing HuggingFace datasets and uploading the processed version to a new or existing repository.

Parameters:

• input_path (str)

• config (PreprocessorConfig)

• converter_factory (ConverterFactory | None)

__init__(input_path, config, converter_factory=None)

Initialize HuggingFace preprocessor.

Parameters:

• input_path (str) – HuggingFace repository ID (e.g., ‘username/dataset’).

• config (PreprocessorConfig) – Preprocessor configuration.

• converter_factory (ConverterFactory | None) – Optional converter factory (for DI/testing).

Return type:

None

Methods

__init__(input_path, config[, converter_factory])	Initialize HuggingFace preprocessor.
create_xmlconverter()	Create XmlConverter for HuggingFace source using factory.
preprocess()	Preprocess HuggingFace dataset.
segment_images()	Segment images in the dataset using YOLO or kraken.

Attributes

config	Get configuration.
converter	Get or create the XmlConverter (lazy initialization).
dataset	Get the current dataset.
state	Get current processor state.