flow_inference.data_handling.HuggingFaceDataHandler

class flow_inference.data_handling.HuggingFaceDataHandler(dataset_name, huggingface_token=None, split=None, cache_dir=None, revision='main')[source]

Download and convert Hugging Face datasets.

Supports repositories with parquet files stored under paths such as data/train/<doc_folder>/*.parquet and data/test/<doc_folder>/*.parquet.

The handler also supports train-only repositories, test-only repositories, and default repositories with parquet files somewhere below data/**. It can preserve and update existing inference_* columns during compatible update uploads.

Parameters:

dataset_name (str)
huggingface_token (str | None)
split (str | Split | Iterable[str] | None)
cache_dir (str | None)
revision (str)

__init__(dataset_name, huggingface_token=None, split=None, cache_dir=None, revision='main')[source]

Initialize the dataset handler.

Parameters:

dataset_name (str) – Hugging Face dataset repository ID to download.
huggingface_token (str | None) – Optional Hugging Face token used for private repositories.
split (Union[str, Split, Iterable[str], None]) – Optional split or splits to load. If omitted, available splits are detected automatically.
cache_dir (Optional[str]) – Optional directory used for downloaded dataset snapshots.
revision (str) – Dataset revision, branch, tag, or commit SHA to download.

Methods

`__init__`(dataset_name[, huggingface_token, ...])	Initialize the dataset handler.
`convert_df_into_hf_dataset`()	Convert the first stored DataFrame split into a Hugging Face Dataset.
`convert_to_list_of_dicts`(dfs)	Convert split DataFrames to record dictionaries.
`count_real_duplicate_excess_lines`(df)	Count only duplicate rows beyond the first occurrence.
`count_real_duplicate_line_groups`(df)	Count duplicate key groups, not rows.
`count_real_duplicate_lines`(df)	Count rows that participate in real duplicate line keys.
`count_real_duplicate_lines_by_split`(dfs)	Count real duplicate lines per split.
`download_hf_dataset`()	Download the configured Hugging Face dataset and load its parquet splits.
`push_to_hub`(upload_repo_name[, private, ...])	Upload the current dataset state to the Hugging Face Hub.
`to_dataframe`()	Convert loaded Hugging Face dataset splits to pandas DataFrames.
`upload_file`(repo_name, target_path, ...)	Upload a single file to a Hugging Face dataset repository.