flow_inference.evaluation.Evaluation

class flow_inference.evaluation.Evaluation(evaluation_repo_name, hf_token, splits=None)[source]

Run CER evaluation for a Hugging Face inference result dataset.

The evaluator downloads a dataset, selects the requested split or a default evaluation split, finds the latest inference column, computes CER against the text ground-truth column, and uploads evaluation artifacts back to the dataset repository.

Parameters:

  • evaluation_repo_name (str)

  • hf_token (str | None)

  • splits (List[str] | None)

__init__(evaluation_repo_name, hf_token, splits=None)[source]

Initialize the evaluator.

Parameters:

  • evaluation_repo_name (str) – Hugging Face dataset repository containing inference results.

  • hf_token (Optional[str]) – Optional Hugging Face token used for private repositories.

  • splits (Optional[List[str]]) – Optional split names to evaluate. If omitted, test is preferred over train.

Methods

__init__(evaluation_repo_name, hf_token[, ...])

Initialize the evaluator.

compute_cer(gt, hyp)

Compute Character Error Rate for predictions.

create_output_files(groundtruth, hypothesis, ...)

Create evaluation artifact files.

load_dataset()

Download the evaluation dataset and convert its splits to DataFrames.

perform_evaluation()

Run the full evaluation pipeline.

select_splits(dfs)

Select the DataFrame rows to evaluate.

upload_readme(dfs, inference_col, cer_score, ...)

Regenerate and upload the dataset README with evaluation metadata.

upload_results(files, timestamp)

Upload evaluation artifact files to the dataset repository.