app.worker

Helper utilities for the preprocessing worker.

The actual Celery task lives in tasks.py. This module retains shared helper functions (e.g. HuggingFace upload) that are used by the task.

Functions

upload_status_to_huggingface(status, ...)

Upload preprocessing status as JSON to the HuggingFace dataset.