pagexml-hf

PageXML-HF (fork from wjbmattingly/transkribus-hf)

Convert (Transkribus) ZIP files (containing XML and images) or local folders with exported XMLs and images or HuggingFace datasets/datasets.Dataset with raw-xml and page-images to HuggingFace datasets with ease.

Overview

pagexml-hf is a Python package that converts (Transkribus) export ZIP files, local folders with exported XMLs and images, or datasets.Dataset (with columns “xml” and “image”) into HuggingFace datasets. It supports multiple export formats and can automatically upload datasets to the HuggingFace Hub. It is a fork of the original transkribus-hf package by wjbmattingly (https://github.com/wjbmattingly/transkribus-hf)

Features

Installation

git clone https://github.com/The-Flow-Project/pagexml-hf
cd pagexml_hf
pip install -e .

or if you use uv:

pip install uv
git clone https://github.com/The-Flow-Project/pagexml-hf
cd pagexml_hf
uv tool install -e .

Export Modes

1. Raw XML (raw_xml)

Exports the original image with the complete PAGE XML content.

Fields:

2. Text (text) - Default

Exports the image with concatenated text from all regions.

Fields:

3. Region (region)

Exports each text region as a separate cropped image.

Fields:

4. Line (line)

Exports each text line as a separate cropped image.

Fields:

5. Window (window) - NEW!

Exports sliding windows of multiple text lines, perfect for data augmentation and multi-line text recognition training.

Configuration:

Fields:

Examples:

Usage

Command Line Interface

# Basic usage - convert and upload to HuggingFace Hub
pagexml-hf path/to/your/transkribus.zip --repo-id username/dataset-name

# Specify export mode
pagexml-hf path/to/your/transkribus.zip --repo-id username/dataset-name --mode region

# Window mode with 3 lines per window, 1 line overlap
pagexml-hf path/to/your/transkribus.zip --repo-id username/dataset-name --mode window --window-size 3 --overlap 1

# Convert to local directory only
pagexml-hf path/to/your/transkribus.zip --local-only --output-dir ./my_dataset

# View statistics only (including window estimates)
transkribus-hf path/to/your/transkribus.zip --stats-only --mode window --window-size 2

# Create private repository
pagexml-hf path/to/your/transkribus.zip --repo-id username/dataset-name --private

# Use custom HuggingFace token
pagexml-hf path/to/your/transkribus.zip --repo-id username/dataset-name --token your_token_here

# Train-test split
pagexml-hf path/to/your/transkribus.zip --repo-id username/dataset-name --split_train 0.8 --split_shuffle --split_seed 42

Python API

from pagexml_hf import XmlConverter, XmlParser

parser = XmlParser()
pages = parser.parse_zip(zip_path="path/to/your/transkribus.zip")
# or parse a local folder with XMLs and images
# pages = parser.parse_folder(folder_path="path/to/your/xml_export")
# or parse a HuggingFace dataset with raw XML and page images
# pages = parser.parse_dataset(dataset)
# or from huggingface hub
# pages = parser.parse_hf_dataset(dataset_repo="username/dataset-name")

# Initialize converter
converter = XmlConverter(pages)

# Get statistics
stats = converter.get_stats()
print(f"Total pages: {stats['total_pages']}")
print(f"Total regions: {stats['total_regions']}")
print(f"Total lines: {stats['total_lines']}")

# Convert to dataset (text mode)
dataset = converter.convert(export_mode='text')
print(f"Created dataset with {len(dataset)} examples")

# Convert to different modes
region_dataset = converter.convert(export_mode='region')
line_dataset = converter.convert(export_mode='line')
xml_dataset = converter.convert(export_mode='raw_xml')

# NEW: Window mode with different configurations
window_2_dataset = converter.convert(export_mode='window', window_size=2, overlap=0)
window_3_overlap_dataset = converter.convert(export_mode='window', window_size=3, overlap=1)
window_4_dataset = converter.convert(export_mode='window', window_size=4, overlap=2)

print(f"2-line windows: {len(window_2_dataset)} examples")
print(f"3-line windows (1 overlap): {len(window_3_overlap_dataset)} examples")
print(f"4-line windows (2 overlap): {len(window_4_dataset)} examples")

# Upload to HuggingFace Hub
repo_url = converter.upload_to_hub(
    dataset=window_3_overlap_dataset,
    repo_id="wjbmattingly/my-transkribus-windows",
    private=False
)
print(f"Dataset uploaded: {repo_url}")

# Convert and upload in one step
repo_url = converter.convert_and_upload(
    repo_id="username/my-transkribus-dataset",
    export_mode="window",
    window_size=2,
    overlap=1,
    private=False
)

Transkribus ZIP Structure

The package expects Transkribus ZIP files with the following structure:

transkribus_export.zip
├── project1/
│   ├── image1.jpg
│   ├── image2.jpg
│   └── page/
│       ├── image1.xml
│       └── image2.xml
├── project2/
│   ├── image3.jpg
│   └── page/
│       └── image3.xml
└── ...

XML and Images Folder Structure

The package can handle folders with the following structure:

xml_export/
├── project1/
│   ├── image1.jpg
│   ├── image2.jpg
│   └── page/
│       ├── image1.xml
│       └── image2.xml
├── project2/
│   ├── image3.jpg
│   └── page/
│       └── image3.xml
└── ...

Window Mode Use Cases

The window mode is particularly useful for:

  1. Data Augmentation: Generate more training examples from existing data
  2. Multi-line Text Recognition: Train models to recognize multiple lines at once
  3. Reading Order Training: Train models to understand line sequences
  4. Flexible Context: Adjust context size (1-4+ lines) based on your needs
  5. Overlapping Context: Create overlapping examples for better generalization

Authentication

To upload datasets to HuggingFace Hub, you need to authenticate:

  1. Set environment variable: export HF_TOKEN=your_token_here
  2. Or pass the token directly: --token your_token_here
  3. Or use huggingface-cli login

Requirements

License

MIT License - see LICENSE file for details.

This site is open source. Improve this page.