Preface OCR pipeline
How Cologne dictionary front matter (title pages, prefaces/forewords, abbreviation lists, addenda) is turned into searchable Markdown plus English and Russian editions — and which agent path produced which deliverable.
Public catalogue of finished text: OCR'd prefaces.
Cologne scan index: csldoc dictionaries.
Two related pipelines
| Pipeline | Skill / code | Input | Output | Role |
|---|---|---|---|---|
| CDSL front matter | /cologne-preface-ocr (Claude Code command + Codex twin) | csldoc PNG scans | per-dict prefaces/ (source + EN/RU + *_all.*) | Production path for Cologne dicts |
| PD (Deccan College) | PD/run_ocr.py on branch feat/ocr-v2-pipeline | local PDF front matter | GA.md, AUB.md, RG.md, SWC.md, AOB.md in PD | Separate corpus; Tesseract crop-then-OCR |
Do not treat the PD branch as a rename of the CDSL skill. They share layout ideas (crop-then-OCR, column modes); they do not share the same authoring engine for Cologne scans.
Engines inside the CDSL skill
| Engine | Name in docs | Author of canonical pages? | Mechanism |
|---|---|---|---|
| A | Vision band OCR | Yes | Native-resolution crops (≤ ~1900 px/side) → vision model → faithful transcription |
| B | Tesseract crop-then-OCR (Grok/PD family) | No (audit only on CDSL) | Tesseract `deu |
Hard rule: never auto-promote B text into <code>prefNN.md. B may disagree and still be wrong (especially title names and diacritic-heavy abbreviation keys). Evidence: PWG A vs B comparison (mean Jaccard ~0.51 vs A as gold; prose best, abbrev/title worst).
CDSL phases (/cologne-preface-ocr)
| Phase | What happens |
|---|---|
| 0 | Resolve dict code → local repo (PW→PWK, MW/MWE→MWS, WIL/YAT→Wil-YAT, …) |
| 1 | Discover scan pages from csldoc toctree (trust toctree order, not filename sort) |
| 2 | Download PNGs into prefaces/scans/ |
| 3 | Engine A — OCR each page into <code>prefNN.md (YAML header + transcription) |
| 3.5 | Engine B audit (recommended) — Tesseract pass; write prefaces/AUDIT_A_VS_B.md; re-vision only flagged bands |
| 4 | Translations: .en.md if source ≠ English; always .ru.md |
| 5 | Consolidated *_all.* via build_combined.py |
| 6 | prefaces/README.md + root README surface + cost/run notes; refresh ocr-prefaces |
| 7 | Delete temp crops / _audit_b/; commit only if asked |
Phase 3.5 — what B is for
| Signal | Action |
|---|---|
| A long, B empty in a band | Possible A invention or B crop fail → re-vision the band |
| B dense, A short | Possible A skip → re-OCR missing region with A |
| Both disagree on a number/date | Re-vision the locus from the scan |
| Diacritic-only key mismatch | Expected B weakness — do not “fix” A toward B |
| B-only “Institute of Indology…” | Digitizer stamp — ignore (A correctly omits) |
Reference tooling: PD/compare_pwg_a_vs_b.py. Full phase text lives in the skill runbook (Claude: ~/.claude/commands/cologne-preface-ocr.md; Codex: ~/.codex/skills/cologne-preface-ocr/).
Bounded sample quality probe (PWG + PW, 24-07-2026): pre-registered 15-page stratified sample + scan-locus table + DE↔EN sense check — H1559 report (runner: scripts/pref_ocr_en_quality_probe.py). Dual-pass is A-vs-B only; scan-locus is the honest gold-vs-print probe.
Non-negotiable OCR rule (Phase 3)
Never Read a full downsampled page. Scans are 3000–6800 px; downsampling yields fluent but fabricated text. Split columns, cut overlapping horizontal bands, keep each crop ≤ ~1900 px on its longest side.
Deliverable layout (per dictionary)
<prepo>/prefaces/
scans/ # source PNGs
<code>prefNN.md # source-language OCR (Engine A)
<code>prefNN.en.md # English (omit if source is already English)
<code>prefNN.ru.md # Russian
<code>pref_all.*.md # consolidated editions
build_combined.py
README.md
METHODS.md # FAIR: OCR policy + how to cite (pilot: PWG, PW)
AUDIT_A_VS_B.md # optional Phase 3.5 QA
Indexed on the site: OCR'd prefaces (includes the FAIR path map: METHODS · naming authority · legend emit). Pilot METHODS: PWG · PW.
Who OCR'd what: Claude Code vs Grok
Attribution is by production pipeline, not git commit identity (commits often land under the human maintainer account after an agent run). “Produced by /cologne-preface-ocr” in root-README run notes = Claude Code vision path (skill default tier: Fable 5 / Opus 4.8 fallback).
Census of local clones 23-07-2026.
Claude Code — CDSL production (Engine A, /cologne-preface-ocr)
All rows below are canonical edition pages under each repo’s prefaces/ (source + translations as applicable). None of these production files were authored by the Grok/Tesseract path.
| Code | Repo | ~Source pages | Source lang | First landed (approx.) |
|---|---|---|---|---|
| PWG | PWG | 27 | de | 2026-06-16 |
| PW | PWK | 5 | de | 2026-06-17 |
| SCH | SCH | 5 | de | 2026-06-17 |
| MW | MWS | 29 | en | 2026-06-17 |
| MWE | MWS (mwepref*) | 11 | en | 2026-07-03 |
| GRA | GRA | 6 | de | 2026-06-22 |
| CCS | CCS | 6 | de | 2026-06-22 |
| INM | INM | 9 | en | 2026-06-22 |
| IEG | prefaces_ieg | 10 | en | 2026-06-22 |
| SHS | SHS | 2 | en | 2026-06-22 |
| BOP | BOP | 5 | la | 2026-06-22 |
| STC | STC | 5 | fr | 2026-06-22 |
| VCP | VCP | 7 | sa | 2026-06-22 |
| MD | MD | 10 | en | 2026-06-23 |
| CAE | CAE | 6 | en | 2026-06-23 |
| BHS | BHS | 15 | en | 2026-06-23 |
| BOR | BOR | 36 | en | 2026-06-23 |
| VEI | VEI | 16 | en | 2026-06-23 |
| MW72 | MW72 | 25 | en | 2026-06-23 |
| WIL | Wil-YAT | 6 | en | 2026-06-23 |
| WIL 1819 (1st ed.) | WIL (prefaces/) | 53 | en | 2026-08-06 |
| YAT | Wil-YAT | 3 | en | 2026-06-23 |
| KRM | KRM | 24 | sa/en | 2026-06-23 |
| MCI | MCI | 28 | en | 2026-06-23 |
Public index of editions: ocr-prefaces.mdx (csl-guides).
Count (23-07-2026 post-merge): ~32 dictionary codes with production or staging editions on a default branch. Merged that day: AP90 #49, BEN #32, ACC #25, BUR #13.
Claude Code — residue status (after promote + drain)
| Code | Remote home | Status |
|---|---|---|
| GST, PGN, SNP, LAN, AE | gasyoun/prefaces_* on main | Promoted / complete |
| PE | gasyoun/prefaces_pe | Partial (01–04; genealogy 05–26 pending) |
| AP90, BEN, ACC, BUR | sanskrit-lexicon/* default branch | Merged 23-07-2026 |
| SKD, PUI | — | Still cold (multi-scan OCR not run) |
| AP, ABCH, LRV | — | No usable ${code}pref.html |
Local-only staging under the GitHub/ meta-tree is obsolete for GST/PGN/SNP/LAN/AE — use the remotes above.
Grok — production OCR (Engine B primary)
| Work | Location | Notes |
|---|---|---|
| Deccan College PD front matter (GA, AUB, RG, SWC, AOB) | PD · branch feat/ocr-v2-pipeline | Tesseract 5.5 eng+san; crop-then-OCR; layout modes multi_pair / multi_seq / prose_2col. Not a Cologne csldoc dict. |
Grok — tooling / QA / docs (not production CDSL pages)
| Work | Location | Notes |
|---|---|---|
| PWG A vs B bake-off | COMPARISON_PWG_OCR_A_VS_B.md, PD/pwg_a_vs_b/, compare_pwg_a_vs_b.py | Sample pages only; confirms A remains gold for PWG |
| Phase 3.5 audit design | skill runbook + this page | B as comparison layer after Claude Code drafts |
| Skill + public index updates | /cologne-preface-ocr, ocr-prefaces | Inventory refresh, dual-engine wording, partial-residue census |
Grok has not replaced Claude Code as author of any CDSL prefaces/<code>prefNN.md edition as of this census. Future CDSL runs may still use Grok only for Phase 3.5 audit (or a human may run Tesseract locally the same way).
Not yet OCR'd / incomplete
- Partial: PE genealogy tables (pages 05–26).
- Cold multi-scan: SKD, PUI.
- No pref HTML: AP, ABCH, LRV.
Default author path: Claude Code / Engine A, then optional Phase 3.5 audit.
How to run (operators)
# CDSL dictionary (Claude Code or Codex with the skill loaded)
/cologne-preface-ocr PWG
/cologne-preface-ocr AP90
/cologne-preface-ocr all # skips Done inventory rows
# PD PDFs only (Grok / local Python)
cd PD
python run_ocr.py
python run_ocr.py --only AOB.pdf --start-page 20
# Optional bake-off / audit sample (PWG-shaped; adapt for other codes)
python compare_pwg_a_vs_b.py
Requirements for Phase 3.5 / PD: Tesseract 5.x with appropriate traineddata (deu, eng, san, …), pillow, pytesseract (PD also: pymupdf, numpy, beautifulsoup4).
Related pages
- OCR'd prefaces — finished editions and queue
- Catalog — code ↔ repo mapping
- Scans and print — end-user scan access
- Generation pipeline — headword XML pipeline (different system)
Last census: 23-07-2026 · Pipeline docs: Grok 4.5 (grok-4.5) with skill/run-note evidence from Claude Code /cologne-preface-ocr runs.