poison-flower/pf-ocr
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

init

Browse Source
This commit is contained in:
Poison Flower
2026-09-01 00:30:34 +03:00
committed by GitHub
commit 41c6e97cd4
15 changed files with 1912 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+82
View File
@@ -0,0 +1,82 @@
# Light Novel Scan -> EPUB
Turns a folder of scanned light novel page images (vertical Japanese text)
into a proper `.epub`, with furigana rendered as real `<ruby>` markup,
illustrations kept in place, an embedded font, and epub metadata — using a
multimodal LLM to do the OCR instead of a traditional column-segmentation
pipeline.
## Pipeline
```
scanned page images
│
▼
ocr/ module → transcribes each page into pages_txt/<name>.txt
│
▼
(manual step) → sort the .txt files (and any illustration
images) into chapters/chNN_name/ folders
│
▼
epub_builder/ module → assembles chapters/ into a finished .epub
```
1. **[`ocr/`](ocr/README.md)** — batch-OCRs scanned pages into per-page
`.txt` files. Several backends are available (a multimodal LLM via any
OpenAI-compatible API is the recommended default; Google Cloud Vision
and a fully offline `manga-ocr` pipeline are also included).
2. **Manual sorting** — split the resulting pages into chapter folders
(`ch00_frontmatter/`, `ch01_chapter00/`, ...), optionally dropping in a
`cover.jpg` and illustration images alongside the `.txt` files. This
step is manual because automatically detecting chapter boundaries from
OCR'd headers turned out to be unreliable — folder structure is simple
and unambiguous instead.
3. **[`epub_builder/`](epub_builder/README.md)** — assembles the sorted
`chapters/` folder into a valid `.epub`: furigana notation becomes
`<ruby>` markup, images are placed inline, a font from `font/` gets
embedded, and metadata comes from `config.json`.
See each module's own README for setup and usage details.
## Quick start
```bash
# 1. OCR
cd ocr
pip install openai pillow natsort tqdm
cp config.example.json config.json # fill in api_key / base_url / model
python openrouter_ocr.py --input /path/to/scans --output ./out
# 2. Sort ./out/pages_txt/*.txt by hand into epub_builder/chapters/chNN_name/
# 3. Build the epub
cd ../epub_builder
pip install natsort
cp config.example.json config.json # fill in title / author / ...
python build_epub.py
```
## ⚠️ A note on copyright
This repository contains only the **tooling**. It is not meant to, and
should not, be used to host or distribute:
- scanned page images,
- OCR'd text extracted from a copyrighted book,
- or a resulting `.epub` file,
for any book you don't hold the rights to. The `.gitignore` in this repo
already excludes `chapters/`, `pages_txt/`, `out/`, and `*.epub` for this
reason — keep it that way if you fork or extend this project. This tool is
intended for personal-use digitization of books you own, not redistribution.
## Requirements
See [`requirements.txt`](requirements.txt) for the full list. Not every
dependency is needed at once — install only what the OCR backend and
features you're using require (see each module's README).
## Status
This is a personal toolkit, still evolving. Contributions/forks welcome,
but expect rough edges — issues and PRs are handled best-effort.
epub_builder/README.md
+182
View File
@@ -0,0 +1,182 @@
# EPUB builder module
Assembles a `chapters/` folder — laid out by hand into per-chapter
subfolders — into a finished, valid `.epub`, with furigana rendered as
proper `<ruby>` markup, an embedded font, a cover image, and metadata from
`config.json`.
This is step 2 of the pipeline; step 1 is the [`ocr`](../ocr/README.md)
module, which produces the per-page `.txt` files you'll sort into
`chapters/`.
## Layout
```
epub_builder/
chapters/
cover.jpg <- cover image (optional, any "cover.*" directly in chapters/)
ch00_frontmatter/ <- everything before the main text (title page, TOC, ...)
001.txt
002.jpg
ch01_chapter00/
003.txt
004.txt
005.jpg
ch02_chapter01/
006.txt
ch08_backmatter/ <- everything after the main text (afterword, ...)
...
font/
YourFont.ttf <- font to embed (optional)
build_epub.py
furigana.py
config.json
```
## Chapter folder naming: `chNN_name`
- `chNN` — sequence number, determines sort order (`ch00` → `ch01` → ... →
`ch10`; use leading zeros if you have more than 9 chapters so plain
alphabetical sort stays correct).
- `name` — default chapter title (override via `config.json` →
`chapter_titles`).
- Folders with `frontmatter` or `backmatter` anywhere in their name are
**always** ignored: no title, no TOC entry — regardless of any flag or
config setting. Their content is still included in the book, just
without a section label.
Files inside a folder are sorted strictly by the number in the filename
(extension doesn't matter — an image `004.jpg` will land between
`003.txt` and `005.txt`).
## Page layout
- The first line of each `.txt` page (after stripping a leading page
number) is rendered as a larger, bold "page header" — typically an
in-story date/time/chapter marker (relevant if you're OCR-ing a novel
where that detail matters to the plot, e.g. one involving time travel).
- If nothing is left after stripping the page number, two blank lines are
shown instead, so the page's vertical rhythm stays consistent.
- A blank line always follows the header, then the text.
- Each paragraph starts with a full-width space (`　`) — the standard
indent in Japanese typography, embedded directly in the text rather than
relying on CSS `text-indent` (which not every reader honors).
## config.json
```json
{
"title": "Your Book Title",
"author": "Author Name",
"language": "ja",
"identifier": "",
"publisher": "",
"description": "",
"date": "",
"rights": "",
"series": "",
"series_index": "",
"font_family": "",
"vertical": false,
"show_chapter_titles": false,
"output": "book.epub",
"chapter_titles": {
"ch00_frontmatter": "",
"ch01_chapter00": "Chapter 00",
"ch08_backmatter": ""
}
}
```
- `title`, `author`, `language` — core metadata (`title` is required).
- `identifier`, `publisher`, `description`, `date`, `rights` — optional
Dublin Core metadata, only added if non-empty. Leave `identifier` empty
to get a random UUID generated on every build; set it once and keep it
stable if you plan to rebuild the file repeatedly and want reading apps
to recognize it as the same book.
- `series` / `series_index` — Calibre-style series metadata
(`calibre:series` / `calibre:series_index`), recognized by Calibre,
KOReader, and several other reading apps.
- `font_family` — explicit name for the embedded font family (see "Font"
below). Optional — without it, the name is guessed automatically.
- `vertical` — vertical Japanese writing-mode (also toggleable via
`--vertical`).
- `show_chapter_titles` — whether to print the chapter title (`<h1>`) at
the start of each chapter. Defaults to `false` — titles still show up in
the table of contents either way. Toggle via `--show-chapter-titles`.
- `output` — path for the resulting epub, used if `--output` isn't passed.
- `chapter_titles` — folder name -> display title mapping. An empty string
`""` means "no heading, no TOC entry" (this is already the default
behavior for `frontmatter`/`backmatter` folders, so listing them here is
optional, just for clarity).
`frontmatter`/`backmatter` entries in `chapter_titles` are ignored no
matter what — they can't be "turned back on" via config.
## Font
Drop font file(s) (`.ttf`/`.otf`/`.woff`/`.woff2`) into `font/`. Name them
`Regular.ttf` / `Bold.ttf` / `Italic.ttf` / `BoldItalic.ttf` (case
doesn't matter, and the keyword can be part of a longer name, e.g.
`NotoSerifJP-Bold.ttf`) and they'll automatically be unified under one
font-family name with the correct `font-weight`/`font-style`. That means a
bold chapter heading (`<h1>`, and the page-header line) will automatically
pick up `Bold.ttf`, and italics (`<em>`), if any ever show up in the text,
will pick up `Italic.ttf`.
If the filenames don't match this pattern, the old behavior kicks in:
each file gets its own font-family name, and only the first one
(alphabetically) is wired up automatically in `body { font-family }`; the
rest are still embedded in the epub but need manual CSS edits to use.
The script tries to guess a shared family name from the common part of the
filenames (e.g. `NotoSerifJP-Regular.ttf` + `NotoSerifJP-Bold.ttf` ->
`NotoSerifJP`). To set it explicitly, use `config.json`:
```json
{ "font_family": "MyBookFont" }
```
**Check the font's license** before embedding it in a file you intend to
share or publish — personal use on your own devices is usually fine, but
redistribution terms vary by font.
## Running
Simplest case — no arguments, if everything is where it's expected:
```bash
python build_epub.py
```
Or with explicit paths/overrides:
```bash
python build_epub.py \
--pages-dir ./chapters \
--font-dir ./font \
--config ./config.json \
--output ./my_book.epub \
--show-chapter-titles \
--vertical
```
`--split-pages` makes each file its own xhtml document inside the epub,
instead of merging every file in a folder into one.
## `furigana.py` as a standalone preview tool
Useful for quickly checking how a single OCR'd page will render before
running the full build:
```bash
python furigana.py --input page.txt --output preview.xhtml
# or a quick one-liner check:
echo "本文《ほんぶん》デザイン" | python furigana.py --stdin
```
## Requirements
```bash
pip install natsort
```
epub_builder/__init__.py
+13
View File
@@ -0,0 +1,13 @@
"""
EPUB builder module: assembles a chapters/ folder (per-page .txt files from
the ocr module, plus optional illustrations) into a finished, valid .epub —
with furigana rendered as proper <ruby> markup, an embedded font, a cover
image, and metadata pulled from config.json.
Modules:
furigana.py Aozora Bunko furigana notation -> HTML <ruby> markup,
plus the shared page-parsing/rendering logic. Also
runnable standalone as a single-page preview tool.
build_epub.py Assembles the whole book. Runnable directly:
python -m epub_builder.build_epub
"""
epub_builder/build_epub.py
+665
View File
@@ -0,0 +1,665 @@
#!/usr/bin/env python3
"""
Builds an EPUB from a chapters/ folder laid out by hand into per-chapter
subfolders, embedding a font from font/ and metadata from config.json.
Expected layout (next to this script):
epub_builder/
chapters/
cover.jpg <- cover image (optional, any "cover.*" directly in chapters/)
ch00_frontmatter/ <- everything before the main text
001.txt
002.jpg
ch01_chapter00/
003.txt
004.txt
005.jpg
ch02_chapter01/
006.txt
ch08_backmatter/
...
font/
YourFont.ttf <- font to embed (optional)
build_epub.py
furigana.py
config.json <- metadata and build settings
Chapter folder naming: chNN_name
- chNN determines sort order.
- name is the default chapter title (can be overridden via
"chapter_titles" in config.json).
- Folders whose name (as a whole, or the part after chNN_) contains
"frontmatter" or "backmatter" are ALWAYS ignored: no title, no TOC
entry, no in-body heading — regardless of any flag or config
setting. Their content is still included in the book, just without
a section label.
Page layout:
- The first line of each .txt page (after stripping a leading page
number) is rendered as a larger, bold "page header" — usually an
in-story date/time/chapter marker, not just page-numbering noise.
- If nothing is left after stripping the page number, two blank lines
are shown instead, so the page's vertical rhythm stays consistent.
- A blank line always follows the header, then the text.
- Each paragraph starts with a full-width space indent (standard in
Japanese typography), not a tab character that HTML/EPUB would
collapse anyway.
Chapter titles are shown in the table of contents by default, but NOT as
an in-body <h1> heading — pass --show-chapter-titles (or set
"show_chapter_titles": true in config.json) to also print them at the
start of each chapter.
Usage:
python build_epub.py
# (all settings are read from config.json / chapters/ / font/ next to this script)
# or with explicit paths/overrides:
python build_epub.py --pages-dir ./chapters --font-dir ./font \\
--config ./config.json --output ./book.epub --show-chapter-titles
Requirements:
pip install natsort
(furigana.py must live in the same folder as this script)
"""
import argparse
import html
import json
import re
import sys
import uuid
import zipfile
from pathlib import Path
from natsort import natsorted
sys.path.insert(0, str(Path(__file__).resolve().parent))
from furigana import parse_page, page_to_paragraphs_html # noqa: E402
SCRIPT_DIR = Path(__file__).resolve().parent
TEXT_EXT = {".txt"}
IMAGE_EXT = {".jpg": "image/jpeg", ".jpeg": "image/jpeg", ".png": "image/png",
".webp": "image/webp", ".gif": "image/gif"}
FONT_EXT = {".ttf": "font/ttf", ".otf": "font/otf", ".woff": "font/woff", ".woff2": "font/woff2"}
CHAPTER_FOLDER_RE = re.compile(r"^ch(\d+)_(.+)$", re.IGNORECASE)
CSS_BASE = """
body {{ font-family: {font_stack}; line-height: 1.8; }}
body.vertical {{ writing-mode: vertical-rl; }}
p {{ margin: 0 0 1em 0; }}
p.page-header {{ font-size: 1.3em; font-weight: bold; margin: 1.5em 0 0 0; }}
p.spacer {{ margin: 0 0 1em 0; }}
rt {{ font-size: 0.5em; }}
h1 {{ text-align: center; margin: 2em 0; }}
div.illustration {{ text-align: center; margin: 1em 0; }}
div.illustration img {{ max-width: 100%; max-height: 100%; }}
div.cover {{ text-align: center; margin: 0; padding: 0; }}
div.cover img {{ max-width: 100%; height: auto; }}
{font_faces}
"""
FONT_FACE_TEMPLATE = """@font-face {{
font-family: "{family}";
src: url("../fonts/{filename}") format("{fmt}");
font-weight: {weight};
font-style: {style};
}}
"""
CONTAINER_XML = """<?xml version="1.0" encoding="UTF-8"?>
<container version="1.0" xmlns="urn:oasis:names:tc:opendocument:xmlns:container">
<rootfiles>
<rootfile full-path="OEBPS/content.opf" media-type="application/oebps-package+xml"/>
</rootfiles>
</container>
"""
CHAPTER_TEMPLATE = """<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE html>
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="ja">
<head>
<meta charset="UTF-8"/>
<title>{title}</title>
<link rel="stylesheet" type="text/css" href="../css/style.css"/>
</head>
<body{body_class}>
{h1}
{content}
</body>
</html>
"""
COVER_TEMPLATE = """<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE html>
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="ja">
<head>
<meta charset="UTF-8"/>
<title>Cover</title>
<link rel="stylesheet" type="text/css" href="../css/style.css"/>
</head>
<body>
<div class="cover"><img src="../{img_href}" alt="Cover"/></div>
</body>
</html>
"""
# ---------------------------------------------------------------------------
# Page files: sorting, folder-name parsing
# ---------------------------------------------------------------------------
def numeric_sort_key(path: Path):
"""Sorts files strictly by the number found in the filename, ignoring the extension."""
m = re.search(r"\d+", path.stem)
if m:
return (0, int(m.group()), path.name)
return (1, 0, path.name)
def sort_files(files: list[Path]) -> list[Path]:
return sorted(files, key=numeric_sort_key)
def parse_chapter_folder_name(folder_name: str) -> tuple[str | None, str]:
"""
Parses a folder name of the form "chNN_name" -> (NN, name).
Returns (None, folder_name) if it doesn't match the pattern.
"""
m = CHAPTER_FOLDER_RE.match(folder_name)
if m:
return m.group(1), m.group(2)
return None, folder_name
def is_ignored_section(folder_name: str) -> bool:
"""frontmatter/backmatter sections are always ignored, regardless of any flag."""
normalized = folder_name.lower().replace("_", "").replace("-", "")
return "frontmatter" in normalized or "backmatter" in normalized
def resolve_title(folder_name: str, chapter_titles_map: dict) -> str | None:
"""
Returns the display title for a chapter, or None if the section should
have no heading and no TOC entry (frontmatter/backmatter, or an
explicit empty string in chapter_titles_map).
"""
if is_ignored_section(folder_name):
return None
if folder_name in chapter_titles_map:
value = chapter_titles_map[folder_name].strip()
return value if value else None
_, remainder = parse_chapter_folder_name(folder_name)
default_title = remainder.replace("_", " ").strip()
return default_title if default_title else folder_name
# ---------------------------------------------------------------------------
# Processing the files in one chapter folder (text + images)
# ---------------------------------------------------------------------------
def build_doc_from_folder(files: list[Path], doc_id: str):
"""Returns (content_html, [(arcname, filepath, media_type), ...])."""
content_parts = []
images = []
for fpath in files:
suffix = fpath.suffix.lower()
if suffix in TEXT_EXT:
text = fpath.read_text(encoding="utf-8")
header, paragraphs = parse_page(text)
if not header and not paragraphs:
continue # [NO_TEXT] / empty page
content_parts.append(page_to_paragraphs_html(header, paragraphs))
elif suffix in IMAGE_EXT:
arcname = f"images/{doc_id}_{fpath.stem}{suffix}"
content_parts.append(f'<div class="illustration"><img src="../{arcname}" alt=""/></div>')
images.append((arcname, fpath, IMAGE_EXT[suffix]))
else:
print(f"Skipping unknown file type: {fpath}", file=sys.stderr)
return "\n".join(content_parts), images
def find_cover(pages_dir: Path, explicit_cover: str | None) -> Path | None:
if explicit_cover:
p = Path(explicit_cover)
return p if p.exists() else None
for ext in IMAGE_EXT:
candidates = list(pages_dir.glob(f"cover{ext}")) + list(pages_dir.glob(f"cover{ext.upper()}"))
if candidates:
return candidates[0]
return None
# ---------------------------------------------------------------------------
# Grouping pages into documents (chapters), with the split-pages option
# ---------------------------------------------------------------------------
def group_pages_into_docs(pages_dir: Path, cover_path: Path | None, chapter_titles_map: dict, split_pages: bool):
subdirs = natsorted([d for d in pages_dir.iterdir() if d.is_dir()], key=lambda d: d.name)
root_files = [
p for p in pages_dir.iterdir()
if p.is_file() and p.suffix.lower() in (TEXT_EXT | set(IMAGE_EXT)) and p != cover_path
]
if not subdirs and not root_files:
print(f"No subfolders or page files found in {pages_dir}.", file=sys.stderr)
sys.exit(1)
docs = []
all_images = []
doc_counter = 0
def add_doc_merged(folder_name: str, files: list[Path]):
nonlocal doc_counter
if not files:
return
doc_counter += 1
doc_id = f"doc{doc_counter:03d}_{re.sub(r'[^a-zA-Z0-9]+', '', folder_name) or 'section'}"
content, images = build_doc_from_folder(files, doc_id)
if not content:
doc_counter -= 1
return
title = resolve_title(folder_name, chapter_titles_map)
docs.append({"id": doc_id, "title": title, "content": content})
all_images.extend(images)
def add_docs_split(folder_name: str, files: list[Path]):
nonlocal doc_counter
if not files:
return
folder_title = resolve_title(folder_name, chapter_titles_map)
safe_name = re.sub(r"[^a-zA-Z0-9]+", "", folder_name) or "section"
first_page_in_folder = True
for fpath in files:
doc_counter += 1
doc_id = f"doc{doc_counter:03d}_{safe_name}_{fpath.stem}"
content, images = build_doc_from_folder([fpath], doc_id)
if not content:
doc_counter -= 1
continue
title = folder_title if first_page_in_folder else None
first_page_in_folder = False
docs.append({"id": doc_id, "title": title, "content": content})
all_images.extend(images)
add_doc = add_docs_split if split_pages else add_doc_merged
if root_files:
add_doc("root", sort_files(root_files))
for subdir in subdirs:
files = sort_files(
[p for p in subdir.iterdir() if p.is_file() and p.suffix.lower() in (TEXT_EXT | set(IMAGE_EXT))]
)
add_doc(subdir.name, files)
return docs, all_images
# ---------------------------------------------------------------------------
# Font handling
# ---------------------------------------------------------------------------
def find_fonts(font_dir: Path) -> list[Path]:
if not font_dir.exists():
return []
fonts = [p for p in font_dir.iterdir() if p.is_file() and p.suffix.lower() in FONT_EXT]
return natsorted(fonts, key=lambda p: p.name)
def sanitize_family_name(stem: str) -> str:
return re.sub(r"[^a-zA-Z0-9 _-]+", "", stem).strip() or "EmbeddedFont"
STYLE_KEYWORDS_RE = re.compile(r"(?i)bold[\s_-]*italic|italic[\s_-]*bold|bold|italic|oblique|regular")
def detect_font_role(stem: str) -> str | None:
"""
Detects a font variant from its filename: Regular/Bold/Italic/BoldItalic
(case-insensitive, anywhere in the name). Returns None if it can't be
determined (non-standard filename).
"""
lower = stem.lower()
has_bold = "bold" in lower
has_italic = "italic" in lower or "oblique" in lower
if has_bold and has_italic:
return "bolditalic"
if has_bold:
return "bold"
if has_italic:
return "italic"
if "regular" in lower:
return "regular"
return None
ROLE_WEIGHT_STYLE = {
"regular": ("normal", "normal"),
"bold": ("bold", "normal"),
"italic": ("normal", "italic"),
"bolditalic": ("bold", "italic"),
}
def strip_style_keyword(stem: str) -> str:
"""Removes style keywords from a font filename, returning the remainder (may be empty)."""
stripped = STYLE_KEYWORDS_RE.sub("", stem)
stripped = re.sub(r"[\s_-]+", " ", stripped).strip(" -_")
return stripped
def resolve_font_plan(fonts: list[Path], explicit_family: str | None):
"""
Decides how to wire up the fonts found in font/:
- If every file's variant is recognized (Regular/Bold/Italic/BoldItalic,
with at least one Regular among them), all of them get ONE shared
font-family name with the correct font-weight/font-style, so
<b>/<strong>/<em>/bold headings automatically pick up the right file.
- Otherwise (non-standard filenames): the old behavior — each file gets
its own family name, and only the first one (alphabetically) gets
wired up automatically in the CSS.
Returns a list of tuples (font_path, family, weight, style, is_primary).
"""
if not fonts:
return []
roles = {f: detect_font_role(f.stem) for f in fonts}
all_recognized = all(r is not None for r in roles.values())
has_regular = any(r == "regular" for r in roles.values())
if all_recognized and has_regular:
if explicit_family:
family = explicit_family
else:
stripped_names = [strip_style_keyword(f.stem) for f in fonts]
non_empty = [n for n in stripped_names if n]
family = max(set(non_empty), key=non_empty.count) if non_empty else "MainFont"
family = sanitize_family_name(family) or "MainFont"
plan = []
for f in fonts:
weight, style = ROLE_WEIGHT_STYLE[roles[f]]
is_primary = roles[f] == "regular"
plan.append((f, family, weight, style, is_primary))
return plan
# Fallback: non-standard filenames — separate families
plan = []
for i, f in enumerate(fonts):
family = explicit_family if (explicit_family and i == 0) else sanitize_family_name(f.stem)
plan.append((f, family, "normal", "normal", i == 0))
return plan
# ---------------------------------------------------------------------------
# Assembling the final epub
# ---------------------------------------------------------------------------
def build_epub(docs, all_images, cover_path, fonts: list[Path], output_path: Path,
meta: dict, vertical: bool, show_chapter_titles: bool):
book_id = meta.get("identifier") or f"urn:uuid:{uuid.uuid4()}"
body_class = ' class="vertical"' if vertical else ""
manifest_items = []
spine_items = []
nav_items = []
package_files = {}
# --- Font ---
font_faces_css = ""
font_family_stack = "serif"
if fonts:
font_plan = resolve_font_plan(fonts, meta.get("font_family"))
font_face_rules = []
primary_family = None
variant_labels = []
for font_path, family, weight, style, is_primary in font_plan:
fmt = FONT_EXT[font_path.suffix.lower()].split("/")[-1]
font_face_rules.append(
FONT_FACE_TEMPLATE.format(family=family, filename=font_path.name, fmt=fmt, weight=weight, style=style)
)
arcname = f"fonts/{font_path.name}"
package_files[f"OEBPS/{arcname}"] = font_path.read_bytes()
font_id = "font_" + re.sub(r"[^a-zA-Z0-9]+", "_", font_path.name)
manifest_items.append(f'<item id="{font_id}" href="{arcname}" media-type="{FONT_EXT[font_path.suffix.lower()]}"/>')
variant_labels.append(f"{font_path.name} ({weight}/{style})")
if is_primary and primary_family is None:
primary_family = family
font_faces_css = "\n".join(font_face_rules)
font_family_stack = f'"{primary_family}", serif'
unified = len(set(family for _, family, *_ in font_plan)) == 1 and len(font_plan) > 1
if unified:
print(f'Font: {len(fonts)} variant(s) unified under the name "{primary_family}": '
+ ", ".join(variant_labels))
elif len(fonts) > 1:
print(f"Found {len(fonts)} font file(s). Filenames don't look like Regular/Bold/Italic — "
f"using {fonts[0].name} as the primary one. The rest are embedded in the epub but "
f"not wired up in the CSS automatically — edit style.css by hand if needed.")
css_content = CSS_BASE.format(font_stack=font_family_stack, font_faces=font_faces_css)
# --- Cover ---
cover_meta = ""
guide_xml = ""
if cover_path:
cover_ext = cover_path.suffix.lower()
cover_media = IMAGE_EXT.get(cover_ext, "image/jpeg")
cover_img_arcname = f"images/cover{cover_ext}"
package_files[f"OEBPS/{cover_img_arcname}"] = cover_path.read_bytes()
manifest_items.append(
f'<item id="cover-image" href="{cover_img_arcname}" media-type="{cover_media}" properties="cover-image"/>'
)
package_files["OEBPS/text/cover.xhtml"] = COVER_TEMPLATE.format(img_href=cover_img_arcname)
manifest_items.append('<item id="cover-page" href="text/cover.xhtml" media-type="application/xhtml+xml"/>')
spine_items.append('<itemref idref="cover-page" linear="yes"/>')
cover_meta = '<meta name="cover" content="cover-image"/>'
guide_xml = '<guide><reference type="cover" title="Cover" href="text/cover.xhtml"/></guide>'
# --- Images from chapters ---
for arcname, filepath, media_type in all_images:
package_files[f"OEBPS/{arcname}"] = filepath.read_bytes()
img_id = "img_" + re.sub(r"[^a-zA-Z0-9]+", "_", arcname)
manifest_items.append(f'<item id="{img_id}" href="{arcname}" media-type="{media_type}"/>')
# --- Chapters ---
for doc in docs:
fname = f"text/{doc['id']}.xhtml"
display_title = doc["title"]
xhtml_title = display_title if display_title else " "
h1 = f"<h1>{html.escape(display_title)}</h1>" if (display_title and show_chapter_titles) else ""
package_files[f"OEBPS/{fname}"] = CHAPTER_TEMPLATE.format(
title=html.escape(xhtml_title), body_class=body_class, h1=h1, content=doc["content"],
)
manifest_items.append(f'<item id="{doc["id"]}" href="{fname}" media-type="application/xhtml+xml"/>')
spine_items.append(f'<itemref idref="{doc["id"]}"/>')
if display_title:
nav_items.append(f'<li><a href="{fname}">{html.escape(display_title)}</a></li>')
manifest_extra = (
'<item id="css" href="css/style.css" media-type="text/css"/>\n'
' <item id="nav" href="nav.xhtml" media-type="application/xhtml+xml" properties="nav"/>\n'
' <item id="ncx" href="toc.ncx" media-type="application/x-dtbncx+xml"/>'
)
# --- Additional dc: metadata ---
dc_extra = []
if meta.get("publisher"):
dc_extra.append(f"<dc:publisher>{html.escape(meta['publisher'])}</dc:publisher>")
if meta.get("description"):
dc_extra.append(f"<dc:description>{html.escape(meta['description'])}</dc:description>")
if meta.get("date"):
dc_extra.append(f"<dc:date>{html.escape(meta['date'])}</dc:date>")
if meta.get("rights"):
dc_extra.append(f"<dc:rights>{html.escape(meta['rights'])}</dc:rights>")
if meta.get("series"):
dc_extra.append(f'<meta name="calibre:series" content="{html.escape(meta["series"])}"/>')
if meta.get("series_index"):
dc_extra.append(f'<meta name="calibre:series_index" content="{html.escape(str(meta["series_index"]))}"/>')
content_opf = f"""<?xml version="1.0" encoding="UTF-8"?>
<package xmlns="http://www.idpf.org/2007/opf" unique-identifier="BookId" version="3.0">
<metadata xmlns:dc="http://purl.org/dc/elements/1.1/">
<dc:identifier id="BookId">{book_id}</dc:identifier>
<dc:title>{html.escape(meta['title'])}</dc:title>
<dc:creator>{html.escape(meta['author'])}</dc:creator>
<dc:language>{meta['language']}</dc:language>
{cover_meta}
{chr(10).join(' ' + line for line in dc_extra)}
</metadata>
<manifest>
{manifest_extra}
{chr(10).join(' ' + item for item in manifest_items)}
</manifest>
<spine>
{chr(10).join(' ' + item for item in spine_items)}
</spine>
{guide_xml}
</package>
"""
nav_xhtml = f"""<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE html>
<html xmlns="http://www.w3.org/1999/xhtml" xmlns:epub="http://www.idpf.org/2007/ops" xml:lang="{meta['language']}">
<head><meta charset="UTF-8"/><title>Table of Contents</title></head>
<body>
<nav epub:type="toc" id="toc">
<h1>Table of Contents</h1>
<ol>
{chr(10).join(nav_items)}
</ol>
</nav>
</body>
</html>
"""
nav_points = []
playorder = 0
for doc in docs:
if not doc["title"]:
continue
playorder += 1
nav_points.append(f""" <navPoint id="navpoint-{playorder}" playOrder="{playorder}">
<navLabel><text>{html.escape(doc['title'])}</text></navLabel>
<content src="text/{doc['id']}.xhtml"/>
</navPoint>""")
toc_ncx = f"""<?xml version="1.0" encoding="UTF-8"?>
<ncx xmlns="http://www.daisy.org/z3986/2005/ncx/" version="2005-1">
<head><meta name="dtb:uid" content="{book_id}"/></head>
<docTitle><text>{html.escape(meta['title'])}</text></docTitle>
<navMap>
{chr(10).join(nav_points)}
</navMap>
</ncx>
"""
output_path.parent.mkdir(parents=True, exist_ok=True)
with zipfile.ZipFile(output_path, "w") as zf:
zf.writestr("mimetype", "application/epub+zip", compress_type=zipfile.ZIP_STORED)
zf.writestr("META-INF/container.xml", CONTAINER_XML)
zf.writestr("OEBPS/content.opf", content_opf)
zf.writestr("OEBPS/nav.xhtml", nav_xhtml)
zf.writestr("OEBPS/toc.ncx", toc_ncx)
zf.writestr("OEBPS/css/style.css", css_content)
for arcname, content in package_files.items():
zf.writestr(arcname, content)
print(f"EPUB built: {output_path}")
print(f"Total sections: {len(docs)}")
for doc in docs:
label = doc["title"] if doc["title"] else "(untitled, not in TOC)"
print(f" - {doc['id']}: {label}")
if cover_path:
print(f"Cover: {cover_path}")
if fonts:
print(f"Fonts embedded: {len(fonts)} (primary: {fonts[0].name})")
# ---------------------------------------------------------------------------
# Config and entry point
# ---------------------------------------------------------------------------
def load_config(config_path: Path) -> dict:
if not config_path.exists():
return {}
return json.loads(config_path.read_text(encoding="utf-8"))
def main():
parser = argparse.ArgumentParser(description="Build an EPUB from a chapters/ folder, with metadata from config.json")
parser.add_argument("--pages-dir", default=str(SCRIPT_DIR / "chapters"), help="Folder with chapter subfolders")
parser.add_argument("--font-dir", default=str(SCRIPT_DIR / "font"), help="Folder with the font to embed")
parser.add_argument("--config", default=str(SCRIPT_DIR / "config.json"), help="Path to config.json")
parser.add_argument("--output", default=None, help="Path to the resulting .epub (defaults to config.json's value)")
parser.add_argument("--cover", default=None, help="Explicit cover image path (overrides auto-detection and config.json)")
parser.add_argument("--title", default=None, help="Override title from config.json")
parser.add_argument("--author", default=None, help="Override author from config.json")
parser.add_argument("--language", default=None, help="Override language from config.json")
parser.add_argument("--vertical", action="store_true", default=None, help="Vertical Japanese writing-mode")
parser.add_argument(
"--show-chapter-titles", action="store_true", default=None,
help="Print the chapter title (<h1>) at the start of each chapter. By default it only appears in the TOC."
)
parser.add_argument(
"--split-pages", action="store_true",
help="Make each file its own xhtml document, instead of merging all files in a folder into one."
)
args = parser.parse_args()
config = load_config(Path(args.config))
meta = {
"title": args.title or config.get("title"),
"author": args.author or config.get("author", "Unknown"),
"language": args.language or config.get("language", "ja"),
"identifier": config.get("identifier") or None,
"font_family": config.get("font_family") or None,
"publisher": config.get("publisher"),
"description": config.get("description"),
"date": config.get("date"),
"rights": config.get("rights"),
"series": config.get("series"),
"series_index": config.get("series_index"),
}
if not meta["title"]:
print("No title set — put one in config.json or pass --title.", file=sys.stderr)
sys.exit(1)
vertical = args.vertical if args.vertical is not None else bool(config.get("vertical", False))
show_chapter_titles = (
args.show_chapter_titles if args.show_chapter_titles is not None
else bool(config.get("show_chapter_titles", False))
)
chapter_titles_map = config.get("chapter_titles", {})
output_path = Path(args.output) if args.output else Path(config.get("output", str(SCRIPT_DIR / "book.epub")))
pages_dir = Path(args.pages_dir)
font_dir = Path(args.font_dir)
explicit_cover = args.cover or config.get("cover")
cover_path = find_cover(pages_dir, explicit_cover)
fonts = find_fonts(font_dir)
docs, all_images = group_pages_into_docs(pages_dir, cover_path, chapter_titles_map, args.split_pages)
build_epub(
docs, all_images, cover_path, fonts, output_path,
meta=meta, vertical=vertical, show_chapter_titles=show_chapter_titles,
)
if __name__ == "__main__":
main()
epub_builder/config.example.json
+21
View File
@@ -0,0 +1,21 @@
{
"title": "Your Book Title",
"author": "Author Name",
"language": "ja",
"identifier": "",
"publisher": "",
"description": "",
"date": "",
"rights": "",
"series": "",
"series_index": "",
"font_family": "",
"vertical": false,
"show_chapter_titles": false,
"output": "book.epub",
"chapter_titles": {
"ch00_frontmatter": "",
"ch01_chapter00": "Chapter 00",
"ch08_backmatter": ""
}
}
epub_builder/furigana.py
+184
View File
@@ -0,0 +1,184 @@
#!/usr/bin/env python3
"""
Shared logic for turning OCR page text (see ocr/prompt.txt for the expected
format) into HTML paragraphs, including furigana rendered as proper <ruby>
markup. Used by build_epub.py, and also runnable standalone as a quick
preview/debugging tool for a single page.
Input text format (as produced by the OCR module):
- First line: page header/date/chapter-marker metadata.
- Every following line: one paragraph (no blank-line separators).
Furigana notation on input (Aozora Bunko style):
- base《reading》 Furigana over `base`. `base` is the nearest
contiguous run of kanji immediately before《.
- ｜base《reading》 Explicit start of `base`, needed when it doesn't
match a plain "kanji run" (e.g. it's shorter, or
contains non-kanji characters, or is glued to the
previous word without a natural boundary).
- Several ｜base《reading》 in a row are kept as separate <ruby> groups,
in reading order (e.g. a compound word whose furigana was printed
split across its parts).
Standalone usage:
python furigana.py --input page.txt --output page.xhtml
# or a quick one-off check:
echo "本文《ほんぶん》デザイン" | python furigana.py --stdin
"""
import argparse
import html
import re
import sys
from pathlib import Path
# Kanji range + the iteration mark 々 + a couple of CJK compatibility ranges
KANJI_RUN = r"[\u4e00-\u9fff\u3005\u3007\uf900-\ufaff]+"
# 1) Explicit base boundary via ｜: ｜<anything but ｜《>《reading》
RE_MARKED = re.compile(r"｜([^｜《]+?)《([^》]+)》")
# 2) No ｜: base is the nearest kanji run right before《
RE_AUTO = re.compile(r"(" + KANJI_RUN + r")《([^》]+)》")
# Strips a leading page number like "5 " or "23 " from the page header
LEADING_PAGE_NUMBER_RE = re.compile(r"^\d+\s*")
# Full-width space — the standard paragraph indent in Japanese typography
PARAGRAPH_INDENT = "\u3000"
XHTML_TEMPLATE = """<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE html>
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="ja">
<head>
<meta charset="UTF-8"/>
<title>{title}</title>
<style type="text/css">
{extra_style} p {{ margin: 0 0 1em 0; }}
p.page-header {{ font-size: 1.3em; font-weight: bold; margin: 1.5em 0 0 0; }}
p.spacer {{ margin: 0 0 1em 0; }}
rt {{ font-size: 0.5em; }}
</style>
</head>
<body>
{body}
</body>
</html>
"""
VERTICAL_STYLE = " body { writing-mode: vertical-rl; font-family: serif; }\n"
HORIZONTAL_STYLE = " body { font-family: serif; }\n"
def furigana_to_ruby(text: str) -> str:
"""Replaces Aozora Bunko furigana notation with <ruby><rb>...</rb><rt>...</rt></ruby>."""
def repl(m: re.Match) -> str:
base, reading = m.group(1), m.group(2)
return f"<ruby><rb>{base}</rb><rt>{reading}</rt></ruby>"
text = RE_MARKED.sub(repl, text)
text = RE_AUTO.sub(repl, text)
return text
def parse_page(text: str) -> tuple[str, list[str]]:
"""Returns (header, [paragraphs]) from a page's raw OCR text.
The literal marker "[NO_TEXT]" (written by the OCR module for pages
with no text at all, e.g. illustrations) parses as an empty page.
"""
if text.strip() == "[NO_TEXT]":
return "", []
lines = [ln.strip() for ln in text.strip("\n").split("\n") if ln.strip() != ""]
if not lines:
return "", []
return lines[0], lines[1:]
def strip_leading_page_number(header: str) -> str:
return LEADING_PAGE_NUMBER_RE.sub("", header, count=1)
def page_to_paragraphs_html(header: str, paragraphs: list[str]) -> str:
"""
Renders one page's (header, paragraphs) as HTML <p> elements:
- The header (with its leading page number stripped) is shown in a
larger, bold "page-header" paragraph — in the source novels this
line is usually an in-story date/time/chapter marker, not just
page-numbering noise.
- If the header is empty after stripping the page number, an empty
placeholder line is rendered instead, so the vertical rhythm of the
page stays consistent whether or not there was a header.
- A blank spacer line always follows the header.
- Each paragraph is prefixed with a full-width space (the standard
Japanese paragraph indent) rather than relying on CSS text-indent,
so the indent survives even in readers that ignore that CSS rule.
"""
header = strip_leading_page_number(header).strip()
header_html = furigana_to_ruby(html.escape(header, quote=False)) if header else "&#160;"
out = [
f'<p class="page-header">{header_html}</p>',
'<p class="spacer">&#160;</p>',
]
for para in paragraphs:
indented = PARAGRAPH_INDENT + para
escaped = html.escape(indented, quote=False)
out.append(f"<p>{furigana_to_ruby(escaped)}</p>")
return "\n".join(out)
def convert_page(raw_text: str) -> str:
"""Convenience wrapper: raw OCR page text -> HTML <p> fragment (no <html>/<body> wrapper)."""
header, paragraphs = parse_page(raw_text)
if not header and not paragraphs:
return ""
return page_to_paragraphs_html(header, paragraphs)
def main():
parser = argparse.ArgumentParser(description="Convert Aozora-style furigana notation to XHTML <ruby>")
parser.add_argument("--input", help="Path to a text file (one page's OCR output)")
parser.add_argument("--output", help="Where to write the XHTML (defaults to stdout)")
parser.add_argument("--stdin", action="store_true", help="Read text from stdin (for quick checks)")
parser.add_argument(
"--fragment", action="store_true",
help="Output only the <p> fragment, without wrapping it in a full XHTML document. "
"Useful when the fragment will be inserted into an assembled chapter file elsewhere."
)
parser.add_argument("--title", default="page", help="Document title (for --output, not --fragment)")
parser.add_argument(
"--vertical", action="store_true",
help="Vertical Japanese writing-mode (as in the original scan). "
"Defaults to horizontal (left-to-right) text, which renders more reliably "
"across readers and on mobile."
)
args = parser.parse_args()
if args.stdin:
raw_text = sys.stdin.read()
elif args.input:
raw_text = Path(args.input).read_text(encoding="utf-8")
else:
print("Provide --input <file> or --stdin", file=sys.stderr)
sys.exit(1)
body = convert_page(raw_text)
if args.fragment:
result = body
else:
extra_style = VERTICAL_STYLE if args.vertical else HORIZONTAL_STYLE
result = XHTML_TEMPLATE.format(title=html.escape(args.title), body=body, extra_style=extra_style)
if args.output:
Path(args.output).write_text(result, encoding="utf-8")
print(f"Saved: {args.output}")
else:
print(result)
if __name__ == "__main__":
main()
ocr/README.md
+67
View File
@@ -0,0 +1,67 @@
# OCR module
Batch-transcribes scanned light novel pages (vertical Japanese text) into
per-page `.txt` files. This is step 1 of the pipeline — step 2 is the
[`epub_builder`](../epub_builder/README.md) module, which turns these `.txt`
files into a finished `.epub`.
## Choosing a backend
| Script | Backend | Setup needed | Notes |
|---|---|---|---|
| `openrouter_ocr.py` | Multimodal LLM via any OpenAI-compatible API | API key + base URL | **Recommended.** Reads a whole page at once, no column segmentation needed. |
| `gemini_direct_ocr.py` | Gemini API directly | Google AI Studio API key | Same idea as above, direct instead of via a proxy. |
| `google_vision_ocr.py` | Google Cloud Vision (classic OCR) | Google Cloud project + billing enabled | No LLM context understanding, but solid on clean scans. Free tier covers a typical light novel volume, but Google still requires a billing account to be linked. |
| `local_mangaocr_ocr.py` | [manga-ocr](https://github.com/kha-white/manga-ocr), fully offline | None — no account, no API key | Slower to set up quality-wise: manga-ocr expects short text blocks, so this script auto-segments each page into vertical columns before OCR-ing each one. |
If you have API access to a multimodal model (Gemini, GPT-4V-class models,
etc.) through any provider, `openrouter_ocr.py` is the easiest and generally
gives the best results with the least fuss.
## Setup for `openrouter_ocr.py` (recommended)
```bash
pip install openai pillow natsort tqdm
```
1. Copy `config.example.json` to `config.json` and fill in your `api_key`,
`base_url` (your provider's OpenAI-compatible endpoint, usually ending in
`/v1`), and `model` identifier.
2. Optionally edit `prompt.txt` — it's plain English text, no need to touch
any code to tweak the instructions given to the model.
```bash
python openrouter_ocr.py --input ./pages --output ./out
```
- `--input`: folder with scanned page images (jpg/png/...), named so that
alphabetical sorting matches page order (natural sort is used, so
`page2.jpg` and `page10.jpg` sort correctly too).
- `--output`: folder for results. Creates `pages_txt/<name>.txt` (one file
per page) plus a `combined.md` preview of the whole run.
- If interrupted, just re-run with the same `--output` — pages that already
have a `.txt` file are skipped, so nothing already done gets re-sent
(and re-billed).
- A page that fails to OCR (network error, rate limit, etc.) does **not**
get an empty file written for it, specifically so the next run retries it
instead of silently treating it as done.
- A page the model judges to be empty (illustration-only, blank, or a
cover/technical page) gets a file containing exactly the literal text
`[NO_TEXT]` — this is a deliberate marker, not an OCR failure. The
`epub_builder` module knows to treat it as "no text on this page".
- `--sleep N` adds a delay (seconds) between requests if you're hitting
rate limits.
## Other backends
`gemini_direct_ocr.py`, `google_vision_ocr.py`, and `local_mangaocr_ocr.py`
follow the same `--input`/`--output` convention and produce the same
`pages_txt/*.txt` + `combined.md` output — see each script's own docstring
for backend-specific setup.
## Next step
Once you have a `pages_txt/` folder full of `.txt` files, manually sort the
pages into the folder structure `epub_builder` expects (see
[`epub_builder/README.md`](../epub_builder/README.md)), then run the epub
builder.
ocr/__init__.py
+15
View File
@@ -0,0 +1,15 @@
"""
OCR module: batch-transcribes scanned light novel pages (vertical Japanese
text) into per-page .txt files, ready for the epub_builder module.
Several interchangeable backends are provided as standalone scripts:
- openrouter_ocr.py Recommended. Any OpenAI-compatible API (OpenRouter,
a self-hosted proxy, etc.) with a multimodal model.
- gemini_direct_ocr.py Same idea, but calling the Gemini API directly.
- google_vision_ocr.py Classic OCR via Google Cloud Vision (no LLM).
- local_mangaocr_ocr.py Fully offline, no cloud account, via manga-ocr.
Each script is self-contained and runnable directly, e.g.:
python -m ocr.openrouter_ocr --input ./pages --output ./out
"""
ocr/config.example.json
+5
View File
@@ -0,0 +1,5 @@
{
"api_key": "PUT_YOUR_API_KEY_HERE",
"base_url": "https://your-provider.example.com/v1",
"model": "google/gemini-3.7-flash"
}
ocr/gemini_direct_ocr.py
+146
View File
@@ -0,0 +1,146 @@
#!/usr/bin/env python3
"""
Batch-OCR for scanned pages of a Japanese light novel (vertical text) via the
Gemini API directly (not through a proxy).
Usage:
export GEMINI_API_KEY="your_key"
python gemini_direct_ocr.py --input ./pages --output ./out
Requirements:
pip install google-genai pillow natsort tqdm
If you access Gemini (or another model) through OpenRouter or a similar
OpenAI-compatible proxy instead of a direct Google API key, use
openrouter_ocr.py instead — it's the recommended entry point for this
project and shares the same prompt.txt / config.json workflow.
"""
import argparse
import os
import sys
import time
from pathlib import Path
from google import genai
from google.genai import types
from natsort import natsorted
from PIL import Image
from tqdm import tqdm
IMAGE_EXTS = {".jpg", ".jpeg", ".png", ".webp", ".tif", ".tiff", ".bmp"}
PROMPT = """\
This is a scanned page from a Japanese light novel, printed in vertical text (縦書き).
Transcribe ALL Japanese text on the page, preserving correct reading order
(columns read top-to-bottom, then right-to-left).
Rules:
- Do not translate or summarize — output only the transcribed Japanese text.
- No line numbers, no commentary, no explanations of your own.
- Furigana may be omitted (only the base kanji/kana text is needed).
- If the page has no text at all (illustration-only, blank, cover/technical
page), output exactly: [NO_TEXT]
- Preserve paragraph breaks where the layout clearly shows them.
"""
def ocr_image(client: genai.Client, model: str, path: Path, retries: int = 3) -> str:
img = Image.open(path)
# Downscale very large scans — speeds up and cheapens the request without
# a noticeable loss of OCR quality.
max_dim = 2200
if max(img.size) > max_dim:
ratio = max_dim / max(img.size)
img = img.resize((int(img.width * ratio), int(img.height * ratio)))
last_err = None
for attempt in range(retries):
try:
response = client.models.generate_content(
model=model,
contents=[PROMPT, img],
config=types.GenerateContentConfig(temperature=0),
)
text = (response.text or "").strip()
return text
except Exception as e: # noqa: BLE001
last_err = e
time.sleep(3 * (attempt + 1))
raise RuntimeError(f"Failed to OCR {path.name} after {retries} attempts: {last_err}")
def main():
parser = argparse.ArgumentParser(description="Batch-OCR a light novel via the Gemini API")
parser.add_argument("--input", required=True, help="Folder with scanned page images (jpg/png)")
parser.add_argument("--output", required=True, help="Folder for the OCR results")
parser.add_argument("--model", default="gemini-3.7-flash", help="Gemini model name")
parser.add_argument(
"--api-key", default=None,
help="API key (falls back to the GEMINI_API_KEY environment variable)"
)
parser.add_argument("--start-page", type=int, default=1)
parser.add_argument(
"--sleep", type=float, default=0.0,
help="Delay in seconds between requests (useful if you're hitting rate limits)"
)
args = parser.parse_args()
api_key = args.api_key or os.environ.get("GEMINI_API_KEY")
if not api_key:
print("No API key found. Pass --api-key or set GEMINI_API_KEY.", file=sys.stderr)
sys.exit(1)
input_dir = Path(args.input)
output_dir = Path(args.output)
output_dir.mkdir(parents=True, exist_ok=True)
pages_dir = output_dir / "pages_txt"
pages_dir.mkdir(exist_ok=True)
images = [p for p in input_dir.iterdir() if p.suffix.lower() in IMAGE_EXTS]
images = natsorted(images, key=lambda p: p.name)
if not images:
print(f"No images found in {input_dir}.", file=sys.stderr)
sys.exit(1)
print(f"Pages found: {len(images)}")
client = genai.Client(api_key=api_key)
combined_path = output_dir / "combined.md"
failed = []
with open(combined_path, "w", encoding="utf-8") as combined_f:
for idx, img_path in enumerate(tqdm(images, desc="OCR"), start=args.start_page):
txt_out = pages_dir / f"{img_path.stem}.txt"
if txt_out.exists():
text = txt_out.read_text(encoding="utf-8")
else:
try:
text = ocr_image(client, args.model, img_path)
txt_out.write_text(text, encoding="utf-8")
except Exception as e: # noqa: BLE001
print(f"\nError on {img_path.name}: {e}", file=sys.stderr)
failed.append(img_path.name)
text = ""
# Do NOT write a file to disk on failure — see openrouter_ocr.py
# for the reasoning.
if args.sleep:
time.sleep(args.sleep)
combined_f.write(f"\n\n<!-- page {idx}: {img_path.name} -->\n\n")
combined_f.write(text)
print(f"\nDone. Combined file: {combined_path}")
print(f"Per-page files: {pages_dir}")
if failed:
print(f"\nFailed to OCR {len(failed)} page(s):")
for name in failed:
print(f" - {name}")
print("Re-run the script with the same --output folder — already-done pages will not be redone.")
if __name__ == "__main__":
main()
ocr/google_vision_ocr.py
+133
View File
@@ -0,0 +1,133 @@
#!/usr/bin/env python3
"""
Batch-OCR for scanned pages of a Japanese light novel (vertical text) via
Google Cloud Vision.
Usage:
python google_vision_ocr.py --input ./pages --output ./out --credentials ./key.json
Requirements:
pip install google-cloud-vision natsort tqdm
Google Cloud setup (one-time, ~5-10 minutes):
1. Go to https://console.cloud.google.com/
2. Create a project (or use an existing one)
3. Search for "Vision API" -> Enable
4. Go to "APIs & Services" -> "Credentials" -> "Create Credentials" -> "Service account"
5. Create the service account (role can be left unset, or "Editor")
6. Open the account -> Keys -> Add Key -> JSON -> downloads key.json
7. Point --credentials at that file
Note: as of writing, Google requires a billing account to be enabled on the
project before the Vision API will respond, even though usage stays within
the free tier (1000 requests/month covers ~600 pages comfortably). No charge
should occur unless you exceed that quota.
This is a classic OCR engine (not an LLM) — generally solid for image
quality, but it doesn't understand context the way a multimodal model does.
For light novel pages with dense vertical prose, openrouter_ocr.py /
gemini_direct_ocr.py usually give better results with less setup friction.
"""
import argparse
import io
import os
import sys
import time
from pathlib import Path
from google.cloud import vision
from natsort import natsorted
from tqdm import tqdm
IMAGE_EXTS = {".jpg", ".jpeg", ".png", ".webp", ".tif", ".tiff", ".bmp"}
def ocr_image(client: vision.ImageAnnotatorClient, path: Path, retries: int = 3) -> str:
"""OCRs a single scan, returning text in reading order."""
with io.open(path, "rb") as f:
content = f.read()
image = vision.Image(content=content)
# The "ja" language hint helps the model handle vertical Japanese more accurately
image_context = vision.ImageContext(language_hints=["ja"])
last_err = None
for attempt in range(retries):
try:
response = client.document_text_detection(
image=image, image_context=image_context
)
if response.error.message:
raise RuntimeError(response.error.message)
return response.full_text_annotation.text
except Exception as e: # noqa: BLE001
last_err = e
time.sleep(2 * (attempt + 1))
raise RuntimeError(f"Failed to OCR {path.name} after {retries} attempts: {last_err}")
def main():
parser = argparse.ArgumentParser(description="Batch-OCR a light novel via Google Cloud Vision")
parser.add_argument("--input", required=True, help="Folder with scanned page images (jpg/png)")
parser.add_argument("--output", required=True, help="Folder for the OCR results")
parser.add_argument("--credentials", required=True, help="Path to the service-account key.json")
parser.add_argument(
"--start-page", type=int, default=1, help="Page number to start the header numbering from"
)
args = parser.parse_args()
os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = args.credentials
input_dir = Path(args.input)
output_dir = Path(args.output)
output_dir.mkdir(parents=True, exist_ok=True)
pages_dir = output_dir / "pages_txt"
pages_dir.mkdir(exist_ok=True)
images = [p for p in input_dir.iterdir() if p.suffix.lower() in IMAGE_EXTS]
images = natsorted(images, key=lambda p: p.name)
if not images:
print(f"No images found in {input_dir}.", file=sys.stderr)
sys.exit(1)
print(f"Pages found: {len(images)}")
client = vision.ImageAnnotatorClient()
combined_path = output_dir / "combined.md"
failed = []
with open(combined_path, "w", encoding="utf-8") as combined_f:
for idx, img_path in enumerate(tqdm(images, desc="OCR"), start=args.start_page):
txt_out = pages_dir / f"{img_path.stem}.txt"
# Skip pages already OCR'd — handy if a previous run was interrupted
if txt_out.exists():
text = txt_out.read_text(encoding="utf-8")
else:
try:
text = ocr_image(client, img_path)
txt_out.write_text(text, encoding="utf-8")
except Exception as e: # noqa: BLE001
print(f"\nError on {img_path.name}: {e}", file=sys.stderr)
failed.append(img_path.name)
text = ""
# Do NOT write a file to disk on failure — otherwise the next
# run would see the file exists and skip retrying it.
combined_f.write(f"\n\n<!-- page {idx}: {img_path.name} -->\n\n")
combined_f.write(text)
print(f"\nDone. Combined file: {combined_path}")
print(f"Per-page files: {pages_dir}")
if failed:
print(f"\nFailed to OCR {len(failed)} page(s):")
for name in failed:
print(f" - {name}")
print("Re-run the script with the same --output folder — already-done pages will not be redone.")
if __name__ == "__main__":
main()
ocr/local_mangaocr_ocr.py
+174
View File
@@ -0,0 +1,174 @@
#!/usr/bin/env python3
"""
Fully local, offline batch-OCR for scanned pages of a Japanese light novel
(vertical text) via manga-ocr — no cloud account or credit card required.
Usage:
python local_mangaocr_ocr.py --input ./pages --output ./out
Requirements:
pip install manga-ocr opencv-python pillow natsort tqdm
The first run downloads manga-ocr's model weights (~400 MB) from
HuggingFace; after that everything runs offline. Without a GPU it runs on
CPU, just slower (roughly 1-3 sec per column).
How it works:
1. Each page is cut into vertical text columns (by detecting whitespace
gaps between columns — typical light novel layout).
2. Columns are sorted right-to-left (the reading order for vertical
Japanese text).
3. Each column is OCR'd separately via manga-ocr.
4. Results are joined back into per-page text.
manga-ocr was trained mainly on manga speech bubbles (short text blocks),
not dense full-page prose, so column segmentation matters a lot here for
quality. If segmentation performs poorly on your scans (e.g. unusual
layout), pass --whole-page to feed the model the full page without cutting
it into columns (simpler, but usually lower quality on dense prose).
"""
import argparse
import sys
from pathlib import Path
import cv2
import numpy as np
from natsort import natsorted
from PIL import Image
from tqdm import tqdm
IMAGE_EXTS = {".jpg", ".jpeg", ".png", ".webp", ".tif", ".tiff", ".bmp"}
def find_columns(img_gray: np.ndarray, min_col_width: int = 12, gap_threshold: int = 4):
"""Finds x-ranges of vertical text columns via a pixel-density projection.
Returns a list of (x_start, x_end), sorted RIGHT-TO-LEFT (the reading
order for vertical Japanese text).
"""
# Binarize: text (dark) -> white, background -> black
_, binary = cv2.threshold(img_gray, 0, 255, cv2.THRESH_BINARY_INV + cv2.THRESH_OTSU)
# Dilate vertically a bit to merge characters within a column into one solid strip
kernel = np.ones((25, 1), np.uint8)
dilated = cv2.dilate(binary, kernel, iterations=1)
col_sums = dilated.sum(axis=0) # text density per pixel column
has_text = col_sums > 0
columns = []
x = 0
width = len(has_text)
while x < width:
if has_text[x]:
start = x
while x < width and (has_text[x] or _gap_too_small(has_text, x, gap_threshold)):
x += 1
end = x
if end - start >= min_col_width:
columns.append((start, end))
else:
x += 1
columns.sort(key=lambda c: c[0], reverse=True) # right-to-left
return columns
def _gap_too_small(has_text: np.ndarray, x: int, gap_threshold: int) -> bool:
"""Checks whether a text-free gap is shorter than gap_threshold (to avoid splitting a column needlessly)."""
if has_text[x]:
return False
end = x
while end < len(has_text) and not has_text[end]:
end += 1
return (end - x) < gap_threshold
def ocr_page(mocr, pil_img: Image.Image, whole_page: bool) -> str:
if whole_page:
return mocr(pil_img)
img_np = np.array(pil_img.convert("L"))
columns = find_columns(img_np)
if not columns:
# No columns detected (e.g. an illustration-only page) — fall back to the whole page
return mocr(pil_img)
texts = []
for x_start, x_end in columns:
pad = 4
crop = pil_img.crop((max(0, x_start - pad), 0, min(pil_img.width, x_end + pad), pil_img.height))
text = mocr(crop)
if text.strip():
texts.append(text.strip())
return "\n".join(texts)
def main():
parser = argparse.ArgumentParser(description="Local batch-OCR for a light novel via manga-ocr")
parser.add_argument("--input", required=True, help="Folder with scanned page images (jpg/png)")
parser.add_argument("--output", required=True, help="Folder for the OCR results")
parser.add_argument(
"--whole-page", action="store_true",
help="Skip column segmentation, feed the whole page to the model at once"
)
parser.add_argument("--start-page", type=int, default=1)
args = parser.parse_args()
input_dir = Path(args.input)
output_dir = Path(args.output)
output_dir.mkdir(parents=True, exist_ok=True)
pages_dir = output_dir / "pages_txt"
pages_dir.mkdir(exist_ok=True)
images = [p for p in input_dir.iterdir() if p.suffix.lower() in IMAGE_EXTS]
images = natsorted(images, key=lambda p: p.name)
if not images:
print(f"No images found in {input_dir}.", file=sys.stderr)
sys.exit(1)
print(f"Pages found: {len(images)}")
print("Loading the manga-ocr model (downloads weights on first run, ~400 MB)...")
from manga_ocr import MangaOcr
mocr = MangaOcr()
combined_path = output_dir / "combined.md"
failed = []
with open(combined_path, "w", encoding="utf-8") as combined_f:
for idx, img_path in enumerate(tqdm(images, desc="OCR"), start=args.start_page):
txt_out = pages_dir / f"{img_path.stem}.txt"
if txt_out.exists():
text = txt_out.read_text(encoding="utf-8")
else:
try:
pil_img = Image.open(img_path)
text = ocr_page(mocr, pil_img, args.whole_page)
txt_out.write_text(text, encoding="utf-8")
except Exception as e: # noqa: BLE001
print(f"\nError on {img_path.name}: {e}", file=sys.stderr)
failed.append(img_path.name)
text = ""
# Do NOT write a file to disk on failure — otherwise the next
# run would see the file exists and skip retrying it.
combined_f.write(f"\n\n<!-- page {idx}: {img_path.name} -->\n\n")
combined_f.write(text)
print(f"\nDone. Combined file: {combined_path}")
print(f"Per-page files: {pages_dir}")
if failed:
print(f"\nFailed to OCR {len(failed)} page(s):")
for name in failed:
print(f" - {name}")
print("Re-run the script with the same --output folder — already-done pages will not be redone.")
if __name__ == "__main__":
main()
ocr/openrouter_ocr.py
+191
View File
@@ -0,0 +1,191 @@
#!/usr/bin/env python3
"""
Batch-OCR for scanned pages of a Japanese light novel (vertical text) via any
OpenAI-compatible API (OpenRouter, a self-hosted proxy, etc.).
This is the recommended OCR backend for this project: it lets a multimodal
LLM "read" a page image directly, including vertical Japanese text, without
any column-segmentation preprocessing.
Usage (after filling in config.json):
python openrouter_ocr.py --input ./pages --output ./out
Requirements:
pip install openai pillow natsort tqdm
One-time setup:
1. Copy config.example.json -> config.json
2. Fill in your api_key, base_url and model
3. Optionally edit prompt.txt to fit your book / house style
"""
import argparse
import base64
import io
import json
import os
import sys
import time
from pathlib import Path
from natsort import natsorted
from openai import OpenAI
from PIL import Image
from tqdm import tqdm
IMAGE_EXTS = {".jpg", ".jpeg", ".png", ".webp", ".tif", ".tiff", ".bmp"}
SCRIPT_DIR = Path(__file__).resolve().parent
def load_config(config_path: Path) -> dict:
if not config_path.exists():
return {}
try:
return json.loads(config_path.read_text(encoding="utf-8"))
except json.JSONDecodeError as e:
print(f"Failed to parse {config_path}: {e}", file=sys.stderr)
sys.exit(1)
def load_prompt(prompt_path: Path) -> str:
if not prompt_path.exists():
print(f"Prompt file not found: {prompt_path}", file=sys.stderr)
sys.exit(1)
return prompt_path.read_text(encoding="utf-8").strip()
def image_to_data_url(path: Path, max_dim: int = 2200) -> str:
"""Downscale (if needed) and encode an image as a base64 data URL."""
img = Image.open(path)
if img.mode not in ("RGB", "L"):
img = img.convert("RGB")
if max(img.size) > max_dim:
ratio = max_dim / max(img.size)
img = img.resize((int(img.width * ratio), int(img.height * ratio)))
buf = io.BytesIO()
img.save(buf, format="JPEG", quality=90)
b64 = base64.b64encode(buf.getvalue()).decode("utf-8")
return f"data:image/jpeg;base64,{b64}"
def ocr_image(client: OpenAI, model: str, prompt: str, path: Path, retries: int = 3) -> str:
data_url = image_to_data_url(path)
last_err = None
for attempt in range(retries):
try:
response = client.chat.completions.create(
model=model,
temperature=0,
messages=[
{
"role": "user",
"content": [
{"type": "text", "text": prompt},
{"type": "image_url", "image_url": {"url": data_url}},
],
}
],
)
text = (response.choices[0].message.content or "").strip()
return text
except Exception as e: # noqa: BLE001
last_err = e
time.sleep(3 * (attempt + 1))
raise RuntimeError(f"Failed to OCR {path.name} after {retries} attempts: {last_err}")
def main():
parser = argparse.ArgumentParser(description="Batch-OCR a light novel via an OpenAI-compatible API")
parser.add_argument("--input", required=True, help="Folder with scanned page images (jpg/png)")
parser.add_argument("--output", required=True, help="Folder for the OCR results")
parser.add_argument(
"--config", default=str(SCRIPT_DIR / "config.json"),
help="Path to config.json with api_key/base_url/model (defaults to config.json next to this script)"
)
parser.add_argument(
"--prompt-file", default=str(SCRIPT_DIR / "prompt.txt"),
help="Path to the prompt file (defaults to prompt.txt next to this script)"
)
parser.add_argument("--base-url", default=None, help="Override base_url from config.json")
parser.add_argument("--model", default=None, help="Override model from config.json")
parser.add_argument("--api-key", default=None, help="Override api_key from config.json")
parser.add_argument("--start-page", type=int, default=1)
parser.add_argument(
"--sleep", type=float, default=0.0,
help="Delay in seconds between requests (useful if you're hitting rate limits)"
)
args = parser.parse_args()
config = load_config(Path(args.config))
api_key = args.api_key or config.get("api_key") or os.environ.get("API_KEY")
base_url = args.base_url or config.get("base_url")
model = args.model or config.get("model")
missing = [name for name, val in [("api_key", api_key), ("base_url", base_url), ("model", model)] if not val]
if missing:
print(
f"Missing settings: {', '.join(missing)}. "
f"Fill them in {args.config} (see config.example.json) or pass --api-key/--base-url/--model.",
file=sys.stderr,
)
sys.exit(1)
prompt = load_prompt(Path(args.prompt_file))
input_dir = Path(args.input)
output_dir = Path(args.output)
output_dir.mkdir(parents=True, exist_ok=True)
pages_dir = output_dir / "pages_txt"
pages_dir.mkdir(exist_ok=True)
images = [p for p in input_dir.iterdir() if p.suffix.lower() in IMAGE_EXTS]
images = natsorted(images, key=lambda p: p.name)
if not images:
print(f"No images found in {input_dir}.", file=sys.stderr)
sys.exit(1)
print(f"Pages found: {len(images)}")
client = OpenAI(base_url=base_url, api_key=api_key)
combined_path = output_dir / "combined.md"
failed = []
with open(combined_path, "w", encoding="utf-8") as combined_f:
for idx, img_path in enumerate(tqdm(images, desc="OCR"), start=args.start_page):
txt_out = pages_dir / f"{img_path.stem}.txt"
if txt_out.exists():
text = txt_out.read_text(encoding="utf-8")
else:
try:
text = ocr_image(client, model, prompt, img_path)
txt_out.write_text(text, encoding="utf-8")
except Exception as e: # noqa: BLE001
print(f"\nError on {img_path.name}: {e}", file=sys.stderr)
failed.append(img_path.name)
text = ""
# Do NOT write a file to disk on failure — otherwise the next
# run would see the file exists and assume the page is already
# done, silently skipping a retry forever.
if args.sleep:
time.sleep(args.sleep)
combined_f.write(f"\n\n<!-- page {idx}: {img_path.name} -->\n\n")
combined_f.write(text)
print(f"\nDone. Combined file: {combined_path}")
print(f"Per-page files: {pages_dir}")
if failed:
print(f"\nFailed to OCR {len(failed)} page(s):")
for name in failed:
print(f" - {name}")
print("Re-run the script with the same --output folder — already-done pages will not be redone.")
if __name__ == "__main__":
main()
ocr/prompt.txt
+11
View File
@@ -0,0 +1,11 @@
This is a scanned page from a Japanese light novel, printed in vertical text (縦書き). Transcribe ALL Japanese text on the page, preserving correct reading order (columns read top-to-bottom, then right-to-left).
Rules:
- Mark furigana using Aozora Bunko notation: base_text《reading》, e.g. 本文《ほんぶん》.
- If the furigana applies to only part of a word, or the start of the kanji run is ambiguous, mark the exact start of the base text with a leading ｜, e.g. ｜中《ちゅう》.
- If a compound word has furigana split across its parts (as printed), keep them as separate 《》 groups in reading order, e.g. ｜事務《じむ》｜所《しょ》.
- Include header/margin/footer text exactly as printed (page number, chapter title, in-story date/time stamp, file code). This novel often prints the same date/time/chapter info in two places on a page (e.g. top margin and a vertical strip on the side) — merge these into ONE single header line at the very start of your output, do not repeat it twice.
- Do not translate, summarize, or explain — output only the transcribed Japanese text.
- No line numbers, no commentary, no markdown formatting.
- Each paragraph goes on its own line. Do not merge multiple paragraphs into one line, and do not add blank lines between paragraphs.
- If the page has no text at all (illustration-only, blank, cover), output exactly: [NO_TEXT]
requirements.txt
+23
View File
@@ -0,0 +1,23 @@
# Core — always needed
natsort
Pillow
tqdm
# --- OCR backends: install only the one(s) you actually use ---
# ocr/openrouter_ocr.py (recommended) — any OpenAI-compatible API
openai
# ocr/gemini_direct_ocr.py — direct Gemini API access
# google-genai
# ocr/google_vision_ocr.py — Google Cloud Vision
# google-cloud-vision
# ocr/local_mangaocr_ocr.py — fully offline, no cloud account
# manga-ocr
# opencv-python
# numpy
# --- epub_builder module ---
# Only needs natsort, already listed above.