commit 41c6e97cd43f91967de6b1e3564b4e334b4c4226 Author: Poison Flower Date: Tue Sep 1 00:30:34 2026 +0300 init diff --git a/README.md b/README.md new file mode 100644 index 0000000..ef058bb --- /dev/null +++ b/README.md @@ -0,0 +1,82 @@ +# Light Novel Scan -> EPUB + +Turns a folder of scanned light novel page images (vertical Japanese text) +into a proper `.epub`, with furigana rendered as real `` markup, +illustrations kept in place, an embedded font, and epub metadata — using a +multimodal LLM to do the OCR instead of a traditional column-segmentation +pipeline. + +## Pipeline + +``` +scanned page images + │ + ▼ + ocr/ module → transcribes each page into pages_txt/.txt + │ + ▼ + (manual step) → sort the .txt files (and any illustration + images) into chapters/chNN_name/ folders + │ + ▼ +epub_builder/ module → assembles chapters/ into a finished .epub +``` + +1. **[`ocr/`](ocr/README.md)** — batch-OCRs scanned pages into per-page + `.txt` files. Several backends are available (a multimodal LLM via any + OpenAI-compatible API is the recommended default; Google Cloud Vision + and a fully offline `manga-ocr` pipeline are also included). +2. **Manual sorting** — split the resulting pages into chapter folders + (`ch00_frontmatter/`, `ch01_chapter00/`, ...), optionally dropping in a + `cover.jpg` and illustration images alongside the `.txt` files. This + step is manual because automatically detecting chapter boundaries from + OCR'd headers turned out to be unreliable — folder structure is simple + and unambiguous instead. +3. **[`epub_builder/`](epub_builder/README.md)** — assembles the sorted + `chapters/` folder into a valid `.epub`: furigana notation becomes + `` markup, images are placed inline, a font from `font/` gets + embedded, and metadata comes from `config.json`. + +See each module's own README for setup and usage details. + +## Quick start + +```bash +# 1. OCR +cd ocr +pip install openai pillow natsort tqdm +cp config.example.json config.json # fill in api_key / base_url / model +python openrouter_ocr.py --input /path/to/scans --output ./out + +# 2. Sort ./out/pages_txt/*.txt by hand into epub_builder/chapters/chNN_name/ + +# 3. Build the epub +cd ../epub_builder +pip install natsort +cp config.example.json config.json # fill in title / author / ... +python build_epub.py +``` + +## ⚠️ A note on copyright + +This repository contains only the **tooling**. It is not meant to, and +should not, be used to host or distribute: +- scanned page images, +- OCR'd text extracted from a copyrighted book, +- or a resulting `.epub` file, + +for any book you don't hold the rights to. The `.gitignore` in this repo +already excludes `chapters/`, `pages_txt/`, `out/`, and `*.epub` for this +reason — keep it that way if you fork or extend this project. This tool is +intended for personal-use digitization of books you own, not redistribution. + +## Requirements + +See [`requirements.txt`](requirements.txt) for the full list. Not every +dependency is needed at once — install only what the OCR backend and +features you're using require (see each module's README). + +## Status + +This is a personal toolkit, still evolving. Contributions/forks welcome, +but expect rough edges — issues and PRs are handled best-effort. diff --git a/epub_builder/README.md b/epub_builder/README.md new file mode 100644 index 0000000..a9b1601 --- /dev/null +++ b/epub_builder/README.md @@ -0,0 +1,182 @@ +# EPUB builder module + +Assembles a `chapters/` folder — laid out by hand into per-chapter +subfolders — into a finished, valid `.epub`, with furigana rendered as +proper `` markup, an embedded font, a cover image, and metadata from +`config.json`. + +This is step 2 of the pipeline; step 1 is the [`ocr`](../ocr/README.md) +module, which produces the per-page `.txt` files you'll sort into +`chapters/`. + +## Layout + +``` +epub_builder/ + chapters/ + cover.jpg <- cover image (optional, any "cover.*" directly in chapters/) + ch00_frontmatter/ <- everything before the main text (title page, TOC, ...) + 001.txt + 002.jpg + ch01_chapter00/ + 003.txt + 004.txt + 005.jpg + ch02_chapter01/ + 006.txt + ch08_backmatter/ <- everything after the main text (afterword, ...) + ... + font/ + YourFont.ttf <- font to embed (optional) + build_epub.py + furigana.py + config.json +``` + +## Chapter folder naming: `chNN_name` + +- `chNN` — sequence number, determines sort order (`ch00` → `ch01` → ... → + `ch10`; use leading zeros if you have more than 9 chapters so plain + alphabetical sort stays correct). +- `name` — default chapter title (override via `config.json` → + `chapter_titles`). +- Folders with `frontmatter` or `backmatter` anywhere in their name are + **always** ignored: no title, no TOC entry — regardless of any flag or + config setting. Their content is still included in the book, just + without a section label. + +Files inside a folder are sorted strictly by the number in the filename +(extension doesn't matter — an image `004.jpg` will land between +`003.txt` and `005.txt`). + +## Page layout + +- The first line of each `.txt` page (after stripping a leading page + number) is rendered as a larger, bold "page header" — typically an + in-story date/time/chapter marker (relevant if you're OCR-ing a novel + where that detail matters to the plot, e.g. one involving time travel). +- If nothing is left after stripping the page number, two blank lines are + shown instead, so the page's vertical rhythm stays consistent. +- A blank line always follows the header, then the text. +- Each paragraph starts with a full-width space (` `) — the standard + indent in Japanese typography, embedded directly in the text rather than + relying on CSS `text-indent` (which not every reader honors). + +## config.json + +```json +{ + "title": "Your Book Title", + "author": "Author Name", + "language": "ja", + "identifier": "", + "publisher": "", + "description": "", + "date": "", + "rights": "", + "series": "", + "series_index": "", + "font_family": "", + "vertical": false, + "show_chapter_titles": false, + "output": "book.epub", + "chapter_titles": { + "ch00_frontmatter": "", + "ch01_chapter00": "Chapter 00", + "ch08_backmatter": "" + } +} +``` + +- `title`, `author`, `language` — core metadata (`title` is required). +- `identifier`, `publisher`, `description`, `date`, `rights` — optional + Dublin Core metadata, only added if non-empty. Leave `identifier` empty + to get a random UUID generated on every build; set it once and keep it + stable if you plan to rebuild the file repeatedly and want reading apps + to recognize it as the same book. +- `series` / `series_index` — Calibre-style series metadata + (`calibre:series` / `calibre:series_index`), recognized by Calibre, + KOReader, and several other reading apps. +- `font_family` — explicit name for the embedded font family (see "Font" + below). Optional — without it, the name is guessed automatically. +- `vertical` — vertical Japanese writing-mode (also toggleable via + `--vertical`). +- `show_chapter_titles` — whether to print the chapter title (`

`) at + the start of each chapter. Defaults to `false` — titles still show up in + the table of contents either way. Toggle via `--show-chapter-titles`. +- `output` — path for the resulting epub, used if `--output` isn't passed. +- `chapter_titles` — folder name -> display title mapping. An empty string + `""` means "no heading, no TOC entry" (this is already the default + behavior for `frontmatter`/`backmatter` folders, so listing them here is + optional, just for clarity). + +`frontmatter`/`backmatter` entries in `chapter_titles` are ignored no +matter what — they can't be "turned back on" via config. + +## Font + +Drop font file(s) (`.ttf`/`.otf`/`.woff`/`.woff2`) into `font/`. Name them +`Regular.ttf` / `Bold.ttf` / `Italic.ttf` / `BoldItalic.ttf` (case +doesn't matter, and the keyword can be part of a longer name, e.g. +`NotoSerifJP-Bold.ttf`) and they'll automatically be unified under one +font-family name with the correct `font-weight`/`font-style`. That means a +bold chapter heading (`

`, and the page-header line) will automatically +pick up `Bold.ttf`, and italics (``), if any ever show up in the text, +will pick up `Italic.ttf`. + +If the filenames don't match this pattern, the old behavior kicks in: +each file gets its own font-family name, and only the first one +(alphabetically) is wired up automatically in `body { font-family }`; the +rest are still embedded in the epub but need manual CSS edits to use. + +The script tries to guess a shared family name from the common part of the +filenames (e.g. `NotoSerifJP-Regular.ttf` + `NotoSerifJP-Bold.ttf` -> +`NotoSerifJP`). To set it explicitly, use `config.json`: + +```json +{ "font_family": "MyBookFont" } +``` + +**Check the font's license** before embedding it in a file you intend to +share or publish — personal use on your own devices is usually fine, but +redistribution terms vary by font. + +## Running + +Simplest case — no arguments, if everything is where it's expected: + +```bash +python build_epub.py +``` + +Or with explicit paths/overrides: + +```bash +python build_epub.py \ + --pages-dir ./chapters \ + --font-dir ./font \ + --config ./config.json \ + --output ./my_book.epub \ + --show-chapter-titles \ + --vertical +``` + +`--split-pages` makes each file its own xhtml document inside the epub, +instead of merging every file in a folder into one. + +## `furigana.py` as a standalone preview tool + +Useful for quickly checking how a single OCR'd page will render before +running the full build: + +```bash +python furigana.py --input page.txt --output preview.xhtml +# or a quick one-liner check: +echo "本文《ほんぶん》デザイン" | python furigana.py --stdin +``` + +## Requirements + +```bash +pip install natsort +``` diff --git a/epub_builder/__init__.py b/epub_builder/__init__.py new file mode 100644 index 0000000..c0fdded --- /dev/null +++ b/epub_builder/__init__.py @@ -0,0 +1,13 @@ +""" +EPUB builder module: assembles a chapters/ folder (per-page .txt files from +the ocr module, plus optional illustrations) into a finished, valid .epub — +with furigana rendered as proper markup, an embedded font, a cover +image, and metadata pulled from config.json. + +Modules: + furigana.py Aozora Bunko furigana notation -> HTML markup, + plus the shared page-parsing/rendering logic. Also + runnable standalone as a single-page preview tool. + build_epub.py Assembles the whole book. Runnable directly: + python -m epub_builder.build_epub +""" diff --git a/epub_builder/build_epub.py b/epub_builder/build_epub.py new file mode 100644 index 0000000..15ee4da --- /dev/null +++ b/epub_builder/build_epub.py @@ -0,0 +1,665 @@ +#!/usr/bin/env python3 +""" +Builds an EPUB from a chapters/ folder laid out by hand into per-chapter +subfolders, embedding a font from font/ and metadata from config.json. + +Expected layout (next to this script): + + epub_builder/ + chapters/ + cover.jpg <- cover image (optional, any "cover.*" directly in chapters/) + ch00_frontmatter/ <- everything before the main text + 001.txt + 002.jpg + ch01_chapter00/ + 003.txt + 004.txt + 005.jpg + ch02_chapter01/ + 006.txt + ch08_backmatter/ + ... + font/ + YourFont.ttf <- font to embed (optional) + build_epub.py + furigana.py + config.json <- metadata and build settings + +Chapter folder naming: chNN_name + - chNN determines sort order. + - name is the default chapter title (can be overridden via + "chapter_titles" in config.json). + - Folders whose name (as a whole, or the part after chNN_) contains + "frontmatter" or "backmatter" are ALWAYS ignored: no title, no TOC + entry, no in-body heading — regardless of any flag or config + setting. Their content is still included in the book, just without + a section label. + +Page layout: + - The first line of each .txt page (after stripping a leading page + number) is rendered as a larger, bold "page header" — usually an + in-story date/time/chapter marker, not just page-numbering noise. + - If nothing is left after stripping the page number, two blank lines + are shown instead, so the page's vertical rhythm stays consistent. + - A blank line always follows the header, then the text. + - Each paragraph starts with a full-width space indent (standard in + Japanese typography), not a tab character that HTML/EPUB would + collapse anyway. + +Chapter titles are shown in the table of contents by default, but NOT as +an in-body

heading — pass --show-chapter-titles (or set +"show_chapter_titles": true in config.json) to also print them at the +start of each chapter. + +Usage: + python build_epub.py + # (all settings are read from config.json / chapters/ / font/ next to this script) + + # or with explicit paths/overrides: + python build_epub.py --pages-dir ./chapters --font-dir ./font \\ + --config ./config.json --output ./book.epub --show-chapter-titles + +Requirements: + pip install natsort + (furigana.py must live in the same folder as this script) +""" + +import argparse +import html +import json +import re +import sys +import uuid +import zipfile +from pathlib import Path + +from natsort import natsorted + +sys.path.insert(0, str(Path(__file__).resolve().parent)) +from furigana import parse_page, page_to_paragraphs_html # noqa: E402 + +SCRIPT_DIR = Path(__file__).resolve().parent + +TEXT_EXT = {".txt"} +IMAGE_EXT = {".jpg": "image/jpeg", ".jpeg": "image/jpeg", ".png": "image/png", + ".webp": "image/webp", ".gif": "image/gif"} +FONT_EXT = {".ttf": "font/ttf", ".otf": "font/otf", ".woff": "font/woff", ".woff2": "font/woff2"} + +CHAPTER_FOLDER_RE = re.compile(r"^ch(\d+)_(.+)$", re.IGNORECASE) + +CSS_BASE = """ +body {{ font-family: {font_stack}; line-height: 1.8; }} +body.vertical {{ writing-mode: vertical-rl; }} +p {{ margin: 0 0 1em 0; }} +p.page-header {{ font-size: 1.3em; font-weight: bold; margin: 1.5em 0 0 0; }} +p.spacer {{ margin: 0 0 1em 0; }} +rt {{ font-size: 0.5em; }} +h1 {{ text-align: center; margin: 2em 0; }} +div.illustration {{ text-align: center; margin: 1em 0; }} +div.illustration img {{ max-width: 100%; max-height: 100%; }} +div.cover {{ text-align: center; margin: 0; padding: 0; }} +div.cover img {{ max-width: 100%; height: auto; }} +{font_faces} +""" + +FONT_FACE_TEMPLATE = """@font-face {{ + font-family: "{family}"; + src: url("../fonts/{filename}") format("{fmt}"); + font-weight: {weight}; + font-style: {style}; +}} +""" + +CONTAINER_XML = """ + + + + + +""" + +CHAPTER_TEMPLATE = """ + + + + +{title} + + + +{h1} +{content} + + +""" + +COVER_TEMPLATE = """ + + + + +Cover + + + +
Cover
+ + +""" + + +# --------------------------------------------------------------------------- +# Page files: sorting, folder-name parsing +# --------------------------------------------------------------------------- + +def numeric_sort_key(path: Path): + """Sorts files strictly by the number found in the filename, ignoring the extension.""" + m = re.search(r"\d+", path.stem) + if m: + return (0, int(m.group()), path.name) + return (1, 0, path.name) + + +def sort_files(files: list[Path]) -> list[Path]: + return sorted(files, key=numeric_sort_key) + + +def parse_chapter_folder_name(folder_name: str) -> tuple[str | None, str]: + """ + Parses a folder name of the form "chNN_name" -> (NN, name). + Returns (None, folder_name) if it doesn't match the pattern. + """ + m = CHAPTER_FOLDER_RE.match(folder_name) + if m: + return m.group(1), m.group(2) + return None, folder_name + + +def is_ignored_section(folder_name: str) -> bool: + """frontmatter/backmatter sections are always ignored, regardless of any flag.""" + normalized = folder_name.lower().replace("_", "").replace("-", "") + return "frontmatter" in normalized or "backmatter" in normalized + + +def resolve_title(folder_name: str, chapter_titles_map: dict) -> str | None: + """ + Returns the display title for a chapter, or None if the section should + have no heading and no TOC entry (frontmatter/backmatter, or an + explicit empty string in chapter_titles_map). + """ + if is_ignored_section(folder_name): + return None + + if folder_name in chapter_titles_map: + value = chapter_titles_map[folder_name].strip() + return value if value else None + + _, remainder = parse_chapter_folder_name(folder_name) + default_title = remainder.replace("_", " ").strip() + return default_title if default_title else folder_name + + +# --------------------------------------------------------------------------- +# Processing the files in one chapter folder (text + images) +# --------------------------------------------------------------------------- + +def build_doc_from_folder(files: list[Path], doc_id: str): + """Returns (content_html, [(arcname, filepath, media_type), ...]).""" + content_parts = [] + images = [] + + for fpath in files: + suffix = fpath.suffix.lower() + if suffix in TEXT_EXT: + text = fpath.read_text(encoding="utf-8") + header, paragraphs = parse_page(text) + if not header and not paragraphs: + continue # [NO_TEXT] / empty page + content_parts.append(page_to_paragraphs_html(header, paragraphs)) + elif suffix in IMAGE_EXT: + arcname = f"images/{doc_id}_{fpath.stem}{suffix}" + content_parts.append(f'
') + images.append((arcname, fpath, IMAGE_EXT[suffix])) + else: + print(f"Skipping unknown file type: {fpath}", file=sys.stderr) + + return "\n".join(content_parts), images + + +def find_cover(pages_dir: Path, explicit_cover: str | None) -> Path | None: + if explicit_cover: + p = Path(explicit_cover) + return p if p.exists() else None + for ext in IMAGE_EXT: + candidates = list(pages_dir.glob(f"cover{ext}")) + list(pages_dir.glob(f"cover{ext.upper()}")) + if candidates: + return candidates[0] + return None + + +# --------------------------------------------------------------------------- +# Grouping pages into documents (chapters), with the split-pages option +# --------------------------------------------------------------------------- + +def group_pages_into_docs(pages_dir: Path, cover_path: Path | None, chapter_titles_map: dict, split_pages: bool): + subdirs = natsorted([d for d in pages_dir.iterdir() if d.is_dir()], key=lambda d: d.name) + root_files = [ + p for p in pages_dir.iterdir() + if p.is_file() and p.suffix.lower() in (TEXT_EXT | set(IMAGE_EXT)) and p != cover_path + ] + + if not subdirs and not root_files: + print(f"No subfolders or page files found in {pages_dir}.", file=sys.stderr) + sys.exit(1) + + docs = [] + all_images = [] + doc_counter = 0 + + def add_doc_merged(folder_name: str, files: list[Path]): + nonlocal doc_counter + if not files: + return + doc_counter += 1 + doc_id = f"doc{doc_counter:03d}_{re.sub(r'[^a-zA-Z0-9]+', '', folder_name) or 'section'}" + content, images = build_doc_from_folder(files, doc_id) + if not content: + doc_counter -= 1 + return + title = resolve_title(folder_name, chapter_titles_map) + docs.append({"id": doc_id, "title": title, "content": content}) + all_images.extend(images) + + def add_docs_split(folder_name: str, files: list[Path]): + nonlocal doc_counter + if not files: + return + folder_title = resolve_title(folder_name, chapter_titles_map) + safe_name = re.sub(r"[^a-zA-Z0-9]+", "", folder_name) or "section" + first_page_in_folder = True + for fpath in files: + doc_counter += 1 + doc_id = f"doc{doc_counter:03d}_{safe_name}_{fpath.stem}" + content, images = build_doc_from_folder([fpath], doc_id) + if not content: + doc_counter -= 1 + continue + title = folder_title if first_page_in_folder else None + first_page_in_folder = False + docs.append({"id": doc_id, "title": title, "content": content}) + all_images.extend(images) + + add_doc = add_docs_split if split_pages else add_doc_merged + + if root_files: + add_doc("root", sort_files(root_files)) + + for subdir in subdirs: + files = sort_files( + [p for p in subdir.iterdir() if p.is_file() and p.suffix.lower() in (TEXT_EXT | set(IMAGE_EXT))] + ) + add_doc(subdir.name, files) + + return docs, all_images + + +# --------------------------------------------------------------------------- +# Font handling +# --------------------------------------------------------------------------- + +def find_fonts(font_dir: Path) -> list[Path]: + if not font_dir.exists(): + return [] + fonts = [p for p in font_dir.iterdir() if p.is_file() and p.suffix.lower() in FONT_EXT] + return natsorted(fonts, key=lambda p: p.name) + + +def sanitize_family_name(stem: str) -> str: + return re.sub(r"[^a-zA-Z0-9 _-]+", "", stem).strip() or "EmbeddedFont" + + +STYLE_KEYWORDS_RE = re.compile(r"(?i)bold[\s_-]*italic|italic[\s_-]*bold|bold|italic|oblique|regular") + + +def detect_font_role(stem: str) -> str | None: + """ + Detects a font variant from its filename: Regular/Bold/Italic/BoldItalic + (case-insensitive, anywhere in the name). Returns None if it can't be + determined (non-standard filename). + """ + lower = stem.lower() + has_bold = "bold" in lower + has_italic = "italic" in lower or "oblique" in lower + if has_bold and has_italic: + return "bolditalic" + if has_bold: + return "bold" + if has_italic: + return "italic" + if "regular" in lower: + return "regular" + return None + + +ROLE_WEIGHT_STYLE = { + "regular": ("normal", "normal"), + "bold": ("bold", "normal"), + "italic": ("normal", "italic"), + "bolditalic": ("bold", "italic"), +} + + +def strip_style_keyword(stem: str) -> str: + """Removes style keywords from a font filename, returning the remainder (may be empty).""" + stripped = STYLE_KEYWORDS_RE.sub("", stem) + stripped = re.sub(r"[\s_-]+", " ", stripped).strip(" -_") + return stripped + + +def resolve_font_plan(fonts: list[Path], explicit_family: str | None): + """ + Decides how to wire up the fonts found in font/: + - If every file's variant is recognized (Regular/Bold/Italic/BoldItalic, + with at least one Regular among them), all of them get ONE shared + font-family name with the correct font-weight/font-style, so + ///bold headings automatically pick up the right file. + - Otherwise (non-standard filenames): the old behavior — each file gets + its own family name, and only the first one (alphabetically) gets + wired up automatically in the CSS. + + Returns a list of tuples (font_path, family, weight, style, is_primary). + """ + if not fonts: + return [] + + roles = {f: detect_font_role(f.stem) for f in fonts} + all_recognized = all(r is not None for r in roles.values()) + has_regular = any(r == "regular" for r in roles.values()) + + if all_recognized and has_regular: + if explicit_family: + family = explicit_family + else: + stripped_names = [strip_style_keyword(f.stem) for f in fonts] + non_empty = [n for n in stripped_names if n] + family = max(set(non_empty), key=non_empty.count) if non_empty else "MainFont" + family = sanitize_family_name(family) or "MainFont" + + plan = [] + for f in fonts: + weight, style = ROLE_WEIGHT_STYLE[roles[f]] + is_primary = roles[f] == "regular" + plan.append((f, family, weight, style, is_primary)) + return plan + + # Fallback: non-standard filenames — separate families + plan = [] + for i, f in enumerate(fonts): + family = explicit_family if (explicit_family and i == 0) else sanitize_family_name(f.stem) + plan.append((f, family, "normal", "normal", i == 0)) + return plan + + +# --------------------------------------------------------------------------- +# Assembling the final epub +# --------------------------------------------------------------------------- + +def build_epub(docs, all_images, cover_path, fonts: list[Path], output_path: Path, + meta: dict, vertical: bool, show_chapter_titles: bool): + book_id = meta.get("identifier") or f"urn:uuid:{uuid.uuid4()}" + body_class = ' class="vertical"' if vertical else "" + + manifest_items = [] + spine_items = [] + nav_items = [] + package_files = {} + + # --- Font --- + font_faces_css = "" + font_family_stack = "serif" + if fonts: + font_plan = resolve_font_plan(fonts, meta.get("font_family")) + font_face_rules = [] + primary_family = None + variant_labels = [] + + for font_path, family, weight, style, is_primary in font_plan: + fmt = FONT_EXT[font_path.suffix.lower()].split("/")[-1] + font_face_rules.append( + FONT_FACE_TEMPLATE.format(family=family, filename=font_path.name, fmt=fmt, weight=weight, style=style) + ) + arcname = f"fonts/{font_path.name}" + package_files[f"OEBPS/{arcname}"] = font_path.read_bytes() + font_id = "font_" + re.sub(r"[^a-zA-Z0-9]+", "_", font_path.name) + manifest_items.append(f'') + variant_labels.append(f"{font_path.name} ({weight}/{style})") + if is_primary and primary_family is None: + primary_family = family + + font_faces_css = "\n".join(font_face_rules) + font_family_stack = f'"{primary_family}", serif' + + unified = len(set(family for _, family, *_ in font_plan)) == 1 and len(font_plan) > 1 + if unified: + print(f'Font: {len(fonts)} variant(s) unified under the name "{primary_family}": ' + + ", ".join(variant_labels)) + elif len(fonts) > 1: + print(f"Found {len(fonts)} font file(s). Filenames don't look like Regular/Bold/Italic — " + f"using {fonts[0].name} as the primary one. The rest are embedded in the epub but " + f"not wired up in the CSS automatically — edit style.css by hand if needed.") + + css_content = CSS_BASE.format(font_stack=font_family_stack, font_faces=font_faces_css) + + # --- Cover --- + cover_meta = "" + guide_xml = "" + if cover_path: + cover_ext = cover_path.suffix.lower() + cover_media = IMAGE_EXT.get(cover_ext, "image/jpeg") + cover_img_arcname = f"images/cover{cover_ext}" + package_files[f"OEBPS/{cover_img_arcname}"] = cover_path.read_bytes() + manifest_items.append( + f'' + ) + package_files["OEBPS/text/cover.xhtml"] = COVER_TEMPLATE.format(img_href=cover_img_arcname) + manifest_items.append('') + spine_items.append('') + cover_meta = '' + guide_xml = '' + + # --- Images from chapters --- + for arcname, filepath, media_type in all_images: + package_files[f"OEBPS/{arcname}"] = filepath.read_bytes() + img_id = "img_" + re.sub(r"[^a-zA-Z0-9]+", "_", arcname) + manifest_items.append(f'') + + # --- Chapters --- + for doc in docs: + fname = f"text/{doc['id']}.xhtml" + display_title = doc["title"] + xhtml_title = display_title if display_title else " " + h1 = f"

{html.escape(display_title)}

" if (display_title and show_chapter_titles) else "" + package_files[f"OEBPS/{fname}"] = CHAPTER_TEMPLATE.format( + title=html.escape(xhtml_title), body_class=body_class, h1=h1, content=doc["content"], + ) + manifest_items.append(f'') + spine_items.append(f'') + if display_title: + nav_items.append(f'
  • {html.escape(display_title)}
  • ') + + manifest_extra = ( + '\n' + ' \n' + ' ' + ) + + # --- Additional dc: metadata --- + dc_extra = [] + if meta.get("publisher"): + dc_extra.append(f"{html.escape(meta['publisher'])}") + if meta.get("description"): + dc_extra.append(f"{html.escape(meta['description'])}") + if meta.get("date"): + dc_extra.append(f"{html.escape(meta['date'])}") + if meta.get("rights"): + dc_extra.append(f"{html.escape(meta['rights'])}") + if meta.get("series"): + dc_extra.append(f'') + if meta.get("series_index"): + dc_extra.append(f'') + + content_opf = f""" + + + {book_id} + {html.escape(meta['title'])} + {html.escape(meta['author'])} + {meta['language']} + {cover_meta} + {chr(10).join(' ' + line for line in dc_extra)} + + + {manifest_extra} + {chr(10).join(' ' + item for item in manifest_items)} + + + {chr(10).join(' ' + item for item in spine_items)} + + {guide_xml} + +""" + + nav_xhtml = f""" + + +Table of Contents + + + + +""" + + nav_points = [] + playorder = 0 + for doc in docs: + if not doc["title"]: + continue + playorder += 1 + nav_points.append(f""" + {html.escape(doc['title'])} + + """) + + toc_ncx = f""" + + + {html.escape(meta['title'])} + +{chr(10).join(nav_points)} + + +""" + + output_path.parent.mkdir(parents=True, exist_ok=True) + with zipfile.ZipFile(output_path, "w") as zf: + zf.writestr("mimetype", "application/epub+zip", compress_type=zipfile.ZIP_STORED) + zf.writestr("META-INF/container.xml", CONTAINER_XML) + zf.writestr("OEBPS/content.opf", content_opf) + zf.writestr("OEBPS/nav.xhtml", nav_xhtml) + zf.writestr("OEBPS/toc.ncx", toc_ncx) + zf.writestr("OEBPS/css/style.css", css_content) + for arcname, content in package_files.items(): + zf.writestr(arcname, content) + + print(f"EPUB built: {output_path}") + print(f"Total sections: {len(docs)}") + for doc in docs: + label = doc["title"] if doc["title"] else "(untitled, not in TOC)" + print(f" - {doc['id']}: {label}") + if cover_path: + print(f"Cover: {cover_path}") + if fonts: + print(f"Fonts embedded: {len(fonts)} (primary: {fonts[0].name})") + + +# --------------------------------------------------------------------------- +# Config and entry point +# --------------------------------------------------------------------------- + +def load_config(config_path: Path) -> dict: + if not config_path.exists(): + return {} + return json.loads(config_path.read_text(encoding="utf-8")) + + +def main(): + parser = argparse.ArgumentParser(description="Build an EPUB from a chapters/ folder, with metadata from config.json") + parser.add_argument("--pages-dir", default=str(SCRIPT_DIR / "chapters"), help="Folder with chapter subfolders") + parser.add_argument("--font-dir", default=str(SCRIPT_DIR / "font"), help="Folder with the font to embed") + parser.add_argument("--config", default=str(SCRIPT_DIR / "config.json"), help="Path to config.json") + parser.add_argument("--output", default=None, help="Path to the resulting .epub (defaults to config.json's value)") + parser.add_argument("--cover", default=None, help="Explicit cover image path (overrides auto-detection and config.json)") + parser.add_argument("--title", default=None, help="Override title from config.json") + parser.add_argument("--author", default=None, help="Override author from config.json") + parser.add_argument("--language", default=None, help="Override language from config.json") + parser.add_argument("--vertical", action="store_true", default=None, help="Vertical Japanese writing-mode") + parser.add_argument( + "--show-chapter-titles", action="store_true", default=None, + help="Print the chapter title (

    ) at the start of each chapter. By default it only appears in the TOC." + ) + parser.add_argument( + "--split-pages", action="store_true", + help="Make each file its own xhtml document, instead of merging all files in a folder into one." + ) + args = parser.parse_args() + + config = load_config(Path(args.config)) + + meta = { + "title": args.title or config.get("title"), + "author": args.author or config.get("author", "Unknown"), + "language": args.language or config.get("language", "ja"), + "identifier": config.get("identifier") or None, + "font_family": config.get("font_family") or None, + "publisher": config.get("publisher"), + "description": config.get("description"), + "date": config.get("date"), + "rights": config.get("rights"), + "series": config.get("series"), + "series_index": config.get("series_index"), + } + if not meta["title"]: + print("No title set — put one in config.json or pass --title.", file=sys.stderr) + sys.exit(1) + + vertical = args.vertical if args.vertical is not None else bool(config.get("vertical", False)) + show_chapter_titles = ( + args.show_chapter_titles if args.show_chapter_titles is not None + else bool(config.get("show_chapter_titles", False)) + ) + chapter_titles_map = config.get("chapter_titles", {}) + + output_path = Path(args.output) if args.output else Path(config.get("output", str(SCRIPT_DIR / "book.epub"))) + + pages_dir = Path(args.pages_dir) + font_dir = Path(args.font_dir) + + explicit_cover = args.cover or config.get("cover") + cover_path = find_cover(pages_dir, explicit_cover) + fonts = find_fonts(font_dir) + + docs, all_images = group_pages_into_docs(pages_dir, cover_path, chapter_titles_map, args.split_pages) + + build_epub( + docs, all_images, cover_path, fonts, output_path, + meta=meta, vertical=vertical, show_chapter_titles=show_chapter_titles, + ) + + +if __name__ == "__main__": + main() diff --git a/epub_builder/config.example.json b/epub_builder/config.example.json new file mode 100644 index 0000000..736eecf --- /dev/null +++ b/epub_builder/config.example.json @@ -0,0 +1,21 @@ +{ + "title": "Your Book Title", + "author": "Author Name", + "language": "ja", + "identifier": "", + "publisher": "", + "description": "", + "date": "", + "rights": "", + "series": "", + "series_index": "", + "font_family": "", + "vertical": false, + "show_chapter_titles": false, + "output": "book.epub", + "chapter_titles": { + "ch00_frontmatter": "", + "ch01_chapter00": "Chapter 00", + "ch08_backmatter": "" + } +} diff --git a/epub_builder/furigana.py b/epub_builder/furigana.py new file mode 100644 index 0000000..1054852 --- /dev/null +++ b/epub_builder/furigana.py @@ -0,0 +1,184 @@ +#!/usr/bin/env python3 +""" +Shared logic for turning OCR page text (see ocr/prompt.txt for the expected +format) into HTML paragraphs, including furigana rendered as proper +markup. Used by build_epub.py, and also runnable standalone as a quick +preview/debugging tool for a single page. + +Input text format (as produced by the OCR module): + - First line: page header/date/chapter-marker metadata. + - Every following line: one paragraph (no blank-line separators). + +Furigana notation on input (Aozora Bunko style): + - base《reading》 Furigana over `base`. `base` is the nearest + contiguous run of kanji immediately before《. + - |base《reading》 Explicit start of `base`, needed when it doesn't + match a plain "kanji run" (e.g. it's shorter, or + contains non-kanji characters, or is glued to the + previous word without a natural boundary). + - Several |base《reading》 in a row are kept as separate groups, + in reading order (e.g. a compound word whose furigana was printed + split across its parts). + +Standalone usage: + python furigana.py --input page.txt --output page.xhtml + # or a quick one-off check: + echo "本文《ほんぶん》デザイン" | python furigana.py --stdin +""" + +import argparse +import html +import re +import sys +from pathlib import Path + +# Kanji range + the iteration mark 々 + a couple of CJK compatibility ranges +KANJI_RUN = r"[\u4e00-\u9fff\u3005\u3007\uf900-\ufaff]+" + +# 1) Explicit base boundary via |: |《reading》 +RE_MARKED = re.compile(r"|([^|《]+?)《([^》]+)》") +# 2) No |: base is the nearest kanji run right before《 +RE_AUTO = re.compile(r"(" + KANJI_RUN + r")《([^》]+)》") + +# Strips a leading page number like "5 " or "23 " from the page header +LEADING_PAGE_NUMBER_RE = re.compile(r"^\d+\s*") + +# Full-width space — the standard paragraph indent in Japanese typography +PARAGRAPH_INDENT = "\u3000" + +XHTML_TEMPLATE = """ + + + + +{title} + + + +{body} + + +""" + +VERTICAL_STYLE = " body { writing-mode: vertical-rl; font-family: serif; }\n" +HORIZONTAL_STYLE = " body { font-family: serif; }\n" + + +def furigana_to_ruby(text: str) -> str: + """Replaces Aozora Bunko furigana notation with .......""" + def repl(m: re.Match) -> str: + base, reading = m.group(1), m.group(2) + return f"{base}{reading}" + + text = RE_MARKED.sub(repl, text) + text = RE_AUTO.sub(repl, text) + return text + + +def parse_page(text: str) -> tuple[str, list[str]]: + """Returns (header, [paragraphs]) from a page's raw OCR text. + + The literal marker "[NO_TEXT]" (written by the OCR module for pages + with no text at all, e.g. illustrations) parses as an empty page. + """ + if text.strip() == "[NO_TEXT]": + return "", [] + lines = [ln.strip() for ln in text.strip("\n").split("\n") if ln.strip() != ""] + if not lines: + return "", [] + return lines[0], lines[1:] + + +def strip_leading_page_number(header: str) -> str: + return LEADING_PAGE_NUMBER_RE.sub("", header, count=1) + + +def page_to_paragraphs_html(header: str, paragraphs: list[str]) -> str: + """ + Renders one page's (header, paragraphs) as HTML

    elements: + - The header (with its leading page number stripped) is shown in a + larger, bold "page-header" paragraph — in the source novels this + line is usually an in-story date/time/chapter marker, not just + page-numbering noise. + - If the header is empty after stripping the page number, an empty + placeholder line is rendered instead, so the vertical rhythm of the + page stays consistent whether or not there was a header. + - A blank spacer line always follows the header. + - Each paragraph is prefixed with a full-width space (the standard + Japanese paragraph indent) rather than relying on CSS text-indent, + so the indent survives even in readers that ignore that CSS rule. + """ + header = strip_leading_page_number(header).strip() + + header_html = furigana_to_ruby(html.escape(header, quote=False)) if header else " " + + out = [ + f'

    ', + '

     

    ', + ] + + for para in paragraphs: + indented = PARAGRAPH_INDENT + para + escaped = html.escape(indented, quote=False) + out.append(f"

    {furigana_to_ruby(escaped)}

    ") + + return "\n".join(out) + + +def convert_page(raw_text: str) -> str: + """Convenience wrapper: raw OCR page text -> HTML

    fragment (no / wrapper).""" + header, paragraphs = parse_page(raw_text) + if not header and not paragraphs: + return "" + return page_to_paragraphs_html(header, paragraphs) + + +def main(): + parser = argparse.ArgumentParser(description="Convert Aozora-style furigana notation to XHTML ") + parser.add_argument("--input", help="Path to a text file (one page's OCR output)") + parser.add_argument("--output", help="Where to write the XHTML (defaults to stdout)") + parser.add_argument("--stdin", action="store_true", help="Read text from stdin (for quick checks)") + parser.add_argument( + "--fragment", action="store_true", + help="Output only the

    fragment, without wrapping it in a full XHTML document. " + "Useful when the fragment will be inserted into an assembled chapter file elsewhere." + ) + parser.add_argument("--title", default="page", help="Document title (for --output, not --fragment)") + parser.add_argument( + "--vertical", action="store_true", + help="Vertical Japanese writing-mode (as in the original scan). " + "Defaults to horizontal (left-to-right) text, which renders more reliably " + "across readers and on mobile." + ) + args = parser.parse_args() + + if args.stdin: + raw_text = sys.stdin.read() + elif args.input: + raw_text = Path(args.input).read_text(encoding="utf-8") + else: + print("Provide --input or --stdin", file=sys.stderr) + sys.exit(1) + + body = convert_page(raw_text) + + if args.fragment: + result = body + else: + extra_style = VERTICAL_STYLE if args.vertical else HORIZONTAL_STYLE + result = XHTML_TEMPLATE.format(title=html.escape(args.title), body=body, extra_style=extra_style) + + if args.output: + Path(args.output).write_text(result, encoding="utf-8") + print(f"Saved: {args.output}") + else: + print(result) + + +if __name__ == "__main__": + main() diff --git a/ocr/README.md b/ocr/README.md new file mode 100644 index 0000000..ac5f71b --- /dev/null +++ b/ocr/README.md @@ -0,0 +1,67 @@ +# OCR module + +Batch-transcribes scanned light novel pages (vertical Japanese text) into +per-page `.txt` files. This is step 1 of the pipeline — step 2 is the +[`epub_builder`](../epub_builder/README.md) module, which turns these `.txt` +files into a finished `.epub`. + +## Choosing a backend + +| Script | Backend | Setup needed | Notes | +|---|---|---|---| +| `openrouter_ocr.py` | Multimodal LLM via any OpenAI-compatible API | API key + base URL | **Recommended.** Reads a whole page at once, no column segmentation needed. | +| `gemini_direct_ocr.py` | Gemini API directly | Google AI Studio API key | Same idea as above, direct instead of via a proxy. | +| `google_vision_ocr.py` | Google Cloud Vision (classic OCR) | Google Cloud project + billing enabled | No LLM context understanding, but solid on clean scans. Free tier covers a typical light novel volume, but Google still requires a billing account to be linked. | +| `local_mangaocr_ocr.py` | [manga-ocr](https://github.com/kha-white/manga-ocr), fully offline | None — no account, no API key | Slower to set up quality-wise: manga-ocr expects short text blocks, so this script auto-segments each page into vertical columns before OCR-ing each one. | + +If you have API access to a multimodal model (Gemini, GPT-4V-class models, +etc.) through any provider, `openrouter_ocr.py` is the easiest and generally +gives the best results with the least fuss. + +## Setup for `openrouter_ocr.py` (recommended) + +```bash +pip install openai pillow natsort tqdm +``` + +1. Copy `config.example.json` to `config.json` and fill in your `api_key`, + `base_url` (your provider's OpenAI-compatible endpoint, usually ending in + `/v1`), and `model` identifier. +2. Optionally edit `prompt.txt` — it's plain English text, no need to touch + any code to tweak the instructions given to the model. + +```bash +python openrouter_ocr.py --input ./pages --output ./out +``` + +- `--input`: folder with scanned page images (jpg/png/...), named so that + alphabetical sorting matches page order (natural sort is used, so + `page2.jpg` and `page10.jpg` sort correctly too). +- `--output`: folder for results. Creates `pages_txt/.txt` (one file + per page) plus a `combined.md` preview of the whole run. +- If interrupted, just re-run with the same `--output` — pages that already + have a `.txt` file are skipped, so nothing already done gets re-sent + (and re-billed). +- A page that fails to OCR (network error, rate limit, etc.) does **not** + get an empty file written for it, specifically so the next run retries it + instead of silently treating it as done. +- A page the model judges to be empty (illustration-only, blank, or a + cover/technical page) gets a file containing exactly the literal text + `[NO_TEXT]` — this is a deliberate marker, not an OCR failure. The + `epub_builder` module knows to treat it as "no text on this page". +- `--sleep N` adds a delay (seconds) between requests if you're hitting + rate limits. + +## Other backends + +`gemini_direct_ocr.py`, `google_vision_ocr.py`, and `local_mangaocr_ocr.py` +follow the same `--input`/`--output` convention and produce the same +`pages_txt/*.txt` + `combined.md` output — see each script's own docstring +for backend-specific setup. + +## Next step + +Once you have a `pages_txt/` folder full of `.txt` files, manually sort the +pages into the folder structure `epub_builder` expects (see +[`epub_builder/README.md`](../epub_builder/README.md)), then run the epub +builder. diff --git a/ocr/__init__.py b/ocr/__init__.py new file mode 100644 index 0000000..099cc92 --- /dev/null +++ b/ocr/__init__.py @@ -0,0 +1,15 @@ +""" +OCR module: batch-transcribes scanned light novel pages (vertical Japanese +text) into per-page .txt files, ready for the epub_builder module. + +Several interchangeable backends are provided as standalone scripts: + +- openrouter_ocr.py Recommended. Any OpenAI-compatible API (OpenRouter, + a self-hosted proxy, etc.) with a multimodal model. +- gemini_direct_ocr.py Same idea, but calling the Gemini API directly. +- google_vision_ocr.py Classic OCR via Google Cloud Vision (no LLM). +- local_mangaocr_ocr.py Fully offline, no cloud account, via manga-ocr. + +Each script is self-contained and runnable directly, e.g.: + python -m ocr.openrouter_ocr --input ./pages --output ./out +""" diff --git a/ocr/config.example.json b/ocr/config.example.json new file mode 100644 index 0000000..15eea5c --- /dev/null +++ b/ocr/config.example.json @@ -0,0 +1,5 @@ +{ + "api_key": "PUT_YOUR_API_KEY_HERE", + "base_url": "https://your-provider.example.com/v1", + "model": "google/gemini-3.7-flash" +} diff --git a/ocr/gemini_direct_ocr.py b/ocr/gemini_direct_ocr.py new file mode 100644 index 0000000..5292379 --- /dev/null +++ b/ocr/gemini_direct_ocr.py @@ -0,0 +1,146 @@ +#!/usr/bin/env python3 +""" +Batch-OCR for scanned pages of a Japanese light novel (vertical text) via the +Gemini API directly (not through a proxy). + +Usage: + export GEMINI_API_KEY="your_key" + python gemini_direct_ocr.py --input ./pages --output ./out + +Requirements: + pip install google-genai pillow natsort tqdm + +If you access Gemini (or another model) through OpenRouter or a similar +OpenAI-compatible proxy instead of a direct Google API key, use +openrouter_ocr.py instead — it's the recommended entry point for this +project and shares the same prompt.txt / config.json workflow. +""" + +import argparse +import os +import sys +import time +from pathlib import Path + +from google import genai +from google.genai import types +from natsort import natsorted +from PIL import Image +from tqdm import tqdm + +IMAGE_EXTS = {".jpg", ".jpeg", ".png", ".webp", ".tif", ".tiff", ".bmp"} + +PROMPT = """\ +This is a scanned page from a Japanese light novel, printed in vertical text (縦書き). +Transcribe ALL Japanese text on the page, preserving correct reading order +(columns read top-to-bottom, then right-to-left). + +Rules: +- Do not translate or summarize — output only the transcribed Japanese text. +- No line numbers, no commentary, no explanations of your own. +- Furigana may be omitted (only the base kanji/kana text is needed). +- If the page has no text at all (illustration-only, blank, cover/technical + page), output exactly: [NO_TEXT] +- Preserve paragraph breaks where the layout clearly shows them. +""" + + +def ocr_image(client: genai.Client, model: str, path: Path, retries: int = 3) -> str: + img = Image.open(path) + # Downscale very large scans — speeds up and cheapens the request without + # a noticeable loss of OCR quality. + max_dim = 2200 + if max(img.size) > max_dim: + ratio = max_dim / max(img.size) + img = img.resize((int(img.width * ratio), int(img.height * ratio))) + + last_err = None + for attempt in range(retries): + try: + response = client.models.generate_content( + model=model, + contents=[PROMPT, img], + config=types.GenerateContentConfig(temperature=0), + ) + text = (response.text or "").strip() + return text + except Exception as e: # noqa: BLE001 + last_err = e + time.sleep(3 * (attempt + 1)) + raise RuntimeError(f"Failed to OCR {path.name} after {retries} attempts: {last_err}") + + +def main(): + parser = argparse.ArgumentParser(description="Batch-OCR a light novel via the Gemini API") + parser.add_argument("--input", required=True, help="Folder with scanned page images (jpg/png)") + parser.add_argument("--output", required=True, help="Folder for the OCR results") + parser.add_argument("--model", default="gemini-3.7-flash", help="Gemini model name") + parser.add_argument( + "--api-key", default=None, + help="API key (falls back to the GEMINI_API_KEY environment variable)" + ) + parser.add_argument("--start-page", type=int, default=1) + parser.add_argument( + "--sleep", type=float, default=0.0, + help="Delay in seconds between requests (useful if you're hitting rate limits)" + ) + args = parser.parse_args() + + api_key = args.api_key or os.environ.get("GEMINI_API_KEY") + if not api_key: + print("No API key found. Pass --api-key or set GEMINI_API_KEY.", file=sys.stderr) + sys.exit(1) + + input_dir = Path(args.input) + output_dir = Path(args.output) + output_dir.mkdir(parents=True, exist_ok=True) + pages_dir = output_dir / "pages_txt" + pages_dir.mkdir(exist_ok=True) + + images = [p for p in input_dir.iterdir() if p.suffix.lower() in IMAGE_EXTS] + images = natsorted(images, key=lambda p: p.name) + + if not images: + print(f"No images found in {input_dir}.", file=sys.stderr) + sys.exit(1) + + print(f"Pages found: {len(images)}") + + client = genai.Client(api_key=api_key) + + combined_path = output_dir / "combined.md" + failed = [] + + with open(combined_path, "w", encoding="utf-8") as combined_f: + for idx, img_path in enumerate(tqdm(images, desc="OCR"), start=args.start_page): + txt_out = pages_dir / f"{img_path.stem}.txt" + + if txt_out.exists(): + text = txt_out.read_text(encoding="utf-8") + else: + try: + text = ocr_image(client, args.model, img_path) + txt_out.write_text(text, encoding="utf-8") + except Exception as e: # noqa: BLE001 + print(f"\nError on {img_path.name}: {e}", file=sys.stderr) + failed.append(img_path.name) + text = "" + # Do NOT write a file to disk on failure — see openrouter_ocr.py + # for the reasoning. + if args.sleep: + time.sleep(args.sleep) + + combined_f.write(f"\n\n\n\n") + combined_f.write(text) + + print(f"\nDone. Combined file: {combined_path}") + print(f"Per-page files: {pages_dir}") + if failed: + print(f"\nFailed to OCR {len(failed)} page(s):") + for name in failed: + print(f" - {name}") + print("Re-run the script with the same --output folder — already-done pages will not be redone.") + + +if __name__ == "__main__": + main() diff --git a/ocr/google_vision_ocr.py b/ocr/google_vision_ocr.py new file mode 100644 index 0000000..d34d9de --- /dev/null +++ b/ocr/google_vision_ocr.py @@ -0,0 +1,133 @@ +#!/usr/bin/env python3 +""" +Batch-OCR for scanned pages of a Japanese light novel (vertical text) via +Google Cloud Vision. + +Usage: + python google_vision_ocr.py --input ./pages --output ./out --credentials ./key.json + +Requirements: + pip install google-cloud-vision natsort tqdm + +Google Cloud setup (one-time, ~5-10 minutes): + 1. Go to https://console.cloud.google.com/ + 2. Create a project (or use an existing one) + 3. Search for "Vision API" -> Enable + 4. Go to "APIs & Services" -> "Credentials" -> "Create Credentials" -> "Service account" + 5. Create the service account (role can be left unset, or "Editor") + 6. Open the account -> Keys -> Add Key -> JSON -> downloads key.json + 7. Point --credentials at that file + +Note: as of writing, Google requires a billing account to be enabled on the +project before the Vision API will respond, even though usage stays within +the free tier (1000 requests/month covers ~600 pages comfortably). No charge +should occur unless you exceed that quota. + +This is a classic OCR engine (not an LLM) — generally solid for image +quality, but it doesn't understand context the way a multimodal model does. +For light novel pages with dense vertical prose, openrouter_ocr.py / +gemini_direct_ocr.py usually give better results with less setup friction. +""" + +import argparse +import io +import os +import sys +import time +from pathlib import Path + +from google.cloud import vision +from natsort import natsorted +from tqdm import tqdm + +IMAGE_EXTS = {".jpg", ".jpeg", ".png", ".webp", ".tif", ".tiff", ".bmp"} + + +def ocr_image(client: vision.ImageAnnotatorClient, path: Path, retries: int = 3) -> str: + """OCRs a single scan, returning text in reading order.""" + with io.open(path, "rb") as f: + content = f.read() + + image = vision.Image(content=content) + # The "ja" language hint helps the model handle vertical Japanese more accurately + image_context = vision.ImageContext(language_hints=["ja"]) + + last_err = None + for attempt in range(retries): + try: + response = client.document_text_detection( + image=image, image_context=image_context + ) + if response.error.message: + raise RuntimeError(response.error.message) + return response.full_text_annotation.text + except Exception as e: # noqa: BLE001 + last_err = e + time.sleep(2 * (attempt + 1)) + raise RuntimeError(f"Failed to OCR {path.name} after {retries} attempts: {last_err}") + + +def main(): + parser = argparse.ArgumentParser(description="Batch-OCR a light novel via Google Cloud Vision") + parser.add_argument("--input", required=True, help="Folder with scanned page images (jpg/png)") + parser.add_argument("--output", required=True, help="Folder for the OCR results") + parser.add_argument("--credentials", required=True, help="Path to the service-account key.json") + parser.add_argument( + "--start-page", type=int, default=1, help="Page number to start the header numbering from" + ) + args = parser.parse_args() + + os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = args.credentials + + input_dir = Path(args.input) + output_dir = Path(args.output) + output_dir.mkdir(parents=True, exist_ok=True) + pages_dir = output_dir / "pages_txt" + pages_dir.mkdir(exist_ok=True) + + images = [p for p in input_dir.iterdir() if p.suffix.lower() in IMAGE_EXTS] + images = natsorted(images, key=lambda p: p.name) + + if not images: + print(f"No images found in {input_dir}.", file=sys.stderr) + sys.exit(1) + + print(f"Pages found: {len(images)}") + + client = vision.ImageAnnotatorClient() + + combined_path = output_dir / "combined.md" + failed = [] + + with open(combined_path, "w", encoding="utf-8") as combined_f: + for idx, img_path in enumerate(tqdm(images, desc="OCR"), start=args.start_page): + txt_out = pages_dir / f"{img_path.stem}.txt" + + # Skip pages already OCR'd — handy if a previous run was interrupted + if txt_out.exists(): + text = txt_out.read_text(encoding="utf-8") + else: + try: + text = ocr_image(client, img_path) + txt_out.write_text(text, encoding="utf-8") + except Exception as e: # noqa: BLE001 + print(f"\nError on {img_path.name}: {e}", file=sys.stderr) + failed.append(img_path.name) + text = "" + # Do NOT write a file to disk on failure — otherwise the next + # run would see the file exists and skip retrying it. + + combined_f.write(f"\n\n\n\n") + combined_f.write(text) + + print(f"\nDone. Combined file: {combined_path}") + print(f"Per-page files: {pages_dir}") + if failed: + print(f"\nFailed to OCR {len(failed)} page(s):") + for name in failed: + print(f" - {name}") + print("Re-run the script with the same --output folder — already-done pages will not be redone.") + + +if __name__ == "__main__": + main() diff --git a/ocr/local_mangaocr_ocr.py b/ocr/local_mangaocr_ocr.py new file mode 100644 index 0000000..2f35d42 --- /dev/null +++ b/ocr/local_mangaocr_ocr.py @@ -0,0 +1,174 @@ +#!/usr/bin/env python3 +""" +Fully local, offline batch-OCR for scanned pages of a Japanese light novel +(vertical text) via manga-ocr — no cloud account or credit card required. + +Usage: + python local_mangaocr_ocr.py --input ./pages --output ./out + +Requirements: + pip install manga-ocr opencv-python pillow natsort tqdm + +The first run downloads manga-ocr's model weights (~400 MB) from +HuggingFace; after that everything runs offline. Without a GPU it runs on +CPU, just slower (roughly 1-3 sec per column). + +How it works: + 1. Each page is cut into vertical text columns (by detecting whitespace + gaps between columns — typical light novel layout). + 2. Columns are sorted right-to-left (the reading order for vertical + Japanese text). + 3. Each column is OCR'd separately via manga-ocr. + 4. Results are joined back into per-page text. + +manga-ocr was trained mainly on manga speech bubbles (short text blocks), +not dense full-page prose, so column segmentation matters a lot here for +quality. If segmentation performs poorly on your scans (e.g. unusual +layout), pass --whole-page to feed the model the full page without cutting +it into columns (simpler, but usually lower quality on dense prose). +""" + +import argparse +import sys +from pathlib import Path + +import cv2 +import numpy as np +from natsort import natsorted +from PIL import Image +from tqdm import tqdm + +IMAGE_EXTS = {".jpg", ".jpeg", ".png", ".webp", ".tif", ".tiff", ".bmp"} + + +def find_columns(img_gray: np.ndarray, min_col_width: int = 12, gap_threshold: int = 4): + """Finds x-ranges of vertical text columns via a pixel-density projection. + + Returns a list of (x_start, x_end), sorted RIGHT-TO-LEFT (the reading + order for vertical Japanese text). + """ + # Binarize: text (dark) -> white, background -> black + _, binary = cv2.threshold(img_gray, 0, 255, cv2.THRESH_BINARY_INV + cv2.THRESH_OTSU) + + # Dilate vertically a bit to merge characters within a column into one solid strip + kernel = np.ones((25, 1), np.uint8) + dilated = cv2.dilate(binary, kernel, iterations=1) + + col_sums = dilated.sum(axis=0) # text density per pixel column + has_text = col_sums > 0 + + columns = [] + x = 0 + width = len(has_text) + while x < width: + if has_text[x]: + start = x + while x < width and (has_text[x] or _gap_too_small(has_text, x, gap_threshold)): + x += 1 + end = x + if end - start >= min_col_width: + columns.append((start, end)) + else: + x += 1 + + columns.sort(key=lambda c: c[0], reverse=True) # right-to-left + return columns + + +def _gap_too_small(has_text: np.ndarray, x: int, gap_threshold: int) -> bool: + """Checks whether a text-free gap is shorter than gap_threshold (to avoid splitting a column needlessly).""" + if has_text[x]: + return False + end = x + while end < len(has_text) and not has_text[end]: + end += 1 + return (end - x) < gap_threshold + + +def ocr_page(mocr, pil_img: Image.Image, whole_page: bool) -> str: + if whole_page: + return mocr(pil_img) + + img_np = np.array(pil_img.convert("L")) + columns = find_columns(img_np) + + if not columns: + # No columns detected (e.g. an illustration-only page) — fall back to the whole page + return mocr(pil_img) + + texts = [] + for x_start, x_end in columns: + pad = 4 + crop = pil_img.crop((max(0, x_start - pad), 0, min(pil_img.width, x_end + pad), pil_img.height)) + text = mocr(crop) + if text.strip(): + texts.append(text.strip()) + + return "\n".join(texts) + + +def main(): + parser = argparse.ArgumentParser(description="Local batch-OCR for a light novel via manga-ocr") + parser.add_argument("--input", required=True, help="Folder with scanned page images (jpg/png)") + parser.add_argument("--output", required=True, help="Folder for the OCR results") + parser.add_argument( + "--whole-page", action="store_true", + help="Skip column segmentation, feed the whole page to the model at once" + ) + parser.add_argument("--start-page", type=int, default=1) + args = parser.parse_args() + + input_dir = Path(args.input) + output_dir = Path(args.output) + output_dir.mkdir(parents=True, exist_ok=True) + pages_dir = output_dir / "pages_txt" + pages_dir.mkdir(exist_ok=True) + + images = [p for p in input_dir.iterdir() if p.suffix.lower() in IMAGE_EXTS] + images = natsorted(images, key=lambda p: p.name) + + if not images: + print(f"No images found in {input_dir}.", file=sys.stderr) + sys.exit(1) + + print(f"Pages found: {len(images)}") + print("Loading the manga-ocr model (downloads weights on first run, ~400 MB)...") + + from manga_ocr import MangaOcr + mocr = MangaOcr() + + combined_path = output_dir / "combined.md" + failed = [] + + with open(combined_path, "w", encoding="utf-8") as combined_f: + for idx, img_path in enumerate(tqdm(images, desc="OCR"), start=args.start_page): + txt_out = pages_dir / f"{img_path.stem}.txt" + + if txt_out.exists(): + text = txt_out.read_text(encoding="utf-8") + else: + try: + pil_img = Image.open(img_path) + text = ocr_page(mocr, pil_img, args.whole_page) + txt_out.write_text(text, encoding="utf-8") + except Exception as e: # noqa: BLE001 + print(f"\nError on {img_path.name}: {e}", file=sys.stderr) + failed.append(img_path.name) + text = "" + # Do NOT write a file to disk on failure — otherwise the next + # run would see the file exists and skip retrying it. + + combined_f.write(f"\n\n\n\n") + combined_f.write(text) + + print(f"\nDone. Combined file: {combined_path}") + print(f"Per-page files: {pages_dir}") + if failed: + print(f"\nFailed to OCR {len(failed)} page(s):") + for name in failed: + print(f" - {name}") + print("Re-run the script with the same --output folder — already-done pages will not be redone.") + + +if __name__ == "__main__": + main() diff --git a/ocr/openrouter_ocr.py b/ocr/openrouter_ocr.py new file mode 100644 index 0000000..d58565a --- /dev/null +++ b/ocr/openrouter_ocr.py @@ -0,0 +1,191 @@ +#!/usr/bin/env python3 +""" +Batch-OCR for scanned pages of a Japanese light novel (vertical text) via any +OpenAI-compatible API (OpenRouter, a self-hosted proxy, etc.). + +This is the recommended OCR backend for this project: it lets a multimodal +LLM "read" a page image directly, including vertical Japanese text, without +any column-segmentation preprocessing. + +Usage (after filling in config.json): + python openrouter_ocr.py --input ./pages --output ./out + +Requirements: + pip install openai pillow natsort tqdm + +One-time setup: + 1. Copy config.example.json -> config.json + 2. Fill in your api_key, base_url and model + 3. Optionally edit prompt.txt to fit your book / house style +""" + +import argparse +import base64 +import io +import json +import os +import sys +import time +from pathlib import Path + +from natsort import natsorted +from openai import OpenAI +from PIL import Image +from tqdm import tqdm + +IMAGE_EXTS = {".jpg", ".jpeg", ".png", ".webp", ".tif", ".tiff", ".bmp"} +SCRIPT_DIR = Path(__file__).resolve().parent + + +def load_config(config_path: Path) -> dict: + if not config_path.exists(): + return {} + try: + return json.loads(config_path.read_text(encoding="utf-8")) + except json.JSONDecodeError as e: + print(f"Failed to parse {config_path}: {e}", file=sys.stderr) + sys.exit(1) + + +def load_prompt(prompt_path: Path) -> str: + if not prompt_path.exists(): + print(f"Prompt file not found: {prompt_path}", file=sys.stderr) + sys.exit(1) + return prompt_path.read_text(encoding="utf-8").strip() + + +def image_to_data_url(path: Path, max_dim: int = 2200) -> str: + """Downscale (if needed) and encode an image as a base64 data URL.""" + img = Image.open(path) + if img.mode not in ("RGB", "L"): + img = img.convert("RGB") + if max(img.size) > max_dim: + ratio = max_dim / max(img.size) + img = img.resize((int(img.width * ratio), int(img.height * ratio))) + + buf = io.BytesIO() + img.save(buf, format="JPEG", quality=90) + b64 = base64.b64encode(buf.getvalue()).decode("utf-8") + return f"data:image/jpeg;base64,{b64}" + + +def ocr_image(client: OpenAI, model: str, prompt: str, path: Path, retries: int = 3) -> str: + data_url = image_to_data_url(path) + + last_err = None + for attempt in range(retries): + try: + response = client.chat.completions.create( + model=model, + temperature=0, + messages=[ + { + "role": "user", + "content": [ + {"type": "text", "text": prompt}, + {"type": "image_url", "image_url": {"url": data_url}}, + ], + } + ], + ) + text = (response.choices[0].message.content or "").strip() + return text + except Exception as e: # noqa: BLE001 + last_err = e + time.sleep(3 * (attempt + 1)) + raise RuntimeError(f"Failed to OCR {path.name} after {retries} attempts: {last_err}") + + +def main(): + parser = argparse.ArgumentParser(description="Batch-OCR a light novel via an OpenAI-compatible API") + parser.add_argument("--input", required=True, help="Folder with scanned page images (jpg/png)") + parser.add_argument("--output", required=True, help="Folder for the OCR results") + parser.add_argument( + "--config", default=str(SCRIPT_DIR / "config.json"), + help="Path to config.json with api_key/base_url/model (defaults to config.json next to this script)" + ) + parser.add_argument( + "--prompt-file", default=str(SCRIPT_DIR / "prompt.txt"), + help="Path to the prompt file (defaults to prompt.txt next to this script)" + ) + parser.add_argument("--base-url", default=None, help="Override base_url from config.json") + parser.add_argument("--model", default=None, help="Override model from config.json") + parser.add_argument("--api-key", default=None, help="Override api_key from config.json") + parser.add_argument("--start-page", type=int, default=1) + parser.add_argument( + "--sleep", type=float, default=0.0, + help="Delay in seconds between requests (useful if you're hitting rate limits)" + ) + args = parser.parse_args() + + config = load_config(Path(args.config)) + + api_key = args.api_key or config.get("api_key") or os.environ.get("API_KEY") + base_url = args.base_url or config.get("base_url") + model = args.model or config.get("model") + + missing = [name for name, val in [("api_key", api_key), ("base_url", base_url), ("model", model)] if not val] + if missing: + print( + f"Missing settings: {', '.join(missing)}. " + f"Fill them in {args.config} (see config.example.json) or pass --api-key/--base-url/--model.", + file=sys.stderr, + ) + sys.exit(1) + + prompt = load_prompt(Path(args.prompt_file)) + + input_dir = Path(args.input) + output_dir = Path(args.output) + output_dir.mkdir(parents=True, exist_ok=True) + pages_dir = output_dir / "pages_txt" + pages_dir.mkdir(exist_ok=True) + + images = [p for p in input_dir.iterdir() if p.suffix.lower() in IMAGE_EXTS] + images = natsorted(images, key=lambda p: p.name) + + if not images: + print(f"No images found in {input_dir}.", file=sys.stderr) + sys.exit(1) + + print(f"Pages found: {len(images)}") + + client = OpenAI(base_url=base_url, api_key=api_key) + + combined_path = output_dir / "combined.md" + failed = [] + + with open(combined_path, "w", encoding="utf-8") as combined_f: + for idx, img_path in enumerate(tqdm(images, desc="OCR"), start=args.start_page): + txt_out = pages_dir / f"{img_path.stem}.txt" + + if txt_out.exists(): + text = txt_out.read_text(encoding="utf-8") + else: + try: + text = ocr_image(client, model, prompt, img_path) + txt_out.write_text(text, encoding="utf-8") + except Exception as e: # noqa: BLE001 + print(f"\nError on {img_path.name}: {e}", file=sys.stderr) + failed.append(img_path.name) + text = "" + # Do NOT write a file to disk on failure — otherwise the next + # run would see the file exists and assume the page is already + # done, silently skipping a retry forever. + if args.sleep: + time.sleep(args.sleep) + + combined_f.write(f"\n\n\n\n") + combined_f.write(text) + + print(f"\nDone. Combined file: {combined_path}") + print(f"Per-page files: {pages_dir}") + if failed: + print(f"\nFailed to OCR {len(failed)} page(s):") + for name in failed: + print(f" - {name}") + print("Re-run the script with the same --output folder — already-done pages will not be redone.") + + +if __name__ == "__main__": + main() diff --git a/ocr/prompt.txt b/ocr/prompt.txt new file mode 100644 index 0000000..4e7f1a5 --- /dev/null +++ b/ocr/prompt.txt @@ -0,0 +1,11 @@ +This is a scanned page from a Japanese light novel, printed in vertical text (縦書き). Transcribe ALL Japanese text on the page, preserving correct reading order (columns read top-to-bottom, then right-to-left). + +Rules: +- Mark furigana using Aozora Bunko notation: base_text《reading》, e.g. 本文《ほんぶん》. +- If the furigana applies to only part of a word, or the start of the kanji run is ambiguous, mark the exact start of the base text with a leading |, e.g. |中《ちゅう》. +- If a compound word has furigana split across its parts (as printed), keep them as separate 《》 groups in reading order, e.g. |事務《じむ》|所《しょ》. +- Include header/margin/footer text exactly as printed (page number, chapter title, in-story date/time stamp, file code). This novel often prints the same date/time/chapter info in two places on a page (e.g. top margin and a vertical strip on the side) — merge these into ONE single header line at the very start of your output, do not repeat it twice. +- Do not translate, summarize, or explain — output only the transcribed Japanese text. +- No line numbers, no commentary, no markdown formatting. +- Each paragraph goes on its own line. Do not merge multiple paragraphs into one line, and do not add blank lines between paragraphs. +- If the page has no text at all (illustration-only, blank, cover), output exactly: [NO_TEXT] diff --git a/requirements.txt b/requirements.txt new file mode 100644 index 0000000..d2867ed --- /dev/null +++ b/requirements.txt @@ -0,0 +1,23 @@ +# Core — always needed +natsort +Pillow +tqdm + +# --- OCR backends: install only the one(s) you actually use --- + +# ocr/openrouter_ocr.py (recommended) — any OpenAI-compatible API +openai + +# ocr/gemini_direct_ocr.py — direct Gemini API access +# google-genai + +# ocr/google_vision_ocr.py — Google Cloud Vision +# google-cloud-vision + +# ocr/local_mangaocr_ocr.py — fully offline, no cloud account +# manga-ocr +# opencv-python +# numpy + +# --- epub_builder module --- +# Only needs natsort, already listed above.