feat(ocr): unify config, add manga translation pipeline and context lookahead
- Consolidate module configs into root config.example.json with ocr, vision, and epub sections - Split LLM OCR workflows into novel_ocr.py (prose) and manga_ocr_llm.py (manga) - Remove gemini_direct_ocr.py in favor of OpenAI-compatible API endpoints - Support direct manga translation via --translate, --target-lang, and glossary.md - Add bidirectional context support: past translations (--context-pages) and lookahead Japanese text (--context-pages-ahead) - Add per-page JSON audit logging under logs/ and expose OpenAI sampling parameters
This commit is contained in:
Poison Flower
+32
-26
@@ -3,7 +3,7 @@
|
||||
Assembles a `chapters/` folder — laid out by hand into per-chapter
|
||||
subfolders — into a finished, valid `.epub`, with furigana rendered as
|
||||
proper `<ruby>` markup, an embedded font, a cover image, and metadata from
|
||||
`config.json`.
|
||||
the `epub` section of the repo's root `config.json`.
|
||||
|
||||
This is step 2 of the pipeline; step 1 is the [`ocr`](../ocr/README.md)
|
||||
module, which produces the per-page `.txt` files you'll sort into
|
||||
@@ -12,6 +12,7 @@ module, which produces the per-page `.txt` files you'll sort into
|
||||
## Layout
|
||||
|
||||
```
|
||||
config.json <- shared with the ocr module; see below for the "epub" section
|
||||
epub_builder/
|
||||
chapters/
|
||||
cover.jpg <- cover image (optional, any "cover.*" directly in chapters/)
|
||||
@@ -30,7 +31,6 @@ epub_builder/
|
||||
YourFont.ttf <- font to embed (optional)
|
||||
build_epub.py
|
||||
furigana.py
|
||||
config.json
|
||||
```
|
||||
|
||||
## Chapter folder naming: `chNN_name`
|
||||
@@ -38,8 +38,8 @@ epub_builder/
|
||||
- `chNN` — sequence number, determines sort order (`ch00` → `ch01` → ... →
|
||||
`ch10`; use leading zeros if you have more than 9 chapters so plain
|
||||
alphabetical sort stays correct).
|
||||
- `name` — default chapter title (override via `config.json` →
|
||||
`chapter_titles`).
|
||||
- `name` — default chapter title (override via `config.json`'s
|
||||
`epub.chapter_titles`).
|
||||
- Folders with `frontmatter` or `backmatter` anywhere in their name are
|
||||
**always** ignored: no title, no TOC entry — regardless of any flag or
|
||||
config setting. Their content is still included in the book, just
|
||||
@@ -64,26 +64,31 @@ Files inside a folder are sorted strictly by the number in the filename
|
||||
|
||||
## config.json
|
||||
|
||||
This module reads the `epub` section of the shared `config.json` at the
|
||||
repo root (copy `config.example.json` there if you haven't already):
|
||||
|
||||
```json
|
||||
{
|
||||
"title": "Your Book Title",
|
||||
"author": "Author Name",
|
||||
"language": "ja",
|
||||
"identifier": "",
|
||||
"publisher": "",
|
||||
"description": "",
|
||||
"date": "",
|
||||
"rights": "",
|
||||
"series": "",
|
||||
"series_index": "",
|
||||
"font_family": "",
|
||||
"vertical": false,
|
||||
"show_chapter_titles": false,
|
||||
"output": "book.epub",
|
||||
"chapter_titles": {
|
||||
"ch00_frontmatter": "",
|
||||
"ch01_chapter00": "Chapter 00",
|
||||
"ch08_backmatter": ""
|
||||
"epub": {
|
||||
"title": "Your Book Title",
|
||||
"author": "Author Name",
|
||||
"language": "ja",
|
||||
"identifier": "",
|
||||
"publisher": "",
|
||||
"description": "",
|
||||
"date": "",
|
||||
"rights": "",
|
||||
"series": "",
|
||||
"series_index": "",
|
||||
"font_family": "",
|
||||
"vertical": false,
|
||||
"show_chapter_titles": false,
|
||||
"output": "book.epub",
|
||||
"chapter_titles": {
|
||||
"ch00_frontmatter": "",
|
||||
"ch01_chapter00": "Chapter 00",
|
||||
"ch08_backmatter": ""
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
@@ -131,10 +136,10 @@ rest are still embedded in the epub but need manual CSS edits to use.
|
||||
|
||||
The script tries to guess a shared family name from the common part of the
|
||||
filenames (e.g. `NotoSerifJP-Regular.ttf` + `NotoSerifJP-Bold.ttf` ->
|
||||
`NotoSerifJP`). To set it explicitly, use `config.json`:
|
||||
`NotoSerifJP`). To set it explicitly, use `config.json`'s `epub` section:
|
||||
|
||||
```json
|
||||
{ "font_family": "MyBookFont" }
|
||||
{ "epub": { "font_family": "MyBookFont" } }
|
||||
```
|
||||
|
||||
**Check the font's license** before embedding it in a file you intend to
|
||||
@@ -143,7 +148,8 @@ redistribution terms vary by font.
|
||||
|
||||
## Running
|
||||
|
||||
Simplest case — no arguments, if everything is where it's expected:
|
||||
Simplest case — no arguments, if everything is where it's expected
|
||||
(`config.json` one level up, at the repo root):
|
||||
|
||||
```bash
|
||||
python build_epub.py
|
||||
@@ -155,7 +161,7 @@ Or with explicit paths/overrides:
|
||||
python build_epub.py \
|
||||
--pages-dir ./chapters \
|
||||
--font-dir ./font \
|
||||
--config ./config.json \
|
||||
--config ../config.json \
|
||||
--output ./my_book.epub \
|
||||
--show-chapter-titles \
|
||||
--vertical
|
||||
|
||||
@@ -23,7 +23,9 @@ Expected layout (next to this script):
|
||||
YourFont.ttf <- font to embed (optional)
|
||||
build_epub.py
|
||||
furigana.py
|
||||
config.json <- metadata and build settings
|
||||
|
||||
config.json (at the repo root, shared with the ocr module — has an
|
||||
"epub" section with metadata and build settings)
|
||||
|
||||
Chapter folder naming: chNN_name
|
||||
- chNN determines sort order.
|
||||
@@ -53,11 +55,12 @@ start of each chapter.
|
||||
|
||||
Usage:
|
||||
python build_epub.py
|
||||
# (all settings are read from config.json / chapters/ / font/ next to this script)
|
||||
# (chapters/ and font/ are read next to this script; config.json is
|
||||
# read from the repo root, one level up)
|
||||
|
||||
# or with explicit paths/overrides:
|
||||
python build_epub.py --pages-dir ./chapters --font-dir ./font \\
|
||||
--config ./config.json --output ./book.epub --show-chapter-titles
|
||||
--config ../config.json --output ./book.epub --show-chapter-titles
|
||||
|
||||
Requirements:
|
||||
pip install natsort
|
||||
@@ -79,6 +82,7 @@ sys.path.insert(0, str(Path(__file__).resolve().parent))
|
||||
from furigana import parse_page, page_to_paragraphs_html # noqa: E402
|
||||
|
||||
SCRIPT_DIR = Path(__file__).resolve().parent
|
||||
ROOT_DIR = SCRIPT_DIR.parent
|
||||
|
||||
TEXT_EXT = {".txt"}
|
||||
IMAGE_EXT = {".jpg": "image/jpeg", ".jpeg": "image/jpeg", ".png": "image/png",
|
||||
@@ -592,16 +596,25 @@ def build_epub(docs, all_images, cover_path, fonts: list[Path], output_path: Pat
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def load_config(config_path: Path) -> dict:
|
||||
"""Loads the shared config.json and returns its "epub" section.
|
||||
|
||||
Falls back to treating the whole file as the epub config if there's no
|
||||
"epub" key, so a bare {"title": ...} style file still works.
|
||||
"""
|
||||
if not config_path.exists():
|
||||
return {}
|
||||
return json.loads(config_path.read_text(encoding="utf-8"))
|
||||
data = json.loads(config_path.read_text(encoding="utf-8"))
|
||||
return data.get("epub", data) if isinstance(data, dict) else {}
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description="Build an EPUB from a chapters/ folder, with metadata from config.json")
|
||||
parser.add_argument("--pages-dir", default=str(SCRIPT_DIR / "chapters"), help="Folder with chapter subfolders")
|
||||
parser.add_argument("--font-dir", default=str(SCRIPT_DIR / "font"), help="Folder with the font to embed")
|
||||
parser.add_argument("--config", default=str(SCRIPT_DIR / "config.json"), help="Path to config.json")
|
||||
parser.add_argument(
|
||||
"--config", default=str(ROOT_DIR / "config.json"),
|
||||
help="Path to config.json with an \"epub\" section. Defaults to config.json at the repo root."
|
||||
)
|
||||
parser.add_argument("--output", default=None, help="Path to the resulting .epub (defaults to config.json's value)")
|
||||
parser.add_argument("--cover", default=None, help="Explicit cover image path (overrides auto-detection and config.json)")
|
||||
parser.add_argument("--title", default=None, help="Override title from config.json")
|
||||
|
||||
@@ -1,21 +0,0 @@
|
||||
{
|
||||
"title": "Your Book Title",
|
||||
"author": "Author Name",
|
||||
"language": "ja",
|
||||
"identifier": "",
|
||||
"publisher": "",
|
||||
"description": "",
|
||||
"date": "",
|
||||
"rights": "",
|
||||
"series": "",
|
||||
"series_index": "",
|
||||
"font_family": "",
|
||||
"vertical": false,
|
||||
"show_chapter_titles": false,
|
||||
"output": "book.epub",
|
||||
"chapter_titles": {
|
||||
"ch00_frontmatter": "",
|
||||
"ch01_chapter00": "Chapter 00",
|
||||
"ch08_backmatter": ""
|
||||
}
|
||||
}
|
||||
@@ -1,6 +1,6 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Shared logic for turning OCR page text (see ocr/prompt.txt for the expected
|
||||
Shared logic for turning OCR page text (see ocr/prompt_novel.txt for the expected
|
||||
format) into HTML paragraphs, including furigana rendered as proper <ruby>
|
||||
markup. Used by build_epub.py, and also runnable standalone as a quick
|
||||
preview/debugging tool for a single page.
|
||||
|
||||
Reference in New Issue
Block a user