update

2026-09-18 01:00:58 +03:00
10 changed files with 843 additions and 456 deletions
@@ -1,39 +0,0 @@
 # --- Python ---
 __pycache__/
 *.pyc
 *.pyo
 .venv/
 venv/
 *.egg-info/
 # --- Secrets / personal settings ---
 # The real config file contains API keys or book-specific metadata.
 # Only config.example.json is meant to be committed.
 /config.json
 # The real glossary is book-specific content (character names, terms).
 # Only glossary.example.md is meant to be committed.
 /ocr/glossary.md
 # --- Generated OCR output ---
 out/
 pages_txt/
 combined.md
 # --- Request/response logs (debugging only, can contain full prompts) ---
 /logs/
 # --- Book content: scans, sorted chapters, fonts, and the built epub ---
 # This repo is tooling only — never commit someone else's copyrighted
 # book pages, extracted text, embedded fonts, or the resulting epub.
 epub_builder/chapters/*
 !epub_builder/chapters/.gitkeep
 epub_builder/font/*
 !epub_builder/font/.gitkeep
 *.epub
 # --- OS / editor cruft ---
 .DS_Store
 Thumbs.db
 .idea/
 .vscode/
@@ -56,10 +56,11 @@ See each module's own README for setup and usage details.
 cp config.example.json config.json
 # fill in ocr.api_key / ocr.base_url / ocr.model, epub.title / epub.author / ...
-# 1a. OCR a light novel
+# 1a. OCR (or translate) a light novel
 cd ocr
 pip install openai pillow natsort tqdm
 python novel_ocr.py --input /path/to/scans --output ./out
 python novel_ocr.py --input /path/to/scans --output ./out --translate --target-lang Russian
 # 1b. ...or OCR/translate manga instead
 python manga_ocr_llm.py --input /path/to/scans --output ./out
@@ -12,7 +12,16 @@ and cross-page continuity context:
 | Pure-OCR alternative | `google_vision_ocr.py --mode novel` | `google_vision_ocr.py --mode manga`, `local_mangaocr_ocr.py` |
 | Layout assumed | Dense running prose, read top-to-bottom then right-to-left | Speech bubbles / narration boxes / SFX scattered across panels |
 | Output | Continuous transcribed text per page | Numbered list, one entry per bubble, in manga reading order |
-| Translation | — | `--translate`, with glossary + cross-page context |
+| Translation | `--translate`, with glossary + cross-page context | `--translate`, with glossary + cross-page context |
 `novel_ocr.py` and `manga_ocr_llm.py` share their translate/state/logging
 infrastructure via [`common.py`](common.py) — the only thing genuinely
 different between them is layout (reading order, output format), which is
 exactly why they're still two scripts rather than split along an
 OCR-vs-translate axis: translating manga still needs to know about panels
 and bubble types, and translating a novel would still want the same
 glossary/context machinery, so the layout split is what actually avoids
 duplicated logic.
 For a light novel, this is step 1 of the pipeline — step 2 is
 [`epub_builder`](../epub_builder/README.md), which turns the resulting
@@ -25,9 +34,12 @@ targets prose light novels and doesn't lay out manga pages.
 Reads each page with a multimodal LLM via any OpenAI-compatible API
 (OpenRouter, a direct provider endpoint, a self-hosted proxy, etc. —
 including Gemini, GPT-4V-class models, or anything else exposed through
-such an endpoint) and transcribes the vertical Japanese prose. Kept
+such an endpoint) and transcribes the vertical Japanese prose — or
-deliberately simple — transcription only, no translate/glossary/context —
+translates it directly with `--translate` (same glossary/context
-since that's genuinely all a novel page needs.
+machinery as `manga_ocr_llm.py`, via the shared [`common.py`](common.py)).
 Kept deliberately simpler than `manga_ocr_llm.py` on the transcription
 side — no bubble types/reading-order logic — since a novel page is one
 linear column of prose, not scattered panels.
 ```bash
 pip install openai pillow natsort tqdm
@@ -39,32 +51,57 @@ pip install openai pillow natsort tqdm
   identifier. This same file is shared with `manga_ocr_llm.py` and the
   `epub_builder` module. `temperature`/`max_tokens`/`top_p`/
   `reasoning_effort` are optional — see [API parameters](#api-parameters).
-2. Optionally edit `prompt_novel.txt` — plain English text, no need to
+2. Optionally edit `prompt_novel.txt` (transcription) or
-   touch any code to tweak the instructions given to the model.
+   `prompt_novel_translate.txt` (`--translate`) — plain English text, no
   need to touch any code to tweak the instructions given to the model.
 ```bash
 python novel_ocr.py --input ./pages --output ./out
 # ...or translate directly instead of transcribing:
 python novel_ocr.py --input ./pages --output ./out --translate --target-lang Russian
 ```
 - `--input`: folder with scanned page images (jpg/png/...), named so that
  alphabetical sorting matches page order (natural sort is used, so
  `page2.jpg` and `page10.jpg` sort correctly too).
- `--output`: folder for results. Creates `pages_txt/<name>.txt` (one file
+- `--output`: folder for results. Creates `pages_txt/<name>.txt` (one
-  per page) plus a `combined.md` preview of the whole run.
+  file per page), a `combined.md` preview of the whole run, and a
  `state.json` manifest (see below).
 - If interrupted, just re-run with the same `--output` — pages that already
  have a `.txt` file are skipped, so nothing already done gets re-sent
  (and re-billed).
- A page that fails to OCR (network error, rate limit, etc.) does **not**
+- A page that fails to OCR (network error, rate limit, truncated/empty
-  get an empty file written for it, specifically so the next run retries it
+  response, etc.) does **not** get a file written for it, specifically so
-  instead of silently treating it as done.
+  the next run retries it instead of silently treating it as done.
 - A page the model judges to be empty (illustration-only, blank, or a
  cover/technical page) gets a file containing exactly the literal text
  `[NO_TEXT]` — this is a deliberate marker, not an OCR failure. The
  `epub_builder` module knows to treat it as "no text on this page".
 - `--pages "3,7-12,20"` limits the run to just those pages (1-based
  positions in the sorted page list) — combine with `--redo` to re-OCR/
  re-translate just a few pages, e.g. after tweaking the prompt or
  glossary, without redoing the whole book.
 - `--redo` re-runs pages even if a `.txt` already exists for them.
 - `--start-page N` skips everything before page N entirely (not just a
  relabelling — pages before it are never sent).
 - `--no-state` ignores `state.json` and falls back to plain "file exists =
  done" behaviour.
 - `--sleep N` adds a delay (seconds) between requests if you're hitting
  rate limits.
 - `--config` overrides the config.json path (defaults to the repo root).
- `--prompt-file` overrides the prompt file (defaults to `prompt_novel.txt`).
+- `--prompt-file` overrides the prompt file (defaults to `prompt_novel.txt`,
  or `prompt_novel_translate.txt` with `--translate`).
 ### `state.json`
 Both `novel_ocr.py` and `manga_ocr_llm.py` write a `state.json` manifest to
 `--output`, recording (per page) the model, a hash of the exact prompt
 used, token usage, and a timestamp. This is what makes `--redo`/`--pages`
 targeted re-runs possible, and it's also how the script notices (and warns
 about, non-fatally) pages that were transcribed/translated under a prompt
 or glossary that's since changed — it won't re-OCR them on its own, but
 tells you to consider `--redo` for them.
 ## `manga_ocr_llm.py` (recommended for manga)
@@ -100,7 +137,8 @@ Output format is a numbered list per page:
 where entries are ordered manga-style (panels right-to-left top-to-bottom;
 bubbles within a panel the same way), and TYPE is one of DIALOGUE,
-THOUGHT, NARRATION, or SFX. Same `--input`/`--output`/`--sleep`/`--config`/
+THOUGHT, NARRATION, or SFX. Same `--input`/`--output`/`--pages`/`--redo`/
 `--start-page`/`--no-state`/`--sleep`/`--config`/
 `--prompt-file` conventions as `novel_ocr.py`.
 ### Translation (`--translate`)
@@ -114,16 +152,24 @@ THOUGHT, NARRATION, or SFX. Same `--input`/`--output`/`--sleep`/`--config`/
 - Use a different `--output` folder than your transcription run — both
  write to `pages_txt/<name>.txt`, and the script skips a page whose
  output file already exists, so reusing the same folder would just
-  return the old transcription instead of translating.
+  return the old transcription instead of translating. (`--redo` re-runs
  it in place if you'd rather overwrite than use a separate folder.)
 - Everything under Glossary/Continuity context/Lookahead context below
  applies identically to `novel_ocr.py --translate` — the `--glossary`/
  `--context-pages`/`--target-lang` flags are the exact same flags,
  implemented once in [`common.py`](common.py). `--context-pages-ahead`
  is manga-only for now (see that section for why light novel prose
  doesn't need it as much).
 ### Glossary
 For a consistent translation of recurring names and terms, copy
 [`glossary.example.md`](glossary.example.md) to `glossary.md` right here in
 the `ocr/` folder (next to the scripts, not with your page images — a
-single stable file you keep building up across books/chapters, which also
+single stable file you keep building up across books/chapters, shared by
-makes it a natural fit for an editing UI later). It's auto-detected and
+both `novel_ocr.py` and `manga_ocr_llm.py`, and a natural fit for an
-sent along with every `--translate` request, e.g.:
+editing UI later). It's auto-detected and sent along with every
 `--translate` request, e.g.:
 ```markdown
 ## Names
@@ -257,6 +303,19 @@ with the OpenAI-compatible API the other scripts use):
 { "vision": { "credentials": "/path/to/service-account-key.json" } }
 ```
 ## System/user prompt split (caching)
 For `novel_ocr.py` and `manga_ocr_llm.py`, every request is sent as a
 `system` message (the format instructions, and, with `--translate`, the
 glossary — identical on every page of a run) plus a `user` message (the
 page image, and, with `--context-pages`/`--context-pages-ahead`, that
 page's sliding-window context). Most OpenAI-compatible providers cache a
 repeated `system` prefix, so a 300-page book isn't re-billed/re-processed
 for the same instructions on every single page — only the truly per-page
 part (image + context) is new each time. This is implemented once in
 `common.py`'s `ocr_image()`/`build_system_prompt()`/`build_context_block()`
 and used identically by both scripts.
 ## Request/response logs
 `novel_ocr.py`, `manga_ocr_llm.py`, and `google_vision_ocr.py` each write
@@ -1,27 +1,36 @@
 """
 OCR module: batch-transcribes (or translates) scanned pages into per-page
 .txt files. Light novel and manga pages are handled by separate scripts,
-since their layouts need genuinely different logic (dense running prose
+since their LAYOUT needs genuinely different logic (dense running prose
-vs. scattered, typed speech bubbles that benefit from translation +
+vs. scattered, typed speech bubbles) — but both support --translate with
-glossary + cross-page continuity context).
+a glossary and cross-page continuity context, sharing that (and all the
 config/state/logging plumbing) via common.py rather than duplicating it.
 Novel (light novel, dense running prose):
 - novel_ocr.py          Any OpenAI-compatible API (OpenRouter, a direct
                          provider endpoint, a self-hosted proxy, etc.)
-                          with a multimodal model. Recommended.
+                          with a multimodal model. Transcribe or
                          --translate. Recommended.
 - google_vision_ocr.py  Classic OCR via Google Cloud Vision (no LLM,
                          --mode novel).
 Manga (speech bubbles / narration boxes / SFX):
 - manga_ocr_llm.py      Any OpenAI-compatible API. Transcribe or
-                          --translate, with --glossary and --context-pages
+                          --translate, with --glossary, --context-pages,
-                          for cross-page continuity. Recommended.
+                          and --context-pages-ahead for cross-page
                          continuity. Recommended.
 - google_vision_ocr.py  Classic OCR via Google Cloud Vision (no LLM,
                          --mode manga; transcription only, no translation).
 - local_mangaocr_ocr.py Fully offline, no cloud account, via manga-ocr
                          (also supports --mode novel via column
                          segmentation, transcription only).
 common.py: shared infrastructure used by novel_ocr.py and manga_ocr_llm.py
    — config/state/logging plumbing, the OpenAI-compatible request helper
    (with a system/user prompt split for provider-side caching), and the
    --translate helpers (glossary, sliding-window context). Not meant to
    be run directly.
 Each script is self-contained and runnable directly, e.g.:
    python -m ocr.novel_ocr --input ./pages --output ./out
    python -m ocr.manga_ocr_llm --input ./pages --output ./out --translate
@@ -0,0 +1,403 @@
 """
 Shared infrastructure for the OCR scripts (novel_ocr.py, manga_ocr_llm.py).
 Everything here is genuinely identical between the two "page layout"
 scripts — config/state/logging plumbing, the OpenAI-compatible request
 helper, and the translate-only helpers (glossary, sliding-window context).
 What's deliberately NOT here is anything that depends on page layout
 (reading order, output format, the prompt files themselves) — that stays
 in each script, since novel and manga pages need genuinely different
 logic there.
 system/user prompt split (for provider-side prompt caching):
    Everything that stays IDENTICAL across every page of a run — the
    format/transcription instructions, and (for --translate) the glossary
    — is sent as the `system` message. Everything that changes per page —
    the image, and any sliding-window context — goes in `user`. Most
    OpenAI-compatible providers cache a repeated system prefix, so with a
    long book this avoids re-billing/re-processing the same instructions
    on every single page. Per-page context (previous/upcoming pages) is
    NOT cached-friendly by nature (it changes every page), so it belongs
    in `user`, never appended to the cacheable `system` text.
 """
 import base64
 import io
 import json
 import re
 import sys
 import time
 from datetime import datetime
 from pathlib import Path
 from openai import OpenAI
 from PIL import Image
 IMAGE_EXTS = {".jpg", ".jpeg", ".png", ".webp", ".tif", ".tiff", ".bmp"}
 STATE_FILE = "state.json"
 # ---------------------------------------------------------------------------
 # config.json
 # ---------------------------------------------------------------------------
 def load_config_section(config_path: Path, section: str) -> dict:
    """Loads the shared config.json and returns one top-level section
    (e.g. "ocr"). Falls back to treating the whole file as that section if
    the key is missing, so a bare {"api_key": ...} style file still works.
    """
    if not config_path.exists():
        return {}
    try:
        data = json.loads(config_path.read_text(encoding="utf-8"))
    except json.JSONDecodeError as e:
        print(f"Failed to parse {config_path}: {e}", file=sys.stderr)
        sys.exit(1)
    return data.get(section, data) if isinstance(data, dict) else {}
 def load_prompt_file(prompt_path: Path) -> str:
    if not prompt_path.exists():
        print(f"Prompt file not found: {prompt_path}", file=sys.stderr)
        sys.exit(1)
    return prompt_path.read_text(encoding="utf-8").strip()
 # ---------------------------------------------------------------------------
 # Page selection (--pages "3,7-12,20", --start-page)
 # ---------------------------------------------------------------------------
 def parse_page_selection(spec: str, images: list) -> set | None:
    """Parses a --pages selection like "3,7-12,20" into a set of page stems.
    Numbers refer to the 1-based position in the naturally-sorted page list,
    so "3" is the third page, not a file literally named "3". Returns None
    when the spec is empty/absent, meaning "no explicit selection".
    """
    if not spec:
        return None
    selected = set()
    for part in spec.split(","):
        part = part.strip()
        if not part:
            continue
        m = re.fullmatch(r"(\d+)\s*(?:-\s*(\d+))?", part)
        if not m:
            raise ValueError(f"Invalid --pages entry: {part!r} (expected N or N-M)")
        start = int(m.group(1))
        end = int(m.group(2)) if m.group(2) else start
        if start < 1 or end < start:
            raise ValueError(f"Invalid --pages range: {part!r}")
        for n in range(start, end + 1):
            if n > len(images):
                raise ValueError(f"--pages refers to page {n}, but only {len(images)} page(s) were found")
            selected.add(images[n - 1].stem)
    return selected
 # ---------------------------------------------------------------------------
 # state.json manifest (resume/redo tracking, prompt-change detection)
 # ---------------------------------------------------------------------------
 def prompt_fingerprint(prompt: str) -> str:
    """Short stable hash of the prompt text, used to detect prompt edits."""
    import hashlib
    return hashlib.sha256(prompt.encode("utf-8")).hexdigest()[:12]
 def load_state(output_dir: Path) -> dict:
    """Loads the pages manifest, mapping page stem -> what produced it.
    Returns {} if the manifest is missing or unreadable; callers must treat
    a page with no entry as "unverified" rather than "stale".
    """
    path = output_dir / STATE_FILE
    if not path.exists():
        return {}
    try:
        data = json.loads(path.read_text(encoding="utf-8"))
    except (json.JSONDecodeError, OSError) as e:  # noqa: BLE001
        print(f"Warning: could not read {path} ({e}); existing pages are reused unverified.", file=sys.stderr)
        return {}
    return data.get("pages", {}) if isinstance(data, dict) else {}
 def save_state(output_dir: Path, pages: dict, meta: dict) -> None:
    """Atomically rewrites the pages manifest (temp file + replace)."""
    path = output_dir / STATE_FILE
    tmp = path.with_suffix(".json.tmp")
    tmp.write_text(
        json.dumps({"meta": meta, "pages": pages}, ensure_ascii=False, indent=2, default=str),
        encoding="utf-8",
    )
    tmp.replace(path)
 def warn_if_prompt_changed(state: dict, images: list, prompt_hash: str, prompt_file_name: str) -> None:
    """Warns (does not fail) if some already-transcribed pages were done
    with a prompt whose hash no longer matches the current one."""
    if not state:
        return
    stale = [
        p.stem for p in images
        if p.stem in state and state[p.stem].get("prompt_hash") not in (None, prompt_hash)
    ]
    if stale:
        print(
            f"Note: {len(stale)} page(s) were transcribed with a different prompt "
            f"({prompt_file_name} changed). Use --redo to re-OCR them.",
            file=sys.stderr,
        )
 # ---------------------------------------------------------------------------
 # Request/response logging
 # ---------------------------------------------------------------------------
 def write_log(log_dir: Path, page_name: str, entry: dict) -> None:
    """Writes one JSON log file per request/response.
    The base64 image payload itself is never included (megabytes of no
    debugging value, one per page) — only its path/size — but everything
    else (full prompt text, model params, every retry attempt, the full raw
    API response) is recorded as-is.
    """
    log_dir.mkdir(parents=True, exist_ok=True)
    timestamp = datetime.now().strftime("%Y%m%d_%H%M%S_%f")
    safe_name = re.sub(r"[^A-Za-z0-9_.-]", "_", page_name)
    log_path = log_dir / f"{timestamp}_{safe_name}.json"
    log_path.write_text(json.dumps(entry, ensure_ascii=False, indent=2, default=str), encoding="utf-8")
 # ---------------------------------------------------------------------------
 # Request params (CLI flag > config.json > default)
 # ---------------------------------------------------------------------------
 def build_request_kwargs(config: dict, args) -> tuple[dict, dict]:
    """Resolves OpenAI-API-style request params: CLI flag > config.json > a
    sensible default. Returns (kwargs, extra_body) — standard params go
    straight into the request; reasoning_effort goes through extra_body
    since support for it varies by provider/model and extra_body is the
    designed passthrough for exactly that.
    """
    temperature = args.temperature if args.temperature is not None else config.get("temperature", 0)
    max_tokens = args.max_tokens if args.max_tokens is not None else config.get("max_tokens")
    top_p = args.top_p if args.top_p is not None else config.get("top_p")
    reasoning_effort = args.reasoning_effort or config.get("reasoning_effort")
    kwargs = {"temperature": temperature}
    if max_tokens is not None:
        kwargs["max_tokens"] = max_tokens
    if top_p is not None:
        kwargs["top_p"] = top_p
    extra_body = {}
    if reasoning_effort:
        extra_body["reasoning_effort"] = reasoning_effort
    return kwargs, extra_body
 # ---------------------------------------------------------------------------
 # Image encoding
 # ---------------------------------------------------------------------------
 def image_to_data_url(path: Path, max_dim: int = 2200) -> str:
    """Downscale (if needed) and encode an image as a base64 data URL."""
    img = Image.open(path)
    if img.mode not in ("RGB", "L"):
        img = img.convert("RGB")
    if max(img.size) > max_dim:
        ratio = max_dim / max(img.size)
        img = img.resize((int(img.width * ratio), int(img.height * ratio)))
    buf = io.BytesIO()
    img.save(buf, format="JPEG", quality=90)
    b64 = base64.b64encode(buf.getvalue()).decode("utf-8")
    return f"data:image/jpeg;base64,{b64}"
 # ---------------------------------------------------------------------------
 # LLM OCR call, with system/user split for prompt caching
 # ---------------------------------------------------------------------------
 def usage_dict(response) -> dict | None:
    """Extracts token usage from a response, tolerating providers that omit it."""
    usage = getattr(response, "usage", None)
    if usage is None:
        return None
    try:
        data = usage.model_dump() if hasattr(usage, "model_dump") else dict(usage)
    except Exception:  # noqa: BLE001
        return None
    return {k: v for k, v in data.items() if v is not None} or None
 def ocr_image(
    client: OpenAI, model: str, system_prompt: str, path: Path,
    request_kwargs: dict, extra_body: dict,
    user_text: str = None, retries: int = 3, log_entry: dict | None = None,
 ) -> tuple[str, dict]:
    """OCRs (or translates) one page image.
    system_prompt: the part of the prompt that's IDENTICAL across every
        page of this run (format instructions, glossary) — sent as the
        `system` message so OpenAI-compatible providers can cache it
        instead of reprocessing it on every page.
    user_text: optional per-page text that varies from page to page
        (sliding-window context) — sent alongside the image in `user`.
        Never put per-page content in system_prompt, or every page
        invalidates the cache instead of hitting it.
    Returns (text, meta) where meta records what the provider reported
    about the accepted response (finish_reason, usage) so the caller can
    persist it (e.g. in the pages manifest).
    """
    data_url = image_to_data_url(path)
    user_content = [{"type": "image_url", "image_url": {"url": data_url}}]
    if user_text:
        user_content.insert(0, {"type": "text", "text": user_text})
    messages = [
        {"role": "system", "content": system_prompt},
        {"role": "user", "content": user_content},
    ]
    if log_entry is not None:
        logged_user_content = [
            {"type": "text", "text": user_text} if user_text else None,
            {"type": "image_url", "image_url": {"url": f"<omitted: {path.name}, {path.stat().st_size} bytes>"}},
        ]
        log_entry["request"] = {
            "model": model,
            **request_kwargs,
            "extra_body": extra_body or None,
            "messages": [
                {"role": "system", "content": system_prompt},
                {"role": "user", "content": [c for c in logged_user_content if c]},
            ],
        }
        log_entry["attempts"] = []
    last_err = None
    for attempt in range(retries):
        try:
            response = client.chat.completions.create(
                model=model,
                messages=messages,
                extra_body=extra_body or None,
                **request_kwargs,
            )
            choice = response.choices[0]
            text = (choice.message.content or "").strip()
            finish_reason = getattr(choice, "finish_reason", None)
            # A truncated or empty response must not be accepted as a
            # successful transcription — this file would otherwise be
            # written to disk and silently skipped on every future retry.
            if finish_reason == "length":
                raise RuntimeError(f"response truncated (finish_reason=length, {len(text)} chars)")
            if not text:
                raise RuntimeError(f"empty response (finish_reason={finish_reason})")
            if log_entry is not None:
                try:
                    raw_response = response.model_dump()
                except Exception:  # noqa: BLE001
                    raw_response = {"content": text}
                log_entry["attempts"].append({"attempt": attempt + 1, "success": True, "response": raw_response})
            return text, {"finish_reason": finish_reason, "usage": usage_dict(response)}
        except Exception as e:  # noqa: BLE001
            last_err = e
            if log_entry is not None:
                log_entry["attempts"].append({"attempt": attempt + 1, "success": False, "error": str(e)})
            time.sleep(3 * (attempt + 1))
    raise RuntimeError(f"Failed to OCR {path.name} after {retries} attempts: {last_err}")
 # ---------------------------------------------------------------------------
 # Translation helpers: glossary + sliding-window context (--translate only)
 # ---------------------------------------------------------------------------
 def find_glossary(script_dir: Path, explicit_path: str) -> Path | None:
    """Resolves the glossary path: explicit --glossary if given, otherwise
    an auto-detected glossary.md next to the ocr scripts themselves (a
    stable, book-independent location — handy for a future UI to edit)."""
    if explicit_path:
        path = Path(explicit_path)
        if not path.exists():
            print(f"Glossary file not found: {path}", file=sys.stderr)
            sys.exit(1)
        return path
    auto_path = script_dir / "glossary.md"
    return auto_path if auto_path.exists() else None
 def build_system_prompt(base_prompt: str, target_language: str = None, glossary: str = None) -> str:
    """Builds the cacheable `system` prompt: format instructions, with the
    target language substituted in and the glossary appended.
    Deliberately excludes per-page context (see build_context_block) —
    that changes every page and must go in `user`, or every page would
    invalidate the provider's cached system prefix instead of hitting it.
    """
    prompt = base_prompt
    if target_language:
        prompt = prompt.replace("{target_language}", target_language)
    if glossary:
        prompt += (
            "\n\n---\n\n"
            "Glossary: use these established translations for the following "
            "names/terms for consistency. Anything not listed here, translate "
            "naturally.\n\n" + glossary
        )
    return prompt
 def build_context_block(past_pages: list, ahead_pages: list) -> str:
    """Builds the per-page `user` text carrying sliding-window context:
    - past_pages: the last N *translated* pages, for continuity (character
      voice, an ongoing exchange, pronouns/referents that only make sense
      given what was just said).
    - ahead_pages: the next N pages' raw *Japanese* transcription (never a
      draft translation — see the ocr/README.md design note on why), for
      context that only becomes clear from what's about to happen (a
      twist, who's actually being addressed, a joke that pays off a page
      later).
    Both are lists of (page_name, text) tuples, oldest/nearest first.
    Returns "" if there's nothing to add.
    This text is per-page by nature and must never be folded into the
    cacheable system prompt (see build_system_prompt) — it changes on
    every single page, so appending it to system would defeat prompt
    caching entirely instead of the intended "cache the instructions,
    vary only the image/context" split.
    """
    if not past_pages and not ahead_pages:
        return ""
    parts = []
    if past_pages:
        parts.append(
            "Context: translations of the immediately preceding page(s), for "
            "continuity only (character voice, an ongoing conversation, "
            "pronouns/referents). Do NOT re-translate or repeat any of this "
            "in your output — translate only the NEW page shown in the image.\n"
        )
        for name, text in past_pages:
            parts.append(f"\n[Previous page: {name}]\n{text}")
    if ahead_pages:
        if parts:
            parts.append("\n\n---\n\n")
        parts.append(
            "Context: the ORIGINAL JAPANESE (not a translation) of the "
            "upcoming page(s) that follow the one you're translating now. "
            "Use this only to correctly resolve things that depend on what "
            "happens next — pronoun/referent gender, who a line is actually "
            "addressed to, a setup whose payoff lands later. Do NOT "
            "translate or otherwise output any of this — translate only the "
            "CURRENT page shown in the image.\n"
        )
        for name, text in ahead_pages:
            parts.append(f"\n[Upcoming page ({name}), original Japanese]\n{text}")
    return "".join(parts)
@@ -8,10 +8,9 @@ anything else exposed through such an endpoint.
 Manga pages split their text across scattered speech bubbles, thought
 bubbles, narration boxes, and SFX rather than running prose, so this script
 has genuinely different logic from novel_ocr.py: it transcribes each page
-as a numbered, typed, reading-order list of bubbles, and — since that's
+as a numbered, typed, reading-order list of bubbles. The --translate
-what most people actually want out of a manga OCR pass — can translate
+machinery (glossary, sliding-window context) is shared infrastructure from
-directly instead, with a glossary for consistent names/terms and a sliding
+ocr/common.py — the same helpers novel_ocr.py uses for its own --translate.
 window of previous pages' translations for continuity.
 Usage (after filling in config.json):
    python manga_ocr_llm.py --input ./pages --output ./out
@@ -36,6 +35,11 @@ Usage (after filling in config.json):
    python manga_ocr_llm.py --input ./pages --output ./out \\
        --translate --target-lang Russian --context-pages-ahead 1
    # re-OCR/re-translate just a few pages (e.g. after editing the prompt
    # or the glossary), without redoing the whole book:
    python manga_ocr_llm.py --input ./pages --output ./out \\
        --translate --pages "12,40-42" --redo
 Requirements:
    pip install openai pillow natsort tqdm
@@ -46,18 +50,27 @@ One-time setup:
    3. Optionally edit prompt_manga.txt (transcription) or
       prompt_manga_translate.txt (--translate) to fit your book / house style
 System/user prompt split: the format instructions (and, with --translate,
 the glossary) are sent as the `system` message, since they're identical on
 every page of a run — most OpenAI-compatible providers cache a repeated
 system prefix, so a long book isn't re-billed/re-processed for the same
 instructions on every single page. Only the image and any sliding-window
 context (which does vary page to page) go in `user`. See ocr/common.py's
 module docstring for more.
 Every request/response is logged as one JSON file under logs/ at the repo
 root (--log-dir to change, --no-log to disable) — full prompt, model
 params, every retry attempt, and the full raw API response (the base64
 image itself is never included, only its path/size).
 Progress/resume state: like novel_ocr.py, this writes a state.json
 manifest to --output (model, prompt hash, usage, timestamps) so pages
 transcribed under a now-stale prompt/glossary can be spotted, and so
 --pages/--redo can target just the pages that actually need re-running.
 """
 import argparse
 import base64
 import io
 import json
 import os
 import re
 import sys
 import time
 from collections import deque
@@ -69,124 +82,12 @@ from openai import OpenAI
 from PIL import Image
 from tqdm import tqdm
 IMAGE_EXTS = {".jpg", ".jpeg", ".png", ".webp", ".tif", ".tiff", ".bmp"}
 SCRIPT_DIR = Path(__file__).resolve().parent
 ROOT_DIR = SCRIPT_DIR.parent
-sys.path.insert(0, str(SCRIPT_DIR))  # so `from local_mangaocr_ocr import ...` works when run from elsewhere
+sys.path.insert(0, str(SCRIPT_DIR))  # so `import common` / `from local_mangaocr_ocr import ...` work when run from elsewhere
 import common
-
+IMAGE_EXTS = common.IMAGE_EXTS
 def load_config(config_path: Path) -> dict:
    """Loads the shared config.json and returns its "ocr" section.
    Falls back to treating the whole file as the ocr config if there's no
    "ocr" key, so a bare {"api_key": ...} style file still works.
    """
    if not config_path.exists():
        return {}
    try:
        data = json.loads(config_path.read_text(encoding="utf-8"))
    except json.JSONDecodeError as e:
        print(f"Failed to parse {config_path}: {e}", file=sys.stderr)
        sys.exit(1)
    return data.get("ocr", data) if isinstance(data, dict) else {}
 def build_request_kwargs(config: dict, args: argparse.Namespace) -> tuple[dict, dict]:
    """Resolves OpenAI-API-style request params: CLI flag > config.json > a
    sensible default. Returns (kwargs, extra_body) — standard params go
    straight into the request; reasoning_effort goes through extra_body
    since support for it varies by provider/model and extra_body is the
    designed passthrough for exactly that.
    """
    temperature = args.temperature if args.temperature is not None else config.get("temperature", 0)
    max_tokens = args.max_tokens if args.max_tokens is not None else config.get("max_tokens")
    top_p = args.top_p if args.top_p is not None else config.get("top_p")
    reasoning_effort = args.reasoning_effort or config.get("reasoning_effort")
    kwargs = {"temperature": temperature}
    if max_tokens is not None:
        kwargs["max_tokens"] = max_tokens
    if top_p is not None:
        kwargs["top_p"] = top_p
    extra_body = {}
    if reasoning_effort:
        extra_body["reasoning_effort"] = reasoning_effort
    return kwargs, extra_body
 def load_prompt(prompt_path: Path, target_language: str = None, glossary: str = None) -> str:
    if not prompt_path.exists():
        print(f"Prompt file not found: {prompt_path}", file=sys.stderr)
        sys.exit(1)
    prompt = prompt_path.read_text(encoding="utf-8").strip()
    if target_language:
        prompt = prompt.replace("{target_language}", target_language)
    if glossary:
        prompt += (
            "\n\n---\n\n"
            "Glossary: use these established translations for the following "
            "names/terms for consistency. Anything not listed here, translate "
            "naturally.\n\n" + glossary
        )
    return prompt
 def find_glossary(explicit_path: str) -> Path:
    """Resolves the glossary path: explicit --glossary if given, otherwise
    an auto-detected glossary.md next to the ocr scripts themselves (a
    stable, book-independent location — handy for a future UI to edit)."""
    if explicit_path:
        path = Path(explicit_path)
        if not path.exists():
            print(f"Glossary file not found: {path}", file=sys.stderr)
            sys.exit(1)
        return path
    auto_path = SCRIPT_DIR / "glossary.md"
    return auto_path if auto_path.exists() else None
 def build_context_block(past_pages: list, ahead_pages: list) -> str:
    """Builds a prompt suffix with:
    - past_pages: the last N *translated* pages, for continuity (character
      voice, an ongoing exchange, pronouns/referents that only make sense
      given what was just said).
    - ahead_pages: the next N pages' raw *Japanese* transcription (never a
      draft translation — see the ocr/README.md design note on why), for
      context that only becomes clear from what's about to happen (a
      twist, who's actually being addressed, a joke that pays off a page
      later).
    Both are lists of (page_name, text) tuples, oldest/nearest first.
    """
    if not past_pages and not ahead_pages:
        return ""
    parts = []
    if past_pages:
        parts.append(
            "\n\n---\n\n"
            "Context: translations of the immediately preceding page(s), for "
            "continuity only (character voice, an ongoing conversation, "
            "pronouns/referents). Do NOT re-translate or repeat any of this "
            "in your output — translate only the NEW page shown in the image.\n"
        )
        for name, text in past_pages:
            parts.append(f"\n[Previous page: {name}]\n{text}")
    if ahead_pages:
        parts.append(
            "\n\n---\n\n"
            "Context: the ORIGINAL JAPANESE (not a translation) of the "
            "upcoming page(s) that follow the one you're translating now. "
            "Use this only to correctly resolve things that depend on what "
            "happens next — pronoun/referent gender, who a line is actually "
            "addressed to, a setup whose payoff lands later. Do NOT "
            "translate or otherwise output any of this — translate only the "
            "CURRENT page shown in the image.\n"
        )
        for name, text in ahead_pages:
            parts.append(f"\n[Upcoming page ({name}), original Japanese]\n{text}")
    return "".join(parts)
 def get_japanese_context(
@@ -224,8 +125,8 @@ def get_japanese_context(
            model = ahead_ocr_state["model"]
            request_kwargs = ahead_ocr_state["request_kwargs"]
            extra_body = ahead_ocr_state["extra_body"]
-            prompt = ahead_ocr_state["prompt"]
+            system_prompt = ahead_ocr_state["system_prompt"]
-            text = ocr_image(client, model, prompt, img_path, request_kwargs, extra_body)
+            text, _meta = common.ocr_image(client, model, system_prompt, img_path, request_kwargs, extra_body)
    except Exception as e:  # noqa: BLE001
        print(f"\nWarning: couldn't OCR {img_path.name} for lookahead context: {e}", file=sys.stderr)
        return ""
@@ -236,93 +137,6 @@ def get_japanese_context(
    return text
 def write_log(log_dir: Path, page_name: str, entry: dict) -> None:
    """Writes one JSON log file per request/response.
    The base64 image payload itself is never included (megabytes of no
    debugging value, one per page) — only its path/size — but everything
    else (full prompt text, model params, every retry attempt, the full raw
    API response) is recorded as-is.
    """
    log_dir.mkdir(parents=True, exist_ok=True)
    timestamp = datetime.now().strftime("%Y%m%d_%H%M%S_%f")
    safe_name = re.sub(r"[^A-Za-z0-9_.-]", "_", page_name)
    log_path = log_dir / f"{timestamp}_{safe_name}.json"
    log_path.write_text(json.dumps(entry, ensure_ascii=False, indent=2, default=str), encoding="utf-8")
 def image_to_data_url(path: Path, max_dim: int = 2200) -> str:
    """Downscale (if needed) and encode an image as a base64 data URL."""
    img = Image.open(path)
    if img.mode not in ("RGB", "L"):
        img = img.convert("RGB")
    if max(img.size) > max_dim:
        ratio = max_dim / max(img.size)
        img = img.resize((int(img.width * ratio), int(img.height * ratio)))
    buf = io.BytesIO()
    img.save(buf, format="JPEG", quality=90)
    b64 = base64.b64encode(buf.getvalue()).decode("utf-8")
    return f"data:image/jpeg;base64,{b64}"
 def ocr_image(
    client: OpenAI, model: str, prompt: str, path: Path,
    request_kwargs: dict, extra_body: dict,
    retries: int = 3, log_entry: dict = None,
 ) -> str:
    data_url = image_to_data_url(path)
    if log_entry is not None:
        log_entry["request"] = {
            "model": model,
            **request_kwargs,
            "extra_body": extra_body or None,
            "messages": [
                {
                    "role": "user",
                    "content": [
                        {"type": "text", "text": prompt},
                        {"type": "image_url", "image_url": {"url": f"<omitted: {path.name}, {path.stat().st_size} bytes>"}},
                    ],
                }
            ],
        }
        log_entry["attempts"] = []
    last_err = None
    for attempt in range(retries):
        try:
            response = client.chat.completions.create(
                model=model,
                messages=[
                    {
                        "role": "user",
                        "content": [
                            {"type": "text", "text": prompt},
                            {"type": "image_url", "image_url": {"url": data_url}},
                        ],
                    }
                ],
                extra_body=extra_body or None,
                **request_kwargs,
            )
            text = (response.choices[0].message.content or "").strip()
            if log_entry is not None:
                try:
                    raw_response = response.model_dump()
                except Exception:  # noqa: BLE001
                    raw_response = {"content": text}
                log_entry["attempts"].append({"attempt": attempt + 1, "success": True, "response": raw_response})
            return text
        except Exception as e:  # noqa: BLE001
            last_err = e
            if log_entry is not None:
                log_entry["attempts"].append({"attempt": attempt + 1, "success": False, "error": str(e)})
            time.sleep(3 * (attempt + 1))
    raise RuntimeError(f"Failed to OCR {path.name} after {retries} attempts: {last_err}")
 def main():
    parser = argparse.ArgumentParser(description="Batch-OCR/translate manga pages via an OpenAI-compatible API")
    parser.add_argument("--input", required=True, help="Folder with scanned page images (jpg/png)")
@@ -381,7 +195,7 @@ def main():
             "original-Japanese .txt files, named exactly like the page image "
             "stems (e.g. page005.txt for page005.jpg) — hand-prepared ones "
             "are used as-is; OCR'd-on-the-fly ones are cached here too. "
-             "Defaults to <output>/context_src/"
+             "Defaults to <o>/context_src/"
    )
    parser.add_argument(
        "--prompt-file", default=None,
@@ -402,7 +216,23 @@ def main():
        help="Override reasoning_effort from config.json (e.g. low/medium/high — support "
             "depends on the model/provider; omitted from the request unless set)"
    )
-    parser.add_argument("--start-page", type=int, default=1)
+    parser.add_argument(
        "--start-page", type=int, default=1,
        help="1-based index of the first page to process (pages before it are skipped entirely)"
    )
    parser.add_argument(
        "--pages", default=None,
        help="Only process these pages, e.g. \"3,7-12,20\" (1-based positions in the sorted page "
             "list). Combine with --redo to re-OCR/re-translate just those pages."
    )
    parser.add_argument(
        "--redo", action="store_true",
        help="Re-OCR/re-translate pages even if output already exists (respects --pages if given)"
    )
    parser.add_argument(
        "--no-state", action="store_true",
        help="Ignore state.json and fall back to plain \"file exists = done\" behaviour"
    )
    parser.add_argument(
        "--sleep", type=float, default=0.0,
        help="Delay in seconds between requests (useful if you're hitting rate limits)"
@@ -418,7 +248,7 @@ def main():
    )
    args = parser.parse_args()
-    config = load_config(Path(args.config))
+    config = common.load_config_section(Path(args.config), "ocr")
    api_key = args.api_key or config.get("api_key") or os.environ.get("API_KEY")
    base_url = args.base_url or config.get("base_url")
@@ -433,12 +263,12 @@ def main():
        )
        sys.exit(1)
-    request_kwargs, extra_body = build_request_kwargs(config, args)
+    request_kwargs, extra_body = common.build_request_kwargs(config, args)
    if args.translate:
        default_prompt_name = "prompt_manga_translate.txt"
        target_language = args.target_lang
-        glossary_path = find_glossary(args.glossary)
+        glossary_path = common.find_glossary(SCRIPT_DIR, args.glossary)
        glossary = glossary_path.read_text(encoding="utf-8").strip() if glossary_path else None
        if glossary_path:
            print(f"Using glossary: {glossary_path}")
@@ -447,15 +277,22 @@ def main():
        target_language = None
        glossary = None
    prompt_file = Path(args.prompt_file) if args.prompt_file else SCRIPT_DIR / default_prompt_name
-    prompt = load_prompt(prompt_file, target_language, glossary)
+    base_prompt = common.load_prompt_file(prompt_file)
    system_prompt = common.build_system_prompt(base_prompt, target_language, glossary)
    prompt_hash = common.prompt_fingerprint(system_prompt)
    # Plain transcription system prompt for OCR'ing lookahead pages — never
    # the translate prompt (we want the raw Japanese, not a draft
    # translation — see the design note in common.build_context_block).
    ahead_system_prompt = (
        system_prompt if not args.translate
        else common.build_system_prompt(common.load_prompt_file(SCRIPT_DIR / "prompt_manga.txt"))
    )
    use_ahead_context = args.translate and args.context_pages_ahead > 0
    ahead_ocr_state = {}
    if use_ahead_context and args.ahead_ocr_backend == "llm":
-        # Plain transcription prompt/params for OCR'ing lookahead pages —
+        ahead_ocr_state["system_prompt"] = ahead_system_prompt
        # never the translate prompt (we want the raw Japanese, not a
        # draft translation — see the design note in build_context_block).
        ahead_ocr_state["prompt"] = load_prompt(SCRIPT_DIR / "prompt_manga.txt")
        ahead_ocr_state["request_kwargs"] = request_kwargs
        ahead_ocr_state["extra_body"] = extra_body
        # client/model are filled in below, once the real client exists.
@@ -473,7 +310,33 @@ def main():
        print(f"No images found in {input_dir}.", file=sys.stderr)
        sys.exit(1)
-    print(f"Pages found: {len(images)}")
+    if args.start_page < 1:
        print("--start-page must be >= 1.", file=sys.stderr)
        sys.exit(1)
    total_found = len(images)
    try:
        selected = common.parse_page_selection(args.pages, images) if args.pages else None
    except ValueError as e:
        print(str(e), file=sys.stderr)
        sys.exit(1)
    # --start-page skips pages entirely (matches novel_ocr.py); numbering in
    # combined.md picks back up from --start-page regardless.
    if args.start_page > 1:
        images = images[args.start_page - 1:]
    if selected is not None:
        images = [p for p in images if p.stem in selected]
    if not images:
        print("No pages left to process after applying --start-page/--pages.", file=sys.stderr)
        sys.exit(1)
    print(f"Pages found: {total_found}; processing {len(images)}")
    state = {} if args.no_state else common.load_state(output_dir)
    if state:
        common.warn_if_prompt_changed(state, images, prompt_hash, prompt_file.name)
    log_dir = None if args.no_log else Path(args.log_dir)
@@ -490,16 +353,21 @@ def main():
    combined_path = output_dir / "combined.md"
    failed = []
    redone = 0
    totals = {}
    with open(combined_path, "w", encoding="utf-8") as combined_f:
        for pos, img_path in enumerate(tqdm(images, desc="OCR")):
            idx = args.start_page + pos
            txt_out = pages_dir / f"{img_path.stem}.txt"
            has_txt = txt_out.exists()
            # A page is skipped if its file exists, unless --redo was passed.
            # --no-state only disables the manifest, not this rule.
            needs_run = args.redo or not has_txt
-            if txt_out.exists():
+            if not needs_run:
                text = txt_out.read_text(encoding="utf-8")
            else:
                page_prompt = prompt
                context_used_past = []
                context_used_ahead = []
                if use_context and recent_context:
@@ -509,8 +377,7 @@ def main():
                        jp_text = get_japanese_context(future_img, context_src_dir, args.ahead_ocr_backend, ahead_ocr_state)
                        if jp_text and jp_text != "[NO_TEXT]":
                            context_used_ahead.append((future_img.stem, jp_text))
-                if context_used_past or context_used_ahead:
+                user_text = common.build_context_block(context_used_past, context_used_ahead) or None
                    page_prompt = prompt + build_context_block(context_used_past, context_used_ahead)
                log_entry = {
                    "timestamp": datetime.now().isoformat(),
@@ -525,10 +392,27 @@ def main():
                    "base_url": base_url,
                } if log_dir is not None else None
                try:
-                    text = ocr_image(client, model, page_prompt, img_path, request_kwargs, extra_body, log_entry=log_entry)
+                    text, meta = common.ocr_image(
                        client, model, system_prompt, img_path, request_kwargs, extra_body,
                        user_text=user_text, log_entry=log_entry,
                    )
                    txt_out.write_text(text, encoding="utf-8")
                    state[img_path.stem] = {
                        "model": model,
                        "prompt_hash": prompt_hash,
                        "finish_reason": meta.get("finish_reason"),
                        "usage": meta.get("usage"),
                        "chars": len(text),
                        "timestamp": datetime.now().isoformat(),
                    }
                    if has_txt:
                        redone += 1
                    usage = meta.get("usage") or {}
                    for key, val in usage.items():
                        if isinstance(val, int):
                            totals[key] = totals.get(key, 0) + val
                except Exception as e:  # noqa: BLE001
-                    print(f"\nError on {img_path.name}: {e}", file=sys.stderr)
+                    tqdm.write(f"Error on {img_path.name}: {e}", file=sys.stderr)
                    failed.append(img_path.name)
                    text = ""
                    if log_entry is not None:
@@ -537,7 +421,7 @@ def main():
                    # run would see the file exists and assume the page is already
                    # done, silently skipping a retry forever.
                if log_entry is not None:
-                    write_log(log_dir, img_path.stem, log_entry)
+                    common.write_log(log_dir, img_path.stem, log_entry)
                if args.sleep:
                    time.sleep(args.sleep)
@@ -550,12 +434,30 @@ def main():
            combined_f.write(f"\n\n<!-- page {idx}: {img_path.name} -->\n\n")
            combined_f.write(text)
        if not args.no_state:
            common.save_state(output_dir, state, {
                "backend": "manga_ocr_llm",
                "model": model,
                "translate": args.translate,
                "target_lang": args.target_lang if args.translate else None,
                "prompt_file": prompt_file.name,
                "prompt_hash": prompt_hash,
                "pages_total": total_found,
                "updated": datetime.now().isoformat(),
            })
    print(f"\nDone. Combined file: {combined_path}")
    print(f"Per-page files: {pages_dir}")
    if use_ahead_context:
        print(f"Cached lookahead Japanese context: {context_src_dir}")
    if log_dir is not None:
        print(f"Request/response logs: {log_dir}")
    if redone:
        print(f"Re-OCR'd: {redone} page(s)")
    if totals:
        print("Tokens: " + ", ".join(f"{k}={v}" for k, v in sorted(totals.items())))
    if not args.no_state:
        print(f"Manifest: {output_dir / common.STATE_FILE}")
    if failed:
        print(f"\nFailed to OCR {len(failed)} page(s):")
        for name in failed:
@@ -1,22 +1,27 @@
 #!/usr/bin/env python3
 """
-Batch-OCR for scanned pages of a Japanese light novel (vertical text) via any
+Batch-OCR (or batch-translate) for scanned pages of a Japanese light novel
-OpenAI-compatible API (OpenRouter, a direct provider endpoint, a
+(vertical text) via any OpenAI-compatible API (OpenRouter, a direct
-self-hosted proxy, etc.) — including Gemini, GPT-4V-class models, or
+provider endpoint, a self-hosted proxy, etc.) — including Gemini,
-anything else exposed through such an endpoint.
+GPT-4V-class models, or anything else exposed through such an endpoint.
 This lets a multimodal LLM "read" a page image directly, including
 vertical Japanese text, without any column-segmentation preprocessing.
-Deliberately simple: light novel pages are dense running prose, and all
+Deliberately simple compared to manga_ocr_llm.py: light novel pages are
-this script needs to do is transcribe them accurately. It has no
+dense running prose read in one linear order, so there's no bubble
-translate/glossary/context-page machinery — see manga_ocr_llm.py for that,
+type/reading-order logic here — see manga_ocr_llm.py for that, which is a
-which is a genuinely different job (short, scattered dialogue that needs
+genuinely different job (scattered speech bubbles across panels). The
-continuity handling).
+--translate machinery (glossary, sliding-window context) is shared
 infrastructure from ocr/common.py, used the same way by both scripts.
 Usage (after filling in config.json):
    python novel_ocr.py --input ./pages --output ./out
    # translate instead of transcribing:
    python novel_ocr.py --input ./pages --output ./out \\
        --translate --target-lang Russian
 Requirements:
    pip install openai pillow natsort tqdm
@@ -24,7 +29,16 @@ One-time setup:
    1. Copy config.example.json (repo root) -> config.json
    2. Fill in the "ocr" section: api_key, base_url, model (temperature/
       max_tokens/top_p/reasoning_effort are optional — see config.example.json)
-    3. Optionally edit prompt_novel.txt to fit your book / house style
+    3. Optionally edit prompt_novel.txt (transcription) or
       prompt_novel_translate.txt (--translate) to fit your book / house style
 System/user prompt split: the format instructions (and, with --translate,
 the glossary) are sent as the `system` message, since they're identical on
 every page of a run — most OpenAI-compatible providers cache a repeated
 system prefix, so a long book isn't re-billed/re-processed for the same
 instructions on every single page. Only the image (and, with --translate
 --context-pages, a bit of sliding-window context) varies per page and goes
 in `user`. See ocr/common.py's module docstring for more.
 Every request/response is logged as one JSON file under logs/ at the repo
 root (--log-dir to change, --no-log to disable) — full prompt, model
@@ -33,163 +47,27 @@ image itself is never included, only its path/size).
 """
 import argparse
 import base64
 import io
 import json
 import os
 import re
 import sys
 import time
 from collections import deque
 from datetime import datetime
 from pathlib import Path
 from natsort import natsorted
 from openai import OpenAI
 from PIL import Image
 from tqdm import tqdm
-IMAGE_EXTS = {".jpg", ".jpeg", ".png", ".webp", ".tif", ".tiff", ".bmp"}
+sys.path.insert(0, str(Path(__file__).resolve().parent))  # so `import common` works when run from elsewhere
 import common
 IMAGE_EXTS = common.IMAGE_EXTS
 SCRIPT_DIR = Path(__file__).resolve().parent
 ROOT_DIR = SCRIPT_DIR.parent
 def load_config(config_path: Path) -> dict:
    """Loads the shared config.json and returns its "ocr" section.
    Falls back to treating the whole file as the ocr config if there's no
    "ocr" key, so a bare {"api_key": ...} style file still works.
    """
    if not config_path.exists():
        return {}
    try:
        data = json.loads(config_path.read_text(encoding="utf-8"))
    except json.JSONDecodeError as e:
        print(f"Failed to parse {config_path}: {e}", file=sys.stderr)
        sys.exit(1)
    return data.get("ocr", data) if isinstance(data, dict) else {}
 def load_prompt(prompt_path: Path) -> str:
    if not prompt_path.exists():
        print(f"Prompt file not found: {prompt_path}", file=sys.stderr)
        sys.exit(1)
    return prompt_path.read_text(encoding="utf-8").strip()
 def build_request_kwargs(config: dict, args: argparse.Namespace) -> tuple[dict, dict]:
    """Resolves OpenAI-API-style request params: CLI flag > config.json > a
    sensible default. Returns (kwargs, extra_body) — standard params go
    straight into the request; reasoning_effort goes through extra_body
    since support for it varies by provider/model and extra_body is the
    designed passthrough for exactly that.
    """
    temperature = args.temperature if args.temperature is not None else config.get("temperature", 0)
    max_tokens = args.max_tokens if args.max_tokens is not None else config.get("max_tokens")
    top_p = args.top_p if args.top_p is not None else config.get("top_p")
    reasoning_effort = args.reasoning_effort or config.get("reasoning_effort")
    kwargs = {"temperature": temperature}
    if max_tokens is not None:
        kwargs["max_tokens"] = max_tokens
    if top_p is not None:
        kwargs["top_p"] = top_p
    extra_body = {}
    if reasoning_effort:
        extra_body["reasoning_effort"] = reasoning_effort
    return kwargs, extra_body
 def write_log(log_dir: Path, page_name: str, entry: dict) -> None:
    """Writes one JSON log file per request/response.
    The base64 image payload itself is never included (megabytes of no
    debugging value, one per page) — only its path/size — but everything
    else (full prompt text, model params, every retry attempt, the full raw
    API response) is recorded as-is.
    """
    log_dir.mkdir(parents=True, exist_ok=True)
    timestamp = datetime.now().strftime("%Y%m%d_%H%M%S_%f")
    safe_name = re.sub(r"[^A-Za-z0-9_.-]", "_", page_name)
    log_path = log_dir / f"{timestamp}_{safe_name}.json"
    log_path.write_text(json.dumps(entry, ensure_ascii=False, indent=2, default=str), encoding="utf-8")
 def image_to_data_url(path: Path, max_dim: int = 2200) -> str:
    """Downscale (if needed) and encode an image as a base64 data URL."""
    img = Image.open(path)
    if img.mode not in ("RGB", "L"):
        img = img.convert("RGB")
    if max(img.size) > max_dim:
        ratio = max_dim / max(img.size)
        img = img.resize((int(img.width * ratio), int(img.height * ratio)))
    buf = io.BytesIO()
    img.save(buf, format="JPEG", quality=90)
    b64 = base64.b64encode(buf.getvalue()).decode("utf-8")
    return f"data:image/jpeg;base64,{b64}"
 def ocr_image(
    client: OpenAI, model: str, prompt: str, path: Path,
    request_kwargs: dict, extra_body: dict,
    retries: int = 3, log_entry: dict = None,
 ) -> str:
    data_url = image_to_data_url(path)
    if log_entry is not None:
        log_entry["request"] = {
            "model": model,
            **request_kwargs,
            "extra_body": extra_body or None,
            "messages": [
                {
                    "role": "user",
                    "content": [
                        {"type": "text", "text": prompt},
                        {"type": "image_url", "image_url": {"url": f"<omitted: {path.name}, {path.stat().st_size} bytes>"}},
                    ],
                }
            ],
        }
        log_entry["attempts"] = []
    last_err = None
    for attempt in range(retries):
        try:
            response = client.chat.completions.create(
                model=model,
                messages=[
                    {
                        "role": "user",
                        "content": [
                            {"type": "text", "text": prompt},
                            {"type": "image_url", "image_url": {"url": data_url}},
                        ],
                    }
                ],
                extra_body=extra_body or None,
                **request_kwargs,
            )
            text = (response.choices[0].message.content or "").strip()
            if log_entry is not None:
                try:
                    raw_response = response.model_dump()
                except Exception:  # noqa: BLE001
                    raw_response = {"content": text}
                log_entry["attempts"].append({"attempt": attempt + 1, "success": True, "response": raw_response})
            return text
        except Exception as e:  # noqa: BLE001
            last_err = e
            if log_entry is not None:
                log_entry["attempts"].append({"attempt": attempt + 1, "success": False, "error": str(e)})
            time.sleep(3 * (attempt + 1))
    raise RuntimeError(f"Failed to OCR {path.name} after {retries} attempts: {last_err}")
 def main():
-    parser = argparse.ArgumentParser(description="Batch-OCR a light novel via an OpenAI-compatible API")
+    parser = argparse.ArgumentParser(description="Batch-OCR/translate a light novel via an OpenAI-compatible API")
    parser.add_argument("--input", required=True, help="Folder with scanned page images (jpg/png)")
    parser.add_argument("--output", required=True, help="Folder for the OCR results")
    parser.add_argument(
@@ -198,8 +76,9 @@ def main():
             "Defaults to config.json at the repo root."
    )
    parser.add_argument(
-        "--prompt-file", default=str(SCRIPT_DIR / "prompt_novel.txt"),
+        "--prompt-file", default=None,
-        help="Path to the prompt file (defaults to prompt_novel.txt next to this script)"
+        help="Path to the prompt file (defaults to prompt_novel.txt, or "
             "prompt_novel_translate.txt with --translate)"
    )
    parser.add_argument("--base-url", default=None, help="Override base_url from config.json")
    parser.add_argument("--model", default=None, help="Override model from config.json")
@@ -215,7 +94,23 @@ def main():
        help="Override reasoning_effort from config.json (e.g. low/medium/high — support "
             "depends on the model/provider; omitted from the request unless set)"
    )
-    parser.add_argument("--start-page", type=int, default=1)
+    parser.add_argument(
        "--start-page", type=int, default=1,
        help="1-based index of the first page to process (pages before it are skipped entirely)"
    )
    parser.add_argument(
        "--pages", default=None,
        help="Only process these pages, e.g. \"3,7-12,20\" (1-based positions in the sorted page "
             "list). Combine with --redo to re-OCR just those pages."
    )
    parser.add_argument(
        "--redo", action="store_true",
        help="Re-OCR pages even if a transcription already exists (respects --pages if given)"
    )
    parser.add_argument(
        "--no-state", action="store_true",
        help="Ignore state.json and fall back to plain \"file exists = done\" behaviour"
    )
    parser.add_argument(
        "--sleep", type=float, default=0.0,
        help="Delay in seconds between requests (useful if you're hitting rate limits)"
@@ -229,9 +124,35 @@ def main():
        "--no-log", action="store_true",
        help="Disable request/response logging entirely"
    )
    # --- Translation (mirrors manga_ocr_llm.py) ---
    parser.add_argument(
        "--translate", action="store_true",
        help="Translate the page instead of transcribing the Japanese "
             "(uses prompt_novel_translate.txt by default)"
    )
    parser.add_argument(
        "--target-lang", default="English",
        help="Target language for --translate, written out as a plain name "
             "(e.g. Russian, English, Spanish). Default: English"
    )
    parser.add_argument(
        "--glossary", default=None,
        help="[--translate only] Path to a glossary.md of established name/term "
             "translations, sent along with the prompt for consistency (see "
             "ocr/glossary.example.md). Auto-detected as glossary.md next to "
             "this script if not given. Shared with manga_ocr_llm.py."
    )
    parser.add_argument(
        "--context-pages", type=int, default=2,
        help="[--translate only] Include the last N translated pages in the "
             "prompt for continuity (character voice, ongoing narration, "
             "pronouns/referents). 0 disables it. Default: 2"
    )
    args = parser.parse_args()
-    config = load_config(Path(args.config))
+    config = common.load_config_section(Path(args.config), "ocr")
    api_key = args.api_key or config.get("api_key") or os.environ.get("API_KEY")
    base_url = args.base_url or config.get("base_url")
@@ -246,8 +167,24 @@ def main():
        )
        sys.exit(1)
-    request_kwargs, extra_body = build_request_kwargs(config, args)
+    request_kwargs, extra_body = common.build_request_kwargs(config, args)
-    prompt = load_prompt(Path(args.prompt_file))
+
    if args.translate:
        default_prompt_name = "prompt_novel_translate.txt"
        target_language = args.target_lang
        glossary_path = common.find_glossary(SCRIPT_DIR, args.glossary)
        glossary = glossary_path.read_text(encoding="utf-8").strip() if glossary_path else None
        if glossary_path:
            print(f"Using glossary: {glossary_path}")
    else:
        default_prompt_name = "prompt_novel.txt"
        target_language = None
        glossary = None
    prompt_file = Path(args.prompt_file) if args.prompt_file else SCRIPT_DIR / default_prompt_name
    base_prompt = common.load_prompt_file(prompt_file)
    system_prompt = common.build_system_prompt(base_prompt, target_language, glossary)
    prompt_hash = common.prompt_fingerprint(system_prompt)
    input_dir = Path(args.input)
    output_dir = Path(args.output)
@@ -262,34 +199,95 @@ def main():
        print(f"No images found in {input_dir}.", file=sys.stderr)
        sys.exit(1)
-    print(f"Pages found: {len(images)}")
+    if args.start_page < 1:
        print("--start-page must be >= 1.", file=sys.stderr)
        sys.exit(1)
    total_found = len(images)
    try:
        selected = common.parse_page_selection(args.pages, images) if args.pages else None
    except ValueError as e:
        print(str(e), file=sys.stderr)
        sys.exit(1)
    # --start-page now actually skips pages rather than only relabelling them.
    if args.start_page > 1:
        images = images[args.start_page - 1:]
    if selected is not None:
        images = [p for p in images if p.stem in selected]
    if not images:
        print("No pages left to process after applying --start-page/--pages.", file=sys.stderr)
        sys.exit(1)
    print(f"Pages found: {total_found}; processing {len(images)}")
    state = {} if args.no_state else common.load_state(output_dir)
    if state:
        common.warn_if_prompt_changed(state, images, prompt_hash, prompt_file.name)
    log_dir = None if args.no_log else Path(args.log_dir)
    use_context = args.translate and args.context_pages > 0
    recent_context = deque(maxlen=args.context_pages) if use_context else None
    client = OpenAI(base_url=base_url, api_key=api_key)
    combined_path = output_dir / "combined.md"
    failed = []
    redone = 0
    totals = {}
    with open(combined_path, "w", encoding="utf-8") as combined_f:
        for idx, img_path in enumerate(tqdm(images, desc="OCR"), start=args.start_page):
            txt_out = pages_dir / f"{img_path.stem}.txt"
            has_txt = txt_out.exists()
            # A page is skipped if its file exists, unless --redo was passed.
            # --no-state only disables the manifest, not this rule.
            needs_run = args.redo or not has_txt
-            if txt_out.exists():
+            if not needs_run:
                text = txt_out.read_text(encoding="utf-8")
            else:
                user_text = None
                context_used = []
                if use_context and recent_context:
                    context_used = list(recent_context)
                    user_text = common.build_context_block(context_used, [])
                log_entry = {
                    "timestamp": datetime.now().isoformat(),
                    "backend": "novel_ocr",
                    "page": img_path.name,
                    "translate": args.translate,
                    "target_lang": args.target_lang if args.translate else None,
                    "glossary_used": bool(glossary),
                    "context_pages_used": [name for name, _ in context_used],
                    "model": model,
                    "base_url": base_url,
                } if log_dir is not None else None
                try:
-                    text = ocr_image(client, model, prompt, img_path, request_kwargs, extra_body, log_entry=log_entry)
+                    text, meta = common.ocr_image(
                        client, model, system_prompt, img_path, request_kwargs, extra_body,
                        user_text=user_text, log_entry=log_entry,
                    )
                    txt_out.write_text(text, encoding="utf-8")
                    state[img_path.stem] = {
                        "model": model,
                        "prompt_hash": prompt_hash,
                        "finish_reason": meta.get("finish_reason"),
                        "usage": meta.get("usage"),
                        "chars": len(text),
                        "timestamp": datetime.now().isoformat(),
                    }
                    if has_txt:
                        redone += 1
                    usage = meta.get("usage") or {}
                    for key, val in usage.items():
                        if isinstance(val, int):
                            totals[key] = totals.get(key, 0) + val
                except Exception as e:  # noqa: BLE001
-                    print(f"\nError on {img_path.name}: {e}", file=sys.stderr)
+                    tqdm.write(f"Error on {img_path.name}: {e}", file=sys.stderr)
                    failed.append(img_path.name)
                    text = ""
                    if log_entry is not None:
@@ -298,17 +296,43 @@ def main():
                    # run would see the file exists and assume the page is already
                    # done, silently skipping a retry forever.
                if log_entry is not None:
-                    write_log(log_dir, img_path.stem, log_entry)
+                    common.write_log(log_dir, img_path.stem, log_entry)
                if args.sleep:
                    time.sleep(args.sleep)
            # Feed this page's (already-completed-or-just-translated) text
            # forward as context for the following pages. Blank/[NO_TEXT]
            # pages carry no useful continuity, so skip adding those.
            if use_context and text.strip() and text.strip() != "[NO_TEXT]":
                recent_context.append((img_path.stem, text.strip()))
            combined_f.write(f"\n\n<!-- page {idx}: {img_path.name} -->\n\n")
            combined_f.write(text)
        # Written once per run (not per page) so an interrupted run simply
        # leaves the previous manifest in place rather than a partial one.
        if not args.no_state:
            common.save_state(output_dir, state, {
                "backend": "novel_ocr",
                "model": model,
                "translate": args.translate,
                "target_lang": args.target_lang if args.translate else None,
                "prompt_file": prompt_file.name,
                "prompt_hash": prompt_hash,
                "pages_total": total_found,
                "updated": datetime.now().isoformat(),
            })
    print(f"\nDone. Combined file: {combined_path}")
    print(f"Per-page files: {pages_dir}")
    if log_dir is not None:
        print(f"Request/response logs: {log_dir}")
    if redone:
        print(f"Re-OCR'd: {redone} page(s)")
    if totals:
        print("Tokens: " + ", ".join(f"{k}={v}" for k, v in sorted(totals.items())))
    if not args.no_state:
        print(f"Manifest: {output_dir / common.STATE_FILE}")
    if failed:
        print(f"\nFailed to OCR {len(failed)} page(s):")
        for name in failed:
@@ -0,0 +1,28 @@
 This is a scanned page from a Japanese light novel, printed in vertical
 text (縦書き). Read the page in correct reading order (columns top-to-bottom,
 then right-to-left).
 Instead of transcribing the Japanese, TRANSLATE all running prose on the
 page into {target_language}.
 Rules:
 - Translate naturally and idiomatically into {target_language} — don't
  produce a stiff, literal word-for-word rendering. Preserve each
  character's tone/register (casual, formal, rude, childish, archaic, etc.)
  and the narrator's voice as far as {target_language} allows.
 - Keep honorifics (-san, -kun, -chan, senpai, etc.) untranslated if
  {target_language} has no natural equivalent, unless the whole line reads
  better without them — use your judgment.
 - Include any header/margin/footer text (page number, chapter title,
  in-story date/time stamp) translated as well, as the first line of your
  output. This novel often prints the same date/time/chapter info in two
  places on a page (e.g. top margin and a vertical strip on the side) —
  merge these into ONE single header line, do not repeat it twice.
 - Do not include the original Japanese text, romaji, or any commentary —
  output only the translation.
 - No line numbers, no markdown formatting.
 - Each paragraph goes on its own line, matching the paragraph breaks of
  the original. Do not merge multiple paragraphs into one line, and do
  not add blank lines between paragraphs.
 - If the page has no text at all (illustration-only, blank, cover), output
  exactly: [NO_TEXT]