accessible-text-analyst

Accessible Text Analyst

A multilingual NLP pipeline designed for accessibility with screen readers (NVDA, JAWS, VoiceOver, Narrator). All code, console output, and reports are crafted to be parsed cleanly by assistive technology: no ANSI colors, no emoji, no progress bars, no box-drawing characters.

Other languages: polski · русский · suomi · íslenska · italiano

i18n status (v1.0). Only the README has been translated. The Jupyter notebook narrative, all console output, CLAUDE.md, and the inline code comments remain in their original languages (mostly Russian, with Polish in the patches and the shamanic layer). Translating those is deliberately scoped for v1.1 — see release_notes.md.

Project contents

• accessible_text_analyst.ipynb — a Jupyter notebook with the complete analysis pipeline (42 cells: 21 code + 21 markdown; the in-notebook narrative is in Russian). It writes two accessibility artefacts (accessible_text.html, accessible_text.docx) where every paragraph and every foreign-language sentence carries its own lang attribute so screen readers and TTS engines switch voice automatically — even offline.
• generate_report.py — a post-processing script that turns the executed notebook into a single accessible HTML file (analysis_report.html). It wraps foreign-corpus fragments in <span lang="target_lang"> and, regardless of corpus language, hardcodes <span lang="en"> around technical English content (POS tags, NER labels, spaCy/Hugging Face model identifiers, ASCII filenames). Inline code and code blocks in the narrative are blanket-tagged lang="en". In reader mode (remove_noise: true) it truncates the notebook's per-document diagnostic loops to the first few documents instead of dumping hundreds of them.
• run_all.py — a launcher for the whole free local chain: it registers the analyst_env Jupyter kernel if it is missing, runs the notebook headlessly (without the interactive question cell) and saves its outputs into the .ipynb, then runs generate_report.py, shamanic_pipeline.py and, for a Finnish corpus, the Book of the Unsaid scan. It never starts a paid step (LLM rituals, ElevenLabs voices, the Book's draft/fill).
• generate_diagnostic.py — an independent generator of a full diagnostic report (diagnostic_report.html) built directly from the notebook's CSV/JSON exports (not from generate_report.py's output). A navigable, screen-reader-first structure: a <nav> table of contents plus sections — each with its own heading and lists — for an overview, topics with their paragraphs, theses, named entities grouped by type, and top keywords. Structural labels follow ui_lang; corpus fragments get <span lang="…">, NER labels <span lang="en">.
• generate_md.py — converts the generated analysis_report.html into gemini_notebook_report.md for ingestion into Gemini Notebook (formerly NotebookLM). The accessibility spans are unwrapped since Gemini Notebook does not consume them.
• shamanic_pipeline.py (optional) — a non-LLM post-processor that turns the notebook's CSV/JSON exports into ritual text artefacts (oracle_script.txt, lore_fragments/, raw_roots_chant.txt, prophecies.txt, and — when the optional sentiment analysis is enabled — emotional_undertow.txt), all fully localized across the six supported languages.
• shamanic_ai.py (optional, LLM-driven) — calls an LLM (OpenAI by default; OpenRouter or any OpenAI-compatible endpoint via llm_provider) to generate four narrative voices (Katla, Vieno, Lumi, Sami) on top of the same exports. When the optional sentiment.csv is present, Vieno builds her chant on the text's emotional arc instead of raw sentences. Requires the chosen provider's key in golden_key.env.
• shamanic_voice.py (optional, requires ElevenLabs) — a standalone, run-once dispatcher that synthesizes the four narrative artefacts produced by shamanic_ai.py into .mp3 audio via the ElevenLabs API, logs each success/failure to the console, and exits (no bot, no server). Needs ELEVENLABS_API_KEY in golden_key.env and a voices mapping in the config.
• shamanic_locale.py — the localization bundle for both shamanic scripts (templates, headers, and Lumi's fallback strings in all six supported languages).
• ksiega_niedomowien.py (optional, Finnish only, agent-assisted) — the Book of the Unsaid (Sanomattomien kirja): turns raw_roots_chant.txt into an incrementally growing cryptic diary whose gaps a blind model could not guess. It is the deterministic helper for the skill in skills/ksiega-niedomowien/SKILL.md; its console output follows ui_lang.

What the pipeline does

1. Loads text from a file (.pdf, .txt, .docx, .md, .html, or images: .png, .jpg, .jpeg, .tif, .tiff, .bmp, .webp) or from a URL — stripping repeated headers/footers, sidebars, "Related articles" sections, and similar boilerplate. Scanned PDFs and image inputs fall back to OCR (pypdfium2 + easyocr).
2. Detects the corpus language with lingua-language-detector. The dominant language drives the spaCy pipeline; the same detector is reused per paragraph and per sentence to power language-tagged export.
3. Loads spaCy models in two stages. First, the dominant language's model processes the entire corpus end-to-end (tokenization → stop-words → lemmatization → POS → NER). Then cell_multilang_pass re-runs the per-language NLP passes only for paragraphs whose detected language differs from the dominant one — loading each non-dominant model on demand through get_nlp(), which is wrapped in functools.lru_cache(maxsize=2). The LRU cache only matters during this second stage; at most two models live in RAM at any time.
4. Tokenization → stop-word filtering → lemmatization → POS tagging → NER (two-stage multilingual with per-sentence model dispatch).
5. Vector representations: Bag of Words, TF-IDF + auto-query search with cosine ranking.
6. Structure: sentences → paragraphs (3–6 sentences each) → theses (the best sentence per paragraph). Each paragraph and sentence is tagged with its detected ISO 639-1 code.
7. Topic modeling with KMeans over spaCy paragraph vectors.
8. (Optional, off by default) Per-paragraph sentiment scoring with cardiffnlp/twitter-xlm-roberta-base-sentiment, written to sentiment.csv. The result is never shown to the user — it exists only as feed for the shamanic layer. See Configuration (enable_sentiment).
9. CSV/JSON export + a textual summary report + an accessibility-tagged HTML and DOCX export + a global HTML report (analysis_report.html).

Supported languages

Code	Language	spaCy model	NER / vectors
pl	Polish	pl_core_news_lg	spaCy
ru	Russian	ru_core_news_lg	spaCy
en	English	en_core_web_lg	spaCy
it	Italian	it_core_news_lg	spaCy
fi	Finnish	fi_core_news_lg	spaCy
is	Icelandic	spacy.blank("is") + Hugging Face	mideind/IceBERT-base (vectors), mideind/icelandic-ner-MIM-GOLD-22 (NER)

There is no full spaCy model for Icelandic, so two Hugging Face models are wired into a blank pipeline. The first download requires ~700 MB of disk space and an internet connection; subsequent runs reuse the HF cache.

Optional Finnish morphology upgrade (omorfi). The statistical lemmatizer in fi_core_news_lg struggles with Finnish agglutinative morphology — it routinely produces garbage lemmas such as kytköknen or pisttää. When the pure-Python omorfi package is installed and its FST automata downloaded (see Installation step 5), the notebook appends an omorfi_lemmas component that overrides lemmas and UD morphological features with dictionary analyses wherever they agree with the spaCy tagger's part of speech (kytköksistä → kytkös, Euroopassa → Eurooppa, pistät → pistää). Without the package or the automata the pipeline silently keeps the stock spaCy behaviour.

Language detection

Detection runs at three levels:

1. Corpus-dominant — cell_langdet votes per document and picks the dominant language as LANG. Drives the spaCy pipeline.
2. Per-paragraph (para_langs) — used as the <p lang="..."> attribute in accessible_text.html and as the default <w:lang> for each paragraph in accessible_text.docx.
3. Per-sentence (sent_langs) — used to wrap individual sentences in <span lang="..."> inside mixed-language paragraphs (HTML) or to set <w:lang> on per-sentence runs (DOCX).

Per-paragraph and per-sentence detection uses a deliberately conservative _safe_detect() wrapper in cell_para. Trusting raw lingua output causes audible artefacts: lingua misclassifies short Latin fragments dominated by proper names ("Igor de Lendorf" → it, "Mut se mies, Igor" → en), and a screen reader switching to an English voice for "Igor de Lendorf" then back to Finnish for the next sentence sounds worse than reading the proper name with a Finnish accent.

The wrapper applies these rules in order:

1. Cyrillic anywhere in the fragment → ru. Unambiguous; works on fragments as short as 3 characters.
2. Fragment shorter than 30 characters → fallback LANG. Lingua is unreliable on short Latin fragments.
3. Fragment contains LANG's diacritics (e.g. ä/ö for fi, ąęć for pl) → fallback LANG. A Finnish sentence with an English quote is still Finnish.
4. Lingua agrees with LANG → keep it.
5. Lingua disagrees but the detected language has its OWN diacritics in the fragment (à for it, ż for pl, ð for is) → trust lingua.
6. Lingua disagrees and the detected language has no characteristic diacritics in the fragment (typically en) → fallback LANG.

The motivation for rule 6 is purely phonetic: an English voice reading Finnish/Polish/Italian fragments produces noticeable distortion ("bełkot fonemów"), whereas a Finnish/Polish/Italian voice reading an English quote with a slight accent is intelligible and unobtrusive. The trade-off is that purely-English sentences embedded in a non-English corpus are read in the dominant voice instead of being switched to English. We accept this trade-off because, in practice, screen-reader users have reported the asymmetric quality cost.

Outliers that survive the heuristic are listed in the cell_para output for manual review (Оставшиеся outlier-предложения...). Same diacritic logic governs the per-<code> heuristic (_classify_code_lang) in generate_report.py and the per-segment lingua fallback (_lingua_word_fallback) for path-like ASCII identifiers.

Requirements

• Python 3.12 or newer (the dependency set is tested on Python 3.14).
• ~1.5 GB of free disk space for spaCy _lg models (one per language you install). Add ~700 MB if you opt into Icelandic support (Hugging Face transformers + torch + IceBERT + MIM-GOLD-22).
• Add ~700 MB if you opt into OCR support (easyocr pulls torch plus its detection and recognition models on the first OCR call). If you also enabled Icelandic, the torch cost is shared between the two.
• Add ~130 MB if you opt into the Finnish morphology upgrade (the omorfi FST automata downloaded by omorfi-download).
• An internet connection on the first run (model downloads).

Environment setup

Pick one of the three options below. All three end with the same pip install -r requirements.txt + spaCy model downloads.

Option A — local venv (recommended for everyday use)

# Linux / macOS
python3 -m venv .venv
source .venv/bin/activate

# Windows (PowerShell)
python -m venv .venv
.venv\Scripts\Activate.ps1

The .venv/ directory is in .gitignore. From inside the activated venv, proceed to Installation below.

Option B — Anaconda / Miniconda

conda create -n accessible-text-analyst python=3.14
conda activate accessible-text-analyst

Then proceed to Installation below. Anaconda's bundled pip works fine — there is no Conda-specific package list for this project.

Option C — Google Colab (no local install)

Open a new Colab notebook and paste the following into the first cell. This clones the repo, installs the dependencies, and downloads the spaCy models you need.

!git clone https://github.com/<your-fork>/accessible_text_analyst.git
%cd accessible_text_analyst
!pip install -r requirements.txt
!python -m spacy download en_core_web_lg
# Add the lines below only for the languages you actually need:
# !python -m spacy download pl_core_news_lg
# !python -m spacy download ru_core_news_lg
# !python -m spacy download it_core_news_lg
# !python -m spacy download fi_core_news_lg

Then upload your config.json (or config.ini) via the Files panel, set source_file, and run the notebook. Note: Colab sessions are ephemeral — downloaded models and export_results/ artefacts disappear when the runtime is recycled. For long-form analysis, mount Google Drive and write the output there.

Installation

# 1. Python dependencies
pip install -r requirements.txt

# 2. spaCy models — install only the languages you actually need.
# The *_lg variants are required because topic modeling depends on word vectors.
python -m spacy download pl_core_news_lg
python -m spacy download ru_core_news_lg
python -m spacy download en_core_web_lg
python -m spacy download it_core_news_lg
python -m spacy download fi_core_news_lg

# 3. (Optional) Icelandic support — uncomment `transformers` and `torch`
# in requirements.txt and re-run `pip install -r requirements.txt`.
# IceBERT and MIM-GOLD-22 are then pulled from Hugging Face on first run.

# 4. (Optional) Sentiment analysis — uncomment `transformers`, `torch`,
# `sentencepiece`, `protobuf` and `tiktoken` in requirements.txt, re-run the
# install, and set "enable_sentiment": true in config.json. The model
# (~1.1 GB) is pulled from Hugging Face on first run and cached afterwards.
# All five packages are needed together to build the XLM-RoBERTa tokenizer.

# 5. (Optional) Finnish morphology upgrade — uncomment `omorfi` in
# requirements.txt, re-run the install, then download the FST automata
# (~130 MB of *.hfst files unpacked into the repo root; they are gitignored).
# Run it from the repo root:
omorfi-download

# 6. (Optional) Check the installed versions against the bounds in
# requirements.txt and against PyPI. It only warns, it never blocks:
python audyt_zaleznosci.py

Configuration

Copy the example configuration to a local file. Both config.json and config.ini are accepted — the file content is JSON in either case; the .ini extension is purely a usability concession to non-technical Windows users for whom .json has no default file handler.

# Pick one:
cp config.example.json config.json
cp config.example.ini  config.ini

config.json and config.ini are both in .gitignore, so each user keeps their own local copy.

Contents:

{
  "source_file": "C:/path/to/document.pdf",
  "custom_patterns": [],
  "remove_noise": true,
  "ocr_languages": ["en"],
  "enable_sentiment": false,
  "ui_lang": "",
  "lumi_katla_lines": null,
  "lumi_vieno_lines": null,
  "llm_provider": "",
  "llm_model": "",
  "llm_base_url": "",
  "kirja_llm_provider": "",
  "kirja_drafter_model": "",
  "kirja_filler_model": "",
  "voices": {
    "katla": "",
    "vieno": "",
    "lumi": "",
    "sami": ""
  }
}

Key	Type	Purpose
source_file	string	Path to a file (.pdf, .txt, .docx, .md, .html, or an image) or a URL (http://, https://). An empty string or a missing file falls back to the built-in example corpus.
custom_patterns	string[]	Optional regular expressions stripped from the raw text (running heads, footers, repetitive boilerplate). Example: ["Editorial: .*", "Copyright \\d{4}"].
remove_noise	boolean	Toggles generate_report.py between reader-facing mode (true, hides Hugging Face/torch loading bars and lemmatization/POS tables) and full diagnostic mode (false).
ocr_languages	string[]	Languages for easyocr (used only when a PDF is a scan or the source is an image). Within a single easyocr.Reader you can only mix languages of the same script — e.g. ["ru", "en"] for Cyrillic or ["en", "pl", "it", "fi", "is"] for Latin.
enable_sentiment	boolean	Turns on the optional per-paragraph sentiment analysis (false by default). Requires the commented sentiment dependencies (see Installation step 4). Writes sentiment.csv; the result is never shown to the user, it only feeds the shamanic layer (the emotional_undertow.txt ritual and Vieno's chant). On any load failure it silently skips — no fallback.
ui_lang	string	UI language for console output and the generated report's <html lang> attribute (pl / en / ru / fi / is / it). Empty string or unknown code → falls back to en. Independent from the analysed corpus language, which is detected automatically.
lumi_katla_lines	integer or null	Optional ornament cap for shamanic_ai.py's Lumi dispatch: how many non-empty lines of Katla's monologue Lumi sees. null or missing key = full content; integer N > 0 = first N lines.
lumi_vieno_lines	integer or null	Same as lumi_katla_lines, but for Vieno's echo chant.
voices	object	Maps each shamanic voice (katla / vieno / lumi / sami) to an ElevenLabs voice ID, consumed by shamanic_voice.py. A voice with an empty or missing ID is skipped. Only needed if you run the audio dispatcher.
llm_provider	string	LLM provider for shamanic_ai.py and the Book: openai (default), openrouter or openai_compat. Each provider reads only its own key from golden_key.env — see LLM providers and keys below.
llm_model	string	Model for that provider. Empty = the provider's default (gpt-6.1-sol / openai/gpt-6.1-sol); required for openai_compat.
llm_base_url	string	Endpoint address, read only by openai_compat (e.g. http://localhost:1234/v1). Ignored by openai and openrouter, whose addresses are fixed.
kirja_llm_provider	string	Optional provider override for the Book (ksiega_niedomowien.py draft / fill). Empty = llm_provider.
kirja_drafter_model	string	Model for draft (the Book's API drafter). Empty = the provider's default, or llm_model when the Book uses the global provider.
kirja_filler_model	string	Model for fill (the Book's blind gap-filler). Same fallback as kirja_drafter_model.

LLM providers and keys. A key is bound to one host and is never sent anywhere else: openai → OPENAI_API_KEY → api.openai.com; openrouter → OPENROUTER_API_KEY → openrouter.ai; openai_compat → LLM_COMPAT_API_KEY → llm_base_url. The provider is chosen explicitly and never inferred from an address, so pointing one script at another service can neither leak the first service's key nor break the other script. The pre-v1.7.0 key kirja_filler_base_url (which sent OPENAI_API_KEY to any address) is refused with a migration message. On OpenRouter, model ids carry a vendor prefix (openai/gpt-4o, anthropic/claude-sonnet-5.5), and the account's privacy settings let you exclude providers that may train on your prompts.

Sentiment analysis — CPU cost. With enable_sentiment: true, the per-paragraph pass is computationally heavy on CPU — roughly comparable to running local Whisper speech-to-text on CPU. On older or thermally-constrained machines this sustained load can be a real strain on the processor; enable it deliberately, ideally on a machine with a GPU or with headroom for prolonged full-load work.

ui_lang — scope of localization. The console output of the standalone scripts (generate_report.py, generate_md.py, generate_diagnostic.py, shamanic_pipeline.py, shamanic_ai.py, ksiega_niedomowien.py) is localized via ui_lang. The notebook stdout is not localized — it is hardcoded Russian (only the theses filename follows the detected corpus language). The reader-mode analysis_report.html keeps the notebook's Russian narrative (its root is <html lang="ru">); the fully ui_lang-localized, user-facing artefact is diagnostic_report.html produced by generate_diagnostic.py. Translations in fi / is / it are drafted but unreviewed — please report rough edges on GitHub.

Windows paths and regex — important. The config content is JSON, and JSON has no raw-string syntax. A single backslash escapes the next character (\U, \d, \n are special), so a Windows path written "C:\Users\marek\doc.pdf" will produce a JSON parse error. Two ways to write it correctly:

• Forward slashes (simplest, works on Windows too): "C:/Users/marek/doc.pdf".
• Double backslashes: "C:\\Users\\marek\\doc.pdf".

The same rule applies to every regular expression in custom_patterns: write "\\d{4}", not "\d{4}"; write "Copyright \\d{4}", not "Copyright \d{4}". There is no r"…" raw-string form in JSON.

Running

The numbered scripts below are intended to be run in order: each one consumes artefacts produced by the previous step.

Shortcut: the whole local chain (run_all.py)

# notebook -> generate_report.py -> shamanic_pipeline.py -> Book scan (Finnish only)
python run_all.py
# reuse the outputs already saved in the notebook
python run_all.py --skip-notebook

Steps 1, 2 and 4 below, plus the Book of the Unsaid scan for a Finnish corpus, in one command. Run it with the Python of the repo's .venv: if the analyst_env kernel is missing, it is registered for that interpreter. The chain stops at the first failing step; exit code 3 means that the source gate of the report refused (see step 2).

1. Notebook (mandatory)

jupyter notebook accessible_text_analyst.ipynb
# (Cell → Run All)
# Headless (CI): cells 0-39 via nbclient, skipping the interactive cell_qa_rag
python -c "import nbformat as f; from nbclient import NotebookClient as C; nb=f.read('accessible_text_analyst.ipynb',4); c=C(nb,timeout=None); c.create_kernel_manager(); c.start_new_kernel(); c.start_new_kernel_client(); [c.execute_cell(x,i) for i,x in enumerate(nb.cells[:40])]; f.write(nb,'accessible_text_analyst.ipynb'); c.km.shutdown_kernel(now=True)"

This is a sequential pipeline with shared global state. Do not reorder cells, and do not run them out of order. The notebook writes export_results/<project>/sentences.csv, paragraphs.csv, theses.csv, keywords_tfidf.csv, paragraphs_with_topics.csv, topic_keywords.json, entities.csv, accessible_text.html, accessible_text.docx, and theses.txt.

2. HTML reports (recommended)

python generate_report.py
# → export_results/<project>/analysis_report.html

generate_report.py reads cell outputs directly from the .ipynb file, so the HTML report must be generated from a freshly executed notebook. requirements.txt lists nbstripout — if it has been activated in the local git config, notebook outputs are stripped on commit. Generate the report before committing, or disable nbstripout for your workflow.

Source gate. Before writing anything, generate_report.py checks that the notebook outputs belong to the source_file in the config, and refuses otherwise (exit code 3, nothing written). Three checks: the source path and the export folder printed by the notebook must match the config, and the saved run must have reached the end of the export cell; paragraphs.csv and sentences.csv must not be newer than the .ipynb (otherwise the outputs of the last run were never saved — typical of running the notebook in VS Code without saving it); and the page openings printed by the notebook must occur in the source file (compared as a stream of letters, so joined hyphenation, removed page numbers and running headers do not matter). When the content cannot be compared — scans read by OCR, images, URLs, a garbled PDF text layer — the first two checks decide and the console says so. The report also records whether omorfi took part in the run (<meta name="ata-omorfi">); the Book of the Unsaid reads it.

For a structured, full diagnostic view built straight from the CSV/JSON exports:

python generate_diagnostic.py
# → export_results/<project>/diagnostic_report.html

generate_diagnostic.py reads only the exported CSV/JSON, so — unlike generate_report.py — it does not need a freshly-executed notebook in memory, only the exports the notebook wrote. The result is a navigable document (table of contents, headings, lists) covering topics, theses, entities grouped by type, and top keywords.

3. Gemini Notebook-friendly Markdown (optional)

python generate_md.py
# → export_results/<project>/gemini_notebook_report.md

This converts the HTML report into a Markdown file with the accessibility spans unwrapped (Gemini Notebook — formerly NotebookLM — does not consume <span lang="…">). Run this only if you want to feed the report into Gemini Notebook.

4. Shamanic non-LLM post-processor (optional)

python shamanic_pipeline.py
# → export_results/<project>/audio_scripts/oracle_script.txt
#                                          /raw_roots_chant.txt
#                                          /prophecies.txt
#                                          /lore_fragments/intercepted_log_T*_P*.txt
#                                          /emotional_undertow.txt   # only if sentiment.csv exists

Generates ritual text artefacts directly from the notebook's CSV/JSON exports — no LLM call, no network. All strings come from shamanic_locale.py and are fully localized across the six supported languages. When the optional sentiment.csv is present, it also produces emotional_undertow.txt — a per-paragraph "mood tide" plus a balance summary; without it, that one ritual is simply skipped.

5. Shamanic LLM post-processor (optional, requires an LLM API key)

python shamanic_ai.py
# → export_results/<project>/audio_scripts/katla_entity_monologue.txt
#                                          /vieno_echoes_chant.txt
#                                          /lumi_final_report.txt
#                                          /sami_energetic_spark.txt

This requires the key of the chosen provider (llm_provider, default openai). Create golden_key.env in the project root with the key you need:

OPENAI_API_KEY=sk-...          # llm_provider: openai (default)
OPENROUTER_API_KEY=sk-or-...   # llm_provider: openrouter
LLM_COMPAT_API_KEY=...         # llm_provider: openai_compat (llm_base_url)

golden_key.env matches *.env in .gitignore, so it will not be committed. The four voices run in sequence:

1. Katla transmutes the entity list (entities.csv) into a monologue of frozen Northern spirits.
2. Vieno chants over the keyword/topic list. By default she weaves in five raw sentences from the corpus as foreign-dimension echoes; when the optional sentiment.csv is present she instead reads the text's emotional arc (per-paragraph moods, downsampled to fit the model's token limit on long corpora) and lets it shape the chant's dynamics.
3. Lumi reads prophecies.txt (mandatory) plus Katla's monologue and Vieno's chant (optional ornament) and produces the final dispatch. Localized fallback strings cover the case where Katla or Vieno are missing.
4. Sami reads Lumi's report and delivers a high-energy synthesis with the spark of hope or call to action.

6. Shamanic audio dispatcher (optional, requires ElevenLabs)

python shamanic_voice.py
# → export_results/<project>/audio_scripts/katla_entity_monologue.mp3
#                                          /vieno_echoes_chant.mp3
#                                          /lumi_final_report.mp3
#                                          /sami_energetic_spark.mp3

A standalone, run-once script that reads the four narrative artefacts produced by shamanic_ai.py and synthesizes each into an .mp3 via the ElevenLabs eleven_multilingual_v2 model (which auto-detects the spoken language, so no language parameter is passed). It logs each success or failure to the console and exits — there is no bot, no server, no interactive session. It needs two things:

• ELEVENLABS_API_KEY in golden_key.env (the same file that holds the LLM keys).
• a voices mapping in config.json / config.ini that assigns an ElevenLabs voice ID to each of katla, vieno, lumi, sami. A voice whose ID is empty or whose artefact is missing is skipped (run shamanic_ai.py first); one voice failing does not abort the others.

Privacy & cost. Like shamanic_ai.py with OpenAI, this script sends artefact text to a third-party service (ElevenLabs) for synthesis, and ElevenLabs is a paid API. The content leaves your machine — do not run it on material you cannot share externally.

7. The Book of the Unsaid (optional, Finnish only, agent-assisted)

python ksiega_niedomowien.py scan              # which export_results/ folders qualify
python ksiega_niedomowien.py scan --blocks     # also blocks done / open / left (slow; automatic with --folder)
python ksiega_niedomowien.py status            # progress per folder: entries, obvious blocks, open packets
python ksiega_niedomowien.py tidy              # entries replaced by a newer entry of the same block (preview; --apply cleans up)
python ksiega_niedomowien.py prepare --folder <name> --limit 3
#   → export_results/_kirja_tyotila/<id>/packet.json
python ksiega_niedomowien.py draft <id>        # optional: draft via API + seal, up to 2 retries
python ksiega_niedomowien.py seal <id>         # validates draft.json, writes blind.json
python ksiega_niedomowien.py fill <id>         # optional: blind guesses via API, then judge
python ksiega_niedomowien.py judge <id>        # → export_results/Sanomattomien_kirja.md
# <id> = the packet id printed by prepare, the start of the folder name, or the packet folder path

A game built on raw_roots_chant.txt. For each block of 3–6 consecutive roots an LLM writes a misty Finnish diary entry with gaps whose answers are verbatim spans of the source text — they must occur in both accessible_text.html and accessible_text.docx, the two versions of the source of truth. A separate, blind model then tries to fill the gaps from the entry alone; every gap it guesses counts as obvious and its fragment is removed. What remains — KATLA (the warning system) reading the roots and VIENO (the shamanic narrator) with the unsaid gaps — is appended to export_results/Sanomattomien_kirja.md. The answer key goes to Sanomattomien_kirja.avain.json and the state to Sanomattomien_kirja.tila.json; re-running never duplicates entries.

The script is the deterministic half (filters, blocks, source of truth, validation, judging, the Book). The creative half — writing draft.json — is done either by an agent following skills/ksiega-niedomowien/SKILL.md or by an API model through draft.

Input filters. Only export_results/ subfolders with a Finnish name, <html lang="fi">, a Finnish chant and no grammar leak qualify; anything else is refused with a reason. The roots.header line is stripped when present. The grammar leak check catches corpora processed before omorfi was enabled (v1.5.0): over the full keywords_tfidf.csv it counts inflected forms that survived lemmatization plus spaCy's garbage lemmas, and rejects the folder above 1.0 per 100 terms. Requires omorfi (Installation step 5). When the folder has a current analysis_report.html, a refusal for a grammar leak says what the report knows: omorfi took part in the run (then the engine itself — the Finnish lemma correction in the notebook — may need fixing; report the examples), omorfi was missing (install it and re-run the notebook), or the run had no omorfi step at all. Without a current report it says that this is unknown.

Why Finnish only. The game lives on what lemmatization leaves unsaid, so it only makes sense where reducing a word to its lemma removes real meaning. Finnish is agglutinative: what Polish or Russian express partly with separate prepositions, and English mostly with word order and function words, Finnish packs into the word itself — fifteen cases (talossa "in the house", talosta "out of the house", taloon "into the house", talolle "to the house", talotta "without a house"), possessive suffixes (taloni "my house", talonsa "his/her/their house"), clitics (-ko question, -kin "also", -han emphasis) and verb forms carrying person, tense, mood and the passive at once (tehtäisiinkö "would it be done?"). The chant reduces all of that to TALO and TEHDÄ: who, where, from where, to where, whose and whether all vanish — exactly the gaps a listener has to fill. In Polish and Russian the lemma drops the inflectional ending, but much of the relation sits in a separate word (the preposition), so the lemma never carried it alone; English loses little beyond number and tense. Icelandic (four cases, largely governed by prepositions) and Italian (prepositions and articles, little nominal inflection) sit closer to the Polish/Russian and the English patterns respectively. Technically the tool also needs a dictionary analyser to prove that a chant is grammar-free and to match roots back to inflected text; in this pipeline only Finnish has one (omorfi), and the thresholds were calibrated on real Finnish corpora.

API drafter (draft). Lowers the entry threshold: no agent is needed. The script sends the packet (the chant, the roots and the source sentences from truth.sentences) to the configured provider, writes draft.json and seals it; when seal rejects the draft, the model gets it back together with the validator's concrete objections, up to 2 retries (3 attempts). Exit codes match seal (0 sealed, 4 still rejected — fix the last draft.json by hand or with the skill). An existing draft.json is never overwritten without --force. The skill remains the stronger mode: unlimited retries, with the agent reading the objections as it goes. One rule cannot be validated deterministically in either mode: do not attribute to real people facts or accusations that are not in the source sentences. The drafter prompt and the skill both state it, but nothing checks it — read entries about real people before you share them.

Blind filler (fill). Sends only blind.json — never the answers or the source — to the configured provider, saves guesses.json and runs judge right away. Provider: kirja_llm_provider (empty = llm_provider); model: kirja_filler_model; --provider / --model override both. judge on its own is meant for guesses from a blind subagent. Both draft and fill ask for a JSON schema (structured outputs) first; when the endpoint rejects it they fall back to plain JSON mode, then to the prompt alone. fill retries twice when the reply is unusable or misses a gap. Every unusable reply is saved in the packet folder (draft_reply_N.txt / fill_reply_N.txt, with finish_reason and any refusal), and the console says where.

Connecting the skill.

• Claude Code: copy skills/ksiega-niedomowien/ to ~/.claude/skills/ (personal) or to .claude/skills/ in your checkout. This repo gitignores .claude/, which is why the skill ships under skills/.
• Claude apps: zip the ksiega-niedomowien/ folder (the folder itself must be the archive root) and add it under Customize → Skills; code execution must be enabled, and the session needs access to your checkout, because the omorfi automata and export_results/ live on your disk. Use the Code tab with a local session: in the plain chat mode or in a cloud session, the files the agent reads — source excerpts included — are copied to a cloud environment.
• Other agents: SKILL.md is a plain Markdown procedure with YAML front matter; any agent that can run local commands in the repo can follow it. If it cannot spawn an independent subagent, use fill for the blind step.
• A chat-only assistant without local command execution (e.g. a custom GPT) cannot run it: omorfi and its ~130 MB of automata have to run next to your data.

Privacy & cost. Three levels, from the least to the most data: fill sends only the entry's misty text (not the source); draft sends verbatim source sentences (truth.sentences) and prints how many before the call; an agent following the skill reads source excerpts and sends them to its own model provider — and, in a cloud session, works on a cloud copy of the files. All API calls are paid. Use them only on material you can share with those providers.

Output

Every analyzed corpus gets its own subdirectory under export_results/, named after the source file (path and extension stripped) or after a domain_slug for URLs. The built-in example corpus uses export_results/_default/.

Each subdirectory contains:

File	Produced by	Description
sentences.csv	notebook	every sentence with its index and paragraph assignment
paragraphs.csv	notebook	paragraphs (3–6 sentences each)
theses.csv, theses.txt	notebook	one thesis per paragraph (highest-TF-IDF sentence); the .txt filename follows the corpus language (e.g. tezy.txt for Polish, тезисы.txt for Russian)
keywords_tfidf.csv	notebook	keywords (uni/bi/trigrams) with TF-IDF weights
paragraphs_with_topics.csv	notebook	paragraphs with their assigned KMeans topic
topic_keywords.json	notebook	keywords per topic
entities.csv	notebook	all named entities and their labels
sentiment.csv	notebook (optional)	per-paragraph sentiment (para_id, label, score, lang) — only when enable_sentiment is true; feed for the shamanic layer, never shown to the user
accessible_text.html	notebook	paragraph- and sentence-level lang attributes — screen readers switch voice automatically per fragment
accessible_text.docx	notebook	the same content with <w:lang> set per Run (pl-PL, ru-RU, en-US, it-IT, fi-FI, is-IS) — Word and SAPI use it offline, no online detector required
analysis_report.html	generate_report.py	global accessible HTML report
diagnostic_report.html	generate_diagnostic.py	structured diagnostic report (topics, theses, entities by type, keywords) from the CSV/JSON exports
gemini_notebook_report.md	generate_md.py	Gemini Notebook-ready Markdown (formerly NotebookLM)
audio_scripts/*.txt	shamanic_pipeline.py, shamanic_ai.py	ritual / narrative text artefacts
audio_scripts/*.mp3	shamanic_voice.py (optional)	the four narrative voices synthesized to audio via ElevenLabs

Accessibility

This is the project's central value. Everything below is intentional and must be preserved when the pipeline is modified:

• No ANSI colors, emoji, or pseudographics in stdout. A screen reader reads every character literally — [OK] instead of "🟢", --- instead of "───━━━".
• Silenced progress bars (tqdm, transformers, torch) — cell_model._silence_hf_progress() mutes Hugging Face/torch loading bars and warnings before pulling IceBERT and MIM-GOLD-22 for Icelandic; cell_corpus._silence_ocr_progress() does the same for easyocr.
• Per-paragraph and per-sentence lang tagging in the export. accessible_text.html and accessible_text.docx carry an ISO 639-1 code on every paragraph (and on every sentence inside mixed-language paragraphs / runs). NVDA, JAWS, Narrator, VoiceOver, Word, and SAPI all honour those tags and switch voice automatically — even offline.
• Hardcoded English tagging in the HTML report. generate_report.py always wraps technical English fragments — POS tags (NOUN, VERB, ADJ), NER labels (PER, ORG, [orgName]), spaCy / Hugging Face model identifiers, ASCII filenames, code blocks — in <span lang="en">, so they are no longer pronounced through the document-default Russian voice.
• Per-fragment foreign-corpus tagging in HTML. When the corpus is non-Russian, structured outputs (sentence excerpts, keyword lists, topic words, RAG ranking rows, lemmatization tables) are wrapped in <span lang="target_lang">.
• Linear HTML structure (<main>, proper heading hierarchy).

Reading the output as a non-Russian user

The notebook narrative and the print() output of most cells are written in Russian. Two characteristics of the Jupyter UI make that hostile to non-Russian screen-reader users, and the pipeline ships specific workarounds for each:

• Jupyter in a browser hardcodes lang="en" on the document, and the "Translate this page?" prompt never fires on localhost pages anyway. A Russian-narrated notebook is therefore announced in an English TTS voice — total chaos. The pragmatic mitigation is the VS Code Jupyter extension: its list-view announces only code cell / markdown cell labels with no ISO tagging, you can press Enter through code cells and ↓ past markdown cells without listening to their bodies, and the hardcoded-English problem is confined to the output view (Ctrl+Shift+↓) — which you can choose not to enter.
• generate_report.py is effectively mandatory for non-Russian readers. It produces analysis_report.html with <html lang="ru"> and the full set of per-fragment <span lang="…"> tags. Opened in a real browser (not from inside Jupyter), it triggers the browser's "Translate this page?" prompt for the Russian narration while still switching TTS voice per fragment for corpus content and for technical English terms (POS, NER, model names).
• Interactive Q&A writes qa_results.html and opens it for you. cell_qa_rag still prints the query and top-3 matches to stdout — but it also writes the same content to export_results/<project>/qa_results.html and calls webbrowser.open() so the page lands in the system browser, where translation and per-fragment voice switching both work. The output view in Jupyter, where the hardcoded lang="en" makes the Russian wrappers unreadable to a screen reader, is therefore no longer the only path to the result. Each new query overwrites the file.

Known limitations

• Loanword origins are not tagged. Language tagging (the lang attributes, see Language detection) covers only fragments written in a foreign script: Greek letters, Latin inside a Cyrillic text. An explicit origin marker (греч., łac., z niemieckiego, имеет греческое происхождение) is not linked to the borrowed word, and a word transliterated into the host script (фразис in a Russian text) keeps the corpus language. Such markers have no fixed form (abbreviation, adjective, a whole phrase), and tagging a Cyrillic фразис as Greek would make the screen reader switch to a Greek voice, which is worse than the Russian default.
• Finnish: the first word of a sentence. At the start of a sentence every word is capitalized, and the spaCy tagger sometimes takes an imperative or an interjection for a proper name: Anna kuulua. ("let it be heard") becomes the name Anna, Oota, miten… ("wait, how…") the name Oota. The omorfi correction never overrides a word tagged as a name, because otherwise real names would turn into common words (Tuskia → tuska, "anguish"). Such a word keeps its written form as its lemma and can show up in lemma lists, keywords and the root chant (anna, oota). When the tagger calls the word an interjection instead, omorfi supplies the verb (Oota → oottaa, since v1.11.1). The Book of the Unsaid's grammar-leak check does not count a keyword that the source also uses as a name in mid-sentence (Anna Maria …).

Repository layout

accessible_text_analyst/
├── accessible_text_analyst.ipynb   # main pipeline (42 cells)
├── generate_report.py              # HTML report generator (reader view)
├── generate_diagnostic.py          # diagnostic HTML report from CSV/JSON exports
├── generate_md.py                  # Gemini Notebook Markdown converter
├── shamanic_pipeline.py            # optional: non-LLM ritual post-processor
├── shamanic_ai.py                  # optional: LLM-driven ritual narrators
├── llm_klient.py                   # LLM provider and key selection (shared)
├── shamanic_voice.py               # optional: ElevenLabs audio dispatcher (run-once)
├── shamanic_locale.py              # localization bundle for the shamanic layer
├── ksiega_niedomowien.py           # optional: the Book of the Unsaid (Finnish only)
├── skills/ksiega-niedomowien/SKILL.md  # agent skill for ksiega_niedomowien.py
├── config.example.json             # config template, JSON extension (versioned)
├── config.example.ini              # config template, INI extension (versioned)
├── config.json / config.ini        # your local configuration (gitignored)
├── golden_key.env                  # LLM and ElevenLabs keys (gitignored)
├── requirements.txt                # pip dependencies
├── CLAUDE.md                       # guide for Claude Code
├── README.md                       # this file (canonical, English)
├── README_pl.md, README_ru.md, …   # translations (link back at the top)
├── release_notes.md                # release notes appended in reverse-chrono order
└── export_results/                 # analysis outputs (gitignored)
    ├── Sanomattomien_kirja.md      # the Book of the Unsaid, appended incrementally
    └── <project_name>/
        ├── sentences.csv
        ├── paragraphs.csv
        ├── theses.csv
        ├── theses.txt              # name follows corpus language
        ├── accessible_text.html
        ├── accessible_text.docx
        ├── analysis_report.html
        ├── diagnostic_report.html
        ├── gemini_notebook_report.md
        └── audio_scripts/…

License

Released under the MIT License. The pipeline is distributed as source only — no binaries — and the notebook is, by its nature, editable cell-by-cell through the browser or the VS Code Jupyter extension. The MIT license simply makes explicit what the format already implies.