kitaru-ai-newsletter

---

An AI-powered newsletter generation workflow built with Kitaru, Gemini, MCP, and Firecrawl.

Kitaru AI Newsletter is an automated AI-news curation and newsletter generation system.

The project collects AI-related information, uses specialized research stages to identify important stories and resources, generates a structured newsletter, reviews and evaluates the generated content, asks for human approval, and finally converts the approved Markdown newsletter into HTML.

The workflow is orchestrated using Kitaru checkpoints, allowing individual stages of the workflow to be tracked, cached, retried, run concurrently, and composed into a durable pipeline — with a durable human-in-the-loop approval gate sitting between AI generation and publishing.

---

---

Features

• Automated AI news research
• Big-story selection
• Quick AI updates
• AI research paper discovery
• GitHub repository discovery
• AI tutorial generation
• AI product discovery
• AI social-post research
• AI newsletter editing
• Automated newsletter review
• Newsletter quality evaluation
• Durable human-in-the-loop approval via kitaru.wait(), resumable via CLI even if the process is killed
• Markdown → HTML conversion
• MCP-based external tool integration
• Local newsletter and artifact storage (both local files and Kitaru's own artifact store)
• Kitaru checkpoint-based workflow execution with caching and automatic retries
• Concurrent research fan-out via .submit()

---

Architecture

The system is organized as a research → editorial → approval → publishing pipeline.

flowchart TB

    USER["User / CLI"]

    FLOW["Kitaru Flow<br/>newsletter()"]

    PLAN["Research Planning<br/>create_research_plan (checkpoint)"]

    MCP["MCP Integration Layer"]

    FIRECRAWL["Firecrawl MCP Server"]

    FIRECRAWL_API["Firecrawl API"]

    NEWSCP["research_news (checkpoint)"]

    BIG["research_big_story (submit)"]
    QUICK["research_quick_updates (submit)"]
    PAPERS["research_papers (submit)"]
    GITHUB["research_github (submit)"]
    PRODUCTS["research_products (submit)"]
    XPOSTS["research_x (submit)"]
    TUTORIAL["research_tutorial (checkpoint)"]

    MERGE["Merged Research Data"]

    EDIT["edit_newsletter (checkpoint)"]

    REVIEW["review_newsletter (checkpoint)"]

    EVAL["evaluate_newsletter (checkpoint)"]

    WAIT["kitaru.wait()<br/>flow-scope approval gate"]

    PUBLISH["publish_newsletter (checkpoint)"]

    MD["newsletter.md"]

    HTML["newsletter.html"]

    USER --> FLOW

    FLOW --> PLAN

    PLAN --> MCP

    MCP --> FIRECRAWL
    FIRECRAWL --> FIRECRAWL_API
    FIRECRAWL_API --> NEWSCP

    NEWSCP --> BIG
    NEWSCP --> QUICK
    NEWSCP -.independent, submitted concurrently.-> PAPERS
    NEWSCP -.independent, submitted concurrently.-> GITHUB
    NEWSCP -.independent, submitted concurrently.-> PRODUCTS
    NEWSCP -.independent, submitted concurrently.-> XPOSTS

    BIG --> TUTORIAL
    QUICK --> TUTORIAL
    PAPERS --> TUTORIAL
    GITHUB --> TUTORIAL

    BIG --> MERGE
    QUICK --> MERGE
    PAPERS --> MERGE
    GITHUB --> MERGE
    TUTORIAL --> MERGE
    PRODUCTS --> MERGE
    XPOSTS --> MERGE

    MERGE --> EDIT

    EDIT --> MD
    EDIT --> REVIEW

    REVIEW --> EVAL

    EVAL --> WAIT

    WAIT -->|Approved| PUBLISH
    WAIT -->|Rejected| STOP["Workflow stopped"]

    PUBLISH --> HTML

Note: research_big_story and research_quick_updates depend on research_news; research_papers, research_github, research_products, and research_x are independent of research_news and of each other. All six are dispatched with .submit() and awaited with .result().load() before research_tutorial (which depends on all of them) runs.

---

How the Workflow Works

Research Plan
      ↓
News Collection (checkpoint)
      ↓
Research Fan-Out (.submit() — 6 concurrent checkpoints)
      ↓
Tutorial Generation (depends on fan-out results)
      ↓
Merge Research
      ↓
Newsletter Generation
      ↓
Review
      ↓
Evaluation
      ↓
kitaru.wait() — Human Approval (flow scope, durable, resumable)
      ↓
Markdown → HTML
      ↓
Published Newsletter

---

1. Research Planning

plan = create_research_plan()

@checkpoint(retries=2). Separates planning from research execution.

---

2. Research Collection

Previously, the raw Firecrawl call sat directly in the flow body — outside any checkpoint boundary, meaning it couldn't be retried, cached, or replayed independently. It is now wrapped as its own checkpoint:

news = research_news(plan)

Kitaru Checkpoint (retries=2)
       ↓
MCP Integration
       ↓
Firecrawl MCP Server
       ↓
Firecrawl API
       ↓
Web Research

---

3. Research Sections — Concurrent Fan-Out

The six independent research sources are dispatched concurrently with .submit() instead of one after another:

big_story_f = research_big_story.submit(news)
quick_updates_f = research_quick_updates.submit(news)
papers_f = research_papers.submit("Artificial Intelligence")
github_f = research_github.submit("Artificial Intelligence")
products_f = research_products.submit("AI Artificial Intelligence")
x_posts_f = research_x.submit("latest AI artificial intelligence")

.submit() returns a future. .result() on that future returns a checkpoint output handle, not the materialized value — .load() is required to get the concrete object:

big_story = big_story_f.result().load()
quick_updates = quick_updates_f.result().load()
papers = papers_f.result().load()
github = github_f.result().load()
products = products_f.result().load()
x_posts = x_posts_f.result().load()

Each of the six checkpoints still runs with @checkpoint(retries=2) — concurrency doesn't trade away durability.

Tutorial depends on the fan-out results, so it runs after they resolve:

tutorial = research_tutorial({
    "big_story": big_story,
    "quick_updates": quick_updates,
    "research_papers": papers,
    "github_repositories": github,
})

Observed local-runner behavior: on Kitaru's local default stack (Runner: default (local)), wall-clock overlap between submitted checkpoints was limited in testing — timestamps showed near-sequential execution rather than full parallelism. The .submit()/.result() pattern used here matches Kitaru's documented API exactly; the limited speedup appears to be a property of the local single-machine runner rather than a code issue, and the same code should parallelize further on stacks with multi-worker or isolated-step support.

---

4. Merge Research

research = {
    "big_story": big_story,
    "quick_updates": quick_updates,
    "research_papers": papers,
    "github_repositories": github,
    "tutorial": tutorial,
    "ai_products": products,
    "x_posts": x_posts,
}

(Values here are already materialized via .load() above — no further loading needed before passing into edit_newsletter.)

---

5. Newsletter Generation

draft = edit_newsletter(research)

@checkpoint(retries=2). Inside this checkpoint, the merged research bundle is also persisted through Kitaru's own artifact store (not just the local .md file):

kitaru.save("research_bundle", research_data, type="output")
kitaru.save("newsletter_draft_raw", newsletter, type="output")

kitaru.save() can only be called from inside a @checkpoint — calling it from flow scope raises KitaruContextError. Valid type= values are blob, context, input, output, prompt, response (not arbitrary strings).

---

6. Newsletter Review

reviewed = review_newsletter(draft)

---

7. Newsletter Evaluation

evaluation = evaluate_newsletter(reviewed).load()

The .load() here is required because the result is consumed directly in flow scope (printed, sliced) rather than being passed straight into another checkpoint.

---

8. Human Approval — kitaru.wait()

Earlier versions of this project used a @checkpoint that called Python's input() for approval. This does not use Kitaru's actual human-in-the-loop primitive, and kitaru.wait() is explicitly rejected inside @checkpoint bodies — waits must be called at flow scope, since a pause signal can't pass cleanly through a checkpoint step.

The approval logic is now a plain function called directly from the flow body (not a checkpoint):

approved = request_approval(evaluation)

approved = kitaru.wait(
    name="newsletter_approval",
    schema=bool,
    question="Approve this newsletter for publishing?",
    timeout=3600,
    metadata={"evaluation_preview": evaluation[:280]},
)

On Windows, kitaru's local daemon is not supported, so interactive in-terminal prompting for wait() isn't available — the execution instead pauses in a waiting state and must be resolved from a second terminal:

kitaru executions input <exec_id> --value true

The running process automatically detects the resolved input via polling and resumes without needing a separate kitaru executions resume call. This was verified live: pausing an execution, resolving it from a second terminal, and confirming the first process resumed and completed cleanly.

If rejected, publishing is skipped.

---

9. Newsletter Publishing

html = publish_newsletter(reviewed)

Markdown → HTML conversion is done locally (not via another LLM call) — deterministic, faster, cheaper, and independent of Gemini availability at publish time. The HTML is also persisted through Kitaru's artifact store:

kitaru.save("published_newsletter_html", full_html, type="output")

---

Kitaru Usage

Kitaru is used as the actual workflow orchestration and durability layer, not just imported for @flow/@checkpoint scaffolding.

Kitaru Capability	Used	Implementation
@flow	✅	newsletter()
@checkpoint	✅	Every research/editorial/publishing stage
@checkpoint(retries=...)	✅	All external-API-calling checkpoints; verified live against a real Gemini 503 and a real 429 quota error
Checkpoint caching	✅	Verified across runs (create_research_plan, research_news reused cached results)
Checkpoint output handles + .load()	✅	Used consistently wherever a checkpoint output is consumed in flow scope rather than passed to another checkpoint
.submit() / .result() concurrency	✅	Six independent research checkpoints dispatched concurrently
kitaru.wait()	✅	Human approval gate, called at flow scope (not inside a checkpoint)
kitaru.save()	✅	Research bundle, newsletter draft, and published HTML persisted as tracked artifacts
kitaru.log()	✅	Flow-level stage tracking and per-checkpoint metadata (item counts, draft length, LLM latency/token counts)
Typed checkpoint outputs	✅	Annotated[...]
Human approval, durable	✅	kitaru.wait() — resumable via kitaru executions input / CLI even across a killed process
Local execution	✅	Tested locally on Windows (with the daemon/interactive-prompt caveat noted above)
MCP integration	✅	Firecrawl integration
kitaru.llm()	⚠️ Not used	kitaru.llm() only supports openai/*, anthropic/*, ollama/*, and openrouter/* model providers — Gemini isn't natively supported. LLM calls go through the Gemini SDK directly, with a manual kitaru.log() wrapper recording latency/token usage as the closest available substitute. Routing through openrouter/google/gemini-* would enable native kitaru.llm() tracking if desired.

---

MCP Integration

Kitaru
   ↓
MCP Integration
   ↓
Firecrawl MCP Server
   ↓
Firecrawl API
   ↓
Web Research

Potential future integrations (not currently implemented): GitHub, arXiv, X, Google Drive, Gmail.

---

Why Kitaru?

A conventional Python script could execute the same functions sequentially, but this project uses Kitaru specifically for:

• Durability — checkpoints can be retried automatically (retries=) without re-running the whole pipeline, and results are cached across runs.
• Concurrency — independent research sources run via .submit() instead of blocking one after another.
• A real pause/resume boundary — kitaru.wait() lets a human gate publishing without keeping a process blocked forever or losing state if it's killed.
• Observability — kitaru.log() and kitaru.save() make intermediate reasoning and data inspectable per-execution via the CLI, instead of scattered print()/local files only.

---

Technology Stack

Technology	Version / Status	Purpose
Python	3.12	Application runtime
Kitaru	0.21.0	Workflow orchestration
MCP Python SDK	via mcp[cli]	MCP integration
Markdown	—	Markdown → HTML
Gemini	gemini-flash-latest (configurable via GEMINI_MODEL)	AI reasoning and generation
Google GenAI SDK	Installed	Gemini API client
Firecrawl MCP	npx -y firecrawl-mcp	Web research
ZenML	Kitaru dependency	Pipeline execution underneath Kitaru

---

Project Structure

kitaru-ai-newsletter/
│
├── workflows/
│   └── newsletter.py
│
├── tasks/
│   ├── planning.py
│   ├── research_news.py           # wraps the Firecrawl call as a checkpoint
│   ├── research_big_story.py
│   ├── research_quick_updates.py
│   ├── research_papers.py
│   ├── research_github.py
│   ├── research_tutorial.py
│   ├── research_products.py
│   ├── research_x.py
│   ├── editing.py
│   ├── reviewing.py
│   ├── evaluation.py
│   ├── human_approval.py          # kitaru.wait(), NOT a @checkpoint
│   └── publisher.py
│
├── agents/
│   ├── big_story.py
│   ├── quick_updates.py
│   ├── editor.py
│   ├── reviewer.py
│   ├── evaluator.py
│   ├── tutorial.py
│   └── publisher.py
│
├── integrations/
│   ├── firecrawl.py
│   └── filesystem.py
│
├── utils/
│   └── gemini.py                  # manual kitaru.log() usage tracking
│
├── models.py
├── config.py
├── prompts/
│
├── outputs/
├── tests/
│
├── .env
├── pyproject.toml
└── README.md

---

Human-in-the-Loop Design

AI Generation
      ↓
AI Review
      ↓
AI Evaluation
      ↓
kitaru.wait() — Human Decision
      ↓
Publishing

The system does not directly publish AI-generated content. The approval gate is a genuine Kitaru wait(), not a blocking input() call — meaning the pending decision is a durable execution state, inspectable and resolvable via kitaru executions list / kitaru executions input even if the original process is gone.

---

Error Handling and Development

During development, retries=2 was verified against two different real external failures without any code changes needed to recover:

• A transient Gemini 503 UNAVAILABLE ("model experiencing high demand") — recovered on the second attempt.
• A Gemini free-tier 429 RESOURCE_EXHAUSTED quota error — correctly retried twice, then surfaced a clean failure after exhausting retries=2, rather than crashing the whole pipeline unpredictably.

Note: Kitaru's retries= does not currently honor the API's suggested Retry-After delay — it retries immediately (Sleeping for 0 seconds) rather than backing off, which is worth knowing when debugging rate-limit-driven failures specifically.

Because the workflow is separated into checkpoints, previously completed stages remain cached while later stages are iterated on and re-run.

---

Running the Project

.venv\Scripts\activate
kitaru init      # first time only
python -m workflows.newsletter

The workflow runs through research, editing, review, and evaluation, then pauses at the kitaru.wait() approval gate. On Windows, resolve it from a second terminal:

kitaru executions input <exec_id> --value true

---

Environment Variables

GEMINI_API_KEY=your_gemini_api_key
GEMINI_MODEL=gemini-flash-latest   # optional override
FIRECRAWL_API_KEY=your_firecrawl_api_key

Never commit real API keys.

---

Outputs

newsletter.md
newsletter_reviewed.md
evaluation_report.md
newsletter.html

Plus tracked artifacts inside Kitaru's own artifact store (research_bundle, newsletter_draft_raw, published_newsletter_html), inspectable via:

kitaru executions get <exec_id>
kitaru executions logs <exec_id>

---

Known Platform Limitations

• Windows daemon: Kitaru's background daemon isn't supported on Windows (Kitaru ! Daemon functionality is currently not supported on Windows.). This disables interactive in-terminal wait() prompting; wait() still works, but must be resolved via kitaru executions input from a second terminal/session.
• Local runner concurrency: the default local stack's Runner: default (local) showed limited real overlap between .submit()-dispatched checkpoints in testing. The code is written correctly per Kitaru's documented concurrency API; true parallel speedup likely requires a stack with multi-worker/isolated-step support.
• kitaru.llm() provider support: native tracked LLM calls are limited to openai/*, anthropic/*, ollama/*, openrouter/*. Gemini is called directly via the Google GenAI SDK with manual kitaru.log()-based usage tracking as a substitute.

---

Future Improvements

MCP Integrations

• GitHub MCP integration
• arXiv MCP integration
• X MCP integration

Newsletter Quality

• Stronger fact-checking to reduce hallucinated sources/model names in generated content
• Structural validation that all required newsletter sections are present before review, with automatic re-prompt if any are missing

Infrastructure

• Deploy on a stack with genuine multi-worker concurrency to validate .submit() speedup
• Scheduled newsletter generation
• Email delivery

---

Design Philosophy

1. Separate responsibilities — Research ≠ Editing ≠ Review ≠ Evaluation ≠ Publishing.
2. Make workflow stages explicit — Kitaru checkpoints provide clear, durable boundaries.
3. Keep deterministic operations deterministic — Markdown → HTML doesn't need an LLM.
4. Keep humans in the loop — via a real durable wait(), not a blocking script prompt.
5. Keep external tools modular — MCP as the integration layer.

---

Example Workflow Code

@flow
def newsletter():

    kitaru.log(stage="started")

    plan = create_research_plan()

    news = research_news(plan)

    big_story_f = research_big_story.submit(news)
    quick_updates_f = research_quick_updates.submit(news)
    papers_f = research_papers.submit("Artificial Intelligence")
    github_f = research_github.submit("Artificial Intelligence")
    products_f = research_products.submit("AI Artificial Intelligence")
    x_posts_f = research_x.submit("latest AI artificial intelligence")

    big_story = big_story_f.result().load()
    quick_updates = quick_updates_f.result().load()
    papers = papers_f.result().load()
    github = github_f.result().load()
    products = products_f.result().load()
    x_posts = x_posts_f.result().load()

    tutorial = research_tutorial({
        "big_story": big_story,
        "quick_updates": quick_updates,
        "research_papers": papers,
        "github_repositories": github,
    })

    research = {
        "big_story": big_story,
        "quick_updates": quick_updates,
        "research_papers": papers,
        "github_repositories": github,
        "tutorial": tutorial,
        "ai_products": products,
        "x_posts": x_posts,
    }

    draft = edit_newsletter(research)

    reviewed = review_newsletter(draft)

    evaluation = evaluate_newsletter(reviewed).load()

    approved = request_approval(evaluation)   # calls kitaru.wait() internally

    if approved:
        return publish_newsletter(reviewed)

    return "Newsletter rejected."

---

What Makes This Project Interesting

The project demonstrates an AI application structured as a durable, checkpointed, concurrent, human-gated workflow rather than a single prompt-to-output script:

Plan
 ↓
Research (checkpointed, retried)
 ↓
Specialized AI Selection (concurrent via .submit())
 ↓
Structured Research
 ↓
Editorial Generation
 ↓
Review
 ↓
Evaluation
 ↓
kitaru.wait() — durable Human Approval
 ↓
Publishing

Kitaru provides durability, concurrency, and the human-approval boundary; MCP provides standardized external tool connectivity; Gemini provides AI reasoning and generation.

---

License

MIT License

---

Acknowledgements

Built using:

• Kitaru
• ZenML
• Model Context Protocol (MCP)
• Google Gemini
• Firecrawl
• Python

---

Author

Pratham Harer

Medium Blog (Detailed Project Explanation) Read the full case study here