calibrated-copilot

Calibrated Copilot

A Gemini 3 financial agent that knows how sure it should be — calibrated, risk-adjusted answers instead of confident hallucinations.

Built for the Google Cloud Rapid Agent Hackathon (June 11, 2026). A Gemini 3 agent (gemini-3.1-pro-preview-customtools) built with the Google Agent Development Kit (ADK) that integrates the MongoDB Atlas MCP server and gives calibrated, risk-adjusted financial guidance.

Built by Ruiyang Zhang (passed all three CFA Program exams) — calibration & risk methodology is my professional focus. github.com/ryonzhang · ruiyang.co

---

The problem

The #1 failure of AI in finance is overconfidence — models state precise numbers they can't back, with the same false confidence whether the data is rich or thin. In finance, an overconfident wrong answer is worse than no answer.

The idea

Calibrated Copilot does the opposite. For any portfolio question or trade idea it returns:

1. A calibrated probability with explicit confidence — and states the basis for it.
2. A risk-adjusted recommendation — position sizing proportional to confidence (fractional-Kelly intuition), with an exposure / drawdown sanity check.
3. An honest abstention — "insufficient or conflicting evidence — I won't give a confident call" — when data is missing or signals disagree.
4. Citations for every number it used. It never states a confident number it didn't retrieve.

Trust through calibration, not bravado.

---

Architecture

                ┌─────────────────────────────────────────────┐
   User  ─────► │             Gemini 3 agent (ADK)            │
   question     │   system prompt enforces calibration:       │
                │   • retrieve before claiming                │
                │   • probability + explicit confidence       │
                │   • abstain on thin / conflicting data      │
                └───────────────┬─────────────────┬───────────┘
                                │                 │
                  ┌─────────────▼──────┐   ┌──────▼─────────────────┐
                  │  MongoDB MCP tool  │   │  deterministic tools   │
                  │  (Atlas)           │   │  • risk_tool:          │
                  │  • portfolio       │   │    fractional-Kelly    │
                  │  • market          │   │    sizing + drawdown   │
                  │  real data only    │   │  • risk_contribution:  │
                  │                    │   │    vol-weighted risk   │
                  └────────────────────┘   └────────────────────────┘
                                │                 │
                                └────────┬────────┘
                                         ▼
                              Calibrated, cited answer

The deterministic risk_tool keeps the math out of the LLM's head — sizing and drawdown are computed, not hallucinated.

---

Setup

Prerequisites

• Python 3.10+
• A Gemini API key (aistudio.google.com)
• A MongoDB Atlas cluster (free tier M0 works) with a connection string
• Node.js (the MongoDB MCP server runs via npx)

Install

cd calibrated-copilot
python -m venv .venv
# Windows: .venv\Scripts\activate   |   macOS/Linux: source .venv/bin/activate
pip install -r requirements.txt

Configure

cp .env.example .env
# then edit .env and fill in:
#   GEMINI_API_KEY=...
#   MONGODB_URI=mongodb+srv://user:pass@cluster.mongodb.net/

Seed the database

python seed_data.py

This loads a small sample portfolio + market snapshot into the calibrated_copilot database (portfolio and market collections).

Run the demo

python demo.py

Or talk to the agent interactively:

python -m calibrated_copilot.agent

---

Demo path (2-minute video)

The scripted demo runs three questions that show all three behaviours:

#	Question	What it shows
a	"Should I add to my BTC position?"	Calibrated probability + confidence-scaled position size from risk_tool
b	"Will the Fed cut rates next week and how should I trade it?"	Abstention — no data in Atlas, agent declines to give a confident call
c	"What's my portfolio's risk concentration?"	Computes concentration from real Atlas holdings

See docs/demo-script.md for the full 2-minute video script.

---

Project layout

calibrated-copilot/
├── README.md
├── LICENSE                 # MIT
├── requirements.txt
├── .env.example
├── .gitignore
├── seed_data.py            # load sample portfolio + market into Atlas
├── demo.py                 # 3 scripted demo questions
├── calibrated_copilot/
│   ├── __init__.py
│   ├── agent.py            # the ADK Gemini agent + MCP wiring
│   ├── prompts.py          # the calibration system prompt (the differentiator)
│   └── risk_tool.py        # deterministic risk_tool + risk_contribution
├── docs/
│   └── demo-script.md
└── SUBMISSION.md           # Devpost submission text

---

How it works (the calibration contract)

The system prompt (calibrated_copilot/prompts.py) binds the agent to a strict contract:

• Retrieve before claiming. Any number about the user's portfolio or the market must come from the MongoDB MCP tool. No number → no claim.
• Quantify confidence. Every directional/forecast statement carries a probability and a confidence band, plus the basis ("based on 90-day realized vol of X").
• Size by confidence. Recommendations call risk_tool, which returns a fractional-Kelly position size that shrinks toward zero as confidence drops, plus a drawdown flag.
• Measure risk, not just weight. Concentration questions call risk_contribution, which reports volatility-weighted risk shares (so a 30%-capital, 62%-vol Bitcoin sleeve is correctly flagged as ~49% of portfolio risk) plus a risk-based Herfindahl index.
• Abstain honestly. If the data is missing or signals conflict, the agent says so and gives no confident call.

---

What's next

• More MCP data sources (filings, news, alt-data)
• Live market feeds
• A reliability dashboard tracking the agent's own calibration over time (Brier score on its past probabilities)

---

License

MIT — see LICENSE.