empirica-iris

Empirica Iris

Epistemic browser automation with SVG spatial indexing.

Iris extends Empirica to bring epistemic gating to browser automation. It vectorizes page layouts into SVG spatial indexes for precise coordinate targeting, compiles reusable browser lessons into executable action sequences, and gates all browser mutations through Empirica's noetic-praxic assessment framework.

The name: the iris controls what passes through the eye. Sentinel gates code actions; Iris gates visual interactions.

Why Iris?

Browser automation is the most ambiguous environment for AI. Two problems:

1. No gating — Browser mutations (clicks, form fills, navigation) bypass epistemic assessment entirely. An AI can click "Send" or "Delete" with zero understanding of consequences.
2. No spatial precision — LLMs guess coordinates from screenshots. No structured spatial representation exists for reasoning about page layout.

Iris solves both:

Problem	Solution	Mechanism
No gating	Browser Sentinel	NOETIC/PRAXIC classification for all browser MCP tools
No spatial precision	SVG PageMap	Vectorized spatial index — text tokens, not image tokens
Repeated work	Batch Execution	Compile lesson + map into N actions with 2 screenshots instead of N+1

Installation

pip install empirica-iris

With MCP server support:

pip install "empirica-iris[mcp]"

Development:

pip install -e ".[dev,mcp]"

Requires empirica >= 1.6.0 and Python >= 3.10.

Architecture

Iris is a separate package that depends on Empirica. It can be used as:

• Python library — Import directly for programmatic use
• MCP server — python -m iris.mcp.server for any MCP-compatible client
• Claude Code plugin — PreToolUse hooks for browser gating

empirica-iris/
  iris/
    mapper/       # SVG spatial indexing (PageMap, extraction, staleness)
    sentinel/     # Browser tool classification + gating
    batch/        # Lesson compilation + execution
    lessons/      # Browser lesson schema + recording + storage
    context/      # Page context for CHECK injection
    mcp/          # MCP server with 8 tools
  plugins/
    claude-code-integration/
      hooks/      # browser-sentinel.py PreToolUse hook

Core Components

1. SVG Mapper

Converts a page's accessibility tree and bounding boxes into an SVG spatial index.

Data flow:

Browser DOM
  -> JavaScript: getBoundingClientRect() for all interactive elements
  -> Parse: ref_id -> {role, name, bounds, selector}
  -> Generate SVG: labeled rectangles at precise viewport positions
  -> PageMap: {url, viewport, svg, elements[], content_hash}

The SVG is a spatial index, not a visual replica. Colored rectangles with labels — cheaper than screenshots (text vs image tokens), more precise for coordinate targeting.

Key types:

from iris.mapper.page_map import PageMap, MappedElement, Bounds, Viewport

page_map.find_by_role("button")       # All buttons
page_map.find_by_name("Sign In")      # Exact name match
page_map.find_nearest(400, 300)       # Closest element center
page_map.find_at_point(350, 120)      # Elements containing point
page_map.summary()                     # Compact dict for CHECK context

Staleness detection: PageMaps have a content hash (SHA256 of element positions + names). A lightweight JS fingerprint (element count + position hash of top 20 elements) enables fast validity checks without full re-extraction.

2. Browser Sentinel

Classifies all mcp__claude-in-chrome__* browser tools as NOETIC (observation) or PRAXIC (mutation), then gates praxic actions through Empirica's assessment framework.

Classification:

Phase	Tools
NOETIC (always allowed)	read_page, find, get_page_text, read_console_messages, read_network_requests, tabs_context_mcp, shortcuts_list, computer[screenshot|zoom|wait]
PRAXIC (require CHECK)	computer[click|type|key|scroll|...], form_input, navigate, javascript_tool, upload_image, resize_window

The computer tool is inspected by its action parameter. javascript_tool uses heuristic JS analysis: read-only DOM queries are noetic; DOM mutations and network calls are praxic.

Gating logic for praxic tools:

1. No PREFLIGHT -> deny (must open measurement window)
2. POSTFLIGHT already submitted -> deny (transaction closed)
3. Auto-proceed: know >= 0.70 and uncertainty <= 0.35 -> allow
4. No CHECK -> deny
5. CHECK = "investigate" -> deny (keep exploring)
6. CHECK = "proceed" -> allow

Fail-open on errors (sentinel crash doesn't block work).

3. Batch Execution

Compiles a BrowserLesson + PageMap into an executable BatchPlan, eliminating the screenshot-per-action loop.

Without Iris (N actions = N+1 screenshots):

screenshot -> analyze -> click -> wait -> screenshot -> analyze -> type -> ...

With Iris (N actions = 2 screenshots):

iris_map -> iris_match -> iris_compile -> iris_execute -> iris_verify

Fuzzy element matching resolves lesson step targets against PageMap elements:

Priority	Strategy	Confidence
1	Exact ref_id	1.0
2	CSS selector	0.95
3	Role + name	0.9
4	Role + proximity	0.7
5	Fuzzy name	0.6

Execution threshold: Coverage >= 70% and min confidence >= 0.4. Staleness checkpoints every N steps (configurable, default 5).

4. Lesson Recording + Storage

Browser lessons extend Empirica's lesson schema with spatial targeting fields:

from iris.lessons.browser_schema import BrowserLesson, BrowserLessonStep, BrowserActionType

lesson = BrowserLesson(
    name="github_login",
    domain="github.com",
    url_pattern=r"github\.com/login",
)
lesson.steps.append(BrowserLessonStep(
    order=0,
    action_type=BrowserActionType.TYPE,
    element_role="textbox",
    element_name="login",
    action_value="username",
))

Supported actions: CLICK, TYPE, FORM_INPUT, NAVIGATE, SCROLL, KEY, WAIT, SCREENSHOT

Storage: JSON file-based persistence in .empirica/iris-lessons/:

from iris.lessons.storage import LessonStore

store = LessonStore()
store.save(lesson)                              # .empirica/iris-lessons/github_login.json
store.load("github_login")                      # -> BrowserLesson
store.list_lessons()                            # -> ["github_login", ...]
store.search_by_domain("github.com")            # -> [BrowserLesson, ...]
store.search_by_url("https://github.com/login") # -> [BrowserLesson, ...]

MCP Server

Iris exposes 8 tools via the Model Context Protocol:

Tool	Phase	Description
iris_map	NOETIC	Generate SVG PageMap from current browser tab
iris_stale	NOETIC	Check if PageMap is still valid (lightweight fingerprint)
iris_find	NOETIC	Find element in PageMap by role/name/selector/ref_id
iris_match	NOETIC	Match lesson against PageMap (coverage assessment)
iris_compile	NOETIC	Compile lesson + PageMap into executable BatchPlan
iris_execute	PRAXIC	Execute compiled BatchPlan against browser
iris_verify	NOETIC	Verify execution result (URL, element count, staleness)
iris_record	PRAXIC	Start/stop recording browser actions as lesson

Two-phase tools: iris_map, iris_stale, and iris_verify use a two-phase pattern because the MCP server has no direct browser access:

• Phase 1: Returns JavaScript to execute in the browser
• Phase 2: Accepts the JS output and processes it (e.g., process_map_result())

Running the MCP Server

Standalone:

python -m iris.mcp.server

In Claude Code .mcp.json:

{
  "mcpServers": {
    "empirica-iris": {
      "command": "python3",
      "args": ["-m", "iris.mcp.server"]
    }
  }
}

Claude Code Plugin

Wire the browser sentinel into Claude Code's PreToolUse hooks in ~/.claude/settings.json:

{
  "PreToolUse": [{
    "matcher": "mcp__claude-in-chrome__.*",
    "hooks": [{
      "type": "command",
      "command": "python3 /path/to/empirica-iris/plugins/claude-code-integration/hooks/browser-sentinel.py",
      "timeout": 10
    }]
  }]
}

Typical Workflow

With saved lessons (batch mode):

iris_map -> iris_match -> iris_compile -> CHECK -> iris_execute -> iris_verify

Ad-hoc actions:

iris_map -> iris_find -> CHECK -> click/type (using coordinates from find) -> iris_verify

Recording new lessons:

iris_record start -> [perform browser actions] -> iris_record stop

Testing

pytest tests/ -q

226 tests across 6 modules: mapper, batch, classifier, sentinel, lessons, MCP integration.

License

MIT