Client
Paying customer
Industry
Analytics operations
Service
Workflow automation
Runs on
Windows desktop, screen vision plus OS events
Status
Shipped to a paying customer
The problem
Giving a model raw mouse clicks produces unreliable automation. Scripts written per application break on every change. Analysts still did the repetitive Tableau work by hand.
What we built
An orchestrator agent that plans at the task level and a designer agent that executes pre-trained compound actions. Each action was trained against screen-vision input and OS-level Windows event data, so it has a predictable outcome. Built before tool-calling was the default agent pattern.
Tableau authoring lives behind a desktop GUI with no usable automation surface, and handing an LLM raw clicks produced unreliable behavior; training compound actions with predictable outcomes, verified against both screen vision and OS-level Windows events, made the agent's steps checkable and therefore reliable.
How it works, step by step
Tableau Desktop window
source
Tableau Desktop on Windows
The application under control
Screen capture stream
source
Windows GDI/DXGI frame grabs
Vision input per action step
Windows event tap
source
Win32 hooks + UI Automation events
OS-level ground truth signals
Task request intake
source · assumed
Local task queue / operator prompt
What the user wants built
Claude 2
Decompose a Tableau authoring request into an ordered plan of named compound actions
Guardrail: Can only emit action names present in the compound action library; free-form click coordinates are rejected by the schema validator
Claude Instant
Select the next compound action and fill its parameters from current UI state
Guardrail: Preconditions must match observed UI state before the input driver fires; bounded retries then escalate to orchestrator
Non-LLM: template matching plus OCR over captured frames, fused with UI Automation accessibility tree
Resolve on-screen controls to stable identifiers
Guardrail: Vision match and OS event/UIA node must agree on the target control or the step is marked unresolved
Non-LLM: rule-based assertion engine over expected Windows events and screen regions
Verify each compound action produced its trained outcome
Guardrail: No further action dispatched until the postcondition passes or the run halts for the operator
What a run produces
Run trace excerpt: build a sales-by-region dashboard sheet. Values illustrative; the shape is real.
| Step | Compound action | Verification signal | Status |
|---|---|---|---|
| 1 | open_data_source(connection) | UIA: data pane populated; window title changed | pass |
| 2 | drag_field_to_shelf(Region, Columns) | Event: drop completed; screen: pill visible on Columns | pass |
| 3 | drag_field_to_shelf(Sales, Rows) | Event: drop completed; screen: axis rendered | pass |
| 4 | set_mark_type(Bar) | Screen assertion failed - Marks dropdown still open | retry then pass |
| 5 | rename_and_save_sheet(Sales by Region) | Event: file write; title bar updated | pass |
What it did not do
- Only tasks expressible as combinations of pre-trained compound actions are possible; novel UI flows require training a new action first
- Tightly coupled to the Windows desktop build of Tableau - UI or version changes can invalidate trained actions and their screen assertions
- It is a desktop driver, not a Tableau API client: it has no server-side guarantees, and unattended runs still halt for an operator when a postcondition cannot be met
How it was measured
- Replay harness: each compound action run repeatedly from its trained starting state, scored on whether the expected Windows event sequence and post-action screen assertion both fire
- End-to-end task replay: recorded task requests re-executed and compared against the reference workbook produced by a human analyst performing the same steps
- Failure-mode drills: deliberate perturbations (dialog popups, window resize, slow data source load) to confirm the verifier halts or retries rather than clicking blind
Built and shipped by the same team you would talk to
Have a workflow like this one?
Thirty minutes. We map where it is stuck and what a first build looks like.
Book a free AI audit
Next story · AI education