Building a desktop AI agent with Anthropic Computer Use and Tauri

When Anthropic released Computer Use last year, every demo showed Claude navigating a browser in a sandboxed VM. Cool, but limited. I wanted to see what happens when you give it a real desktop with real apps. So I built Juno.

Juno is a native macOS app. You describe a task -- type it or say it out loud -- and an AI agent takes over your mouse and keyboard. It can open apps, fill forms, browse the web, organize files. Anything you'd do manually.

Here's what I learned building it.

Why native, not web

The obvious approach would be a browser extension or an Electron wrapper. I went with Tauri (Rust backend, lightweight webview frontend) for a few reasons:

The tradeoff is platform lock-in. Juno currently only runs on macOS. Windows and Linux are possible but would need separate accessibility implementations.

The agent architecture

Early versions had a single agent loop: take screenshot, send to Claude, get back actions, execute them, repeat. This works for simple tasks but falls apart on anything complex.

The current design uses a multi-agent system:

Orchestrator
  ├── Desktop Agent (mouse, keyboard, app control)
  ├── Browser Agent (web navigation, content extraction)
  └── File Agent (filesystem operations)

The Orchestrator sees your request, decides which specialist(s) to invoke, and coordinates results. Specialists run with isolated memory -- they can't interfere with each other.

This matters for tasks like "research the top 5 coffee shops near me and put them in a spreadsheet." The Browser Agent handles the research. The Desktop Agent opens the spreadsheet app. The File Agent can save/organize the result. The Orchestrator stitches it together.

Screenshots and the Computer Use API

The core loop for any Computer Use agent is:

1. 1.Take a screenshot
2. 2.Send it to Claude with the current task context
3. 3.Claude returns tool calls (click at x,y; type text; scroll; etc.)
4. 4.Execute the tool calls
5. 5.Take another screenshot to verify
6. 6.Repeat until done

The tricky parts:

Voice control

Juno uses a custom Tauri plugin wrapping OpenAI's Whisper model for local speech-to-text. The plugin:

• Runs the whisper.cpp library in a background thread
• Monitors the microphone continuously
• Detects speech vs. silence using energy thresholds
• Transcribes speech segments and sends them to the agent

All of this happens on-device. No audio is sent to any server. The wake word ("Hey Juno" by default) is just a string match on the transcription output, which is simple but works well enough.

The main challenge was getting the voice-activity detection right. Too sensitive and it triggers on keyboard typing. Too conservative and it cuts off the beginning of sentences. I ended up using a combination of energy threshold + minimum duration + silence gap detection.

Security model

Giving an AI agent mouse and keyboard access is, obviously, a security concern. Juno has a tiered permission system:

By default, Juno runs at Level 3. Users can lower it for specific tasks or raise it when they trust the operation. There's also a tool approval system where certain sensitive operations (deleting files, sending messages, making purchases) require explicit user confirmation before the agent proceeds.

The CLI: giving other AI agents desktop superpowers

One thing I didn't plan for initially but turned out to be really useful: packaging Juno's capabilities as an MCP (Model Context Protocol) server.

This starts an MCP server that any compatible AI agent can connect to. Claude Code, Cursor, Codex -- they can all use it to take screenshots, read the accessibility tree, and control the desktop.

The use case: you're pair-programming with an AI agent and it needs to check something in a browser, or look at a design in Figma, or verify that the app you just built actually works visually. Instead of you describing what you see, the agent just looks.

What I'd do differently

Try it

macOS 14+ required. It's in beta -- rough edges exist. If you're interested in AI desktop automation or Computer Use, I'd appreciate feedback.