typesafe-computer-use

MacTap is a macOS menu-bar app that turns one, two, or three knocks on a compatible MacBook chassis or the desk beneath it into keyboard shortcuts and other mapped actions. It supports left- and right-edge layouts, configurable sensitivity and actions, presets such as Daily, Coding, Capture, Media, and Focus, and per-application overrides. Tap classification runs on-device, with gesture mappings stored locally and nothing uploaded. It requires macOS 14.6 or later and a MacBook whose motion sensor is exposed as an SPU HID device, including M2-and-later models and M1 Pro, Max, or Ultra systems; some M1 models, including the original M1 Air, are unsupported. The app requires Accessibility permission for actions to send shortcuts, is distributed as a signed and notarized macOS app, and is licensed under the MIT license.

jev-use is a macOS voice and text computer-use harness built on Jev. It reads the frontmost app's Accessibility tree rather than taking screenshots, sends the command, app and window names, visible targets, and recent actions to Jev, then selects and performs operations such as clicking, typing, pressing keys, scrolling, opening items, and arranging windows. After each operation it reads the Accessibility tree again and continues until the task is complete, blocked, or waiting; low-confidence or destructive actions stop for confirmation. The app supports hold-to-talk, typed commands, hands-free listening, and an optional “Hey Jev” wake phrase, with speech handled by Apple Speech. It requires macOS 14.2 or later and Xcode, has no external dependencies, and stores the TypeSafe API key in the macOS Keychain.

macOS Harness is an experimental, MIT-licensed, macOS-only Python harness from browser-use that gives an LLM or coding agent raw primitives for controlling a Mac in one persistent Python process, without app-specific tools, recipes, or framework rails. Its six core primitives—`see`, `key`, `type`, `click`, `ax`, and `script`—capture native and Electron app windows, send keyboard and coordinate input directly to an application process, expose Apple Accessibility and Apple Events, execute AppleScript, and draw a click-through pointer without moving the physical cursor. The agent can write missing task logic in ordinary Python during execution. The same process provides access to a real, logged-in Chrome browser through Browser Harness, as well as the local filesystem and shell commands through Python, `Path`, and `subprocess`. It can capture background application windows without bringing them to the foreground and does not activate or raise the target app. The project includes installation and skill-registration workflows, a `doctor` command for checking required macOS permissions, and verification instructions. Anonymous telemetry is enabled by default and records the CLI command category, success, duration, package version, operating system and architecture, and detected agent client; the project states that it does not record prompts, app names, screenshots, UI text, scripts, paths, or window titles, and provides `macos-harness telemetry disable` to turn telemetry off.

FluidUse is an open-source macOS library and demo application for local computer use on Apple silicon. It reads forms in running Mac applications or browsers through the macOS Accessibility API, uses an on-device decision model to match visible fields with supplied profile data, and types or selects the resulting answers in the real application. A WebFormDriver supports embedded WKWebView forms. The project converts CUA-S1-FORMS to Core ML and runs it through FluidAudio; it also includes laya-based typed decision models for choice, score, and yes/no-style questions. DocumentEntities can turn PDF or text files containing label-and-value lines into a profile, while a predetermined-answer sheet handles question-style fields outside the model's decisions. The repository states that processing stays on the machine and that the tool does not read résumés, reason about dropdown options, write free text, upload files, or click Submit unless enabled. FluidUse is licensed under Apache 2.0 and requires Accessibility access for its launching terminal.

TypeSafe Mario is an experimental controller that uses TypeSafe's Jev model to play the original Super Mario Bros. from structured emulator state rather than screenshots. A parser converts emulator telemetry and RAM into object-centric JSON describing Mario's motion and jump trajectory, nearby enemies, terrain, response timing, recent control results, and episode progress; Jev then selects one of a small set of legal NES controller actions, after which the emulator advances and the loop repeats. The system also runs separate Choice, Noul, and Score judgments over each state: Choice selects the next controller macro, Noul estimates whether a forward jump is useful, and Score measures immediate danger for visualization. It provides a demo state inspector, a live dashboard with action probabilities, confidence, latency, jump probability, danger score, reward, and parsed state, plus JSONL decision logs containing model state, debug state, timing, and game outcomes. It requires Python 3.13 or newer, a TypeSafe API key, and a lawful local emulator and game setup; the repository does not include the Nintendo ROM or other copyrighted game data.