Apple Vision Text Recognition (VNRecognizeTextRequest)

jev-use is a macOS voice and text computer-use harness built on Jev. It reads the frontmost app's Accessibility tree rather than taking screenshots, sends the command, app and window names, visible targets, and recent actions to Jev, then selects and performs operations such as clicking, typing, pressing keys, scrolling, opening items, and arranging windows. After each operation it reads the Accessibility tree again and continues until the task is complete, blocked, or waiting; low-confidence or destructive actions stop for confirmation. The app supports hold-to-talk, typed commands, hands-free listening, and an optional “Hey Jev” wake phrase, with speech handled by Apple Speech. It requires macOS 14.2 or later and Xcode, has no external dependencies, and stores the TypeSafe API key in the macOS Keychain.

BetterVoice is an experimental, open-source macOS menu-bar app for local voice dictation with screen context. It transcribes speech locally and captures the full display whenever the user circles an area with the pointer, marking each capture with a blue pointer trail and pulse while preserving multiple references in order. The app inserts the transcript into the selected text field when macOS permits it and pastes captured images when both speech and screen context are available. It supports quick notes and long explanations, configurable microphones, dictation languages, circle-detection thresholds, and keyboard shortcuts. English uses a local Parakeet model by default, while other languages use a separately downloaded multilingual model; an optional English-only grammar-cleanup feature runs locally through quantized ONNX weights.

FluidUse is an open-source macOS library and demo application for local computer use on Apple silicon. It reads forms in running Mac applications or browsers through the macOS Accessibility API, uses an on-device decision model to match visible fields with supplied profile data, and types or selects the resulting answers in the real application. A WebFormDriver supports embedded WKWebView forms. The project converts CUA-S1-FORMS to Core ML and runs it through FluidAudio; it also includes laya-based typed decision models for choice, score, and yes/no-style questions. DocumentEntities can turn PDF or text files containing label-and-value lines into a profile, while a predetermined-answer sheet handles question-style fields outside the model's decisions. The repository states that processing stays on the machine and that the tool does not read résumés, reason about dropdown options, write free text, upload files, or click Submit unless enabled. FluidUse is licensed under Apache 2.0 and requires Accessibility access for its launching terminal.

Voice Memos is a built-in audio recording application developed by Apple Inc. for iPhone, iPad and Mac. It records and edits audio clips, organizes recordings, and can sync them across devices via iCloud.