phoneuse

Phone Harness is an open-source harness that lets AI agents such as Claude Code, Codex, or other LLMs control a real iPhone or Android phone without a jailbreak, injected app, Xcode, WebDriverAgent, or on-device installation. It exposes helpers for opening apps, reading screen content, tapping, scrolling, typing, and verifying actions. On iPhone, it uses the macOS iPhone Mirroring window as the transport: screenshots are captured from the window, Apple's Vision framework performs OCR and returns tap-ready coordinates, and HID-level CGEvents provide taps, long presses, drags, flicks, scrolling, Unicode typing, and shortcuts. On Android, it uses ADB over USB or Wi-Fi, combining screenshots with the phone's accessibility tree for text and element bounding boxes. Actions can be verified by capturing the screen again, with the screenshot serving as the ground truth rather than relying on a DOM. Each invocation is self-contained rather than relying on a daemon.

ESP32 iPhone Agent is an open-source AI automation tool that lets a vision model control a real iPhone without installing an app, using a computer, AirPlay mirroring, and an ESP32-S3 board acting as a USB mouse and keyboard. UxPlay receives the AirPlay stream, a GStreamer pipeline converts it into JPEG frames, and the computer sends relative pointer, typing, scrolling, and key commands to the board over Wi-Fi. A planner model creates a short plan from the task and current screen, while an actor model examines each subsequent screenshot and selects actions such as tapping, swiping, drawing, typing, pressing keys, going home, or finishing. The server tracks and recalibrates the relative pointer, checks whether the screen changed after each action, and can retry or re-plan after missed actions, repeated actions, dropped links, or invalid model responses. The project provides a Python/FastAPI server and browser-based interface with live phone control, logs, screenshots with action overlays, manual controls, calibration, and an emergency stop. It runs on macOS or Linux with an OpenAI-compatible vision model, either remote or local; the phone requires AssistiveTouch but no Mac, developer account, jailbreak, or installed app. The repository is licensed under MIT.

An ESP32-S3 firmware and MCP server for the M5Stack StopWatch that lets an external phone or computer agent control an iPhone. The client observes the iPhone screen and decides the next action; the StopWatch executes MCP commands such as taps, double-clicks, long presses, drags, scrolling, keyboard presses, ASCII text entry, and stopping or revoking control. The firmware runs no model and does not provide screenshots. The client connects to the StopWatch MCP endpoint over USB networking or Wi-Fi, while the StopWatch sends input to the iPhone over USB or experimental Bluetooth Low Energy. It supports portrait-normalized coordinates, one-pointer input, US-layout ASCII typing, and iPhones with AssistiveTouch enabled. The repository includes a setup skill for identifying the board, flashing firmware, pairing MCP, and adapting the software to other ESP32 boards. The project is integrated with the client-side Anan agent but also supports other HTTP MCP clients. Wi-Fi requires a 2.4 GHz local network; BLE input is experimental, and Chinese or other Unicode text input is not supported. Existing code is GPL-3.0-only, with an AGPL-3.0-or-later control-avatar component and other third-party notices retained in the repository.

Poki is an open-source tool that lets AI agents operate a real iPhone through a local MCP server and agent, without jailbreaking or additional hardware. It connects an iPhone to a Mac, exposes device-control tools to MCP clients such as Claude Code and Codex, and sends screen contents to the configured model service. Poki can also run a standalone terminal agent and authorized iMessage or Telegram channels. It requires macOS 15 or later, full Xcode, Node.js 24 or later, and a physical iPhone paired with the Mac over USB; the project documents limits including portrait-only support for verified physical devices. It is licensed under AGPL-3.0-only.