Skip to main content

macOS desktop automation

The macOS Desktop Skill lets an agent observe the current logged-in desktop and, within the permissions you grant, click, type, use keyboard shortcuts, or drag.

Docker, remote VPS hosts, system-level background services, and logged-out sessions cannot control the real macOS desktop.

1. Install AgentDock​

Follow the macOS installation and run AgentDock as the current logged-in user. Docker, a LaunchDaemon, and a remote Linux instance cannot control the current Mac desktop.

2. Install the Desktop Skill​

Send this instruction to a client that is already connected to AgentDock:

Read the desktop entry from the official AgentDock Skills catalog, verify the package against its digest, then install and activate it:
https://raw.githubusercontent.com/uvwt/agentdock-skills/main/catalog.json

The Desktop Skill is versioned and released independently in uvwt/agentdock-skills. The catalog pins its download URL and SHA-256 digest. After installation, ask whether the desktop Skill is active.

3. Grant macOS permissions​

  1. Grant Screen Recording permission to the terminal or host application that actually runs AgentDock.
  2. Grant Accessibility permission when clicking or typing is required.
  3. Restart the relevant terminal or application so the permission takes effect.

Grant these permissions only to the process that runs AgentDock, not to unrelated applications.

How to use it​

Describe the goal directly:

Inspect the current window and explain this error message.
Find the settings page in this application, but ask before changing anything.
Fill this text into the form without submitting it.

The agent should observe the current application and window first, perform the action, and observe or capture a screenshot again to verify the result.

Operations that require confirmation​

The following actions usually need an additional confirmation:

  • Send messages, email, or files.
  • Delete content.
  • Upload, pay, or authorize access.
  • Change system permissions or account settings.
  • Write sensitive information to the clipboard.

Privacy​

Desktop screenshots may contain notifications, chats, accounts, and file contents. Do not publish, synchronize, or commit screenshots and runtime artifacts automatically.

Skill development is covered in the Contributor guide.