GitHub launches computer use in Copilot, advising developers to try direct tools first

GitHub launched computer use in public preview on Thursday, October 1, giving Copilot CLI and its desktop app the ability to operate applications on macOS and Windows. The agent can read an app's content and click, type, scroll and drag, including in older, GUI-only software with no API, command-line interface or MCP integration.

An expense report in Safari served as GitHub's launch demonstration, but the company lists other scenarios: summarizing information in a legacy application, updating a presentation, entering data and moving information between apps. The feature is available from the terminal or the Copilot app.

Computer use vs. direct tools

Enabling computer use in Copilot CLI activates a bundled plugin with its own MCP server. It works in local sessions, reading application content through the operating system's accessibility tree and taking screenshots when it needs visual context.

GitHub nevertheless recommends direct tools wherever possible: if an API, MCP server, terminal command, filesystem tool or dedicated browser tool can handle the task, it typically gives more structured information and more predictable results than desktop interaction. OpenAI president Greg Brockman argued a broader case last month: agents could use the same interfaces as people, sparing the industry a connector for every piece of software.

Approvals and enterprise controls

Developers enable the feature with /computer in Copilot CLI or through the Computer Use settings in the Copilot app. On macOS it also requires Accessibility permission to control apps and Screen Recording permission to inspect windows when visual context is needed. The CLI session's permission mode determines whether Copilot asks before accessing an app. When prompted, developers can allow access for the current session, choose “Always allow” for future sessions or decline, and deny rules override both automatic and saved approvals.

Approvals saved in the CLI carry over to the desktop app on the same computer. Removing an app from the always-allowed list clears its approval for future sessions but leaves access already granted in a running session intact. Enterprise policy overrides a developer's local preference: if managed settings block computer use, the CLI reports the feature as unavailable, and through managed-settings.json owners can control whether developers may bypass approval prompts across the Copilot app, CLI and VS Code. The default-enablement policy for Business and Enterprise does not change the preview's opt-in status; it starts applying to unconfigured features on October 22, but excludes previews.

Reliability depends on the interface

GitHub warns that a change in timing or window state can cause Copilot to repeat an action or stall, and the agent may choose the wrong control, type into the wrong field or struggle with dynamic interfaces and complex workflows.