cua_desktop_operator_skill

Windows desktop operation skill for MCP-capable AI agents. Use when the task requires observing the current desktop, launching or focusing apps, clicking or typing in Windows GUI apps, using reusable desktop macros, or running an observe-plan-act-verify loop through the local desktop-operator MCP server.

CUA Desktop Operator Skill

Use this skill when the user wants desktop control on Windows through a local MCP server.

What this skill expects

  • A local Python runtime with the desktop operator dependencies installed.
  • The desktop-operator MCP server available over stdio.
  • The agent can inspect screenshot artifacts returned by desktop_observe.
  • Temporary artifacts should be cleaned after task success unless the user asked to keep them.

If the current client does not expose the desktop_* MCP tools yet:

  1. Read references/mcp-client-setup.md.
  2. If the client is Codex, Claude Code, Cursor, or OpenCode, configure the local MCP server before attempting the desktop task.
  3. If the client is something else, stop and ask the user to add the MCP server manually, then send them the config snippet from references/mcp-client-setup.md.
  4. After configuration, reload the client if needed and verify that desktop-operator is visible before starting the task.

If the runtime is not ready:

  1. Run scripts/setup_runtime.ps1.
  2. Run scripts/start_mcp_server.ps1.

Core workflow

Use this loop for almost every GUI task:

  1. Call desktop_observe.
  2. Inspect the screenshot, active window, visible windows, and optional cropped window image.
  3. Choose the smallest next step.
  4. Prefer the highest-signal tool that fits:
    • desktop_focus_window before keyboard input
    • desktop_click_relative before absolute clicks
    • desktop_uia_* when a stable control is visible in UIA
    • desktop_run_macro when the action pattern is reusable
  5. After each mutating action, call desktop_observe or desktop_validate_state.
  6. Repeat until the success condition is visible.
  7. If the task is complete and the user did not ask to keep screenshots, logs, or debug traces, call desktop_cleanup_artifacts.

Tool selection order

Use tools in this order unless the task clearly needs something else:

  1. desktop_observe
  2. desktop_find_window or desktop_list_windows
  3. desktop_focus_window
  4. desktop_run_macro for stable repeated flows
  5. desktop_click_relative / desktop_uia_click
  6. desktop_paste_text for Chinese or mixed-language text
  7. desktop_type_text only for short plain text
  8. desktop_validate_state

Action policy

  • Keep actions small and reversible.
  • Prefer one to three tool calls before observing again.
  • Use desktop_wait when the UI is loading instead of stacking blind clicks.
  • Use match_index whenever multiple similar windows exist.
  • Use desktop_run_macro with macro_id="__catalog__" to inspect the built-in macro catalog.
  • Clean temporary artifacts after successful completion unless the user explicitly asked to retain them.

Safety policy

  • Do not run destructive OS actions unless the user explicitly asked for them.
  • Validate focus before typing or pressing submit keys.
  • If a click misses once, re-observe before trying again.
  • If UIA lookup is slow or empty, return to screenshot-driven interaction.

References

  • MCP tool details: references/mcp-tool-catalog.md
  • MCP client setup: references/mcp-client-setup.md
  • Interaction patterns: references/interaction-patterns.md
  • Macro catalog: references/macro-catalog.md
  • Failure recovery: references/failure-recovery.md
  • Cross-agent notes: references/compatibility.md