Desktop Control Skill

Generic macOS desktop automation. No app-specific logic — just primitives.

Prerequisites

Run once:

bash skills/desktop-control/setup.sh

Modes

• Human mode (default) — curved mouse movement, randomised delays, per-character typing. Use when posting to Reddit or any site with bot detection.
• Fast mode (--fast) — direct clicks, no delays, instant typing. Use for internal automation where bot detection doesn't matter (e.g. checking Gmail, navigating internal tools).

desktop-control --fast click "Submit"   # instant
desktop-control click "Submit"          # human-like (~2.4s per click)

Agent Workflow — The Core Loop

Scan → Act → Scan → Act → Scan → Verify. Repeat until done.

Every single action must be preceded and followed by a scan. Never assume the screen state — always check.

SCAN → read all elements → decide what to do
ACT  → click / type / scroll
SCAN → confirm the action worked, see what changed
ACT  → next step
SCAN → verify final state

Full example — post a comment on Reddit

# Step 1: Navigate
desktop-control focus-url "https://reddit.com/r/example/comments/abc/post_title/"

# Step 2: SCAN — read the full page, find the comment box
desktop-control scan
# → read output, find "Join the conversation" or similar

# Step 3: Click the comment box
desktop-control click "Join the conversation"

# Step 4: SCAN again — confirm the box is active/focused
desktop-control scan
# → confirm the text input is ready

# Step 5: Type the comment
desktop-control type "Your comment here with https://docverify.app"

# Step 6: SCAN again — find the submit button
desktop-control scan
# → find "Comment" or "Post" button

# Step 7: Click submit
desktop-control click "Comment"

# Step 8: SCAN to verify the comment appeared
desktop-control scan
# → confirm your comment text is now visible on the page

Rules

1. Scan before every action — never click something you haven't just seen in a scan
2. Scan after every action — confirm the state changed as expected
3. Read the full scan output — every element matters, don't filter or skip
4. If something doesn't appear — scroll and scan again, or wait-for then scan
5. Never write Python — use only CLI commands; use run-task for sequences
6. Never assume — the page may have loaded differently, changed, or shown an error

When things go wrong

# Element not found? Scroll down and scan again
desktop-control scroll down --amount 3
desktop-control scan

# Page still loading? Wait then scan
desktop-control wait-for "element you expect" --timeout 15
desktop-control scan

# Unexpected state? Screenshot to see exactly what happened
desktop-control screenshot --output /tmp/debug.png
desktop-control scan

CLI (available globally as desktop-control)

# Window/App focus
desktop-control focus "Google Chrome"           # bring app to front
desktop-control focus-url "https://example.com" # navigate + focus Chrome
desktop-control focus-url "https://..." --app "Safari"
desktop-control active-app                      # print frontmost app name

# Screen capture & element detection
desktop-control screenshot [--output /tmp/screen.png]
desktop-control scan [--json]                   # detect all elements with labels

# Find elements
desktop-control find "Submit"                   # fuzzy label match, prints coords
desktop-control find "comment box"

# Interact
desktop-control click "Post"                    # click by label (fresh scan)
desktop-control click "820,450"                 # click by x,y coords
desktop-control type "hello world"              # human-like typing
desktop-control key return                      # send keystroke (return, tab, escape, etc)
desktop-control scroll down [--amount 3]

# Run a task sequence from JSON
desktop-control run-task tasks/my-task.json

Python API

import sys
sys.path.insert(0, 'skills/desktop-control')
from desktop_control import DesktopControl

dc = DesktopControl()
dc.focus("Google Chrome")
dc.focus_url("https://example.com")
elements = dc.find_all_elements()        # returns list of {label, bbox, center_x, center_y, confidence}
el = dc.find_element("Submit")           # fuzzy match, returns dict or None
dc.click_element("Submit")              # find + click
dc.type_text("hello world")
dc.scroll("down", amount=3)
app = dc.get_focused_app()

Task JSON format

[
  {"action": "focus", "app": "Google Chrome"},
  {"action": "focus-url", "url": "https://...", "app": "Google Chrome"},
  {"action": "wait", "seconds": 3},
  {"action": "click", "query": "Sign in"},
  {"action": "type", "text": "my text here"},
  {"action": "key", "key": "return"},
  {"action": "scroll", "direction": "down", "amount": 5},
  {"action": "verify", "query": "Success", "timeout": 10},
  {"action": "click", "x": 820, "y": 450}
]

How it works

1. Screenshot → /usr/sbin/screencapture
2. YOLOv8 (icon_detect model) → finds icon/button bounding boxes
3. Florence-2 (icon_caption model) → captions each crop with a human-readable label
4. easyocr → extracts text regions
5. cliclick → executes mouse moves/clicks and keyboard input with randomised delays

Paths

• Venv: ~/.openclaw/tools/desktop-control/.venv
• Models: ~/.openclaw/models/omniparser/ + ~/.openclaw/models/florence2-base/
• CLI: ~/.local/bin/desktop-control (symlink)
• Tasks: skills/desktop-control/tasks/ (put your JSON task files here)

Performance

• First run: ~25–30s (models load into memory)
• Subsequent scans (models cached): ~5–8s
• Detects 400+ elements on a typical desktop

Notes

• transformers must stay at 4.49.0 — v5.x breaks Florence-2
• All task logic (what to do with which app) lives in task JSON files, not in this skill
• This skill has no knowledge of any specific app, website, or workflow