By Sagar Shankaran, Founder of CallSphere
Understand when to use Microsoft UFO for Windows desktop automation versus browser tools like Playwright or Selenium, with use cases for legacy apps, native software, and hybrid approaches.
Key takeaways
Modern web automation is a solved problem. Playwright, Selenium, and Puppeteer can interact with any web application through well-defined DOM APIs. But a large portion of enterprise computing still happens in Windows desktop applications — ERP systems, medical records software, CAD tools, legacy accounting packages, and internal tools built with WinForms, WPF, or even MFC.
These applications have no DOM, no CSS selectors, and no REST APIs. They exist only as native Windows processes with graphical interfaces. This is the gap UFO fills.
Browser automation tools operate on the DOM — the structured tree of HTML elements that represents a web page. Their core capabilities include:
flowchart LR
GOAL(["High level goal"])
PLAN["Planner LLM"]
SCREEN["Screen capture<br/>every step"]
VLM["Vision LLM<br/>reads UI state"]
ACT{"Action type"}
CLICK["Click coordinate"]
TYPE["Type text"]
KEY["Keyboard shortcut"]
GUARD["Safety filter<br/>allow lists"]
OS[("OS sandbox<br/>ephemeral VM")]
DONE(["Goal verified"])
GOAL --> PLAN --> SCREEN --> VLM --> ACT
ACT --> CLICK --> GUARD
ACT --> TYPE --> GUARD
ACT --> KEY --> GUARD
GUARD --> OS --> SCREEN
OS --> DONE
style PLAN fill:#4f46e5,stroke:#4338ca,color:#fff
style GUARD fill:#f59e0b,stroke:#d97706,color:#1f2937
style OS fill:#ede9fe,stroke:#7c3aed,color:#1e1b4b
style DONE fill:#059669,stroke:#047857,color:#fff
# Playwright: Easy and reliable for web apps
from playwright.sync_api import sync_playwright
with sync_playwright() as p:
browser = p.chromium.launch()
page = browser.new_page()
page.goto("https://app.example.com")
# CSS selectors, text selectors, role selectors
page.click("button.submit")
page.fill("input[name='email']", "user@example.com")
page.wait_for_selector(".success-message")
This works beautifully for web applications. But consider these scenarios where it fails completely:
Scenario 1: Legacy ERP System — A company runs SAP GUI, a native Windows application. There is no browser version available. Playwright cannot see or interact with SAP GUI windows.
Hear it before you finish reading
Talk to a live CallSphere AI voice agent in your browser — 60 seconds, no signup.
Scenario 2: Desktop Accounting Software — QuickBooks Desktop stores data locally and has a native Windows interface. The web version exists but lacks features the accounting team depends on.
Scenario 3: CAD/Engineering Tools — AutoCAD, SolidWorks, and MATLAB are desktop applications with complex custom-rendered UIs.
Scenario 4: File System Operations with GUI — Renaming, moving, and organizing files through File Explorer with specific right-click operations and property modifications.
UFO uses the Windows UI Automation (UIA) framework combined with visual understanding:
# UFO approach: Works with any Windows application
# No DOM, no CSS selectors — just visual understanding
# Task: Automate a legacy inventory management application
task = """
In the Inventory Manager application:
1. Click the 'New Item' button in the toolbar
2. In the Item Name field, type 'Widget Pro X200'
3. Set the category dropdown to 'Electronics'
4. Enter 150 in the Quantity field
5. Enter 29.99 in the Unit Price field
6. Click Save
"""
# UFO handles this by:
# 1. Taking a screenshot of the application
# 2. Identifying labeled UI controls
# 3. Asking GPT-4V which control matches "New Item button"
# 4. Executing the click
# 5. Repeating for each step
| Dimension | Playwright | UFO |
|---|---|---|
| Target | Web browsers | Windows desktop apps |
| Element ID | DOM selectors | Vision + UIA tree |
| Reliability | Very high (deterministic) | Moderate (model-dependent) |
| Speed | Fast (direct API) | Slower (LLM per step) |
| Cost | Free | $0.01-0.03 per action |
| UI resilience | Breaks on selector changes | Adapts visually |
Use UFO when:
Use Playwright/Selenium when:
Still reading? Stop comparing — try CallSphere live.
CallSphere ships complete AI voice agents per industry — 14 tools for healthcare, 10 agents for real estate, 4 specialists for salons. See how it actually handles a call before you book a demo.
Many real-world workflows span both web and desktop applications. A common pattern is using Playwright for web portions and UFO for desktop portions:
from playwright.sync_api import sync_playwright
import subprocess
def hybrid_workflow():
"""Download report from web app, process in desktop Excel."""
# Phase 1: Web automation with Playwright
with sync_playwright() as p:
browser = p.chromium.launch(headless=False)
page = browser.new_page()
page.goto("https://analytics.company.com")
page.fill("#username", "analyst@company.com")
page.fill("#password", "secure-password")
page.click("button[type='submit']")
# Download the report
with page.expect_download() as download_info:
page.click("text=Export to Excel")
download = download_info.value
file_path = download.path()
browser.close()
# Phase 2: Desktop automation with UFO
# Open the downloaded file in Excel and process it
subprocess.run([
"python", "-m", "ufo",
"--task",
f"Open {file_path} in Excel. "
"Create a pivot table from the data. "
"Add a chart showing monthly trends. "
"Save the workbook."
])
print("Hybrid workflow completed")
Common enterprise applications that lack web interfaces or APIs:
For these applications, UFO provides an automation path that simply did not exist before vision-capable LLMs.
Yes. Electron apps are rendered by Chromium but run as desktop applications. They expose UIA elements, so UFO can interact with them. However, since Electron apps are essentially web apps in a wrapper, you might also consider using Playwright with the Electron-specific API for better performance and reliability.
UFO is not designed for test automation. Each step requires an LLM API call (200-2000ms latency) plus screenshot processing. A 10-step task takes 20-60 seconds. For automated testing, use dedicated testing frameworks. UFO is best for workflow automation, data entry, and one-off tasks.
#DesktopVsWeb #PlaywrightAlternative #LegacyAutomation #EnterpriseRPA #MicrosoftUFO #HybridAutomation #WindowsApps
Written by
Sagar Shankaran· Founder, CallSphere
LinkedInSagar Shankaran is the founder of CallSphere, where he builds production AI voice and chat agents deployed across healthcare, hospitality, real estate, and home services. He writes about agentic AI, LLM engineering, and shipping voice agents that handle real calls in production.
See how AI voice agents work for your industry. Live demo available -- no signup required.
Build a working computer-use agent with the OpenAI Computer Use tool — clicks, types, scrolls a real browser — then evaluate task success on a benchmark suite.
Build a browser agent with LangGraph and Playwright that does multi-step web tasks, then ground-truth its work with visual diffs and DOM-based evaluators.
How Singapore-based companies are using ChatGPT Operator 2.0 for cross-border APAC workflows — pricing, latency, and regulatory considerations in 2026.
Detailed comparison of ChatGPT Operator 2.0, Browserbase, and Skyvern for production browser automation in 2026 — pricing, accuracy, and DX.
What ChatGPT Operator 2.0's developer API actually costs and supports in production — task templates, scheduled runs, and where it beats Browserbase.
The official puppeteer MCP went unmaintained; the MCP team recommends Playwright. We compare both, look at withLinda/puppeteer-real-browser-mcp-server for stealth, and show the browsing-agent loop.
© 2026 CallSphere Inc. All rights reserved.
Made within San Francisco
Watch how CallSphere handles real customer calls, schedules appointments, and processes payments — live.