AI Computer Use Agents: How AI Works in Apps and Browsers

Most office routine does not live in modern cloud apps with tidy APIs. It lives in Excel, QuickBooks Desktop, supplier portals, online banking, government websites and that one legacy system nobody wants to touch. AI computer use agents tackle this differently: they work in the interface like a person, looking at the screen, clicking and typing. Here is how it works, what such an agent can do, where its limits are and how you keep an eye on it.

What "AI that uses a computer" means#

There are two ways to connect AI to the systems your business runs on.

Through an API. One program exchanges data directly with another. It is fast and reliable, but only if the service has an API, you are on a plan that includes it and someone has built the integration.

Through the interface. The AI agent gets a Windows computer and uses it like an employee: opens the program, finds the right button, enters data and checks the result. No API is needed. If a person can do it with a mouse and keyboard, the agent can too.

The second approach is what people mean by a computer use agent. In Super Intellect this is the job of the PC operator role: a general-purpose AI employee that opens any program or website and does the work following your instructions. You will find it, along with more specialized roles, in the AI employee catalog.

How the agent sees the screen and acts#

Every task is a loop of four steps that repeats many times.

1. 1Look. The agent takes a screenshot and works out what is on it: windows, buttons, fields, tables and text.
2. 2Decide. It compares what it sees with the task: where it is now, what the next step is, whether something went wrong.
3. 3Act. It moves the cursor and clicks, types, uses keyboard shortcuts, scrolls and drags files.
4. 4Check. It looks at the screen again: did the right window open, did the file save, is there an error message?

That checking step is what separates an agent from old-school macros that clicked blindly at fixed coordinates. If a window opens in the wrong place, a pop-up appears or a page loads slowly, the agent notices and adjusts.

Where it helps most: software without an API#

The browser deserves its own mention. Plenty of services technically have an API, but it costs extra, covers only part of the features or needs a developer. The agent simply opens the website and works in it like a staff member: fills in forms, collects data from pages and downloads reports.

The dedicated Windows workspace#

The agent does not run on your computer. It has its own dedicated Windows workspace on the platform's servers, with a desktop, a browser, office software and folders. The workspace belongs to one AI employee, together with its share of SUPER II capacity, and nobody else uses it.

What that means in practice:

• Isolation. The agent cannot see your personal files and does not get in your way. You work at your computer, it works at its own.
• Stability. The workspace does not shut down overnight or go to sleep, so long tasks run to the end.
• Your software. You can install the programs a job needs, such as a supplier's desktop client. The agent does not install software without your permission.
• Your logins. Credentials are kept in an encrypted vault and used at the moment of sign-in.

When a job is large, employees in a team split it by role and take turns in the team's workspace. That is how an AI team for business operates. For work to run at the same time you need another team, and that is why paid plans are priced per team: one team, one workspace.

Watching the agent from your dashboard#

An agent you cannot see is understandably unsettling. So the dashboard shows you its screen.

• Open a task and you see the agent's screen as a screenshot that refreshes every few seconds, so you know which program is open and what it is filling in.
• If it heads the wrong way, stop the task and tell it what to change.
• When the task finishes, a log remains: which programs were opened, what was entered (passwords excluded) and which files were created.

It is worth watching the first runs of a new task in full. That takes about as long as walking a new hire through it, and afterwards a glance at the log and a spot check of results is enough.

Three computer use scenarios#

Purchase orders from email into QuickBooks#

Orders arrive as free-form emails and someone keys them in by hand. The agent does the same: opens the inbox, reads the email, creates the document and checks the total.

1. 1Opens the shared inbox and finds new order emails.
2. 2Looks up the customer and flags it if the customer is missing.
3. 3Creates a sales order with the right items and quantities.
4. 4Compares the total with the email and saves the order as a draft.
5. 5Logs in a spreadsheet which orders went in and which need attention.

One weekly report from several dashboards#

Your sales data sits in three or four places, each with its own export. The agent logs into each, downloads the report, combines everything in Excel and adds up the totals.

Filling in web forms#

Supplier applications, carrier pickup requests, marketplace registrations: all of them are forms filled in from the same data. The agent takes the data from a spreadsheet or your company profile, fills in the form and shows it to you before submitting.

Limitations worth knowing#

A computer use agent is a powerful tool, not a magic one. Where it is weaker:

• Speed. It works at a human pace. Where an API moves a thousand records in seconds, the agent needs time, but you need no developer.
• CAPTCHAs and bot protection. The agent does not get around them and calls a person instead.
• Two-factor authentication. Codes go to the team member you choose, who confirms the sign-in.
• Interface changes. When a program updates and buttons move, the agent usually adapts, but watch the first runs after an update.
• Irreversible actions. Payments, signing and deletions wait for a person's approval.

How to get started#

1. 1Pick one repetitive computer task that takes someone an hour or two a day.
2. 2Describe it step by step: which programs, where the data comes from, where the result goes.
3. 3Create separate logins for the agent with minimal rights.
4. 4Run it on one example and watch the screen from your dashboard.
5. 5Refine the instructions and scale up.

See ready-made role bundles in solutions, compare plans on the pricing page, and sign up free to try task wording before you add a workspace.