AI Agents Completing Tasks: Give Them Minutes, Not Hours
Here are two true statements about the same AI agents, measured weeks apart by the same research lab.
On the original OSWorld A fixed set of tasks used to compare AI systems against each other. Useful for tracking progress over time, and routinely mistaken for a measure of how the thing will behave on your actual work. of real computer tasks, the best systems now score above 85%: better than the roughly 72% a human scores on the same set. On OSWorld 2.0, released on 26 June 2026, the best system in the world completes 20.6%.
Nothing got worse in between. The test got longer.
That single fact is the most useful thing you can know about agents right now, because it converts directly into a rule for what to hand one.
What actually changed between the two tests
OSWorld 2.0 is 108 workflows across seven professional domains, built by the same team at XLANG Lab. The tasks aren’t more obscure. They’re more drawn out.
| OSWorld 1.0 | OSWorld 2.0 | |
|---|---|---|
| Steps the agent takes | about 30 | 318 |
| How long a human needs | minutes | ~1.6 hours (median) |
| Best completion score | above 85% | 20.6% |
That middle row is the whole story. A task that takes you ten minutes is a different species from one that takes you an afternoon, and the second kind is where agents currently fall apart.
The researchers call this a A task with many steps stretched over a long run, where later steps depend on what happened in earlier ones. The opposite is a short, self-contained job: find this, reformat that, file it there. task, and their finding is that nothing on the market handles them end to end. At a 500-step budget, no system completed more than 21%.
Agents don’t crash. They stall
Here’s the detail that will match your own experience.
The best system scored 20.6% on finishing tasks, but 54.8% on partial credit. Across every system tested, partial scores clustered between 20% and 55%.
So agents are not flailing. On a long job they get somewhere between a fifth and half of the way through, do that part competently, and then fail to land it: usually by losing track of what it already did, because nothing on the screen states it outright and it has to infer the current state from what it can see. Which is exactly the frustrating experience people describe: the agent did most of the work, and you still had to finish it yourself.
That’s worse than it sounds, because a half-finished task isn’t half a task. You have to read what it did, work out where it stopped, decide whether the earlier steps are trustworthy, and then complete it, and reviewing someone else’s abandoned work is often slower than doing it yourself.
Why length is the thing that breaks them
The arithmetic is unforgiving, and it’s the same reason a long chain of anything fails.
Suppose an agent gets each individual step right 99% of the time. That sounds excellent. Over 30 steps, it finishes cleanly about 74% of the time. Over 318 steps: OSWorld 2.0’s average. It finishes about 4% of the time.
(That arithmetic is mine, and it’s a simplification: real errors aren’t independent, and agents can recover from some mistakes. But the shape is right, and it’s why “more steps” is not a small change.)
This is also why demos are so persuasive and Tuesday is so disappointing. A demo is a twelve-step task. Your actual workflow is a hundred-step task with three decisions in the middle that depend on things nobody wrote down.
What to do with this on Tuesday
The rule that falls out of the data is simple: measure the task in the time it would take you, and keep it under about half an hour.
Three ways to use that:
- Hand over minutes, not afternoons. “Pull these figures from the last four reports and put them in this table” is a good agent task. “Prepare the quarterly review” is not, not because it’s harder, but because it’s longer.
- Cut long jobs into short ones with a checkpoint between. Three twenty-minute tasks you look at in turn will beat one ninety-minute task you look at once. You’re re-anchoring the agent each time instead of letting errors compound.
- Expect to finish it yourself, and budget for that. Given that partial credit runs far ahead of completion, the realistic plan is that the agent gets you most of the way. Treat it as a strong first draft rather than a delivered result, and check it properly: the errors will be concentrated in the steps you weren’t watching.
None of this says agents aren’t useful. An 85% success rate on short, bounded computer tasks is genuinely good, and that’s most of the busywork in a normal week. It says the useful zone has a shape, and the shape is short.
The honest caveats
Benchmarks aren’t your job. OSWorld 2.0 uses self-hosted websites and set tasks, and your work has context the test doesn’t: some of which makes things easier, some much harder.
And this will move. OSWorld 1.0 went from 12% in April 2024 to above 85% by mid-2026, so the 20.6% figure is a snapshot of August 2026, not a permanent ceiling. What’s less likely to change quickly is the pattern: whenever a benchmark gets harder in the direction of real work, scores fall a long way, and long-horizon reliability is the hardest part of the problem.
Which is a good reason to treat any vendor number as a question rather than an answer. Ask which benchmark, and how long the tasks were.
The takeaway
When someone tells you AI agents complete 85% of tasks, they aren’t lying, and when someone tells you the best model manages 20%, they aren’t either. Both are real results on real tests. The tests differed in how long the work took.
So the practical version is short enough to remember: if the job would take you more than half an hour, don’t hand the whole thing to an agent. Break it up, put a checkpoint between the pieces, and plan to finish it yourself.
Sources: OSWorld 2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Tasks (opens in a new tab) (XLANG Lab, arXiv 2606.29537, released 26 June 2026). For the 108 tasks across seven domains, the ~1.6-hour median human completion time, the 318 average tool calls versus about 30 in OSWorld 1.0, the 20.6% best binary completion and 54.8% best partial score, and the finding that no system exceeded 21% at a 500-step budget. The stalling mechanism is the paper’s too. It names “implicit-state inference” among the challenge phenomena its tasks are built to expose, alongside cross-source reasoning and dynamic environments. OSWorld 1.0 scores and the ~72% human baseline are from that benchmark’s public leaderboard. The compounding-error arithmetic and the half-hour rule are mine, not the paper’s: the researchers describe the failures, they don’t prescribe a task length.
Related: What AI ‘Agents’ Actually Do, and What They Can Do for Your Day is the plain-English explanation of what these are and which categories of work they suit. The First Job to Give an AI Agent, and the One to Keep is how to pick that first task, and the length rule here is the sharpest filter to apply to it. How AI Agents Work: A Loop, Not a Brain is the mechanism underneath all of this. Why step count is the variable that matters at all.