Skip to main content
← /writing
  • #genai
  • #productivity
  • #engineering-leadership

I Asked My AI Agents for Self-Reviews

Six personal agents can finish useful jobs, but I still coordinate their work. Their self-reviews exposed overlapping assignments, uncertain savings, and missing evidence.

Vinny Carpenter15 min read2.8k words

For almost the last decade, I read self-reviews from engineers. The useful ones named specific work, admitted a miss, and asked for a harder assignment. The weak ones listed activity and called it impact.

This time I asked six AI agents. I use Claude Duet, OpenAI Dots, Grok Bot, Meta's Muse, Instinct, and Wajo's Fo every day. Each works in the background across some part of my email, calendar, or accounts. I wrote about the category in The Return of the Personal Agent, and a month later I wanted to know what their work was adding up to.

The agents have completed useful jobs, but I still carry the context that connects them. Until I count that coordination, their individual savings tell me little about what the whole system returns.

Each agent described its own work well. None of them seemed to know it had five coworkers.

The handoff was me

Instinct bought three playoff tickets and got them into my Vivid Seats app. A parking pass failed to transfer, and the support exchange left that problem unresolved. Fo called Vivid Seats the next day and resolved it.

Both agents did useful work: Instinct completed the purchase, and Fo handled the follow-up call. The handoff between them was me, relaying the problem from one chat to another.

I was still the only one who could see across their assignments, and that's a problem I know well from infrastructure teams. Useful work happens locally, and every handoff depends on whoever remembers the whole picture.

At home, that person was me, because I'd put several capable agents to work without building any reliable way to coordinate them.

What I asked

I sent each agent the same message:

I'm writing a blog post about the AI agents I use day to day. Looking only at what you've actually done for me since I started using you, how have you made me more productive? Give me three specific examples with a rough estimate of time saved, one thing you got wrong or couldn't do, and one kind of work I'm not handing you today that I should. Don't describe your features in general. If you don't have enough history with me to answer honestly, say so.

Then I asked each one a different follow-up: Duet about approvals, Fo about its calls, and Instinct about finishing tasks without asking me anything.

The six had different histories, workloads, and permissions, and their replies mixed work I recognized with claims I haven't checked yet. The time estimates below are theirs. Naming a mistake made an answer more useful, but it didn't prove the rest of the account was accurate.

Six accounts of useful work

Claude Duet

Duet had about nine days of history and estimated that it saved me 4 to 6 hours. It scheduled release posts for three apps and drafted a week of X posts I approved as written.

Its most memorable miss was insisting the dot in my work email address was a typo. I spent time talking it out of that while paying another agent called Dots. It also named its daily package list as its noisiest update.

Duet was the only agent that asked me to supervise it less. It suggested approving a week of social posts in one batch, which would cut interruptions while keeping the publishing decision with me.

OpenAI Dots

Dots claimed 45 to 90 minutes across three examples, with limits attached to each. It had researched a desk lamp but didn't know whether I bought it. Much of its work involved security triage, such as checking a GSD Task Manager dependency warning against the lockfile before calling it low priority.

When I corrected its understanding of an AWS identity, it changed its advice and explained why. It also said its first-week record was too thin to support a claim of improvement, and that admission made the review more useful.

Grok Bot

The Grok routine in this review checks my LinkedIn posts and comments every weekday at 9:05 a.m. and messages me when something needs a reply. It estimated 1 to 1.5 hours saved each week.

Sign-in checks blocked it on some mornings, and one post loaded only four of 26 comments. Its most useful observation was about quiet days. If it sent nothing, I couldn't tell whether it had found nothing or failed to run, and that distinction mattered more than the time estimate.

Muse

Muse estimated 8 to 10 hours saved over seven weeks through its morning briefing and package checks. It added another 3 to 4 hours for tracking the Brewers' postseason, including gate times and ticket rules.

Its miss was telling me a treadmill promotion ended five days before it did, a mistake it caught and corrected itself. Muse argued that I noticed the big projects while the hours accumulated in "the quiet daily filtering" of routine messages. That's plausible, but I need a way to check what the filtering catches and what it misses.

Fo

Fo's self-review covered five weeks, four business calls, and one dinner booking. Two of those calls helped move my old VoIP number to a new carrier, exactly the kind of phone-tree errand I'd put off for a month. It flagged a duplicate iCloud storage charge, about $120 a year, without being asked, and it hasn't spent any of my money.

Its miss was a morning briefing I never wanted, which it ran in spoken and written versions for a week before I canceled them. I needed to define the job before letting the routine settle in.

As I noted in my earlier personal-agent essay, Fo can involve human assistance. I'm describing the service's outcomes, without claiming every step was automated.

Instinct

Instinct handled the ticket purchase from my first request to tickets in the Vivid Seats app the next morning. I made the budget decisions. It needed another approval because its first quote omitted tax, and it called that interruption its own mistake.

Instinct also added an unexpected flourish: it wrote its self-review as a blog post in my first person. I asked for an answer and got a ghostwriter.

A denominator changed the review

My follow-up to Instinct produced the most useful record. It described 13 consecutive task requests from Oct. 18 through Nov. 3, including the unfinished work.

A row of 13 squares for Instinct's 13 consecutive task requests from Oct. 18 through Nov. 3, as Instinct reported them. Eight are marked finished, two are half filled as partial with gaps named, and three are empty because they didn't finish. A box below reads: zero clarifying questions, which doesn't measure approvals, corrections, or follow-up. A green banner reads: next review, the same period, clear outcome definitions, and unfinished work included.

Those classifications give me a record to inspect. They still need checking, and the sample can't tell me how another agent would perform on different work.

Instinct reported zero clarifying questions across the sample. That doesn't tell me how much approval, correction, or follow-up the work needed. A task can fail without a question, and a useful task can require a decision from me.

The other five gave me the three examples I requested, and I'd asked Instinct an additional question about completion. Its answer changed what I want from all six next time: the same review period, clear outcome definitions, and unfinished work included. That's a better basis for deciding what to keep than choosing the agent that writes the most convincing review.

Six agents and no org chart

Putting the answers together exposed several shared assignments.

A grid of five jobs against six agents. Duet, Muse, and Instinct each describe a morning briefing, and Duet, Dots, and Muse each describe package tracking; both are recurring routines. Muse, Fo, and Instinct describe work on the subscription audit during the review period and on the Brewers postseason around particular events. Dots and Muse describe ongoing deadline watching. A green banner reads: every job needs an accountable owner and evidence of completion.

This grid records involvement, which needs a closer look before I call it duplication. Researching tickets, buying them, and resolving a transfer problem can be complementary jobs. Three versions of the same morning briefing need a different justification, and Fo had also run a briefing until I canceled it.

Some overlaps were less obvious. Muse described filtering the summary email from my own nightly GitHub routine, which might be useful filtering, but I still want that record when something fails. Muse also suggested drafting routine email replies, work Duet already does. The closest any agent came to noticing a coworker was Muse flagging my Grok Bot session as a duplicate calendar entry.

I spent years asking teams to name the owner of a service and agree on how recurring work should run. I hadn't applied that discipline to my own agents. I was assigning work, carrying the context between them, and then counting their individual estimates of time saved. When two teams own the same job, you don't fix it with better self-reviews. You fix the org chart.

Every job needs an accountable owner and evidence of completion. When responsibility changes, the handoff needs to carry the context. One agent can own the outcome and involve another. What I want to reduce is the work of reconstructing the situation each time responsibility changes.

The hours still need accounting

In Where the Saved Hour Goes, I argued that prompting, review, correction, rework, and coordination must come out of gross savings. The same accounting applies here.

I spend an hour or more a day with these agents, learning their capabilities, approving work, correcting reports, and relaying problems. That puts my rough estimate at seven hours or more a week. Part of that is the cost of evaluating six products and part is recurring supervision. I haven't separated the two well enough to claim a steady operating cost.

Their estimates have limits too. Duet's nine days and Muse's seven weeks cover different work, and a number transfer or a playoff purchase doesn't recur every week. If several agents claim time for the same outcome, I can't count the same avoided effort several times. And my prompt asked for three examples, not a complete account of their work.

A side-by-side comparison. On the left, what the agents reported: Duet, 4 to 6 hours in nine days; Muse, 8 to 10 hours over seven weeks; Grok, 1 to 1.5 hours a week; Dots, 45 to 90 minutes in total. On the right, what I spend: an hour or more a day, learning, approving, correcting, and relaying. Below, why the two don't net out yet: different periods, one-off jobs, shared outcomes, and three examples rather than a full account.

Coordination appears to consume much of the time they claim to save. I haven't established a dependable net gain, and a precise-looking total would hide more than it explained.

Learning is part of why I use these tools, and handing off a phone-tree errand has value beyond minutes saved. But those are different benefits. I want to know whether the routines themselves return attention once the learning and setup settle down.

The subscription charges are easier to count. I pay $200 a month for Claude Ultra, $100 for ChatGPT Pro 100, and $30 for X Subscription Premium. Those are $330 in plans I already bought for coding. The six agents currently add no separate subscription charge on my accounts, and Muse, Fo, and Instinct are free at my current level of use.

Some background work draws from the allowances I use for coding. Duet's background checks count against Claude plan limits. Grok Bot shares a weekly pool with Grok and Cursor under my X Subscription. Dot conversations don't count toward ChatGPT limits, but tasks in Codex or ChatGPT Work do.

So far, that shared usage hasn't pushed me into a limit. I also don't assume today's free access will last. The price and the access I grant both need to make sense as the services change.

A quiet routine still needs evidence

Several misses involved incorrect reports. Duet questioned a correct email address, Grok overstated how new a comment was, and Muse got a promotion deadline wrong before correcting it.

The failure that keeps pulling me back into the work is a confident report about the wrong state. Those errors are hardest to catch when I don't already know the answer, and the cases I notice can't tell me how often I miss one.

Filtering is harder still. Muse says much of its value comes from routine messages I no longer need to read. I can check that work, but checking it risks consuming the attention the filtering was supposed to save.

Grok suggested a one-line message on quiet days. That would separate a routine that ran from one that failed to run. But a check that loaded only part of a comment thread could still send a reassuring update.

I need a record of when the routine ran, what it reached, and whether it finished. An incomplete check should stay visibly incomplete. The record should be available without requiring six new reassurance messages every day, and I should be able to sample the underlying work.

Access problems belong in that record too. Apple sign-in stopped two agents on the same subscription. Duet couldn't find the OpenTable availability I needed, and LinkedIn sign-in checks interrupted Grok. Those are observed limits, and they don't establish why each service restricted or failed the interaction.

I explored those incentives in Your Agent Is Bad for Their Business. Here, the immediate requirement is simpler: an agent should show where its view of the work ends.

What I'm testing next

I want to find out whether clearer ownership and better records reduce the attention these routines require. I'll start with the work already assigned before adding the next job each agent suggested.

First, each recurring job gets one accountable owner. I'll choose who handles the morning briefing, package tracking, and other overlapping routines. Other agents can contribute when needed, but their role and the handoff should be explicit. My existing GitHub routine already has a job; another agent should have a clear reason to act on the same work.

Second, handoffs need a short record that another agent can use. It should include the request, decisions already made, work completed, unresolved issue, next owner, and supporting evidence. For the parking pass, that would let the next agent see what arrived, what failed to transfer, and what support had already tried.

An ownership map can reduce duplicate assignments, but it can't make isolated products exchange state. Some handoffs may remain manual, and their cost belongs in the next review. Giving every agent the same instructions won't, by itself, create the connection between them.

Third, routine status should live in a place I can inspect. I want to see the last check, its coverage, its outcome, and anything blocked or incomplete. Missing or failed checks should reach me. Routine success should leave evidence without demanding another conversation.

Fourth, I'll use the same questions and outcome definitions for each review. I want a consecutive sample of requests, including partial work and tasks that stopped at a permission boundary. An appropriate stop can respect my instructions while leaving the requested outcome unfinished, and both facts matter.

I'll also record where I intervened and how much attention it took. An approval that settles a spending decision is useful. Repeating an approval because a quote omitted tax is rework. A single autonomy percentage would blur that distinction.

These are changes to test. A more detailed record will only help if the benefit exceeds the work of maintaining it.

The record should survive the agent

In The Agent Is Not the Platform, I argued for an engineering system that remains useful when the agent changes. The same principle applies to these personal routines.

The ownership map, decisions, and execution history should remain usable if I replace an agent. Moving the whole record into whichever product I favor this month would make the next switch harder.

OpenAI envisions teams of dots working together. That could help within one product. My ticket-and-parking-pass handoff crossed products, so I still need a way for the relevant context and evidence to travel between them.

I expect to use fewer agents as I learn which jobs each handles well. A narrow job done reliably is worth keeping. Another overlapping routine needs to justify the attention it adds.

Next month, I'll compare completed work with the attention it required. Each recurring job will have an owner, and each claimed outcome will need evidence. I'll keep the agents that reduce my work, including the work of managing them.

// found this useful? share it

Post on X Share to LinkedIn
Vinny Carpenter

Written by Vinny Carpenter

VP Engineering · 30+ years building software

I lead engineering teams building cloud-native platforms at a Fortune 100 company. I write about engineering leadership, AI-assisted development, platform strategy, and the hard lessons that come from shipping at scale.

keep reading