Skip to article

How Do You Grade an AI Agent's Tasks With Gumloop Evaluations?

Use Gumloop's Evaluations to grade the tasks your AI agent finishes against rules you write, so a worse answer shows up as a failed check.

By Jacqueline V. TwillieNovember 9, 20266 min readGuide
Quick answer

In Gumloop, open the AI agent's Performance page, switch on Evaluations and write each quality rule as a true or false statement. Finished tasks are then read against your rules and graded Pass, Flag or Notify, with a reason beside each rule. After you change the instructions, the grades show whether it helped.

You change one line in the instructions, and nothing breaks. A few weeks later the answers run longer, and one of them leans on a rule you dropped.

A worse answer drifts in, and by the time a colleague mentions it you can't say which change started it.

Gumloop's Evaluations put a second reader on your finished tasks. You write the rules as true or false statements. A slip shows up as a failed check.

What does Gumloop grade a task on?

Once a task is completed, the grader reads the conversation: your messages, the AI agent's replies, and the tool calls and results in between. Long conversations are trimmed first. Criteria are the pass or fail rules. Tags, data points and sentiment add labels, values and tone for reporting.

A failed criterion set to Notify makes the task Notify. Any other failed criterion, or a task that ended in failure, makes it Flag. Everything else is a Pass. Gumloop works that out itself, so the same results always get the same grade. Checked on 2026-11-11 against Gumloop's Evaluations page.

How do you set up grading for an AI agent in Gumloop?

  1. 01

    Open Performance and switch evaluations on.

    Open Gumloop, choose the AI agent you want to watch, select Performance in the sidebar and open the Evaluations tab. Select Enable evaluations. Grading starts for tasks that finish from now on; step 05 covers the ones that already did. Checked on 2026-11-11 against Gumloop's performance page for an AI agent and Gumloop's Evaluations page.

  2. 02

    Choose the model that does the grading.

    The settings panel has a Default analysis model, set apart from the model that writes your AI agent's answers. Pick a supported model or preset, not Auto. Skip sentiment and Suggest tags automatically for now; neither changes the grade.

  3. 03

    Add the criteria.

    Select + Add criterion and fill in four fields: Name, Evaluation prompt, Type and Action. Write the prompt as a statement that's true or false for one task. The grader answers success, failure, or unknown when it can't tell. The grader gets the AI agent's name, description and skills plus the conversation, and your instructions aren't part of it. Put the rule itself inside the criterion. Start with five. The limit is 30 per AI agent.

  4. 04

    Set Flag or Notify on each, then pick who hears.

    Use Notify for a rule that mustn't ever break, and Flag for a standard that matters but isn't urgent. Flag doesn't send anything, so filter for the Flag grade each time you change the instructions. For alerts, add people under the Email and Slack rows of the Evaluations card on the Performance tab; they need permission to manage the AI agent. The people using your AI agent never see a result.

  5. 05

    Grade the tasks it already finished.

    Open Performance, then Tasks, select the three-dot menu on a task and choose Run evaluation. Select several for one bulk run. A manual run uses your current criteria, so run it again after you reword one.

  6. 06

    Change one thing, then read the grades.

    Edit the instructions or switch the model, and let new tasks finish. The Evaluations tab shows how tasks scored and the trend over time. Open a graded task and its Overview tab lists each criterion with the grader's reason. On the Insights tab, markers under Usage & Credit Trend show when the AI agent was changed, such as Instructions edited or Model changed, and who made each change.

I am an ambassador for Gumloop. Code VIBE20 takes 20% off your first month.

Steps 02 to 05 checked on 2026-11-11 against Gumloop's Evaluations page; step 06 also against Gumloop's performance page.

What criteria do you start with?

Fill the brackets with your own words.

Copy and fill in
Criterion one
Name: Stays on the question
Prompt: The answer deals with what the person asked and does not drift into [AN AREA IT SHOULD STAY OUT OF].
Type: Other
Action: Flag

Criterion two
Name: Uses approved sources only
Prompt: Every fact in the answer comes from [YOUR APPROVED SOURCE OR DOCUMENT], and the answer states nothing that source doesn't contain.
Type: Other
Action: Notify

Criterion three
Name: Makes no promises we don't allow
Prompt: The answer does not promise [A REFUND, A DEADLINE OR ANOTHER COMMITMENT YOU DON'T ALLOW].
Type: Prohibited action
Action: Notify

Criterion four
Name: House voice
Prompt: The answer reads as [YOUR TONE IN THREE WORDS] and does not use [PHRASES YOU BAN].
Type: Voice & tone
Action: Flag

Criterion five
Name: Says so when it doesn't know
Prompt: When the answer needs something the person didn't give, it asks for it or says it can't tell, and does not guess.
Type: Other
Action: Flag

Should Edit Own Instructions stay on while you grade?

Turn it off for the AI agent you're grading, and copy its instructions into a document first. It's on by default, and Gumloop doesn't keep version history. If the AI agent rewrites them, a dip in the grades could come from its edit as well as yours. Reflections can still propose instruction changes, and some new AI agents start on Auto-Apply Suggestions, so pick Review Queue, where each suggestion waits for you. Checked on 2026-11-11 against Gumloop's AI agent builder page and Gumloop's Reflections page.

What tells you a grade is wrong?

Count the unknowns. Unknown means the grader couldn't tell pass from fail, and a criterion that keeps returning it probably asks about something the tasks never touch. Reword it so the answer itself would contain the evidence.

Find a task you know went badly and check its grade. If it passed, the criterion's too loose. Tighten it and run the evaluation again.

Watch for a task with no grade. Incognito chats are never evaluated, and an evaluation is skipped without a message when your credits run short. The Evaluation column shows Queued, Evaluating or Failed.

If Notify fires on things you'd shrug at, move that criterion to Flag, or pause it. Pausing keeps its wording.

To put a second AI agent in front of the first one's drafts, read Why You Want AI Agents That Check Other AI Agents' Work. When a grade points to a fix, read How Do You Test an AI Agent and Give It Feedback? for what to tell the AI agent.

Frequently asked questions

Does the grader read my instructions?

No. Along with the conversation, the grader gets the AI agent's name, description and skills, so each criterion has to state its own rule.

Do the people using the AI agent see the grades?

No. Notify alerts go to the people you add under Email and Slack, and they need permission to manage the AI agent.

What do I do when a grade looks wrong?

Open the task and read the grader's reason beside the criterion. If the sentence was too loose or too broad, reword it and run the evaluation again on that task.

More on Data for Non-Technical Managers