Agent Eval

Agent Eval runs locally from a Python package published to PyPI: agent-regress-cli version 0.1.8. It speaks MCP over stdio, so the client starts it as a program and talks to it through standard input and output. It needs Python; clients usually start it with uvx (from uv) or after pip install — the usual command is uvx agent-regress-cli.

In the mcpServers JSON format that many desktop and editor MCP clients read, the entry looks like this (placeholders in angle brackets):

{
  "mcpServers": {
    "agent-eval": {
      "command": "uvx",
      "args": [
        "agent-regress-cli"
      ]
    }
  }
}

Derived from the registry entry, not tested here. What the server does, and on what terms, is set by its publisher; check its repository or website before giving it access to your accounts or files. How to add an MCP server to an assistant · Before you connect