iris-eval/mcp-server
MCP-native agent evaluation and observability server with trace logging, output quality evaluation, cost tracking, 12 built-in eval rules, real-time dashboard, and PII detection.
Ads
Overview
MCP-native agent evaluation and observability server with trace logging, output quality evaluation, cost tracking, 12 built-in eval rules, real-time dashboard, and PII detection.
Installation
npx @iris-eval/mcp-server --demo
Configuration
{
"mcpServers": {
"iris-eval": {
"command": "npx",
"args": ["@iris-eval/mcp-server", "--dashboard"]
}
}Capabilities
- Log agent runs
Ask to record an execution with `log_trace`, including spans, tool calls, token usage, and cost in USD.
- Score output quality
Use `evaluate_output` to check completeness, relevance, safety, and cost against 13 built-in rules.
- Query trace history
Retrieve stored runs with `get_traces`, filtering by time range, pagination, and other criteria.
- Manage custom rules
Deploy new eval rules with `deploy_rule` or remove them via `delete_rule` to tailor scoring.
- Run LLM-as-judge
Invoke `evaluate_with_llm_judge` for semantic scoring across five templates, with a hard per-eval cost cap.
- Verify citations
Use `verify_citations` to extract and fact-check cited sources against claims via an LLM judge.
Tags
- development
- awesome-list
Related MCP servers
ingero-io/ingero
eBPF-based GPU causal observability agent with MCP server. Traces CUDA Runtime/Driver APIs and host kernel events to build causal chains explaining GPU latency.
adanb13/cirdan
AI infrastructure cartographer & MCP server: fingerprints, graphs, and watches the live infrastructure an agent can reach (Docker, Kubernetes, cloud, IaC) and detects incidents.
agentkitai/agentlens
Tamper-evident observability for AI agents: a SHA-256 hash-chained audit log with chain verification and signed export (EU AI Act Art. 12). Instrument any agent with zero code via `npx -y @agentlensai/mcp`; also ingests OpenTelemetry GenAI traces.
alilxxey/openobserve-community-mcp
Read-only MCP server for OpenObserve Community Edition via REST API. Search logs, traces, stream schemas, and dashboards without requiring the Enterprise license.