When Harbor shipped alongside Terminal-Bench 2.0, it solved the right problem at the right time: a clean, container-first abstraction for trials and jobs, sandbox adapters for Daytona, Modal, E2B, Runloop, and GKE, and an honest CLI that lets anyone run harbor run --dataset ... --agent ... -n 100 and get useful results. For a researcher with an API key and a laptop, that’s everything you need.
But the moment you try to run Harbor as a team — engineers, research scientists, infra folks who don’t all live in the terminal — the rough edges show up. Who launched run 1037? What model did it use? Where are the logs? Did the run that died at 03:14 because of a Daytona auth blip ever come back? The CLI doesn’t answer those questions, and we don’t think it should — that’s not what Harbor is for.
So we built Harbormaster: a web platform that wraps Harbor and runs it on our own Kubernetes cluster. It is decisively not a replacement for Harbor. It is a thin shell around harbor run, the SQLAlchemy job tracking, and the artifact upload conventions, designed for the case where your benchmark suite is shared infrastructure rather than a personal script. Everything Harbor knows how to do, Harbormaster does — it just gives you a submit form, a dashboard, an audit trail, and a leaderboard at the end of it.
This post explains how it composes with Harbor, what it taught us about Harbor’s behavior under sustained load, and the specific bugs and edge cases we’d like to push back into the project.
What it is, concretely
The user-facing surface is small on purpose. You pick an agent (Claude Code, Gemini CLI, Codex), a dataset, a model, and a concurrency level, and you hit submit. From there:
The frontend — a React SPA served by nginx — POSTs to a FastAPI backend that writes a
BenchmarkRunrow to Postgres.The backend creates a Kubernetes
Jobwhose pod is a “runner” image that has Harbor installed.That runner builds a
JobConfigprogrammatically and calls Harbor’s Python API.Harbor does what Harbor does: it fans out trial pods at the requested concurrency, talks to the agent, runs the verifier, and writes the trajectory and reward to the configured artifact root.
We watch the artifact root — backed by S3 in production, MinIO locally — and stream per-trial state back into Postgres so the dashboard can show live progress, logs, and a running leaderboard.
The infrastructure is intentionally boring: EKS with Karpenter for node autoscaling, an ECR pull-through cache so trial pods don’t hammer Docker Hub, Traefik IngressRoute with OIDC for auth, and k3d + MinIO for local development so contributors can iterate without a cloud account. None of this is novel by itself. The point is that gluing it to Harbor turned out to be straightforward, which is a real compliment to Harbor’s design — the BaseEnvironment interface and the Trial/Job model gave us clean seams to build against.
The runner-as-Job pattern
The single most useful pattern we found is treating a Harbor invocation as a Kubernetes Job, not as a long-lived service. Each benchmark run is its own Job, with its own pod, its own resource envelope, and its own lifecycle. Harbor’s per-trial concurrency lives one level below: the runner pod calls into Harbor, and Harbor — via whichever environment adapter you’ve configured — fans out to N parallel trial pods on the same cluster.
Two things fall out of this naturally that we wanted:
Resilience to runner restarts. Harbor already uploads trial artifacts trial-by-trial as they finish. We lean on this. If a runner pod gets evicted mid-run (Karpenter consolidation, spot reclaim, OOM), the next reconciliation reads the artifact root, sees which trials have completed, and the dashboard reflects the partial state correctly. The run isn’t “lost” just because the orchestrator went away. This is mostly Harbor’s design doing the work — we just don’t get in its way.
Resource isolation per run. A misbehaving Replicationbench run that wants to allocate 32 GB of RAM doesn’t take down the dashboard or the API, because the runner is in its own pod with its own limits. The Kubernetes Job boundary turns out to be a much better blast radius than “long-running daemon that orchestrates everything.”
This isn’t a new idea — it’s basically how Argo Workflows and Tekton model things — but we want to call it out for the Harbor community because it’s a pattern that maps cleanly onto Harbor’s existing JobConfig/TrialConfig abstractions without requiring any changes to the framework. If you already run Kubernetes, this is a low-friction way to operationalize Harbor for a team.
What we ran
We used Harbormaster to run three frontier coding agents — Claude Code (claude-opus-4-6), Gemini CLI (gemini-3.1-pro-preview), and Codex (gpt-5.4) — across roughly a dozen public datasets each. The full numbers are in the appendix, but the headline results, on the datasets where all three completed:
Those numbers aren’t the point of this post — they’re a snapshot, and we’ll publish a proper write-up once SWE-Bench Verified completes for all three agents. What is the point: running this volume of trials end-to-end through Harbor, on our own cluster, on our own dime, gave us a much richer view of where Harbor is solid and where it has sharp edges.
What we’d like to push back upstream
Three findings are worth surfacing for the project, and we’ll be opening issues on each.
/logs/artifacts directory missing in trial environments. Several of our runs on SWE-Bench Multilingual hit a 53–86% verifier failure rate that resolved to a missing artifacts directory in the trial container. The verifier expected /logs/artifacts to exist; in some task environments it didn’t, and the verifier failed fast rather than creating the directory. Both Claude Code and Gemini CLI runs were affected, which suggests it’s a task-environment issue, not an agent issue. We’ve worked around it locally by ensuring the directory exists in our runner image preflight, but a fix in the relevant adapter would be cleaner.
Aider Polyglot verifier errors at scale. On Aider Polyglot at n_concurrent=4, we saw 88 verifier errors out of 225 trials for Claude Code (94 of 225 for Gemini CLI). The pattern looked race-y — re-running affected trials individually passed. We suspect contention on a shared verifier resource at high concurrency, but we haven’t fully isolated it. We’ll attach our trajectory bundles to the issue.
Pod OOM on Replicationbench HPC tasks. A handful of Replicationbench tasks run physics or ML simulations that allocate gigabytes of memory before Harbor finishes attaching. The pod gets killed before the trial can even start, which means the trial doesn’t show up as a failure in the usual sense — it just disappears. Two changes would help: a configurable per-task memory ceiling (so we can opt these tasks into a higher limit) and clearer surfacing of “pod OOMKilled before trial start” as a distinct failure class in the trial output.
None of these are dealbreakers. They’re the kind of thing you only find by running the framework hard, in an environment that isn’t the reference Daytona setup, against the full heterogeneity of the public benchmark catalog. We’d rather find them and report them than work around them silently.
What’s next
Three things on our roadmap that we think are interesting for the community:
We’re publishing the runner image and the K8s Job templates. The web UI itself has too much of our internal auth model baked in to open-source as-is, but the runner — the part that turns “Harbor on a laptop” into “Harbor on a cluster” — is general, and we’ll have it on GitHub by month’s end. If you have an EKS or GKE cluster and want a starting point that isn’t “stand up Daytona,” that’s the artifact to grab.
We want to contribute a richer EKS-flavored environment adapter alongside the existing GKE one. The two clouds aren’t identical (IRSA vs. Workload Identity, ECR vs. Artifact Registry), and the differences matter for image pull behavior at high concurrency. We’ve worked through them; we’d like that work to live in harbor-framework/harbor rather than only in our fork.
And we’d like to keep running the public catalog. The numbers above represent perhaps two weeks of cluster time. There are clear gaps — Codex hasn’t finished SWE-Bench Verified, Gemini CLI never completed it across three attempts, and Replicationbench is blocked on the OOM issue above. We’ll push through those and publish complete leaderboards as runs land.
If you’re building something similar — we’d love to compare notes. Harbor’s whole bet is that a shared standard makes the ecosystem move faster, and operationalizing that standard for teams is exactly the kind of thing the community can do better together than any one of us can do alone.
Over time we will be using Harbormaster to run as much of the Harbor suite as possible. Our work at Neurometric is heavily tied to task-specific AI and so, running Harbor on lots of models to understand the smallest, the fastest, and the most cost effective models for each task will help us move our product forward. As we go through this, we will make sure we publish the results. Stay tuned.