Firebase's Agent Skills Now Start With a Failing Eval

Firebase’s product team has changed how it writes the instructions that tell AI coding agents how to use Firebase, moving from best-guess prompt authoring to a test-first loop borrowed from software engineering: write the failing eval first, then build the skill that passes it, according to a Firebase blog post published Aug 11 by product manager Charlotte Liang.

---

What changed

Firebase’s Agent Skills library, shipped in February, packages Firebase-specific instructions for Claude Code, Cursor, Gemini CLI and other assistants so they write correct Firebase code with less trial and error. Liang’s post says the team originally wrote those skills the way most people write documentation: guessing at what an agent might get wrong and writing instructions to head it off.

The new process replaces guesswork with a four-step loop, per the post:

1. Write the eval first: define the task and set strict LLM-as-a-judge criteria for what a correct Firebase implementation looks like, before any skill exists to pass it.
2. Establish a baseline: run the same task through an agent with zero Firebase skills installed and record its failure rate.
3. Write the skill: add the procedural steps, mandatory flags and specific “gotchas” that target the exact way the baseline agent failed.
4. Hill climb: re-run the eval, keeping the change only if the score goes up and discarding it if it doesn’t.

Firebase gives agents two ways to act on its platform: the CLI, which agents can call directly from a terminal, and MCP servers, which expose tools with explicit definitions an agent can reason about. The post frames skills as the layer that sits above both, teaching an agent when and how to reach for the CLI or an MCP tool rather than replacing either. Liang’s post says the eval loop did double duty: pushing an agent through the same tasks over and over surfaced friction in Firebase’s own CLI and MCP servers, not just in the wording of the skill files.

---

Why it matters

Firerun’s take: one vendor tuning its prompts isn’t much of a story on its own. “Model-as-judge” evaluation used to be a research technique reserved for teams training foundation models. Applying that same rigor, a scored baseline, a fixed rubric, a discard-if-it-doesn’t-improve rule, to writing developer docs is a sign that “does the agent actually get this right” is becoming a checkable claim instead of a vibe. For teams writing their own internal Claude Code or Cursor instructions, the four-step loop above is a directly reusable template, not just a Firebase-specific story.

Key Takeaways

• Firebase now writes agent skills with an eval-first loop: write the eval and success bar, measure a baseline failure rate with no skills installed, write the skill to fix the specific failure, then keep the change only if the eval score improves.
• The approach uses LLM-as-a-judge scoring against a defined success bar, not manual review of agent transcripts.
• Running agents through the eval loop repeatedly surfaced friction in Firebase’s own CLI and MCP servers, feeding fixes back into the tools themselves, not just the skill text.
• The work builds on Firebase’s Agent Skills library, which shipped in February for Claude Code, Cursor and Gemini CLI.