Where BAML Fell Short for Finnbud (And What We Kept)

I started building Finnbud with BAML.

At the time it was the right call. I was tired of hand-written OpenAI response schemas, inline prompt strings, and asRecord() casts scattered across the API. BAML promised deterministic output, generated TypeScript types, and prompts that diffed like code. For a while, it delivered.

Then Finnbud grew past the prototype stage. We added receipt parsing, inline spending answers, report generation, app-help routing, and eventually voice calls. Prompts became a product surface. We needed fast iteration, regression tests, observability, and the ability to ship prompt improvements without redeploying the API.

This post is not about moving away from BAML. We still use it. It is about where BAML fell short for us, what we moved to Langfuse, and the split that made both tools work together.

What BAML Did for Finnbud Early On

BAML solved real problems.

Before BAML	After BAML
Inline prompt literals with no history	`.baml` files that diffed in PRs
Hand-written JSON schemas passed to `response_format`	Generated types from BAML class definitions
50-200 lines of manual `isRecord()` / coercion per flow	Generated parser that fails closed on schema mismatch
Prompts tested only by mocking the LLM client	`baml test` running real prompts against fixtures
Normalization logic duplicated across services	Shared BAML types and one generated client

For small, well-defined functions like classifying chat intent, validating a receipt image, or simple extraction, BAML was genuinely nice. Schema-Aligned Parsing removed a whole class of "the model returned a shape we did not expect" bugs. If your problem is structured output from an LLM without writing a parser, BAML is a good answer.

That was enough for the first few months of Finnbud.

Where It Fell Short for Us

The break did not happen all at once. It happened as the prompt surface grew.

Finnbud's main parser, ParseText, has to understand user messages, route intent, extract transactions, handle follow-ups, classify categories against a user-defined list, decide when to call client tools, and write back natural-sounding confirmations. The prompt is long. The behavior is nuanced. And it changes almost every week.

What I needed was a loop:

See what the model was doing with real Finnbud traffic.
Build a dataset of real inputs and expected outputs.
Edit the prompt.
Run the candidate against the dataset.
Compare it to the current production version.
Promote the winner without a deploy.

BAML helped with step 4 in a small way. It did not help with the rest.

Prompt Changes Required a Deploy

In BAML, the prompt lives in a .baml file. To change it, you edit the file, run baml generate, run tests, commit, open a PR, merge, and deploy.

That workflow is fine for code. It is painful for prompts.

Prompts are not like functions. They are closer to copy, design, or heuristic tuning. I often need several small experiments in an hour to see which phrasing reduces a class of errors. Requiring a full deploy cycle for each experiment slows the feedback loop to a crawl. It also means only engineers can iterate, because the prompt is buried in source code.

We moved our prompts to Langfuse. Now a prompt has versions and labels. The production API always fetches the production label. Someone can edit a prompt in the Langfuse UI, save it as a new version, test it against a staging label, and promote it to production. No deploy. No baml generate. No generated client diff.

For Finnbud, that change alone was worth the split.

BAML Tests Did Not Scale for Us

BAML's inline test blocks are convenient for small functions. You write a test inside the same .baml file, give it sample arguments, and run baml test.

The problem is that every test duplicates the full prompt. Our parse-text.baml file ballooned to almost 700 KB because the prompt instructions were copied into every test case. That is not a test suite. That is a maintenance hazard.

Worse, the tests did not give us what we actually needed: pass rates, regression baselines, and comparisons between prompt versions. They mostly told us "this input does not crash."

We needed a benchmark. Langfuse datasets let us collect real Finnbud inputs, attach expected outputs, and run experiments against them. We can compare a candidate prompt to production on the same dataset, compute pass rates, and gate merges on thresholds (for example, require 85% of dataset items to match expected outputs before promoting). That is evaluation infrastructure, not unit tests.

Observability Took Extra Work

BAML has a Collector that captures token usage and raw request/response pairs, but only when you use its one-shot API. As soon as we moved to BAML's Modular API, which we needed to inject Langfuse-managed prompts at runtime, the Collector stopped giving us what we needed.

We ended up writing our own plumbing around the Modular API:

const request = await b.request.ParseText(/* args */);
const rawResponse = await sendBamlRequest(request);
const result = b.parse.ParseText(extractModelOutputText(rawResponse));

That meant we also had to extract token usage and raw request/response ourselves, instead of relying on BAML's built-in Collector. All of that exists because we needed to see what the model actually sent and received, and BAML's built-in path did not fit our architecture.

Langfuse, by contrast, is built for this. We now trace every model call with raw request/response, token usage, latency, and the exact prompt version that produced it. When a Finnbud user gets an odd response, we look at the trace instead of adding logs.

For us, that observability is not a nice-to-have. It is how we improve the product.

Templating Became Awkward

BAML prompts use Jinja-style templating. That works inside BAML. But once we moved prompts to Langfuse, we discovered that Langfuse cannot run BAML's Jinja. The templating languages do not compose.

We ended up with placeholder tokens inside our Langfuse prompts:

const CLIENT_TOOLS_JSON_PLACEHOLDER = '[[CLIENT_TOOLS_JSON]]';
const HELP_MANIFEST_PLACEHOLDER = '[[HELP_MANIFEST]]';

Then we do string replacement in TypeScript before passing the resolved prompt to BAML:

const instructions = prompt.text
  .replaceAll(CLIENT_TOOLS_JSON_PLACEHOLDER, clientToolsJson ?? '[]')
  .replaceAll(HELP_MANIFEST_PLACEHOLDER, helpManifest ?? 'none');

It works, but it is awkward. It is also a sign that we were using two tools for one job.

The Adapter Tax Added Up

BAML generates its own types, which did not match Finnbud's shared contracts exactly. We have shared types in packages/shared that mobile and web depend on. BAML's enums came back PascalCase while our contracts use lowercase.

Every BAML function needed a thin adapter layer. Here is one of the simpler ones:

function toContractStatus(value: BamlEntryStatus): 'ok' | 'needs_review' | 'reject' {
  switch (value) {
    case BamlEntryStatus.Ok:          return 'ok';
    case BamlEntryStatus.NeedsReview: return 'needs_review';
    case BamlEntryStatus.Reject:      return 'reject';
  }
}

Ten of these in one file. Then another file duplicates two of them because the same mapping is needed in a different flow. We shared some across files, but others got copied. That is not BAML's fault, but it is real code to maintain. The generated types were safe, but they were not free.

Could we have avoided it? Maybe, but every option traded one problem for another:

Make shared contracts match BAML's naming. That couples your public API to BAML's codegen style. If you drop BAML later, the naming leaks into mobile and web.
Make BAML match shared contracts. BAML's @alias only affects the prompt text, not the generated TypeScript. You cannot control the output casing.
Auto-generate the adapters. Now you have a codegen step managing another codegen step. And the adapters are not always 1:1 mappings. Some filter nulls, validate formats, or reshape nested objects. The generator needs custom rules anyway.
Use Zod instead of BAML types. You lose Schema-Aligned Parsing. Zod validates and throws. BAML repairs and returns.

We accepted the adapter tax because the code is thin, stable, and easy to test. Each mapper is a switch or a flatMap. It changes only when the BAML schema changes, which is rare. The real cost is not the code. It is the mental overhead of remembering which naming convention lives where.

The Split: What We Kept and What We Moved

Here is where we landed. We did not throw BAML away. We kept it where it is strong and moved what it could not handle to Langfuse.

What we kept in BAML:

Schema definitions and type generation. BAML's class and enum definitions in types.baml generate clean TypeScript types that match the schema exactly.
Schema-Aligned Parsing. When the model returns malformed JSON, BAML's repair layer fixes it deterministically (no extra AI call, no latency hit). Truncated output, trailing commas, missing optional fields, type mismatches: BAML handles all of it. Zod validates and throws. BAML repairs and returns.
The request templates in baml_src/clients.baml and function signatures in .baml files. These generate the typed client and the HTTP request shape.

What we moved to Langfuse:

Prompt management and versioning. Prompts live in Langfuse, labeled production or staging, versioned and promoted without deploys.
Observability. Every model call is traced with raw request/response, token usage, latency, and the prompt version that produced it.
Evaluation. Langfuse datasets hold regression cases. Experiments compare candidate prompts against production with pass rates and thresholds.
User feedback signals. Corrections from Finnbud users are recorded as scores, so we can find regressions quickly.

What we use BAML's Modular API for: We stopped using BAML's one-shot API (b.ParseText(...)) because it tied prompts to code. Instead, we use the Modular API to keep the parser and drop the prompt runner. We chose the Modular API over dropping BAML entirely because Schema-Aligned Parsing is the one thing we could not easily replace. Zod validates and throws on malformed JSON. BAML repairs it. That repair layer saves us from building a custom JSON fixer for truncated output, trailing commas, missing optional fields, and type mismatches. The Modular API gives us that repair without the prompt lock-in.

const prompt = await getLangfusePrompt('parse-text');
const instructions = prompt.text
  .replaceAll(CLIENT_TOOLS_JSON_PLACEHOLDER, clientToolsJson ?? '[]')
  .replaceAll(HELP_MANIFEST_PLACEHOLDER, helpManifest ?? 'none');

const request = await b.request.ParseText(/* args, instructions */);
const rawResponse = await sendBamlRequest(request);
const result = b.parse.ParseText(extractModelOutputText(rawResponse));

The prompt comes from Langfuse. The request is built and sent manually. The response is parsed with BAML's Schema-Aligned Parser.

Where BAML Still Makes Sense

I would still reach for BAML in a few situations:

Early-stage MVPs where you want type-safe LLM outputs without setting up a separate observability stack.
Small, stable prompt surfaces that do not change often and do not need production iteration.
Teams without dedicated prompt/observability infrastructure who want one tool that covers parsing and basic testing.
Flows where Schema-Aligned Parsing is the hardest problem and you would rather not maintain a parser yourself. I have seen teams ship extraction pipelines where BAML's parser paid for itself in the first week.

If your AI layer is one or two prompt calls and you deploy twice a month, BAML alone is probably a net win.

The Mental Model

BAML is a type-safe prompt runner with a strong parser. Its parser is excellent. Its prompt management is code-centric and requires deploys.

Langfuse is an AI operations platform. It is excellent at managing, observing, and evaluating AI behavior over time.

For Finnbud, the second job became more important than the first. But we kept BAML for the parser because nothing else does what Schema-Aligned Parsing does without an AI call.

Final Point

I did not move away from BAML. I moved the parts of BAML that stopped scaling for us to Langfuse, and kept the part that still earns its place: the parser.

BAML is a great on-ramp for structured LLM output. Its Schema-Aligned Parsing repairs malformed JSON deterministically, which is more than Zod or OpenAI's structured outputs do alone. But it is not a prompt management system, an observability platform, or an evaluation framework. Those are different jobs.

For Finnbud, those jobs stopped being optional. So we gave them to a tool built for them, and kept BAML where it works.

Enjoyed this article?