Building With the OpenAI API: What I Learned Shipping Three Products

The gap between "works in a notebook" and "runs in production" is larger for LLM applications than for most software. The failure modes are different, the economics are different, and the debugging experience is genuinely strange in ways you do not anticipate until you are in it.

After shipping three products on the OpenAI API — a code review tool, a document summarizer, and a conversational onboarding flow — these are the lessons that stayed.

Model selection is a cost decision first

The instinct when building is to reach for the most capable model. In production, that instinct is expensive and usually wrong. The models differ meaningfully in capability for hard tasks, but for the majority of LLM features — extraction, summarization, classification, straightforward generation — a faster and cheaper model is good enough.

My current default:

• GPT-4o mini for high-volume, lower-stakes calls: extraction, classification, summarizing short documents, structured output from well-defined inputs.
• GPT-4o for reasoning-heavy tasks: code generation, complex synthesis, anything where errors are expensive.

The cost difference between these can be 20x. A feature that is trivially affordable at 1,000 requests/day becomes a budget problem at 100,000. Design for the target scale from the start, not from the prototype.

Streaming is not optional for user-facing features

A non-streaming response from a large model takes 5–15 seconds. Users read that as broken. Streaming starts showing content within 1–2 seconds, and the perceived performance is entirely different even if the total time is the same.

import OpenAI from "openai";
 
const client = new OpenAI();
 
const stream = await client.chat.completions.create({
  model: "gpt-4o",
  messages: [{ role: "user", content: prompt }],
  stream: true,
});
 
for await (const chunk of stream) {
  const delta = chunk.choices[0]?.delta?.content ?? "";
  process.stdout.write(delta); // or send to client via SSE
}

For anything a user watches in real time, streaming is table stakes. The engineering cost is low and the UX difference is large.

Rate limits will surprise you

The OpenAI API enforces limits on requests per minute (RPM) and tokens per minute (TPM). These limits vary by tier and model, and they are lower than you expect at the start. A feature that works fine in testing will start returning 429s under real concurrency.

What actually helps:

• Implement retry with exponential backoff. The official SDKs have this built in (maxRetries option), but check that it is configured correctly for your use case.
• Use a queue for non-interactive workloads. Batch jobs — processing uploads, running background analysis — should go through a queue with rate-limit-aware workers rather than calling the API in a tight loop.
• Monitor your usage dashboard. Set up alerts before you hit your rate limit, not after. A 429 storm from a legitimate traffic spike is nearly indistinguishable from a runaway bug.

Costs need active monitoring

LLM API costs are harder to reason about than database or compute costs because they depend on the content of requests — token counts vary per user, per prompt, per document. You cannot budget from first principles alone.

What works: log token counts on every call, aggregate daily by feature, and set a billing alert in the OpenAI dashboard at 80% of your expected monthly budget. The alert will fire once, teach you which feature is expensive, and pay for itself.

The expensive surprises I have seen: system prompts that were much longer than they needed to be (counted on every request), conversation history that grew unbounded over a session, and a batch job that accidentally sent full documents instead of summaries.

Evals are not optional at scale

The thing that makes LLM bugs hard is that they are statistical, not deterministic. A prompt change that improves average output quality can make a specific edge case worse in ways you will not see until a user reports it.

Even a simple eval suite — fifty representative inputs with expected output shapes or criteria — catches regressions before they ship. Build it early, before the codebase is large enough that changing a prompt feels risky.

The thing that does not change

The API itself is stable and well-designed. The client libraries are solid. The fundamentals — completion, structured output, tool use, streaming — are reliable enough to build on. The hard parts of building LLM applications are the same hard parts as building any other software: cost, reliability, and the feedback loop that tells you when something breaks.