Teaching my quiz app to make up its own questions

I have a side project called certflow, a little app for practicing certification exams. It has a pool of curated questions I wrote and reviewed by hand, which is exactly as fun as it sounds. So naturally, at some point, I had the idea every developer eventually has when faced with tedious manual work: what if the computer did it for me?

The plan was to let an AI generate extra exam questions on demand. The plan worked. The interesting part, the part I did not see coming, was everything that happened after it worked.

The easy 80%: making questions appear

Wiring up generation was the quick bit. certflow already had an AI layer for its tutor feature, so producing questions was mostly a matter of asking nicely and parsing the result into the same shape as my hand-written ones. A generator takes the exam's topics, asks the model for a batch, and hands back objects that look exactly like pool questions.

Crucially, the generation runs server-side, through an API route, not in the browser. This is one of those decisions that feels like bureaucratic box-ticking until the one time it saves you. Question generation needs credentials and a model endpoint, and the browser is the last place on earth you want either of those. So the flow is simple: the client asks the server for questions, the server talks to the model, and the client never sees anything it should not.

certflow supports a configurable AI backend (I run it against different providers depending on the day and my mood), so the settings page grew a small Advanced Options panel: temperature, max tokens, and a base URL. That last field is the quietly useful one, because it lets me point the same code at a local model or a hosted one without touching a line of source.

export interface AISettings {
  temperature: number  // 0–1, how adventurous the model gets
  maxTokens: number
  baseUrl: string      // point it at whatever speaks the same protocol
}

At this point I had a quiz that could invent an endless supply of questions. Victory. Except.

The hard 20%: not lying to the user

Here is the thing about a machine that generates plausible exam questions: it generates plausible exam questions. Plausible is not the same as correct. A model can produce a question that looks impeccable, uses all the right vocabulary, has four tidy answers, and is quietly, confidently wrong. It is the overconfident colleague of software: never in doubt, occasionally right.

My hand-curated questions carry a different kind of trust. I checked those. If a user is grinding through a practice exam the night before the real thing, the difference between "a human vetted this" and "a language model was feeling creative" is not a footnote. It is the whole point.

So the real feature was never "generate questions." It was be honest about where each question came from. Every question now carries its provenance, and the live question card wears it openly:

type QuestionSource = "curated" | "ai-generated"
// the UI shows a small badge; nobody has to guess

A small badge, almost nothing to build. But it changes the contract with the user from "trust this quiz" to "here is exactly what you are looking at, decide for yourself." I would rather show the seams than paper over them.

Letting the user hold the dial

Once provenance was visible, the natural next step was to stop deciding the mix for everyone. The generator had always supported a pool-to-generated ratio, but the value was hardcoded to 0.3 and buried in the code like a light switch installed inside a wall. Not useful to anyone who actually lives there.

So I pulled it out into the open as a target percentage. Someone who trusts only the vetted questions can sit at 100% curated. Someone who has exhausted the pool and just wants volume before an exam can crank up the generated share. The important bit is the graceful fallback: if the pool cannot fill its quota, or the generator comes back empty-handed, the app fills the gap with whatever it does have instead of presenting the user with a proud, empty exam.

That last sentence took far longer to get right than the badge, the settings panel, and the ratio slider combined. Fallbacks are where the honest work hides.

What the exercise actually taught me

I set out to add a feature called "AI question generation." What I actually built was a set of small honesties: run the model where its secrets are safe, label what it produces, let the user decide how much of it to trust, and never pretend the app has questions it does not.

The generating was the afternoon. The not-lying was the week. And that ratio, now that I write it down, feels like it applies to a great deal more than one quiz app.

certflow, badges and fallbacks and all, lives at github.com/Light2288/certflow for the curious.