I build Bison, a chat app that compares AI models side by side. Today I wrote the small script behind our public comparison pages, and it is useful on its own: one prompt, four models, all answers in parallel. This is the core of it, tested with Node 20+ and OpenRouter (one API key, many providers).
const KEY = process.env.OPENROUTER_API_KEY; const MODELS = [ { id: "openai/gpt-6.2-sol", name: "GPT" }, { id: "anthropic/claude-sonnet-5.5", name: "Claude" }, { id: "google/gemini-4-argon", name: "Gemini" }, { id: "x-ai/grok-4.7", name: "Grok" }, ]; async function ask(model, prompt) { for (let attempt = 0; attempt < 3; attempt++) { const res = await fetch("https://openrouter.ai/api/v1/chat/completions", { method: "POST", headers: { Authorization: `Bearer ${KEY}`, "Content-Type": "application/json" }, body: JSON.stringify({ model, max_tokens: 3000, reasoning: { effort: "low" }, messages: [{ role: "user", content: prompt }], }), }); const text = (await res.json()).choices?.[0]?.message?.content; if (text) return text.trim(); await new Promise((r) => setTimeout(r, 1500 * (attempt + 1))); } return "(no answer)"; } const prompt = process.argv[2]; const answers = await Promise.all(MODELS.map((m) => ask(m.id, prompt))); MODELS.forEach((m, i) => console.log(`\n=== ${m.name} ===\n${answers[i]}`)); Run it with OPENROUTER_API_KEY=... node compare.mjs "Explain inflation with a grocery example in under 100 words". Check the current model IDs in the OpenRouter catalogue first, they change often.
The trap: reasoning models can return an empty answer
My first version used max_tokens: 600. One of the models kept coming back with content: null and finish_reason: "length". The cause: models that think before answering spend part of the token budget on hidden reasoning, so a small limit gets used up before a single visible word is written. The fix was two lines: give a much bigger max_tokens than the answer needs, and lower the reasoning effort (reasoning: { effort: "low" }). Always handle content === null instead of assuming a string.
A second small one: some providers reject temperature above 1, so keep it at 1 or below.
Why compare at all
Single answers hide differences you only see side by side: in my small runs one model averaged about 116 words per answer and another about 181 for the same "under 170 words" instruction. I wrote up the longer version with full answers here: the 10-prompt comparison. If you would rather not run scripts, Bison does the same thing in a UI (free plan, no card), but the snippet above is all you need to start.
What would you add: streaming, a judge model, cost tracking?


