This is a submission for the Hacktoberfest Open-Source AI Challenge Week 1: Touch Grass
What I Built
I built Offscreen — an AI-powered outdoor discovery journal.
The idea is simple: instead of giving people another reason to stay on their phones, the app gives them a reason to put their phones down.
Every session starts with a small outdoor challenge:
Capture the shadow of a person walking past you.
Or:
Spot a piece of faded, hand-painted advertising that has almost vanished into a building's walls.
The challenge might take five minutes or twenty-five.
Then comes the important part:
Put the phone down. Go outside.
When you come back, you bring a photograph of what you discovered. Gemma looks at the photograph, interprets what you found, and scores the discovery.
The app deliberately has no timer, no feed, and nothing to scroll. The shortest part of the experience is supposed to be the time spent in the app.
The goal isn't to make people spend more time with an AI.
It's to give them a reason to spend less time looking at a screen.
Demo
Live demo: https://offscreen-one.vercel.app/
Judge instructions: create an account with any email address and a password of at least 8 characters. There is no email verification step, so you can be signed in within seconds.
Demo video (63s, the real app end to end):
Your browser does not support the video tag.The complete experience is:
Get a challenge → Put the phone down → Go outside → Take a photo → Come back → Let Gemma evaluate your discovery.
Code
GitHub repository:
Offscreen
A photo walk, one challenge at a time.
Outside, Not Online — an AI-powered outdoor discovery journal built for the Hacktoberfest 2026 Open-Source AI Challenge: "Touch Grass".
You get one small, interesting real-world challenge. You leave the app, find something, photograph it, and come back. Gemma looks at the photo, judges whether it satisfies the challenge, writes a short reflection, and awards discovery points.
The design goal is that you leave the app. There is no feed, no streak pressure no notifications, and no infinite scroll.
Open app → Today's challenge → Close the app, go outside → Photograph → Gemma looks → Score + reflection → Journal → Come back tomorrow Watch it
The actual demo — the real app, real Gemma, real photograph, no cuts or narration:
The 21-second launch cut — a recreation of the product's screens, rendered from HTML:
The demo is the honest…
The project is an AI-powered web application with the AI workflow separated from the user interface, orchestrated as two explicit state graphs rather than a single opaque agent loop.
How I Built It
The core of the project is Gemma, Google's open-weight model.
Gemma writes every challenge, looks at every photograph, and decides every score. Nothing in the app works without it.
The application uses LangGraph to orchestrate the workflow. There are two graphs, and both are deliberately boring:
challenge: START → load_context → generate_challenge → END discovery: START → load_challenge → analyze_photo → evaluate_discovery → generate_feedback → save_discovery → update_profile → END There is no autonomous loop and no self-reflection. Each node runs once. The whole system has exactly one conditional edge, and it is an error guard: if the challenge can't be found, the graph jumps straight to END, because there is no point paying for a vision call to judge a photograph against nothing.
Three decisions were made against the obvious implementation.
1. One multimodal call, not four. The textbook pipeline is identify → describe → evaluate → feedback. That's four model calls, and it lets the model contradict its own description. Instead, a single request returns the description and the evaluation together. The two-stage path is still in the codebase behind a flag so the two can be compared.
2. Gemma ignores the function schema. My first live call used the provider's structured-output helper. Gemma ignored the JSON schema I declared and returned its own key names — asked for title, it gave me challenge_title. So the schema is now written into the prompt as an annotated JSON skeleton, the reply is parsed and validated locally with Pydantic, and a validation failure buys the model exactly one chance to fix itself. Nothing the model returns is trusted.
3. Personalization you can measure. "Favour what the user likes" left to the model is a suggestion it may ignore, and you cannot check it afterwards. So the category is chosen before generation — about 60% weighted toward the user's favourites and 40% uniform exploration — and handed to the model as a constraint it cannot override. A test draws 2,000 seeds and asserts the ratio. A score isn't a vibe here; it's a number somebody checked.
The backend is FastAPI, Pydantic, and SQLite through SQLAlchemy; the frontend is Next.js. There are 176 tests, all hermetic — no test makes a network call, none touches the real database, and the model is stubbed everywhere. The suite runs in about eighteen seconds and costs nothing, which meant I could run it after every single change.
Why Does Open Innovation Matter?
The honest version of this answer starts with a limitation: this MVP runs against Google's hosted AI Studio API, so photographs do leave the machine. I'm not claiming local inference or privacy properties I haven't demonstrated.
What open AI actually bought me was three concrete things.
1. It made the project possible in the time I had. A multimodal model I could call with an API key, in an afternoon, for free at this scale, is what turned "an idea about putting the phone down" into something I could iterate on. The interesting work went into prompt design, evaluation, and agent structure rather than into access. For a project whose entire premise is an AI loop, that's the difference between shipping and not.
2. Gemma's actual behaviour shaped the architecture. Asked for a JSON schema through the provider's function calling, it returned its own field names. A closed API would have given me that same result — but with an open model I could inspect the failure, decide to stop trusting the tooling, and build the fix into my own prompt and validation layer. Every subsequent discovery followed the same pattern: see what the model does, then design around it. I now know it hedges with "this appears to be" when it's unsure, which is exactly the behaviour the prompts demand, and I know it doesn't reliably — both of which I learned by testing, not by reading a model card.
3. The model is a configuration value, not an architecture. The model id lives in an environment variable. That made one optimisation obvious: challenge writing is mostly text generation and barely uses vision, so splitting a small model for challenges and the larger one only for judging became a straightforward next step rather than a rewrite. It's only obvious because swapping models isn't a rewrite.
LangGraph matters for the same reason — the reason this agent is explainable is that its control flow is an open graph I can read. When the graph made a mistake, I found it because the whole path was six lines long and visible. That determinism is a property of choosing an open harness, not something I wrote.
What Actually Happened When I Used It
This is the part I found most interesting, and most of it is wrong.
The judge is genuinely strict. I submitted a photograph of a walking shadow against a challenge asking for a dated plaque on a building. It came back 0/10, completed: false, with: "This is a great capture of city life and textures, but I couldn't spot a plaque or date in this frame!" No score inflation, no encouragement to try again — it just told me the truth. That mattered more to me than a high score would have, because the obvious failure mode for an AI judge is being agreeable.
It invented an object. On the same photograph it correctly described the low angle, the mid-stride legs, and the warm afternoon light — and then titled the discovery "Streetlight Silhouettes." There is no streetlight in that frame. My prompts explicitly forbid inventing things that aren't visible, and it did anyway. This is a real hallucination that shipped, and it's the clearest example of why an AI evaluator needs a human who actually goes outside and checks.
Its confidence is meaningless. Both submissions returned confidence: 1.0 — for the correct verdict and for the wrong one. A field that's always 1.0 carries no information at all. It's either a prompt problem or a calibration problem, and right now I can't tell which. The fix is a small labelled set of my own photographs with scores I'd defend, and measuring the model against them.
Real usage found bugs that 169 tests missed. The profile showed a streak of 0 and 0 completed challenges forever. Two separate bugs, in the same function: SQLite returns naive datetimes even from timezone-aware columns, so a timezone comparison always failed — meaning the streak would have worked perfectly on PostgreSQL, and the field I'd have migrated to later. And a counter was declared and returned by the API but never incremented anywhere in the codebase. Both only appeared once a real photograph went through the system. The tests had been asserting on in-memory objects; the new ones round-trip through the database, which is exactly the gap that let them through.
The latency is real. Generating a challenge takes about 40 seconds on the free tier. That's why the app pre-generates tomorrow's challenge while you're reading today's one — the wait happens once, quietly, instead of in front of you every morning.
My Agent Session
I built Offscreen from the initial idea through implementation and submission planning with an agentic coding workflow — twenty-two commits, phase by phase, from a written specification.
You can see the development process here:
hi, read @spec.md and then start bilding the app, phase by phase, ensuring proper commits for each feature built
I'll start by reading the spec.
No git repo yet. Starting Phase 0 (repo scaffolding).
Now the backend skeleton.
Now the frontend.
No API key in the environment — I'll build Phase 1 fully with an injectable client + tests, then need a key from you for the live check.
Now the prompt modules and the test endpoint.
Now schemas + the /api/test/gemma endpoint.
Now tests for the Gemma service with a fake client.
git's author must be me
Author fixed — the Phase 0 commit is now yours (Harsh-0986 <[REDACTED]>), and it's set in local repo config so all future commits are too. (I won't pass -c user.name=... again.)
Now back to the 3 failing Phase 1 tests.
Now let me verify the live endpoint behaves correctly with no key, then commit Phase 1.
Phase 0 and 1 are committed. One blocker before I call Phase 1 done:
I need a GEMINI_API_KEY (AI Studio). There's none in the environment and no .env yet.
aistudio.google.com/apikey → Create API key Put it in .env at the repo root (git-ignored):
GEMINI_API_KEY=your_key_here GEMMA_MODEL=gemma-4-26b-a4b-it Then I'll run the acceptance check:
cd backend && .venv/bin/uvicorn app.main:app --reload curl -X POST localhost:8000/api/test/gemma # → {"success":true, "structured_ok":true, ...} Status:
| Phase | State |
|---|---|
| 0 — Repo setup | ✅ 821ffad — both apps run, /health ok, landing page serves |
| 1 — Gemma connection | ⚠️ code + 17 tests pass; live call needs your key |
| 2–10 | not started |
Two notes on Phase 1:
- The model id gemma-4-26b-a4b-it from the spec is a placeholder — the live call will tell us if that exact id resolves. If it 404s I'll swap it via GEMMA_MODEL only, as the spec instructs.
- I added GEMMA_RETRY_BASE_DELAY as an env-tunable (default 1.5s) so tests don't actually sleep.
Two things worth your call before I continue:
1. Do you want to paste the key now, or should I press on with Phase 2 (challenge generator — fully unit-testable offline) and verify Phase 1 when you have the key?
2. GEMMA_MODEL — do you have a specific Gemma 4 model id from AI Studio you want pinned, or should I take whatever the key's project exposes?
added key, test
don't read env, just start
Fair — I'll make the config pick up backend/.env on its own instead of me reading it.
Gemma ignores the function-calling schema and invents its own keys. I'll switch the default to instructed-JSON with explicit schema rendering + one repair retry.
The repair loop caught a bad category and fixed it. Now the live endpoint:
Let me also remove the duplicated repair loop in the two call paths.
Now re-verify live (the enum now reaches the model) and commit.
Phase 1 acceptance met — live Gemma call works.
{"success":true,"model":"gemma-4-26b-a4b-it","structured_ok":true, "sample":"Take a deep breath and enjoy the fresh air."} Three real problems surfaced that only a live call could have caught:
-
.envlocation — your key was inbackend/.env, config only read the repo root. Now reads both. -
CORS_ORIGINScrashed the app — pydantic-settings tries to JSON-decode dotenv values, sohttp://localhost:3000was a hardSettingsError. Now parsed as a comma string. - Gemma ignores the function schema — this was the big one.
with_structured_outputreturned{"challenge_title": "Nature...", ...}, not my schema. Fixed by making the default path instructed-JSON (schema rendered into the prompt as an annotated skeleton → parsed → Pydantic-validated) plus one self-repair attempt. The repair loop earned its keep on the first live run: the model returnedcategory: "Exploration", failed validation, and corrected itself to a valid category. Old behaviour available viaGEMMA_STRUCTURED_OUTPUT_MODE=tooling.
gemma-4-26b-a4b-it resolves fine — no model swap needed.
Committed as 03a07b0. 22 tests pass, ruff clean.
Next: Phase 2 — Challenge Generator. That means the LangGraph state graph (START → LoadContext → GenerateChallenge → END), build_challenge_graph(), POST /api/challenges/generate returning a validated challenge, and a dependency-injected model so the tests stay offline.
Two decisions I'd like your input on before I build it, since both shape the DB work in Phase 5:
- Single-user MVP or multi-user? The spec's schema has
user_idon challenges and discoveries, but there's no auth in the MVP (spec §3 lists it as a non-goal). I'd default to a single implicit user (created on first request) so/journaland/profilework today without auth, while keepinguser_idcolumns so multi-user is a later migration. Agreed? - Does "today's challenge" expire?
/api/challenges/todayimplies one challenge per day. I'd generate on first request and reuse until the next UTC day, so a refresh doesn't burn a new challenge. Yes?
multi user, yes
Multi-user without auth it is — anonymous per-browser identity via a UUID header, real user_id on everything, auth deferred.
Now the API schemas and routes for challenges.
Now tests for the graph and endpoint (offline, fake service).
Adding the global error handler so no stack trace ever reaches the client (SPEC §26):
Now verify Phase 2 live against the real model:
Live works, but 35.9s against a <10s target (SPEC §43). Let me find out why.
bud leave it, it can consume time since it's a free api
Fair enough — dropping the latency optimization. Committing Phase 2.
Phase 2 committed (89b5627) — 36 tests, verified live.
Two things I noticed in the live run that you should know about, because they'll bite in Phase 5:
The duplicate guard can't work yet. User A's 2nd challenge was "a single leaf with a hole", the 3rd from user B was "a single leaf turned yellow" — all three leaves. The guard only compares against persisted history, and there is none until Phase 5. So the spec's "two consecutive challenges must not be identical" acceptance criterion is not actually met yet — it's coded and unit-tested against injected history, but not true end-to-end. I'll re-verify it after the DB lands.
35.9s per challenge, dropped at your instruction. Worth remembering that this eats your demo's 2-minute script budget if the frontend chains calls.
Next up: Phase 3 — Photo Analysis (upload + validation + multimodal Gemma, no scoring yet). That means the image upload endpoint, MIME/size validation with sanitized UUID filenames, and image resizing before it goes to the model.
Two questions on that:
-
/submitneeds achallenge_id, but nothing is persisted yet. Options: (a) generate an ephemeral in-memory challenge handle now and make it a real FK in Phase 5, or (b) pull the DB forward into Phase 3 so photo submission has something real to reference. The spec puts DB in Phase 5, but (a) means building a throwaway id layer. I lean (b) — pull the DB forward, since every remaining phase touches it. OK? - Test photos — do you have a few real outdoor photos on this machine I can use as fixtures for the AI test cases in SPEC §41 (obvious success, obvious failure, ambiguous, unrelated, poor quality)? That matters a lot for Phase 4 accuracy, and SPEC §37/§38 require real photos for the demo anyway.
pull db, and leave the testing, i'll do it at the end of the app
Pulling the DB forward. I'll generate synthetic fixture images programmatically for plumbing tests — real photo testing is yours at the end.
no need to create images, that we'll see later
Understood — no fixtures. Continuing the DB layer.
Now wire the graph to the DB and make /generate and /today persist.
Now repository tests for the DB layer itself:
Found it — a real coupling bug: preferences only update if a user row exists. Preferences should always record.
Now live-verify persistence + the challenge-reuse behaviour, including the dedup criterion that failed before:
hey, let's move testing to the end the last phase
Understood — no more live calls from me until the final phase. Committing the DB work on unit tests alone.
Now Phase 3 — Photo Analysis: upload, validation, and multimodal Gemma. No scoring yet, no live calls.
Now the photo schemas, analysis service, and upload endpoint:
Now tests for image validation and the upload endpoint — using in-memory images, no fixture files on disk:
Found a real trap: SQLite in-memory uses a per-thread connection, so the TestClient's thread got a different empty database.
Phase 3 committed (c385caf) — 85 tests, no live calls made.
Notable finds along the way:
- StaticPool bug in tests — in-memory SQLite gives each thread its own connection, so TestClient's thread was hitting a different, empty database than the test session. Silent and would have made ownership tests lie.
- Ownership is enforced at the photo layer too — passing another user's challenge_id yields "no challenge context" rather than leaking their prompt.
Next: Phase 4 — Challenge Evaluation. This is the heart of the app: the discovery graph
START → load_challenge → analyze_photo → evaluate_discovery → generate_feedback → save_discovery → update_profile → END All model calls stay injectable and offline-tested. Scoring, the self-repair loop, and the "appears to be" hedging rule (SPEC §14) get exercised by fakes now; real-photo accuracy waits for your final phase.
Two decisions I want your call on first, because they're visible in the product:
One multimodal call or two? SPEC §43 says prefer one call, but §12–13 describe analysis and evaluation as separate prompts. Combining them is faster and avoids the model contradicting its own description, at the cost of a less interesting prompt. I'd do one call that returns description + evaluation together, and keep the two-stage path behind a flag for comparison in your real testing. OK?
Should a failed challenge be saved to the journal? SPEC §15 says "every completed challenge creates a journal entry", which reads as successes only — but a score of 2/10 with kind feedback is motivating, and hiding misses feels dishonest. I'd save everything, and let the journal show
completed: falsewith a muted treatment. Yes?
prefer 1, ok
Both confirmed. Building Phase 4 with the single-call pipeline.
Now the discovery graph nodes:
Now the discovery graph:
image_bytes in the state violates SPEC §9's "state should be serializable". Moving it into the graph closure instead:
Now the discovery/journal/profile endpoints:
Now the offline test suite for the evaluation graph and endpoints:
Three real bugs found. Fixing: duplicate field placement, and the graph kept calling the model after a missing challenge, and a wrong error type on failure.
Phase 4 committed (efb28fa) — 106 tests, still no live calls.
The most important bug this phase caught: the graph kept calling the model after load_challenge failed. A user submitting against a nonexistent challenge would have burned a paid vision call and asked Gemma to judge a photo against nothing. A conditional edge now short-circuits to END — the only branching in the graph, and it's an error guard, not autonomy.
Backend MVP is now feature-complete against SPEC §19. Remaining:
| Phase | Scope | State |
|---|---|---|
| 6 | Journal frontend | not started |
| 7 | Personalization (favour top categories ~60%) | plumbing exists, not tuned |
| 8 | Frontend polish + animations | not started |
| 9–10 | Your real outdoor testing, demo data, empty/error states | yours |
Next I'll build Phase 6 + 7 together, since the journal UI is small and personalization is a small backend change — one commit each.
For Phase 7 there's a judgement call I want your input on. The spec says roughly 60% familiar / 40% exploration, but my bump_preference just adds +1 per discovery, which makes the ratio emergent rather than enforced. Two ways to do it:
- Prompt-level (what the spec implies): tell the model your favourite categories and let it choose. Zero code, but the model may ignore the ratio — and every generation is a coin flip.
- Deterministic category selection: pick the category ourselves (60% weighted pick from your top categories, 40% uniform from all 11), then constrain the model to it. The 60/40 split becomes measurable, and generations get cheaper because we skip the repair retry. But challenges may feel samey, since a category can only go so many ways.
I'd go with a hybrid: deterministic category pick, but only ~50% of the time — otherwise force a full-choice generation. Best of both, still measurable. Which do you want?
Also, one thing I noticed and should flag: the app currently has no way to change display_name, and no endpoint exists for it. The profile is anonymous. Do you want a tiny PATCH /api/profile for a name, or leave it anonymous for the hackathon?
second, yes
Option 2 it is — deterministic 60/40 category selection, plus the profile name endpoint.
Now wire it into the generate_challenge node:
Now the PATCH /api/profile name endpoint and personalization tests:
Now tests that personalization actually reaches the prompt and that consecutive challenges differ (SPEC §35):
The invalid-category path raised a raw ValidationError out of the graph. Making it degrade safely instead:
Adding a test for the profile-name endpoint, then committing:
Phase 7 committed (e92916f) — 128 tests, still no live calls.
Your deterministic option paid off immediately. Two bugs it exposed:
-
build_challenge_graphsilently ignoredselect_category— the constraint was never reaching the prompt at all. Caught only because the tests inject a fake picker. - An invalid category override raised a raw
ValidationErrorout of the graph, which would have surfaced as an unhandled 500. Now refused, with the model's own category kept.
The 60/40 split is now measurable, not aspirational — a test draws 2000 seeds and asserts familiar lands in 55–65%.
Next: Phase 6 — the frontend. This is the first time I'll write real UI, and it's the bulk of what makes the demo land. Before I start, three decisions that shape every screen:
Identity in the browser. The backend needs
X-User-Id. I'll generate a UUID on first visit, keep it inlocalStorage, and attach it in the API client — no login screen, matching the "anti-social" tone. Agreed?Fonts. SPEC §21 says large typography, photography, whitespace. I'd pair a heavy grotesque for the big statements (Inter Tight or Archivo) with a quiet serif for reflections — serif for the reflection text specifically sells "journal" over "app". Do you have a preference, or shall I pick?
Colour. SPEC §21 doesn't state a palette. I want to avoid the generic AI default of purple gradients on dark. My instinct: warm paper white background, near-black text, one mossy-green accent, and the photographs carry all the colour. It's on-brief for "anti-social media" — the app recedes, the photo dominates. Yes, or do you want something bolder?
Also worth deciding: /today takes ~35s to generate on a cold cache. That means the /today screen needs a genuinely good loading state ("Finding something interesting..." per SPEC §38), and the challenge should probably be generated in advance rather than on first view. I'd add a "tomorrow's challenge" pre-generation in the background — cheap to build now, and it removes the wait from the demo. Want that?
- login, 2. your wish, 3. your wish, bonus yes
Login it is — real accounts. That changes the backend's identity model, so I'll do that first as its own commit.
Now the repositories and auth routes:
Now replace the anonymous identity dependency with real auth:
Now update the routes that used the anonymous header:
My string replacements got sloppy. Rewriting both route files cleanly:
Now update the tests to the account model:
Now the endpoint tests — swapping X-User-Id headers for real bearer tokens:
The regex pass is getting fragile. Let me rewrite the endpoint section of that file directly.
This file has too many patch scars. Rewriting it cleanly:
Now auth tests:
Auth committed (2447240) — 162 tests. Anonymous header identity is fully gone; every route requires a bearer token now.
Security choices worth naming:
- PBKDF2-HMAC-SHA256 (600k rounds) from stdlib — no native bcr...
Related posts
Three dates that catch a hard-coded time-zone bug in a session clock
Laravel N+1 Query Problem: How to Find and Fix It
LongCat-2.5 at $0.30/$1.20: Thinking Locks Temperature, 1M Context, and Vendor Benches Between Opus 4.8 and 5
Shipped: two packages, a registry listing, and the first seals
AI Agent Memory Needs Garbage Collection, Not Just Storage
