Learning Dad's Garden: From 4 Minutes of Phone Video to a Printed Watering Sheet, Offline

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 1: Touch Grass

My dad has looked after our pot plants for years. On Sunday morning he filmed them on his phone and explained each pot as he went.

At the bush with tag 12, he said: "Give regular amount of water, the usual amount."

He knows exactly how much that is. I didn't. So I built a tool that finds every line like that and asks him.

It's called Learning Dad's Garden. It turns his walk-through into a printed checklist I can take outside. Each pot gets a row, with a tick box for each day it needs something. Gemma 4 does the reading, on my laptop. Before anything prints, Dad answers the questions a beginner would ask.

What I Built

I'm taking up Dad's hobby, and he knows what every pot needs. Some of his instructions are exact. Plant with tag 4 gets "200 to 250 ml of water daily". For the pot with tag 9 he said "one cup of water or one and a half cup of water to be accurate, the green cup we have", which is exact, as long as nobody moves the green cup. Others only make sense if you already know the garden: "the usual amount", "water it regularly", "this plantation". To him, those are complete instructions.

Learning Dad's Garden turns a walk-through like that into something I can follow:

1. Dad films each part of the garden and talks, the way he would explain it to me in person.
2. "Learning Dad's Garden" turns his speech into text and splits it into instructions. Then it checks that every amount and time came from his words.
3. A gap check lists everything I couldn't act on. Dad answers on a local web page that shows the video frame and plays his voice from that moment.
4. The sheet prints. Each row has the tags, what to do, how much and when, a box for each day it's due, and Dad's exact words underneath.
5. I take the sheet outside, water, and tick. Each evening Dad checks the pots and marks every row: done right, missed or done wrong. My dad is grading my homework again.

The screen time comes first but it stays small. The routine lives on paper. This week I walk from pot to pot with the sheet and a pencil, and the laptop stays shut.

It works for anyone learning a garden from someone who knows it by feel. The command line calls them the owner and the helper. In our house, that's Papa and me.

The top of our sheet. Tag 12's "usual amount" is now half a glass of water, in Dad's typed words. What he said on Sunday sits under the row.

Demo

Code

GitHub URL

The licence is MIT.

It's built on faster-whisper, Ollama, Gemma 4, PyAV, Pillow, Jinja2 and headless Chromium for the PDF.

How I Built It

The model has one narrow job, and plain Python checks its work at every step. Pipeline :

What Gemma is allowed to do

Gemma gets the transcript of one part of the garden and a short list of rules. The first rule sets the boundary:

You are not a gardening expert. Never add advice, amounts, times or conditions that the speaker did not say.
(Well, neither am I, which is the whole reason this project exists.)

Ollama's structured output holds Gemma to a JSON schema. For each instruction, Gemma returns the exact quote, the tag numbers, the kind (do, don't, do only if, or a note), a short command, the amount, the timing and the condition. It also lists any words a helper couldn't act on. An example:

{ "quote": "water this daily in the morning or in the evening anytime when you are free give regular amount of water, the usual amount", "pots_said": "", "pots": [], "kind": "do", "what": "Water", "amount": "regular amount of water, the usual amount", "when": "daily in the morning or in the evening anytime when you are free", "condition": "", "vague": [ {"kind": "which_pot", "phrase": "this"}, {"kind": "how_much", "phrase": "regular amount of water, the usual amount"} ] } 

Gemma flagged both gaps on its own. It couldn't tell which pot "this" was, or how much "the usual amount" is. Both became questions for Dad.

I kept Gemma's thinking on, which is its default, with temperature 0 and an 8,192-token context. Gemma 4 E4B can take audio as input, but Whisper still does the listening. The tool needs two things Whisper gives for every word. The time lets the review page play the right clip. The confidence lets the sheet mark words Whisper wasn't sure of. I also give Whisper a list of words from Indian gardens, like tulsi, money plant, curry leaves, lota, etc. so it expects them.

Every value has to be in Dad's words

A model that restructures text will sometimes try to help. So nothing Gemma returns goes straight onto the sheet. First, the code finds each quote in the transcript, word for word or close to it, with a similarity of at least 0.8. Then it checks every amount, time and condition. A value must be copied from the instruction's own words, or from elsewhere in the same video. Every number in it must appear there too. Anything else is deleted.

def ground_field(value: str, quote: str, transcript: str) -> str | None: """Where a field value came from. "said": copied from the instruction's own words. "inferred": copied from elsewhere in the same note (for example "do the same for 26"); the owner is asked to confirm it. None: not found; the value must be dropped. """ if not value or not value.strip(): return None if is_extractive(value, quote) and numbers_ok(value, quote): return "said" if is_extractive(value, transcript) and numbers_ok(value, transcript): return "inferred" return None 

"Inferred" covers lines like "do the same for 26". The value is real but comes from another sentence, so the tool asks Dad to confirm it.

Days come from rules

The model never decides days. Plain rules turn the timing words into days on the sheet. "Daily" means every day. "Every alternate day", "one day gap" and "every 2 days" all mean days 1, 3, 5 and 7 on a 7-day sheet. That makes 11 rows on odd days and 5 on even days. A striped box means nothing to do that day.

The questions

The gap check rebuilds its questions from the instructions whenever anything changes, and it never stores a question as the truth. An answer changes the instruction. "It's fine as is" dismisses that one question. The questions come from five places:

• Gemma's own flags, like the two above.
• A phrase list of things people say instead of numbers. Three of its lines:

 ("how_much", r"\b(?:as usual|the usual(?: amount| way| quantity)?|like (?:usual|always|i do|before)|the same amount)\b"), ("how_often", r"\b(?:regularly|sometimes|occasionally|now and then|from time to time|often|frequently" r"|once in a while|on and off)\b"), ("when_to_act", r"\bdepend(?:s|ing)?(?: on| upon)?(?: how| what| whether| if| the)(?: \w+){0,4}"), 

• Structure rules: a watering row with no amount, two amounts joined by "or" (tag 9's "100 to 200 ml or 250 ml"), a command that is only a verb and an amount (tag 11's "Keep around 200 ml"), and a Do NOT line that also has a timing.
• Coverage: tags that no instruction names.
• Whisper: a word it heard with less than 5% confidence gets a question if it lands in a field on the sheet. In Dad's quotes it gets a "(?)", as in "amount brown (?) should be not more than 200 ml".

Every question carries its moment in the video. The page shows the frame and plays Dad's voice from the first word of the instruction to the last. He doesn't have to remember what "this" meant, because he sees what the camera saw. For pot with tag 12 he typed "This is tag number 12", then "that is usually 1/2 glass of water".

The tag 6 question, on a copy of the project without our videos, so the frame and audio player are missing. The green line repeats the previous answer back.

For the record, Plant (Tag #12) usual amount is half a glass of water. It took a 19-second video, a local AI model and one question to find that out. I could also have just asked him.

Why Does Open Innovation Matter?

The videos show our front door, our pots and my dad's voice. With Gemma and Whisper running on my laptop, none of it leaves the house, and after the first model download a full run works with the Wi-Fi off. Each run costs nothing, so I could re-run it as often as I liked. Anyone taking over a garden can run it too. It needs Docker and, for speed, an NVIDIA GPU, with Gemma 4 E2B as the smaller fallback.

Prize Categories

Best Use of Gemma. Gemma 4 E4B does all the reading. It splits Dad's speech into instructions, flags the words a beginner can't act on and, as an option, reads tag numbers in video frames. It runs locally through Ollama.