Building Production-Ready Open-Weights AI Agents with Gemma, Google ADK, and Cloud Run GPUs

Figure 1: The four foundational pillars I used to take an open-weights Gemma agent from a local prototype to governed enterprise production on Google Cloud.

Most AI tutorials stop at a basic "Hello World" chatbot running on a laptop. However, when engineering teams try to move open-weights models like Google’s Gemma 3 (gemma-3-4b-it, gemma-3-12b-it, gemma-3-27b-it) into real enterprise production, they immediately hit four hard engineering questions:

Grounding: How do I connect Gemma to live enterprise data (like Cloud Firestore) so it doesn't hallucinate?
Serverless GPU Scaling: How do I serve Gemma on GPUs (Cloud Run + vLLM + Cloud Storage FUSE) so it scales to zero when idle instead of burning thousands of dollars a month?
Security & Governance: How do I guarantee the agent cannot leak API keys, fall for prompt injections, or accidentally take down a production service?
Testing & Optimization: How do I unit-test an AI agent's tool calls in CI/CD, and how do I fine-tune a small, fast Gemma 4B model with LoRA on Vertex AI so it performs like a 27B model?

I wrote this article to serve as a complete, end-to-end Enterprise Reference Blueprint that answers all four questions using a single, working project (CloudGuard) verified in Google Cloud Shell (all working code is available in my cloudguard-gemma-agent GitHub repository).

Where This Blueprint Is Useful in Production

DevOps, SRE & Automated Incident Response: On-call engineers often waste 20–30 minutes during an outage manually querying logs, searching old postmortems, and finding remediation runbooks. CloudGuard automatically pulls live incident telemetry from Cloud Firestore (get_service_incident), matches symptoms against historical postmortems in its Long-Term Memory Bank (MEM-2026-014), and executes approved mitigations in staging (mitigate_runaway_workload).
Cloud FinOps & GPU Quota Governance: A single misconfigured ML batch job or unthrottled retry loop can spin up dozens of GPUs overnight. CloudGuard checks real-time regional GPU utilization (NVIDIA L4 and RTX PRO 6000) and monthly budget burn (check_gpu_quota_and_cost), immediately scaling down runaway staging workloads and calculating hourly dollar savings ($36.12/hr).
Regulated Enterprises Requiring Data Sovereignty (Banking, Healthcare, Government, SecOps): Organizations prohibited from sending internal incident logs, credentials, or infrastructure schemas to external third-party APIs can deploy open-weights Gemma inside their own Google Cloud project behind --no-allow-unauthenticated IAM and Model Armor guardrails (cloudguard_agent/guardrails.py), where accidentally pasted GCP API keys (AIza...) are redacted before reaching the model.
AI Engineering & MLOps CI/CD Pipelines: Instead of relying on manual "vibe checks" when prompts or tools change, my ADK Eval harness (scripts/run_eval.py) scores Tool Trajectory Accuracy (1.00) and Response Factuality (1.00) on every build, while my Vertex AI QLoRA pipeline (scripts/prepare_and_verify_lora.py) lets teams replace a 54 GB VRAM model (Gemma 27B) with an 8.9 GB VRAM model (Gemma 4B + LoRA) running in ~51ms on a single NVIDIA L4 GPU.

Here are the six stages of the production lifecycle I will walk through:

Build: Scaffolding a model-agnostic Google ADK agent in Python that runs against local Gemma weights (via Ollama) during development and seamlessly switches to Cloud Run GPUs or Vertex AI Model Garden in production.
Connect: Authoring native Python function-calling tools backed by Cloud Firestore to inspect live production incidents and regional GPU FinOps quotas.
Scale: Deploying vLLM on Cloud Run Serverless GPUs (NVIDIA L4 / RTX PRO 6000) with Cloud Storage FUSE for zero-download weight streaming, paired with Short-Term Session State and a Long-Term Memory Bank.
Govern: Enforcing defense-in-depth security with private Cloud Run IAM service accounts, Model Armor secret redaction (before_model_callback), and deterministic RBAC tool policies (before_tool_callback).
Optimize: Automating CI/CD release gates with ADK Eval (tool_trajectory_avg_score and response_match_score) and OpenTelemetry tracing for Google Cloud Trace.
Specialize: Fine-tuning a compact google/gemma-3-4b-it model using QLoRA on Vertex AI so it matches gemma-3-27b-it tool-calling accuracy (1.00) at 2.8x faster latency and 6x lower GPU memory.

Figure 2: End-to-end request lifecycle for my CloudGuard Incident & FinOps Agent across Cloud Run, vLLM, Cloud Firestore, Model Armor guardrails, and Cloud Trace.

Part 1: Platform Foundations & Scaffolding the First Gemma Agent

Prerequisites & Google Cloud Setup

To follow along with my setup, you will need:

A Google Cloud Project with billing enabled.
The Google Cloud SDK (gcloud) installed and authenticated.
Python 3.10+ and uv (the fast Python package and virtual environment manager I use across this project).
Ollama installed locally (ollama pull gemma3:4b or ollama pull gemma3:27b) for zero-cost local prototyping before deploying to Google Cloud GPUs.

First, I set my project and region environment variables and enabled the required Google Cloud APIs:

export PROJECT_ID="your-gcp-project-id" export REGION="us-central1" gcloud config set project $PROJECT_ID gcloud config set compute/region $REGION gcloud services enable \ aiplatform.googleapis.com \ run.googleapis.com \ artifactregistry.googleapis.com \ firestore.googleapis.com \ cloudtrace.googleapis.com \ iam.googleapis.com

Next, I configured Application Default Credentials (ADC) so my local Python environment can authenticate with Google Cloud services:

gcloud auth application-default login

Scaffolding the Project (Local or 1-Click in Google Cloud Shell)

You can either clone and run this project directly inside Google Cloud Shell (which comes pre-authenticated with gcloud and Python in your browser) by clicking Open in Google Cloud Shell, or initialize the workspace locally with uv:

Step 1: Launching the RobinaMirbahar/cloudguard-gemma-agent repository directly inside Google Cloud Shell.

Step 2: Authorizing Google Cloud Shell credentials while the repository clones into the Cloud Shell workspace.

Step 3: Executing the CloudGuard environment setup, database seeding, and ADK evaluation commands inside the Google Cloud Shell terminal.

mkdir cloudguard-gemma-agent && cd cloudguard-gemma-agent uv init

I structured the repository so that agent orchestration, native tools, memory management, security guardrails, and evaluation suites are cleanly separated (and runnable offline via a deterministic local simulation harness before touching cloud GPUs):

cloudguard-gemma-agent/ ├── pyproject.toml ├── .env.example ├── cloudguard_agent/ │ ├── __init__.py │ ├── agent.py # Core ADK LlmAgent & Gemma LiteLLM configuration │ ├── tools.py # Native Python tools backed by Cloud Firestore │ ├── memory.py # Short-Term Session & Long-Term Memory Bank │ ├── guardrails.py # Model Armor & RBAC policy callbacks │ └── finetuning.py # Vertex AI QLoRA dataset & job spec builder ├── deploy/ │ ├── Dockerfile.vllm │ ├── Dockerfile.vllm.lora │ └── deploy_cloud_run_gpu.sh ├── scripts/ │ ├── seed_firestore.py │ ├── simulate_parts.py │ ├── run_eval.py │ └── prepare_and_verify_lora.py └── tests/ └── eval_dataset.json

Repository structure of RobinaMirbahar/cloudguard-gemma-agent showing the modular separation across agent orchestration, Firestore tools, memory, guardrails, deployment manifests, and evaluation scripts.

In pyproject.toml, I declared google-adk, litellm, google-cloud-firestore, and opentelemetry-sdk:

[project] name = "cloudguard-gemma-agent" version = "0.1.0" description = "Enterprise SecOps & FinOps Triage Agent powered by Open-Weights Gemma & Google ADK on Cloud Run GPUs" readme = "README.md" requires-python = ">=3.10" dependencies = [ "google-adk>=1.0.0",  "google-genai>=1.0.0",  "google-cloud-firestore>=2.19.0",  "litellm>=1.60.0",  "pydantic>=2.10.0",  "python-dotenv>=1.0.1",  "opentelemetry-api>=1.29.0",  "opentelemetry-sdk>=1.29.0", ]

Then I locked and installed the dependencies:

uv sync

Decoupling Local vs. Cloud GPU Inference (`.env.example`)

One of the strongest architectural benefits of Google ADK is its model-agnostic LiteLlm integration. By defining a .env configuration file, I can point my agent to a local Ollama server on localhost:11434 during development and switch to a self-hosted vLLM endpoint on Cloud Run GPUs (or Vertex AI Model Garden) in production without changing a single line of agent orchestration code:

# Google Cloud Project Settings GOOGLE_CLOUD_PROJECT="cloudguard-enterprise-demo" GOOGLE_CLOUD_LOCATION="us-central1" # Gemma Inference Backend Mode: "local_ollama" | "cloud_run_vllm" GEMMA_BACKEND_MODE="local_ollama" # Local Prototyping via Ollama OLLAMA_API_BASE="http://localhost:11434" GEMMA_MODEL_NAME="ollama_chat/gemma3:27b" # Production Cloud Run GPU vLLM Endpoint GEMMA_VLLM_ENDPOINT_URL="https://gemma-vllm-service-xyz-uc.a.run.app/v1" GEMMA_VLLM_MODEL_ID="google/gemma-3-27b-it" # Firestore Configuration FIRESTORE_DATABASE="(default)" USE_LOCAL_MOCK_DB="true"

Defining the CloudGuard Agent Persona (`cloudguard_agent/agent.py`)

When building agents with open-weights models like Gemma, explicit, numbered system instructions are essential. Here is how I defined the agent's identity, tool bindings, and strict operational constraints in cloudguard_agent/agent.py:

import os from google.adk.agents import LlmAgent from google.adk.models.lite_llm import LiteLlm from .tools import ( check_gpu_quota_and_cost, get_service_incident, list_active_alerts, mitigate_runaway_workload, ) CLOUDGUARD_SYSTEM_INSTRUCTION = """You are CloudGuard, an Enterprise SecOps & FinOps Incident Response Agent running on Google Cloud and powered by open-weights Gemma. Your primary responsibilities: 1. Incident Triage: Retrieve real-time operational and FinOps telemetry using `get_service_incident` and `list_active_alerts`. 2. GPU FinOps & Capacity: Verify regional NVIDIA L4 and RTX PRO 6000 GPU availability and monthly budget burn using `check_gpu_quota_and_cost`. 3. Safe Remediation: When authorized, mitigate runaway staging workloads using `mitigate_runaway_workload`. 4. Enterprise Governance Compliance: - NEVER execute automated scaling or termination in `production` without explicit `sre-incident-commander` escalation. - Always cite the exact Incident ID, affected service name, hourly USD burn rate, and recommended runbook ID in your response. """ def build_adk_root_agent() -> LlmAgent: backend_mode = os.getenv("GEMMA_BACKEND_MODE", "local_ollama") if backend_mode == "cloud_run_vllm": model_id = f"openai/{os.getenv('GEMMA_VLLM_MODEL_ID', 'google/gemma-3-27b-it')}" api_base = os.getenv("GEMMA_VLLM_ENDPOINT_URL") else: model_id = os.getenv("GEMMA_MODEL_NAME", "ollama_chat/gemma3:27b") api_base = os.getenv("OLLAMA_API_BASE", "http://localhost:11434") gemma_model = LiteLlm(model=model_id, api_base=api_base) return LlmAgent( name="cloudguard_gemma_agent", model=gemma_model, description="Enterprise SecOps and FinOps incident triage agent powered by Gemma.", instruction=CLOUDGUARD_SYSTEM_INSTRUCTION, tools=[ get_service_incident, list_active_alerts, check_gpu_quota_and_cost, mitigate_runaway_workload, ], ) root_agent = build_adk_root_agent()

Part 2: Native Python Tools & Connecting to Cloud Firestore

Figure 3: How Google ADK introspects my Python type hints and docstrings to drive structured Gemma tool calls against Cloud Firestore.

An isolated LLM knows nothing about active Cloud Run services, internal SRE runbooks, or current monthly GPU FinOps burn rates. To bridge that gap, Google ADK automatically converts standard Python functions into JSON Schema tool definitions for Gemma.

My Engineering Rule for Open-Weights Gemma: Compact open-weights models (Gemma 4B, 12B, 27B) rely heavily on explicit Args: descriptions and concrete format examples (such as 'INC-YYYY-NNNN') in Python docstrings to construct valid JSON tool arguments. I treat Python docstrings as first-class prompt engineering.

Implementing the Operational Tools (`cloudguard_agent/tools.py`)

I built four native Python tools backed by Cloud Firestore (plus a zero-config local JSON schema mirror (USE_LOCAL_MOCK_DB="true") so the entire test suite can be verified locally before provisioning cloud resources):

get_service_incident(incident_id: str): Retrieves severity, symptom, GPU type, hourly USD burn rate, and remediation runbook for a specific incident ticket.
list_active_alerts(environment: str = "all"): Lists open incidents filtered by production, staging, or all.
check_gpu_quota_and_cost(region: str): Checks regional NVIDIA L4 and RTX PRO 6000 quota limits, active usage, and monthly FinOps budget percentage.
mitigate_runaway_workload(...): Scales down a runaway Cloud Run service or batch job and records the hourly cost savings.

def get_service_incident(incident_id: str) -> dict[str, Any]: """Retrieves full operational and FinOps telemetry for a specific incident ID. Use this tool whenever a user asks about a specific incident ticket (e.g., 'INC-2026-0891' or 'INC-2026-0904'), its symptoms, severity, or recommended runbook. Args: incident_id: The unique incident identifier formatted as 'INC-YYYY-NNNN'. Returns: A dictionary containing the incident status, severity, affected service, GPU usage, hourly burn rate, and recommended remediation runbook. """ use_mock = os.getenv("USE_LOCAL_MOCK_DB", "true").lower() == "true" normalized_id = incident_id.strip().upper() if not use_mock: from google.cloud import firestore db = firestore.Client(project=os.getenv("GOOGLE_CLOUD_PROJECT")) doc = db.collection("incidents").document(normalized_id).get() if not doc.exists: return {"status": "error", "message": f"Incident {normalized_id} not found in Firestore."} return {"status": "success", "incident": doc.to_dict()} db_data = _load_db() incident = db_data.get("incidents", {}).get(normalized_id) if not incident: return { "status": "error", "message": f"Incident '{normalized_id}' not found. Valid IDs: {list(db_data.get('incidents', {}).keys())}", } return {"status": "success", "incident": incident} def check_gpu_quota_and_cost(region: str) -> dict[str, Any]: """Checks current Cloud Run GPU quota utilization and monthly FinOps spend for a GCP region. Use this tool when evaluating whether a region has available NVIDIA L4 or RTX PRO 6000 GPUs or when checking monthly budget consumption. Args: region: The Google Cloud region identifier (e.g., 'us-central1', 'europe-west4'). Returns: A dictionary with GPU quota limits, active usage, and budget utilization metrics. """ db_data = _load_db() clean_region = region.strip().lower() quota_info = db_data.get("quotas", {}).get(clean_region) if not quota_info: return { "status": "error", "message": f"No quota telemetry found for region '{clean_region}'. Supported: us-central1, europe-west4.", } l4_free = quota_info["nvidia_l4_limit"] - quota_info["nvidia_l4_in_use"] budget_pct = round( (quota_info["current_month_spend_usd"] / quota_info["monthly_budget_usd"]) * 100, 1 ) return { "status": "success", "region": clean_region, "available_nvidia_l4_gpus": l4_free, "budget_utilization_percent": budget_pct, "details": quota_info, }

Seeding the Database (`scripts/seed_firestore.py`)

When connecting to a live Cloud Firestore instance in Google Cloud, you can provision the database in Native mode:

gcloud firestore databases create \ --location=us-central1 \ --type=firestore-native

I then ran scripts/seed_firestore.py to populate the incidents and quotas collections:

uv run python scripts/seed_firestore.py

Here is the terminal output from running the database seeder:

========================================================================== CloudGuard Database Seeder | Project: cloudguard-enterprise-demo Target Mode: Local JSON Store (Firestore Schema Mirror) ========================================================================== [OK] Wrote seed collections to: .../cloudguard-gemma-agent/data/mock_firestore.json Seeded Collection Summary [incidents]: INCIDENT ID SEVERITY ENVIRONMENT SERVICE NAME BURN ($/HR) -------------------------------------------------------------------------- INC-2026-0891 P2 staging ml-batch-indexer-dev $42.50 INC-2026-0904 P1 production payment-gateway-prod $18.00 INC-2026-0912 P3 production gemma-rag-worker-prod $4.80 Seeded Collection Summary [quotas]: REGION L4 IN USE / LIMIT RTX 6000 IN USE / LIMIT MONTHLY SPEND -------------------------------------------------------------------------- us-central1 12/16 1/4 $3,840 / $5,000 europe-west4 2/8 0/2 $910 / $2,500 ==========================================================================

Running scripts/seed_firestore.py inside Google Cloud Shell to populate the incidents and quotas collections.

Part 3: Scaling with Cloud Run Serverless GPUs, Vertex AI Model Garden & Dual Memory

Figure 4: Streaming Gemma weights from Cloud Storage FUSE into Cloud Run Serverless GPUs with vLLM, backed by Short-Term Session State and a Long-Term Enterprise Memory Bank.

Moving an open-weights Gemma model from a developer laptop into enterprise production on Google Cloud offers two primary deployment patterns: Cloud Run Serverless GPUs with vLLM (for scale-to-zero serverless containers) or Vertex AI Model Garden (for one-call managed GPU endpoints).

Option A: Serverless GPU Serving on Cloud Run with vLLM & Cloud Storage FUSE

Instead of baking multi-gigabyte model weights into a container image — or downloading them from Hugging Face on every cold start — I stage the Gemma weights in a regional Google Cloud Storage (GCS) bucket and mount that bucket at /mnt/models using Cloud Storage FUSE.

VRAM Sizing Note on Google Cloud Run GPUs: A single NVIDIA L4 GPU on Cloud Run provides 24 GB of VRAM, which easily fits google/gemma-3-4b-it (~8.4 GB in bfloat16), google/gemma-3-12b-it, or a 4-bit AWQ quantized google/gemma-3-27b-it. To run unquantized bfloat16 google/gemma-3-27b-it (~54 GB VRAM) on a single serverless instance, use --gpu-type=nvidia-rtx-pro-6000 on Cloud Run or deploy on Vertex AI Model Garden.

Here is my deploy/Dockerfile.vllm configuration:

FROM vllm/vllm-openai:latest ENV HF_HOME=/mnt/models \ VLLM_WORKER_MULTIPROC_METHOD=spawn \ PORT=8080 ENV MODEL_PATH="/mnt/models/gemma-3-27b-it" EXPOSE 8080 ENTRYPOINT ["python3", "-m", "vllm.entrypoints.openai.api_server"] CMD ["--model", "/mnt/models/gemma-3-27b-it", \ "--served-model-name", "google/gemma-3-27b-it", \ "--host", "0.0.0.0", \ "--port", "8080", \ "--dtype", "bfloat16", \ "--max-model-len", "8192", \ "--gpu-memory-utilization", "0.92", \ "--enable-auto-tool-choice", \ "--tool-call-parser", "pythonic"]

And here is the gcloud run deploy command from deploy/deploy_cloud_run_gpu.sh mounting the GCS FUSE volume:

gcloud run deploy gemma-vllm-service \ --image="us-docker.pkg.dev/${PROJECT_ID}/cloudguard-repo/gemma-vllm:latest" \ --region="${REGION}" \ --service-account="${VLLM_SA}" \ --no-allow-unauthenticated \ --cpu=8 \ --memory=32Gi \ --gpu=1 \ --gpu-type=nvidia-l4 \ --no-gpu-zonal-redundancy \ --min-instances=0 \ --max-instances=3 \ --concurrency=8 \ --add-volume="name=model-weights,type=cloud-storage,bucket=${MODEL_BUCKET},readonly=true" \ --add-volume-mount="volume=model-weights,mount-path=/mnt/models" \ --port=8080

Option B: Managed Deployment via Vertex AI Model Garden

Alternatively, if you prefer a managed endpoint provisioned directly from Vertex AI Model Garden, you can deploy google/gemma3@gemma-3-4b-it or google/gemma3@gemma-3-27b-it programmatically using the Vertex AI Python SDK:

import vertexai from vertexai import model_garden vertexai.init(project="your-gcp-project-id", location="us-central1") # Official Vertex AI Model Garden identifier for Gemma 3 open_model = model_garden.OpenModel("google/gemma3@gemma-3-4b-it") endpoint = open_model.deploy( machine_type="g2-standard-12", accelerator_type="NVIDIA_L4"Source：https://dev.to/gde/building-production-ready-open-weights-ai-agents-with-gemma-google-adk-and-cloud-run-gpus-2mfgMy favorites

PreviousWhy we hand over the domain, code and accountsNextSpreadsheet or Invoicing App? An Honest Checklist for Freelancers

`Related posts`


Copilot Fusion Takes Local Models, and the Offline Boundary Is Somewhere Else
GitHub published an experimental preview on October 28, 2026 that lets Copilot Fusion add eligible local models to its s
0dev.to AI
A CTO Checklist for Evaluating AI Features Before Production Release
An AI feature that works during a product demonstration is not necessarily ready for production. It may generate inaccur
0dev.to AI
Humanoid Robot Demo Makes the Case for Keeping the Human Form
Humanoid Robot Demo Makes the Case for Keeping the Human Form A new demonstration video of a humanoid robot has reignite
0dev.to AI
Supabase anon key vs service role key: what each one opens
TL;DR. The anon key, now called the publishable key, is meant to be public. It maps to the anon or authenticated role, a
0dev.to Security
After the eMudhra CA compromise: audit your own TLS stack in 30 minutes
On 17 October 2026 a researcher told the root programs that the private keys of eight eMudhra intermediate certificate a
0dev.to Security
Exposed Building Automation Services: Bavaria and DACH
On 02.11.2026 I used Shodan to measure how much building automation in Germany is directly reachable from the internet. 
0dev.to Security