Part 2: 4-Gate AST Verification & Static Call Inventories (4B Edge NPU vs. Frontier)

In Part 1, we introduced SpaghettiLens-AI (v0.5.0), showing how running a compact 4B parameter model (qwen3-4b-instruct-2507) on a local Snapdragon NPU exposed multi-tier orchestration weaknesses in agentic frameworks.

Following community feedback—specifically noting that models of different scales may pass the same routing tests while altering how they chain tool calls—we packaged our verification gates and a static call-inventory extractor into the repository (dev branch).

This article documents our two public synthetic fixtures, our 4 scoped verification gates, an AST-based static call inventory extractor, and an exploratory comparison between our local 4B NPU setup and Google's gemini-3.8-flash.

1. What Is Reproducible Offline vs. Live

Because readers evaluating this repository will not generally have access to a local Qualcomm Snapdragon X Elite NPU (npurun) and may not wish to run 100 paid cloud API evaluations, we split our benchmark scripts into two distinct modes:

1. Offline Gate Verifier & Telemetry Replay (benchmarks/run_eval.py --mode dry-run): Runs in <0.2s without hardware or API keys. It executes the real DeterministicSandboxVerifier against representative candidate code templates (VALID_REFACTORED_SERVICE, FLOAT_HAZARD_CANDIDATE, SQL_INJECTION_CANDIDATE, and TRUNCATED_SYNTAX_CANDIDATE) and replays pre-recorded baseline parameters (BASELINE_PROFILES) across 5 configurations $\times$ 2 fixtures $\times$ 10 iterations (100 simulated fixture-runs) using a fixed pseudo-random seed (--seed 42).
2. Single Illustrative Live Cloud Run (benchmarks/run_live_cloud_trace.py): Connects to Google AI Studio's OpenAI-compatible endpoint (gemini-3.8-flash, temperature=0.0) to execute a single live refactoring task (N=1) and extract its static call inventory.

Use plain text when copying the commands below into your terminal:

git clone https://github.com/giuseppecamiolo-dev/spagnettilens.git cd spagnettilens git checkout dev # 1. Run the offline gate verifier and replay harness (5 configs x 2 fixtures x 10 runs = 100 traces) python3 benchmarks/run_eval.py --mode dry-run --runs 10 --seed 42 # 2. Run the offline unit test suite python3 -m pytest tests/ -v # 3. (Optional) Execute a single illustrative live run against gemini-3.8-flash export GEMINI_API_KEY="your_api_key" python3 benchmarks/run_live_cloud_trace.py 

Methodological Note: Passing --seed 42 makes the offline dry-run replay deterministic across machines. It does not make live LLM API calls deterministic; live runs via run_live_cloud_trace.py are subject to provider-side sampling and network variance.

2. Public Synthetic Fixtures & The 4 Scoped Verification Gates

To test modernization logic without exposing proprietary code, benchmarks/run_eval.py includes two small synthetic fixtures:

• legacy_billing.pl (Perl 5 CGI/DBI snippet): Contains inline SQL interpolation (INSERT INTO ledger_entries (user_id, total) VALUES ($uid, $total)) and IEEE 754 floating-point currency math ($total = $subtotal * (1 + $tax_rate)).
• legacy_auth_order.php (PHP 5.6 snippet): Contains raw $_POST string concatenation inside mysql_query().

A candidate Python microservice only scores as a Pass if it clears four sequential checks in DeterministicSandboxVerifier:

1. Gate 1 — Syntactic & Callable Validity: Verifies that the generated Python string parses cleanly via ast.parse() and defines at least one callable function (ast.FunctionDef).
2. Gate 2 — Scoped SQL Construction Heuristic: Walks the AST to reject ast.JoinedStr (f-strings) and ast.BinOp (+ or % concatenation) whose string literals match SELECT|INSERT|UPDATE|DELETE|DROP, and checks for the presence of SQLAlchemy/bind substrings ("text(", "select(", or ":uid"). Scope limitation: This is a narrow static heuristic for the tested fixtures, not a complete SQL-injection proof. It can miss unsafe queries constructed through helper functions or .format(), and its substring check is specific to our target schema.
3. Gate 3 — Scoped Financial Precision Heuristic: Walks the AST to ban float(...) calls and checks that Decimal and ROUND_HALF_UP appear in the source.
4. Gate 4 — Isolated Behavioral & Input-Validation Check: Executes calculate_invoice_total(subtotal, tax_rate) inside an isolated exec() namespace against test vectors:
   • Decimal Quantization: Evaluates Decimal("0.05") * (Decimal("1.00") + Decimal("0.10")) = Decimal("0.055"), verifying that .quantize(Decimal("0.01"), rounding=ROUND_HALF_UP) occurs on the final gross calculation to return Decimal("0.06").
   • Service-Boundary Input Validation: Verifies that negative values ("-10.00") and non-numeric strings (such as '100.00; DROP TABLE users;--') raise a ValueError when parsed via Decimal(str(subtotal_str)). Note: Rejecting a SQL-formatted string inside calculate_invoice_total tests numeric input parsing at the service boundary, not database parameterization.

3. Exploratory Five-Configuration Summary

The table below shows the summary output produced by benchmarks/run_eval.py --mode dry-run --runs 10 --seed 42, where each row represents 20 fixture-runs (10 repetitions $\times$ 2 fixtures).

Data Provenance:

• Rows 1–4 (qwen3-4b-instruct-2507): Parameterized in BASELINE_PROFILES from our local development experiments on a Qualcomm Snapdragon X Elite NPU (npurun, INT4 AWQ, configured with max_seq_len=4096), testing 10 repetitions of each synthetic fixture.
• Row 5 (gemini-3.8-flash): Token count (7,532 cumulative input tokens and 366 output tokens across 2 steps: 2,387 in / 142 out on Step 1, 5,145 in / 224 out on Step 2) was captured from a live execution of run_live_cloud_trace.py on legacy_billing.pl in November 2026 via Google AI Studio's OpenAI-compatible endpoint (https://generativelanguage.googleapis.com/v1beta/openai/, temperature=0.0). The 19/20 pass rate and 14.2s latency in BASELINE_PROFILES are simulated reference baselines in the dry-run harness, as client-side retries and free-tier rate limits (HTTP 503) caused wall-clock step durations during our live Gemini run to exceed 750 seconds per step.

*Row 5 pass rate and latency are dry-run reference parameters in BASELINE_PROFILES; the token count (7,532 input + 366 output = 7,898 total tokens) comes from our single live verification run (N=1).

Configuration Caveat: To operate within the 4,096-token configured context limit on the local NPU, the Hardened 3-Tier qwen3-4b setup uses custom code-first Jinja2 prompts and a sliding-window memory pruner. Therefore, the local and cloud configurations do not share strictly identical prompts or memory state.

4. Extracting Static Call Inventories (CodeAgent)

When inspecting a smolagents CodeAgent, reading ActionStep.tool_calls only returns the internal executor wrapper:

Tool Trajectory Recorded: ['python_interpreter', 'python_interpreter'] 

Because CodeAgent writes and executes Python snippets directly, calls to registered tools (extract_sql_schema, synthesize_safe_fastapi_service) appear as Python function calls inside step.code_action.

To inspect which tools the model invoked in code, benchmarks/run_live_cloud_trace.py implements extract_static_call_inventory(agent):

def extract_static_call_inventory(agent) -> list[str]: known_callables = ( set(getattr(agent, "tools", {}).keys()) | set(getattr(agent, "managed_agents", {}).keys()) | {"final_answer"} ) inventory: list[str] = [] for step in agent.memory.steps: if isinstance(step, ActionStep) and getattr(step, "code_action", None): try: tree = ast.parse(step.code_action) calls = [ node for node in ast.walk(tree) if isinstance(node, ast.Call) and isinstance(node.func, ast.Name) and node.func.id in known_callables ] calls.sort(key=lambda n: (getattr(n, "lineno", 0), getattr(n, "col_offset", 0))) for call_node in calls: inventory.append(call_node.func.id) except SyntaxError: pass return inventory 

Scope of this Extractor: Sorting ast.Call nodes by (lineno, col_offset) produces a static source-order call inventory, not a dynamic runtime execution trace. If a model writes conditional branches, loops, or helper functions inside step.code_action, static AST inspection will list calls in source order regardless of whether a branch executed at runtime.

Observed Step-Chaining Difference on legacy_billing.pl

In our live run of benchmarks/run_live_cloud_trace.py with gemini-3.8-flash:

• Step 1 (2,387 input tokens | 142 output tokens):

 extracted_schema = extract_sql_schema(legacy_code=legacy_code) print(extracted_schema) 

• Step 2 (5,145 input tokens | 224 output tokens):

 safe_code = synthesize_safe_fastapi_service(schema_json=extracted_schema) print(safe_code) final_answer(safe_code) 

• Extracted Static Call Inventory: ['extract_sql_schema', 'synthesize_safe_fastapi_service', 'final_answer'] across 2 steps (7,532 cumulative input tokens, 366 output tokens).

In this experiment, gemini-3.8-flash chained synthesize_safe_fastapi_service(...) and final_answer(...) inside a single Python block in Step 2. By contrast, under our local 4B configuration—where Jinja2 templates restrict agents to one tool call per step to avoid exceeding the 4,096-token context limit—the model invokes the synthesis tool in Step 2, observes the output, and calls final_answer in Step 3. Furthermore, the unconstrained Step 2 input size (5,145 tokens) illustrates the prompt-growth pressure that necessitated sliding-window pruning on our 4,096-token local setup.

5. Conclusion

From this exploratory evaluation, we draw two practical takeaways for our codebase:

1. Constrained Contexts Surface Prompt and Routing Overhead Early: Operating under a 4,096-token configured context limit on local 4B hardware forced us to confront prompt growth, truncated JSON outputs, and multi-tier routing visibility—issues that did not cause immediate failures when running the same task against a larger-context cloud model in our test.
2. Combine Scoped AST Gates with Static Call Inventories: Even when two configurations produce equivalent final code, inspecting the AST of step.code_action alongside scoped syntax, SQL, and Decimal checks reveals how different models and prompt constraints alter multi-step tool chaining.

Repository and benchmark scripts: github.com/giuseppecamiolo-dev/spagnettilens (dev branch).