Introduction

I've been experimenting with LLM agents that control Blender. The goal: have Claude generate a complete YouTube video—script, 3D scenes, animations, camera moves, and render—without human intervention. The result is a stack of Python components that bridge natural language to Blender's Python API. This post outlines the architecture, the code, and the hard limits I hit.

Problem

Blender is a GUI application with a massive Python API (bpy). An LLM can emit Python code, but directly executing it is dangerous and brittle. You need:

  • Safety: Arbitrary code execution can delete files or crash the system.
  • State management: Blender scenes are complex; the agent must track objects, materials, and timelines.
  • Iterative refinement: A single prompt won't produce a polished video. The agent must plan, execute, observe, and fix.
  • Rendering: Final output requires invoking Blender's render engine, which can take hours.

Naively asking Claude to "write a Blender script" yields code that often fails due to API changes or missing context.

Solution

I built a tool-using agent loop (inspired by ReAct-style prompting) where Claude calls a set of Python functions that manipulate Blender via bpy. The agent runs inside Blender's Python environment (or a subprocess with blender --background). The loop terminates when the agent calls a finish tool or exceeds a step limit.

Key components:

  1. Blender Bridge: A JSON-RPC server running inside Blender that executes whitelisted operations.
  2. Tool Definitions: Python functions decorated with metadata for Claude's tool-use API.
  3. Agent Loop: Sends conversation + tool results to Claude, parses tool calls, executes them, and appends results.
  4. Renderer: A final tool that triggers bpy.ops.render.render(animation=True).

This avoids eval of arbitrary code. Each tool is a specific, auditable function.

Implementation

1. Blender Bridge (inside Blender)

Run this script in Blender's scripting tab or via blender --background --python bridge.py. It listens on a local socket for JSON commands.

# bridge.py import bpy import json import socket import threading HOST = '127.0.0.1' PORT = 65432 # Whitelisted operations def op_create_cube(name, location): bpy.ops.mesh.primitive_cube_add(location=location) obj = bpy.context.active_object obj.name = name return {'name': obj.name, 'location': list(obj.location)} def op_set_camera(location, rotation): cam = bpy.context.scene.camera if not cam: bpy.ops.object.camera_add() cam = bpy.context.active_object bpy.context.scene.camera = cam cam.location = location cam.rotation_euler = rotation return {'camera': cam.name} def op_render(output_path, frame_start=1, frame_end=1): bpy.context.scene.frame_start = frame_start bpy.context.scene.frame_end = frame_end bpy.context.scene.render.filepath = output_path bpy.ops.render.render(animation=True) return {'rendered': output_path} OPERATIONS = { 'create_cube': op_create_cube, 'set_camera': op_set_camera, 'render': op_render, } def handle_client(conn): with conn: data = conn.recv(4096).decode('utf-8') if not data: return try: msg = json.loads(data) op = msg.get('operation') args = msg.get('args', {}) if op not in OPERATIONS: response = {'error': f'Unknown operation: {op}'} else: result = OPERATIONS[op](**args) response = {'result': result} except Exception as e: response = {'error': str(e)} conn.sendall(json.dumps(response).encode('utf-8')) def start_server(): with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s: s.bind((HOST, PORT)) s.listen() print(f'Bridge listening on {HOST}:{PORT}') while True: conn, _ = s.accept() threading.Thread(target=handle_client, args=(conn,)).start() # Start server in a background thread so Blender's UI remains responsive threading.Thread(target=start_server, daemon=True).start()

Security note: This bridge executes only whitelisted functions. Never expose it to untrusted networks. Bind to 127.0.0.1 and use authentication if needed.

2. Tool Definitions for Claude

Define tools in a format compatible with Anthropic's tool-use API.

# tools.py tools = [ { "name": "create_cube", "description": "Create a cube mesh at a given location", "input_schema": { "type": "object", "properties": { "name": {"type": "string"}, "location": {"type": "array", "items": {"type": "number"}, "minItems": 3, "maxItems": 3} }, "required": ["name", "location"] } }, { "name": "set_camera", "description": "Set the active camera's location and rotation (Euler angles in radians)", "input_schema": { "type": "object", "properties": { "location": {"type": "array", "items": {"type": "number"}, "minItems": 3, "maxItems": 3}, "rotation": {"type": "array", "items": {"type": "number"}, "minItems": 3, "maxItems": 3} }, "required": ["location", "rotation"] } }, { "name": "render", "description": "Render an animation to the given output path", "input_schema": { "type": "object", "properties": { "output_path": {"type": "string"}, "frame_start": {"type": "integer"}, "frame_end": {"type": "integer"} }, "required": ["output_path"] } } ]

3. Agent Loop

This loop sends messages to Claude, executes tool calls via the bridge, and appends results. It terminates when Claude calls a finish tool (not shown) or after max_steps.

# agent.py import json import socket import anthropic client = anthropic.Anthropic(api_key="YOUR_API_KEY") def call_bridge(operation, args): with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s: s.connect(('127.0.0.1', 65432)) s.sendall(json.dumps({'operation': operation, 'args': args}).encode('utf-8')) response = s.recv(4096).decode('utf-8') return json.loads(response) def run_agent(prompt, max_steps=20): messages = [{"role": "user", "content": prompt}] for step in range(max_steps): response = client.messages.create( model="claude-3-5-sonnet-20241022", max_tokens=1024, tools=tools, messages=messages ) # Append assistant's response  messages.append({"role": "assistant", "content": response.content}) tool_calls = [block for block in response.content if block.type == "tool_use"] if not tool_calls: print("No tool calls, ending.") break for tool_call in tool_calls: if tool_call.name == "finish": print("Agent finished.") return result = call_bridge(tool_call.name, tool_call.input) messages.append({ "role": "user", "content": [{ "type": "tool_result", "tool_use_id": tool_call.id, "content": json.dumps(result) }] }) else: print(f"Reached max steps ({max_steps}).")

4. Orchestration

Start Blender with the bridge, then run the agent from a separate process.

blender --background --python bridge.py & python agent.py

Key Takeaways

  • Tool whitelisting beats code generation: Instead of letting Claude write arbitrary bpy code, expose a small set of safe operations. This prevents catastrophic mistakes and makes debugging easier.
  • Explicit termination: The agent loop stops on a finish tool call or a step limit. Without this, it could run indefinitely.
  • Security first: Never eval LLM output. My bridge uses a whitelist and runs locally. If you must execute generated code, sandbox it (e.g., RestrictedPython or a container).
  • Iterative refinement is key: The agent can observe results (e.g., object locations) and adjust. For a full video, you'd add tools for keyframing, text, and compositing.
  • Rendering is the bottleneck: Blender renders can take hours. The agent should plan the entire scene before triggering a render.

This stack is a starting point. With more tools (e.g., add_keyframe, set_material), Claude can direct increasingly complex animations. The code is available on GitHub—link in comments.