How to Build an Autonomous Blog Writer with CrewAI and Python

Those who have just started writing, and even experienced writers, face this problem on a day-to-day basis.

They sit in front of a laptop screen for a couple of hours before rage-quitting over

• what to write,
• how to write it,
• which research papers are the best,
• and how to format the post.

The main problem is that writing requires patience, research, and good structure before it can be considered a solid piece of information.

To overcome this, we are going to learn how to automate the task and reduce the effort and hassle of writing.

We will also make the process more interesting, even for experienced writers.

Disclaimer: Think of AI as an assistant, not a replacement for your effort.

Please don't use these tools to spam blogs created purely for monetary gain.

What is CrewAI

CrewAI is an open-source framework and an enterprise platform used to build multi-agent AI systems that orchestrate and collaborate on a complex task, where each agent has a defined role for handling different parts of a job.

You can consider CrewAI as an organisation, agents as the employees of that organisation, and tools as the skill set of those agents.

Main Components of CrewAI

• Crew: A crew is a team of agents specialised in a single task. It can comprise a single agent or multiple agents, depending on the use case. Just as an organisation can have multiple departments, CrewAI can have multiple crews.
• Agents: Agents are the employees responsible for completing a specialised task, such as research, drafting, editing, etc.
• Tools: Tools are the helping hands of agents and carry out certain tasks, such as web search, API calls, etc.

Prerequisites

• AI API keys (paid) or Ollama local setup (free)
• A code editor (VS Code, Cursor, or similar)
• Concepts of generative AI and prompt engineering
• CrewAI basics
• Python basics

LLM Setup

CrewAI requires the configuration of an LLM, so gather the API keys and required details from your LLM provider and write them down in a .env file later, in the format below.

AZURE_OPENAI_API_KEY=your-azure-key AZURE_OPENAI_ENDPOINT=https://your-resource-name.openai.azure.com/ AZURE_OPENAI_API_VERSION=2024-10-21 AZURE_OPENAI_DEPLOYMENT=your-deployment-name 

For my development, I am going to use Ollama, a free, open-source tool for running LLMs locally.

You can use it too, and refer to my blog, How I Built My Own Voice Agent Prototype Using Ollama, ElevenLabs & Whisper?, for setting up Ollama on your local system.

OLLAMA_MODEL=llama3.1:8b OLLAMA_BASE_URL=http://localhost:11434 OTEL_SDK_DISABLED=true 

Python Setup

Once your LLM credentials are written down securely, we can proceed with setting up Python on our system.

Python is a programming language used for the development of many kinds of coding projects, and AI/ML is one of them.

You can download it by visiting the official website.

After downloading and setting up Python on your system, create an empty folder and open it inside your code editor.

Make sure you have installed the Python extension in the editor for better and faster development and debugging.

With that, the Python setup is complete.

To set up CrewAI, we need to install the dependencies of CrewAI in our project using the uv package manager.

Open the terminal and install the uv package manager using the command below.

#Windows powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex" #Linux curl -LsSf https://astral.sh/uv/install.sh | sh #MacOS curl -LsSf https://astral.sh/uv/install.sh | sh 

Once the package manager has installed, restart your terminal and verify it using the below command.

if the command fails, there might be an issue with the installation process

uv --version 

After installing the package manager, use the following command to set up the boilerplate of your project.

uv init 

After initialising, you need to install a few dependencies, so use the following command to download CrewAI and other libraries.

uv add crewai ddgs requests beautifulsoup4 python-dotenv markdown xhtml2pdf python-docx pyyaml 

CrewAI Setup

Once Python and all its libraries are in place, we can start building our project using CrewAI.

Use the following command to set up your blog writing flow.

crewai create 

After running the command, you will be prompted with a form where you need to provide basic information about the project you want to build with CrewAI.

I will cover each approach in a different blog.

For our use case, we are going to create a flow named Blogwriting.

What would you like to create? crew A team of AI agents working together → flow A deterministic workflow with full control over agents and crews tool A custom tool for the CrewAI Tool Repository skill An agent skill with instructions and optional assets template A remote project template from the CrewAI gallery Name of your flow › BlogWriting 

Once your CrewAI foundation is created, you will see a similar folder structure in your code editor.

AutonomousBlogWriting/ .venv/ .gitignore .python-version main.py pyproject.toml README.md uv.lock blogwriting/ .env .gitignore pyproject.toml README.md src/ blogwriting/ __init__.py main.py crew.py crews/content_crew config/ agents.yaml tasks.yaml content_crew.py tools/ __init__.py custom_tool.py 

Our first task is to set up the environment variables, so go to AutonomousBlogWriting/blogwriting/.env and paste your LLM credentials in the format I showed you above.

Code

Now that every element is in place, we are going to start combining all those things to build our project.

If you get stuck during this process, I recommend you fork my Git repo to get a better understanding of the code

GitHub — Autonomous Writer Git Repository

Config

To build an organisation, we first need employees who are specialists in their respective domains, so go to AutonomousBlogWriting/blogwriting/src/blogwriting/crews/content_crew/config and start updating the YAML files.

agents.yaml — In this file, you need to define all the agents (employees) you want to work with in your organisation.

Each agent has three main components:

• role — The title of your agent
• goal — The responsibilities of your agent
• backstory — short summary of how your agent will perform its task

We are going to create four agents: researcher, writer, editor, and formatter.

You can always refer to my Git repo for reference, as I have defined all four agents in my YAML file.

tasks.yaml — In this file, we are going to define all the respective tasks your agents will perform to generate the expected output.

An agent can have a single task or multiple tasks, depending on the use case.

We are going to create four tasks, one for each agent's goal.

Similar to agents, tasks also have three main components:

• description — A detailed structure of the steps the agent has to take to generate the desired results
• expected_output — The format in which the agent will generate the result. It can be normal text, a markdown file, code, etc.
• agent — The name of the agent you want this task to be assigned to.

Tool

Agents require tools to fetch details from the internet, make API calls, fetch data from databases, and much more.

We need to create a web search tool that lets us find the most relevant information about the topic the user is searching for.

For this, we are going to create a custom tool inside AutonomousBlogWriting/blogwriting/src/blogwriting/tools and name it web_search_tool.py.

It helps us fetch a dictionary of URLs related to the topic and then read the details from those URLs to create a new blog as the output.

from typing import Type import requests from bs4 import BeautifulSoup from crewai.tools import BaseTool from ddgs import DDGS from pydantic import BaseModel, Field class WebSearchInput(BaseModel): query: str = Field(..., description="The search query, e.g. 'latest trends in RAG'") max_results: int = Field(5, description="Number of results to return (1-10)") class WebSearchTool(BaseTool): name: str = "web_search" description: str = ( "Search the web for up-to-date information. Returns title, URL and snippet " "for each result. Use it to research topics and collect sources." ) args_schema: Type[BaseModel] = WebSearchInput def _run(self, query: str, max_results: int = 5) -> str: max_results = max(1, min(int(max_results), 10)) try: results = DDGS().text(query, max_results=max_results) except Exception as e: return f"Search failed: {e}. Try rephrasing the query or retry shortly." if not results: return "No results found. Try a broader or differently worded query." return "\n\n".join( f"[{i}] {r.get('title', '').strip()}\n" f"URL: {r.get('href', '')}\n" f"Snippet: {r.get('body', '').strip()}" for i, r in enumerate(results, 1) ) class PageReaderInput(BaseModel): url: str = Field(..., description="Full URL of the page to read (include https://)") max_chars: int = Field(4000, description="Maximum characters of page text to return") class PageReaderTool(BaseTool): name: str = "read_webpage" description: str = ( "Fetch a web page and return its main text. Use on a promising URL from " "web_search when the snippet isn't enough." ) args_schema: Type[BaseModel] = PageReaderInput def _run(self, url: str, max_chars: int = 4000) -> str: try: resp = requests.get( url, timeout=15, headers={"User-Agent": "Mozilla/5.0 (compatible; BlogFlowBot/1.0)"}, ) resp.raise_for_status() except Exception as e: return f"Could not fetch {url}: {e}" soup = BeautifulSoup(resp.text, "html.parser") for tag in soup(["script", "style", "nav", "footer", "header", "aside", "form"]): tag.decompose() text = " ".join(soup.get_text(separator=" ").split()) return text[:max_chars] + ("... [truncated]" if len(text) > max_chars else "") 

LLM and Crew

Everything around the project is set up.

Now we have to build our crew and flow, then merge them into main.py.

Start your Ollama server and create a file named config.py inside AutonomousBlogWriting/blogwriting/src/blogwriting.

import os from crewai import LLM from dotenv import load_dotenv load_dotenv() def get_llm(temperature: float = 0.4) -> LLM: return LLM( model=f"ollama/{os.getenv('OLLAMA_MODEL', 'llama3.1:8b')}", base_url=os.getenv("OLLAMA_BASE_URL", "http://localhost:11434"), temperature=temperature, ) 

We are going to call our LLM from this file and set it in our content_crew.py file, where we are going to define our agents and tasks in CrewAI and call them based on the agent role and task description.

Below is our content_crew.py file. You need to update it in AutonomousBlogWriting/blogwriting/src/blogwriting/crews/content_crew/content_crew.py.

from pathlib import Path import yaml from crewai import Agent, Crew, Process, Task from config import get_llm CONFIG_DIR = Path(__file__).parent / "config" AGENTS = yaml.safe_load((CONFIG_DIR / "agents.yaml").read_text(encoding="utf-8")) TASKS = yaml.safe_load((CONFIG_DIR / "tasks.yaml").read_text(encoding="utf-8")) def run_step(task_key: str, inputs: dict, tools=None, temperature=0.4, max_iter=None) -> str: task_cfg = dict(TASKS[task_key]) agent_key = task_cfg.pop("agent") # "agent: researcher" -> look up in agents.yaml  extra = {"max_iter": max_iter} if max_iter else {} agent = Agent( config=AGENTS[agent_key], tools=tools or [], llm=get_llm(temperature), allow_delegation=False, verbose=True, **extra, ) task = Task(config=task_cfg, agent=agent) crew = Crew(agents=[agent], tasks=[task], process=Process.sequential, verbose=True) return str(crew.kickoff(inputs=inputs).raw).strip() # fills {topic}, {draft}, etc. 

Exporters

exporters.py is a file that we are going to use to store the LLM response in our desired format, such as Markdown, DOCX, PDF, etc.

Create a new file named exporters.py in AutonomousBlogWriting/blogwriting/src/blogwriting.

For now, this file supports a limited number of formats that the LLM response can be converted to.

If you need more, you can update this file according to your needs.

import re from pathlib import Path from typing import Callable, Dict, Tuple import markdown as md_lib Meta = Dict[str, str] CSS = """ @page { size: A4; margin: 2cm; } body { font-family: Helvetica, Arial, sans-serif; font-size: 11pt; line-height: 1.5; color: #222; } h1 { font-size: 22pt; margin-bottom: 4pt; } h2 { font-size: 15pt; margin-top: 18pt; } h3 { font-size: 12pt; } a { color: #1a5fb4; } code, pre { font-family: Courier; font-size: 9.5pt; background-color: #f2f2f2; } blockquote { margin-left: 12pt; color: #555; } table { border-collapse: collapse; } td, th { border: 1px solid #bbb; padding: 4pt; } """ def split_front_matter(text: str) -> Tuple[Meta, str]: m = re.match(r"^---\s*\n(.*?)\n---\s*\n(.*)$", text.strip(), re.S) if not m: return {}, text.strip() meta: Meta = {} for line in m.group(1).splitlines(): if ":" in line: k, v = line.split(":", 1) meta[k.strip().lower()] = v.strip().strip("'\"") return meta, m.group(2).strip() def slugify(text: str) -> str: return re.sub(r"[^a-z0-9]+", "-", text.lower()).strip("-")[:60] or "blog-post" def _html_doc(meta: Meta, body: str) -> str: html_body = md_lib.markdown(body, extensions=["extra", "sane_lists", "tables", "fenced_code"]) title = meta.get("title", "Blog Post") return ( f"<html><head><meta charset='utf-8'><title>{title}</title>" f"<style>{CSS}</style></head><body>{html_body}</body></html>" ) def _to_md(meta: Meta, body: str, path: Path, raw: str) -> None: path.write_text(raw.strip() + "\n", encoding="utf-8") def _to_html(meta: Meta, body: str, path: Path, raw: str) -> None: path.write_text(_html_doc(meta, body), encoding="utf-8") def _to_pdf(meta: Meta, body: str, path: Path, raw: str) -> None: from xhtml2pdf import pisa with open(path, "wb") as f: status = pisa.CreatePDF(src=_html_doc(meta, body), dest=f, encoding="utf-8") if status.err: raise RuntimeError("PDF generation failed") def _to_txt(meta: Meta, body: str, path: Path, raw: str) -> None: text = re.sub(r"\[([^\]]+)\]\(([^)]+)\)", r"\1 (\2)", body) text = re.sub(r"^#{1,6}\s*", "", text, flags=re.M) text = re.sub(r"(\*\*|__|`)", "", text) path.write_text(text.strip() + "\n", encoding="utf-8") _INLINE = re.compile(r"(\*\*.+?\*\*|`.+?`|\*.+?\*)") def _add_runs(par, text: str) -> None: text = re.sub(r"\[([^\]]+)\]\(([^)]+)\)", r"\1 (\2)", text) for part in _INLINE.split(text): if not part: continue if part.startswith("**") and part.endswith("**") and len(part) > 4: par.add_run(part[2:-2]).bold = True elif part.startswith("`") and part.endswith("`") and len(part) > 2: par.add_run(part[1:-1]).font.name <...