Postmortem
$ tail -f /dev/regret — What 30 Days of Autonomous AI Agents Silently Cost Us
A real-world post-mortem on unbounded growth, silent cost leaks, and the architectural defaults nobody warns you about when running autonomous LLM agents at home.
A real-world post-mortem on unbounded growth, silent cost leaks, and the housekeeping patterns that only reveal themselves on day 30.
The Setup
It starts simply enough: you build a few Python agents that run on a schedule. One reads the news and emails you a sentiment report. One checks your stock portfolio every morning. One sends your kids a daily science fact over WhatsApp. A personal assistant that reads your Gmail, creates calendar events, and reminds you about appointments.
The whole stack is elegant:
OpenClaw Gateway (Node.js, port 18789) ← WhatsApp + Telegram delivery
↕
System cron (7 scheduled jobs) ← trigger Python agents
↕
Python agents (LangGraph pipelines) ← grok-4 + ollama for reasoning
↕
Email (Brevo SMTP) + WhatsApp + iMessage ← delivery channels
You test it for a week. Everything works. You put it on a Mac mini, enable auto-start, and forget about it.
Then day 30 arrives.
The Problems
Problem 1 — The Hourly Cost Bomb
The platform had a cron system for triggering AI conversations on a schedule. Three jobs had been configured at some point:
| Job | Schedule | What it did |
|---|---|---|
| Stock Trend Report | Every hour | Triggered a full grok-4 conversation |
| Daily Stock Summary | 8 AM daily | Triggered a full grok-4 conversation |
| Email Scanner | 9 AM daily | Triggered a full grok-4 conversation |
The hourly job alone was firing 24 times per day. Each conversation loaded the full workspace context — ~39,000 bytes of markdown files (~10,000 tokens) — before generating any response.
The math:
24 calls/day × 10,000 input tokens = 240,000 input tokens/day
From one background job. Before the model generated a single output token.
Meanwhile, Python agents were already running via system cron and covering the same tasks — using the local free ollama model. The AI cron jobs were entirely redundant.
Problem 2 — Context Bloat: The 198-Line Timezone File
Every session loaded 11 workspace markdown files totaling 39,278 bytes (~10,000 tokens) injected into the system prompt before the user types a single word.
The worst offender: TIMEZONE_CONFIG.md — 198 lines, 5,376 bytes — containing:
- A full table of Apple Reminder entries from March 2026
- UTC → CDT conversion examples
- Architecture diagrams of which cron jobs use which timezone
- A compliance checklist:
✅ All cron jobs use 'tz': 'America/Chicago'
All of this loaded into every session — including sessions about recipes or "what time is Shivi's tennis class."
The actual timezone information needed at runtime: one line.
America/Chicago, UTC-5 CDT
Problem 3 — Gateway Logs Growing at 580 KB/Day
The Node.js gateway writes two log files continuously:
~/.openclaw/logs/gateway.log → 7.6 MB
~/.openclaw/logs/gateway.err.log → 6.6 MB
14.2 MB in ~24 days of uptime. No rotation. No size limit.
Projected over a year: 8 GB of gateway logs from a process assumed to be invisible.
There was a complication: these files are held open by the running process. If you rm a file that a process has open, the process continues writing to the deleted inode. The file disappears from ls but the disk space is never reclaimed until the process exits.
Problem 4 — Session History Growing Forever
Every conversation creates a .jsonl file:
~/.openclaw/agents/main/sessions/
d53a6413-210b-4ad6-b25f-6f002a31a611.jsonl → 178 KB
3c9536ef-4e38-48f5-9adf-91275e64ccb3.jsonl → 136 KB
... 13 files, 7.7 MB total
Oldest file: March 18. No deletion ever.
At this trajectory: ~320 MB of session history per year. From a process assumed to be ephemeral.
Problem 5 — The 9.4 GB Orphan Log File
When a feature was decommissioned, its LaunchAgent was deleted. Its log destination was not.
~/Library/Logs/openclaw/personal-agent-bot-error.log → 9.4 GB
Nine-point-four gigabytes. From a process removed weeks earlier.
Problem 6 — The Expiring caffeinate
The Mac mini needed to stay awake 24/7 for scheduled jobs to run reliably. Someone had run:
caffeinate -s -t 300
-t 300 means: prevent sleep for 300 seconds. Five minutes. Then sleep prevention expires.
This left the machine potentially sleeping after 5 minutes of idle — silently dropping cron jobs at 3 AM while nobody was watching.
The Numbers
Before and After
| Metric | Before | After |
|---|
13 Best Practices for Running LLM Agents at Scale
Context Window
1. Every byte in always-loaded context is a cost multiplier. Workspace/system prompt files are loaded on every call. 39 KB × 24 background calls/day = 936 KB of redundant input per day. Keep them lean.
2. Prefer facts over explanations in always-loaded context.
Chicago tz = UTC-5 CDT is one line. A 198-line timezone document with architecture diagrams and compliance checklists is not system prompt material.
3. contextPruning only trims conversation history, not workspace.
Most LLM frameworks prune old tool results and messages from the conversation window. They don't touch your always-loaded system context. You have to manage that separately.
4. One source of truth per fact. When the same data lives in multiple workspace files, they drift out of sync and both consume tokens.
Log Management
5. Every service that logs needs a rotation policy on day one. Not day 30. Not when you notice the disk is 90% full. On day one.
6. Never rm a log file held open by a running process.
# Safe in-place truncation
tail -c 5242880 ~/.openclaw/logs/gateway.log > /tmp/gw.tmp
cat /tmp/gw.tmp > ~/.openclaw/logs/gateway.log
rm /tmp/gw.tmp
7. Zero-byte log files are safe to delete. A cron job that ran successfully and produced no output writes a zero-byte log. Clean them up.
8. Build --dry-run into your maintenance scripts.
Before the first real run of a housekeeping script, preview what it would delete.
Data Retention
9. Processed ID lists have natural expiry. An email UID from 3 weeks ago can never match a new email. Cap the list by count rather than implementing timestamp-based expiry.
10. Pipeline output directories are caches, not archives. After 30 days, it's archaeology. Keep 30 days inline; archive anything older if you need it.
11. Conversation session files have diminishing value with age. 30-day rolling retention is the standard. Older sessions are rarely read and are expensive to load if any tooling ever indexes them.
Cost Architecture
12. Every background cron job that calls an LLM is a cost multiplier by its frequency. Hourly LLM job = 24× daily cost. Quadruple-check the schedule before enabling anything that runs more than once per day with a cloud LLM.
13. System cron + local LLM for pipelines; cloud LLM for reasoning only. Data pipeline jobs (fetch → format → deliver) don't need reasoning. They need reliability. Run them with ollama locally. Reserve the cloud LLM for tasks that genuinely need judgment: conversation, triage, synthesis.
The Key Insight
The fix isn't clever engineering — it's discipline applied systematically:
- Every file that grows needs a ceiling.
- Every process that logs needs rotation.
- Every token in always-loaded context needs to earn its place.
- Cloud LLM calls need to be counted, not assumed to be free.
Build the housekeeping in on day one. You'll thank yourself on day 30.
Built on OpenClaw platform with grok-4 (xAI) + ollama (llama3.1:8b) + LangGraph + system cron on macOS.