AI Agent Mastery
1. Agent Architecture Patterns
Single Agent Pattern
┌──────────────────────────────────────────────┐
│ Single Agent │
│ ┌─────────┐ ┌─────────┐ ┌──────────────┐ │
│ │ LLM │ │ Tools │ │ Memory │ │
│ │ (Brain) │──│(Hands) │──│ (Context) │ │
│ └─────────┘ └─────────┘ └──────────────┘ │
│ │ │ │ │
│ └──────────────┴─────────────┘ │
│ │ │
│ User Interface │
│ (Chat/CLI/API) │
└──────────────────────────────────────────────┘
When to use: Simple tasks, single-domain, no parallelism needed
Example: Basic Q&A bot, simple tool-user
Multi-Agent Orchestration Pattern
┌──────────────────────────────────────────────────────────┐
│ Orchestrator Agent │
│ ┌────────────────────────────────────────────────────┐ │
│ │ Task Queue → Decompose → Route → Merge → Respond │ │
│ └────────────────────────────────────────────────────┘ │
│ │ │ │ │ │
│ ┌────┘ ┌─────┘ ┌──────┘ ┌─────┘ │
│ ▼ ▼ ▼ ▼ │
│ ┌─────┐ ┌─────┐ ┌─────────┐ ┌──────────┐ │
│ │Worker│ │Worker│ │Researcher│ │Code Agent│ │
│ │ #1 │ │ #2 │ │ #3 │ │ #4 │ │
│ └─────┘ └─────┘ └─────────┘ └──────────┘ │
│ │ │ │ │ │
│ └───────────┴──────────┴─────────────┘ │
│ │ │
│ Result Summary │
└──────────────────────────────────────────────────────────┘
When to use: Complex multi-domain tasks, parallelism needed
Example: Research + code + review pipeline, data pipeline
Agent Spawning (delegate_task)
User Request
│
▼
Orchestrator decomposes
│
├── delegate_task(goal="Research X", toolsets=["web"])
│ └── Subagent researches → returns summary
│
├── delegate_task(goal="Write code for Y", toolsets=["terminal","file"])
│ └── Subagent codes → returns final code
│
└── delegate_task(goal="Review code Z", toolsets=["terminal","file"])
└── Subagent reviews → returns report
│
▼
Orchestrator merges results → Final response
Delegation Limits
┌─────────────────────────────────────────────────┐
│ Parameter │ Value │ Notes │
├─────────────────────────────────────────────────┤
│ Max concurrent │ 3 │ Per user │
│ Max spawn depth │ 1 │ No nesting │
│ Subagent memory │ None │ Fresh context │
│ Subagent tools │ Restricted │ Whitelist only │
│ Timeout │ 600s │ 10 min max │
│ Return format │ Summary │ Not raw output │
└─────────────────────────────────────────────────┘
2. MCP (Model Context Protocol) Mastery
MCP Architecture
┌──────────────────────────────────────────────────────────┐
│ MCP Architecture │
│ │
│ ┌──────────────┐ ┌──────────────┐ │
│ │ Hermes Agent │────▶│ MCP Client │ │
│ │ (Host) │ │ (Native) │ │
│ └──────────────┘ └──────┬───────┘ │
│ │ stdio / HTTP │
│ ┌──────────┴──────────┐ │
│ │ │ │
│ ┌─────▼─────┐ ┌──────▼──────┐ │
│ │ MCP Server│ │ MCP Server │ │
│ │ (Filesystem)│ │ (Database) │ │
│ │ Tools: │ │ Tools: │ │
│ │ - read │ │ - query │ │
│ │ - write │ │ - insert │ │
│ │ - search │ │ - update │ │
│ └────────────┘ └─────────────┘ │
└──────────────────────────────────────────────────────────┘
MCP Server Configuration
# ~/.hermes/config.yaml — MCP servers section
mcp:
servers:
# Filesystem access
- name: filesystem
command: npx
args: ["-y", "@modelcontextprotocol/server-filesystem", "/root/workspace"]
# Web fetch
- name: fetch
command: npx
args: ["-y", "@modelcontextprotocol/server-fetch"]
# PostgreSQL database
- name: postgres
command: npx
args: ["-y", "@modelcontextprotocol/server-postgres"]
env:
POSTGRES_URL: "postgresql://user:pass@localhost:5432/mydb"
# Custom server
- name: my-custom-server
command: python
args: ["-m", "my_mcp_server"]
env:
API_KEY: "${MY_API_KEY}"
Building Custom MCP Servers
# my_mcp_server/__init__.py — Custom MCP server with FastMCP
from fastmcp import FastMCP
mcp = FastMCP("my-custom-server")
@mcp.tool()
def get_server_status(server_ip: str) -> str:
"""Get health status of a server."""
import subprocess
result = subprocess.run(
["ssh", server_ip, "uptime; df -h / | tail -1"],
capture_output=True, text=True, timeout=30
)
return result.stdout
@mcp.tool()
def deploy_config(server_ip: str, config_path: str) -> str:
"""Deploy a configuration file to a remote server."""
import subprocess
result = subprocess.run(
["rsync", "-avz", config_path, f"{server_ip}:/opt/app/config.yaml"],
capture_output=True, text=True, timeout=60
)
return result.stdout
@mcp.resource()
def server_list() -> str:
"""List all servers in the fleet."""
return """US:51.83.223.88:9377
TR:51.83.223.88:9378
PK-C2:51.83.223.88:9379
PK-C3:51.83.223.88:9380
HK-UK2:152.114.193.86:18789"""
if __name__ == "__main__":
mcp.run()
MCP Server Quality Checklist
┌──────────────────────────────────────────────────────────────────┐
│ Checkpoint │ Description │ Required │
├──────────────────────────────────────────────────────────────────┤
│ Outcome-oriented tools │ Focus on what, not how │ ✅ Every tool │
│ Flat parameters │ No nested objects │ ✅ All params │
│ Actionable errors │ isError + details │ ✅ Error cases │
│ Token-efficient output │ Concise responses │ ✅ Under 2K │
│ Composable outputs │ Structured data │ ✅ JSON/table │
│ Idempotent operations │ Same input = same out│ ✅ Write ops │
│ Dry-run support │ Preview before exec │ ✅ Destructive │
│ Type-safe schemas │ Zod/Pydantic types │ ✅ All params │
└──────────────────────────────────────────────────────────────────┘
3. ACP (Agent Communication Protocol)
ACP Transaction Flow
┌───────────────────────────────────────────────────────────┐
│ ACP Transaction │
│ │
│ Client Agent Provider Agent │
│ ┌──────────┐ ┌──────────┐ │
│ │ │───REQUEST────────▶│ │ │
│ │ Intent │ │ Process │ │
│ │ │◀──RESPONSE──────────│ │ │
│ └──────────┘ └──────────┘ │
│ │ │ │
│ ▼ ▼ │
│ Verify Result Update Ledger │
│ (cryptographic (payment receipt) │
│ signature) │
└───────────────────────────────────────────────────────────┘
x402 Payment Protocol
┌───────────────────────────────────────────────────────────┐
│ x402 Payment Flow │
│ │
│ 1. Agent discovers paid API endpoint │
│ 2. Agent checks wallet balance (USDC on Base) │
│ 3. Agent sends HTTP request with x402 header │
│ ├── X-Payment: version=1, amount=0.01, asset=USDC │
│ ├── X-Payment-Address: 0x1234... │
│ └── X-Payment-Signature: 0xabcd... │
│ 4. Provider verifies payment on-chain │
│ 5. Provider returns requested data │
│ 6. Agent processes response │
│ │
│ Three-party settlement: │
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
│ │ Agent │────▶│ Facilitator│────▶│ Provider │ │
│ │ (Payer) │ │ (Escrow) │ │ (Payee) │ │
│ └──────────┘ └──────────┘ └──────────┘ │
│ $0.01 $0.001 $0.009 │
│ paid fee received │
└───────────────────────────────────────────────────────────┘
4. Memory Systems Deep Dive
Memory Architecture
┌─────────────────────────────────────────────────────────┐
│ Memory Hierarchy │
│ │
│ ┌───────────────────────────────────────────────────┐ │
│ │ L1: Conversation Context (current session) │ │
│ │ - Last few turns of conversation │ │
│ │ - Active skills, tools, workspace │ │
│ │ - Volatile, cleared on session end │ │
│ └───────────────────────────────────────────────────┘ │
│ │ │
│ ┌───────────────────────────────────────────────────┐ │
│ │ L2: Session Memory (session_search) │ │
│ │ - Searchable past conversations │ │
│ │ - LLM-generated summaries │ │
│ │ - Expires after ~90 days │ │
│ └───────────────────────────────────────────────────┘ │
│ │ │
│ ┌───────────────────────────────────────────────────┐ │
│ │ L3: Persistent Memory (memory tool) │ │
│ │ - User preferences (user target) │ │
│ │ - Environment facts (memory target) │ │
│ │ - Survives across sessions │ │
│ │ - Budget: ~4.4K tokens (keep compact) │ │
│ └───────────────────────────────────────────────────┘ │
│ │ │
│ ┌───────────────────────────────────────────────────┐ │
│ │ L4: Long-Term Memory (hindsight) │ │
│ │ - Semantic search across all stored facts │ │
│ │ - Entity resolution & knowledge graph │ │
│ │ - Reflect/reason across memories │ │
│ │ - Persistent, indexed, unlimited │ │
│ └───────────────────────────────────────────────────┘ │
│ │ │
│ ┌───────────────────────────────────────────────────┐ │
│ │ L5: Procedural Memory (skills) │ │
│ │ - Reusable workflows & approaches │ │
│ │ - SKILL.md format (frontmatter + markdown) │ │
│ │ - Auto-loaded per-turn when relevant │ │
│ │ - Persist via skill_manage │ │
│ └───────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────┘
When to Use Each Memory Type
What to store? ─┬─ User preference/habit ──→ memory (user target)
├─ Environment fact (IPs, paths) ──→ memory (memory target)
├─ Procedural workflow ──→ skill_manage (create skill)
├─ Temporary task state ──→ todo tool
├─ Past conversation ──→ session_search (recall)
├─ Important fact (retrievable) ──→ hindsight_retain
├─ Semantic search needed ──→ hindsight_recall
├─ Cross-memory reasoning ──→ hindsight_reflect
└─ Will be stale in <7 days ──→ DON'T STORE
Memory Anti-Patterns
❌ BAD: Store everything in memory
→ Budget overflows, irrelevant facts injected
❌ BAD: Store imperative instructions
→ "Always respond in Urdu" — becomes directive, overrides user
❌ BAD: Store temporary state
→ "PR #123 merged" — stale in days, use session_search
❌ BAD: Duplicate across memory types
→ Same fact in memory AND hindsight — wastes tokens
✅ GOOD: Declarative facts
→ "User prefers Roman Urdu" — survives, doesn't override
✅ GOOD: Environment facts
→ "HK ICE-1 runs CamoFox on port 9377" — stable, useful
✅ GOOD: Procedural knowledge as skills
→ Complex workflows → SKILL.md → auto-loaded
5. Prompt Engineering for Agents
System Prompt Design Patterns
┌──────────────────────────────────────────────────────────┐
│ Pattern │ Best For │ Example │
├──────────────────────────────────────────────────────────┤
│ Role + Constraint │ Focused task │ "You are a │
│ │ │ senior SRE. │
│ │ │ Respond in │
│ │ │ bullet pts." │
├──────────────────────────────────────────────────────────┤
│ Few-shot Examples │ Consistent fmt │ "Input: X │
│ │ │ Output: Y │
│ │ │ Input: A │
│ │ │ Output: B" │
├──────────────────────────────────────────────────────────┤
│ Chain-of-Thought │ Reasoning │ "Think step │
│ │ │ by step." │
├──────────────────────────────────────────────────────────┤
│ React Pattern │ Tool use │ "Reason, │
│ │ │ then act, │
│ │ │ then observe" │
├──────────────────────────────────────────────────────────┤
│ Structured Output │ JSON/XML needed │ "Respond in │
│ │ │ JSON: {key}" │
└──────────────────────────────────────────────────────────┘
Task requires? ─┬─ File operations ──→ write_file, read_file, patch, search_files
├─ Code execution ──→ execute_code (batch 3+ calls)
├─ Shell commands ──→ terminal (foreground or background)
├─ Web browsing ──→ browser_navigate, browser_snapshot, browser_click
├─ Research ──→ web_search, browser, delegate_task with web toolset
├─ Memory ──→ memory, hindsight_retain, session_search
├─ Skills ──→ skill_view, skill_manage, skills_list
├─ Multi-step ──→ delegate_task (subagent with isolated context)
├─ Scheduled ──→ cronjob (create, list, update)
├─ User input ──→ clarify (with choices)
└─ Communication ──→ send_message (WhatsApp/Telegram/Discord)
6. Multi-Agent Orchestration
Orchestration Patterns
┌───────────────────────────────────────────────────────────┐
│ Pattern 1: Sequential Pipeline │
│ Agent_A → Agent_B → Agent_C → Final │
│ Use: Research → Write → Review │
│ │
│ Pattern 2: Fan-out/Fan-in (Parallel) │
│ ┌→ Agent_A ─┐ │
│ Orchestrator ─→ Agent_B ──→ Merge → Final │
│ └→ Agent_C ─┘ │
│ Use: Multi-source research, parallel code review │
│ │
│ Pattern 3: Hierarchical Delegation │
│ Orchestrator → Manager_A → Worker_1 │
│ → Worker_2 │
│ Use: Complex projects with sub-projects │
│ │
│ Pattern 4: Council (Adversarial) │
│ ┌→ Agent_A (pro) ─┐ │
│ Judge ← Agent_B (con) ──← Final │
│ └→ Agent_C (neut) ─┘ │
│ Use: Decision making, code review, design critique │
│ │
│ Pattern 5: Event-Driven (Cron) │
│ Cron triggers Agent → Agent processes → Sends notification │
│ Use: Monitoring, daily reports, alerts │
└───────────────────────────────────────────────────────────┘
Delegation Best Practices
# ✅ GOOD: Specific goal, relevant context, appropriate tools
delegate_task(
goal="Research the latest Python 3.13 features and create a summary document",
context="Focus on performance improvements and new syntax. Output in markdown.",
toolsets=["web", "file"]
)
❌ BAD: Vague goal, no context, too many tools
delegate_task(
goal="Research something",
toolsets=["web", "terminal", "file", "browser", "vision"]
)
✅ GOOD: Parallel tasks with independent contexts
delegate_task(tasks=[
{"goal": "Research Python 3.13 features", "toolsets": ["web"]},
{"goal": "Research Rust 2024 features", "toolsets": ["web"]},
{"goal": "Compare both in a table", "context": "Use research from tasks 1 and 2"}
])
7. Evaluation & Quality Frameworks
LLM Council Evaluation (10 Dimensions)
┌──────────────────────────────────────────────────────────────┐
│ Dimension │ What It Measures │ Weight │ 10pts │
├──────────────────────────────────────────────────────────────┤
│ 1. Design & │ Structural soundness│ Architecture │ 10 │
│ Architecture │ patterns, modularity │ decisions │ │
│ 2. Flowcharts & │ Visual decision │ Diagrams, │ 10 │
│ Diagrams │ trees, architecture │ flow charts │ │
│ 3. Implementation │ Real code examples, │ Copy-paste │ 10 │
│ Patterns │ copy-paste ready │ quality │ │
│ 4. Language & │ Breadth of topics, │ Coverage │ 10 │
│ Coverage │ tools covered │ depth │ │
│ 5. Security │ Security practices, │ Hardening, │ 10 │
│ │ secrets, hardening │ vuln aware │ │
│ 6. Testing │ Test approaches, │ Validation, │ 10 │
│ Strategy │ load, chaos, unit │ edge cases │ │
│ 7. Edge Cases & │ Error handling, │ Gotchas, │ 10 │
│ Pitfalls │ failure modes │ pitfalls │ │
│ 8. Performance │ Tuning, scaling, │ Efficiency, │ 10 │
│ Optimization │ efficiency │ benchmarks │ │
│ 9. Deployment │ Production deploy, │ Rollout │ 10 │
│ │ rollout strategies │ strategies │ │
│ 10. Documentation │ Clarity, structure, │ References, │ 10 │
│ │ references │ examples │ │
├──────────────────────────────────────────────────────────────┤
│ TOTAL │ │ │ 100 │
│ Threshold: 80+/100 │ PASS │ │ │
│ Mastery: 95+/100 │ EXCELLENT │ │ │
└──────────────────────────────────────────────────────────────┘
Scoring with OpenRouter
import json, subprocess
def score_skill(skill_name, content, model="google/gemini-2.0-flash-001"):
"""Score a skill using LLM council evaluation."""
prompt = f"""You are an expert evaluator scoring skills for an AI agent.
Evaluate on 10 dimensions (10 points each, 100 total):
Design & Architecture
Flowcharts/Diagrams
Implementation Patterns
Language/Coverage
Security
Testing Strategy
Edge Cases & Pitfalls
Performance Optimization
Deployment
Documentation
Reply ONLY in this format:
DESIGN: N/10
FLOWCHARTS: N/10
IMPLEMENTATION: N/10
COVERAGE: N/10
SECURITY: N/10
TESTING: N/10
EDGE_CASES: N/10
PERFORMANCE: N/10
DEPLOYMENT: N/10
DOCUMENTATION: N/10
TOTAL: N/100
SKILL CONTENT:
{content}"""
result = subprocess.run([
"curl", "-s", "--max-time", "180",
"https://openrouter.ai/api/v1/chat/completions",
"-H", f"Authorization: Bearer {OPENROUTER_KEY}",
"-H", "Content-Type: application/json",
"-d", json.dumps({
"model": model,
"messages": [{"role": "user", "content": prompt}],
"max_tokens": 500,
"temperature": 0.1
})
], capture_output=True, text=True, timeout=200)
return parse_scores(json.loads(result.stdout))
8. Production Deployment
Agent Service Architecture
┌──────────────────────────────────────────────────┐
│ Production Deployment │
│ │
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
│ │ Nginx │ │ Agent │ │ Redis │ │
│ │ (SSL) │──▶│ Gateway │──▶│ (Cache) │ │
│ │ :443 │ │ :8788 │ │ :6379 │ │
│ └──────────┘ └────┬─────┘ └──────────┘ │
│ │ │
│ ┌────────┴────────┐ │
│ │ │ │
│ ┌────▼────┐ ┌─────▼────┐ │
│ │ Browser │ │ LLM │ │
│ │ Stack │ │ Providers│ │
│ │ :9377-78│ │ (API) │ │
│ └─────────┘ └──────────┘ │
│ │
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
│ │ MariaDB │ │ Hindsight│ │ WA Bridge│ │
│ │ :3306 │ │ :9077 │ │ :9375 │ │
│ └──────────┘ └──────────┘ └──────────┘ │
└──────────────────────────────────────────────────┘
Systemd Service Best Practices
# /etc/systemd/system/hermes-gateway.service
[Unit]
Description=Hermes Agent Gateway
After=network.target
Wants=redis.service
[Service]
Type=simple
User=root
WorkingDirectory=/usr/local/lib/hermes-agent
ExecStart=/usr/local/bin/hermes gateway start
Restart=always
RestartSec=10
KillMode=control-group # Kills gateway + bridge.js together
Environment=NODE_ENV=production
Environment=HERMES_LOG_LEVEL=info
Resource limits
LimitNOFILE=65536
MemoryHigh=4G
MemoryMax=8G
[Install]
WantedBy=multi-user.target
Monitoring & Alerting
#!/bin/bash
/opt/scripts/agent-health-check.sh
Run via cron every 5 minutes
GATEWAY_URL="http://localhost:8788/health"
ALERT_WEBHOOK="https://your-webhook-url"
Check gateway
if ! curl -sf "$GATEWAY_URL" > /dev/null; then
curl -sf "$ALERT_WEBHOOK" -d '{"text":"🚨 Hermes Gateway DOWN!"}'
systemctl restart hermes-gateway
fi
Check disk
DISK_USAGE=$(df -h / | tail -1 | awk '{print $5}' | sed 's/%//')
if [ "$DISK_USAGE" -gt 80 ]; then
curl -sf "$ALERT_WEBHOOK" -d "{\"text\":\"⚠️ Disk usage at ${DISK_USAGE}%\"}"
fi
Check memory
MEM_USAGE=$(free | grep Mem | awk '{print int($3/$2*100)}')
if [ "$MEM_USAGE" -gt 85 ]; then
curl -sf "$ALERT_WEBHOOK" -d "{\"text\":\"⚠️ Memory usage at ${MEM_USAGE}%\"}"
fi
9. Testing Agents
Agent Testing Decision Tree
What to test? ─┬─ Tool invocation ──→ Unit test with mocked tools
├─ Prompt adherence ──→ Eval rubric with golden outputs
├─ Multi-step reasoning ──→ Integration test with real LLM
├─ Error recovery ──→ Fault injection test
├─ Memory persistence ──→ Session restart test
├─ Subagent delegation ──→ Isolation + result verification
├─ Cron job execution ──→ Schedule test + verify output
└─ Browser automation ──→ Screenshot diff test
Test Patterns
# Unit test: Tool invocation
def test_memory_add():
result = memory(action="add", target="user", content="Test preference")
assert result["status"] == "success"
Integration test: Multi-step agent execution
def test_agent_research_flow():
result = delegate_task(
goal="Research Python 3.13 features",
toolsets=["web"]
)
assert result["status"] == "completed"
assert len(result["summary"]) > 100
Eval rubric: Prompt adherence
def test_prompt_follows_instructions():
response = agent.chat("List 3 things about Python. Respond in JSON.")
data = json.loads(response) # Should be valid JSON
assert len(data) == 3
10. Troubleshooting & Best Practices
Agent Not Responding
Agent not responding? ─┬─ Gateway down? ──→ systemctl restart hermes-gateway
├─ Provider error? ──→ Check config.yaml, test API key
├─ Context overflow? ──→ Clear session, reduce skill count
├─ Tool timeout? ──→ Increase timeout, check network
└─ Bridge disconnected? ──→ Restart WA bridge, re-scan QR
Memory Issues
Memory issues? ─┬─ Budget overflow? ──→ Keep entries under 500 chars
├─ Wrong target? ──→ user for preferences, memory for facts
├─ Stale data? ──→ Don't store data expiring <7 days
└─ Duplicate entries? ──→ Use replace with old_text, not add
Skill Not Loading
Skill not loading? ─┬─ Invalid YAML? ──→ Validate frontmatter with yaml.safe_load
├─ No name/description? ──→ Add frontmatter fields
├─ Over 100KB? ──→ Split into multiple skills
├─ Wrong directory? ──→ Check ~/.hermes/skills/ vs optional-skills/
└─ Gateway not restarted? ──→ systemctl restart hermes-gateway
┌───────────────────────────────────────────────────────────────────┐
│ Practice │ Impact │ Priority │
├───────────────────────────────────────────────────────────────────┤
│ Keep memory under 4.4K │ Stable context │ CRITICAL │
│ Use delegate_task for 3+ │ -70% context usage │ HIGH │
│ tool calls │ │ │
│ Batch with execute_code │ -60% round-trips │ HIGH │
│ Use patch over write_file │ Smaller diffs │ HIGH │
│ Use search_files over grep │ Faster + less output│ MEDIUM │
│ Read files with limit/offset │ Don't read 100K │ MEDIUM │
│ Lazy-load skills per-turn │ -20K context │ MEDIUM │
│ Use background terminal │ Non-blocking ops │ LOW │
│ Cache browser snapshots │ -90% tokens │ LOW │
│ Compress old sessions │ -50% context │ LOW │
└───────────────────────────────────────────────────────────────────┘