smart-fetch

Fetch URLs and extract only what matters. For AI agents.

57KB page → 10KB. 81% reduction. $0.01 vs $0.07.

Why

AI agents waste tokens on irrelevant web content. The OpenAI chat completions docs are 57KB of markdown—but you only need the messages parameter format.

smart-fetch distills pages to just the content matching your intent:

• 77% average context reduction
• 50-84% cheaper than feeding raw markdown to Claude Opus
• Free fetching via playbooks, Firecrawl fallback for JS-heavy sites
• Preserves code verbatim xno hallucinated examples

Quick Start

# Install
bun install -g github:celebi/agent-web-suite/smart-fetch

# Use
smart-fetch "https://docs.stripe.com/api/authentication" "how to authenticate with API keys"

Benchmarks

Real results from production documentation sites:

Average: 77% reduction in 7.4s

See BENCHMARK.md for full analysis including cost breakdowns.

Cost

Per-fetch breakdown

500 pages/month comparison

Based on benchmark averages: 17.3KB raw → 3.5KB distilled (77% reduction)

Without smart-fetch (full markdown → Opus):

500 pages × 4,325 tokens avg = 2.16M input tokens
2.16M × $15/1M (Opus) = $32.44

With smart-fetch (distilled → Opus):

Gemini input  (reads full):     2.16M × $0.30/1M  = $0.65
Gemini output (distilled):      437K  × $2.50/1M  = $1.09
Opus input    (reads distilled): 437K  × $15/1M   = $6.56
                                          Total   = $8.30

The context window savings compound: those 1.72M tokens saved are tokens you're not paying for on every subsequent turn in your conversation.

How It Works

URL + Intent
     ↓
playbooks fetch (free)
     ↓
[sparse content?] → Firecrawl fallback ($0.015)
     ↓
Gemini 2.5 Flash distillation (~$0.003/10KB)
     ↓
Crystallized markdown (stdout)

Installation

Prerequisites

• Bun runtime
• playbooks CLI (for free fetching)

Install

bun install -g github:celebi/agent-web-suite/smart-fetch

Or clone:

git clone https://github.com/celebi/agent-web-suite.git
cd agent-web-suite/smart-fetch
bun install

Environment Variables

# Required - for content distillation
export GEMINI_API_KEY="your-gemini-api-key"

# Optional - fallback for JS-heavy pages
export FIRECRAWL_API_KEY="your-firecrawl-api-key"

Usage

smart-fetch "<url>" "<intent>"

Output goes to stdout. Status messages go to stderr.

Examples

# Extract auth setup from API docs
smart-fetch "https://docs.stripe.com/api/authentication" \
  "how to authenticate API requests with examples"

# Get specific function documentation
smart-fetch "https://react.dev/reference/react/useState" \
  "useState parameters, return value, and usage example"

# Extract installation steps
smart-fetch "https://bun.sh/docs/installation" \
  "installation commands for macOS"

Intent Tips

Specificity drives reduction quality:

Benchmarking

smart-fetch --metrics "<url>" "<intent>"

Returns JSON with timing, size metrics, and noRelevantContent flag.

Agent Integration

For Cursor or Claude Desktop, see SKILL.md for agent skill configuration that auto-triggers on URL + intent patterns.

Troubleshooting

License

MIT