Key Takeaways
- Claude Code's built-in web fetch has significant limitations when dealing with JavaScript-heavy sites, anti-bot protections, and large-scale scraping tasks.
- Firecrawl solves these problems by returning scraped data in LLM-friendly markdown format with customizable schemas, reducing token usage and improving accuracy.
- Eight actions are available: scrape, crawl, search, extract, agent, map, batch scrape, and browser interact. Each fits different use cases depending on whether you have URLs and how much autonomy you want.
- Agent mode is the most powerful but uses the most credits. It autonomously decides which actions to take (search, extract, map) to fulfill your request.
- Browser Interact is a new feature that spins up live Chromium sessions for clicking, typing, and scrolling, similar to Playwright.
Concepts to Understand
Why Standard Web Fetch Fails
Claude Code's default web fetch only reads static HTML. Many modern websites render content dynamically with JavaScript or implement anti-bot protections (like CAPTCHAs and rate limiting). This means web fetch often returns empty shells or gets blocked entirely.
How Firecrawl Improves Data Extraction
Firecrawl converts complex web pages into clean markdown with a schema you define upfront. Instead of dumping thousands of lines of HTML into Claude Code's context window, you get only the structured fields you need (product name, price, rating, etc.).
Choosing the Right Action
- Scrape: You have a specific URL, want all content from that page
- Crawl: You have a starting URL, want to traverse the entire site
- Search: You do not have a URL, need Firecrawl to find and scrape
- Extract: You want structured JSON output from a page
- Agent: You want full autonomy, Firecrawl decides the approach
Open Source vs. Hosted
Firecrawl is open source, but the self-hosted version loses the proprietary "Fire Engine" (anti-bot bypass), agent mode, and browser interact. Self-hosting requires Docker knowledge and is best for simple scraping tasks that do not need anti-bot capabilities.
Performance Comparison
- SimilarWeb test: Firecrawl returned full competitive data in 42 seconds. Standard web fetch hung for 5+ minutes and returned nothing.
- Yellow Pages test: Firecrawl pulled 16 plumber results in 53 seconds. Web fetch hit repeated 403 errors.
- Amazon test: Firecrawl completed in 45 seconds vs. 5.5 minutes for standard web fetch.
Getting Started
- Create a Firecrawl account (free plan includes 500 credits)
- Install the Firecrawl CLI and skills into Claude Code
- Authenticate with your API key
- Use natural language prompts in Claude Code to invoke Firecrawl actions