Give your AI agent live web access with Firecrawl

A model does not have the live page. It has whatever you pasted, plus whatever it memorized when it was trained. If the answer depends on today's docs, a price table, or a changelog, you have to fetch the page and put the text in the prompt. Firecrawl is a fetch built for that job. You send a URL. You get markdown, or HTML, instead of a browser you have to drive yourself.

This tutorial does three things. It calls the current scrape API from Node. It connects the hosted Firecrawl MCP server to an MCP client such as Cursor, so the agent can call scrape without you writing a wrapper. It then compares that request shape with an Apify Actor already listed on IQ.net, the Website Content Crawler. Apify has no affiliate link on IQ.net. The only Apify link in this guide is that catalogue entry. There is no button to the Apify homepage.

The Firecrawl link on this page includes an offer of 10% off the first purchase.

Scrape one URL from Node

Install the current SDK. The package name is firecrawl.

npm install firecrawl

Create a key in the Firecrawl dashboard and put it in the environment as FIRECRAWL_API_KEY. Do not commit the key and do not paste it into a chat. The SDK reads the variable, or you can pass apiKey to the constructor.

import { Firecrawl } from "firecrawl";

const firecrawl = new Firecrawl({ apiKey: process.env.FIRECRAWL_API_KEY });

const doc = await firecrawl.scrape("https://example.com", {
  formats: ["markdown"],
});

const markdown = doc.markdown ?? "";
if (!markdown) {
  console.dir(doc, { depth: 4 });
  throw new Error("No markdown field. Read the logged object and use the field that holds the page text.");
}
console.log(markdown.slice(0, 1500));

Save that as scrape.mjs and run it with node scrape.mjs. example.com is a good first target because the page is tiny and you can see whether the client is authenticated. Firecrawl's docs also show constructing the client with no key for a rate-limited subset of methods, including scrape. Use a key as soon as the call is part of something you run on a schedule. A keyless call is a demo, not a production path, and the limit is Firecrawl's, not something this page can quote.

If doc.markdown is empty but the log shows the text nested under another property, keep the log line and read that property. SDK releases have moved response fields before. The contract you should rely on is the one in the request: you asked for markdown, and the body contains the page in that format. Pin the firecrawl package version in package.json once you know which field your installed version returns.

The same call over HTTP, which is what you want from a language without the SDK, or from the n8n HTTP Request node:

curl -sS -X POST "https://api.firecrawl.dev/v2/scrape" \
  -H "Authorization: Bearer $FIRECRAWL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url":"https://example.com","formats":["markdown"]}'

A 401 means the bearer token is missing or is not a Firecrawl key. A JSON body without url is a validation error. Read the error message. Do not retry a 401 in a loop.

Open Firecrawl

What to ask for

formats: ["markdown"] is the right default for an agent. Markdown keeps headings and links and drops most of the navigation chrome. Ask for html as well only when you still need a table structure the markdown conversion flattened. Asking for both costs more of the response size. Start with markdown, and add a second format when you have a page that proves you need it.

Use scrape when you already know the URL. That is the common agent case: the user pasted a link, or a previous step produced one. Firecrawl also documents crawl, for walking a section of a site, and map, for listing URLs, plus search. Those are different tools. A crawl with a high page limit will run longer and return more text than you can put in one prompt. If you crawl, set a limit, then pick the few URLs the agent actually needs and scrape those.

Pass a page you are allowed to read. Firecrawl will not fix a URL that requires a login you do not have, and you should not hand it credentials in the query string. If the page is empty in a browser until a button is clicked, look at Firecrawl's documented scrape actions rather than pretending the first HTML snapshot is the whole document. Actions are a list of steps such as a click. Use them when you have seen the page and know the selector. Do not invent selectors.

A small wrapper keeps agents from sending the entire internet:

export async function readPage(url) {
  const parsed = new URL(url);
  if (parsed.protocol !== "https:") throw new Error("Only https URLs");
  const doc = await firecrawl.scrape(parsed.href, { formats: ["markdown"] });
  const markdown = (doc.markdown ?? "").slice(0, 20000);
  return { url: parsed.href, markdown };
}

The slice is a budget, not a claim about Firecrawl's maximum. Twenty thousand characters is already a lot of prompt. Log the original length if you truncate, so you can tell a short page from a page you cut.

Connect the MCP server

MCP is the path that lets the agent call Firecrawl itself. Firecrawl documents a hosted server. For a client you configure by hand, point it at https://mcp.firecrawl.dev/v2/mcp and send the key as a bearer header. In Cursor, that is an entry in ~/.cursor/mcp.json or in .cursor/mcp.json inside the project. Prefer the user-level file if the key should not sit in a repository.

{
  "mcpServers": {
    "firecrawl": {
      "url": "https://mcp.firecrawl.dev/v2/mcp",
      "headers": {
        "Authorization": "Bearer fc-YOUR_API_KEY"
      }
    }
  }
}

Do not put the key in the URL, and do not put it in the agent chat. If the client can sign in through a browser, Firecrawl documents a second endpoint, https://mcp.firecrawl.dev/v2/mcp-oauth, with no header. Use that when a person is present to approve the connection. Use the bearer header when the client must run without a browser prompt.

There is also a local stdio server, npx -y firecrawl-mcp, which reads FIRECRAWL_API_KEY from the environment. Use the hosted URL unless you have a reason to run the process yourself. One Firecrawl server entry is enough. Two entries with the same tools confuse the client about which key is live.

Firecrawl's own setup notes say a keyless connection can expose a smaller tool list, including scrape and search, and a key unlocks the wider set, subject to the account. After you save the config, open the client's MCP tool list and confirm firecrawl is connected before you ask it to fetch anything. A missing tool is a config problem. Retrying the prompt will not create the server.

A prompt that stays inside what the tool can do:

Use the Firecrawl scrape tool on https://example.com and return markdown.
Do not invent page content. If the tool errors, show the error and stop.

For a docs site, a second prompt is map, then a short list, then scrape:

Use Firecrawl to map https://docs.example.com and list up to 15 URLs.
I will pick which ones to scrape. Do not crawl the whole site.

That pause is the difference between a useful agent and a bill you did not expect. The model is bad at knowing when to stop walking links.

Open Firecrawl

How this differs from an Apify crawl

Apify is a different shape, and IQ.net lists it as catalogue records rather than as an affiliate. The comparison here is the Website Content Crawler, an Actor whose description on the index is crawling sites and extracting text for models and RAG pipelines. IQ.net has not run that Actor. The listing is the publisher's description.

You use that Actor when the job is "start at this URL, follow links under this host, and give me a dataset of pages." You start a run, wait for it, and then read the dataset. Inputs are the Actor's own fields, typically a list of start URLs and a page cap. The result is many records, not one HTTP response. That is a good fit for building a corpus. It is a clumsy fit for "the user just pasted one link, answer now."

Firecrawl scrape is the second case. One URL, one document, the call returns when the page is ready. Firecrawl crawl is closer to the Actor when you do want many pages, but you still stay inside one API and one key. Pick the tool from the job shape:

Do not send the agent to an Apify marketing page from this tutorial. If you want another Actor, search the IQ.net catalogue and open that service record. The Website Content Crawler record is the one that matches this comparison.

Put it behind a real agent

A practical arrangement for a coding agent:

1. The MCP server is configured with the key outside the chat.
2. The system prompt says to scrape only URLs the user supplied or URLs from a previous tool result.
3. The agent quotes from the markdown and links back to the URL.
4. If the tool returns an error, the agent shows it. It does not fill the gap from memory and call it a quote.

For a backend job, skip MCP and call readPage from the section above. MCP is for an interactive client. A server should use the SDK or the REST call, with the key in the server environment, and should log the URL and the status, not the key.

Cache when the same URL is fetched again within a short window. Firecrawl documents its own caching behavior on the MCP path. Your application cache can be as simple as a map from URL to markdown with a timestamp. Invalidate it when the user says the page changed. Do not cache error bodies.

If you also run the Bright Data workflow from the n8n tutorial, keep the roles separate. Bright Data's Web Unlocker returns the page body and leaves cleaning to you. Firecrawl returns markdown aimed at the model. Use the unlocker when you already have an n8n workflow and you want control of the proxy zone. Use Firecrawl when the next consumer is a prompt. Using both on the same URL is a debugging step, not a design.

Failure checks

• Empty markdown on a page you can see in a browser. The content may be behind a click or a consent wall. Try a documented action, or fetch a more specific URL, such as the article itself rather than the homepage.
• The MCP tool list does not include Firecrawl. The JSON is in the wrong file, the header name is not Authorization, or a second config is overriding it. Reload the client after you save.
• The agent claims it read a page but never called a tool. Say so in the prompt: no tool call, no claim. Check the client's tool trace.
• A crawl that "never finishes." You omitted a page limit. Cancel it and set one.
• Secrets in the prompt. Rotate the key. The next config goes in the MCP file or the environment, not in the message.

The catalogue record for the tool is Firecrawl on IQ.net. The offer on the button is 10% off the first purchase.

Open Firecrawl