Scraping APIs for AI agents: ScraperAPI vs Scrapingdog vs Browse AI

Agents that "read the web" usually need one of three shapes. The first is a URL in and HTML out, which you parse. The second is the same kind of fetch, exposed as MCP tools so the model can call it without you writing HTTP. The third is a robot you train once in a browser, then run by id. ScraperAPI, Scrapingdog, and Browse AI line up with those shapes. This is not a ranking. IQ.net does not score providers. It is a way to pick the shape that matches the job, with a request you can actually send.

Each name above goes to that product's page on IQ.net. The buttons go to the affiliate URLs stored for those listings. Do not swap in a homepage link with the tracking parameter removed. The parameter is the whole point of the affiliate URL, and a lookalike link is a different URL.

ScraperAPI: a URL in, HTML out

ScraperAPI's synchronous API is a GET. You pass api_key and url. The response body is the page. Their docs also describe an async endpoint, a proxy port, structured endpoints, and an MCP server. Start with the synchronous GET. It is the call an agent makes when it already knows the address.

curl --get "https://api.scraperapi.com" \
  --data-urlencode "api_key=$SCRAPERAPI_KEY" \
  --data-urlencode "url=https://example.com"

--data-urlencode matters. If you paste the target URL into the query string by hand, a target that already has ? and & can be parsed as extra ScraperAPI parameters. Their docs also say to keep ScraperAPI's own parameters before url when you build the string yourself. The curl above avoids that class of bug.

Two optional parameters show up constantly, and both are documented:

• render=true runs the page in a browser so JavaScript can execute. Leave it off until the HTML you got is an empty shell.
• country_code=us asks for a specific region. Use a code they document. Do not guess.

premium=true and session_number are also documented. Premium asks for a higher-quality proxy pool. A session number keeps a client on the same IP for a short series of requests. Their docs say sessions expire about 15 minutes after the last use. Use a session when the site ties a page to the previous request, not as a default.

Their documentation recommends a client timeout around 70 seconds so a slow target has time to return. Set that on the HTTP client. A 10 second timeout will look like ScraperAPI is down when the target was only slow.

From Node, the same request:

const target = new URL("https://api.scraperapi.com");
target.searchParams.set("api_key", process.env.SCRAPERAPI_KEY);
target.searchParams.set("url", "https://example.com");

const response = await fetch(target, { signal: AbortSignal.timeout(70000) });
if (!response.ok) {
  throw new Error(`ScraperAPI ${response.status}: ${await response.text()}`);
}
const html = await response.text();
console.log(html.slice(0, 500));

The key stays in the environment. The model can be given html after you strip it down, or you can extract the fields you need with a parser before the model sees anything. Do not ask the model to "figure out the API key."

ScraperAPI's docs also describe structured endpoints for sites they have a parser for, and an MCP server. Use a structured endpoint when you want fields instead of HTML and they publish one for that site. Use the raw GET when the page is arbitrary. The catalogue page is ScraperAPI.

Open ScraperAPI

Scrapingdog: the same shape, plus MCP tools

Scrapingdog's general web scraping API is also a GET. You pass the key, the target URL, and whether the fetch should execute JavaScript.

curl "https://api.scrapingdog.com/scrape?api_key=$SCRAPINGDOG_API_KEY&url=https://example.com&dynamic=false"

dynamic=false is the static fetch. If the body you get is a short shell and the text you care about only appears after JavaScript runs in a normal browser, call the same endpoint with dynamic=true. Do not flip it on for every URL. A rendered fetch is slower and you should spend it on pages that need it.

URL-encode the target if it contains a query string. In Node:

const endpoint = new URL("https://api.scrapingdog.com/scrape");
endpoint.searchParams.set("api_key", process.env.SCRAPINGDOG_API_KEY);
endpoint.searchParams.set("url", "https://example.com");
endpoint.searchParams.set("dynamic", "false");
const response = await fetch(endpoint, { signal: AbortSignal.timeout(90000) });
if (!response.ok) throw new Error(`Scrapingdog ${response.status}: ${await response.text()}`);
console.log((await response.text()).slice(0, 500));

The MCP server is why this product has a different kind of catalogue entry. IQ.net lists Scrapingdog as that server. The published package is scrapingdog-mcp. It reads SCRAPINGDOG_API_KEY from the environment and does not take the key as a tool argument, so the model does not see it. Scrapingdog's own write-up shows an npx launch. A client config looks like this:

{
  "mcpServers": {
    "scrapingdog": {
      "command": "npx",
      "args": ["-y", "scrapingdog-mcp"],
      "env": {
        "SCRAPINGDOG_API_KEY": "your_api_key_here"
      }
    }
  }
}

The server exposes the general scrape plus a set of site-specific endpoints Scrapingdog documents, including Google result pages. You do not need to memorize every tool name. Ask for the page, and let the client show you which tool it called. If the tool list is empty, the process did not start. Check that Node 18 or newer is on the path the client uses, and that the key is in env, not in the prompt.

A prompt that matches the tool:

Use the Scrapingdog tools to fetch https://example.com as static HTML.
Return the page title and the first paragraph. If the tool returns an empty shell, say so. Do not switch to a rendered fetch unless I ask.

The affiliate link for the product is the one with deal=iqnet. Documentation pages on the same site are reference material. The button is the link to give a person.

Open Scrapingdog

Browse AI: record the path, then run a task

Browse AI is the no-code shape. You teach a robot by clicking through a page in their recorder, or you start from a prebuilt robot. You name the text you want. After the robot is approved, the API runs it. The agent does not rediscover the selectors on every call. That is the right trade when the page has a click path, a menu, or a logged-in view you have already taught, and the wrong trade when every URL is new.

Their getting-started guide walks through a key and a task. The pieces you need:

1. Create the robot in the Browse AI dashboard and approve it. An unapproved robot will not run through the API.
2. Create an API key from the API tab. Store it as BROWSE_AI_KEY.
3. Copy the robot id.
4. POST a task. The input object must use the parameter names that robot expects. Their guide's example uses originUrl. If your robot's API snippet shows a different key, use that key. A plausible name that the robot does not define will not scrape anything useful.

curl -sS -X POST "https://api.browse.ai/v2/robots/$ROBOT_ID/tasks" \
  -H "Authorization: Bearer $BROWSE_AI_KEY" \
  -H "Content-Type: application/json" \
  -d '{"inputParameters":{"originUrl":"https://example.com"}}'

The response includes a task id and a status such as running. Fetch the task until it finishes:

curl -sS "https://api.browse.ai/v2/robots/$ROBOT_ID/tasks/$TASK_ID" \
  -H "Authorization: Bearer $BROWSE_AI_KEY"

On success, the extracted fields are under capturedTexts, using the names you gave them in the recorder. status tells you whether to keep polling. Poll on a timer of a few seconds, and stop after a bound you choose, rather than looping forever. A non-success status is the result. Show it to the person who asked. Do not substitute a guessed price or title.

You can also list robots with GET https://api.browse.ai/v2/robots and check GET https://api.browse.ai/v2/status while you are setting the key up. Both take the same bearer header. Webhooks exist if you would rather have Browse AI POST the finished task to you. Point the webhook at an endpoint you control, and verify the payload before you write it into a database.

Browse AI is listed at its IQ.net page. The dashboard is where the robot is trained. The API is how an agent runs a robot that already exists.

Open Browse AI

Which shape to give the agent

Use the job, not a leaderboard.

ScraperAPI and Scrapingdog overlap on the "one URL" job. Pick ScraperAPI when you are writing the HTTP client yourself and you may later use their structured endpoints. Pick Scrapingdog when the consumer is an MCP client and you want the key to stay in the server environment. Running both against the same URL is a reasonable test while you integrate. It is a waste as a permanent architecture.

Browse AI does not replace either of them for arbitrary URLs. A robot is a saved procedure. If the user can paste any link, a URL API is the one that can accept it. If the user always wants the same table from the same site, a robot will be more stable than asking a model to re-derive the selectors every morning.

None of these calls should receive a secret inside the prompt. The pattern for all three is identical: the key is in the environment or in a credential store, the tool or the HTTP client attaches it, and the model sees the page or the extracted fields.

A thin agent wrapper

This function hides the choice behind one interface so the rest of the agent does not care which vendor fetched the page. It does not try to be a framework. It shows the branch you actually have to write.

export async function fetchPage({ vendor, url }) {
  if (vendor === "scraperapi") {
    const endpoint = new URL("https://api.scraperapi.com");
    endpoint.searchParams.set("api_key", process.env.SCRAPERAPI_KEY);
    endpoint.searchParams.set("url", url);
    const response = await fetch(endpoint, { signal: AbortSignal.timeout(70000) });
    if (!response.ok) throw new Error(await response.text());
    return { vendor, url, body: await response.text() };
  }
  if (vendor === "scrapingdog") {
    const endpoint = new URL("https://api.scrapingdog.com/scrape");
    endpoint.searchParams.set("api_key", process.env.SCRAPINGDOG_API_KEY);
    endpoint.searchParams.set("url", url);
    endpoint.searchParams.set("dynamic", "false");
    const response = await fetch(endpoint, { signal: AbortSignal.timeout(90000) });
    if (!response.ok) throw new Error(await response.text());
    return { vendor, url, body: await response.text() };
  }
  throw new Error("Browse AI needs a robot id, not a bare URL. Run a task instead.");
}

Browse AI stays outside that function on purpose. Forcing it into fetchPage(url) throws away the robot id, which is the part that makes it work. Give the agent a second function, runRobot({ robotId, originUrl }), that POSTs the task and polls. The model then has two tools with honest names, instead of one tool that fails half the time.

Log vendor, the target URL, the status code, and the latency. Do not log the key or the full HTML if that HTML might contain personal data. Keep the raw body only as long as the extraction step needs it.

Limits of this comparison

This page does not quote prices, credit allotments, or success rates. Those change, and a number copied into a tutorial becomes a false claim the week after. Read the current plan on each product before you depend on a volume. The same goes for which sites are covered by a structured endpoint. If the docs do not list it, the generic URL fetch is the honest tool.

Also skip the fantasy that an agent should pick a vendor by itself on every request. You pick the shape once, from the table above, and the agent operates inside it. A model that can silently switch from a static GET to a rendered browser to a recorded robot will be impossible to debug.

The three catalogue records, again, are ScraperAPI, Scrapingdog, and Browse AI.

Open ScraperAPI

Open Scrapingdog

Open Browse AI