Crawl

The Crawl API allows you to crawl websites and get data from multiple pages in a single request. Starting from a URL, it can navigate through the site and extract content from linked pages.

For detailed usage, checkout the Crawl API Reference.

Hyperbrowser exposes endpoints for starting a crawl request and for getting its status and results. By default, crawling is handled in an asynchronous manner of first starting the job and then checking its status until it is completed. However, with our SDKs, we provide a simple function that handles the whole flow and returns the data once the job is completed.

Installation

npm install @hyperbrowser/sdk dotenv

yarn add @hyperbrowser/sdk dotenv

pip install hyperbrowser python-dotenv

uv add hyperbrowser python-dotenv

Usage

import { Hyperbrowser } from "@hyperbrowser/sdk";
import { config } from "dotenv";

config();

const client = new Hyperbrowser({
  apiKey: process.env.HYPERBROWSER_API_KEY,
});

const main = async () => {
  // Handles both starting and waiting for crawl job response
  const crawlResult = await client.crawl.startAndWait({
    url: "https://example.com",
    maxPages: 10,
    followLinks: true,
  });
  console.log("Crawl result:", crawlResult);
};

main();

import os
from dotenv import load_dotenv
from hyperbrowser import Hyperbrowser

# Load environment variables from .env file
load_dotenv()

# Initialize Hyperbrowser client
client = Hyperbrowser(api_key=os.getenv("HYPERBROWSER_API_KEY"))


def main():
    # Start crawling and wait for completion
    crawl_result = client.crawl.start_and_wait(
        {
            "url": "https://example.com",
            "max_pages": 10,
            "follow_links": True,
        }
    )
    print("Crawl result:\n", crawl_result.model_dump_json(indent=2))


main()

import os
from dotenv import load_dotenv
from hyperbrowser import Hyperbrowser
from hyperbrowser.models import StartCrawlJobParams

# Load environment variables from .env file
load_dotenv()

# Initialize Hyperbrowser client
client = Hyperbrowser(api_key=os.getenv("HYPERBROWSER_API_KEY"))


def main():
    # Start crawling and wait for completion
    crawl_result = client.crawl.start_and_wait(
        StartCrawlJobParams(
            url="https://example.com",
            max_pages=10,
            follow_links=True,
        )
    )
    print("Crawl result:\n", crawl_result.model_dump_json(indent=2))


main()

# Start Crawl Job
curl -X POST https://api.hyperbrowser.ai/api/crawl \
    -H 'Content-Type: application/json' \
    -H 'x-api-key: <YOUR_API_KEY>' \
    -d '{
        "url": "https://example.com",
        "maxPages": 10,
        "followLinks": true
    }'

# Get Crawl Job Status
curl https://api.hyperbrowser.ai/api/crawl/{jobId}/status \
    -H 'x-api-key: <YOUR_API_KEY>'

# Get Crawl Job Status and Data
curl https://api.hyperbrowser.ai/api/crawl/{jobId} \
    -H 'x-api-key: <YOUR_API_KEY>'

Response

The Start Crawl Job POST /crawl endpoint will return a jobId in the response which can be used to get information about the job in subsequent requests.

{
  "jobId": "962372c4-a140-400b-8c26-4ffe21d9fb9c"
}

The Get Crawl Job Status GET /crawl/{jobId}/status will return the following data:

{
  "status": "completed"
}

The Get Crawl Job GET /crawl/{jobId} will return the following data:

{
  "jobId": "962372c4-a140-400b-8c26-4ffe21d9fb9c",
  "status": "completed",
  "totalCrawledPages": 10,
  "data": [
    {
      "metadata": {
        "title": "Example Page",
        "description": "A sample webpage",
        "url": "https://example.com"
      },
      "markdown": "# Example Page\nThis is content..."
    }
  ]
}

The status of a crawl job can be one of pending, running, completed, failed. The results will be an array of scraped pages in the data field.

Each crawled page has it’s own status of completed or failed and can have it’s own error field, so be cautious of that.

To see the full schema, checkout the API Reference.

Crawl Options

You can configure various options for the crawl job:

maxPages: Maximum number of pages to crawl (default: 10, max: 100)
followLinks: Whether to follow links on the crawled pages (default: true)
ignoreSitemap: Whether to ignore the sitemap (default: false)

import { Hyperbrowser } from "@hyperbrowser/sdk";
import { config } from "dotenv";

config();

const client = new Hyperbrowser({
  apiKey: process.env.HYPERBROWSER_API_KEY,
});

const main = async () => {
  const crawlResult = await client.crawl.startAndWait({
    url: "https://example.com",
    maxPages: 50,
    followLinks: true,
    ignoreSitemap: false,
  });
  console.log("Crawl result:", crawlResult);
};

main();

import os
from dotenv import load_dotenv
from hyperbrowser import Hyperbrowser

# Load environment variables from .env file
load_dotenv()

# Initialize Hyperbrowser client
client = Hyperbrowser(api_key=os.getenv("HYPERBROWSER_API_KEY"))


# Start crawling and wait for completion
crawl_result = client.crawl.start_and_wait(
    {
        "url": "https://example.com",
        "max_pages": 50,
        "follow_links": True,
        "ignore_sitemap": False,
    }
)
print("Crawl result:\n", crawl_result.model_dump_json(indent=2))

import os
from dotenv import load_dotenv
from hyperbrowser import Hyperbrowser
from hyperbrowser.models import StartCrawlJobParams

# Load environment variables from .env file
load_dotenv()

# Initialize Hyperbrowser client
client = Hyperbrowser(api_key=os.getenv("HYPERBROWSER_API_KEY"))


# Start crawling and wait for completion
crawl_result = client.crawl.start_and_wait(
    StartCrawlJobParams(
        url="https://example.com", max_pages=50, follow_links=True, ignore_sitemap=False
    )
)
print("Crawl result:\n", crawl_result.model_dump_json(indent=2))

Session Configurations

You can also provide configurations for the session that will be used to execute the crawl job, such as using a proxy or solving CAPTCHAs. To see all the different available session parameters, checkout the API Reference or Session Parameters.

import { Hyperbrowser } from "@hyperbrowser/sdk";
import { config } from "dotenv";

config();

const client = new Hyperbrowser({
  apiKey: process.env.HYPERBROWSER_API_KEY,
});

const main = async () => {
  const crawlResult = await client.crawl.startAndWait({
    url: "https://example.com",
    maxPages: 10,
    followLinks: true,
    sessionOptions: {
      useProxy: true,
      solveCaptchas: true,
      proxyCountry: "US",
    },
  });
  console.log("Crawl result:", crawlResult);
};

main();

import os
from dotenv import load_dotenv
from hyperbrowser import Hyperbrowser

load_dotenv()

client = Hyperbrowser(api_key=os.getenv("HYPERBROWSER_API_KEY"))


crawl_result = client.crawl.start_and_wait(
    {
        "url": "https://example.com",
        "max_pages": 10,
        "follow_links": True,
        "session_options": {"use_proxy": True, "solve_captchas": True},
    }
)
print("Crawl result:", crawl_result)

import os
from dotenv import load_dotenv
from hyperbrowser import Hyperbrowser
from hyperbrowser.models import StartCrawlJobParams, CreateSessionParams

load_dotenv()

client = Hyperbrowser(api_key=os.getenv("HYPERBROWSER_API_KEY"))


crawl_result = client.crawl.start_and_wait(
    StartCrawlJobParams(
        url="https://example.com",
        max_pages=10,
        follow_links=True,
        session_options=CreateSessionParams(use_proxy=True, solve_captchas=True),
    )
)
print("Crawl result:", crawl_result)

Using proxy and solving CAPTCHAs will slow down the crawl so use it only if necessary.

Scrape Configurations

You can also provide optional scrape options for the crawl job such as the formats to return, only returning the main content of the page, setting the maximum timeout for navigating to a page, etc.

import { Hyperbrowser } from "@hyperbrowser/sdk";
import { config } from "dotenv";

config();

const client = new Hyperbrowser({
  apiKey: process.env.HYPERBROWSER_API_KEY,
});

const main = async () => {
  const crawlResult = await client.crawl.startAndWait({
    url: "https://example.com",
    scrapeOptions: {
      formats: ["markdown", "html", "links"],
      onlyMainContent: false,
      timeout: 10000,
    },
  });
  console.log("Crawl result:", crawlResult);
};

main();

import os
from dotenv import load_dotenv
from hyperbrowser import Hyperbrowser

# Load environment variables from .env file
load_dotenv()

# Initialize Hyperbrowser client
client = Hyperbrowser(api_key=os.getenv("HYPERBROWSER_API_KEY"))


# Start crawling and wait for completion
crawl_result = client.crawl.start_and_wait(
    {
        "url": "https://example.com",
        "scrape_options": {
            "formats": ["html", "links", "markdown"],
            "only_main_content": False,
            "timeout": 10000,
        },
    }
)
print("Crawl result:", crawl_result)

import os
from dotenv import load_dotenv
from hyperbrowser import Hyperbrowser
from hyperbrowser.models import ScrapeOptions, StartCrawlJobParams

# Load environment variables from .env file
load_dotenv()

# Initialize Hyperbrowser client
client = Hyperbrowser(api_key=os.getenv("HYPERBROWSER_API_KEY"))


# Start crawling and wait for completion
crawl_result = client.crawl.start_and_wait(
    StartCrawlJobParams(
        url="https://example.com",
        scrape_options=ScrapeOptions(
            formats=["html", "links", "markdown"],
            only_main_content=False,
            timeout=10000,
        ),
    )
)
print("Crawl result:", crawl_result)

Hyperbrowser’s CAPTCHA solving and proxy usage features require being on a PAID plan.

For a full reference on the crawl endpoint, checkout the API Reference.

Get Started

Browser Sessions

Session Configuration

Web API

Agents

HyperAgent Usage

Scraping

Integrations

Installation

Usage

Response

Crawl Options

Session Configurations

Scrape Configurations

​Installation

​Usage

​Response

​Crawl Options

​Session Configurations

​Scrape Configurations

Installation

Usage

Response

Crawl Options

Session Configurations

Scrape Configurations