Advanced Guide

This guide shows how to use the Web APIs to fetch pages, search the web, and extract structured data. It also documents the most important parameters for advanced use cases.

You can also see dedicated pages for Fetch and Search. For full schemas, see the API Reference.

Fetching a web page

With just a URL, you can extract page contents in your chosen formats using the /api/web/fetch endpoint.

import { Hyperbrowser } from "@hyperbrowser/sdk";
import { config } from "dotenv";

config();

const client = new Hyperbrowser({
  apiKey: process.env.HYPERBROWSER_API_KEY,
});

const main = async () => {
  const result = await client.web.fetch({
    url: "https://example.com",
    outputs: {
      formats: ["markdown", "links"],
    },
  });
  console.log("Fetch result:", result);
};

main();

import os
from dotenv import load_dotenv
from hyperbrowser import Hyperbrowser

load_dotenv()

client = Hyperbrowser(api_key=os.getenv("HYPERBROWSER_API_KEY"))

# Fetch a page
result = client.web.fetch(
    {
        "url": "https://example.com",
        "outputs": {"formats": ["markdown", "links"]},
    }
)
print("Fetch result:\n", result.model_dump_json(indent=2))

import os
from dotenv import load_dotenv
from hyperbrowser import Hyperbrowser
from hyperbrowser.models import FetchParams, FetchOutputOptions

load_dotenv()

client = Hyperbrowser(api_key=os.getenv("HYPERBROWSER_API_KEY"))

# Fetch a page
result = client.web.fetch(
    FetchParams(
        url="https://example.com",
        outputs=FetchOutputOptions(formats=["markdown", "links"]),
    )
)
print("Fetch result:\n", result.model_dump_json(indent=2))

curl -X POST https://api.hyperbrowser.ai/api/web/fetch \
    -H 'Content-Type: application/json' \
    -H 'x-api-key: <YOUR_API_KEY>' \
    -d '{
        "url": "https://example.com",
        "outputs": {
            "formats": ["markdown", "links"]
        }
    }'

Fetch Response

The fetch response includes a jobId, overall status, and the outputs under data:

{
  "jobId": "962372c4-a140-400b-8c26-4ffe21d9fb9c",
  "status": "completed",
  "data": {
    "metadata": {
      "title": "Example Domain",
      "sourceURL": "https://example.com"
    },
    "markdown": "# Example Domain\n\nThis domain is for use in illustrative examples...",
    "links": [
      "https://www.iana.org/domains/example"
    ]
  }
}

Output Formats

The outputs.formats array specifies what data to return. You can mix simple strings with output objects:

Output	Description
`markdown`	Page content as Markdown
`html`	Raw HTML
`links`	All links on the page
`screenshot`	Screenshot (configurable)
`json`	AI-extracted structured data
`branding`	Visual brand profile—colors, fonts, logo, components, personality

Screenshot with Options

{
  "url": "https://example.com",
  "outputs": {
    "formats": [
      "markdown",
      {
        "type": "screenshot",
        "fullPage": true,
        "format": "webp",
        "cropToContent": false
      }
    ]
  }
}

Screenshot cropping rules:

fullPage and cropToContent are mutually exclusive.
If both cropToContentMaxHeight and cropToContentMinHeight are set, max must be ≥ min.
Crop dimensions must be between 100 and 8000 pixels.

JSON Extraction with Schema

{
  "url": "https://example.com",
  "outputs": {
    "formats": [
      {
        "type": "json",
        "schema": {
          "type": "object",
          "properties": {
            "heading": { "type": "string" },
            "description": { "type": "string" }
          },
          "required": ["heading", "description"],
          "additionalProperties": false
        }
      }
    ]
  }
}

Fetch Options

The Fetch API supports comprehensive configuration for controlling browser behavior, navigation, and outputs.

string

required

The URL of the page to fetch.

string

"none" | "auto" | "ultra" — controls the default proxy/stealth posture.

Array<string | object>

Output formats to return (e.g. ["markdown", "links"]), plus structured outputs like {"type":"json","schema":...}.

string

"none" | "basic" | "advanced" — Sanitizes output for easier downstream parsing.

string[]

default:"[]"

CSS selectors to include. Only matching elements are returned.

string[]

default:"[]"

CSS selectors to exclude from output.

object

Pre-seed localStorage / sessionStorage as key/value pairs.

Load condition: "load", "domcontentloaded", or "networkidle".

Milliseconds to wait after navigation completes before collecting outputs (0–30000).

Max time (ms) to wait for navigation (1–60000).

number

Cache control for fetch results. Set 0 to bypass cache reads.

object

Localize via proxy location (country, state, city). If set, proxy is enabled automatically.

string

Reuse an existing browser profile.

string

Enable CAPTCHA solving.

object

Set viewport dimensions (width, height).

Example: Fetch with Advanced Options

const result = await client.web.fetch({
  url: "https://example.com",
  stealth: "ultra",
  outputs: {
    formats: ["html"],
    sanitize: "basic",
    excludeSelectors: ["nav", "footer", "aside"],
    storageState: {
      localStorage: {
        "example:key": "example:value",
      },
    },
  },
  navigation: {
    waitUntil: "networkidle",
    waitFor: 2000,
    timeoutMs: 60000,
  },
  cache: {
    maxAgeSeconds: 3600,
  },
  browser: {
    location: {
      country: "US",
      state: "CA",
    },
  },
});

import os
from dotenv import load_dotenv
from hyperbrowser import Hyperbrowser

load_dotenv()

client = Hyperbrowser(api_key=os.getenv("HYPERBROWSER_API_KEY"))

result = client.web.fetch(
    {
        "url": "https://example.com",
        "stealth": "ultra",
        "outputs": {
            "formats": ["html"],
            "sanitize": "basic",
            "exclude_selectors": ["nav", "footer", "aside"],
            "storage_state": {"local_storage": {"example:key": "example:value"}},
        },
        "navigation": {
            "wait_until": "networkidle",
            "wait_for": 2000,
            "timeout_ms": 60000,
        },
        "cache": {"max_age_seconds": 3600},
        "browser": {"location": {"country": "US", "state": "CA"}},
    }
)

print("Fetch result:\n", result.model_dump_json(indent=2))

import os
from dotenv import load_dotenv
from hyperbrowser import Hyperbrowser
from hyperbrowser.models import (
    FetchParams,
    FetchOutputOptions,
    FetchNavigationOptions,
    FetchCacheOptions,
    FetchBrowserOptions,
    FetchBrowserLocationOptions,
    FetchStorageStateOptions,
)

load_dotenv()

client = Hyperbrowser(api_key=os.getenv("HYPERBROWSER_API_KEY"))

result = client.web.fetch(
    FetchParams(
        url="https://example.com",
        stealth="ultra",
        outputs=FetchOutputOptions(
            formats=["html"],
            sanitize="basic",
            exclude_selectors=["nav", "footer", "aside"],
            storage_state=FetchStorageStateOptions(
                local_storage={"example:key": "example:value"}
            ),
        ),
        navigation=FetchNavigationOptions(
            wait_until="networkidle", wait_for=2000, timeout_ms=60000
        ),
        cache=FetchCacheOptions(max_age_seconds=3600),
        browser=FetchBrowserOptions(
            location=FetchBrowserLocationOptions(country="US", state="CA")
        ),
    )
)

print("Fetch result:\n", result.model_dump_json(indent=2))

curl -X POST https://api.hyperbrowser.ai/api/web/fetch \
  -H 'Content-Type: application/json' \
  -H 'x-api-key: <YOUR_API_KEY>' \
  -d '{
    "url": "https://example.com",
    "stealth": "ultra",
    "outputs": {
      "formats": ["html"],
      "sanitize": "basic",
      "excludeSelectors": ["nav", "footer", "aside"],
      "storageState": {
        "localStorage": {
          "example:key": "example:value"
        }
      }
    },
    "navigation": {
      "waitUntil": "networkidle",
      "waitFor": 2000,
      "timeoutMs": 60000
    },
    "cache": {
      "maxAgeSeconds": 3600
    },
    "browser": {
      "location": {
        "country": "US",
        "state": "CA"
      }
    }
  }'

Searching the web

Use the Search API to query the web and get structured results—titles, URLs, and descriptions—ready to use in your application.

import { Hyperbrowser } from "@hyperbrowser/sdk";
import { config } from "dotenv";

config();

const client = new Hyperbrowser({
  apiKey: process.env.HYPERBROWSER_API_KEY,
});

const main = async () => {
  const result = await client.web.search({
    query: "hyperbrowser browser automation",
  });
  console.log("Search result:", result);
};

main();

import os
from dotenv import load_dotenv
from hyperbrowser import Hyperbrowser

load_dotenv()

client = Hyperbrowser(api_key=os.getenv("HYPERBROWSER_API_KEY"))

# Search the web
result = client.web.search({"query": "hyperbrowser browser automation"})
print("Search result:\n", result.model_dump_json(indent=2))

import os
from dotenv import load_dotenv
from hyperbrowser import Hyperbrowser
from hyperbrowser.models import WebSearchParams

load_dotenv()

client = Hyperbrowser(api_key=os.getenv("HYPERBROWSER_API_KEY"))

# Search the web
result = client.web.search(WebSearchParams(query="hyperbrowser browser automation"))
print("Search result:\n", result.model_dump_json(indent=2))

curl -X POST https://api.hyperbrowser.ai/api/web/search \
    -H 'Content-Type: application/json' \
    -H 'x-api-key: <YOUR_API_KEY>' \
    -d '{
        "query": "hyperbrowser browser automation"
    }'

Search Response

The response includes search results with titles, URLs, and descriptions:

{
  "status": "completed",
  "data": {
    "query": "hyperbrowser browser automation",
    "results": [
      {
        "title": "Hyperbrowser - Browser Automation Platform",
        "url": "https://hyperbrowser.ai",
        "description": "Hyperbrowser provides cloud browsers for AI agents and web scraping..."
      },
      {
        "title": "Getting Started with Hyperbrowser",
        "url": "https://docs.hyperbrowser.ai/quickstart",
        "description": "Learn how to use Hyperbrowser for browser automation..."
      }
    ]
  }
}

Search Options

The Search API supports pagination, location-based results, and advanced filtering.

string

required

The search query (max 500 characters).

number

Page number for pagination (1-100, default: 1).

number

Cache control—cached results older than this are treated as stale. Set to 0 to bypass cache reads.

string

ISO-2 country code (e.g., "US", "GB", "DE").

string

State code (for supported countries).

string

City name (max 200 characters).

Advanced search filters allow you to refine your search results:

boolean

Wrap query in quotes for exact match.

boolean

Use semantic search.

string[]

Terms to exclude from results.

string[]

Terms to prioritize in results.

string

Filter by file type: pdf, doc, docx, xls, xlsx, ppt, pptx, html.

string

Limit results to a specific site.

string

Exclude results from a specific site.

string

Search term must appear in page title.

string

Search term must appear in URL.

exactPhrase and semanticPhrase cannot both be true.

Example: Search with Filters

const result = await client.web.search({
  query: "machine learning tutorials",
  page: 1,
  location: {
    country: "US",
    state: "CA",
  },
  filters: {
    excludeTerms: ["beginner"],
    filetype: "pdf",
    site: "arxiv.org",
  },
});

import os
from dotenv import load_dotenv
from hyperbrowser import Hyperbrowser

load_dotenv()

client = Hyperbrowser(api_key=os.getenv("HYPERBROWSER_API_KEY"))

result = client.web.search(
    {
        "query": "machine learning tutorials",
        "page": 1,
        "location": {"country": "US", "state": "CA"},
        "filters": {
            "exclude_terms": ["beginner"],
            "filetype": "pdf",
            "site": "arxiv.org",
        },
    }
)

print("Search result:\n", result.model_dump_json(indent=2))

import os
from dotenv import load_dotenv
from hyperbrowser import Hyperbrowser
from hyperbrowser.models import (
    WebSearchParams,
    WebSearchLocation,
    WebSearchFilters,
)

load_dotenv()

client = Hyperbrowser(api_key=os.getenv("HYPERBROWSER_API_KEY"))

result = client.web.search(
    WebSearchParams(
        query="machine learning tutorials",
        page=1,
        location=WebSearchLocation(country="US", state="CA"),
        filters=WebSearchFilters(
            exclude_terms=["beginner"],
            filetype="pdf",
            site="arxiv.org",
        ),
    )
)

print("Search result:\n", result.model_dump_json(indent=2))

curl -X POST https://api.hyperbrowser.ai/api/web/search \
  -H 'Content-Type: application/json' \
  -H 'x-api-key: <YOUR_API_KEY>' \
  -d '{
    "query": "machine learning tutorials",
    "page": 1,
    "location": {
      "country": "US",
      "state": "CA"
    },
    "filters": {
      "excludeTerms": ["beginner"],
      "filetype": "pdf",
      "site": "arxiv.org"
    }
  }'

Combining Search and Fetch

A common pattern is to search for pages and then fetch their content. Here’s how to do it:

import { Hyperbrowser } from "@hyperbrowser/sdk";

const client = new Hyperbrowser({
  apiKey: process.env.HYPERBROWSER_API_KEY,
});

const main = async () => {
  // First, search for pages
  const searchResult = await client.web.search({
    query: "hyperbrowser documentation",
    filters: {
      site: "docs.hyperbrowser.ai",
    },
  });

  // Then fetch content from the top results
  const fetchPromises = searchResult.data.results.slice(0, 3).map((result) =>
    client.web.fetch({
      url: result.url,
      outputs: {
        formats: ["markdown"],
      },
    })
  );

  const fetchedPages = await Promise.all(fetchPromises);

  fetchedPages.forEach((page, index) => {
    console.log(`\n--- Page ${index + 1}: ${searchResult.data.results[index].title} ---`);
    console.log(page.data.markdown);
  });
};

main();

import os
from dotenv import load_dotenv
from hyperbrowser import Hyperbrowser

load_dotenv()

client = Hyperbrowser(api_key=os.getenv("HYPERBROWSER_API_KEY"))

# First, search for pages
search_result = client.web.search(
    {
        "query": "hyperbrowser documentation",
        "filters": {"site": "docs.hyperbrowser.ai"},
    }
)

# Then fetch content from the top results
for i, result in enumerate(search_result.data.results[:3]):
    print(f"\n--- Page {i + 1}: {result.title} ---")

    fetch_result = client.web.fetch(
        {
            "url": result.url,
            "outputs": {"formats": ["markdown"]},
        }
    )

    print(fetch_result.data.markdown)

import os
from dotenv import load_dotenv
from hyperbrowser import Hyperbrowser
from hyperbrowser.models import (
    WebSearchParams,
    WebSearchFilters,
    FetchParams,
    FetchOutputOptions,
)

load_dotenv()

client = Hyperbrowser(api_key=os.getenv("HYPERBROWSER_API_KEY"))

# First, search for pages
search_result = client.web.search(
    WebSearchParams(
        query="hyperbrowser documentation",
        filters=WebSearchFilters(site="docs.hyperbrowser.ai"),
    )
)

# Then fetch content from the top results
for i, result in enumerate(search_result.data.results[:3]):
    print(f"\n--- Page {i + 1}: {result.title} ---")

    fetch_result = client.web.fetch(
        FetchParams(
            url=result.url,
            outputs=FetchOutputOptions(formats=["markdown"]),
        )
    )

    print(fetch_result.data.markdown)

# First, search for pages
SEARCH_RESPONSE=$(curl -s -X POST https://api.hyperbrowser.ai/api/web/search \
  -H 'Content-Type: application/json' \
  -H 'x-api-key: <YOUR_API_KEY>' \
  -d '{
    "query": "hyperbrowser documentation",
    "filters": {
      "site": "docs.hyperbrowser.ai"
    }
  }')

# Extract first URL from results (requires jq)
URL=$(echo $SEARCH_RESPONSE | jq -r '.data.results[0].url')

# Fetch the page content
curl -X POST https://api.hyperbrowser.ai/api/web/fetch \
  -H 'Content-Type: application/json' \
  -H 'x-api-key: <YOUR_API_KEY>' \
  -d "{
    \"url\": \"$URL\",
    \"outputs\": {
      \"formats\": [\"markdown\"]
    }
  }"

Browser & Stealth

Both Fetch and Search APIs support stealth configuration for bypassing anti-bot measures:

Stealth (stealth): "none" | "auto" | "ultra"
Location (browser.location): { country, state, city } (enables proxy automatically)
CAPTCHA (browser.solveCaptchas): boolean
Profiles (browser.profileId): reuse an existing profile
Screen (browser.screen): { width, height }

Using proxy and CAPTCHA solving increases latency. Enable only when needed.

Using a custom location costs 10 credits per search. Without a location specified, searches default to US and cost 5 credits.

Best Practices

Caching

Use cache.maxAgeSeconds (Fetch) or maxAgeSeconds (Search) to control caching behavior:

Set to 3600 (1 hour) for frequently updated content
Set to 86400 (24 hours) for static content
Set to 0 to always fetch fresh data

Error Handling

Always check the status field in responses:

const result = await client.web.fetch({ url: "https://example.com" });

if (result.status === "failed") {
  console.error("Fetch failed:", result.error);
} else if (result.status === "completed") {
  console.log("Success:", result.data);
}

Rate Limiting

The Web APIs are rate-limited. Use the SDK’s built-in retry logic or implement your own exponential backoff strategy.

Output Sanitization

Use outputs.sanitize to clean up HTML/Markdown:

"none": No sanitization (fastest)
"basic": Remove scripts, styles, and common noise
"advanced": Aggressive cleaning for LLM processing

Selector Filtering

Use includeSelectors and excludeSelectors to focus on specific content:

{
  "url": "https://example.com/blog",
  "outputs": {
    "formats": ["markdown"],
    "includeSelectors": ["article", "main"],
    "excludeSelectors": ["nav", "footer", "aside", ".advertisement"]
  }
}

Get Started

Browser Sessions

Session Configuration

Web API

Agents

HyperAgent Usage

Scraping

Webhooks

Integrations

Fetching a web page

Fetch Response

Output Formats

Screenshot with Options

JSON Extraction with Schema

Fetch Options

Example: Fetch with Advanced Options

Searching the web

Search Response

Search Options

Search Filters

Example: Search with Filters

Combining Search and Fetch

Browser & Stealth

Best Practices

Caching

Error Handling

Rate Limiting

Output Sanitization

Selector Filtering

​Fetching a web page

​Fetch Response

​Output Formats

​Screenshot with Options

​JSON Extraction with Schema

​Fetch Options

​Example: Fetch with Advanced Options

​Searching the web

​Search Response

​Search Options

​Search Filters

​Example: Search with Filters

​Combining Search and Fetch

​Browser & Stealth

​Best Practices

​Caching

​Error Handling

​Rate Limiting

​Output Sanitization

​Selector Filtering

Fetching a web page

Fetch Response

Output Formats

Screenshot with Options

JSON Extraction with Schema

Fetch Options

Example: Fetch with Advanced Options

Searching the web

Search Response

Search Options

Search Filters

Example: Search with Filters

Combining Search and Fetch

Browser & Stealth

Best Practices

Caching

Error Handling

Rate Limiting

Output Sanitization

Selector Filtering