Skip to main content

Documentation Index

Fetch the complete documentation index at: https://hyperbrowser.ai/docs/llms.txt

Use this file to discover all available pages before exploring further.

Hyperbrowser integrates seamlessly with OpenAI and Anthropic’s function calling APIs, enabling you to enhance your AI applications with web scraping and crawling capabilities. This guide shows how to set up and use Hyperbrowser’s scrape, crawl, and extract tools with OpenAI (and Anthropic-compatible definitions).

​
Setup

​
Installation

npm install @hyperbrowser/sdk dotenv openai

​
Setup your Environment

To use Hyperbrowser with your code, you need an API Key from the dashboard. Add it to your .env as HYPERBROWSER_API_KEY. You will also need an OPENAI_API_KEY.

​
Code

import OpenAI from "openai";
import { Hyperbrowser } from "@hyperbrowser/sdk";
import { WebsiteCrawlTool, WebsiteScrapeTool,WebsiteExtractTool } from "@hyperbrowser/sdk/tools";
import { config } from "dotenv";

config();

// Initialize clients
const hb = new Hyperbrowser({ apiKey: process.env.HYPERBROWSER_API_KEY });
const oai = new OpenAI({
  apiKey: process.env.OPENAI_API_KEY,
});

async function handleToolCall(tc) {
  console.log("Handling tool call");

  try {
    const args = JSON.parse(tc.function.arguments);
    console.log(`Tool call ID: ${tc.id}`);
    console.log(`Function name: ${tc.function.name}`);
    console.log(`Function args: ${JSON.stringify(args, null, 2)}`);
    console.log("-".repeat(50));

    if (
      tc.function.name === WebsiteCrawlTool.openaiToolDefinition.function.name
    ) {
      const response = await WebsiteCrawlTool.runnable(hb, args);
      return {
        tool_call_id: tc.id,
        content: response,
        role: "tool",
      };
    } else if (
      tc.function.name === WebsiteScrapeTool.openaiToolDefinition.function.name
    ) {
      const response = await WebsiteScrapeTool.runnable(hb, args);
      return {
        tool_call_id: tc.id,
        content: response,
        role: "tool",
      };
    } else if (
      tc.function.name === WebsiteExtractTool.openaiToolDefinition.function.name
    ) {
      const response = await WebsiteExtractTool.runnable(hb, args);
      return {
        tool_call_id: tc.id,
        content: response,
        role: "tool",
      };
    } else {
      return {
        tool_call_id: tc.id,
        content: "Unknown tool call",
        role: "tool",
      };
    }
  } catch (e) {
    console.error(e);
    return {
      role: "tool",
      tool_call_id: tc.id,
      content: `Error occurred: ${e}`,
    };
  }
}

const messages = [
  {
    role: "user",
    content: "What does Hyperbrowser.ai do? Provide citations.",
  },
];

async function openaiChat() {
  while (true) {
    const resp = await oai.beta.chat.completions.parse({
      model: "gpt-4o-mini",
      messages: messages,
      tools: [
        WebsiteCrawlTool.openaiToolDefinition,
        WebsiteScrapeTool.openaiToolDefinition,
        WebsiteExtractTool.openaiToolDefinition,
      ],
    });

    const choice = resp.choices[0];
    messages.push(choice.message);

    if (choice.finish_reason === "tool_calls") {
      for (const tc of choice.message.tool_calls) {
        const result = await handleToolCall(tc);
        messages.push(result);
      }
    } else if (choice.finish_reason === "stop") {
      console.log(choice.message.content);
      break;
    } else {
      throw new Error("Unknown Error Occurred");
    }
  }
}

openaiChat();
Hyperbrowser exposes a WebsiteCrawlTool, a WebsiteScrapeTool, and a WebsiteExtractTool to be used for OpenAI and Anthropic function calling. Each class provides a tool definition for both OpenAI and Anthropic that defines the schema which can be passed into the tools argument. Then, in the handleToolCall/handle_tool_call function, we parse the tool call arguments and dispatch the appropriate tool class runnable function based on the function name which returns the result of the scrape or crawl in formatted markdown.