Crawl4AI

Crawl4aiTools enable an Agent to perform web crawling and scraping tasks using the Crawl4ai library.

​Prerequisites

The following example requires the crawl4ai library.

pip install -U crawl4ai

​Example

The following agent will scrape the content from the https://docs.projectbit.ca/ webpage:

cookbook/tools/crawl4ai_tools.py

from bitca.agent import Agent
from bitca.tools.crawl4ai_tools import Crawl4aiTools

agent = Agent(tools=[Crawl4aiTools(max_length=None)], show_tool_calls=True)
agent.print_response("Tell me about https://github.com/bitca/bitca.")

​Toolkit Params

Parameter	Type	Default	Description
max_length	int	1000	Specifies the maximum length of the text from the webpage to be returned.

​Toolkit Functions

Function	Description
web_crawler	Crawls a website using crawl4ai’s WebCrawler. Parameters include ‘url’ for the URL to crawl and an optional ‘max_length’ to limit the length of extracted content. The default value for ‘max_length’ is 1000.