Crawl4AI
Crawl4aiTools enable an Agent to perform web crawling and scraping tasks using the Crawl4ai library.
Prerequisites
The following example requires the crawl4ai library.
pip install -U crawl4aiExample
The following agent will scrape the content from the https://docs.projectbit.ca/ webpage:
cookbook/tools/crawl4ai_tools.py
from bitca.agent import Agent
from bitca.tools.crawl4ai_tools import Crawl4aiTools
agent = Agent(tools=[Crawl4aiTools(max_length=None)], show_tool_calls=True)
agent.print_response("Tell me about https://github.com/bitca/bitca.")Toolkit Params
Parameter
Type
Default
Description
max_length
int
1000
Specifies the maximum length of the text from the webpage to be returned.
Toolkit Functions
Function
Description
web_crawler
Crawls a website using crawl4ai’s WebCrawler. Parameters include ‘url’ for the URL to crawl and an optional ‘max_length’ to limit the length of extracted content. The default value for ‘max_length’ is 1000.
Last updated