Spider

SpiderTools is an open source web Scraper & Crawler that returns LLM-ready data. To start using Spider, you need an API key from the Spider dashboard.

​Prerequisites

The following example requires the spider-client library.

pip install -U spider-client

​Example

The following agent will run a search query to get the latest news in USA and scrape the first search result. The agent will return the scraped data in markdown format.

cookbook/tools/spider_tools.py

from bitca.agent import Agent
from bitca.tools.spider import SpiderTools

agent = Agent(tools=[SpiderTools()])
agent.print_response('Can you scrape the first search result from a search on "news in USA"?', markdown=True)

​Toolkit Params

Parameter	Type	Default	Description
max_results	int	-	The maximum number of search results to return
url	str	-	The url to be scraped or crawled

​Toolkit Functions

Function	Description
search	Searches the web for the given query.
scrape	Scrapes the given url.
crawl	Crawls the given url.