> ## Documentation Index
> Fetch the complete documentation index at: https://phidatainc-redirect-agent-platform-overview.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Spider

> Use Spider tools with Agno agents.

Enable Agno agents to go beyond page-scraping by crawling entire websites and returning structured, "LLM-ready" Markdown data with Spider tools.

## Prerequisites

To use Spider, you need an API key from the Spider dashboard.

```python theme={null}

from agno.agent import Agent
from agno.tools.spider import SpiderTools

# ---------------------------------------------------------------------------
# Create Agent
# ---------------------------------------------------------------------------


# Example 1: All functions available (default behavior)
agent_all = Agent(
    name="Spider Agent - All Functions",
    tools=[SpiderTools(optional_params={"proxy_enabled": True})],
    instructions=["You have access to all Spider web scraping capabilities."],
    markdown=True,
)

# Example 2: Include specific functions only
agent_specific = Agent(
    name="Spider Agent - Search Only",
    tools=[SpiderTools(enable_crawl=False, optional_params={"proxy_enabled": True})],
    instructions=["You can only search the web, no scraping or crawling."],
    markdown=True,
)


# Use the default agent for examples
agent = agent_all

# ---------------------------------------------------------------------------
# Run Agent
# ---------------------------------------------------------------------------
if __name__ == "__main__":
    agent.print_response(
        'Can you scrape the first search result from a search on "news in USA"?'
    )
```

## Run the Example

```bash theme={null}
# Clone and setup repo
git clone https://github.com/agno-agi/agno.git
cd agno/cookbook/91_tools

# Create and activate virtual environment
./scripts/demo_setup.sh
source .venvs/demo/bin/activate

python spider_tools.py
```

For details, see [Spider cookbook](https://github.com/agno-agi/agno/blob/main/cookbook/91_tools/spider_tools.py).