> ## Documentation Index
> Fetch the complete documentation index at: https://docs.toolip.io/llms.txt
> Use this file to discover all available pages before exploring further.

# How to Use Toolip with Scrapy

> Integrate Toolip with Scrapy to enhance your web scraping workflows. This guide provides a step-by-step configuration process to enable secure and anonymous connections for your Scrapy projects.

<img src="https://mintcdn.com/toolip-d3ca8ce9/jm4qNwxMk9v2AMyx/integration-guides/img/toolip_integrations/Scrapy.png?fit=max&auto=format&n=jm4qNwxMk9v2AMyx&q=85&s=db2d19001707cd66080574c50f18cbe3" alt="title" width="720" height="200" data-path="integration-guides/img/toolip_integrations/Scrapy.png" />

## What is Scrapy?

**Scrapy** is a powerful Python-based framework for web scraping and data extraction. Designed for speed and scalability, Scrapy helps developers crawl websites and collect structured data efficiently. By integrating **Toolip** proxies into Scrapy, you can enhance your scraping tasks with secure, anonymous, and geo-targeted connections.

Prefer an enterprise-grade guide? Learn more about Oculus Proxies <a href="https://docs.oculusproxies.com/integration-guides/scrapy" target="_blank" rel="noopener">Scrapy Integration Guide</a>.

<Tip>
  Targeting search engines like Google, Bing, or Yandex requires a specialized proxy to ensure stable access and avoid blocks. **Toolip’s Search Engine ISP Proxies** are designed specifically for this, providing reliable performance where standard proxies may fail. If your proxy test isn't working on search engines, switching to **Search Engine ISP Proxies** can resolve the issue.
</Tip>

## How to Set Up and Start a Scrapy Project

<Steps>
  <Step title="Install Prerequisites">
    1\. **Install Python**: Download and install the latest version from [python.org](https://www.python.org/).

    2\. **Install Scrapy**: Open your terminal and run:

    ```bash theme={null}
    pip install scrapy
    ```
  </Step>

  <Step title="Retrieve Toolip Proxy Credentials">
    Log in to your [Toolip dashboard](https://toolip.io/dashboard/page/plans) and retrieve your proxy details (Host, Port, Username, and Password).

    If using geo-specific proxies, format your username as `your-username-country-XX` (e.g., `your-username-country-US` for a US-based proxy).
  </Step>

  <Step title="Create a Scrapy Project">
    1\. Open a terminal and create a new Scrapy project by running:

    ```bash theme={null}
    scrapy startproject myproject
    ```

    2\. Navigate into the project folder:

    ```bash theme={null}
    cd myproject
    ```
  </Step>

  <Step title="Generate a Scrapy Spider">
    1\. Run the following command to create a spider:

    ```bash theme={null}
    scrapy genspider ToolipExample http://httpbin.org/ip
    ```

    2\. This will generate a spider template inside `spiders/ToolipExample.py`.
  </Step>

  <Step title="Configure Toolip Proxy">
    Open `spiders/ToolipExample.py` and modify it as follows:

    ```python theme={null}
    import scrapy

    class ToolipExampleSpider(scrapy.Spider):
        name = "ToolipExample"
        start_urls = ['http://httpbin.org/ip']

        def start_requests(self):
            # Define Toolip proxy
            proxy = "http://[USERNAME]:[PASSWORD]@[HOST]:[PORT]"  # Replace with your credentials

            # Use the proxy for all requests
            for url in self.start_urls:
                yield scrapy.Request(url, meta={'proxy': proxy})

        def parse(self, response):
            yield {
                'proxy_ip': response.text
            }
    ```

    Replace `[USERNAME]`, `[PASSWORD]`, `[HOST]`, and `[PORT]` with your Toolip proxy credentials.
  </Step>

  <Step title="Run the Scrapy Spider">
    1\. Navigate to your Scrapy project directory:

    ```bash theme={null}
    cd myproject
    ```

    2\. Start the spider:

    ```bash theme={null}
    scrapy crawl ToolipExample
    ```

    3\. To save results to a file, use:

    ```bash theme={null}
    scrapy crawl ToolipExample -o output.json
    ```
  </Step>

  <Step title="Verify Proxy Integration">
    1\. Check if the spider successfully scrapes an IP address from Toolip proxies.

    2\. Example output:

    ```json theme={null}
    [
        {
            "proxy_ip": "{\n  \"origin\": \"123.45.67.89\"\n}"
        }
    ]
    ```

    3\. If you saved the output, open `output.json` to inspect the results.
  </Step>
</Steps>

With **Toolip** proxies integrated into **Scrapy**, your web scraping tasks become more secure, private, and efficient. Whether you’re collecting geo-specific data, managing high-volume scraping jobs, or avoiding detection, Toolip provides the stability and anonymity you need. Start scraping smarter with **Toolip** and **Scrapy** today!