How to Use Toolip with Scrapy

1. Install Python: Download and install the latest version from python.org.

2. Install Scrapy: Open your terminal and run:

pip install scrapy

Log in to your Toolip dashboard and retrieve your proxy details (Host, Port, Username, and Password).

If using geo-specific proxies, format your username as your-username-country-XX (e.g., your-username-country-US for a US-based proxy).

1. Open a terminal and create a new Scrapy project by running:

scrapy startproject myproject

2. Navigate into the project folder:

cd myproject

1. Run the following command to create a spider:

scrapy genspider ToolipExample http://httpbin.org/ip

2. This will generate a spider template inside spiders/ToolipExample.py.

Open spiders/ToolipExample.py and modify it as follows:

import scrapy

class ToolipExampleSpider(scrapy.Spider):
    name = "ToolipExample"
    start_urls = ['http://httpbin.org/ip']

    def start_requests(self):
        # Define Toolip proxy
        proxy = "http://[USERNAME]:[PASSWORD]@[HOST]:[PORT]"  # Replace with your credentials

        # Use the proxy for all requests
        for url in self.start_urls:
            yield scrapy.Request(url, meta={'proxy': proxy})

    def parse(self, response):
        yield {
            'proxy_ip': response.text
        }

Replace [USERNAME], [PASSWORD], [HOST], and [PORT] with your Toolip proxy credentials.

1. Navigate to your Scrapy project directory:

cd myproject

2. Start the spider:

scrapy crawl ToolipExample

3. To save results to a file, use:

scrapy crawl ToolipExample -o output.json

1. Check if the spider successfully scrapes an IP address from Toolip proxies.

2. Example output:

[
    {
        "proxy_ip": "{\n  \"origin\": \"123.45.67.89\"\n}"
    }
]

3. If you saved the output, open output.json to inspect the results.