​
What is Scrapy?

Scrapy is a powerful Python-based framework for web scraping and data extraction. Designed for speed and scalability, Scrapy helps developers crawl websites and collect structured data efficiently. By integrating Toolip proxies into Scrapy, you can enhance your scraping tasks with secure, anonymous, and geo-targeted connections.

​
How to Set Up and Start a Scrapy Project

Step 0. Prerequisites

Before you begin, ensure you have:

1. Python Installed:

  • Download and install the latest version from python.org.

2. Scrapy Installed: Run the following command in your terminal to install Scrapy:

pip install scrapy

3. Toolip Proxy Credentials:

  • Log in to your Toolip dashboard and retrieve your proxy details (Host, Port, Username, and Password).

  • For region-specific proxies, modify your username using the format your-username-country-XX (e.g., your-username-country-US for a US proxy).

Step 1. Create or Open Your Scrapy Project

1. If you don’t have a Scrapy project, create one by running:

   scrapy startproject myproject

Replace “myproject” with a name that reflects the purpose of your project, such as “toolip_test” or “web_scraper”.

2. Navigate to your project folder:

   cd myproject

Step 2. Generate a Spider

1. Use Scrapy’s command to create a spider:

 scrapy genspider <spider_name> <target_url>

For example, to scrape httpbin.org/ip, you can run:

scrapy genspider ToolipExample http://httpbin.org/ip

2. This generates a basic spider template located in the spiders/ directory of your project. It looks something like this:

import scrapy

class ToolipExampleSpider(scrapy.Spider):
  name = "ToolipExample"
  allowed_domains = ["httpbin.org/ip"]
  start_urls = ["http://httpbin.org/ip"]

  def parse(self, response):
      pass

Step 3. Configure Toolip Proxies

1. Open the generated spider file in a text editor (spiders/ToolipExample.py) and update it to include Toolip proxy settings. Here’s an example:

import scrapy

class ToolipExampleSpider(scrapy.Spider):
    name = "ToolipExample"
    start_urls = ['http://httpbin.org/ip']

    def start_requests(self):
        # Define the Toolip proxy
        proxy = "http://[USERNAME]:[PASSWORD]@[HOST]:[PORT]"  # Replace with your Toolip proxy details

        # Use the proxy for all requests
        for url in self.start_urls:
            yield scrapy.Request(url, meta={'proxy': proxy})

    def parse(self, response):
        # Parse and return the IP address
        yield {
            'proxy_ip': response.text
        }

2. Replace [USERNAME], [PASSWORD], [HOST], and [PORT] with your Toolip credentials. If you need a country-specific proxy, modify the username (e.g., your-username-country-US).

Step 4. Run Your Scrapy Spider

1. Navigate to the project directory in your terminal:

cd myproject

2. Run the spider:

scrapy crawl ToolipExample

3. To save the output to a file, use:

scrapy crawl ToolipExample -o output.json

Step 5. Verify the Output

1. If everything is configured correctly, the spider will display the IP address of the Toolip proxy it’s using. Example output:

[
    {
        "proxy_ip": "{\n  \"origin\": \"123.45.67.89\"\n}"
    }
]

2. Open the output.json file (if you used the -o flag) to review the scraped data.

With Toolip proxies integrated into Scrapy, your web scraping tasks become more secure, private, and efficient. Whether you’re collecting geo-specific data, managing high-volume scraping jobs, or avoiding detection, Toolip provides the stability and anonymity you need. Start scraping smarter with Toolip and Scrapy today!

ParseHubScreaming Frog