Changing IP address in Python

How to change your IP address with python requests

Most times our webscraper get banned from visiting a website, this might be as a result of the website detecting that our webscraper is a bot, or that our webscraper is making multiple calls to thier servers. So in this case the need to change our IP address becomes a necessity. In this tutorial we will learn how we can change our IP address in Python using proxies.

Getting started:

We will need the Python third party module requests, so you can install it by running the following command on your terminal.

pip3 install requests

Like I said earlier we will use proxies to browse the web,this will enable us pass as different people accessing the web. For this tutorial we will use a list of free proxies gotten from here.

Python’s requests module allows us to use proxies when need be; and this follow a special format for parsing it to requests functions.
This format is designed as follows.

{"protocol":"ip:port"}

For example:

http_proxy  = {"http":"10.10.1.10:3128"}
https_proxy = {"https":"10.10.1.11:1080"}
ftp_proxy   = {"ftp":"10.10.1.10:3128"}

How to send requests through a Proxy using Requests:

First things first in other to change our IP address with Python, we will write a script that will scrape this proxies from here. For each proxy we scrape we test it for availability then save it in a text file.

import requests
 import random
 from lxml.html import fromstring
 def get_proxies():
     url = 'https://free-proxy-list.net/'
     response = requests.get(url)
     parser = fromstring(response.text)
     proxies = set()
     for i in parser.xpath('//tbody/tr')[:10]:
         if i.xpath('.//td[7][contains(text(),"yes")]'):
             proxy = ":".join([i.xpath('.//td[1]/text()')[0], i.xpath('.//td[2]/text()')[0]])
             proxies.add(proxy)
     return(list(proxies))
 def test_proxy(proxies):
     url = 'https://httpbin.org/ip'
     for id,proxy in enumerate(proxies):
         print("Request #%d"%id)
         try:
             response = requests.get(url,proxies={"http": proxy, "https": proxy})
             print(response.json())
             with open("proxy.txt","a+") as px:
                 px.write(proxy)
                 px.write("\n")
         except requests.exceptions.ProxyError:
             print("Skipping: PROXY ERROR")
         except:
             print("Skipping: CONNECTION ERROR")
 proxies = get_proxies()
 print(proxies)
 print(test_proxy(proxies))

Here are my results as at when I ran the script above:

Getting web proxies

We are going start making web requests using our scraped proxies. We will parse this proxies to our requests Session object.

import os, random
import requests

path = os.path.split(file)[0]+"/"
proxies = [i.strip() for i in open(path+"proxy.txt")]

for i in range(len(proxies)):
     #Select the proxy to use
     bot = requests.Session()
     proxy = random.choice(proxies)
     bot.proxies = {"http": proxy, "https": proxy}
     print(f"Using {proxy} as proxy")
     try:
         response = bot.get(url)
         print("Coookies :",response.cookies)
         print("Request was sent from :",print(response.json()))
     except requests.exceptions.ProxyError:
         print("Proxy Error") print("\n"*5)

To test our code we will visit http://ip.nefsc.noaa.gov/, because the website tell’s us our IP address each time we visit.

Here are the results returned by our small code after each visit.

results for changing IP address in Python

Conclusion:

This is not the end there are a variety of things you can do with hiding or changing your IP address, you can scrape sites which are not available to you in your region and the best of it all the Anonymity it offers.

References:

free/public list proxies .

More on webscraping click here.

Python requests and proxies stackoverflow.

4 comments

  1. Throughout this awesome pattern of things you actually secure an A for hard work. Exactly where you misplaced us was first on all the specifics. As people say, details make or break the argument.. And it couldn’t be more correct right here. Having said that, allow me reveal to you exactly what did work. Your article (parts of it) is definitely pretty persuasive which is probably the reason why I am making the effort in order to comment. I do not make it a regular habit of doing that. Second, although I can certainly see a leaps in reasoning you make, I am not really sure of how you appear to connect the points which help to make your final result. For the moment I shall yield to your point but wish in the foreseeable future you link the dots better.

Leave a Reply

Your email address will not be published. Required fields are marked *