Requests in Python – Request Web Pages Using Python

By /
Requests Method In Python

Requests in Python is an elegant library that lets you send HTTP/1.1 requests to web pages via Python.

It is officially supported by both Python 2.7 and 3.5+. Advance features like Keep – Alive, Connection Pooling, Sessions with persistent cookies, Browser Style SSL verification make it the go-to choice for developers.

In this article, we shall learn more about some of these features and how you can get started with using the Python Requests module to create web requests.

How to Install Requests in Python?

Installing requests in Python is easy and straight forward to follow. You can have several approaches to install a module in Python. But for this article, we’ll show you how to use it with pip module.

Open your terminal or command prompt (if you are a windows user) and type in the following command.

pip install requests 
#Or (if the first command doesn't work) use:
pip3 install requests

It should successfully install requests module in your device.

Using Requests in Python

To understand how requests module, works we need to know the fundamentals of what happens when we browse the web and how it instantly shows you the content you were hoping to see.

Every time, you click on a link, we send an HTTP (Hypertext Transfer Protocol), request to the server of the requests page.

Upon getting the request, the server sends us back the right content we requested from it. The two most useful HTTP requests we’re gonna learn is GET and POST requests.

In the following section, we will learn how to use these methods with the requests library. But first, we need to import it on your script or interpreter.

import requests

1. GET Request

This method is used to indicate we are requesting the contents of our selected URL from the server. So, let’s say we want to get the homepage of google using our HTTP requests.

Type the following line.

r = requests.get("http://google.com")

Here’s what the single line of code does: It sends an HTTP GET request to Google’s homepage via the get() method where the URL is provided as the argument. And the response object is stored in our ‘r’ variable.

The instance of our Response object further classifies the retained data and stores them within proper attributes. Here’s an example below

print(r.status_code) 
#The output provides the status code for the url. For a success full attempt, the result is 200

print(r.headers)
#The following attribute returns a python dictionary containing key-value pair of the headers

print(r.text)
#This command prints out the response content from the server or Static Source Code. 

print(r.encoding)
r.encoding = 'utf-8' 
#Requests library also allows you to see or change the encoding of the response content.

2. Passing Parameters using the GET Method

Often a single GET method doesn’t let us find all the information we need access to, so we need to pass additional parameters with our original get request.

Parameters are mainly key-value pairs of data enclosed in a tuple or list. We can send it using the params parameter of our get() method.

See the syntax to follow along. 

import requests 
payload = {'key1': 'value1', 'key2': 'value2'}
r = requests.get('http://httpbin.org/get', params=payload)
print(r.text)

Output:

{
  "args": {
    "key1": "value1",
    "key2": "value2"
  },
  "headers": {
    "Accept": "*/*",
    "Accept-Encoding": "gzip, deflate",
    "Host": "httpbin.org",
    "User-Agent": "python-requests/2.22.0",
    "X-Amzn-Trace-Id": "Root=1-5f9a64d1-2abfc74b2725386140a897e3"
  },
  "origin": 0.0.0.0, 
  "url": "http://httpbin.org/get?key1=value1&key2=value2"
}

3. POST Request

Unlike GET Requests in Python, the POST Method in HTTP requires a payload to be posted with it. This method is used to send data to a server instead of directly retreving it. Within our requests library, we can access POST using post() method.

Have a quick look at the syntax:

import requests 
payload = {'key1': 'value1', 'key2': 'value2'}
r = requests.post("https://httpbin.org/post", data=payload)
print(r.text)

Output:

{
  "args": {},
  "data": "",
  "files": {},
  "form": {
    "key1": "value1",
    "key2": "value2"
  },
  "headers": {
    "Accept": "*/*",
    "Accept-Encoding": "gzip, deflate",
    "Content-Length": "23",
    "Content-Type": "application/x-www-form-urlencoded",
    "Host": "httpbin.org",
    "User-Agent": "python-requests/2.22.0",
    "X-Amzn-Trace-Id": "Root=1-5f9a6726-276da087230912e01dd5dcd7"
  },
  "json": null,
  "origin": [REDACTED],
  "url": "https://httpbin.org/post"
}

Some Advanced Features of Requests in Python

Our article mainly focuses on the two most basic yet highly important HTTP methods. But the requests module supports a multitude of such methods such as PUT, PATCH, DELETE etc.

One of the key reasons, the ‘requests’ module is so famous among developers is advance features like:

  1. The Sessions Object: It is mainly used to store the same cookies among different requests overall providing a faster response.
  2. Support for SOCKS Proxies: Although you need to install a separate dependency (called ‘requests[socks]’), it can greatly help your performance for multiple requests especially if the server rate limits your IP.
  3. SSL Verification: You can force check if a website properly supports SSL using requests by providing an extra argument “verify=True” within the get() method. If the website fails to show proper support for SSL, the script will throw an error.

Conclusion

Whether be it web-scraping or other HTTP related work, the requests module is the most popular option out there.

The only thing where the requests module falls short is handling dynamic changes in the source code of a page as the module is not designed to execute javascript commands.

Hope this article served you a basic idea about what the module does.

Resources

You can read more about the module on their official documentation site: https://requests.readthedocs.io/en/latest/

Their Official Github Repo: https://github.com/psf/requests