We often talk about how to make a GET request and fetch crucial data from any website but in this post, we will talk about how we can send a POST request with Python. Web Scraping is not all about a GET request, many times you have to submit some data to the host server in order to retrieve the information.
But again one thing will remain common in this tutorial too and that is Requests library. As we all know it is a widely used library due to its simplicity and public support on multiple technical fronts. Using this library you can easily interact with any web API, scrape web pages, or send data easily to the host server.
We will start with basic steps and cover advanced topics like session management & handling cookies.
How to send a basic POST request?
We can easily make a POST request using .post() method of the requests library. We will be using httpbin.org website for this tutorial.
# import requests package
import requests
# Define the URL
url = "http://httpbin.org/post"
# Send the POST request
response = requests.post(url)
This is the most basic way of sending a POST request.
How to send a POST request with JSON data?
In the last step, we saw how we can make a POST request. But with POST request we have to send some data too. Sending JSON data is common when interacting with APIs that expect data in JSON format. We’ll use the httpbin.org/post URL as our testing endpoint. So, let’s see how we can send data in JSON format.
import requests
import json
# Define the URL
url = "http://httpbin.org/post"
# Define the JSON data you want to send in the POST request
data = {
"name": "John Doe",
"email": "johndoe@example.com"
}
# Convert the data dictionary to a JSON string
json_data = json.dumps(data)
# Send the POST request with JSON data
response = requests.post(url, data=json_data)
print(response.text)
- We create a dictionary
datacontaining the key-value pairs that we want to send in JSON format. - We use
json.dumps(data)to convert thedatadictionary into a JSON string. This step is essential because therequests.post()method expects the data to be in the form of a JSON string.
Once you run it you will see this.
How to send headers with POST request?
The above example worked but in real-world scenarios when you will be dealing with commercial APIs this will fail due to the absence of a proper header. You have to tell the host server about the format of the data that you are going to send to their database. This will help them properly receive the data otherwise many servers will send you a 4xx error.
import requests
url = 'http://httpbin.org/post'
headers = {
'Content-Type': 'application/json'
}
data = {
'first_name': 'Joe',
'Last_name': 'Biden'
}
response = requests.post(url, headers=headers, json=data)
print(response.json())
Here we have used Content-type header whose value is application/json. I have made another change, I think you might have already noticed it. I have used json paramter to pass the JSON-encoded object directly without using the json.dumps() function. This simply the code a little.
Once you run this code you will get the same response as above. You can even read our complete guide on web scraping with Python to get a complete idea of how headers actually function.
How to send FORM data with POST request?
Let’s say you want to scrape something that is behind an auth wall. That auth wall consists of a simple form that expects the correct username and password from you. This can be easily done using POST request. You just need to use the correct content type.
Generally, the value of the content-type header is application/x-www-form-urlencoded while submitting a form.
import requests
# Define the URL of the login endpoint
url = 'https://example.com/login'
# Create a dictionary containing the form data
form_data = {
'username': 'your_username',
'password': 'your_password'
}
# Define headers including the "Content-Type" header
headers = {
'Content-Type': 'application/x-www-form-urlencoded'
}
# Send the POST request with the form data and headers
response = requests.post(url, data=form_data, headers=headers)
You can read this guide on how to scrape data behind authentication with Python for more details.
How to upload files with POST request?
In this case, you just have to use the file parameter of the requests. The best part of this you don’t typically set the Content-Type header manually for file uploads. Instead, requests will automatically determine the correct Content-Type based on the file being uploaded.
import requests
# Define the URL where you want to upload the file
url = 'https://example.com/upload'
# Create a dictionary containing the file to upload
files = {
'file_field_name': ('sample.png', open('path/to/your/file.png', 'rb'))
}
# Send the POST request with the file
response = requests.post(url, files=files)
Session & cookie management with POST request
We have to create a session object that will retain information like cookies and other session-related data. The session object will help with state management.
import requests
# Create a session object
session = requests.Session()
# Define the login URL
login_url = 'https://example.com/login'
# Define the data to be sent in the POST request for login
login_data = {
'username': 'your_username',
'password': 'your_password',
}
# Send a POST request to log in
login_response = session.post(login_url, data=login_data)
session.close()
You can even store the cookies in a variable and use it to visit other pages of the website. With that, you can make HTTP GET requests to specific pages to retrieve the data.
After completing the requests, it’s good practice to close the session using session.close() to release any resources associated with it.
Conclusion
By mastering these concepts, you’ve gained the knowledge and skills needed to interact with web services, web APIs, and web applications effectively. Sending POST requests is a fundamental aspect of web development and data retrieval, making this knowledge invaluable in your programming journey.
As you continue to explore Python and web development, you’ll find that sending POST requests opens the door to a wide range of possibilities, from interacting with third-party APIs to building your own web applications. Remember to refer back to this guide whenever you need a refresher or a quick reference on POST requests in Python.
I hope you like this tutorial and if you do then please do not forget to share it with your friends and on your social media.
