Google Jobs is a job aggregator that collects data from multiple job listing platforms. If you’re building a job board, scraping Google Jobs should be your priority.
Some other use cases of scraping it could be market research, that can further help you track trends in hiring, monitor demand for specific skills, or analyze job availability across industries.
In this read, we’ll scrape Google Jobs using Node.js and see how to avoid getting blocked when the platform detects scraping.
Let’s get started!
Let's Scrape Google Jobs using Nodejs
In this section, we will prepare a basic script to scrape the first 10 Google Jobs listings, including their title, location, and other details.
For this, we have first to install some libraries so we can move forward:
- Unirest JS — For scraping raw HTML data.
- Cheerio JS — For parsing the raw HTML data.
We will also download a CSS selector agent to easily select the elements from the HTML so we don’t have to deal with the complex HTML structure.
Please open the link below in your browser so that we can select the HTML tags for the required elements.
https://www.google.com/search?q=web+developer+in+mumbai&ibp=htl;jobs&hl=en
We will be using Unirest JS to extract the raw HTML data and parse this data with the help of Cheerio JS.
Now, make a GET request using Unirest JS on the target URL.
const unirest = require("unirest");
const cheerio = require("cheerio");
const getJobsData = async () => {
const url = "https://www.google.com/search?q=web+developer+in+mumbai&ibp=htl;jobs&hl=en";
const response = await unirest
.get(url)
.header({"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Safari/537.36"})
const $ = cheerio.load(response.body);
Step-by-step explanation:
- In the fifth line, we made a GET request to the target URL.
- In the following line, we passed User-Agent as a header with the URL, so our bot can mimic a real organic user.
- Next, we load the response in a Cheerio instance variable.
Now, we will prepare our parser by searching the tags with the help CSS selector gadget, as stated above in the above section where we stated what all requirements could be needed to scrape.
All the jobs tabs you saw above are inside an HTML tag li.iFjolb . So its parser will look like this:
let jobs_results = [];
$(".iFjolb").each((i,el) => {
jobs_results.push({
})
})
Then we will start selecting the tags using the CSS SelectorGadget to scrape the required data.
After adding this tag to our scraper, it looks like this:
let jobs_results = [];
$(".iFjolb").each((i,el) => {
jobs_results.push({
title: $(el).find(".PUpOsf").text()
})
})
Then we will parse the company name.
From the above image, we found the tag for the company name as .vNEEBe.
Similarly, we can find the tags for the other required data fields, which makes our parser look like this:
let jobs_results = [];
$(".iFjolb").each((i,el) => {
jobs_results.push({
title: $(el).find(".PUpOsf").text(),
company_name: $(el).find(".vNEEBe").text(),
location: $(el).find(".vNEEBe+ .Qk80Jf").text(),
via: $(el).find(".Qk80Jf+ .Qk80Jf").text(),
})
if($(el).find(".KKh3md").length)
{
jobs_results[i].extensions = [];
$(el).find(".KKh3md .LL4CDc").each((j,el) => {
jobs_results[i].extensions[j] = $(el).text()
})
jobs_results[i].extensions = jobs_results[i].extensions?.slice(jobs_results[i].extensions?.length/2)
}
})
Result:
let jobs_results = [];
$(".iFjolb").each((i,el) => {
jobs_results.push({
title: $(el).find(".PUpOsf").text(),
company_name: $(el).find(".vNEEBe").text(),
location: $(el).find(".vNEEBe+ .Qk80Jf").text(),
via: $(el).find(".Qk80Jf+ .Qk80Jf").text(),
})
if($(el).find(".KKh3md").length)
{
jobs_results[i].extensions = [];
$(el).find(".KKh3md .LL4CDc").each((j,el) => {
jobs_results[i].extensions[j] = $(el).text()
})
jobs_results[i].extensions = jobs_results[i].extensions?.slice(jobs_results[i].extensions?.length/2)
}
})
Using Scrapingdog's Google Jobs API
For a limited project, Python can be used alone to scrape data from Google Jobs. However, if you are looking to integrate this data or want to scale up this extraction process, API would be required.
We at Scrapingdog provide a Google job scraping API to get you the data in JSON format.
Here is how you can use this API:
const axios = require('axios');
axios.get('https://api.scrapingdog.com/google_jobs?query=jobs+in+london&api_key=APIKEY')
.then(response => {
console.log(response.data);
})
.catch(error => {
console.log(error);
});
Result:
{
"meta": {
"gl": "us",
"q": "jobs in london",
"api_key": "APIKEY"
},
"jobs_results": [
{
"title": "Barista | London",
"company_name": "Blank Street",
"location": "London, UK",
"via": "via Greenhouse",
"description": "About Blank Street\n\nAt Blank Street, we believe great coffee should be an everyday ritual. With shops across London, Brooklyn, Manhattan, Boston and DC, we’re the first-ever brand to offer affordable high-quality coffee.\n\nBlank Street originated in an effort to change the specialty coffee status quo. Starting with small-format shops and continuing with a limited menu and top-of-the-line tech, we’ve always been focused on simplifying the coffee experience. We partner with amazing local vendors and brands, and have some of the best baristas out there on our team.\nLove coffee and customer service? Keep reading.\n\nWhat's Brewing...\n\nBaristas at Blank Street must be able to work independently, while also thriving in a team environment. The right person for this role is a proven people-person, taking pride in giving amazing service experiences. You are customer-centric and believe in serving a carefully curated menu with passion and precision, making every effort to give your customers... exactly what they need, every day, in their way.\n\nOur Values\n• Magic is in the Details: in everything we do we value attention to detail, going the extra mile & thinking about everything that makes a moment meaningful\n• Move as One: We look for inclusive and respectful team members who strive to be the best team player & who over-communicates to ensure understanding\n• “My House is your House” Hospitality: We look for individuals who welcome guests into the cafe the same way they would invite in a friend, who are obsessed with presentation, making sure our cafes are always clean and welcoming & who provide stellar service, even when things are busy or stressful\n\nWho you are:\n• A friendly and enthusiastic team player with a passion for excellent customer service; you’re always ready to make your customers’ day\n• A reliable employee who is able to effectively manage time and priorities, including during busy rush periods where a sense of urgency is necessary\n• Passionate about creating delicious coffee and other cafe beverages\n• A strong independent problem solver with proven multi-tasking and communication skills\n• Someone who is curious, adaptable and always willing to learn\n• Comfortable working in a team or independently\n• Comfortable maintaining store safety\n\nWhat you'll own:\n• Delight our customers with consistent, welcoming and engaging customer service, taking the opportunity to turn every customer into a regular\n• Work within a world-class coffee program using the best coffee equipment (the Eversys cameos and shot masters) and products in the industry to prepare the tastiest drinks\n• Complete all training to ensure proper measures are in place to achieve the correct handling of food and beverage to retain its freshness and quality\n• Have systems in place to avoid wastage of product with both food and beverage, following and adjusting par levels where needed in line with business peaks and lows\n• Adhere to Health and Safety Regulations and Food Safety Standards at all times\n• Full and complete knowledge and adherence to all product, service and brand training playbooks\n• Participates in all initiatives with the Operations and marketing teams to increase foot traffic, new customers, daily sales, and operational profitability and develop a loyal neighbourhood customer base\n\nRequirements:\n• Experience in the customer service or hospitality industry\n• Availability that meets the needs of our cafes\n• Part-Time: 24 hours per week, 3 days of availability required and must be available to work 2 peak days (Peak days: Friday, Saturday, Sunday)\n• Full Time: 35 hours per week, 4-5 days of availability required and must be available to work 2 peak days (Peak days: Friday, Saturday, Sunday)\n• Weekend and holiday availability preferred\n• You must be authorised to work in the UK\n\nPerks:\n• £12.00 - £13.00\n• Industry-leading pay, bi-weekly payments, pension scheme\n• Barista training program and exciting growth opportunities\n• Unlimited drinks and free food - when on shift\n• 50% off all drinks and food - when you are off your shift\n• Great work-life balance\n• Referral incentive scheme - £250 when you refer a friend\n\nApplication & Interview Process:\n• Online Questionnaire\n• Recruiter Phone Interview\n• Hiring Manager In-Person Interview\n• Offer",
"url": "https://www.google.com/search?ibp=htl;jobs&q=jobs+in+london&htidocid=5efcMuKLmz4lae9WAAAAAA%3D%3D&hl=en-US&kgs=1a84be9da335179c&shndl=-1&shem=ssic&source=sh/x/im/textlists/detail/m1/1#fpstate=tldetail&htivrt=jobs&htiq=jobs+in+london&htidocid=5efcMuKLmz4lae9WAAAAAA%3D%3D",
"extensions": [
"Full-time and Part-time",
"No degree mentioned"
],
"apply_links": [
{
"title": "Apply on Greenhouse",
"link": "https://boards.greenhouse.io/blankstreet/jobs/5210435003?utm_campaign=google_jobs_apply&utm_source=google_jobs_apply&utm_medium=organic"
},
{
"title": "Apply on LinkedIn",
"link": "https://uk.linkedin.com/jobs/view/barista-london-at-blank-street-3667913102?utm_campaign=google_jobs_apply&utm_source=google_jobs_apply&utm_medium=organic"
},
{
"title": "Apply on Monster.co.uk",
"link": "https://www.monster.co.uk/job-openings/barista-london-eng--4c52d2ff-80e3-48a4-ae79-0e6935f36fd6?utm_campaign=google_jobs_apply&utm_source=google_jobs_apply&utm_medium=organic"
},
{
"title": "Apply on Left Lane Job Board",
"link": "https://jobs.leftlanecap.com/companies/blank-street/jobs/15383388-barista-london?utm_campaign=google_jobs_apply&utm_source=google_jobs_apply&utm_medium=organic"
},
{
"title": "Apply on Base10 Job Board",
"link": "https://careers.base10.vc/companies/blank-street/jobs/15383388-barista-london?utm_campaign=google_jobs_apply&utm_source=google_jobs_apply&utm_medium=organic"
},
{
"title": "Apply on General Catalyst Job Board",
"link": "https://jobs.generalcatalyst.com/companies/blank-street/jobs/15383388-barista-london?utm_campaign=google_jobs_apply&utm_source=google_jobs_apply&utm_medium=organic"
},
{
"title": "Apply on Talentify",
"link": "https://www.talentify.io/job/barista-london-england-blank-street-5210435003?utm_campaign=google_jobs_apply&utm_source=google_jobs_apply&utm_medium=organic"
},
{
"title": "Apply on ZipRecruiter UK",
"link": "https://www.ziprecruiter.co.uk/jobs/321842903-barista-london-at-blank-street?utm_campaign=google_jobs_apply&utm_source=google_jobs_apply&utm_medium=organic"
}
]
},
Conclusion
In a nutshell, Google Jobs is a vast repository of job posts from various trusted sources that can be used for identifying industry trends and staying informed about the latest market demands around the world.
In this tutorial, we learned to scrape Google Jobs Results with Node JS. Feel free to message me if I missed something.
Additional Resources
- Web Scraping LinkedIn Jobs using Python
- Web Scraping Glassdoor using Python
- Web Scraping Indeed using Python
- Web Scraping Google Auto Complete Suggestions using Nodejs
- How to Scrape Google Search Results using Python
- How to Scrape Google Maps Data using Python
- Web Scraping with Nodejs
- How to Scrape Google Shopping Results using Python
- Web Scraping Google News using Python
