Paid advertisements are an important strategic funnel for growing your customer base and increasing brand awareness and conversion rates.
Businesses can remain very targeted with the data they can get from scraping Google Ads.
Understanding the user intent behind every search query, to monitoring competitors’ rankings and bidding strategies are some of the benefits you get from this data.
In this article, we will learn to scrape Google ads Search Results using Python with Scrapingdog’s Google Search API.
Why Scrape Google Ads?
Google’s massive dominance in the search engine market can drive huge traffic to websites if businesses have the budget and correctly spend over the accurate pinpoints. Scraping Google Ads will provide valuable insights into these pinpoints, and your business will be able to make up-to-the-mark decisions to compete in the market.
Track Competitors
- Which keywords are competitors targeting?
- What messaging are they using to attract customers?
- What is the frequency of their ad placements?
Answers to these questions are necessary to identify gaps in your marketing campaign, the target audience your competitors are looking for, and any weak keywords your competitors are ranking for.
Monitor PPC Investments
Imagine you are running a PPC campaign for an online show retailer. As other competitors were targeting ads, you decided to scrape Google Ads for the keyword “buy running shoes” to track them. Using an API, you extracted the data for this keyword and got these results:
Ad Headlines:
- “Best Running Shoes — Huge Discounts | BrandX”
- “Buy Running Shoes Online — Free Shipping | BrandY”
- “Top Rated Running Shoes — Sale Up to 50% Off | BrandZ”
Descriptions:
- “Shop high-quality running shoes for men and women. Huge discounts this season.”
- “Order running shoes online with free shipping and easy returns.”
- “Comfortable, durable running shoes at the best prices.”
Ad Placement:
- BrandX appears in the 1st position consistently.
- BrandY switches between 2nd and 3rd positions.
- BrandZ appears at the bottom of the first page.
Ad Extensions:
- BrandX uses site links and promotion extensions (e.g., “20% off for first orders”).
- BrandY uses callout extensions (e.g., “Fast Delivery”, “Trusted by Athletes”).
- BrandZ uses no extensions.
You analyzed the data and found some crucial insights like BrandX has been bidding the highest to focus on more clicks as they are consistently appearing higher while BrandZ is not investing much in the budget or has lower bids. Discounts and free shipping are a recurring theme in all the results.
The next step would be to optimize your own PPC strategy. There are many options, you can start by targeting unique points such as “Free same-day delivery” which hasn’t been mentioned by anyone.
You can even bid for a long tail keyword “best running shoes for beginners” which may be less competitive and so on. This may result in higher rankings, much improved CTR, and a great conversion rate.
Scraping Google Ads With Scrapingdog’s Google Search API
The consistent rotation of millions of IPs at the backend is what makes a scraper robust and scalable to produce a reliable data pipeline. Furthermore, it requires steady maintenance of the API for a streamlined and unbreakable flow of data.
That’s a lot of work!
A lot of time and resources to manage the API will be needed. However, at Scrapingdog we have already made a dedicated infrastructure for your needs that handles everything from proxy rotation to headers.
Further in this article, we will build our scraper to extract Google Ads using Scrapingdog’s Google Search API.
Setting up a Scrapingdog Account
Before starting this project, we need to register on Scrapingdog to get our API Key for successful authentication in the API requests.
Registering on the website will redirect you to the dashboard where you can find your API Key.
(Please refer to the image below ⬇️)
Copy this API Key and store it in a safer place as we will use this in a bit.
Here’s a quick demo video tutorial ⬇️ on how you can use Scrapingdog’s Search API to extract Google ADs Data )
Let’s Start Scraping!!
I will be taking up a scenario so that you better understand using the API.
Imagine, you’re a digital marketer of a lead generation company researching for the competition for the keyword “lead generation tools”. You search for this query on Google and find out the space was already occupied by some top companies in your business domain.
Tracking these ads over time can provide valuable insights into these companies’ strategies. You can identify their target customer base, be conscious of their keyword approach, and uncover their bidding strategies. However, this can only be done if we have a reliable API that can regularly update you with the current trends.
Let’s implement this for our project. First, we will import the required libraries, which will help us through the tutorial.
import requests
Then, using the Scrapingdog Google Search API, we will make a GET request for our target keyword, “lead generation tools.”
payload = {'api_key': 'APIKEY', 'query':'lead+generation+tools', 'advance_search': 'true'}
resp = requests.get('https://api.scrapingdog.com/google', params=payload)
data = resp.json()
print(data["ads"])
The payload consists of the various parameters of the Scrapingdog’s Google API. If you wish to add more filters, refer to this documentation. But for now, we have these parameters:
- API Key — A required parameter that will serve as an authentication entity for the API.
- Query — Another required parameter that consists of the target keyword.
- Advance Search — To get featured snippets including ads, knowledge graph, PAA, etc from the response.
Let’s run this script in our terminal and check the results.
"ads": [
{
"position": 1,
"block_position": "top",
"title": "Start Your 14-Day Free Trial - Sales Lead Tool",
"snippet": "Our User-Friendly Design Makes It Easy To Get Started And See Results. Try Pipedrive™ Now. Get Full Access for 14 Days Free. No Credit Card Required. Try Any Feature During Trial. Simple, Flexible Pricing.",
"displayed_link": "https://www.pipedrive.com",
"link": "https://www.pipedrive.com/",
"tracking_link": "https://www.google.com/aclk?sa=l&ai=DChcSEwjM_v7dvMyJAxXxNggFHRifKkkYABAAGgJtZA&co=1&ase=2&gclid=EAIaIQobChMIzP7-3bzMiQMV8TYIBR0YnypJEAAYASAAEgI6xPD_BwE&sig=AOD64_0lQvucrceQN56hnkgkIIOA4YOOQg&q&nis=4&adurl",
"sitelinks": [
{
"title": "Find Your Perfect Plan",
"link": "https://www.pipedrive.com/en/pricing"
},
{
"title": "Pipedrive™ AI",
"link": "https://www.pipedrive.com/en/products/ai-crm"
},
{
"title": "Personalised Onboarding",
"link": "https://www.pipedrive.com/en/products/sales/onboarding"
},
{
"title": "Explore Our Features",
"link": "https://www.pipedrive.com/en/products"
}
]
},
{
"position": 2,
"block_position": "top",
"title": "Free 14 Day Trial - AI Lead Generation - Stop Paying For Leads",
"snippet": "Visual Visitor Matches Company Contact Data With Cutting Edge Visitor Identification Tools",
"displayed_link": "https://www.visualvisitor.com",
"link": "https://www.visualvisitor.com/what-we-do/webid-person/",
"tracking_link": "https://www.google.com/aclk?sa=l&ai=DChcSEwjM_v7dvMyJAxXxNggFHRifKkkYABAEGgJtZA&co=1&ase=2&gclid=EAIaIQobChMIzP7-3bzMiQMV8TYIBR0YnypJEAAYAiAAEgKakfD_BwE&sig=AOD64_0TjdS2ha_g8aQGw4EEbNs73tLnNQ&q&nis=4&adurl"
},
{
"position": 3,
"block_position": "top",
"title": "Try Lead Generation Ads",
"snippet": "Gather high-quality leads — 57% of TikTok users are likely to purchase after viewing a Lead Generation Ad. Start...",
"displayed_link": "https://getstarted.tiktok.com",
"link": "https://getstarted.tiktok.com/",
"tracking_link": "https://www.google.com/aclk?sa=l&ai=DChcSEwjM_v7dvMyJAxXxNggFHRifKkkYABABGgJtZA&co=1&ase=2&gclid=EAIaIQobChMIzP7-3bzMiQMV8TYIBR0YnypJEAAYAyAAEgJuKfD_BwE&sig=AOD64_0eZgCqW_gXJgkhMGIBeDNiIBE9WQ&q&nis=4&adurl",
"sitelinks": [
{
"title": "Affiliate Program",
"link": "https://getstarted.tiktok.com/affiliates"
},
{
"title": "Advertise on TikTok Today",
"link": "https://getstarted.tiktok.com/global-en-videoad"
},
{
"title": "TikTok for Business",
"link": "https://www.tiktok.com/business/en-US"
},
{
"title": "TikTok Lead Ads",
"link": "https://getstarted.tiktok.com/global-en-leadad"
}
]
},
{
"position": 4,
"block_position": "top",
"title": "Lead Generation Tool | B2B Lead Generation Tool",
"snippet": "Seamlessly integrate all your paid channels into one platform and use your CRM data",
"displayed_link": "https://www.metadata.io › tools › b2b",
"link": "https://metadata.io/demandhub-free-trial/lp/",
"tracking_link": "https://www.google.com/aclk?sa=l&ai=DChcSEwjM_v7dvMyJAxXxNggFHRifKkkYABADGgJtZA&co=1&ase=2&gclid=EAIaIQobChMIzP7-3bzMiQMV8TYIBR0YnypJEAAYBCAAEgIzn_D_BwE&sig=AOD64_2doTDhRMNU2Nw-XemHevLKYgVecg&q&nis=4&adurl",
"sitelinks": [
{
"title": "30-Day Free Trial",
"link": "https://metadata.io/demandhub-free-trial/lp/?metadata_cid=2094"
},
{
"title": "Pricing Information",
"link": "https://www.metadata.io/pricing/"
},
{
"title": "Work Smarter Not Harder",
"link": "https://metadata.io/roles/demand-generation-marketers/?metadata_cid=2097"
},
{
"title": "Take A Quick Tour",
"link": "https://metadata.io/platform/?metadata_cid=2098#product-tour"
}
]
},
{
"position": 1,
"block_position": "bottom",
"title": "Try Zendesk®, Free for 14 Days | Start Your Free Trial",
"snippet": "Drive revenue and slow spend with Zendesk's cost-efficient software. Start a free trial. Create meaningful customer connections with Zendesk, the leader in customer experience. Macro-enabled responses. Personalized AI responses.",
"displayed_link": "https://www.zendesk.com",
"link": "https://www.zendesk.com/lp/nb-support",
"tracking_link": "https://www.google.com/aclk?sa=l&ai=DChcSEwjM_v7dvMyJAxXxNggFHRifKkkYABAFGgJtZA&co=1&ase=2&gclid=EAIaIQobChMIzP7-3bzMiQMV8TYIBR0YnypJEAMYASAAEgKc7vD_BwE&sig=AOD64_1EHuqshm7JbraNlnyuqtF4I3scpA&q&nis=4&adurl",
"sitelinks": [
{
"title": "Start A Free Trial",
"link": "https://www.zendesk.com/"
},
{
"title": "Watch Our Free Demo",
"link": "https://www.zendesk.com/lp/demo/"
},
{
"title": "Pricing Plans",
"link": "https://www.zendesk.com/pricing/"
},
{
"title": "Explore Our Pricing Plans",
"link": "https://www.zendesk.com/pricing/support/"
}
],
},
{
"position": 2,
"block_position": "bottom",
"title": "Make 300% More Connections | Dial Prospects Instantly",
"snippet": "Five9®'s Advanced Dialer Options Allow Your Team to Maximize Productivity & Connections. Five9 is Accelerating Sales with Next-Generation Dialer Technology. Learn How Today. 4-Dialing Modes.",
"displayed_link": "https://www.five9.com",
"link": "https://www.five9.com/landing/dialer",
"tracking_link": "https://www.google.com/aclk?sa=l&ai=DChcSEwjM_v7dvMyJAxXxNggFHRifKkkYABACGgJtZA&co=1&ase=2&gclid=EAIaIQobChMIzP7-3bzMiQMV8TYIBR0YnypJEAMYAiAAEgKg3PD_BwE&sig=AOD64_3CDlWrlo4c1YJ8nMTY6wZZNmIvZg&q&nis=4&adurl",
"sitelinks": [
{
"title": "Easy Pricing Bundles",
"link": "https://www.five9.com/products/pricing"
},
{
"title": "2 Minute Demo Video",
"link": "https://www.five9.com/landing/call-center-software"
},
{
"title": "Explore Pricing & Bundles",
"link": "https://www.five9.com/products/pricing"
},
{
"title": "Plan for Capacity",
"link": "https://www.five9.com/resources/whitepaper-five9-guide-to-capacity-planning"
}
],
},
{
"position": 3,
"block_position": "bottom",
"title": "Generate More Sales Leads",
"snippet": "LinkedIn® Sales Navigator — Tap into the power of LinkedIn’s 1B + member network to drive more leads and close deals. Zero in on decision-makers with LinkedIn Sales Navigator. +37% larger sales deals.",
"displayed_link": "https://business.linkedin.com",
"link": "https://business.linkedin.com/sales-solutions/cx/23/08/linkedin-sales-navigator",
"tracking_link": "https://www.google.com/aclk?sa=l&ai=DChcSEwjM_v7dvMyJAxXxNggFHRifKkkYABAMGgJtZA&co=1&ase=2&gclid=EAIaIQobChMIzP7-3bzMiQMV8TYIBR0YnypJEAMYAyAAEgJRsfD_BwE&sig=AOD64_0QmgmF1LgfAUbETJ7LdY0LupLWQw&q&nis=4&adurl",
"sitelinks": [
{
"title": "B2B Sales Leads",
"link": "https://business.linkedin.com/sales-solutions/cx/16/01/b2b-sales-leads-demo-sem"
},
{
"title": "Sales Lead Generation",
"link": "https://business.linkedin.com/sales-solutions/cx/16/01/b2b-sales-leads-demo-sem"
},
{
"title": "Request Free Demo",
"link": "https://business.linkedin.com/sales-solutions/cx/16/01/request-demo-sem"
}
],
}
]
Great! We got 7 Ads — 4 at the top of the page and 3 at the bottom.
This is how you can scrape Google Ads. We have used Python for this project, however, the same can be done in Node JS, Go Lang, Ruby, PHP, and Java.
Moreover, you can implement a NodeCron for tracking Ads which can email or notify you whenever there are any fluctuations in the rankings.
Conclusion
Search engines have massive data, and scraping them or at least one of them can give you an edge over the competitors.
With every other optimization in your marketing channels, optimizing Google Ads is also a must if your competitors are actively getting customers from it!!
While Google provides an official API for accessing Ads data, it has limitations in terms of flexibility, data handling, and cost-effectiveness.
Therefore, using a 3rd-party Google Search Result API offers much better control and customization for your data.
I have also written a guide on scraping google search results using Python, do check it out!!
I hope you liked this article and if you do, please share this article on Social Media.
Thanks for reading it!👋
Additional Resources
