Async Crawler Documentation
Asynchronous Scraping using Webhooks.
Points to consider while using Asynchronous Scraping
- You have to save the webhook URL on your dashboard
- Webhook URL should support POST request and should respond within 5 seconds with NO CONTENT and status as 200, 204 or 201.
- You just have two use an additional param other than the Scraping API in Async API that is scraper
- Using scraper we will identify the Webhook POST URL.
- You can create 7 webhooks.
- You can identify the success of the request by sd-status. It will be 200 for every successful request.
- You can make 5 requests each second.
On Demand Data Delivery.
You provide a webhook endpoint and the scraping data will be delivered there with your total control.
curl "https://api.scrapingdog.com/webhook?api_key=5ea317fee83fsdsd2231583332581a67&url=url&scraper=scraperName"
This will generate a Scraping Id as response as shown in Results.
Results
{"sid":"8566d6fe-92dc-45a3-b5c2-8910c71bb766"}