Async Crawler Documentation

Asynchronous Scraping using Webhooks.

Points to consider while using Asynchronous Scraping

  • You have to save the webhook URL on your dashboard
  • Webhook URL should support POST request and should respond within 5 seconds with NO CONTENT and status as 200, 204 or 201.
  • You just have two use an additional param other than the Scraping API in Async API that is scraper
  • Using scraper we will identify the Webhook POST URL.
  • You can create 7 webhooks.
  • You can identify the success of the request by sd-status. It will be 200 for every successful request.
  • You can make 5 requests each second.

On Demand Data Delivery.

You provide a webhook endpoint and the scraping data will be delivered there with your total control.

curl ""

This will generate a Scraping Id as response as shown in Results.