We collect data from websites, including protected ones: products and prices, jobs, résumés, quotes. Collection runs on a schedule, and progress is tracked in one panel.

What we did

  1. Schedule

    Daily, weekly, or monthly runs with priorities.

  2. Job status

    Which jobs are running and which are finished, with logs and statistics.

  3. Scrapy spiders

    Uploading, editing, and deploying new versions to Scrapyd.

  4. Server resources

    CPU and memory for each scraper.

  5. Control panel on Next.js and Django, with data and jobs stored in PostgreSQL.

Screenshot of Scraping sites with bot protection (#2)
Scraping sites with bot protection: screen 2
Screenshot of Scraping sites with bot protection (#3)
Scraping sites with bot protection: screen 3

Stack

Python, Scrapy, Scrapyd, SpiderKeeper, Splash, Selenium, Django, Next.js, PostgreSQL.