We collect data from websites, including protected ones: products and prices, jobs, résumés, quotes. Collection runs on a schedule, and progress is tracked in one panel.
What we did
Schedule
Daily, weekly, or monthly runs with priorities.
Job status
Which jobs are running and which are finished, with logs and statistics.
Scrapy spiders
Uploading, editing, and deploying new versions to Scrapyd.
Server resources
CPU and memory for each scraper.
Control panel on Next.js and Django, with data and jobs stored in PostgreSQL.


Stack
Python, Scrapy, Scrapyd, SpiderKeeper, Splash, Selenium, Django, Next.js, PostgreSQL.
8 927 874 48 31