The system completes multi-step forms on sites with bot protection, for example government portals in Russia and other countries. One example is registration on a Spanish government services portal protected by F5 Distributed Cloud Bot Defense.

What we did

  1. Our own headless browser and proxy

    Full control over the browser environment and address rotation.

  2. Step-by-step forms

    Filling in wizards and submitting forms.

  3. Parallel workers

    Parsing and form submission run in separate processes and scale independently.

  4. API

    a page screenshot, fetching HTML, queuing a task, and serving the next task from the queue.

  5. Debugging over VNC

    You can connect to the browser and see where a scenario stopped.

Screenshot of Form autofill on bot-protected sites (#2)
Form autofill on bot-protected sites: screen 2

Stack

Python, Selenium, Scrapy, a custom headless browser build and proxy, VNC.