Michał Frąckowiak has begun to take action against bots and scrapers on Wikidot. Because services like Crom are critical infrastructure for the wiki, it’s important that we know how to avoid blacklists and maintain our ability to retrieve data from the site.
This document will maintain an up-to-date list of necessary countermeasures:
IP shuffling
Ensure requests come from varied IP addresses which look legitimate, ideally residential.
User-Agent
The reported
User-Agentfor requests should look like a common browser user agent, for instance Firefox or Chrome on default settings.
Headers
The
RefererandOriginheaders should be set to appropriate values.
wikidot_token7This should look like an MD5 hash value, occasionally refreshed with a new value.