Wikidot Request Countermeasures

Michał Frąckowiak has begun to take action against bots and scrapers on Wikidot. Because services like Crom are critical infrastructure for the wiki, it’s important that we know how to avoid blacklists and maintain our ability to retrieve data from the site.

This document will maintain an up-to-date list of necessary countermeasures:

• IP shuffling

  • Ensure requests come from varied IP addresses which look legitimate, ideally residential.

• User-Agent

  • The reported User-Agent for requests should look like a common browser user agent, for instance Firefox or Chrome on default settings.

• Headers

  • The Referer and Origin headers should be set to appropriate values.

• wikidot_token7

  • This should look like an MD5 hash value, occasionally refreshed with a new value.