User:GreenC/WaybackMedic 2.5

From Wikipedia, the free encyclopedia
< User:GreenC  (Redirected from Wikipedia:WAYBACKMEDIC)
Jump to navigation Jump to search
Sign first aid.svg
by GreenC

Wayback Medic 2.5 is a bot that adds and maintains links from the list of known webarchive services in use on the English Wikipedia.

Edits made after 2018-12-04 are by version 2.5

The bot operator is User:GreenC. The bot account is User:GreenC bot. The bot (software) is "WaybackMedic".

WaybackMedic Fixes
Fix number Function name Example edit Description Notes Date added
1 fixthespuriousone Example Remove spurious |1= in cite templates. August 2016
2 fixmissingprotocol Example 1. Add https if protocol missing from the URL.
2. Convert existing protocol http to https.
3. Add second-level domain web if missing ( →
4. Add /web/ path ( → In some URLs adding /web/ breaks the link, test for those.
HTTPS per RFC August 2016
3 fixemptyarchive Example 1. If |archiveurl= is empty or missing but |archivedate= has content, attempt to find a working archive URL based on the archive date, otherwise add {{dead link}} if appropriate.
2. If |archivedate= is empty or missing but |archiveurl= has content, generate date value based on timestamp in the archive URL.
3. If |archiveurl= and |archivedate= are empty, remove both and leave a {{dead link}} if appropriate.
August 2016
4 fixbadstatus Example Check all Wayback Machine URLs for response code errors (anything but 200s). If an error code, try for a better URL via the Wayback API - first using accessdate, then using the earliest date available. If none there, check WebCite API. Try Memento API which checks a few dozen other archives. Other techniques undocumented. If still none found, remove |archiveurl= and |archivedate= and add {{dead link}}. August 2016
5 Retired
6 fixemptywayback Example The wayback template is mangled in a certain way. Action: re-assemble. It won't delete multiple instances if they exist in the same ref (as in the Example). August 2016
7 fixencodedurl Example The URL was incorrectly encoded. Fully decode URL and re-encode. August 2016
8 fixdatemismatch Example 1. Ensure |archivedate= matches the snapshot date in the URL
2. Ensure date format matches dmy or mdy if set (retain ymd if in use)
August 2016
9 fixwebcitlong Example
Convert WebCite URL's from short-form to long-form
Convert URL's from short-form to long-form
WebCite Usage January 2017
10 fixstraydt Example Remove stray {{dead link}} template when an archive exists for the link January 2017
11 fixwam Example Merge {{wayback}} and {{webcite}} --> {{webarchive}}
Merge completed February 5, 2017
Webarchive TfM January 2017
12 fixiats Example archive url -> |archive-url) January 2017
13 fixswitchurl Example Move an URL from |url= to |archiveurl= and add |archivedate= if missing. January 2017
14 Retired
15 fixembway Example
1. A {{wayback}} is embedded in a CS template.
2. A {{dead link}} is embedded in a CS template.
January 2017
16 <various> Example Timestamp and/or |archivedate= is 19700101 and/or out-of-bounds. January 2017
17 fixdoubleurl Example URLs are doubled, tripled, etc.. January 2017
18 fixemptywebarchive Example {{webarchive}} |date= is missing or empty value. January 2017
19 fixdoublewebarchive Example Remove duplicate {{webarchive}} instances. January 2017
20 fixembwebarchive Example A {{cite web}} is embedded in a {{webarchive}} January 2017
21 fixarchiveis Example
1. Convert URL's from short-form to long-form
2. Fix URL encoding of broken links Usage January 2017
22 fixitems Example Change "/items/" URLs that are using machine IDs BRFA January 2017
23 encodemag Example Convert MediaWiki encoding to url encoding in URLs (ie. {{!}} and {{=}}) RFC3986 January 2017
24 decodespace Example Convert %20 to +, + to %20, etc.. in URLs that can be repaired this way See also June 2017
25 waytree_trailgarb Example
Remove typical garbage characters found at the end of URLs: .,;:-"l(%XX)('') February 2018
26 fixcommentarchive Example Open-up commented-out archives and add a |deadurl= "yes" or "no" February 2018
27 waytree_x2encoding Example Repair double URL-encoding eg. %3A -> %253A February 2018
28 fixencodebug Example Repair missed URL-encoding of square brackets T186417 February 2018
29 fixiats Example
Restore truncated Wayback URL February 2018
30 fixiats Example Convert |title={title} -> |title=Archived copy T203865 September 2018
31 urlchanger Example Move broken URL to a new working URL and undo previous archives. BOTREQ November 2018
32 cosmetic
Edits that might be cosmetic. Only with other edits.
1. Del trailing # in URLs
2. Del empty archive fields
3. -->
4. Fix double fragments
5. Convert protocol-relative URLs
WP:PRURL, T214855, January 2019
Technical details
  • Changes to URLs are checked against the remote site to ensure they are working
  • Real-time link checks, no link database. However, links are checked over a 24hr period before final upload of diff.
  • Supports many APIs including Internet Archive, Memento, WebCite and "Timemap" APIs at individual services
  • Multiple HTTP header status code checks at the application (WaybackMedic) layer
  • Additional time-out & retries built-in to the web transfer libraries.
  • Additional operating-procedure level checks against network and other errors - bot is semi-supervised in known trouble areas.
  • Multiple redundant checks of the APIs using multiple dates to ensure a page really is unavailable
  • Accepts API results but then verifies by looking at page headers and/or contents
  • The bot is primarily written in Nim (compiles to C source) with support utilities in Awk. Libraries were custom made including a string primitives library for regex, a wiki template parsing library, OAuth library (in awk), a MediaWiki API interface library, a soft404 detector.
  • Due to the nature of the task, running the bot includes a fair amount of supervisory overhead so it requires operator training, though the steps are documented in the source package.


About every 2-3 months, the bot creates a new batch of articles to process, about ~50k-100k, taking about 1-2 weeks to complete, then takes a break before the next batch 2-3 months later. Typically it follows behind IABot editing the same articles IABot did during that 2-3 month period. This is because WaybackMedic started life as a bug fixer for IABot, a task it can still perform as needed. Also because WaybackMedic does not have a dead link checker so it relies on IABot to tag links dead so it knows which ones might be saved.