Help:DFM bots

We have been using a variety of "bot" scripts that interact with MediaWiki and other platforms.

The bots, which are generally triggered by a user command, facilitate repetitive tasks such as copying information from one system to another -- for instance, using the data from our Zotero library to generate a bibliography page in the wiki, or retrieving a page from the wiki and publishing it to the public-facing website.

Installation

Most of these scripts are version-controlled at https://github.com/DriedFishMatters/pywikibot-scripts. To use them:

1. Follow the instructions on the MediaWiki site to install Pywikibot on your computer. If it is a Windows machine, there is the option of using either the built-in Command Prompt or the Windows Subsystem for Linux (WSL); the linux command prompt is preferable because it makes installation of dependencies much easier.
2. Configure a user account for the DFM Wiki on Pywikibot, again following the instructions on MediaWiki.
3. Checkout or download the DFM scripts from GitHub to a myscripts directory inside pywikibot2/scripts/userscripts.
4. From the myscripts directory, install any dependencies by running the command pip install -r requirements.txt
5. Add the following line to your user-config.py:

user_script_paths = ['scripts.userscripts.myscripts']

List of scripts

Script	Description
blog2wiki	Retrieves the RSS feed of recent blog posts and uploads them to the wiki. Used in generating the DFM Newsletter.
compile_tables	Parses information from tables in pages in a given wiki category, where each table has two columns representing keys and values, and generates a new page with a table combining the data from all tables. For example, this was used in automatically generating E-book abstracts summary table out of the pages in Category:E-book synopsis.
docx2wiki	Uploads the content of a Word document to the wiki. Reads an input Word document (docx); uploads new images from the document to the wiki, using an image fingerprint hash algorithm to compare images in the document to those already on the wiki and detect visually similar images (e.g., cropped or resized versions of the same image); converts Zotero citations to wiki templates; then uploads the document text to the wiki. This bot is used in converting Word manuscripts to documents and image sets managed on the DFM wiki, notably (but not exclusively) the Scoping reports. Some manual cleanup of the wiki text is generally required. Metadata for newly imported images will also need to be completed. Note that some images of stacks of dried fish are very similar, so they may be detected as duplicates even if they are slightly different.
featured_pages	Locates the most recently added items in a featured category, and posts a page to the wiki listing those pages with their summaries. Pages are linked using the canonical URL, so the generated wiki page can be redistributed outside the wiki. This is used for the DFM Newsletter.
ical2wiki	Reads an iCalendar file, accessible from a public URL, and updates a wiki page containing a listing of upcoming events from the calendar file. Used on the Calendar page.
report	Downloads a page from the wiki along with high-resolution versions of all the images embedded in that page, retrieves data for any citations on that page that are linked to the Zotero group library, then saves the page as an html document containing front matter, standalone content, notes and bibliography, and table of contents. The resulting document can be used as input for generation of any of the output formats supported by Calibre, including EPUB and PDF. See examples on DFM Working Papers
trello2wiki	Reads data from a Trello board and constructs a wiki page for each label on the board, listing the cards sharing that label and the status of each (taken from the card comments).
trelloattachments	Downloads all the attachments from cards on a Trello board to the local computer, ignoring those that have previously been downloaded. Useful for backups or for redistributing Trello-managed attachments (like e-book chapter revisions) to people who are not using Trello.
wiki2html	Creates a series of static html pages, plus an index with brief extracts, for wiki pages in a given category (in our case, Category:Public. These can then by synchronized with the web server using rsync. For example, see https:/driedfishmatters.org/pub/. Images are downloaded and links are adjusted so that the content can be distributed independently of the wiki, without links to internal content or editing functions.
zotero_bibliography	Downloads reference details from a collection in a Zotero library and generates a complete bibliographic listing of items in that collection, sorted by item type, with abstracts and links to downloadable attachments. This is used on the page Publications, which is exported to https://driedfishmatters.org/pub/publications.html.
zotero_recently_added	Uploads a list of recent items from a Zotero collection to a wiki page. This is used to generate the list of items on the DFM Newsletter page, for example.
zotero2wiki	Retrieves a list of items in a Zotero collection and uploads the details to a page on the wiki using Template:Report. This is used to generate DFM Working papers, for example. Note that the "Cover" metadata field must be placed in the "Extra" field as there is no built-in corresponding property in Zotero.

Examples

Copy Scoping research monitoring data from Trello to the wiki

python pwb.py trello2wiki -key:******** -token:******** -board:HubNzjyl -category: -pagename_prefix:'MEL' -preface:'This document is part of the Dried Fish Matters Monitoring, Evaluation, and Learning workflow. Deliverables listed here are taken from the Terms of Reference (Schedule B) of contracts between the University of Manitoba and partner organizations for Scoping Research. ' -labels:true

Copy e-book section lists to the wiki

python pwb.py trello2wiki -key:******** -token:******** -board:jBONMwoW -category: -preface:'E-book' -comment: -lists:true

Download Trello attachments from e-book board -- this is a wiki bot but it doesn't (yet) interact with the wiki

python pwb.py trelloattachments -key:******** -token:******** -board:jBONMwoW -basedir:/mnt/g/My\ Drive/e-book-submissions

Update the calendar

python pwb.py ical2wiki -calendar:'https://calendar.google.com/calendar/ical/dried.fish.matters%40umanitoba.ca/public/basic.ics' -pagename:Calendar -preface:'{{Calendar intro}}'