View source for Help:DFM bots

We have been using a variety of "bot" scripts that interact with MediaWiki and other platforms.  

The bots, which are generally triggered by a user command, facilitate repetitive tasks such as copying information from one system to another -- for instance, using the data from our Zotero library to generate a bibliography page in the wiki, or retrieving a page from the wiki and publishing it to the public-facing website. 

== Installation ==
Most of these scripts are version-controlled at https://github.com/DriedFishMatters/pywikibot-scripts. To use them: 

# Follow the [[mediawikiwiki:Manual:Pywikibot/Installation|instructions on the MediaWiki site]] to install Pywikibot on your computer. If it is a Windows machine, there is the option of using either the built-in Command Prompt or the Windows Subsystem for Linux (WSL); the linux command prompt is preferable because it makes installation of dependencies much easier. 
# Configure a user account for the DFM Wiki on Pywikibot, again following the instructions on MediaWiki. 
# Checkout or download the DFM scripts from GitHub to a <code>myscripts</code> directory inside <code>pywikibot2/scripts/userscripts</code>.  
# From the <code>myscripts</code> directory, install any dependencies by running the command <code>pip install -r requirements.txt</code> 
# Add the following line to your <code>user-config.py:</code> 

<blockquote>
 user_script_paths = ['scripts.userscripts.myscripts']
</blockquote>

== List of scripts ==
{| class="wikitable"
!Script
!Description
|-
|<code>blog2wiki</code>
|Retrieves the RSS feed of recent blog posts and uploads them to the wiki. Used in generating the DFM Newsletter.
|-
|<code>compile_tables</code>
|Parses information from tables in pages in a given wiki category, where each table has two columns representing keys and values, and generates a new page with a table combining the data from all tables. For example, this was used in automatically generating [[E-book abstracts summary table]] out of the pages in [[:Category:E-book synopsis|Category:E-book synopsis]].
|-
|<code>docx2wiki</code>
|Uploads the content of a Word document to the wiki. Reads an input Word document (docx); uploads new images from the document to the wiki, using an image fingerprint hash algorithm to compare images in the document to those already on the wiki and detect visually similar images (e.g., cropped or resized versions of the same image); converts Zotero citations to wiki templates; then uploads the document text to the wiki. This bot is used in converting Word manuscripts to documents and image sets managed on the DFM wiki, notably (but not exclusively) the Scoping reports. Some manual cleanup of the wiki text is generally required. Metadata for newly imported images will also need to be completed. Note that some images of stacks of dried fish are very similar, so they may be detected as duplicates even if they are slightly different.
|-
|<code>featured_pages</code>
|Locates the most recently added items in a featured category, and posts a page to the wiki listing those pages with their summaries. Pages are linked using the canonical URL, so the generated wiki page can be redistributed outside the wiki. This is used for the DFM Newsletter.
|-
|<code>ical2wiki</code>
|Reads an iCalendar file, accessible from a public URL, and updates a wiki page containing a listing of upcoming events from the calendar file. Used on the [[Calendar]] page.
|-
|<code>report</code>
|Downloads a page from the wiki along with high-resolution versions of all the images embedded in that page, retrieves data for any citations on that page that are linked to the Zotero group library, then saves the page as an html document containing front matter, standalone content, notes and bibliography, and table of contents. The resulting document can be used as input for generation of any of the [https://manual.calibre-ebook.com/faq.html#what-formats-does-calibre-support-conversion-to-from output formats supported by Calibre], including EPUB and PDF. See examples on [[DFM Working Papers]]
|-
|<code>trello2wiki</code>
|Reads data from a Trello board and constructs a wiki page for each label on the board, listing the cards sharing that label and the status of each (taken from the card comments).
|-
|<code>trelloattachments</code>
|Downloads all the attachments from cards on a Trello board to the local computer, ignoring those that have previously been downloaded. Useful for backups or for redistributing Trello-managed attachments (like e-book chapter revisions) to people who are not using Trello.
|-
|<code>wiki2html</code>
|Creates a series of static html pages, plus an index with brief extracts, for wiki pages in a given category (in our case, [[:Category:Public|Category:Public]]. These can then by synchronized with the web server using <code>rsync</code>. For example, see [https://driedfishmatters.org/pub/ https:/driedfishmatters.org/pub/]. Images are downloaded and links are adjusted so that the content can be distributed independently of the wiki, without links to internal content or editing functions.
|-
|<code>zotero_bibliography</code>
|Downloads reference details from a collection in a Zotero library and generates a complete bibliographic listing of items in that collection, sorted by item type, with abstracts and links to downloadable attachments. This is used on the page [[Publications]], which is exported to https://driedfishmatters.org/pub/publications.html. 
|-
|<code>zotero_recently_added</code>
|Uploads a list of recent items from a Zotero collection to a wiki page. This is used to generate the list of items on the [[DFM Newsletter]] page, for example.
|-
|<code>zotero2wiki</code>
|Retrieves a list of items in a Zotero collection and uploads the details to a page on the wiki using [[Template:Report]]. This is used to generate [[DFM Working papers]], for example. Note that the "Cover" metadata field must be placed in the "Extra" field as there is no built-in corresponding property in Zotero.
|}

== Examples ==
Copy Scoping research monitoring data from Trello to the wiki
 python pwb.py trello2wiki -key:******** -token:******** -board:HubNzjyl -category: -pagename_prefix:'MEL' -preface:'This document is part of the Dried Fish Matters Monitoring, Evaluation, and Learning workflow. Deliverables listed here are taken from the Terms of Reference (Schedule B) of contracts between the University of Manitoba and partner organizations for Scoping Research. <!-- DO NOT EDIT! This file is updated by a bot. Please do not edit manually. -->' -labels:true
Copy e-book section lists to the wiki
 python pwb.py trello2wiki -key:******** -token:******** -board:jBONMwoW -category: -preface:'E-book' -comment:'<!-- DO NOT EDIT! This file is updated by a bot. Please do not edit manually. -->' -lists:true
Download Trello attachments from e-book board -- this is a wiki bot but it doesn't (yet) interact with the wiki
 python pwb.py trelloattachments -key:******** -token:******** -board:jBONMwoW -basedir:/mnt/g/My\ Drive/e-book-submissions
Update the calendar
 python pwb.py ical2wiki -calendar:'<nowiki>https://calendar.google.com/calendar/ical/dried.fish.matters%40umanitoba.ca/public/basic.ics'</nowiki> -pagename:Calendar -preface:'<nowiki>{{Calendar intro}}</nowiki>'