I’ve been busy. Since July last year all my spare time has been taken up with a new initiative which I am now proud to launch into “early beta”: a systematic review assistant to help perform the laborious task of classifying thousands of scientific articles.
Sometime ago I watched my professor partner reviewing 5,000 documents manually and boldly claimed I could quickly make an AI to do it for her. I wasn’t exactly wrong, but it was a little bit harder than I thought. The document classification side is partly “just a search engine” but some interesting problems and opportunities emerged from the specific workflow required by the structured review process – such as blinding, teamwork, comparisons and corpus preparation artefacts. These led me to explore a myriad of ML and statistical techniques in various languages and packages. I even did an AI nano-degree MOOC as a refresher.
I have 400 pages of notes, so I will publish some articles on what I learned and how I built the prototype, ranging from classifier algorithm evaluation trials, what I learned from SciKit Learn, designing APIs, designing a scalable application architecture (SOA) under ExpressJS, managing large corpora in MongoDB, integrating Node apps with Python, so much async, so many Promises, abusing Mocha for science, Dockerizing micro-services and just how truly amazing Naive Bayes was for its time. I said I’ve been busy.
I’m now at “Step 3” of the original plan, and am looking for help to take it forward:
a) Alpha testers: scientists who need to do a systematic review, and who are willing to collaborate on helping polish the system to become more of an off-the-shelf product. It’s not finished but I feel it’s already a useful tool. I need feedback from real users to continue iterating the product. Leave a comment to get in touch.
b) Developers: who want to get involved in taking it forward. Check out the repo here: https://github.com/scipilot/sysrev-assist and drop me a line.
Here is the project description from the primary repository:
What is this project?
The aim of this project is to provide a set of tools to help undertake scientific systematic reviews more easily and quickly so they are more likely to be performed at all.
Some scientist such as Ben Goldacre believe systematic reviews are one of the most important steps forward for progressing science itself. “Systematic reviews are one of the great ideas of modern thought. They should be celebrated.” [Bad Science p99]. Academic organisations such as The Cochrane Collaboration already provide rules and guidance, services and tools, but due to gaps in this support during the lengthy review process they are still perceived to be so difficult or laborious that they are not performed as often as they should.
There are commercial offerings such as Covidence and Zotero which offer a well-established range of functionality, some specialised to particular fields. While these are certainly powerful tools, commercial products are sometimes challenging to acquire during research projects.
Our goals:
- to provide free and open software to the science community
- to develop intelligent assistants to automate the laborious aspects of collation and screening
- to establish a community of open-source developers to broaden the creativity and support base
What are Systematic Reviews?
“Systematic reviews are a type of literature review that collects and critically analyzes multiple research studies or papers, using methods that are selected before one or more research questions are formulated, and then finding and analyzing studies that relate to and answer those questions in a structured methodology.[1] They are designed to provide a complete, exhaustive summary of current literature relevant to a research question. Systematic reviews of randomized controlled trials are key in the practice of evidence-based medicine,[2] and a review of existing studies is often quicker and cheaper than embarking on a new study.” https://en.wikipedia.org/wiki/Systematic_review
Systematic reviews can provide the data needed for a meta-analysis, or they can be used as a preparatory stage of any research project to assess the current state of a specific scientific topic.
A typical scenario might be summarised as follows:
- Journal database article search (sourcing articles by keyword, reference/citations)
- Systematic Review Process (filtering thousands down to dozens)
- Data extraction (of experimental methods, results/statistics)
- Meta-analysis and/or further research.
We are focussing on step 2, which itself has a fairly complex set of stages. As mentioned above Cochrane provide good tools and services for steps 1 and 4 but only guidance on how to do 2 and 3, which is up to each researcher to perform.
Outline approach
- Step 1 is to implement a basic web application which can perform a basic review, including article data import, screening process, collaboration and result data export. The application must have a “plugin” architecture to enable future additions.
- Step 2 is to research and develop potential solutions to the perceived roadblocks. For example could screening 5,000 documents be assisted by a natural-language AI? Or could the initial citation/reference searches be improved?
- Step 3 is to widen out the project to collaboration by international, academic developers who will have their own ideas and challenges.
History
The project was originally initiated around 2016 by Dr Nic Badcock and his team (Dept. Cognitive Science, Macquarie University, Sydney, Australia). While performing many systematic reviews, they developed their own software in R and Matlab. The R libraries help to automate the complex task of ingesting and processing articles exported from various online journal database archives. The Matlab GUI allows researchers to screen thousands of articles, keeping track of ratings and comments, while keeping the researcher focused and productive. This first version has been used successfully in several collaborative projects.
In early 2017 Pip Jones (programmer, scipilot.org) teamed up with Nic with the idea of adding an AI assistant to the screening process. After considering the scalability of installing and supporting the Matlab-based GUI and limitations of Matlab itself for this purpose, it was decided to first build a web-based GUI which was compatible with the existing R libraries and data-file formats. The MatLab GUI’s information-architecture was duplicated into an Express JS prototype application with a wireframe front-end.
In 2017 a small research fund was granted by Macquarie University to help kickstart the web project.
An initial machine learning (ML) assistant has been added to the web-based GUI after extensive evaluation of various classification techniques. This is implemented as a separate REST API backed by a Python application which utilises the SciKit-Learn libraries. The API provides classification and search services using fairly standard vectorisation (word-embedding) models, term frequency (TF-IDF), principle component analysis for dimensionality reduction (PCA/LSI/LSA) and nearest-neighbour (KNN) correlation. The REST API could be used independently from the web application.
While the initial ML algorithm in place (in the beta version) does have a little special-sauce in the ranking algorithm, it predominately using very basic, standard techniques. This is because after a long evaluation study, I found there not much improvement in classification accuracy (F-Score) in this specific problem domain. I felt it was best left working as simple and performant as possible and without some of the quirks exhibited by the more complex algorithms.
There is huge and exciting scope for improvement in the assistant concept, but it’s not necessarily in the classification and search accuracy of the algorithms. Buy me a beer if you want to know more.
