IATI Tech update: Q1 - Q2 2021

This post has been written by Wendy Thomas, IATI Technical Lead based at Development Initiatives.

Welcome to the IATI Technical Team’s quarterly update. Every quarter, we update our community on the progress made on IATI tech and share our plans for the next quarter. The Technical Team’s quarterly plans are reviewed by IATI’s Governing Board technical focal points. You can read the last quarter's update here.

Over the last quarter, the Technical Team has been busy implementing multiple recommendations from IATI’s Technical Stocktake 2020, to help transform IATI publishing and data services.

Following a successful start to 2021, I am very pleased that our team is on track to meeting the ambitious roadmap of post-Stocktake projects.

Following a successful start to 2021, I am very pleased that our team is on track to meeting the ambitious roadmap of post-Stocktake projects. As agreed (and monitored closely) by the Governing Board,

API Gateway

One of the core recommendations from the Technical Stocktake is to move IATI’s set of individual, siloed technical tools and products towards a single, integrated system architecture. This work will provide a more unified user experience, delivering consistency and eliminating redundancies within the system. As a key step towards implementing this recommendation, the IATI Governing Board agreed to establish an API Gateway.

A new API Gateway will offer technical users a single point of access to IATI’s APIs. This will enable us to:

• Provide all IATI API services, documentation and change logs in one place.
• Better track and understand how our APIs are being used, via analytics and monitoring tools.
• Help identify more real-world use cases which we could better accommodate in future releases of our services.
• Enable us to protect IATI APIs from overuse and abuse.

In Q1, the Technical Team began work to create an integrated system architecture for IATI’s technical services on Microsoft Azure. We started by implementing a prototype of the API Gateway to fully establish the flexibility of Azure’s API management components.

The first resource we will establish on the API Gateway will be a new public API for IATI’s Validator. Future phases will then bring existing API services behind the Gateway, and throughout this process every effort will be made to ensure the minimum disruption to users of those services.

The Technical Team undertook essential engagement with providers of IATI tools and API users to understand the journeys they take across our services, including at IATI’s Virtual Community Exchange (VCE) meeting and through one-to-one engagement. The Community’s feedback has helped in both gathering and better understanding the requirements for APIs.

You can learn more about the API Gateway and see a demonstration by viewing the IATI VCE recording.

Publishing Tool Options

The Technical Stocktake recommended that we deliver a full options analysis for assessing potential approaches to an IATI publishing tool. As a first step, the Technical Team engaged an external consultant (ThresholdWorld) to design and deliver a survey to IATI publishers, to gauge what they require from a publishing tool. The survey and UX (user experience) work began in December 2020 and concluded in February 2021. The goals of this project were to:

• Capture user requirements for an IATI-funded publishing tool; these will be used to draft a detailed Terms of Reference.
• Analyse whether requirements differ for different sized publishers.
• Compare these requirements to existing tools and analyse gaps.
• Use this information to determine if IATI should fund a new publishing tool or back an existing one.

Supported by the IATI Technical Team, Threshold World conducted desk research, initiated one-to-one interviews with stakeholders and publishers, and surveyed all IATI publishers via email. The organisations interviewed were chosen to form a representative sample of all publishers covering a variety of sectors, organisation sizes, geographical regions and publishing tools. At the March 2021 meeting of the IATI Governing Board, the Technical Team presented the findings of the research. Work is underway to implement the next steps of this project and a detailed blog to update IATI’s Community will be published shortly.

Validator

The Technical Team delivered work to improve the IATI Validator tool which automatically checks IATI data files for errors. The Validator’s client-side performance was significantly improved with the release of 1.5.x, which upgraded Angular from a ten-year old, unsupported version (1.7) to the most recent (11.1).

Earlier this quarter we held a consultation through IATI Connect with the IATI Community on the API contract and work is now underway to implement the public Validator API. In parallel, we are porting the existing functionality from v1 to v2 and onto the unified platform.

Semantic Data Layer/d-portal v2

One of the Technical Stocktake recommendations is that we develop a new ‘'semantic data layer,' or a curated dataset of IATI data that provides non-XML users with a way to access, understand and use the data more easily. To do this we will be undertaking a UX project to learn from data users what formats, transformations etc they require to access data most usefully. We will also review and incorporate existing data use case studies and personas that IATI have established so far. The findings from this project will help IATI’s Secretariat to determine the requirements for d-portal v.2.0, with its specification and design planned by the end of 2021 and development and delivery in 2022.

Terms of Reference for this project were advertised in early March and Ventures Consulting selected as the supplier to do the UX work required. The UX work will run through Q2.

Datastore

One of our priorities this quarter has been working with the Datastore’s supplier to address existing performance issues with the product. We have been investigating how to make sure the Datastore is robust and efficient and we will share an update shortly with the community on our progress.

The IATI Governing Board accepted the Technical Team’s recommendation that IATI not store publishers’ XML.

Hosting Publishers’ XML Files

As part of the Technical Stocktake recommendations we were tasked with developing a policy for hosting publishers’ XML files. Different parts of the current IATI system already host publishers’ XML, but do so separately in each tool, and the cached data is not made publicly available.

In investigating this recommendation, the Technical Team reviewed the main driver for this, which is that hosting publishers’ XML could increase resilience for accessing IATI data, if, for example, a publisher’s site is down. We rarely receive reports of this happening however, and increasingly publishers’ platforms are more robust. Another potential benefit is that this would enable us to store historical data and make that available. This is a theoretical benefit, as IATI currently does not host historical data.

We also assessed the disadvantages and risks of IATI hosting publishers’ XML. A fundamental concern is that this move could lead to a lack of clarity of who is responsible for the data, which is the publisher. Also, this would mean users could access the data in two places, which could lead to confusion and the potential for the files not being in sync. Lastly, if a publisher needs to quickly take down their data they currently do so themselves, rather than having to update/remove in multiple places or having to notify IATI, not only causing delays but also the risk of IATI making changes to the data.

Our assessment is that the disadvantages of this move outweigh the advantages. The IATI Governing Board accepted the Technical Team’s recommendation that IATI not store publishers’ XML.

IATI Registry API

Last month the Technical Team launched the enhanced Registry API Documentation on IATI’s website, to help technical users to integrate their tools and systems with the Registry. The documentation includes:

• How to access the API
• The available API endpoints
• Detailed description of the endpoints with plenty of practical examples.
• Available in English and French

Non-Functional Requirements

The Technical Team has developed comprehensive Non-functional Requirements (NFRs) which will be applicable to IATI tools and software going forward. These have been shared with the IATI Governing Board and will shortly be shared with the IATI community and made available on our website.

Review of how publishers publish

A core data quality project for us this year is to conduct a systematic review of how organisations use the IATI Standard to publish their data. This will involve looking at what data they publish and how they adapt their data to the IATI Standard, focusing on specific data fields with known data quality issues, additional to those identified in the Validator. In quarter 1, we wrote the framework for assessment, with methodology on elements to be assessed and a list of publishers to include. The initial quantitative data analysis is underway and will continue in Q2.

Guidance consultations

The final review of the humanitarian guidance was completed in February, after thorough consultation with IATI’s Community. This guidance will help align and import IATI humanitarian data into UN OCHA’s Financial Tracking Service, as well as meeting Grand Bargain transparency commitments. The guidance is now live on the IATI website.

The Technical Team launched the traceability guidance consultation in mid-March. Community members were invited to share feedback on IATI Connect and register for a webinar on 28 April. The consultation will be completed in Q2.

We continue to receive a steady number of helpdesk requests (526 received this quarter) and feedback shows a high satisfaction rate (96%)

Helpdesk requests

We continue to receive a steady number of helpdesk requests (526 received this quarter) and feedback shows a high satisfaction rate (96%). We also note that the number of data use queries continues to increase, and this quarter we provided support to eight direct user requests.

Stay updated with the Technical Team’s work by signing up to IATI’s Newsletter using the form below or by engaging with IATI’s Technical Community of Practice on IATI Connect.