IATI Tech update: Q2 - Q3 2021

This post has been written by Wendy Thomas, IATI Technical Lead based at Development Initiatives.

Welcome to the IATI Technical Team’s quarterly update. Every quarter, we update our community on the progress made on IATI tech and share our plans for the next quarter. The Technical Team’s quarterly plans are reviewed by IATI’s Governing Board technical focal points. You can read the last quarter's update here.

The Technical Team is continuing to implement multiple recommendations from IATI’s Technical Stocktake 2020, to help transform IATI publishing and data services. These are outlined in the roadmap of post-Stocktake projects. As agreed (and monitored) by the Governing Board, this work is essential to ensure the stability and future-proofing of IATI tech.

API Gateway

The API Gateway and accompanying Developer Portal were released in mid September, providing a single access point for all of IATI’s API data services. Delivering the new Gateway is a core recommendation from the Technical Stocktake and will enable the Technical Team to better track API usage and to protect tools from overuse and abuse. The new Validator Public API is the first resource available on the Gateway; we are working to add all IATI APIs over time.

IATI Publishing Tool

As mentioned in a newspost in June, the Technical Stocktake recommended that we deliver a full options analysis for assessing potential approaches to an IATI publishing tool. Based on that options analysis and a user experience (UX) project delivered earlier this year, the IATI Governing Board has decided that IATI should launch an open procurement for the provision of a free-to-use IATI publishing tool. The Secretariat has finalised the Terms of Reference for the procurement, and it has been published online with the deadline of 18 October for submissions.

Validator

IATI Validator v2.0 was released in mid-September, offering an improved user experience by checking data files with greatly increased speed and allowing users to easily identify and prioritise different types of errors.

Included with Version 2.0 is the new public Validator API, which allows organisations to validate IATI documents on demand in a matter of seconds, for the majority of cases. We believe this can become a driver of data quality, enabling publishers to validate their data before publication.

Unified Platform

This technical change moves IATI from a siloed set of technical tools and products to a single, integrated system architecture. IATI’s Validator is the first product to have moved to IATI’s unified platform with its Version 2.0 release. A key demonstrator of the power of the Unified Platform is that, from standstill, we now identify from the Registry all 8000+ IATI documents, download them and validate them in a matter of hours; this contrasts with the previous version of the Validator for which the same would have taken a matter of days. The next iteration of the Datastore will be built on this platform, and our expectation is that users can expect their IATI documents to be validated and indexed in the Datastore well within 24 hours from entry to the Registry.

Semantic Data Layer (SDL) project as research for d-portal v2

In quarter 2 we initiated a project with Ventures Consulting to learn more about how users who are not familiar with IATI use and access IATI data. The project involved a research study with a dozen end-users of IATI data, concentrating on those groups who have not used or have struggled to use IATI data in the past, such as investigative journalists, academics and those working on research roles at civic society organisations.

Outputs from the project include a ‘researcher persona’ with a model for how this group understands IATI data and the challenges they expect to encounter with it. The project also outlined a high-level product strategy for this user group; this will help inform future end-user product requirements and UX designs and will be incorporated in developing requirements for d-portal V2.0. We will share a newspost on the SDL project soon.

Datastore

This summer, the Technical Team began work to bring the Datastore in-house, to be included within IATI’s new unified platform. A Datastore API is scheduled to be launched later this year on the new API Gateway, offering technical users a single point of access for IATI’s API data services. We will then look into a new web interface for the Datastore, after determining what functionality should be included in the Datastore and what should be incorporated in d-portal v2. During this period users can continue to access IATI data using IATI Datastore Classic, an updated version of the original IATI Datastore, or Zimmerman’s iati.cloud

IATI Registry

Maintenance work has been on-going for the Registry with our supplier Derilinx, who have made great progress on bug fixes and issues, resolving a total of 15 issues, including Registry timeout errors. The IATI developer team met with Derilinx to discuss the Registry Solr version vulnerability; Solr Upgrade work is being assessed and planned by the developers to be delivered in quarter 3 - 4. In addition to maintenance work, the key focus for the rest of the year is on delivering an agreed set of user requirement enhancements to the Registry that will help our users have a better user journey when registering to publish data to IATI.

Non-Functional Requirements

The Technical Team has developed comprehensive Non-functional Requirements (NFRs) which will be applicable to IATI tools and software going forward. These have been shared with the IATI Governing Board and are now available on our website.

Review of how publishers publish

A key activity in quarter 2 and 3 has been the systematic review of how publishers publish (looking at what data they publish and how they adapt their data to the IATI Standard), focusing on specific data fields with known data quality issues, additional to those identified in the Validator. Following a set template we completed a quantitative as well as qualitative analysis for a sample of 100 publishers, including governments, multilateral organisations, DFIs, private sector and NGOs. We shared a summary report with initial findings ahead of a session to engage publishers at the IATI Virtual Community Exchange 2 (12-13 October) and additional follow-ups are planned in quarter 4.

Data Quality Index

As part of our commitment to support organisations to publish the highest quality IATI data, we are developing a set of revised data quality measures for all IATI publishers, based on definitions and use cases agreed with our members and community. We launched a consultation on IATI Connect during September and encouraged all stakeholder groups and people with different levels of IATI knowledge to give feedback. We also held a session on the Data Quality Index at IATI’s Virtual Community Exchange 2. You can read the background paper here.

Guidance consultations

IATI traceability guidance was shared with the community for feedback and discussion via a webinar at the end of April. Final review of the updated guidance took place by the end of May and is now live on the IATI website.

There are no specific planned consultations in Quarters 3 and 4; we are awaiting input from the community on any topics they’d like additional guidance on.

Helpdesk requests

We continue to receive a steady number of Helpdesk requests (988 received from April to end of August) and feedback shows a high satisfaction rate (96%). We also note that the number of data use queries continues to increase, and this quarter we supported about 20 direct user requests.

Stay updated with the Technical Team’s work by signing up to IATI’s Newsletter using the form below or by engaging with IATI’s Technical Community of Practice on IATI Connect.