IATI Technical Update September 2022

This post has been written by Wendy Thomas, IATI Technical Lead based at Development Initiatives.

Welcome to the IATI Technical Team update blog, outlining progress made on IATI tech and our plans for the next quarter. The Technical Team’s work plan is regularly reviewed by IATI’s Governing Board technical focal points. You can read our last update here.

This Technical Team update reflects progress so far on our 2022 work plan, which has been revised in light of the developments with the institutional hosting arrangements for IATI. As the mandate of current Secretariat hosts (UNDP, UNOPS and Development Initiatives) concludes on 31 December, the IATI Governing Board has agreed to place some work plan activities on hold. Most of these are activities that, if started, would not be concluded by the end of the current hosting period. For the IATI Technical Team, this means plans for an improved d-portal V2 are on hold, as is any further work on the Data Quality Index. This will not affect our regular activities of servicing existing tools and providing support to publishers through the Helpdesk, which we will maintain operational until new hosts are appointed and ready to take over these tasks. You can read more about the hosting arrangements here.

September 2022 Update

Datastore

The Technical Team launched IATI’s Datastore Search tool in March, providing access to all valid development and humanitarian data published to IATI in CSV, XML and JSON formats via a simple website. Datastore Search replaces the previous Query Builder tool. The tool offers several additional functions to enable complex queries of data published to IATI:

• Robust free text searches
• Flexible searching to allow users to select data across any combination of elements or activities in the IATI Activity Standard
• Extended, non-paginated exports so users can download thousands of activities/budgets/transactions at once.
• Datastore Search utilises the IATI Datastore API Version 3, so data is up to date within 24hrs of publishing in the IATI Registry.

Since the launch we have delivered 5 minor version releases (v1.1.0 – v1.5.0) with enhancements and bug fixes, such as Excel-optimised CSV downloads, validation when importing filters and on advanced searches, and easier to use codelists. We have also upgraded and improved the Datastore Solr infrastructure. You can read about these releases here.

Validator

To ensure more valid IATI data is available in the IATI Datastore, and in response to requests from the IATI Community, we delivered Activity Level Validation Phase I in May.

• Phase I enabled the Validator to individually assess each activity contained within activity files and to exclude only activities with critical errors. Previously, the Validator assessed activity files (which contain multiple activities) and if any data in the file was critically invalid then no data in the file would be able to enter the Datastore. The Validator User Interface was enhanced to indicate whether a file was fully or partially available in the Datastore as a result.
• Phase II of Activity Level Validation is under development and planned to be delivered in Q4. It builds on Phase I by enhancing the Validator API and Validator to show critical IATI XML schema errors inline at an activity level to help publishers pinpoint which activities have critical schema errors (currently, critical schema errors are returned/displayed only at a file level).

As part of our wider work to help publishers improve the quality of their data, we have contacted external publishing tool providers to encourage them to integrate the Validator API and offer them support in that process.

Publishing Tool

The IATI Secretariat and Technical Team have continued to work with Young Innovations to progress the new IATI publishing tool, IATI Publisher. The tool will provide an IATI-owned, free to use publishing tool for small and medium sized organisations. We are pleased that the development of the tool is nearly complete and soon ready for user testing. The Technical Team will work with Young Innovations to test the tool on its target users:

1. Organisations that have previously published IATI data (using an alternative publication tool); and
2. Organisations preparing to publish IATI data for the first time.

If you would like to be involved in testing IATI Publisher, kindly send an email to [email protected].

Registry

Over the last few months we have focused on investigating user login issues and the Registry’s hashing behaviour, which impacts the Unified Platform. We have also started work on adding validation status to the IATI Registry, to enable publishers to see the IATI Validator’s validation status of their datasets, so that they are aware of any errors that need to be fixed.

d-portal

We recently worked with d-portal’s supplier to carry out essential maintenance work; the server was updated to the latest version of Node.js and a NGINX upgrade completed. In addition, the d-portal website headers were adjusted so that CORS (Cross-origin resource sharing) compliant servers can securely retrieve data.

Google's Universal Analytics, the web analytics service used to provide user statistics for d-portal, is being phased out so work was completed to prepare to switch to Google Analytics 4. Research and testing is underway to see if the data collected using the new service is a good replacement. The new GA4 service is now set up and running in parallel to the old service to allow for checking of live data as it is generated.

As mentioned above, further development of d-portal v2 is on hold until new hosting arrangements for IATI are in place.

Data Quality Index

As part of IATI’s commitment to supporting organisations to publish high-quality data, the Secretariat has been developing a set of revised data quality measures for IATI’s 1500+ publishers, which will result in an IATI ‘Data Quality Index’ (DQI). The DQI is being progressed via a series of consultations. In quarter 2 the Secretariat concluded:

• Consultation 2: Data Completeness, with extensive feedback received on measures such as Sustainable Development Goals, Classifications, Identifiers, etc. You can read the summary here.

As mentioned above, further work on the DQI, including the third consultation (Weights and Measures) is on hold until new hosting arrangements are in place.

Helpdesk

Users of the IATI Support Helpdesk may have noticed slower than usual response times in quarter 2 and quarter 3, due to staffing changes on the Technical Team (see staffing updates below). Where possible during this period the Data Use team at UNDP have provided support to data users contacting the Helpdesk, and helped with onboarding of our new team members. We have now recruited additional Business and Data Analysts, which should enable us to return to our usual service response times. Please contact me directly ([email protected]) if you have any concerns with responses via our Helpdesk.

Staffing updates

Sadly we have bade farewell to Business and Data Analysts Audrey Migot-Adholla, Amy Silcock and Andi Szenasi. We have welcomed new members of the team: Christian Kilonzo, an Information Analyst with Development Initiatives; Titi Olukole, a project coordinator and analyst; and David Eade, a business analyst/project manager.

Forthcoming work

Our focus for the rest of this year will be business-as-usual activities to support publishers and improve Data Quality; whilst progressing delivery of the IATI Publishing Tool. The Technical Team is also beginning to work on what will be required to transition the IATI technical estate, helpdesk functions and publishing support to new hosts.

Stay updated

Stay updated with the Technical Team’s work by signing up to IATI’s Newsletter using the form below or by engaging with IATI’s Technical Community of Practice on IATI Connect.