IATI Technical Update January 2023

This post has been written by Wendy Thomas, IATI Technical Lead based at Development Initiatives.

Welcome to the IATI Technical Team update blog, outlining progress made on IATI tech and our plans for the next quarter. The Technical Team’s work plan is regularly reviewed by IATI’s Governing Board technical focal points. You can read our previous update here.

As mentioned in our last update, the IATI Governing Board has agreed to a revised Secretariat work plan, placing some activities on hold during the hosting transition period (view revised and transition workplans here, and planned changes to hosting arrangements here.) This update reflects the Technical Team’s progress on its revised work plan since September 2022 and includes a look ahead to what is on our six-month transition work plan in 2023.

Quarter 4 2022 update

Datastore

During the last quarter we continued to deliver minor version releases to address bug fixes and make enhancements to the IATI Datastore. Many enhancements are inspired by suggestions from members of the IATI community– please do get in touch with us if you have any suggestions on improving our tools!

Highlights of the work on the Datastore during quarter 4 include the following:

• Releasing v3.3.0 of the Datastore API, with a new Structured JSON response option. The IATI Datastore API now provides the ability to receive the response from the activity collection in a structured JSON format, which maintains the hierarchical context of the original IATI Standard XML. This enables data users to more easily and intuitively use IATI data in a familiar JSON format.
• Releasing v1.9.0 of Datastore Search, with several enhancements. For example, we made error messages more user friendly and have enabled metadata on errors to be sent to our developers for debugging purposes. In addition, for IATI elements and attributes that are conditionally associated with codelists, the dropdown fields have been replaced with combination dropdown and text input fields, allowing users to select values from a default codelist, whilst also enabling free-text input.

In November, we worked with UNDP colleagues during a Data Use Drop-In session to provide a detailed hands-on explanation of how to use the features of the Datastore Search. An overview of the session and a recording of the demonstration is available on IATI Connect.

During the last quarter, we compiled usage analytics for the 6 month period since the launch of Datastore Search. These show increases in organic discovery of IATI data and use of the Datastore, as compared to previous time periods. The presentation is available here.

Looking forward, in early 2023 we are preparing for a major version upgrade of Solr to v9 (Datastore API v4) which includes numerous security, stability, and scalability improvements that will provide a better experience for Datastore users.

Validator

This quarter we delivered a wide range of improvements on IATI Validator. Phase II of Activity Level validation was implemented, which checks for schema errors in IATI data at a more granular level.

This quarter we delivered a wide range of improvements on IATI Validator.

Previously, an entire IATI activity file would be checked– each file contains multiple activities and if just one had critically invalid data, then all the activities would fail validation. Now the Validator assesses each activity individually and only activities with critical errors fail validation.

Accompanying this major improvement in the Validator, we made changes to the Validator front end website, which improves the user interface of the tool. It is now easier to find a publisher by first letter of the publisher’s name, using the alphabetical jump buttons. We have also improved viewing and sorting for publishers who have many files and we have improved filtering and viewing for Validation reports, making it easier to find specific activities or errors.

You can read more about these improvements here.

Publishing Tool

Last quarter, the final phase of work on the IATI Publishing Tool progressed on schedule, with completion of internal and community testing of the tool. After soliciting testers via IATI Connect, we were able to work with 15 publisher organisations who provided input on usability of the new publishing tool (as well as comments on their understanding of the IATI Standard itself) and helped us identify any bugs before launch. We are grateful to the community members who provided valuable feedback which has been incorporated into the final release before launch. At the 4th Virtual Community Exchange in November, the Technical Team gave a ‘sneak peek’ presentation on the new IATI Publishing Tool; you can view that session here.

Last week we were pleased to deliver a soft launch of IATI Publisher: publisher.iatistandard.org (17 January 2023). Prior to the launch we identified a number of new publishers and lined them up to be initial users of the tool. We are delighted that within a few days of the launch, our Technical Team supported seven organisations to successfully register and activities have already been published via the new tool.

Looking ahead, we will work with Aidstream’s provider, Young Innovations as well as other tool providers, on the options to migrate existing publishers’ data (where requested) onto IATI Publisher. Read important information about IATI Publisher on our launch newpost.

Registry

The IATI Registry now shows the Validator Status of activities. This allows publishers to be aware of their datasets containing errors that need to be fixed. We also implemented a new feature to allow users to search for pending publishers, making it easier to search for a publisher that has yet to be approved. We have also enabled a feature to give publishers a validation error if the URL to their dataset does not point to an XML file (so they are not able to publish a dataset that is not in XML).

d-portal

After Phase II of Activity Level validation was implemented, we worked with d-portal’s developers from Wetgenes to make sure that links on the d-portal activity pages take users to the Validator to view the validation results of that activity.

Helpdesk and Publisher Support

Between October and December we solved over 200 Helpdesk tickets, with welcome support from UNDP on data use queries. Satisfaction ratings have returned to their usual high levels; with 100% satisfaction from those providing feedback last quarter.

We held in-depth publisher support calls last quarter with many organisations, including the ICRC, UNEP, ILO and World Bank. We also contacted all ‘un-approved publishers’, i.e. organisations who started the publishing process but did not go on to publish their data. This re-engagement has resulted in several new organisations beginning to publish.

Forthcoming work

Our focus for 2023 will be business-as-usual activities to support publishers and improve data quality, as well as maintaining and improving our tools.

The Technical Team, as well as Secretariat colleagues, will be working to prepare for transition to new hosts later in 2023. We have already documented what is required to transition our technical estate; we are now working on documenting the processes to run the IATI Helpdesk, provide publisher support and maintain the IATI Standard.

Stay updated

Stay updated with the Technical Team’s work by signing up to IATI’s Newsletter using the form below or by engaging with IATI’s Technical Community of Practice on IATI Connect.