IATI technical team: our detailed quarterly update

Welcome to the IATI technical team’s second quarterly update. Based at Development Initiatives, it is our job to maintain and improve IATI’s core technical services that support the development and use of the IATI Standard.

Every quarter, we update our community on the progress made on IATI tech and share our plans for the next quarter. As explained in my first blog, the technical team’s quarterly plans are approved by IATI's Governing Board technical focal points.

Update on last quarter...

As you’ll see, our team has been kept very busy delivering the last quarter’s ambitious plan.

Launching french version of IATI website

We successfully launched the French version of the iatistandard.org site in February. Now users can access a wider range of IATI information in French such as news, events, and guidance on using and publishing data. The technical team is responsible for updating the French version of iatistandard.org at least fortnightly to match changes and additions to the English version of the site.

Welcoming IATI’s new Technical Lead

We were delighted to welcome Wendy Thomas, who took over from Bill Anderson in March as IATI’s new Technical Lead.

Wendy joins the team after spending nearly fourteen years at the UK’s Environment Agency, as national lead for metadata, data flow mapping and information services. Wendy now manages the IATI technical team and is responsible for oversight of the IATI Standard. Many thanks to Bill for his hard work, leadership and support of our team.

We also welcomed Francesca who fills our last vacancy for a Senior Developer and our team has been working hard to induct our new colleagues into the complex world of IATI tech!

Updating the SSOT

Throughout the quarter, the team continued our work on updating the SSOT or Single Source of Truth. The SSOT is the collection of rules and descriptions about elements of the IATI Standard. Essentially it is the Standard itself and absolutely central to IATI.

Starting from version 2.01, we have identified all rules contained within the IATI Standard, and ensured that these are captured within the SSOT ruleset. This work has included increasing the number of rules that are considered as ‘machine readable’, which we can check and enforce by code. The same process has happened for version 2.02 and 2.03 of the Standard. Thanks to IATI community member Andy Lulham for helping the IATI tech team with this essential work.

We worked closely with UNOPS to secure a vendor to build IATI’s new validation service to allow publishers, users and developers to check if data published complies with the IATI Standard.

Kicking off the validation project

We worked closely with UNOPS to secure a vendor to build IATI’s new validation service to allow publishers, users and developers to check if data published complies with the IATI Standard. The technical team were part of the review committee that scored the proposals from bidders and we were pleased to announce in March that Data4Development were selected.

The IATI technical team are managing the outsourced development of the new validator and we met Data4Development to kick the project off in March.

Testing the new datastore

During the quarter, we have continued managing the build of IATI’s new datastore with Zimmerman and Zimmerman. The new online data service aims to provide timely and standardised access to all data published to the IATI Standard.

Over the last few weeks, the team has been testing the newly built datastore. We are pleased with the results and will continue to work with Zimmerman and Zimmerman as new features are developed and improved upon.

Updating the Dashboard

We’ve updated the IATI Dashboard methodology to enable data users to understand if publishers are using the “budget-not-provided” attribute in the IATI Standard. This was added to version 2.03 to enable organisations to explain if they cannot provide future budget data for their activity, for example when funding a rapid onset emergency.

The team also met the recommendation from IATI’s Technical Audit to investigate re-building the Dashboard to reduce excessive time that our team spends fixing bugs and maintaining the existing tool. Specifically, we were asked to understand if splitting the Publisher Statistics section away from the Dashboard would make it quicker to rebuild. The team found it would be 70% faster to build locally (on a computer here in the office) and this information will be provided to the IATI Governing Board to help inform decisions on the future of the Dashboard.

Enhancing d-portal

Many improvements were launched on IATI’s online search platform, d-portal. Now, users can easily identify what activities are being delivered in humanitarian crises, the sectors they are being delivered in and the specific type of transactions being made. These new features have been added alongside many other updates, fixes and cosmetic changes throughout the site to meet requests made by d-portal users.

Improving behind-the-scenes tech processes

To ensure the team delivers robust technical services, our developers have continued to dedicate much-needed time to improving our processes behind-the-scenes. Here’s a few examples:

• Refactoring code - getting rid of as much ‘technical debt’ in the coding of our tools as possible. For example re-writing, restructuring or removing outdated and duplicated code which helps reduce the risk of IATI tools breaking.
• Improving deployment processes - automating lengthy and complex manual processes used to implement any changes to IATI’s main website: iatistandard.org. This helps to helps to eliminate the risk of failed deployments or unwanted server downtime, due to server or developer error.
• Consolidating the tech stack - making sure all the components used to build and maintain IATI tools are consistent, for example, ensuring technical languages used across tools are the same.
• Streamlining github repositories - tidying up IATI repositories on the open platform github, which stores the code for each IATI product/project. This helps to ensure our external tech community can more easily understand and contribute suggested changes to IATI tech.

Pulling IATI Annual Report 2018 data

The technical team supported the delivery of the IATI Annual Report 2018 (published last week) by pulling and analysing statistics to assess the changes in data quality over the last year.

Supporting IATI publishers and data users

Over the last quarter, the technical team has continued to provide support to organisations publishing IATI data:

• We received 446 queries via the IATI support helpdesk and resolved on average 168 tickets a month, with a 97% satisfaction rating.
• The team contacted every publisher still using version 1 of the IATI Standard to offer support to move across to version 2.
• We also supported Microsoft’s work developing a common data model for NGOs, to align it with the IATI Standard.

Plans for April - June 2019

Our main focus over the next quarter is ensuring the successful development of IATI’s new datastore and validation services.

Managing the validator build

The IATI technical team will continue managing the outsourced development of the new validator, holding weekly meetings with Data4Development. We will continue collaborating closely to ensure that users receive useful error messages which enable them to ensure their XML files comply with all of IATI’s machine-readable rules. Data4Development plans to complete the project by early July.

This week, some members of the team are meeting both Data4Development and Zimmerman and Zimmerman for face-to-face meetings in the Hague.

Launching new datastore and validator together

These products are going to be launched together. The datastore is on target to be completed by the end of April. However, to ensure it provides data that complies with the IATI Standard, it will connect to the new Validator through an API which is due to be publicly available in July. During this quarter we will focus on completing rigorous testing of both of the new products.

Launching the datastore and validator is going to be a landmark moment for IATI in which two long standing technical needs are going to be addressed.

Launching the datastore and validator is going to be a landmark moment for IATI in which two long standing technical needs are going to be addressed. We are really excited about this and will be working closely with Zimmerman & Zimmerman and Data4Development to make sure the launch goes smoothly and the needs of our community are met.

Enhancing iatistandard.org

The team will begin working on the next phase of the IATI website project in close consultation with UNDP in relation to communications aspects. Working with a consultant, we will be designing a better user experience for accessing all content contained on reference.iatistandard.org and move it to our main site iatistandard.org. This phase will also include writing more understandable descriptions for all elements in the IATI Standard.

There will be an opportunity for our community to get involved in this phase, so look out for information about user testing later in the quarter.

Preparing to deprecate IATI Standard version 1

Version 1 of the IATI Standard is being deprecated on 30 June 2019 and our team will be busy encouraging and supporting all organisations to move to using version 2. We will also need to properly archive all the pages relating to version 1.[3,4,5] of the Standard. Version 1 content will still be accessible, but will no longer be subject to any updates (outside of any essential security) as of 1 July 2019.

Updating IATI tech to Python 3

The technical team uses the programming language, Python, across a number of IATI tools and products. As of 1 January 2020, support for Python version 2.x will officially cease. Our developers will update the IATI technical stack by upgrading to Python 3.x, commencing work now to ensure we meet the deadline.

Continuing d-portal improvements

During this quarter, d-portal’s external developers will be tidying up the tool behind-the-scenes to keep it running well. Now that d-portal is displaying data published on more elements of the IATI Standard, some generic pages will be redesigned so that it remains nice and easy to read.

Implementing French codelists of the IATI Standard

The codelists for the IATI Standard are currently available in English only and this month we will be working on providing translations of the codelists in French. Users will be able to find these in the XML codelists as well as from the CSV download option on our website. Previously only a few codelists had been translated and could be downloaded. However, last year Canada - Global Affairs Canada / Affaires mondiales kindly translated the majority of the codelists, which we will implement into the IATI Standard.

Fixing the Registry

As many users of the IATI Registry will know, there have been recurring problems with the Registry’s metadata not updating, or overwriting itself. IATI’s metadata provides important data about an organisation’s data. For example what data they first published, the date they last updated their data and the number of activities within their files.

It was suggested during IATI’s Technical Audit that we explore the option of managing the metadata provided in the IATI Registry ourselves. This would involve bringing in-house the scripts that populate the IATI specific metadata fields in the Registry from our external provider. As managing this metadata involves an in-depth understanding of IATI, it makes sense for it to sit within the IATI technical team. We will start this work in the later half of the quarter.

We hope you find this update informative and look forward to sharing our progress made in our next IATI Tech quarterly update.