This post has been written by Wendy Thomas, IATI Technical Lead based at Development Initiatives.
this post focuses on the actions that we will take forward from the many recommendations arising from the Stocktake’s workshops
Following this year’s Technical Stocktake, I am pleased to share details about how we will implement the Stocktake recommendations to improve IATI’s technical services. You can read more about the Stocktake here.
Below I present a Roadmap of post-Stocktake projects which form a core part of the IATI Technical Team’s work in 2021 (and beyond). Whilst IATI’s full 2021 workplan is still being finalised; this post focuses on the actions that we will take forward from the many recommendations arising from the Stocktake’s workshops. Our roadmap prioritises work on behind-the-scenes structural changes and building blocks, to enable new work on services for data publishers and users to follow.
To underpin our technical work going forward, we have developed comprehensive Non-functional Requirements (NFRs) which will be applicable to IATI tools and software going forward. Whilst Functional Requirements specify what a technical product or service does to meet user needs, NFRs outline how a service or product is delivered (that is developed, installed, or operated). For example, they could specify the requirement to have User Acceptance Testing (UAT) processes before a product is released. The Stocktake consultant Dan Hughes recommended that IATI devise robust NFRs as a fundamental building block to help us implement an integrated architecture.
What do we need to do?
We are using the Product Quality Model (4.2) of the ISO/IEC 25010:2011 standard as our NFR Framework, which encompasses eight characteristics: functional suitability, performance efficiency, compatibility, usability, reliability, security, maintainability and portability. These will then be developed as required for IATI tools and software.
Who do we need to engage with?
We will share our proposed NFRs with the community via IATI Connect for discussion and feedback (Q1 2021).
Application Programming Interfaces (API’s) are specifications of possible interactions with a software component. We will deliver a new API Gateway to facilitate managing our API’s across platforms, enabling the integrated architecture that the Stocktake recommended and helping us understand service usage. From Stocktake consultant Dan Hughes:
“Initially, current APIs will be publicly published through the API Gateway. Over time additional APIs are added and driven by User Personas and Stories. These microservices are loosely coupled, independent microservices.”
“Initially, current APIs will be publicly published through the API Gateway. Over time additional APIs are added and driven by User Personas and Stories. These microservices are loosely coupled, independent microservices.”
What do we need to do?
We need to map the journeys of our users across IATI services so that we fully understand what is needed for APIs, as well as the different products our users may need the IATI system to interact with. We are starting to do this in Q4 2020. Once the mapping has been completed, there will be a period of engagement with tool providers and API users in Q1 2021. We will then work on and agree the API Gateway’s design, and aim to launch the service later in 2021.
Who do we need to engage with?
We will engage with our users and suppliers, to determine the best methodology for each product.
The Technical Stocktake recommended that we deliver a full options analysis for future IATI data publishing services. As a first step, our Technical Team has engaged an external consultant (ThresholdWorld) to design and deliver a survey to IATI publishers, to gauge what they require from a publishing tool. The survey and UX (user experience) work kicks off this month.
What do we need to do?
In Q1 of 2021, we will analyse the insights from the UX work, draft Terms of Reference and use this as the basis of a detailed options analysis for the Governing Board to consider the best route forward. Technical work will take place once the route forward for a publishing tool has been agreed. Depending on the scale of the work and method of delivery, our aim is to have an IATI-supported publishing tool available by the end of 2021. During this time we will continue to monitor the provision of publishing services by external suppliers, including AidStream. We will also ensure that the Technical Team’s planned work on the IATI Registry API (see below) will complement any future publishing routes.
Who do we need to engage with?
We will engage with a range of publishers throughout the UX work to understand which types of organisations would want to use an IATI-supported publishing tool and what internal systems it would need to integrate with. We will also engage with existing publishing tool suppliers, e.g. AidStream.
What do we need to do?
Following up on the Stocktake recommendation that we should organise tools as decoupled microservices, we will undertake background work during 2021 to decouple the Validator from the Datastore. This will adhere to the microservices architecture as proposed in the Stocktake. By undertaking this essential behind-the-scenes work, this should speed up time taken for users to receive public validation reports.
The Validator and the Datastore will eventually communicate with one another via the API Gateway in the same way that all other products will operate. We will also begin work on creating a public Validator API in 2021 and aim to have this available by the summer.
Who do we need to engage with?
We will engage with Validator users about their requirements for a Validator API.
The Stocktake identified the need for better data access tools for users as a clear area to be addressed, and recommended the development of a new 'semantic data layer.' In simpler terms this is a curated dataset of IATI data that provides non-XML users with a way to access, understand and use the data more easily. Whereas the current d-portal offers this to some extent, it is a tool developed several years ago and now missing all the functionality needed to provide a wider group of users with a better experience using IATI data. In 2021 we will launch a UX project to learn from data users what formats, transformations and visualisations are needed for them to access data most usefully. The findings from the UX project will help us to determine the requirements for d-portal v.2.0, slated for design and specification by the end of 2021 and development and delivery in 2022.
What we need to do?
As mentioned above, to fully understand what data users want from a new data layer / second version of d-portal, we will need to carry out a UX project. We will procure the UX project early in 2021 and we anticipate that the UX work will run from Q2 to Q3. We will then work on drafting designs, defining the scope and finalising the Terms of Reference. By year end, we aim to produce a design and product specification, which will determine the scope of the new data layer / d-portal 2.0 and allow us to begin any necessary procurement. We will continue to host, support and maintain the current version of d-portal throughout this process.
Who do we need to engage with?
We will engage with IATI data users and the wider community throughout this project, starting with getting broad input into the UX work.
What do we need to do?
We will prioritise working with the Datastore’s supplier to address existing performance issues with the product. We do not expect major feature changes in the Datastore in 2021 but will focus our efforts on ensuring the core product is robust and efficient as a core tool that other tools and services can consistently rely on.
Who do we need to engage with?
We will need to engage with the Datastore supplier regarding the delivery of the new API Gateway and the Datastore API.
The Stocktake highlighted a need to improve our publishing statistics methodology and the information that is provided via that methodology. The demand for a new methodology has been raised before, including through the Data Use Working Group. Delivering this project is also in line with IATI’s data quality implementation plan for 2021.
What do we need to do?
In Q2 of 2021, we will begin discussions about a new publishing statistics methodology for IATI. We will review existing user needs for publishing statistics and seek additional feedback as needed. We will then design a new methodology based on user needs, agree what is required in Q3 2021 and deliver new pages by the end of Q4.
Who do we need to engage with?
We will share our proposed new methodology with the community via IATI’s community platform, IATI Connect for discussion and feedback.
Another recommendation from the Stocktake was to develop a policy for hosting publishers’ XML files. Different parts of the current IATI system already host publishers’ XML but do so separately in each tool, and the cached data is not made available if, for example, a publisher’s site goes down.
What do we need to do?
We need to investigate and develop a policy on whether to store a copy of publishers’ XML files and provide an access point to this. We also need to ensure there is a clear set of policies and rules on data storage, retention and data removal, as well as operational processes to do this.
Who do we need to engage with?
Once developed, we will share our proposed policy on hosting publishers’ XML files with the community for discussion and feedback through IATI Connect, early in 2021.
What we need to do?
The Secretariat was asked by the Governing Board to articulate the added value of storing historical data, and based on this, undertake a consultation with the IATI community. This project will also map out the technical implications of storing historical data and devise a proposal for the Board to consider. This piece of work is dependent on agreeing the policy to store publishers’ XML (above), so it will not commence until later in 2021/early in 2022.
Who do we need to consult/engage with?
A full consultation with the IATI community, (publishers and data users) will be required.
I look forward to working with the IATI community to deliver these important post-Stocktake actions over the next year.
A small but important piece of work is already underway to improve the publishing and metadata API documentation for the Registry on our website. This will make it possible for developers to work with the Registry’s APIs. We are working with the supplier to implement this in Q4 2020.
I look forward to working with the IATI community to deliver these important post-Stocktake actions over the next year. For more information, please see the IATI Technical Stocktake Roadmap 2020/21.