Improving IATI data quality with feedback loops

Catalpa International has undertaken research on how to strengthen feedback mechanisms to improve the quality of IATI data. Funded by the IATI Data Use Fund, Catalpa has developed key recommendations for both publishers and the IATI Secretariat. Over the next year, IATI’s Secretariat and Data Use Working Group are committed to implementing these recommendations and support publishers to also take action to drive up the standard of data quality.

Why strengthen feedback?

To date, over 1200 organisations have published data in IATI on over one million development and humanitarian projects.Some data users have cited that issues with the quality of data have prevented them from using it in their work. This is something already identified in IATI’s Strategic Plan 2020-2025 (ES / FR), which states that “good quality data is a prerequisite for data use, while feedback from data users helps to drive improvements in data quality”.

What did the research find?

Catalpa conducted a desk review and interviewed representatives from a range of organisations that have published and used IATI data including governments, multilaterals and civil society organisations, along with technical experts. This research revealed different barriers and drivers that influence data quality, but primarily focused on the role of feedback processes.

The findings confirmed that feedback was a major factor in influencing publishing organisations to improve the quality of their IATI data. However, most publishers interviewed said they received very little feedback on their data and Catalpa identified the following challenges inherent in the process of providing feedback:

1. Feedback requires effort: Providing feedback requires dedicated time and effort from busy data users.
2. Lack of official feedback channels: Several partner countries said they were not aware of official channels for providing feedback.
3. Trust in feedback process: Data users said they were not confident in getting an effective response if they provided feedback. One reason was because publishers provide generic email addresses, and users have doubts that feedback will reach the right person.
4. Reputational risk to those who provide critical feedback: For some, providing feedback through public channels such as GitHub carries with it a personal reputational risk because their feedback may cause the perception of a lack of knowledge or a lack of understanding. Other data users were reluctant to do so, fearing that their comments may be interpreted as the official position of their organisation.

Read IATI Research on Feedback Mechanisms to see all the findings on the ways in which different drivers, barriers and incentives affect data quality, and the impact of current feedback processes on data quality.

7 Recommendations to improve feedback

Catalpa developed seven recommendations for publishers and the IATI Secretariat. The Secretariat and the Data Use Working Group have set out clear actions for implementation (view plans on presentation below). As the core finding of the research is that the burden for identifying data quality issues should not be placed on the end user, the recommendations seek to ensure that feedback mechanisms are implemented earlier in the publishing process to enable major data quality issues to be resolved before data reaches external users.

Recommendations to IATI Publishers

Recommendation 1: Enhance feedback at the point of data entry: Individual agencies should include validation in data entry forms so that data quality issues are detected at the data entry stage. Those entering data should be encouraged to preview their data through simple data visualisations. The report notes that those who know the projects best are often best placed to identify issues.

Recommendation 2: Enhance feedback from publishing team to those entering data: Publishing teams should regularly communicate the relevance of good data quality and data quality indicators to all colleagues. This will help those responsible for entering data to track progress on the quality of data for their projects.

Recommendation 3: Establish feedback loops between publishing teams and local offices in partner countries: Publishers should ensure that feedback loops exist between their headquarters and offices in selected partner countries. They should also expand engagement with their counterparts in country offices to investigate and resolve data discrepancies.

Recommendation 4: Continue to invite feedback from internal users: Publishers should continue to engage with colleagues within their agencies to promote the use of IATI data - both their own data and that of others - and invite feedback on data quality.

To support the implementation of recommendations for publishers, the IATI Secretariat will support discussions on best practice for improving internal feedback mechanisms with publishers through IATI’s new Communities of Practice. The IATI Secretariat will also directly engage with publishers to use the IATI Validator (launched in October 2020), supporting them with improving their data quality and using the Validator in different ways that would help the publishers to improve their own internal feedback mechanisms. For recommendation 3, IATI’s Data Use Working Group in coordination with the Secretariat will launch a project next year to assess potential ways to improve the feedback loops between publishing teams and local offices.

Recommendations to IATI Secretariat

Recommendation 5: Enhance automated feedback to all publishers: Publishers and data users should make better use of existing tools to identify IATI data errors and these tools should be improved. In addition, the report recommends that the assessment of data quality should be expanded and better communicated.

The IATI Secretariat will help implement Recommendation 5 by exploring the possibility for automatic feedback on data quality issues to publishers. The Secretariat will also consider the report’s suggestions as part of their consultation next year to create a new methodology for measuring data quality.

Recommendation 6: Establish targeted feedback to priority publishers The IATI Secretariat should establish criteria for which publishers should be engaged with as a strategic priority. This should identify a list of priority publishers, and establish a systematic process for addressing major data quality issues with these publishers.

IATI’s Secretariat has already begun to engage a list of target publishers to improve their data quality by using the IATI Validator. After learning from this round of engagement, the Secretariat plans to establish systematic regular monitoring and engagement with publishers on data quality.

Recommendation 7: Request and coordinate feedback from external data users: There should be a feedback pop-up dialogue on d-portal and the IATI Datastore, with data feeding into a central feedback tool. The feedback would be sent to a specific publisher and copied to the Secretariat to ensure it has an overview of user feedback.

Next year the IATI Secretariat and Data Use Working Group plan to explore the possibility of building a central feedback tool in line with IATI’s Technical Stocktake implementation roadmap.

Next steps

The IATI Secretariat will work with publishers and data users to implement Catalpa’s recommendations and the Data Use Working Group will play a key role in these efforts. For more information, read the report on IATI Feedback Processes and the presentation on how the recommendations will be implemented.