Ensuring your files are published without access restrictions

Problem

Some organisations host IATI data on websites protected by security tools like Cloudflare, which guard against malicious bots but often block legitimate users from accessing the data too.

This can be misleading: the data appears available — visible in a browser on the publisher's website — but tools such as the IATI Dashboard, Validator, and Datastore are unable to retrieve it.

The solution

The only solution is to remove bot-blocking restrictions from any published IATI files, making them openly accessible. This doesn't mean removing security entirely — tools like Cloudflare allow restrictions to be lifted for specific areas of a site, and files could alternatively be moved to a subdomain with no restrictions.

This approach is both practical and consistent with IATI's principles: it saves time for publishers and IATI staff, reduces errors, and ensures data is available to anyone without barriers.

Blocking genuinely malicious bots — such as those hammering a file thousands of times per second — remains acceptable. The key is that well-behaved bots must not be blocked by default.

Ways to test that your files are accessible

1. Test outside your own network

If your IATI files are hosted on your employer's network, test access from a separate network — not your office connection, and not a home or co-working connection routed through your employer's VPN. Internal and external users are often subject to different security rules, and you need to confirm the files are publicly accessible.

2. Test downloading the file

Open a terminal (Mac/Linux) or Command Prompt (Windows — search from the Start menu). If prompted for permission to make changes, you can safely proceed: the command makes no changes to your system.

You'll need the full URL of the file you want to test.

Run the code below, replacing the example URL with your own:

curl --user-agent "IATI download test" https://www.example.com/iatidata.xml

You should see XML containing some IATI data. If so, the test has passed.

3. Test access over time

The tests above only check access once. For a reliable result, repeat the test over time — some websites allow an initial request but block subsequent ones.

Read more about making your files accessible while using Cloudflare.

Allow-listing won’t work unfortunately

Many security products support allow listing — whitelisting specific trusted bots while blocking all others. This requires organisations to configure exceptions based on each bot's user agent and IP address, and we're often asked to provide this information for our own tools.

However, allow listing is not a workable solution for several reasons.

It's operationally burdensome. IATI's ecosystem includes many products, each with multiple technical identifiers that can change at any time. Keeping this information accurate places an ongoing maintenance burden on both IATI and the reporting organisation's staff.

More fundamentally, it's incompatible with open data. Anyone can build tools that consume IATI data and we cannot provide allow list details for tools we don't control. Organisations who only permit listed bots will inadvertently block third-party products, creating gaps in IATI data coverage and undermining the analysis that depends on it.