How we measure
Webbindex measures the same things on the same websites, using the same method, several times a year. Here we explain which organisations we include, how we measure and what the figures cannot tell us. Results are published in aggregate. No individual organisation is singled out.
Who we measure
Our sample comes from a domain register we have built using open sources:
- Wikidata (websites for municipalities, regions and companies)
- SCB's register of public authorities
- Nasdaq's Large Cap and Mid Cap listings.
- Organisation registration numbers are checked, and each domain is reviewed by a person before it is measured for the first time.
For listed companies, we measure the client-facing website, not the investor website. Organisations that share a website with others or have no domain of their own are not measured. They are reported separately instead. Round 8, in autumn 2026, included 820 organisations, of which 772 had their own website that we could measure.
How we collect data
Data collection runs from Stockholm, so websites see a visitor from Sweden. Our bot openly identifies itself as PigmentWebbindex and never changes its name. It respects robots.txt; if that file cannot be reached, nothing further is fetched. We never try to bypass a block. We visit one website at a time, with at least one second between requests.
For each website, we fetch robots.txt and the homepage both with and without www (to establish which address the website uses), followed by llms.txt. We follow redirects but do not follow links further into the website, and we collect no personal data. All raw data is saved, with a separate row for each attempt.
How to block our bot
We respect robots.txt. Add the following to your robots.txt and we will fetch nothing beyond robots.txt itself. The change takes effect from the next round of measurement. Websites that block us are counted as non-responses and appear only in aggregate figures, never by name.
User-agent: PigmentWebbindexDisallow: /
What we measure
Six checks, each with a version number:
- With or without www: whether the website responds at both addresses or redirects one to the other.
- Presence of llms.txt: whether the file exists and starts with a heading as set out in the proposal.
- llms.txt: a platform default or written for the website: whether the file contains default text from a known platform or plugin. This gives a minimum count, as we only count patterns we have seen.
- robots.txt rules for AI bots: which bots used for training, search and user-initiated queries are allowed to fetch the homepage.
- Content without JavaScript: how many words the homepage HTML contains before any scripts run.
- Structured data on the homepage: whether valid JSON-LD or microdata is present.
Because we save the raw data, we can replace a check with an improved version and recalculate the results without fetching the data again. Previous values are never overwritten.
How we check our own work
Before publishing each round, we fetch 30 randomly selected websites again using an independent tool and compare the results. Round 8 produced 0 discrepancies across 140 verifiable outcomes. This spot check shows that the checks interpret the raw data correctly. It does not show that every website has been correctly selected, which is why we also review the domain register.
Non-response
Each proportion is calculated from the websites that returned a result for that particular check and is reported as "X out of Y". Non-responses fall into three categories: blocked in robots.txt, bot protection or an unexpected response, and no response. We do not weight or scale up the results to compensate. Websites that failed for a temporary reason are tried again after 24 hours. A group with a non-response rate of at least 20 per cent or fewer than 20 responses is not reported as a standalone finding.
Limitations
- We only read the homepage, not the rest of the website.
- The homepage is read without JavaScript. This reflects how most AI bots read a page, but not how a person sees it.
- Websites may show different content to different bots and at different times. We measure what our bot received during data collection.
- The presence of an llms.txt file tells us nothing about whether any AI service reads it.
Rounds
Round 8, Webbindex 2026-11, with data collected on 2 November 2026. More rounds will be added here.