I rewrote a national UX standard so a machine could read it

Audit360, the AI audit platform NeGD uses to evaluate government websites against UX4G, GIGW and Lighthouse. Built by four people. The tool was the easy half.

Product Manager and Product Designer · Team of four · Feb 2025 – Mar 2026, in production

The conversation that started it

India runs more than 60,000 government websites. In 2025, none of them had been systematically evaluated for usability or accessibility. Not audited badly — never audited at all.

I was at an office gathering with NeGD's President and CEO when it came up, and he asked me directly what we should do about it.

I gave him arithmetic instead of an opinion. A manual UX audit took my team seven days per site. One auditor working through 60,000 sites at that rate is a little over eleven hundred years. Even the whole team never finishes.

So the answer could not be audit harder. It had to be a system that evaluated government websites against UX4G, GIGW, WCAG and DBIM without a designer reading every page. He approved it in that conversation. The CEO directed that it be built and shipped under UX4G.

The stated goal was making India's digital ecosystem accessible and intuitive for citizens. The real pressure was narrower and more embarrassing: we owned 480 published guidelines with no structure, no versioning, no changelog, no scoring, and no record of how many websites we had ever audited.

What was actually broken

The manual process worked like this. One designer took one platform and spent seven days on it, alongside their actual project work.

Day one went to understanding the product, its audience and its IA. Days two and three were heuristic evaluation plus GIGW and DBIM. Days four to six were the UX4G guidelines, one at a time, by eye. Day seven was writing it up.

The deliverable was a PDF, emailed to a point of contact at the ministry. If they understood it, they acted on it. If they did not, they called us. A re-audit happened six months later, if it happened.

There was no score. Two designers auditing the same site would produce different findings and we had no way to know how different, because we never tested it. There was no tracker either, which is why I cannot tell you how many audits the team completed before Audit360 existed.

An audit programme that could not audit itself. That is the sentence the whole project came out of.

The standard was the product

UX4G had 480 guidelines. A team of four to five designers audited against them by hand, seven days a site. The obvious read is that the bottleneck was speed.

It wasn't. It was that most of the standard could not be evaluated by anything — a machine or a person.

Three guidelines, verbatim:

Your website is hosted on a secure server.

Text on the homepage is concise and to the point to accommodate smaller screens.

The relationship between controls and their corresponding actions is clear and apparent.

The first is testable in a single request. The third is a judgement two designers could reasonably disagree on. Nothing in the document distinguished them. There was no field saying what evidence a rule needed, which page it applied to, what a pass looked like, or who was supposed to fix it. The guidelines lived in a spreadsheet and on a website, written by a designer and a project manager who had both since moved on, with no version history.

So automating the audit was the second problem. The first was that the standard had never been written to be measured.

I went through all 480 rules one at a time and gave the corpus a schema — 22 fields, every one of them answering a question the pipeline would otherwise have to guess.

Then I collapsed the duplicates. Rules written in different tones that asked for the same thing became one rule. 480 became 411.

What the schema bought was not tidiness. It turned a guideline from a sentence into a routable object: this rule needs a full-page screenshot of the footer, on the homepage, judged by a vision model, and a developer is the one who fixes it. Before the applicability fields existed, we ran the same page through the pipeline repeatedly and got different answers, because a rule with no declared page and no declared evidence gives the model nothing to hold onto. It fills the gap with invention.

Machine-evaluable coverage went from roughly 100 of 480 rules to 260 of 411 — about 21% to 63%. Not one line of that came from a better model. It came from rewriting the input.

The machine that reads it

Once a guideline declares its own evidence, the pipeline stops guessing and starts routing.

A rule tagged DOM gets a structural query. A rule tagged Visual gets a full-page screenshot, captured with Puppeteer. Content rules read text, behavioural rules look at state. The screenshot is not decoration — visual hierarchy, contrast in rendered state and whether a control looks like a control are not recoverable from markup. When DOM extraction fails, the screenshot is also the fallback, so a rule degrades instead of disappearing.

Each page is evaluated in a single call: one JSON prompt carrying every guideline applicable to that page, returning a structured verdict per rule.

The part that looks like a compromise, and is actually the design: a human tells us which pages to audit. The user pastes in up to seven URLs — homepage mandatory, then About Us, Contact Us, Sign In and three of their choosing.

We could have crawled. We chose not to, for three reasons that are specific to this ecosystem. Government sites share no path-naming convention, so about us is aboutus, about-us, or something a vendor invented in 2011. Many of them block crawlers outright. And we cannot hit a ministry's servers at volume without permission we do not have. Seven typed URLs buys accuracy that no classifier would have given us, and it keeps us on the right side of an infrastructure question that is not ours to answer.

One constraint I did not control: the model was chosen for us. Our CTO directed OpenAI at project initiation, and no alternatives were evaluated. It works. But I cannot tell you it was the right call, because nobody ran the comparison, and I did not push for one.

Scoring inside a government

The manual process produced no score at all. Introducing one meant deciding what a government website's number is made of, which turned out to be the most political decision in the product.

The split is UX4G 50%, GIGW UI/UX 25%, Lighthouse 25%. My Project Head and I set it. The defensible reason is proportionality: UX4G contributes over 400 evaluated guidelines where GIGW contributes twenty parameters, so equal weighting would have let twenty rules outvote four hundred.

The uncomfortable reason is that UX4G is our own standard, and we weighted it highest in the score we use to grade other people's websites. Nobody inside challenged it. I would expect a ministry CTO to, and the honest answer is that the weighting reflects our team's experience with government products rather than any external validation.

Three decisions I would defend without hedging:

No single site-level number. You get a score per page and a score per audit, never one figure for an entire domain. A site that submits four pages and a site that submits seven have not taken the same test, and one number would have implied they had.

The leaderboard ranks homepages only. Every government website has a homepage, so it is the one page where the denominator is constant and the comparison is fair. Ranking whole sites would have rewarded whoever submitted the least.

Findings say who, not what. Bands read Excellent, Average and Poor rather than Pass and Fail, because the audience for a low score is a department that has to be willing to run the audit again.

And one gap I will not dress up. AI confidence does not discount the score. A finding the model was unsure about moves the number exactly as much as an SSL check that either passes or does not, and the only mitigation shipped is a disclaimer telling users to verify important information. A disclaimer moves risk onto the user instead of handling it. It is the thing in this product I most want to fix.

I also lost an argument here worth recording. I proposed opening the platform beyond government, including a paid tier for businesses that wanted their own UX health checked, which would have funded the compute and widened the sample enormously. It was rejected — a government body cannot generate revenue that way. Access stayed with government officials and invited members. I still think the reach was worth more than the tidiness, but the constraint was real and it was not mine to overrule.

The five-minute wait

An audit takes about five minutes. That single number defines the product's shape more than any screen does.

Five minutes is too long to watch and too short to schedule. So the run has three exits. You can watch it, with per-page progress that reports 2 of 6 webpages audited rather than an indeterminate spinner. You can leave, and hand us an email address so the report arrives when it is ready. Or it fails, and it says so, with a retry that does not start the whole audit over.

The wait screen also teaches. While the pipeline runs, it shows the site being audited and a piece of UX evidence — users form an opinion about a website's credibility in about 50 milliseconds. We are a UX advocacy organisation with a captive government audience for five minutes. Spending that on a progress bar would have been a waste.

The report itself is organised the way a ministry reads it, not the way the pipeline produces it. A site summary, then per-page scores with a non-compliant count, then every finding grouped by category and collapsed by default. Findings filter to Compliant, Not Compliant, or Not Applicable — the third one matters, because a rule that does not apply to your page is not a rule you passed, and conflating those two is how compliance tools lose trust.

Underneath, a comparison against the previous audit, per page. That view is what turns a report into a loop.

Everything that can fail, has a screen. Unreachable URL, API failure mid-run, restricted audits, a first-time account with nothing in it, and a second attempt that resumes rather than restarts. None of these are interesting to design and all of them are the difference between a demo and a product a government department will use unattended.

Auditing my own work

A year in, I ran the analysis on my own corpus that I had built the product to run on other people's websites.

It failed.

The schema I had designed to eliminate inconsistency had drifted. Severity was carrying five values where it should have carried three — 193 rules said Medium Issue, another 62 just said Medium. AI Support had fourteen values, because two different vocabularies had been mixed into one column: the type of evaluator, and, on 94 rules, a completely separate scale of High / Medium / Low / Partial. Evidence Type had thirty-odd values including free text somebody had typed once — UX review, Data logic, Logic review. Three different spellings meant "applies everywhere."

The interesting part was the shape of it. The drift wasn't scattered. It was clustered in 7 of 32 categories — Performance Optimization, Analytics, Accessibility, Offline Functionality and three others, 104 rules in total. Random carelessness spreads evenly. Clustering means provenance.

Those categories were the ones where the metadata had been AI-populated in a separate pass, and never normalised afterward. The guidelines themselves were human-written and fine. The fields the pipeline actually routes on were not.

That mattered because routing is exact-match. A guideline whose page reads All pages does not match a filter looking for Common. It doesn't error. It just quietly isn't there.

Which led to the finding I least wanted:

Page-wise Applicability held 156 distinct values across 411 rules. The product accepts 7.

Half of those page values weren't page types at all — Pre-SSO, Deep hierarchies, Core journeys, Repeating content types. They were contexts, sitting in the page column, while a perfectly good Context column sat unused beside them. Of the 336 rules a machine can judge, 182 are addressed to pages a user has no way to submit, and they fall outside the automated audit entirely. Whole categories go with them — Lists and Filters, Navigation and IA, Forms and Data Entry.

This is a scope boundary rather than a bug: the automated audit evaluates what it can address and reports the count. But the report says X of 411 evaluated without saying which 411 minus X, so a department reading a strong score has no way to know that navigation was never in scope.

I also shipped a Confidence Weight on every rule, a number between 0.55 and 0.98. It looks like rigour. It is a placeholder. It was never calibrated against outcomes, it doesn't gate a finding, and it doesn't move a score. The honest version is that I designed the field before I had a way to earn it, and left it in.

So I fixed what I could fix, and made the rest visible:

• Normalised the corpus. 405 of 412 rows touched. Severity 5 → 3 values. AI Support 14 → 5, with the automation level split into its own column where it belonged. Platform formats unified. Evidence types reduced to 14 canonical tokens.
• Logged every edit. A Normalization Note column preserving the original value of every cell I changed, and every inferred value flagged for review rather than silently accepted.
• Closed the door behind me. Controlled-vocabulary constraints on the four routing fields, so the next person to add a guideline physically cannot invent a sixth severity. The drift happened because nothing stopped it, not because anyone was careless.

The routing gap is not fixed. Fixing it means separating page from context across 411 rules, and — the part that matters more — never letting an unmatched guideline disappear silently. A compliance tool that skips a check without saying so is worse than one that fails loudly.

The lesson wasn't about spreadsheets. I built a system to make a standard measurable, and then let the system's own inputs go unmeasured. The governance was the deliverable, and I had shipped the schema without it.

What happened

5,463 government websites have been audited since v1 shipped in August 2025.

The comparison worth making is against the process it replaced, with the assumption visible. At seven days a site, 5,463 audits is 38,241 audit-days. Those days were elapsed time rather than full-time effort — designers ran audits alongside their real projects — so across the four to five people who did this work, it is somewhere between twenty-one and twenty-six years of team throughput, delivered in twelve months. The comparison is conservative: an Audit360 run covers up to seven pages where a manual audit covered about five.

5,463

government websites audited in twelve months, by a team of four, against a standard that could not be measured at all the year before.

Now the honest part, and it is pointed.

I have no accuracy baseline. I believe the schema cut false positives sharply, because we watched the same page stop producing contradictory verdicts. But I never built a labelled set, never compared the tool's findings against a human audit of the same site, and cannot give you a false-positive rate. Belief is not evidence and I am not going to present it as evidence.

I have no remediation data yet. The product tracks score history per page, so the question did these websites actually get better is answerable from data that already exists. I have not answered it. Adoption proves the tool gets used; it does not prove a single citizen had a better experience.

And the one that stings. Audit360 exists because nobody at NeGD could answer how many websites we had audited. A year and three versions later, the counter exists — but completion rates, drop-off at the URL form, and per-version usage still do not. I did the design work thoroughly and the product instrumentation incompletely, and defining success metrics was my job.

What I'd do differently

Calibrate the confidence score or delete it. Sample findings across confidence levels, check whether low-confidence findings are actually wrong more often, and either let confidence gate what enters a score or stop displaying a number that implies a rigour it does not have.

Split page from context, and add an explicit unknown. The 182 unaddressable guidelines exist because one column carried two concepts. Separating them is mechanical. The design work is the third state: a check that could not run must say so, because a silent skip reads exactly like a pass.

Build the dispute path. There is no way for a ministry to flag a false positive or mark a finding as an intentional deviation. Every enterprise compliance product needs one, and its absence puts the entire score's credibility on our being right every time.

Surface the fields the report throws away. The schema carries Severity, Enforcement Level and Roles for every rule. The report renders none of them, so a foundational blocker and an advanced nice-to-have look identical, and a developer cannot filter to the work that is theirs. The routing metadata exists to make findings actionable and the last mile flattens it back into a checklist.

Solve version comparability. When a guideline's text changes, every score computed under the old wording becomes quietly incomparable. I have no answer for this yet. It is the hardest open problem in the product and it gets harder with every audit we run.

The parts I could not fit — the arguments about weighting, the things that did not work, what a national standard teaches you about designing with AI — I am happy to walk through. Find me on LinkedIn.