An AI Found a Clinically Dangerous Bug in the Code Another AI Helped Write
What building a lab interpreter taught me about review discipline, honest software, and why healthcare AI needs guardrails instead of blind trust.
A blood glucose of 54 mg/dL should make any clinical system stop and pay attention.
Instead, one version of my code labeled it HIGH.
The bug came down to something deceptively simple: the laboratory result was reported in mg/dL, while the reference range was expressed in mmol/L. The software compared numbers that looked compatible but represented completely different units. The result was an inverted interpretation that could have surfaced to a user if it had escaped review.
It was caught because the development process assumed something important:
Every healthcare AI system will eventually be wrong. The question is whether your engineering process catches it before your users do.
That experience shaped the latest release of HealthClaw Guardrails more than any feature I added.
The problem isn’t reading FHIR
The healthcare industry has spent years making clinical data more accessible through FHIR.
Now AI agents can retrieve laboratory results, medications, diagnoses, and encounters almost instantly. Every major model can summarize records. Every framework promises intelligent healthcare assistants.
But very few conversations focus on what sits between the AI and the patient’s clinical data.
Why build a lab interpreter?
Most patients eventually receive lab results through a portal.
Many of those results arrive before anyone has explained what they mean.
Patients often search the internet, ask AI assistants, or wait anxiously for a phone call from their clinician.
Just an honest explanation written in plain language.
HealthClaw 1.6.0 introduces Observation/$interpret, a FHIR operation that classifies laboratory results using standard HL7 interpretation codes while generating a consumer-friendly explanation.
Four principles guided the implementation
The first principle was to optimize for understanding rather than labels. A person does not benefit from seeing “L” or “H” beside a laboratory value if they still have no idea what that means.
The second was that accuracy determines inclusion. If there was uncertainty around a reference range, the software should decline to interpret rather than guess.
Third, every fallback reference range must have documented provenance. If a source could not be cited, it could not become part of the system. The automated test suite enforces that rule.
Finally, every meaningful feature receives adversarial review.
That last step changed everything.
Breaking your own software on purpose
After implementing the lab interpreter and building the test suite, I started another review. Instead of asking whether the feature worked, I asked how it could fail in ways that looked reasonable.
That review uncovered the glucose bug.
The implementation correctly preferred the laboratory’s own reference range whenever one was available.
Today, the interpreter only trusts a laboratory reference range when the units match the observation. If they do not, the system either uses a validated fallback range with documented provenance or returns an indeterminate result.
Honest software earns trust
One design choice that continues to matter to me is labeling features honestly.
This release also includes support for the NQF 0018 blood pressure control measure through Measure/$evaluate-measure.
It produces a standards-compliant FHIR
MeasureReport.It is also clearly documented as a calculator.
Not a certified electronic clinical quality measure.
That distinction matters.
Healthcare has enough ambiguity without software marketing adding more.
The same philosophy applies throughout the project.
Decision support is not diagnosis.
Interpretation is not medical advice.
A demonstration of a standard is not certification.
Clear boundaries build trust.
Safety is a system property
There is constant discussion about which foundation model performs best. That is an interesting research question. It is not the question I spend most of my time thinking about. I care more about what happens regardless of which model someone chooses.
HealthClaw now supports OpenAI, Gemini, and LangChain through a lightweight adapter layer.
The models can change.
The guardrails do not.
PHI redaction.
Audit logging.
Tenant isolation.
Step-up authentication.
FHIR validation.
Those protections remain on the server where they cannot be bypassed by changing prompts or swapping frameworks.
Healthcare AI should not depend on every application developer remembering to implement the same safety controls.
Those controls belong in infrastructure.
Building in public
HealthClaw Guardrails is released under the MIT License because I believe healthcare infrastructure benefits from public scrutiny.
Clinical software needs engineers.
It also needs clinicians.
The laboratory reference tables are intentionally documented as requiring ongoing clinical review.
That is not unfinished work.
That is an acknowledgement that software should be honest about where human expertise belongs.
If you are a clinician, I would value your review of the interpretation logic and reference ranges.
If you are an engineer, there are plenty of opportunities to contribute. There are several good first issue tickets waiting for anyone interested in healthcare interoperability, FHIR, or AI safety.
The repository is open because I expect people to find things I missed.
That is the point.
Healthcare AI will not become trustworthy because models get larger.
It will become trustworthy because we build systems that assume mistakes will happen and make those mistakes difficult to reach patients.
That is the engineering discipline I hope HealthClaw demonstrates.
If that mission resonates with you, I’d appreciate a star on the repository, a watch to follow future releases, or a subscription here on Substack. I’ll continue documenting what works, what breaks, and what I’m learning as we build AI infrastructure that deserves to be trusted.




Useful because the bug is not just that one AI wrote bad code and another caught it. The workflow lesson is that review has to preserve the reason for the flag, the evidence behind it, and who owns the fix before the code touches a clinical path. In OR-adjacent agents, I would want the same discipline: no silent upgrade from generated output to operational truth.