Hybrid to Cloud
An on-prem hospital platform with cloud failover, built entirely as code.
What happens to a hospital's lab results when its only site goes dark?
Why I'm building this
I've worked 24/7 hospital labs for eight years. When systems go down, results stop moving and clinicians wait. Most hospitals can't move patient data fully to the cloud, so they need on-site systems that fail over safely. I'm building that end to end, in public.
The problem
In February 2021, Winter Storm Uri knocked out power across Texas for days. Hospitals lost power and water pressure, and some moved patients elsewhere. The state counted 246 deaths.
Hospitals can't move patient data fully to the cloud, because of data residency and on-site clinical systems. So when the building fails, the systems inside it fail too. This platform keeps lab quality control running in Azure while the site is down, and brings it home when the site recovers.
Sources: Texas Tribune, Feb 18, 2021; Texas DSHS final death toll of 246 (TPR)
The approach
- Foundation
- An on-prem Proxmox cluster built as code with Ansible, Packer golden images and Terraform. It can be rebuilt from scratch, and the cluster grows by adding a node with code.
- Azure first
- A governed landing zone, a VPN, and Active Directory and DNS on Windows Server. QC Intelligence runs on Kubernetes on-prem and fails over to AKS.
- AWS next
- An Ordering Guardrail app, with region-to-region recovery run by Step Functions and Lambda.
- AI in operations
- Advisory only, never acting on its own. It reviews infrastructure changes, triages incidents, drafts postmortems and explains costs. Every output is logged, capped on spend and checked against a test set.
The apps
QC IntelligenceOn-prem and Azure
Reads lab quality-control results as HL7 messages, applies Westgard rules from configurable rule packs, and charts them as Levey-Jennings charts in Grafana. An AI advisor links failing patterns to maintenance alarms and reagent lots. QC data only, never patient data.
Ordering GuardrailAWS
Catches duplicate and unnecessary lab orders before they're placed: a CMP added on to a BMP already run (it suggests a liver panel add-on instead), or an HbA1c repeated within 3 months. Providers can override with a documented reason, and every override is logged. Synthetic orders only, never patient data.
Targets
- QC back online in the cloud in under 1 hour (RTO)
- No more than 15 minutes of data lost (RPO)
- No patient data in scope; QC records kept tamper-evident for CLIA retention.
- Failover automated, with a human approval step
Progress
1 of 7 stages done
- DoneStage 0 — The on-prem foundation as code
- Up nextZ1 — The Azure landing zone (budgets, tagging, Azure Policy, Entra roles, CI pipelines that sign in without stored keys, a third cluster node added by code, the AI plan reviewer)
- PlannedZ2 — Hybrid networking (VPN, Windows Server Active Directory, DNS and DHCP)
- PlannedZ3 — Kubernetes and QC Intelligence (Argo CD, Prometheus and Grafana, AI incident triage)
- PlannedZ4 — QC on Azure (AKS, PostgreSQL, Azure Arc, Azure Monitor, a manual failover drill)
- PlannedZ5 — Automated failover (health probe, alert, pipeline, approval, AKS)
- PlannedAWS — Landing zone, the Ordering Guardrail, region-to-region recovery with Step Functions and Lambda
Already built and verified
- Proxmox cluster created and joined by Ansible. Every playbook run again reports zero changes.
- Least-privilege automation: Terraform and Packer each get their own scoped API token, which expires after 90 days and was rotated by playbook.
- Secrets encrypted in the repo with SOPS and age, decrypted only into memory, with a paper recovery key proven in a scripted drill.
- Secret scanning (gitleaks) blocks a commit before it lands.
- Golden Ubuntu 24.04 image built by Packer from a signature-verified cloud image.
- A Terraform smoke test caught a real defect, a leftover build SSH key in the image. It was fixed, rebuilt and verified again, with the evidence recorded.
- An Architecture Decision Record for every major choice.
Next milestone
Z1: an Azure landing zone with budgets, Azure Policy guardrails, Entra role-based access, and GitLab pipelines that sign in to Azure and AWS without stored keys.