How we achieve multi-provider DNS redundency

How Hale Stack Built Highly Resilient DNS with Multi‑Provider Syncing

At Hale Stack, we believe that critical infrastructure, especially DNS, cannot depend on a single point of failure. Over the past year, we developed an automation platform that keeps DNS records in sync across up to three DNS providers. We list all of those providers in the registrar’s NS (nameserver) configuration. This setup provides our clients with both redundancy and failover protection.

Here’s how we achieved this and why we think it makes a significant difference.

1. The Challenge: DNS Risk & Provider Dependence

Many organizations rely on a single DNS provider (e.g., Cloudflare, AWS Route 53). If that provider experiences an outage or degradation, DNS resolution can fail. This can potentially take your entire website or service offline.

Manual syncing between DNS providers is error-prone, time-consuming, and does not scale effectively. Registrar-level changes, such as changing nameservers, are rarely automated. Making live changes can also be risky.

We identified this risk as a major weakness, especially for clients whose uptime and reliability are non-negotiable.

2. Our Solution: Management API + Multi‑Provider Sync

Here’s a breakdown of how we built our automated, resilient DNS platform:

• Central DNS Record Source of Truth: We maintain a central repository of DNS records (A, CNAME, MX, TXT, etc.) in our own versioned configuration store. This becomes the “master” of truth. We are also considering future redundancy upgrades in this area.

  
  

• Automated Management API Integration: For each DNS provider we support (up to three per domain), we use their management APIs to programmatically read and write DNS records. Whenever a record changes in our central store, we push updates via the API to all configured DNS providers.

  
  

• Bidirectional Reconciliation:

- On a regular schedule (or triggered by events), we fetch the current DNS state from each provider via API.

- We compare each provider’s records to our central source.

- If there are drifts or discrepancies (for example, if a record was added manually in one provider), we raise alerts.

• Registrar NS Configuration: At the registrar level, we list all of the DNS providers’ nameservers in the NS records. This means the domain is delegating to multiple DNS providers in parallel. If one or two providers are unreachable, DNS queries still succeed via the others.

  
  

• Monitoring & Health Checks: We continuously monitor DNS resolution health: query times, TTLs, record consistency, propagation success, and more. If a provider becomes unhealthy (e.g., consistently slow, failing API, or serving stale data), we trigger alerts and remediation.

  
  

3. How Hale Stack’s DNS Architecture Saved the Day During the Cloudflare Outage

On November 18, 2025, Cloudflare experienced a significant global outage that impacted core parts of its network, including traffic routing, proxies, and service delivery. More info here: The Cloudflare Blog.

Here’s what happened and how Hale Stack’s DNS design helped our clients stay online:

1. Cloudflare’s Root Cause:

2. According to Cloudflare's incident report, a permissions change in one of their ClickHouse database systems caused a “feature file” used for Bot Management to bloat in size.

3. That oversized file propagated across their network, causing proxy components to fail and leading to elevated HTTP 5xx errors. More info:

5. The outage lasted several hours, with core services recovering only after rolling back the problematic configuration and restarting critical proxy services.




6. Impact on Standard DNS Setups:

7. Many customers rely solely on Cloudflare for their DNS. During the outage, some reported being unable to resolve their domains or experiencing high error rates. More info: Krebs on Security.

9. Because DNS is often delegated exclusively to Cloudflare’s nameservers, if their infrastructure degrades, there is no fallback.




10. Why Did Hale Stack Clients Keep Serving DNS?

11. Thanks to our multi-provider DNS strategy:

12. Even though Cloudflare’s DNS/control plane was impacted, the other DNS providers in our clients’ configuration remained available. Because we had set up up to 3 providers in the registrar’s NS records, recursive resolvers could still query the healthy providers.

13. Our continuous reconciliation ensured that the DNS records on the “backup” providers were up-to-date and fully mirrored with no stale or missing entries.

15. As a result, DNS resolution for our clients’ domains remained intact, and their services kept functioning without interruption, or with only minimal short-term performance impact—less than 10 seconds.




16. Rapid Response & Visibility:

17. Our monitoring systems immediately flagged any anomalies in DNS resolution or increased query latency.

18. Our ops team verified via API that Cloudflare’s provider was degraded.

20. Because of our automated sync, we didn’t need to manually reconfigure DNS or perform emergency registrar work; failover happened at the DNS resolver level naturally!

Key Takeaways & Lessons Learned

• Multi-provider DNS matters: Relying on a single DNS provider introduces systemic risk. Having parallel DNS providers dramatically improves resilience.

• APIs make automation possible: By using management APIs from DNS providers, you can programmatically keep records in sync, reducing risk and manual error.

• Registrar-level redundancy is powerful: Listing all DNS providers’ nameservers at the registrar ensures queries can be answered even if one or more providers go down.

• Observability is critical: Continuous monitoring and reconciliation are key. If a provider drifts, you need to catch it early—not during an outage.

• Practice for failure: The Cloudflare outage wasn’t due to an external attack; it was a configuration and scaling mistake. Even trusted providers can fail in unexpected ways. Hale Stack’s architecture treated DNS as a first-class critical service with redundancy.

  
  

Final Word

At Hale Stack, our goal is to make infrastructure invisible—in the best way. Our clients shouldn’t have to think about DNS when they’re focused on their core business. By building a resilient, automated, multi-provider DNS solution, we turned what is often a vulnerability into a strength.

The recent Cloudflare outage was a real-world proof point: our design worked. DNS stayed up, failover happened seamlessly, and our clients stayed online. If you’re interested in how Hale Stack can help you architect resilient DNS (or broader managed infrastructure), feel free to reach out. We’d be happy to walk you through our approach.