COE = Correction of Error

My previous employer, Amazon, was a big proponent of doing blameless analysis of outages and figuring out what could be done to fix it. I recently had an outage on my servers and wanted to share what went wrong and the fix.

Summary

Starting Thursday until Friday, all TLS requests to a *.technowizardry.net domain would have failed due to a TLS certificate expiration error. Then on Friday, all DNS queries to a *.technowizardry.net zone failed which also caused mail delivery to fail too. This happened because cert-manager had created the acme-challenge TXT record, but the record was not visible to the Internet because the HE DNS was failing to perform an AXFR Zone Transfer from my authoritative DNS server. This was because PowerDNS was unable to bind to port :53 because systemd-resolved was already listening on that port.

Ever wondered how well your HVAC system is working in your home or condo? I did to an unhealthy degree. I want to know not just what’s the temperature, but how often is it running, what’s the supply and return temperatures, etc.? Let’s overengineer another project.

I’ve had enough of Helm. I don’t know who thought string-based templating engines would be a good idea, but I have had one too many indention relate bugs. They’re a source of bug and a pain. Kubernetes YAML files just contain a ton of boiler-plate YAML configuration. Like how many times do I have to specify the labels? Its spec/template/spec for Deployment, but spec/jobTemplate/spec for CronJob. Ain’t nobody got time to remember that.

Enter cdk8s. It’s built-upon CDK, a software development kit that uses standard programming languages, like TypeScript, Python, or Java, as a way to define resources that then get compiled into YAML or JSON to upload to CloudFormation, or in our case, Kubernetes.

Why would you want/need a full programming language just to define some infrastructure? Well, there are some benefits. Let’s go through them.

This blog is a static website compiled using Hugo. Up to this point, I built the website and packaged all of the assets into a Docker container with NGINX which was hosted on my dedicated server cluster.

This worked well and was simple, but I have an upcoming project that I’ll be announcing soon that required dynamic content that nginx + pure static files wasn’t easily able to implement with NGINX.

To fix this, I decided to migrate this blog from NGINX to ASP.NET Core. Here’s how and why.

I posted a link to my blog on Slack and was greeted with HTML entities right in the website summary. I could see certain characters like the apostrophe ’ being encoded as ’.

Here’s how I fixed this problem.