The Rise and Fall of My Home Server

This is an amazing story and a personal experience that I can't wait to share, and it's all about Computer Science!

Amazing Life.

Before I dive into it, let me share some things I've been working on lately. If you've read my previous blog, it was about creating a Cloudflare tunnel and how to route traffic from the domain to the server we want.

I learned a lot while working on it, and a major challenge was creating a new DNS route for every self-hosted application (like Audiobookshelf, Jellyfin, Nextcloud, etc.). I was doing it manually because I didn't want to break anything. Then one day, the idea of using a reverse proxy manager came to me. I thought, "What if I point all the traffic to my server and let the reverse proxy manager handle the distribution?" So, I started looking for tools that could do this.

The application that i choose was Nginx Proxy Manager, or more generally NPM. then i installed it on my server and started to setup.

A simple understanding: While doing this, I was also figuring out how to set up TLS certificates on the system so that all my traffic would be encrypted (honestly, I was annoyed by the browser warnings). After creating the certificates and using them in the proxy manager, I realized that all the traffic routed through the Cloudflare tunnel is already encrypted by HTTPS by default. In hindsight, I shouldn't have spent so much time on this, but at least I learned how to create and use certificates.

Once I installed NPM, I encountered another major issue and realized I should have addressed it earlier. All the applications running on my server (like Audiobookshelf, Jellyfin, Nextcloud, etc.) were set up as separate Docker containers, each with its own Docker network. Anyone familiar with NPM knows that you need to provide a URL for a specific endpoint. For example, audiobooks.12129090.xyz (this is my actual URL) should point to where the application is running. Normally, if the application is on the host environment, you could simply direct audiobooks.12129090.xyz to localhost:13378. However, since it's running in a Docker container, the process is a bit different. This was another important realization for me.

If I had placed all my Docker applications on the same Docker network, I could have accessed them using something like docker.internal.<application_name>. However, since I didn't think of this earlier, I needed to find another solution. Fortunately, I knew about the Docker interface and its IP address. I learned that a common solution is to use 172.17.0.1 to access all the applications running on Docker. (I would have shared a snapshot of the NPM proxy hosts page, but unfortunately, I no longer have the server, and I'll explain why.)

This whole setup cost me at least 100 strands of hair, a week of time, and a lot of coffee. But in the end, I managed to get everything working. Finally, all the traffic was routed to the NPM using a wildcard entry, and the reverse proxy handled everything.

What was the biggest complication that I couldn't figure out quickly?

Feeling confident that everything was set up correctly, I decided to create a Kubernetes cluster using my server, so I installed k3s on both servers. Then, suddenly, the proxy hosts stopped working completely. All the routes showed a 404 not found error when I tried to access them, and I couldn't figure out why. I accessed the applications using the Tailscale IP (where traffic doesn't go through the NPM), and they worked fine. This made me realize there was an issue with the NPM. I started examining its configuration and adjusting various settings. Just when I was about to give up, I discovered that the k3s installation had also installed Traefik, which took over as the reverse proxy manager. Since Traefik didn't have the route configurations, it showed 404 errors. I uninstalled Traefik, reconfigured the NPM, and everything returned to normal after a rollercoaster of emotions. Who in the name of computer science, as a beginner would have thought that Traefik will be installed on side with k3s

The main reason I wanted to install k3s was to play with it. Nothing else, just for the pure joy of playing with servers, trying to break them and rebuild them. Unfortunately, setting it up broke other configurations and taught me a lot of things (I guess this is why people tell you to learn things practically). Finally, I was able to set up the Kubernetes cluster and made my friend’s server an agent to mine, and everything was set up beautifully. I was sharing all the applications with my friends and family, and they were using them without thinking about how much was happening behind the scenes. Setting up the server was just one part; getting the content for the applications, like audiobooks, movies, and TV shows, is another big story. It involves some interesting methods of finding sources, seeding, and peering (if you know what I mean).

So what happened now?

I couldn't share any snapshots of the configurations or applications, and there's a reason for that: I LOST ACCESS TO MY SERVER after doing all this work. It was very disappointing. What happened was, while trying to install a self-hosted music application, the Docker Compose file provided by the developer didn't work. Greedily, I used a Docker file modified by ChatGPT, and it instantly closed port 22. I could still ping the server and trace its route, but port 22 was closed, and I didn't have physical access to the server.

After days of troubleshooting and trying to get it back online, I finally gave up hope. I asked someone I know to turn off the server. And that's it, here I am writing all of this.

Conclusion

What do you expect me to say? It was a hell of a beautiful experience that I had in the process of trying to get a desired state, breaking it, retrying, and doing all kinds of stuff, installing applications, using them, looking for more of them, trying to install k3s, load balancers (bare metal), but in the end, I lost access to the server itself. But it was a must-have experience, and if possible, everyone should experience it someday. It will be pure bliss when the configuration and applications are helping you and heartbreaking when the same thing suddenly stops working, and pure fun while trying to debug and learn while doing it. I definitely had fun while doing it, and I hope you will too :)