27 December 2024
I’ve had a lot of fun writing the Homelab Chronicles, but Chapter 5 will be the final entry in 2024. This chapter catches us up to the current situation. This journey has been exciting and evolved a lot. In Chapter one, I covered my first homelab configuration, why I wanted to go down this rabbit hole further, upgrading my network through my office renovation project, and learning to love Docker. In Chapter two, I set up a reverse proxy for saner access to my services at home and away from home, changed how I managed my DNS, and fell in love with Docker Compose. Chapter three documented my growing application list, developing a backup strategy for the homelab, and improving security on the reverse proxy. Chapter four was a long entry dedicated entirely to my media streaming application, Jellyfin. This chapter covers a single topic too: Fail2ban.
Securing my network is a priority. In previous chapters, the snippets of my Caddyfile was not showing my actual domain for the reverse proxy. I have no illusions that using a random domain would be enough. Security through obscurity1 cannot be the only technique used. As of today, all the applications I’m hosting have their own authentiation framework and I’ve further protected almost all of the endpoints with Authelia. What would happen if someone finds the domain and attempts to login multiple times? They could do either a denial of service or try to brute force access. This is where Fail2Ban comes in.
Fail2Ban scans log files and can be configured to ban IP addresses with too many failed attempts by adding a system firewall rule. Getting this added to my homelab seemed like a good fit. I would need to configure Fail2Ban to read the Authelia logs and the Jellyfin logs since the Jellyfin apps don’t work when Authelia is added to the site.
At this point, I have a confession to make. The progress I’ve made on my Homelab has not been done entirely on my own. I’m leveraging an assistant that has been incredibly helpful. The assistant’s name is ChatGPT. For each application or goal, I would create a new chat thread. I’ve been impressed with how helpful it can be. ChatGPT remembers things from previous chats, so when I’m asking for a Docker Compose file it automatically adds the network that I created in my Docker environment. The reason Fail2Ban is the only thing covered in Chapter five is that LLMs have their limitations. I ran into those with setting up Fail2Ban. At one point, my prompt started with “WHAT IS GOING ON? I THINK I’M LIVING IN A CRAZY WORLD.” ChatGPT successfully talked me off the ledge, but it ultimately did not provide me with the right solution.
So, how did I get here. This prompt started a multi-day voyage:
“With the services I’m running on the Mac mini that are exposed through my reverse proxy, how can I set up fail2ban to protect the services from unauthorized access attempts?”
The conversation turned to determining how I would run the application: natively or through Docker. I chose Docker and ChatGPT gave me the initial steps including a docker-compose.yml file and the default configuration files that can read through Authelia logs.
The Fail2Ban container needs access to the log files from other applications. This means that each log directory must be mounted to the Fail2Ban container. When I configured Authelia, I did not configure its logs to store on my Mac mini. This was the first challenge. ChatGPT helped me change my Authelia setup to store the logs.
First, I changed my Authelia configuration.yml file to add a new section:
log:
level: 'info'
format: 'text'
file_path: /logs/authelia.log
Second, the docker-compose.yml file needed an extra entry in the volumes section to mount that logs directory to a folder on my computer.
I did a down/up with the Docker Compose command. I tried to do an unsuccessful login and I was able to see an entry in the authelia.log:
time="2024-12-12T22:28:49Z" level=error msg="Unsuccessful 1FA authentication attempt by user 'asd'" error="user not found" method=POST path=/api/firstfactor remote_ip=172.18.0.1
I was excited to see the log files and that I’d be able to throw them to a regex on Fail2Ban but after a moment, something didn’t seem right. That IP address was awfully peculiar. After doing a little bit of research, that IP address was the local address of my Docker network. No matter who accessed my services, the IP addresses logged on Caddy and the access issues in Authelia would show the local IP from Docker. If I wanted to ban an IP address, I would need the real IP address.
Everything was attempted. There were dozens of chats back and forth with ChatGPT. I was doing classical search and there seemed to be many things to try.
I’ll save you all the troubleshooting steps, but I tried to run the Docker container using the host mode which is not supported on the Docker for Mac. Things functioned, but it wasn’t ideal. Eventually, I followed the pattern I did with Pihole and installed it outside of Docker. It was simple to move from the Caddy install I had on Docker to the native solution.
brew install caddy
I migrated the Caddyfile to /opt/homebrew/etc/Caddyfile. I changed the location of the log file back into my ~/homelab diretory.
Then I deleted the Docker Caddy install and started the caddy service through Homebrew:
brew services start caddy
Everything functioned as I’d expect. The logs now are recording the real Remote IP addresses.
I took the instructions from ChatGPT and modified the docker-compose.yml to pass in the log files for: Caddy, Authelia, and Jellyfin. I created the jail.local file and the filter.d for records for the three.
To my surprise everything started on first try. I ran the scripts to test the regex and nothing. Absolutely nothing was happening. After hours of troubleshooting, I found the issue was with the volumes on my Docker Compose file. I had the wrong folders. Oops. Once that was corrected, I turned to the email issue. I wanted to get an email when an IP address was banned. Setting up Mailtrap wasn’t working right away, so ChatGPT had me install msmtp on the Docker image. That change worked without a hitch.
Now, back to testing the regex. I really wanted to ensure it worked on Authelia first. I ran the command and nothing. It would fail or I’d get errors. ChatGPT gave me this as the regular expression to work with Authelia:
[Definition]
failregex = .*level=error msg="Unsuccessful 1FA authentication attempt by user '.*'" error=.* method=.* path=.* remote_ip=(?P<host>\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}).*
ignoreregex =
After more hours of troubleshooting, I eventually found examples and the issue was with the HOST portion. The Regular Expression ChatGPT had was technically correct and after multiple back and forth and me complaining I’m living in a crazy world, it would keep giving me it. Fail2Ban did not like breaking out the components of the IP address. This one worked:
[Definition]
failregex = ^.*Unsuccessful 1FA authentication attempt by user .*remote_ip="?<HOST>"?.*
^.*Unsuccessful (TOTP|Duo|U2F) authentication attempt by user .*remote_ip="?<HOST>"?.*
Jellyfin had instructions so adding that to Fail2Ban was trivial.
I spent more time going back and forth with ChatGPT on Fail2Ban and most of the issues were related to macOS not being a perfect citizen in Docker and it’s limitation on what Fail2Ban actually needs. It’s important to remember that the LLMs are incredible but they do have limitations. Sometimes you will need to read through a lot of forum threads and documentation. The promised land isn’t here yet.
This will be the final Homelab Chronicles entry for a little bit. My focus right now is on moving my Python scripts to Docker.
Security through obscurity is helpful, but relying on it can mean that as soon as the URL is leaked (and it will be) then everything is compromised. ↩