Rebuilding This Space

I’ve been wanting to do a writeup on this for a bit now. I had a pesky IPv6 issue with my server, and thought, hey, why not just nuke the whole thing and restart from scratch? I first opened this server in August 2020, and installed YunoHost back then, so the installation has been around for awhile. As a result, lots of cruft built up over time, and somehow things went south with my IPv6 configuration, to the point where YunoHost kept throwing warnings about non-functioning IPv6. As I couldn’t figure out the issue, despite loads of attempts, I thought I’d rebuild everything and reinstall.

I backed everything up to the best of my ability. YunoHost has a nice backup system in place that will let you back up apps, as well as system configurations, so I saved everything. I downloaded all the files from my NextCloud. I thought that most things could be rebuilt. I saved my Bitwarden/Vaultwarden vault to .csv so it could be easily reloaded once I had the server back up and running. That was one of the scariest parts, since I did not want to lose my passwords, y’know? I have my spouse, sister, and ex on the server as well, so I let them know to back up their stuff before I cleared everything out. Once I got confirmation that everything was backed up, well, it was time to take the plunge.

The unfortunate thing is that Contabo, my VPS provider, does not have an option to simply reinstall the OS. The whole server has to be wiped. Since I had everything set up where /home was on its own partition, and / (where the OS lives) was on its own partition, it would have been really nice to just be able to reinstall the OS to / and leave /home untouched. But no dice there. So I started from scratch and started the “reinstall” process. It completed fairly quickly, installing a fresh new installation of Debian 12 “bookworm.” From there, I booted a rescue CD, got into Xfce, and ran GParted to set up partitions. My goal was 60Gb for /, 1.5Tb for /home, and around the same for another partition which I intend to use for encrypted backup. I set up partitions exactly how I wanted them. I restarted, and from there, I had issues logging in. I thought I’d messed something up. In fact, I had, because I’d edited /etc/fstab (which indicates partitions) with the incorrect UUID for /home. Once I fixed that, we were in business.

I logged in via SSH as root, and had access to my server. Great! Now from here, the idea was to install YunoHost, which is not its own OS, but is sort of an “overlay” on top of Debian. The installation process seemed straightforward enough. I hadn’t done this for about 4 years, so I didn’t remember how to do it initially. Once the installation completed, I was left at the command-line, and it suggested I log in and set up my admin user via the web interface. It’s a little scary that YunoHost exposes this interface to the web at large, since anyone could theoretically log in right after it’s installed, and wreak havoc. Nonetheless, I went through the setup process in the web interface, and set up my admin user.

Unfortunately, something went wrong after that, and I wasn’t able to log in with my admin user, no matter how hard I tried. I have no idea what happened. Debian was functional, as I could login via VNC, and the rescue CD, but it just didn’t work correctly. I also had an issue provisioning a Let’s Encrypt SSL cert for my main domain, so my browsers were freaking out when I tried to log in via the web. It was not great.

So we’re back to the drawing board. I hit “reinstall” again to clear everything out and reinstall Debian. That didn’t entirely take, which was frustrating, and I worried I messed everything up. I ended up giving it a day, sleeping on it, and came back to it the next day. The reinstallation “failed,” so I started the process again. Debian reinstalled correctly, and then I started the YunoHost installation. From there, I didn’t use the web interface, but used the YunoHost “postinstall” process from the shell, and set up my admin user from there. I then set up public key authentication and disabled passwords, so I could only log in to my server using a SSH key. This is by far the most secure way to do things if you’re running a VPS, and I highly recommend it. Any hackers or bots trying to access your server or brute-force the password will run into a total brick wall, since it won’t even accept any passwords!

Once YunoHost was set up and I had access to it with my admin user, I could set about restoring the apps I had previously installed. IPv6 worked great out of the box. I had to rebuild some DNS records, but it’s OK, it worked out fine. YunoHost complained a lot, but I eventually got the records correct. I restored just about all of the apps (Vaultwarden [Bitwarden], Wallabag, Navidrome, Metronome). Most of them were just fine. My NextCloud backup, however, completely malfunctioned, and I realized that I’d need to rebuild it from scratch. OK, no problem. I reinstalled NextCloud and set it up to my liking, reinstalling the apps I previously had. Set up bookmarks sync via Floccus to NextCloud Bookmarks. Restored my RSS feeds for NextCloud News.

I actually didn’t restore from my Vaultwarden backup, mostly since I wanted to redo the password and start fresh. I imported my vault from the .csv backup, which worked great, although I had to regenerate my Discord passkey. I also set up the admin interface with a new password and made sure to get my spouse and ex invites. For whatever reason, Vaultwarden/Bitwarden sets up new user accounts by invite emails. The problem is that I had to make sure my mail server DNS records were set up correctly, or mail wouldn’t come through. Thankfully, they’re both set up now on Vaultwarden, and we’re golden. Emails seem to be working fine.

My Wallabag app restored 100% fine. Navidrome too, and Metronome was fine. However, I did have to regenerate the main domain’s Let’s Encrypt SSL cert so that XMPP uploads would work correctly. That was not super intuitive, but it worked out eventually. I also had to mess about a lot with permissions for my music files in the YunoHost Multimedia/ directory, which was an exercise in frustration, as NextCloud’s finicky permissions are not well-documented. But we’re back in business and I can now stream music from my server via Navidrome again.

I also reinstalled LibreTranslate. I think I’ve mentioned my issues with LibreTranslate before. I didn’t reinstall a backup of that because there’s no point. Reinstalling from scratch is just fine, since I’ll need to redownload the language model files anyway. One major flaw of the YunoHost package, at least, is that it only ships with French<->English, but the app has far more languages available. My blog post linked above explains how to get the app to download all the language models, and how to store them in /home, if desired, if you need more space (the models are probably 4-5Gb for all of them). So that was pretty easy to install, thankfully.

It’s always a bit of a struggle making your OS your own, whether on a server or on your physical laptop. I know it took me awhile to get Debian on my laptop to work just how I like it, and the same goes for the server. I unfortunately had to set up my encrypted backup system again from scratch, and I’m slowly working on reuploading and mirroring my laptop on there. That took far less time than I thought it would, mostly because I followed my own tutorial. I mentioned on Mastodon that I write tutorials not only for y’all, but also so I can retrace my steps if needed. My own guide was a lifesaver for re-creating the backup system.

I honestly think everything is just about back to normal. My blog rsync’d swimmingly after I reinstalled “my_webapp,” and was as good as new. I was just happy I could redo things and have a cleaner, fresh start. I am a fan of updating in place, but there is something to be said for completely restarting. As I mentioned before, I would have just reinstalled Debian to / if I could have, but that unfortunately is not an option with my VPS provider. I get it, it’s probably security or somehing, but really, it would be so much easier if I could just boot to a LiveCD, reinstall to /, and leave /home intact and not have to wipe everything else. I am hoping I don’t have to do this again for a long time, or at all.

If you redo your server, I would really recommend changing up all your passwords, regenerating SSH certs, and basically freshening everything up. Not that your previous credentials were necessarily insecure, but why not just make it harder for attackers? My Logwatch emails show a whole lot of people trying to access the VPS every day, and every little bit of security hardening helps. My encrypted backup also uses a keyfile which is only stored on my laptop, so that is also super secure. There is a ton you can do to harden security, and it is well worth it.

I’m just happy the server is back. I’ve had it for a little over 4 years now, as I mentioned, and as I was telling my spouse, I don’t think I realized how integrated it is into my life. I use Nextcloud for contact/calendar sync over DAV, Vaultwarden/Bitwarden for password management, NextCloud Bookmarks for sync, and Navidrome for streaming music. Those are just a few of the things I use. It’s been lovely having it, despite being a pain to run sometimes. Totally worth all the pain here and there, but honestly, most of the time it Just Works(TM), which is best case scenario.

If y’all use any of the services (well, it’s just LibreTranslate for now), let me know what your thoughts are. And as I’ve inquired on Mastodon, I’m down to add more services for the public to use, but I don’t know what people would want. CryptPad seemed promising, but it was a huge pain to set up, so that’s a non-starter. I imagine there are others which would be worth the trouble. I miss running Searx(NG), but it works about half the time for search, which is unacceptable to me. Invidious and Piped have been blocked to hell and back by YouTube’s servers, so they aren’t great either. I’ll keep an eye out for interesting stuff on the YunoHost Apps store, though, and we’ll see, tempus docebit.

I just kinda wrote this more quickly than I expected. I’ve been thinking on this for a few days now, but haven’t written anything up until now. Thanks for reading, friends <3