Congratulations, you're looking at this on the brand new server... At last!
It has been a bit of a rocky journey! We're using a non-standard installation of Ubuntu using a ZFS filesystem. This is a feature that they're looking to roll out in the next LTS 20.04 in a months time, but not to the level we've taken it! The feature will only allow the use of whole drives, not individual sections of drives like we need. We had no end of issues during the move, firstly the server refused to boot as it didn't want to read the /boot/EFI partition. We eventually fixed that using an EFI Shell, simply accessing the fileset and renaming the directory twice did the trick although we still have no idea why!
Then we got everything running and installed ready for the move and during our tests we found the MySQL on ZFS is a NIGHTMARE. The databases were performing around 6 times slower on a server with no traffic than they were on the old, fully loaded server. We tried every trick in the book to tune this but ultimately failed. In the end we had to wipe the new server and start from scratch. This time we reserved a 1TB partition from each drive and used that to create a software Raid10 EXT4 array to place the MySQL data files on. This worked much better
I've learned a hell of a lot about custom partitioning, and a lot more than I ever wanted about EFI Shells. Ultimately we have ended up with a server that:
- More power processors than before
- Same RAM as before (speed and capacity) - 32GB
- 14TB of RaidZ1 ZFS disk array for server and site data (before we had3TB in Raid1 and a single 3TB backup drive)
- Every site (and a lot of key linux areas) reside of their own ZFS volume which gives us data snapshots (point in time recovery). 1/4 drives can fail with no impact on data and minor impact on speed.
- For the sites and key linux areas, the snapshots run hourly and are stored for 48 hours. Daily are stored for 2 weeks and weekly are stored for a month. If we delete something or it breaks we can recover up it up to a month later.
- MySQL sits on a 2TB Raid10 array meaning we can tolerate 1/4 drives (potentially 2/4 drives if it is the right two!) failing with little impact.
- Backups are made daily consisting of all site data and the databases. These are both stored on the server for immediate use and copied remotely to a backup server in another datacentre for redundancy. Akuma also copies these backups to his home server. This gives us 5 copies of the data - 1. Original Data, 2. ZFS Snapshots, 3. Backups on server, 4. Backups on another server, 5. Backups on Akuma's NAS. The maximum amount of data we stand to lose in the event of a dual drive failure is up to 48 hours. If I get clever enough, I'll also work out how to remotely back up the snapshots to my own NAS's ZFS array with incremental updates on the fly. This will have to wait for another day though!
- The server software is vastly updated, although we're still limited to using an older version of PHP due to incompatibilities with our scripts and newer versions.
A lot of you may be wondering what happened to the planned upgrade to Xenforo. Unfortunately (I'm using this word a lot today!) things did not go smoothly. There were many issues that I was unable to overcome. These include:
- No direct upgrade path for the downloads script - we have 2844 files for download (fiction and videos), importing them manually would be a nightmarish job.
- Multiple issues upgrading from such old scripts that needed work arounds.
- Inability to recreate a lot of the current functionality, specifically the front page.
- Escalating costs for plugins.
- Zenforo 2.X was also a very new release at the time and as such contained many "features".
This is still part of the plan, but it has had to take a backseat to real life issues and time constraints. To get the project rolling again will cost around $300 in licence extension (yes, you have to pay annually for plugin and forum licenses if you want to stay up to date with current software and upgrade) fees. It will also rely on either Akuma being able to dedicate a significant amount of time to 1. Learning the software, and 2. Custom coding the missing functionality; or more money for a freelance coder to do the job. We'll have to wait and see how things pan out as right now donations are only just covering the single server fee, I paid the new server setup fee and the extra month for the old server out of my own pocket.
Remaining issues from the move:
- DiaperedAnime lost around 2 hours of data. Unfortunately I was unable to restore the changes made while we tested it on the new server back to the old server due to non-backwards compatible software differences. No other sites lost data.
- AnimeOTK has some broken download files. This was caused by use of non-utf8 characters in the filenames. Transfer and copy commands really don't like non-utf8 characters (in this case they were latin1 characters) so they had to be automatically converted during transfer. This has resulted in around 15 broken downloads - please report them to Akuma if you find them as there is no simple way to locate them without trying every download manually!
