Alright, so this nightmare is finally coming to an end. It all started last Friday, I had been super busy with classes all semester and I was finally getting some time to work on the site. We switched from bitnami during the last upgrade and so this time the upgrade was supposed to be much faster and easier. Just back up the postgresql database, shutdown everything, run a git pull, migrate and pre-compile the assets in the database, update all the gems, check the config files for any new changes, and bring everything back up. However, it didn’t work out quite like that.
First, the path for bluepill was wrong and it was using a path other than rvm’s. Upgraded rvm and all the gems, but not until after fixing some issues with bundler. There was more though, after digging through the logs I found an issue with one of the config files. Fixed that, only to find that discourse refused to load the page, showing a blank every time. That took all of Saturday and much of Sunday morning to figure out. In the process, I restored the database several times, re-installed discourse twice, and then tried bitnami’s discourse app only to get the same result each time. Finally, I managed to narrow it down to an option with our CDN, specifically “auto-minify html”. Not sure why the new upgrade effected that, I’ll be continuing to look into that in the future, but at least this nightmare is finally over and everything is back up and operational. 
I’ve taken a number of measurements now to ensure that our downtime in the future is very unlikely to have downtime for longer than an hour (due to upgrades).
On the brightside, this latest upgrade has over a month of new updates:
- Improved Summary Mode for huge threads
- Improved Mobile Support
- Support for Google Maps onebox
- Updates to over a month of small fixes and enhancements