Hey all. It's been about a week since this place has seen the day of light. We suffered a catastrophic incident related to the Database itself which lead to GDU having to be taken offline. At this point you may be wondering, how the fuck this may have happened? Well, we have the answers.
Tl;dr
What exactly happened? Ran outta disk space, tables got eaten for breakfast by the Server, latest backups were located in a different State than where either of us were present. Wally and gm112 had to go down to hell and fight both God and Devil at the same time. Gm112 died, but Wally won by drinking from the Holy Grail.
What about my Posts, PM's, Profile, Forum Settings, etc.? Nothing important was lost except for a few customization/plugin related things to the forum itself (but nothing on user data).
InnoDB vs. MYISAM (the semi-technical explanation)
The Database itself suffered some corruption that hadn't been completely dealt with from the last hacking incident with the Russians. Which lead to the recreation of GDU's Database composed of a combination between the Russian-hacked version and a backup. Unfortunately given the nature of the attack at that point, some data was just lost to time. Leaving us to the recreation of GDU's Database.
For performance purposes many of the tables for GDU make use of InnoDB as its storage engine rather than MYISAM. There are plenty of pros/cons that make one better than the other depending on use case. But mileage may vary, of course. On May 19, 2018, we were attempting to upgrade the server to more storage and to do some general maintenance to help combat the technical problems on GDU. But, when the server came back online from that upgrade.. things weren't as okay as they seemed. Unfortunately for us, the newest backups were physically in another State, so we were left with having to rebuild the database with what we had on the server.
As a result, internal Table ID's that InnoDB utilizes ended up falling out of sync from the actual InnoDB data file itself. This was a minor problem however, as there is an internal mapping that MySQL utilizes to keep track of every InnoDB table. Then, the server had issues with storage with MySQL(default settings changed over time, breaking system specs) consuming over 60% more space than it should/needed to leading to out of disk space issues. This manifested itself over time as GDU simply not loading. But, this is an important thing to keep in mind as when these events happen, there is an off chance that you are allowing incomplete commits/writes to happen specifically on InnoDB tables(MYISAM isn't included due to differences). So, we were in a state where the tables had no mapping to the InnoDB data files, various InnoDB data files themselves were corrupted with bad writes(thankfully only System tables were affected here!) and GDU was looking relatively bleak.
Recovery was possible because we were able to reconstruct the same exact structure of the old MySQL Database. Through the process of hex editing, importing and re-importing tables again and again just to make the InnoDB Table Mappings match up and all of the table pointers in the InnoDB Data files, eventually the forum began to start showing signs of life.
While every table containing user data has been successfully re-imported, the situation regarding other systemic tables (such as one including routing information for our Status Updates Plugin EDIT: For clarification, the "systemic" plugins are based around URL mappings and basic settings/configuration no actual data regarding the Statuses were lost) have not been fully recovered due to difficulties with MySQL accepting the data that's in there. Upon reviewing the InnoDB Data files for those tables, I found the information was mostly still in tact but what was present in there all could be re-established by simply updating/reinstalling those customizations.
Moving Forward
We are in the process of a full audit of the server and have made efforts in addressing the data storage issue once and for all. And more importantly, to establish a backup system that is a more modern than what was available 10 years ago. Although these are reactionary changes, we will work proactively to help prevent bad events like this from ever happening again after our audit is done. In the mean time, nothing really changes here at GDU. Same ol, same ol. Wally's a troll, etc. HIME
Known Issues
-
Status Update Widget is fucked.This has now been addressed. - GDU Pages Content not enabled/but is present in DB.
- Some other plugins might be fucked