The Collegian website went down for most of the day Dec. 28, 2023, and when it was available again, stories from two weeks prior were erased. While a campus-wide cybersecurity incident is happening right now and is affecting our website at the time of publication, we still want to address what happened last month for transparency.

At 10:15 a.m. Dec. 28, 2023, the website went down because of an internal issue. We run our websites on virtual machines on a bigger server, and we take snapshots of these virtual machines every couple weeks to have backups in case of a major incident. However, our snapshots started to take up so much space they filled up the server’s memory, preventing our virtual machines from running properly and causing our websites to crash.

This is not the first time this has happened, and usually we delete old snapshots to free up storage for the virtual machines to work again. We did apply this, but another problem with the machine arose.

When attempting to turn our virtual machines back on, they did not reboot properly. Every time we turned them on, they would repeatedly shut themselves off and back on again until they eventually stopped.

We have a solution for this situation as well, but in our haste to get the server back up, we pulled our last resort option of reverting to a previous backup from Dec. 13, 2023. This solution deleted all stories between the 13th and the 28th, but on the bright side, our website was back up and running just in time for our Pop-Tarts Bowl post-game coverage.

Within the next two days, we had all our stories reposted with their original dates and almost all sponsored content restored. We have backup documents for all our stories, pictures and content, and with the help of the Wayback Machine, the lost stories were republished to their original states. Except for an added editor’s note and reset view counts, the stories are exact replicas of the originals — including the urls, so social posts and shared links still work.

We apologize for the inconveniences this may have caused, and we are taking steps to ensure this doesn’t happen in the future. We will not pull the last resort option until after affected Collegian Media Group leaders mutually agree to it and only when absolutely necessary. We will also take more preemptive measures to prevent this, including doing a better job of watching our server’s memory.

Thank you for your continued support of the Collegian. If you have any questions, please direct them to IT manager Garrett Brown at [email protected] or to editor-in-chief Carter Schaffer at [email protected]. We will continue to provide transparency when these incidents occur, including when the Jan. 14 campus-wide cybersecurity incident is resolved.