Going into Crisis Mode

If you have been in IT long enough, you end up experiencing a “Crisis Mode” day. You know, you click a button and immediately all hell breaks loose, then instantly, your brain clicks into crises mode. I had another one today. Tried to update our firewall, which normally has no issues, and comes right back up. Not today, that update failed.
The funny thing, it’s failed before and it normally comes down to a service package running that I forgot to turn off, and I figured I would just run it later, it was getting close to start of day for employees, so I ran off to do something else and figured I’d try again another day.

I come back, and 3 people are in my office, yep, you guessed it, no internet. Literally within 5 minutes, 3 people. Instant crises mode. I couldn’t access the firewall through a browser, and hooking up to the console, lots of errors related to files necessary that are missing. It became clear this was not a quick fix.

I kept our old firewall mounted to the wall, just in case a day would come that we needed to failover to it. (good job ken). That got everyone back online and happy. It’s a good bit slower, and can’t handle the normal fiber speeds, but people can work. My boss is typically understanding when things like this happen, however he does not have time to deal with employees screaming in his ear that they can’t work. I was very happy to avoid that.

I had a configuration file backed up, so I knew my life wasn’t going to suck too much. After trying a factory reset, all I received were errors, and the reset button didn’t even register when I tried it. (and yes, I did it correctly).
Finally, I loaded up the latest release onto a flash drive and loaded the firmware. It acted as an in place upgrade (surprisingly). I didn’t even have to load the backup. 4 hours later, crises resolved!