Still Want To Be a SysAdmin?

12-02-03

Azizi waves.

Calin has a tale to tell.

Calin says, "A tale of horror."

Pearl says, "Oh, lots of tales today. Is it about what happened yesterday?"

Calin says, "A tale of how the server was down yesterday, and how I was up till 4am..."

Pearl exclaims, "WHOA!"

Calin says, "And finally of how it all works perfectly by the morning."

Pearl says, "Whew, so it has a good ending, Calin?"

Azizi blinks, "What was wrong with it?"

Calin says, "Oooook."

Calin says, "So, as you may or may not know, I upgraded the kernel recently."

Pearl settles in to listen.

Azizi settles in next to Pearl.

Calin says, "Then I upgraded several packages to get various things to work better (or at all) with the new kernel.

Calin says, "So yesterday, I noticed that the server could not make outgoing connections."

Calin says, "Which was very odd, as we could connect to it."

Pearl remembers.

Calin says, "This meant a couple of things: It could not send mail (though it could receive it), and it could not update splitreflection.com if my dsl went down."

Calin says, "Now, to start with, I was not very smart."

Calin says, "Yesterday from work I futzed with the routing table. Which changed things so we couldn't connect anymore. So I had to run home to fix it."

Calin says, "Then, I decided to restart my router. Also from work. Also a bad idea, as restarting the router means my DSL goes down... and the server couldn't fix its ip address after coming back up."

Calin says, "Which again means we can't connect."

Calin says, "And I was swamped with Work in the afternoon, so I couldn't run home to fix it."

Calin says, "Anyway, I didn't get out of work till about 6, and I rushed home."

Calin says, "I futzed with it for a while, and decided some settings weren't loaded properly on the server. So I figured I'd reboot."

Cal says, "DA Da dahhh."

Calin says, "After rebooting, the network didn't work at all. It seems one of the packages I upgraded went to the trouble of deleting my old network configuration."

- Ka-BOOM!!!

Calin says, "And my auto-detect package hadn't been upgraded and didn't work with the new stuff..."

Calin says, "So I figured I'd boot with my handy linux boot cd (which would get the network going), and then download the newer auto-detect packages."

Calin says, "Well. It seems the cmos battery was dead, and it defaulted to booting first from the hard disk, last from the cd."

Cal asks, "Crunch?"

Calin says, "So I ran out and bought a new battery."

Calin says, "I came home, opened up the system, put in the battery and pushed the power button...."

Calin says, "And waited...."

Calin says, "The drives whirred up, but the keyboard and monitor stayed off."

Calin exclaims, "FUBAR!"

Azizi meeps.

Calin says, "I will probably need to cook that motherboard for a couple more minutes. That's what it did before my brother and I fixed it in the oven."

Calin says, "Bad solder joint somewhere."

Calin says, "So, as you can see, this is going from bad to worse."

Azizi nods.

Calin says, "I couldn't even WORK on the problem I had at first until I fixed the more serious motherboard problem."

Calin says, "So ANYWAY."

Cal says, "hardware failures, one thing even linux has trouble with."

Calin says, "After holding my head in despair for a little while, and crying on my girlfriend's shoulder, I came back to the problem."

Calin says, "Penguin had two hard drives. The root drive, and the media drive."

Calin says, "I took them out of the system, and put the root drive into my USB enclosure, and hooked it up to my desktop computer."

Calin says, "I copied the whole thing to my hard drive."

Calin says, "Then I hooked up the media drive, and moved the contents of the root drive onto there. So now all of Penguin is on one hard drive."

Pearl says, "Whew! What a pain."

Calin says, "I changed my desktop computer's name and IP to match penguin."

Calin says, "I moved my telnet service to a different port, and shut down a few other services."

Cal asks, "so we have a temporary fix?"

Calin says, "I chrooted to the usb drive, and started up all the services."

Calin says, "And here we are."

Calin says, "Yup."

Azizi says, "And now I know why I couldn't connect last night."

Calin says, "And then I went to bed."

Pearl says, "Ugh."

Azizi shakes her head, "What a pain."