Share Your IT Horror Stories 6

[Mike555]

My first IT boss once said to me "No matter how difficult your job may seem at times, you always have it easier than someone else." For some reason those words stuck with me, and I remember them often whenever difficult situations arise at work.

In that spirit...

This is always a topic I like to bring up around fellow IT colleagues. Tell us about the worst situation (horror story) you've ever encountered while working with IT. Everyone has atleast one story to tell.

I'll go first:
I was once hired by a small company to upgrade their entire network from Windows NT to Windows 2003/XP. This company 20 employees but no IT staff in-house. They'd purchased 20 new PCs and a Win2k3 SBS Server, but it was all just sitting around waiting to all be installed. Management told me they wanted to get the server installed first, and then worry about the PCs later.

Among other things I needed to check leading up to the installation, I needed to ensure I knew the local administrator password for each PC. (In order to add an NT machine to a win2k3 domain, you need to log on locally and manually add it most of the time.) I was told by the manager that all local admin passwords were blank. I checked 5 computers to make sure that the passwords were blank, and they were, so I continued under the assumtion that all 20 local admin passwords were blank.

I dedicated an entire Saturday to perform the installation, and when that time came I showed up and began the installation. Everything went fine until I realized that besides the 5 workstations I had initially checked, the other 15 workstations required passwords to log on locally.

To make a long story short, no one knew what the passwords were, not even company management. We were also unable to contact the folks who originally installed the PCs. Because of this I had to take a week off of work at my regular job and stay on site to install all the new XP machines because the NT machines were useless. The one-day project turned into a week-long hell.

Please share your stories too!

--
Mike

 

[IdahoTech]

Here is one that is not to big but for a new job in a new town and state, it was kind of heart stopping in a way...

I work for a small school district in Idaho. When I first came here they were still doing machines by hand. So I talked with the boss and had him get Ghost to simplify the process.

All went well that summer ghosting machines and there was much improvemnt on how they worked and they were running faster than they had in a long time.

This story has a two parter.

When school came around, a teacher was having a real tough time with one of her computers so I said ok I can re-image it and you should be good to go. Well I guess I should of exlplained a little more about what 'RE-IMAGE' meant.

She had power points on the system as well as presentation material for classes and and other teachers and meetings. etc.... So in one fail swoop I wiped her system and all her school work in about 5 minutes...

I know it doesn't seem to bad, but hey for the first time working here it was a nightmare.

Also the second part of the story. I then came to understand that the k-2 school teachers had about 5 years of work on their systems, ouch..... So as you can tell it was a very hard first year, because every time I came near a system the teacher looked at me with worry and said 'What are you going to do to it??' It's amazing how word spreads fast in a small town.

So since that point there is a policy in place that they are responsible for their data, period. We provide the server as backup for them but they are warned that they are the only ones responsible to create a backup. So the tech in each building at the end of the year makes a Burn of their data in their home directory.

IdahoTech
'Only two things are infinite - the Universe and human Stupidity, and I'm not so sure about the Universe' Albert Einstein

http://www.sd60.k12.id.us

 

[Varith]

My favorite horror story was (almost) completely self-inflicted. I was working my first programming job as the first employee for a company that made card access systems (the type that unlock doors). One of the owners of the company had designed the hardware and firmware for the card readers and controller board that connected to the locks and made decisions about whether the door should be unlocked or not. I was hired to write the software that allowed the administration of the system.
There was not a lot of planning for what would turn out to be a fairly large piece of software. Basically, I was shown a few dBase prototypes of what they thought they would need and some brochures on similar systems already out in production. Needless to say I struggled mightily for several months but eventually we got some sales.
Shortly after installing the system at our second customer, I had to release a hurried patch out to their site, an hour's drive away in Manhattan. One of our techs took it out, getting it installed by about 4 pm. He ran the dowload which pushed all of the changes to the controller boards. Every card reader in the building, all 30+ of them, was suddenly informed that it should have a keypad attached and that it should not allow access to users until they had entered their correct PIN number. Unfortunately, the readers did *NOT* have any such hardware. I had introduced a worse bug than the one I fixed and now *NONE* of the doors could be opened.
Ok, time for the tech to go back to the old software (he was smart enough not to have erased that). He runs the download again and....nothing. The readers are still in keypad mode. Ok, now I can panic a little. After a quick consult with the hardware guy, we discover that there is *ANOTHER* bug in the card reader firmware that does not allow the keypad mode flag to be unset once it has been set. The only way to unset it is to power off the reader.
In the meantime the few customer's employees have propped doors open so they can continue work. They are not yet too upset as it is after hours and they have the (very firm) expectation that this will all be fixed before the morning. But our tech, deciding that there was nothing he could do, left.
So, to speed the story, I spent most of that night in the customer's building, unscrewing the readers out of the wall, unplugging them, waiting five minutes to make sure the memory cleared fully, and then plugging them back in. Fun.

 

[gurner]

our company inherited a system installed by a plum, can't comment on the workstations as i don't go near them (type of place you want to work on remotely or send someone else)

thier Domain had been called 'companyname.co.uk' not to clever for an internal domain, but workable.

however, after a complaint they couldn't access their new website (hosted on an ISP), but other 'external' people could. It quickly transpired their DC had been named 'www'

8-P

Gurner

http://www.divineparadox.co.uk

 

[chris13]

from the old days where a 486/33 was 'state of the art':
One day, I had to install an additional harddrive in a development-server (netware-sql). Some weeks later my boss told me, that we had been wondering about the development-server reacting very slowly and that he found the cause. It was me!
Seemed that I had pressed the 'turbo'-switch with my knee without noticing when I had finished my work and the checkups and stood up to leave the room...
Then grabbed a lot of key caps of our Nortel-phones to glue them around all the 'turbo'-switches in our server-room. And the 'reset'-switches, too

 

[jrbarnett]

Gurner - we have virtually that exact situation at my current workplace, and people wonder why we have a dedicated server to host a copy of our public website for internal use only.

John

 

[johnpoole]

thought i would add one, a simple upgrade, 4 backups, never have to many. enter now keycodes to turn on some soom to be unused features and reboot ( i'll be home before the game) system does not boot, pls insert a bootable disk, of course the only disk is the hd. not a problem, restore from the backup tape, (i'll be home before the 7th inning stretch). still non bootable, do a new install and restore cust data from a tape. new installs clean, restore cdb, tape auto formats the hd, blows away the os, replaces it with the backup, nonbootable. not a problem (i'll be home in time for my wife birthday) 104 hours later, my backup came in for a 20 hour shift so i could sleep on site.. (6 days down time) it was only the largest hosp in the state and i had been their trusted super tech for almost a week..

john poole
bellsouth business
columbia,sc

 

[donbott]

I'm working for a telemarketing/phone service outsourcing cmmpany and have been there for about two months. I knew nothing abou phones and very little about networks but I was highly recommended by a guy who used to work with me. I'm in the main computer room changing a station or something, a little nervous because I'm not used to being around all this equipment. I press enter, all the lights flicker and it feels like the whole room moves! Seconds later every technician in the building is running into the room - nothing is working! What did I just do?

Our $100,000.00 UPS had failed at some point and no one knew it. We took a power hit and everything went down. The room did move, at least the floor because of the air conditioners compressors losing power for two seconds and then kicking in again.

DonBott

 

[wvajenm]

This happened to me a few years ago, when I was starting my first job. A temp came in to replace the clerk in our Eng/IT dept. She was supposed to be computer literate, a very nice lady. I was the junior guy on the block, and our dept clerk was responsible for putting out the company newsletter, so I was enlisted to help the new temp the first time out. I began to realize the temp agency may have deceived us about her computer skills when I had to show her how to turn the computer on. That was the highlight of this training session. The program used to publish the news letter required the CD to be put in to this computer that had been custom built to accomate the old CAD drawings on 5 1/4 floppies. So, after instructing the temp to put the CD in, and her repeatly saying it still wouldn't find the CD, I went to investigate. You guessed it, the CD was in the 5 1/4 drive. Being young and this being my first job in IT, I got some experience in taking apart an old drive to retrieve the CD.

 

[MikeLacey]

right...

I used to be second-line support for the IT in some offices about a hundred miles away. They had a local IT dept which did PC & phones; I looked after the UNIX machine with its MFG/Pro database.

Local IT were installing video conferencing and using the computer room - one of these barn like places with 4 servers in one corner - to try it out.

Barn like room - big noisy airconditioning unit. Too noisy to set up the video conferencing really so Local IT person (who I'm not going to name (Sue)) turns it off. No problem, it'll get turned back on after she's finished.

But it didn't.

Full summer, sun comes around the building and heats up the room nicely.

By about 9pm it's up to 33C and the UNIX server shuts down in disgust. So far so good.

By 9.30pm it's hit 35C and the disk-array, which doesn't shut down on its own, has a three disk failure. An HP AutoRAID box if anyone remembers those, nice but vulnerable to simultaneous disk failures.

Anyway - Oracle database is corrupt along with most everything else. 3am next morning sees me fixing file-systems and waiting for the engineer to turn up and replace disks so that I can restore database.

Engineer replaces disks and I start off restore, engineer and I yak for a while (users and local IT people we have known, you can imagine) and then go outside to have a cigarette. I am, I estimate, about an hour away from finishing.

I light a second cigarette and engineer goes inside to check on restore progress. I'm holding burning match in my fingers as engineer says "It's asking for the next tape - where is it and I'll put it in." I burn my fingers quite badly.

You've guessed it I'm sure. No second tape, backup's been overrunning for more than a year now. I don't have a backup for this company's stock control, manufacturing and invoicing system.

After two days I find a two week old on-disk database dump that I can restore from tape without too many errors. It was only there by chance, I'd meant to delete it and it wouldn't have been on any tapes anywhere.

So - my name was mud, they had to re-key two weeks of transactions. I slept for about a day

The company stayed open though thank goodness. 250 people...

Mike

You cannot really appreciate Dilbert unless you've read it in the
original Klingon.

Want great answers to your Tek-Tips questions? Have a look at faq219-2884

 

[chris13]

This has nothing to do with bits an bytes.
Long ago (1991) I was on "alternative service" as helping hand in the technical department of a hospital. Just at the time the hospital had to pass a power outage test. We would have to live without external power for two hours.
Seemed to be easy: There was a really impressive amount of batteries to backup the "red" sockets and operating theaters continuously during the first minutes. Then we had two diesels with more than 500 hp each to drive our own generators (I had problems to open the doors to the machine room, they were sucking air like hell . In the server-room the server-rack (Netware 3.11 with external mirrored HDs) had its own additional UPS as well as the PBX.
The test began, we were really cut off from external power. Batteries jumped in without interruption, the diesel engines woke up and began to roar like hell (WOW!).
Looking at the systems in the server room, we stopped smiling some minutes later: All the additional UPS of the IT-systems and the PBX (now beeing powered by the output of the generators as external source) were constantly switching over between external and internal power! Beep...click...click...beep... at different intervalls between 5 and 20 seconds. We grabbed a scope and did some tests. We discovered that whenever a radiologist acivated his x-ray outfit or one of the elevators were activated, the output of the generators was heavly distorted and had really nothing in common with the usual sine-wave.
The input-filters of the UPSs couldn't handle this and switched to internal battery and back to external source again and again. 30 minutes later the server and the PBX were shut down by their local UPSs even though there was plenty of volts and amperes.
Even worse: The external HDs of the Netware server were not connected to the local UPS. One of the two mirrored drives and one of two power supplies suffered dead...
All in all it was a very good lesson for the tech-department. And I hope that todays medical devices are as tolerant as during these old days when it comes to the waveforms of their power sources...
Always think about the UPS-system when you add more and more consumers.

 

[jrbarnett]

I'll probably get it in the neck for this, as UK based members are well aware of this (as are my colleagues and managers). Relatively small fry compared to some of the others detailed here, but I still think about it; colleagues won't let me forget this easily (and now neither will anybody on TT).

A few months ago, I was involved in a project to output some data from an Access database to external HTML files to an ActiveSync synchronised files folder, which then got synchronised onto an HP Jornada 720 PDA for use in Pocket Internet Explorer.

I went to set up the PDA on my PC to test it, and when I plugged the synchronisation base into my PC the power light went from green to orange and something blew up on the motherboard and funny smells emitting from the base unit, rendering the whole system useless.
Result: Warranty replacement PC from manufacturer.

The replacement unit works fine using the same software and synchronisation base and an identical make/model of PC.

Since then, anytime anyone has wanted whole PC's disabled, they ask me if I can plug something USB into it, or any time USB related items go wrong, its blamed on myself even if there's no way it could possibly be my fault.

John