Share Your IT Horror Stories 6

[Mike555]

My first IT boss once said to me "No matter how difficult your job may seem at times, you always have it easier than someone else." For some reason those words stuck with me, and I remember them often whenever difficult situations arise at work.

In that spirit...

This is always a topic I like to bring up around fellow IT colleagues. Tell us about the worst situation (horror story) you've ever encountered while working with IT. Everyone has atleast one story to tell.

I'll go first:
I was once hired by a small company to upgrade their entire network from Windows NT to Windows 2003/XP. This company 20 employees but no IT staff in-house. They'd purchased 20 new PCs and a Win2k3 SBS Server, but it was all just sitting around waiting to all be installed. Management told me they wanted to get the server installed first, and then worry about the PCs later.

Among other things I needed to check leading up to the installation, I needed to ensure I knew the local administrator password for each PC. (In order to add an NT machine to a win2k3 domain, you need to log on locally and manually add it most of the time.) I was told by the manager that all local admin passwords were blank. I checked 5 computers to make sure that the passwords were blank, and they were, so I continued under the assumtion that all 20 local admin passwords were blank.

I dedicated an entire Saturday to perform the installation, and when that time came I showed up and began the installation. Everything went fine until I realized that besides the 5 workstations I had initially checked, the other 15 workstations required passwords to log on locally.

To make a long story short, no one knew what the passwords were, not even company management. We were also unable to contact the folks who originally installed the PCs. Because of this I had to take a week off of work at my regular job and stay on site to install all the new XP machines because the NT machines were useless. The one-day project turned into a week-long hell.

Please share your stories too!

--
Mike

 

[lespaul]

We spent several YEARS waiting for our new courthouse to be built. In January of this year, the move finally occurred. Did we have a PLAN, of course not!!!

We had an IT department of 8 at that time. We closed the court at 12:00 Friday afternoon, the following Monday was a holiday and the court reopened at 1:00 Tuesday afternoon.

We have a new (CISCO) network built by contractors with a LOT of different configurations and settings compared to our "old" network. We had to set up and configure 200+ PCs, 20+ network printers, we had blades with missing connectors, we had wall sockets with missing connectors, people starting working on wire management before things were finalized and we have a lot of hanging wires now! The IT division actually started working in the new building a week or two before everyone else. We had two contractors (HS Kids) setting up some of the PCs (which we then had to go back to ALL of them and correct). It was a MESS!!!!

I didn't do a lot and I had 65 hours of OT in a 2 1/2 week period. Some of my co-workers had 200+ hours of OT in that month!!

And the judges wondered why it didn't run smoothly Tuesday afternoon!!!

Leslie

 

[philote]

What a timely topic. I work for a small company with about two dozen or so users. Our domain name registration was about to run out so I decided to renew it a couple of days ago. The registrar we were with was charging a lot more for the registration than I could get elsewhere, so I transferred the domain. This meant our nameservers were going to change as well, which I hadn't thought about before hand.

So yesterday, one of the ladies here said she was checking the president's e-mail (which she does for him) and he didn't have any. I thought it was strange and kept getting reports throughout the day about users not getting important e-mail they were expecting. Then it hit me. I logged into the domain registrar's page and updated the nameservers and dns records. I had to send an e-mail out to all the users saying we were having e-mail problems and couldn't get e-mail from outside the company. I also decided to fess up to my boss that it was my poor planning that caused this problem. My conscience wouldn't let me blame someone else.

So now I'm sitting here waiting and praying the DNS settings will take effect as soon as possible.

And here's a lesson I learned from another horror story: When logging off from a Terminal Service session on your company's only server, make sure 'Log Off' is selected instead of 'Restart'.

 

[SgtJarrow]

Okay...mine's a single user issue, but I will never forget it.

This user, who works in the billing review department, comes to me and says she wants a floppy and some instruction to download some files and work some from her home in the evening. I first get approval from management and then proceed to supply her a floppy and instructions. I go back to the IT room and don't think anymore about it. Next day she shows up and says the disk was empty when she got home. We checked the disk and it was blank. I proceed to show here agian, and of course the data is gone the next day.

This goes on for about four days....I have tried swapping out floppy, saving from another computer, etc. Every time we save the data to the floppy and she goes home, it is not there. I was stumped.

So on the fith day, I go through the process one more time. I hand her the floppy and begin to walk back to my desk. About halfway there, I remember that I did not verify the files were on the floppy today, so I head back to her desk. I ask her from the floppy and she reaches up to her file cabinet and pulls the floppy out from under the BIG magnet she was using to secure it there!

Needless to say I was not happy....

=======================================
People think it must be fun to be a super genius, but they don't realize how hard it is to put up with all the idiots in the world. (Calvin from Calvin And Hobbs)

Robert L. Johnson III
CCNA, CCDA, MCSA, CNA, Net+, A+, CHDP
Network Admin/Access Programmer

 

[mrdenny]

This story isn't mine; it's from a friend of mine.

I swear that this is true.

He got a new sales exec a while back, and got the guy a new laptop. He told the guy that it had wireless networking in it so he could use it any where in the office.

After about 3 or 4 days Mr I have an MBA I'm so smart sales exec (I've met the guy) comes to my buddy and says that his laptop is broken. My buddy goes and looks at the laptop and the battery is dead.

He asks the guy when the last time he plugged it in was, and Mr sales exec said that he should have to plug it in because it's wireless.

After smacking him self in the head with a heavy object repeatedly he had to explain that the network connection was wireless, he still had to plug in the power to charge the battery.


Denny

--Anything is possible. All it takes is a little research. (Me)

 

[fredericofonseca]

philote,

It can take a couple of days!! This means that you will be able to send emails (probably) but won't receive many of them.

Good luck

Regards

Frederico Fonseca
SysSoft Integrated Ltd

http://www.syssoft-int.com

 

[mrdenny]

I've got another one as well. This little gem happened to me at a pervious job.

An few hours before a new file was to be released on our digital distribution system our billing system started having issues and couldn't talk to any of the other servers it needed to on the network, sort of.

Instead of a quick fix of rebooting the server I wanted to fix the cause of the problem, instead of just rebooting the server. Silly me I didn't want to have this problem again the next week.

So after about 20 minutes I figure out that something is really wrong with the server. So I get on the phone with the folks at Microsoft. I explain that I'm affecting 22 million customers, and I'm about to have to delay the release of a highly expected file. Needless to say, I'm whisked away to second level support instantly. After about 2 hours on the phone uploading log files and dump files they tell me that the problem is with an extended stored procedure. For those of you who aren't DBAs it's basically a DLL that's been attached to SQL Server to perform additional tasks that Microsoft didn't think of.

So when I tell the developers that there dll is the problem they tell me that Microsoft is full of it, and there is nothing wrong with there DLL. This argument goes on for about an hour or so. At this point everyone is pissed because we're loosing money left and right as no billing requests can be processed. Did I mention that this all happened in 11am not in the middle of the night.

Then our head developer realizes that the version of the DLL that he installed over 2 years before I started working with the company was known to have code issues, and cause memory leaks, and the exact problems that we were seeing.

So needless to say after I strangle him for not speaking up sooner, and replacing the bad dll for a good version of the same dll we were back up and running 12 minutes before the file was to be released. Needless to say I checked every other SQL Server with the dll to see if they had the old one, and the only server that did was the billing system. Only the most important system on the network.

Needless to say the CEO was pissed and the head developer got a good yelling at. Never did get an apology from anyone for yelling at me during this whole mess.


Denny

--Anything is possible. All it takes is a little research. (Me)

 

[MasterRacker]

http://www.computerworld.com/departments/opinions/sharktank/

Jeff
The future is already here - it's just not widely distributed yet...

 

[Onyxpurr]

Hmmmmm...which one out of the fifty should I choose?

At my former company, they decided that the accounting software which we had been using for the past at least 5 years should be exchanged for a newer system. Because of a relationship between our president's wife and the vendor. So a new development team was put together between our company and the new vendor. Everything had to be written from scratch.

Regardless to say there were issues and misunderstandings, but finally they put together an implementation plan in which, thank god for our IT manager, our location was to be the last in line to install the new software.

First week, another location, we hear that someone received a 40-year paycheck instead of a 40-hour paycheck. (Thank god we caught him before he went to Mexico, lol) Anyway, in addition to the error, no code was created to be able to BACK OUT the paycheck. To this day, I'm not sure how they resolved it.

 

[philote]

I know, Frederico. At least its finally the weekend!

 

[chiph]

My first job out of college, they had installed a Novell server. But they didn't put it in an air conditioned room (it had AC during the work week, but by Sunday afternoon the temps in the room were over 85 degrees). After about 18 months, the server started needing a "kick" to start up. Literally.

The heads on the disk drive would stick to the platter, and the only way to get it to spin up was a physical blow to the box.

The manager told me to "just don't reboot it so often".

Chip H.


____________________________________________________________________
If you want to get the best response to a question, please read FAQ222-2244 first

 

[hilfy]

Going back to my DOS/Clipper programming days...

The company that I used to work for had a trainer who, in many ways, was the epitome of "dumb blonde" who thought that she could do support. One afternoon, she was working on a client machine, connected over a modem (pcAnywhere? Carbon Copy? I don't remember...) Anyways, she thought she was deleting one file, but she deleted ALL of the client's data files. The client didn't have a recent backup. I spent several hours using the old DOS Undelete command to try to recover the files. If this had been the first time I'd had to cover her a$$ I wouldn't have been nearly as upset...

-Dell

 

[webmigit]

I'm mostly a programmer, but I wing it pretty well as DBA when the need arises..

One company I worked for last year had two servers.. One was production... the other was a sort of development, they didn't know why they had it, why they were paying for it, but they were and since the CEO liked the owner of the hosting co where the dev server was, they just kept paying for an empty server. I put it to use.

After a while, we got the folks down there to help us with backups because the dedicated host we were at had no policy on backups and no intent to change that... So every two or three days, a programmer there would download a backup of the database.. It was his job because it was his T1 vs. my Cable and it was a big database.

Well the Tuesday before Thanksgiving of last year, he ran a backup... Friday after Thanksgiving the RAID controller fried on the production server and took the live stuff with it... I had a current backup of the code but nothing but his backup of the db... Well we noticed it was three days prior to the crash but put it back in place (What else were we supposed to do?) on a new server, now in the same place as the dev server...

It occurs to my employer at that point that their customer's sites (they were a specialized host of sorts) that orders placed two days after the backup are (duh) not there... And I kid you not, the customer service girl that I routinely worked with asked me with all serious-ness "Isn't there a way you could have predicted the orders that would come in and save the data in the backup?"

I was dumbfounded.

She was mad when I (of course) said no you can't get code to predict the future... she wanted credit card numbers and everything.

ALFII.com

http://www.alfii.com

---------------------
If this post answered or helped to answer your question, please reply with such so that forum members with a similar question will know to use this advice.

 

[webmigit]

A bit of humor, my customers are calling lately: "Your site says that I should change my password, what should I do?"

... Its very hard not run away from people like that ...

ALFII.com

http://www.alfii.com

---------------------
If this post answered or helped to answer your question, please reply with such so that forum members with a similar question will know to use this advice.

 

[jrbarnett]

A routine server disk space expansion (add an extra 2 HDDs to a RAID 5 array and extend the array over the new disks) as it was getting very full happenned to me at a former employer, just over 2 years ago:

Picture the scene, one Wednesday evening. 2 people were to stay behind to do the update. Everyone was kicked off at 6:00 and I said "I'll go home now, you don't need me, I'll be here 9:00 tomorrow in case anything goes wrong" not realising the significance of what I said.

I go home and come in in the morning, as I'm walking up to the office, someone collars me "The disk expansion yesterday went wrong. You're going to have a lovely day" - and I did.
Story from those who were there, about an hour later:
Both disks were inserted into the array together and the command issued to expand the array over the new disks (Novell).
One of the new disks was duff, and at the same time one of the existing 3 disks decided to die. Result: Data screwed -RAID 5 can handle one disk dying, but not 2.
Did we have a backup? From the day prior to the backup, Did we have onsite maintenance for the server, 4hr swap out on components? No.
Result: 500 very unhappy people who've lost a days work, everyone rushing around like headless chickens trying to do as much as possible to keep people happy. Additionally, the server was out for most of this day while repairs were initiated and backups restored.

The following day I was on a training course, then the weekend. Monday was still a bit mad but it had calmed down a lot by then.

Funny thing is, this is not the only time I've seemed to be psychic about a computer system failure.

John

 

[mrdenny]

From back in my Dial up tech support days (this was phone support). For those of you who havn't had the experience of handeling phone support with the general public it's a real treat, let me tell you.

Never fails at least once a week, you would tell some nice old lady (usually a nice old lady) to double click on the my computer icon, and you would head her tap the mouse twice against the glass of the monitor.

Or the time where you would tell someone to turn off the computer, wait 10 seconds (to let everything spin down) and turn it back on. Then 2 seconds later they say, everything is back again. You then spend 5 minutes screwing around before you realise that they turned off the monitor and not the PC.

Another favorite was when someone called tech support and said that there computer wasn't working. So we ran through is it turned on, is the monitor off, then asked them to check the power cables in the back to make sure they were plugged in. I swear the response from the customer was "Hang on a minute, let me get a flashlight, the powers out and it's dark back there." This tech had to spent the next 5-10 minutes explaining to the user that if the power was out there computer wasn't going to work.

I'm sure I've got some more gems as well. Mabey later.

Denny

--Anything is possible. All it takes is a little research. (Me)

 

[ecobb]

http://rinkworks.com/stupid/cs_calls.shtml

One of your stories is close to the bottom of the page...



Hope This Helps!

Ecobb
Beer Consumption Analyst

"My work is a game, a very serious game." - M.C. Escher

 

[mrdenny]

Yeah, it's an old story, that more than one tech has had to deal with.

Denny

--Anything is possible. All it takes is a little research. (Me)

 

[Dollie]

An ongoing nightmare for me.... Male technical support.

I called our e-mail provider yesterday with some issues and they told me to have "our e-mail admin guy" give them a call and they'd be happy to help him. Gritting my teeth, I told them "I *am* the admin guy, Sir. Would you like me to recommend a new ISP to my supervisor or would you like to handle the problem like a professional without getting your manager involved?"

Same company that got upset with me when I found their MX record was misspelled and the tech was furious that I had done a reverse DNS (he thought it was illegal to do that?) on the IP to figure out what our e-mail problem was.

Same company that talks down to me and tells me in a condescending tone "Don't worry, hun, we'll get this nasty problem taken care of", then tells my co-worker (male) all the technical details that I was looking for.

Different company a while back (web host) told me essentially the same thing, to have the "guy in charge" call them and they'd be happy to help, but couldn't do a thing for me.

Are women in IT really that rare, or are men really that rock-headed?

 

[SQLSister]

I got that treatment alot when i was young before I learned to adjust my speech patterns. Many women use a rising tone on the end of statements that is interpreted as a question. (It is a question as women are socialized to seek agreement from others whereas mean are not). Men won't take you seriously if you do this no matter how competent you are. I mention this because many women are totally unaware that they do this.

You might make sure you tell them your title when you call, this may help a few of them.

Some men unfortunately are just sexist no matter what. They will never take you seriously as long as you have a feminine voice. In this case I would call his boss and tell him I was going to move my account unless the technicians started behaving like professionals. Follow it up in writing. I have no problem whatsoever with treating these guys like the dirt that they are. Cutting humor works well, too.

IT is a male profession for the most part especially on the networking side. As my boss told me many years ago when I was an analyst, you have to be thick-skinned to stay in this business. Give back whatever guff you get, even raise the ante a little. (Nobody messes with me in a sexist way now because they know I will turn it around to make them look bad.) But don't let whatever anyone says bother you for more than a few seconds.

Questions about posting. See faq183-874