Share Your IT Horror Stories 6

[Mike555]

My first IT boss once said to me "No matter how difficult your job may seem at times, you always have it easier than someone else." For some reason those words stuck with me, and I remember them often whenever difficult situations arise at work.

In that spirit...

This is always a topic I like to bring up around fellow IT colleagues. Tell us about the worst situation (horror story) you've ever encountered while working with IT. Everyone has atleast one story to tell.

I'll go first:
I was once hired by a small company to upgrade their entire network from Windows NT to Windows 2003/XP. This company 20 employees but no IT staff in-house. They'd purchased 20 new PCs and a Win2k3 SBS Server, but it was all just sitting around waiting to all be installed. Management told me they wanted to get the server installed first, and then worry about the PCs later.

Among other things I needed to check leading up to the installation, I needed to ensure I knew the local administrator password for each PC. (In order to add an NT machine to a win2k3 domain, you need to log on locally and manually add it most of the time.) I was told by the manager that all local admin passwords were blank. I checked 5 computers to make sure that the passwords were blank, and they were, so I continued under the assumtion that all 20 local admin passwords were blank.

I dedicated an entire Saturday to perform the installation, and when that time came I showed up and began the installation. Everything went fine until I realized that besides the 5 workstations I had initially checked, the other 15 workstations required passwords to log on locally.

To make a long story short, no one knew what the passwords were, not even company management. We were also unable to contact the folks who originally installed the PCs. Because of this I had to take a week off of work at my regular job and stay on site to install all the new XP machines because the NT machines were useless. The one-day project turned into a week-long hell.

Please share your stories too!

--
Mike

 

[SQLSister]

Leslie, here's the link

http://forums.prospero.com/foxfirefly

It will seem odd to you that this is a site for a cancelled tv show. (I read somewhere that you like science fiction, if you haven't seen this one you should pick up the dvds.) But we have some of best political discussions around and we have had other people who weren't fans of Firefly join just for the political discussions. YOu can read if you sign on as a guest, but to post you need to create a member profile. Look for me there - I'm HLGEM on this board.

Questions about posting. See faq183-874

 

[lespaul]

Thanks for the link, I'll look for you there! And congratulations on TipMaster of the Week! You are correct, I do like SF/F. A star for paying attention!

Les

 

[SQLSister]

Until you said it I hadn't even noticed I was Tipmaster of the Week. A nice little Monday surprise.

Questions about posting. See faq183-874

 

[Maverickpoet]

Hiya all,

Back when I first started in IT, I worked on a desktop pc and server intergration dept. I worked with a tech that thought he knew it all. We were working on some hotswap servers and he tells me, hey I think this nic is bad. I tell him, "well yank it out and get a good one." He says, "okay!" and to my HORROR, he reaches into the running server and literally rips out the NIC. My jaw drops as I ask him what in the blue blazes he thinks he's doing! He says, "Well I heard you telling the other tech's they were all hotswap components. So you mean I can't pull a NIC from a live server?" Man, I laughed so damn hard.:-D

-Mav

 

[chelseatech]

Here is a good true story from many years ago.

Our top floor computer room had added a bunch of new hardware and was getting a little warm. So we got a contractor in to install some additional air conditioning units on the room.

He came in one afternoon and drilled some holes throught the ceiling to the roof where the external part of the unit was going to be installed. But it was getting late in the day, so he left to continue the job tomorrow.

Did I mention it was one of those old concrete buildings with a flat concrete roof area?

It rained overnight.

We came in the next morning with all the computers sitting in puddles of water. Water was still pouring in from the hole in the ceiling.

Did I also mention that the hole was directly above the new expensive machine that had been generating all the heat?

We carefuly turned off all the computers and lifed up the false floor (now covering a small lake) and stated drying the room out.

Thankfully no major damage done, but more by luck than good planning.

Editor and Publisher of Crystal Clear

http://www.chelseatech.co.nz/pubs.htm

 

[ecobb]

Hey, I forgot about this thread!

I've got a true story for ya. I used to work for a local ISP. We had a residential dailup service, plus high speed and wireless connections for businesses. We were also an application hosting center for the main offices of several major corporations. Anway, we had just finished putting in our remote datacenter in another part of town, and had set everything up to be redundant. The theory was that if the main office was to lose all connection to the outside world, everything would automatically switch over to the remote site and keep running. The customers would never know anything happened. (yeah, right!)

So, once everything was set up and in place, my boss decided to test it out. About 3:00 one afternoon in the middle of the week, he went into the server room and pulled the main plug. The MAIN plug. We immediately lost all connection to everything, he may as well have just cut the power to the entire building. Well, guess what...the redundent system didn't work. Everything crashed, everything died, and everyone screamed. About 350 banks in the southeast US went down, an insurance company (have you seen the duck?) went down, and many, many local businesses went down. His reaction was, "Huh, I guess everything isn't set up right." He plugged it back in and went about his merry way, not even concerned about what he had just done.

Needless to say, all of us poor saps on the helpdesk did not have a good afternoon...



Hope This Helps!

Ecobb
Beer Consumption Analyst

"My work is a game, a very serious game." - M.C. Escher

 

[willir]

I have been real busy for the past few months so I meeds this post.

I have quite a few, but there is one from a long time ago when technology was much less sophisticated. (I will have to be careful not to identify anyone)

I worked for a midsize company in a department that specialized in supporting towns and small municipalities - taxes, water, payroll. In these days, files had to be specified to a certain size to accommodate xx number of records. If you reached the max size xx, you either had to purge data, or resize the file and make it larger. This process was manual, and requried technical support. This provided me with my pay cheque since towns and small municipalities did not have technical employees.

We also worked in a very mixed environment from Unix to propriorty systems. The company provided one-stop shopping by selling the software, training and support.

Well a new mayor is elected to a town, and to cut costs, he fires all the "trained" staff leaving the administrative assistant in charge. (And AA is always in charge, but she was less than 25, perhaps less than 21.) She got a real bad rap -- she had to run payroll, A/P etc. I worked my butt trying to help her. I really felt for the unfair position she was put into. Typically, training for payroll took at least a week, and that was after you knew the concpets about payroll. I mention payroll because this is the toughest, and the most visible task -- employees don't get paid, you read about it in the paper the next day. But she also lacked training for GL, AP, etc.

Along comes property tax time. For those who do not pay property taxes, the way it works...
- An interim billing is run for the year based on last year taxes. Owners are expected to pay their interim bills so the city, town or what ever has cash to pay their bills and employees and look after infrastructure, etc.
- The city council finalizes their budget.
- The final tax billing is run where the tax rate or mill rate is calculated based on the approved budget.

Well, the tax billing is only run two to four times a year based on how the billing is partitioned - two to four payments. And of course, this poor girl was not trained on taxes.

She ran the billing, and mailed out the interim bills without calling for help. Things looked grat -- another hurdle cleared. She felt she was finally learning how to run the various tasks. I was proud of her.

Then, months later, the final billing is run. Everthing still looks good ... until the new treasurer realizes that the tax roll is short by significant coinage, to seven digits!

I was asked to investigate, and found the problem. A file had reached it's max size, and somebody had included an error statement for the INSERT / WRITE to continue with the update. Very bad programming -- never did find out who did this.

The other part of the problem, like with any financial update process, you have to check the registry or print out to ensure was to billed / paid, matched the epxected amount - this was not done.

I am sad to say that the girl was let go. She had fought tooth and nail to cope with the tremendously unfair situation. She had learned an awful lot during her months of pain. And the real cuplret, in my opinion, the guy that thought it was easy to cut costs by cutting the trained people got away scott free with his flawed decision making process.

I felt so bad for the girl, that I sent her a bottle of Grande Marnier.

 

[chiph]

Another story --

I was contracting to a bank in Tennessee, and one of their branches in the eastern part of the state had been built back when Daniel Boone first settled the area. At some point the building had had electricity added -- using the then high-tech cloth wrapped wire run through porcelain insulators.

Fast forward to the early 1990s, when I was involved in automating their branches. We showed up, plugged in about 12 computers (2 servers, 8 teller machines, and 2 customer service rep machines), plus monitors, laser printers, etc.

As soon as we turned on machine #9, we blew all the circuits in the building. No lights, no heat/air, and more importantly, no ATM or alarm system. We had to stay there all night until the branch manager showed up in the morning, and explain what happened.

We then left, feeling bad about it, but there wasn't anything we could do. I heard later that when the electrician came in, he freaked out and called the city inspector, who also freaked out when he saw the 19th century wiring. The branch was closed for about 10 days while they rewired it top to bottom, costing the bank a lot of money.

Chip H.


____________________________________________________________________
Click here to learn Ways to help with Tsunami Relief
If you want to get the best response to a question, please read FAQ222-2244 first

 

[GPOTony]

A friend of mine told me about when his IT Unit was being relocated to another town.

When they arrived to the new building, they had to lug the mainframe up a stairs. Then they realised that the doorway was too narrow. They had to remove the door and get someone with a sledge hammer to widen it by two foot.

 

[HarleyQuinn]

I work in a small team of software developers who create applications for use in a call centre environment.

About a year ago we released a new application for use by middle/senior management. Two of whom had to be physically shown to double click the application's icon on their desktop to load the program...

Harleyquinn

---------------------------------
For tsunami relief donations

 

[spamly]

I'm probably not the first person to complain about mice in the sub-floor. My dim building manager thought he'd take care of them right away by putting baited glue traps under several tiles. ("We ain’t got no money to waste on an exterminator!") Unfortunately, he forgot which tiles he put them under. There's nothing like walking into the server room and smelling rotting mice to really start your day out nicely.

After finally tracking down all the glue traps with the unfortunate dead mice, the smell went away.

Months later the mouse problem resurfaced. This time, learning from his mistake, the building manager marked every tile which he had a glue trap under. He couldn't be bothered with checking every day to see if there was a dead mouse, but when the smell hit again he knew what to do. Checking under marked floor tiles he found nothing- not even a glue trap. You see he had not attached the glue traps to anything so the mice just dragged them across the floor. It took us weeks to track down all the traps this time.

Nice guy, but dumber than a box of hair.

 

[flapeyre]

I once worked for a retail company (now out of business - imagine that!) with a sales manager who had a mania for printed reports. Never mind that his reports alone consumed nearly half a case (5 reams) of paper per week (and many of his reports were were DOUBLE SIDED - like he'd ever have the time to read it all) - he had to have this new one.

Anyway, once he asked for yet another report - this to be printed daily, a very detailed report of sales which duplicated information in the reports he was already getting. The difference? The totals were to be on the front page, not the last page. This, of course, means you have to do all the calcualtions to get the totals before anything gets printed, rather than simply keeping a running tally as you print. The resulting work file was so large that it overwhelmed the system the first night it ran, and at 10:00 AM the next morning, the stores could not use their point-of-sale system because the report was still figuring out the totals, and other vital nightly processes had not yet been run.

We killed the report job, the stores eventually could use their POS system, and we told the sales manger to look at the last sheet, where the totals were, on his current report from now on.

Me transmitte sursum, Caledoni!

 

[lespaul]

In an earlier post, I mentioned the difficulties in our move. Update: After we had been in the building for months it was "found out" that the fire extinguisher equipment in the server room was a normal water sprinkler system!! It has been modified to a foam extinguisher!

Leslie

 

[billbredbenner]

In my early years as a beginning computer operator for a
large state agency (not mentioning the state, of course),
I was cleaning out old,sometimes empty files/directories on
a PR1ME minicomputer that just happened to be our principal
production machine. One thing you have to know about PRIMOS
(the o/s) is that when you create a new disk partition
(mfd, or Master File Directory), it creates 3 empty sub-directories (ufd's, or User File Directories),whether or
not they are going to be used by the o/s.These are CMDNC0,
DOS (not M$DOS) and one other that escapes me right now.They
are only used by the o/s on the very first partition, on the very
first disk, on the very first controller card, etc. to hold
o/s commands that were essential, but external to the
kernal. (Remember, these were the DARK AGES of computerdom,
the 1980's ). Well, along comes lil' ol' me with my then
limited knowledge of commands and their repercussions and I
proceed to type: delete CMDNC0 -no_query -no_verify -force
at the system console, not realizing (until that VERY sickening feeling hit my stomach, "Gee, why doesn't
<ctrl> C stop the process???!!!") that I was in fact wiping
the o/s (WITHOUT a backup no less!!). Needless to say, my
boss, a VERY understanding man, was not very pleased. Years
later we were able to have a good laugh about it, especially after he was able to schmooze a new copy of the
o/s out of the vendor.Sorry for the lengthy post.

Bill

 

[chiph]

After we had been in the building for months it was "found out" that the fire extinguisher equipment in the server room was a normal water sprinkler system!!

Hearing this reminded me of a friend's story. He was a mainframe programmer working at a large firm that was upgrading their Amdahl to an IBM 3090. He was looking over the blueprints for the new "glass house", and asked "Where are you going to put the water pumps"?

The admins had no idea that the 3090 was water-cooled, and hadn't done any planning for the piping, the pumps, nor the heat exchangers. They were upset with him at first (for showing them up), but later came to realize he'd saved them a lot of money & embarassment.

Chip H.


____________________________________________________________________
Click here to learn Ways to help with Tsunami Relief
If you want to get the best response to a question, please read FAQ222-2244 first

 

[KenCunningham]

Bill's earlier post about the PRIME reminds me of the days when I used to work on them and managed to delete the whole of the organisation's personnel system. Fortunately, we had a good backup regime and I spent a pleasant evening swapping reel-to-reels to restore it. Taught me once and for all to always verify where you are before hitting the return on a delete/rm or whatever.

 

[Tarwn]

Funny, I could have sworn I found this link on this forum (seems to go so well), but apprently not:

http://members.optusnet.com.au/white_gold/

-T

 

[Stella740pl]

Tarwn, you did, here: thread655-960691.

 

[billbredbenner]

KenCunningham,

AMEN !!

(p.s. we NOW have one of the most comprehensive backup
programs of any IT office in our state. We back up BEFORE
we run production jobs and AFTER (8mm tapes are CHEAP
insurance.))

Bill

 

[willir]

Bill

I used to work with 8mm tapes, a whole closet full of them. We found that they did fail occasionally. Another disaster in the works??

We moved to DLT tapes which have a lower failure rate.

Which brings me to another story from a past life...

As part of the package, we provided the customer with a menu driven set of Unix admin tools. The backups included 1) System backup (no data), 2) Databackup only, 3) Full backup.

The full system backups would take more than one (8 mm) tape, so customers were advised to perform the database backups. (And yes, training on the differences was provided)

Well, at one site, the database backup started using more than one tape. ...So the person responsible for the backups switched to "System backup (no data)" without discussing it with support (us).

...Then I get a support call for a serious payroll issue. Things had really been messed up so... "Do you have a backup from last night" ... "Yes" ... "Okay, let's do a restore"... Well it took a while but I then realized that the backup tape did not include the database files. Although I was able to implement a work-around to get the pay cheques "out the door", it took weeks of hard work to correct the data issues.

From the "school of hard knocks", I have always been believer in testing your backups.