going into panic/crisis mode 6

[hinesward]

I seem to have a major problem with going into what I would call "crisis mode" or "panic mode." I seem to see even something like a server crash as the end of the world. I spend a lot of time second-guessing myself.

The biggest problem is that I lose access to my own resources when I do this. Thus, I end up making little mistakes and become less cautious. I need to be more patient and more relaxed in this profession.

I remember I once asked a doctor: "How do doctors handle the pressure?" I often like that people depend on me and like being important. Certainly we don't directly deal with life-and-death issues like doctors do. However, I often find that I treat everything as if it is life and death.

I think that one of the hardest things about this work is that IT workers are forced to take a Murphy's Law approach to things. You have to think about all the things that could go wrong and take measures to prevent them. You have to test and retest everything. You can't assume anything.

In over ten years, it seems like I have worked under two types of supervisors: 1. ones that leave me alone and are very accepting. 2.ones that over-supervise, micromanage, and blame me for things that aren't my fault. I would love to work under someone who is in the middle of these extremes.

 

[edfair]

In 48 years I've seen them all, and for part of it I've been some of them.

The best, IMHO, was one who would listen when I raised questions, offer suggestions if he had something to add, admit that he couldn't help if he couldn't, and trust me to do the right thing if we were both charting new paths.

The bad, IMHO, was one who had answers in spite of the questions being different.

And the worst, IMHO, was one who had answers before he had the questions.

I've had several worsts and have had some pleasure in watching then self destruct, by being fired, re-assigned, or taking their company down the tubes.

And I've had two of the bests. Both insisted that I could be better and enabled me to do it. And they are the bosses that I tried to emulate when I had the power.



Ed Fair
Give the wrong symptoms, get the wrong solutions.

 

[SQLSister]

The first rule for avoiding panic mode is to be prepared. Practice recoving from a server crash or restoring a database or whatever. Practice it over and over and over until you can do it in your sleep. Then when the real thing happens, your ingrained memory of how to handle will take over.

If you are that prepared (very few people really are) and still have the panic attacks when something goes wrong, then move to step 2. Stop take ten deep breaths, then go into fix mode.

If that doesn't work, you may be having panic attacks that are biochemical in nature and need medication to help you. There is also a product called Rescue Remedy that has helped me deal with panic attacks when there was no real need to panic and I wan't on the prescription panic attack meds.

Visualization is another technique that can help panic attacks when they are not justified. Visualize the emergency and visualize your self handling it calmly and professionally and successfully. Again this is something you need to do over and over and over again over the course of several months (do it one or more times a day). You are re-programming your brain and that takes time and a whole lot of effort but it can be done.


"NOTHING is more important in a database than integrity." ESquared

 

[hinesward]

Certainly doing drills is a good idea. The problem is there is no time.

 

[SQLSister]

Of course there is time, you just have to get management to agree it is a high priority. There is always time for the high priority items.

If you do the drills, the likelihood is that fixing problems will take less time than it currently does. You also will identify possible sources of further problems while testing that are much better found then than when it is an actual emergency. Then you can fix the process before you need to rely on it.

All this is what you have to sell your employer as to the reason to do drills. Schedule them and set aside a half a day a week to do this and I'll bet in less than a month you will find that the time is easier to come by because of improved efficiency. Even if you stay later for a couple of weeks to do the drills on your own time, I bet you will find that you are more confident when something goes wrong and can fix it easier. Once you have a track record of showing how much the drills help, it is easier to convince management to spend the time.

If all you do now is fight fires, I'll bet you can easily come up with several occasions where having practiced how to do something would have made the problem less likely to happen or take less time to fix when it did happen. Remind them of how awful it was when (fill in the blank) happened and that you want to be proactive about planning for future problems so that the next time the server fails (or whatever) it can be fixed as quickly as possible with the least impact on the day-to-day operations.

What kind of fires do you fight most often now or which are most critical to the operation of the business? Start with drilling how to fix those.

Having a server down for several hours can cost a company a lot of money in lost time and lost orders. Learning how to fix something in the least amount of time (because servers will die sooner or later) is an investment that pays for itself with just one failure.

It's hard to get out of fight fire mode and into prevention mode. But if what you are doing now isn't working for you (and if you are so stressed you are panicing, then things aren't working well), changing how you do things is the only real solution no matter how hard it is to do.

"NOTHING is more important in a database than integrity." ESquared

 

[gbaughma]

I fortunately don't suffer from "panic mode" very often.

I manage to work the problem and have my breakdown afterwards. lol

The only down-side to this is that I can get *very* short tempered if I'm dealing with a crisis (such as a network outage, crashed server, etc.) and if someone interrupts my thought process, I tend to bark at them.

Then I apologize to them.

So, mainly, I've just gotten in the habit of explaining this to people beforehand, and now if there's something that I'm working on fixing, they know to leave me alone and ask me afterwards what happened.



Just my 2¢

"What the captain doesn't realize is that we've secretly replaced his Dilithium Crystals with new Folger's Crystals."

--Greg

http://parallel.tzo.com

 

[hinesward]

Here is an example of me going into "panic mode." I heard a clock ticking in an office. It actually sounded like a hard drive clicking. I thought someone's hard drive had died.

What do I about this?

 

[gbaughma]

hinesward:

Take a vacation. lol

"Out of the ordinary" sounds mess with me, too. I've woke up from a dead sleep in the middle of the night because my aquarium filter started making weird noises.

Keep in mind though.... if someone's hard drive was ticking like that, the user probably would have come to you and said "My computer's making a weird noise....."



Just my 2¢

"What the captain doesn't realize is that we've secretly replaced his Dilithium Crystals with new Folger's Crystals."

--Greg

http://parallel.tzo.com

 

[rosieb]

hinesward [b\] Relax, take a deep breath, and check your backup!

You have, of course, tested your restore routine! And have a full restore procedure fully documented!

Oh, and I fully sympathise - I've just discovered I've to upgrade a portion of a system, with no test system, just suck it and see. Then duck if it all goes horribly wrong!







Rosie
"It doesn't matter how beautiful your theory is, it doesn't matter how smart you are. If it doesn't agree with experiment, it's wrong." Richard Feynman

 

[edfair]

Then there is the time when the backup routine is fully tested and the spare machine is loaded with everything except current data and the 15 minute upload and restore routine goes toes up when the spare machine operating system drive tanks and the process becomes 8 hours of starting over from scratch trying to find parts that work together.

This on the busiest day of the week in a vet clinic with no manual backup in place in spite of my pleading for such over 8 years.

It wasn't my panic.

There is one major difference between techs and doctors:

techs have to take callbacks when the patient expires.

Ed Fair
Give the wrong symptoms, get the wrong solutions.

 

[kjv1611]

[LOL] @ edfair:
==>There is one major difference between techs and doctors:

techs have to take callbacks when the patient expires.

--

"If to err is human, then I must be some kind of human!" -Me

 

[gbaughma]

I've said it before.

I'll say it again.

It's never a question of IF the system fails.

It's WHEN.

Backup... then backup again... then store your backups offsite... and if it's critical, redundancy redundancy redundancy.... did I mention redundancy?





Just my 2¢

"What the captain doesn't realize is that we've secretly replaced his Dilithium Crystals with new Folger's Crystals."

--Greg

http://parallel.tzo.com

 

[edfair]

Yeah, Greg, but no matter how well prepared you are something can always go wrong, at the worst possible moment.

The day you reach quadruple redunduncy is the day you find that the binder in the tape was faulty and your bits from the last month are piles of rust in the tape carriers.

This happened at one of the presidential libraries. Stuff had been stored on tape at the white house. They needed to pull some archived stuff off and found that nothing was left. The one I was servicing proceeded to make duplicates of everything they had stored on the same type of media. Fortunately, they didn't find any bad tape.

Ed Fair
Give the wrong symptoms, get the wrong solutions.

 

[DonQuichote]

Dear hinesward,

Sorry for the late reaction. From the posts, it looks like you are a system administrator or may even a "one man it department" guy. For programmers, I would recommend things like version control and unit testing. They are ways of being prepared.

First, you are not a magician. Neither is a doctor. A doctor cannot keep every patient healthy and cannot even keep every patient alive.

If a hard disk fails, it is not your task to detect it the very same second. In fact, you have a great advantage over the doctor: you can just hand out a spare patient while investigating the current one. Crashing memory, for instance, causes vague pains that are not easily identified. Just tell the user to switch to another PC in such a case and see if it helps.

About time pressure. You can write books about it. If there is no time to "be prepared", there never will be time. But who says there is no time? you or your boss?

I once applied for a programming job. When the boss-to-be saw my resume and the fact that it mentioned hobbies, he spoke his worries about it. When software was released, often hell broke loose. Me having hobbies meant that I could not sacrifice my weekend to fix them. I explained about test environments and unit tests and that "after release" was too late to fix anything, as expensive company data could already be damaged beyond repair.
His only reaction was "you don't have time to do these tests".
He clearly saw me as a fixing slave instead of as a professional. Off course doing those tests SAVES time and increases quality, which saves more time again. So it is up to you, do you want to be a fixing slave or a professional?

For systems, there are monitoring systems. Imagine you get a signal that there is only 10% free temp space on a system. You then have time to investigate, act (cleanup) and take precautions (like asking for a bigger drive). But if you have to react to the fact that no application can create a temp file anymore, the only thing that remains is panic / crisis.

If it is your boss who wants you to run blindly, act there. Talk to him, enlighten him, discuss options with him or if all else fails find a job where you are treated as a professional. But first show you are a professional. That you are not just a running slave but someone who knows how organise things to take the pressure off the situation. Because the situation is tight and I don't think you could handle a major problem if minor already scare you.
This may have a positive side: you need management now. For monitoring systems, for planning, for a colleague or for whatever. If you give the boss the option to really manage something, he might be too distracted to micro-manage you. Micro-management could just mean that your boos has nothing better to do. So give him something.

This situation has probably grown (because the company has grown for instance). So intervene: talk with your boss.

Good luck!


+++ Despite being wrong in every important aspect, that is a very good analogy +++
Hex (in Darwin's Watch)

 

[gbaughma]

DonQuichote:

Great advice. Have a *.

The job that I just took back in February has been, to say the least, a challenge. However, I love challenges. As long as the boss is aware that I'm there to fix things. I'll give you a few examples.

Backups: Pretty much non-existant. They were running on IOmega REV Drives. However, since the REV only backs up 70GB, and one of the servers was larger than that, and the other two REV drives were malfunctioning, there were no backups.

"Main" file server: Running on a PC. With an overheating power supply.

Network: Two buildings, a wireless link in between them, but on the same subnet. All traffic was going back and forth, making the network lag so much as to be pretty much unusable.

Web Server: Outsourced, outdated. One of my projects was to get our deliverable data online.

So......

I split the subnet between the two buildings and dropped in a 10MBps internet connection (they had a 3 MB connection on shared wireless which would die if it rained, let alone snowed)

I ordered a tape autoloader with 8 Ultrium 4 carts

I ordered a new file server with 3 x 1TB drives, in RAID 5 to give 2 TB raided storage and dual power supplies

I ordered a web server with 3x 500GB drives, giving me 1TB of Raid 5.

I developed their entire web system in about a month, including the ability for customers to log on and get their data.

Now, I'm back to "Project mode"; in other words, projects from the bosses like "Some of our customers would like a report in this format" and "New lab equipment coming in, we need import routines written"... stuff like that.

The upgrades and changes that I made were painful, to say the least... a routing issue was keeping the buildings from talking properly for a day until I figured it out, and our Metaframe connections were dropping until I realized that some routers have a "hard time limit" on TCP connections (UGH). That didn't include the cost of about $18,000 that I spent on servers, software, and backup systems; but it was necessary to keep the mission-critical systems for the company working.

But, now the network is running smoothly, backups are running daily, and life in general is good.

I love what I do; it can be downright frustrating at times, but I honestly love what I do.


Just my 2¢

"What the captain doesn't realize is that we've secretly replaced his Dilithium Crystals with new Folger's Crystals."

--Greg

http://parallel.tzo.com

 

[SteveTheGeek]

I'm not sure where I first heard it, but "do you want me to wrestle alligators, or drain the swamp?" can be an interesting and thought-provoking question for your boss.

@EdFair: As others have said, there's two elements of being prepared to deal with technical issues - a structure for fault tolerance, redundancy, and disaster recovery (backups, failovers, automatic notification of impending issues, etc); and an adequate level of experience and familiarity with whatever recovery is necessary. If both of those are adequately in place, you might take a look at ways to deal with panic attack type symptoms, because aside from a mental "uhoh" (or more unprintable four-letter word) there's no real cause for alarm.