VIO and AIX 5.3 1

[hfaix]

Does anyone have AIX 5.3 running stable on vscsi from 2 VIO Servers? By this, I mean two VIO's serving up VSCSI and SEA's to Client Paritions? I'm trying to do it with ESS800 Lun's (Both VIO's see the same LUN). I've got it working in a test environment, but my vscsi0 (from vio1) and vscsi1 (from vio2) on client LPAR's never seem to recover well from simulated failures. Some times my client paritions won't boot unless I shutdown a VIO server first.

It seems like I have had support on speed dial from these new "virtualized" technologies.

I'm just curious to see if anyone is tinkering with this.

 

[dvsweden]

Hi,

I have not tried running two VIO servers sorry. I've spent enought time working on just one!

I am curious if you have a good sense of scaling the processing units and the virtual processors. We have a p5 550 w/ 4 procs and I have 5 lpars up and running but I don't have a good sense for the sanity of my processor settings. I still don't think I quite understand just how the virtual processors are supposed to work...

I've been on the phone with IBM ALOT as well ;0)

Cheers,
David

 

[hfaix]

I agree, I don't really have that great of a feel about virtual processors. I never get the same answer from anyone. I've worked with IBM and with my IBM business partners and everybody differs.

I've been instructed (by support) that the virtual processors really depend on the workload of the application running on the machine. If the machine can MultiThread, then I would say two virtual processors would be the minimum. I typically match my vp's to my physical cpu's (in tenths). If I'm running and LPAR with a .1 min, .2 des, .5 max, then I'll have my vp's set to 1min, 2des, 5max.

I wish i could help more. I'm far and away no expert of virtual processors.

Support hasn't been too good lately (with any of my problems). This stuff is a hair too new.

Are you running your p550 micro-partitioned in prod?

 

[hfaix]

Oh.... and welcome to the board!

 

[dvsweden]

Thanks! I was frustrated trying to find any IBM forums and found this site.

We are not running in production yet, but our timeline is getting short. Lucky for me the vio environment is going to be a test area so we are not as worried about availability (hence no 2 vio servers...)

Have you run into the dynamic lpars being very sensitive to the network? It seems you need a dns (reverse and forward) entry for the dlpar to work correctly. Also, have you done any performance testing at all? Any sense of how the vio environement will work when pushed?

We will have an Oracle based ERP (Banner) so it we will end up with a db server 2 app servers 1 nim server and the vio server for our lpars.
Cheers,
David

 

[axelsche]

Hi David,

yes dlpar is very "Name resolution" sensitive.

We haven't done a lot of performance testing yet, but what we have seen is that it is a good idea to use the uncapped option for processors from the Processor pool. (if you dont't use dedicated procs for VIO).
But of course we don't use it for productive systems right now. For productive use we have to wait for the support of virtual FC disks and HACMP Support for VIO Client LPARS.

Both is not supported yet.(as far as i know)

Axel

 

[hfaix]

Axel -
That's an interesting thought about the support. I've got mine running in a testing phases as well, however, I'm using virtual FC disk. I've called support atleast twice on disk problems, and they've never mentioned anything about my setup as unsupported. I run my vio's boot from san, and serve up vscsi FC disk to LPARs. Seems to work good from one vio. Seems to suck from two VIO's. Maybe I should check on the support stuff I didn't bother to ask.

David -
I haven't had any problems with DNS, however, I always ask for reverse lookup also.

Performance has been pretty solid. I have tested it too much, but nothing seems notibly slow. I've become partial to "lparstat" .

 

[axelsche]

hfaix,

do you use dedicated procs for VIO Server?

Axel

 

[dvsweden]

I am assuming FC is fiber channel. If that is the case, we are also experimenting with fiber channel. VIO seems to work just fine and I talked to IBM today and they told me that if the san is supported on 5.3 then it is supported on vio. Take that with a grain of salt though... The one bummer so far is that as far as we can tell 5.3 won't work with dual host bus adapters for mpio at least with our san (an hp eva3000)

As far as dedicated procs to the vio goes, initially I did have it set up that way but our boss felt it was not needed so rather than dedicate the processor I have 1 desired listed in the shared mode. I guess the difference is that potentially the hypervisor could pull proc from the vio down to it's minimum. I'm not sure my explanation is quite right, still learning this beast! We'll see how it goes...
Cheers,
David

 

[hfaix]

No, I don't use dedicated CPU's for my VIO Server(s). It's pooled, however, they have the highest weight of all the pooled LPAR's. I'm planning on pooling until utilization tells me I need dedicated cpu. Performance hasn't been a problem ( but I haven't pushed them much either)

David, I do have dual HBA's up and going on 5.3, however, I'm using ESS. I even had dual vscsi (via dual hba's) from dual VIO's up for awhile. The vscsi path's weren't failing over very well as I took my vio's up and down, so I scrapped that until IBM get's the kinks worked out.

 

[axelsche]

Hi,

I have found something about VIO and HA Setup.
Perhaps this is the reason why MPIO is not working on your client Partition.

Here the IBM Text:

snip...
If the reserve_policy value is anything other than no_reserve, it must be changed in order to use the device in an MPIO configuration on the client logical partition. To set the attribute properly, use the following command:
chdev -l diskdevicename -a reserve_policy=no_reserve
Failover is the default behavior for MPIO Virtual SCSI disks on the client logical partition. Configuring multipath Virtual SCSI devices on the client logical partition protects the partition against failure of one of the following:
· One physical adapter, cable, or port in one Virtual I/O Server
· One Virtual I/O Server
snip..

I cant test it right now (we have HDS Storage), because I have not enough disks to setup a second VIO Server...
Mhhhm... perhaps I should give it a try with rootvg on FC Disks...

HTH
Axel

 

[dvsweden]

Interesting. Thanks for the ideas. Our vendor is working with us on some driver issue which may help as well (3rd party drivers not ibm)

On a different topic, are you all using partion load manager or just relying on the hypervisor? Another point of confusion for me, it seems they do similar things but in a different manner, the PLM is policy based and the hypervisor doles out base on cap weight. Right?

Do you know if there are any other tools beside lparstat that shows how the procs (virtual and/or physical) are being used between the lpars? The CEC properties does not seem to be terribly helpful.
Cheers,
David

 

[hfaix]

Correct, the reserve policy should be set to no_reserve to get full capability out of MPIO. This is a root command, not a padmin command.

I've been doing this and it seems to work pretty well. I have noticed that even though I can hit one physical LUN from two VIO's, if I serve up vscsi from these vio's (two get two different paths to one disk), then pull the HBA's from one side, sometimes it won't find both paths on a reboot of the CLPAR. Sometimes it won't find either path I few times when testing this out, I had to shutdown one VIO, to get the CLPAR back up.

That's why I've rebuilt my environment with one VIO for now. I am setting the reserve policy now though, just incase I want to use a second vio sever later. I'll wait for IBM to get their act together before I try to take full advantage of this.

 

[alfapet]

Hello everybody!

hfaix, you wrote::

"I run my vio's boot from san, and serve up vscsi FC disk to LPARs. Seems to work good from one vio.".

Do your LPARs boot from SAN?

I have problems with the VIO-server serving SAN-disks as physical volumes used as rootvgs in the LPARs. The VIO-server itself has its rootvg mirrored on two local disks. I have set up a "System Profile" in the HMC that is supposed to start the VIO-server first and then the LPARs in order of importance.

When I activate the system profile the VIO-server starts booting, but before it has finished the other LPARs start booting as well. The result is that the VIO-server boots good, but the LPARs stops at the SMS-menu. When I restart the LPARs using the HMC, they boot well.

My theory is that the LPARs start booting too soon after the VIO-server boot. What do you guys think? Any solution?

 

[hfaix]

Yes, I boot VIO's and Client LPARS from SAN ( ESS 800 )

I honestly haven't messed with the partition auto-boot stuff. I don't have a system profile setup. I prefer booting my p5 to stand-bye, then booting my VIO(s), then booting my CLPARS. Exactly for the reason you mentioned. I wondered about that.

Wish I could help, but I haven't played with that.

 

[alfapet]

Ok. Imagine that the HMC fails (the internal disk or something else), and during the time it is repaired/replaced the electricity fails. In this case the eServer has be able to reboot without HMC interaction. If the default start-order of the LPARs doesn't work, this would be a problem, and the partition would not be accessible.

Have you thought of any solution?

Best regards,
Nils

 

[axelsche]

Hi Nils,

you are right. That would be a desaster with one HMC but
that's the reason why every p5 Server can connect to two HMC's...

I just got another information from my IBM Rep. VIO with FC and HACMP will officialy be supported not before mid of this year (more likely end of this year)

Axel

 

[hfaix]

I have my "distaster" HMC at an off site about 2 miles away. In the case of Power Failure, I plan to use it. That's a good thought though....

Axel, thanks for the post on the VIO with FC and HACMP, luckily, I don't have an HA environment. I do have backup servers, however, they will need to be "quickly" built and brought up to speed. They are usually test machines.

 

[axelsche]

You lucky guy...

 

[alfapet]

hfaix and axel,

What you are saying is that the availability is very low having only one HMC. There must be some way to work around the problematic scenario I described above (Feb 25, 2005)?! Maybe coming code releases will enable to set a default start-order for the partitions used when the p5 reboots. Until then I think it is necessary to have two HMCs for production enviroments!! :-(