Oracle RAC and AIX adapters configuration

[MoreFeo]

Hi, we've got a problem with Oracle RAC on AIX.
I've posted this issue in the Oracle 10g forum (see thread1177-1469271), but with no answers, so we're thinking about seeking for a solution from AIX.

The situation is: Oracle RAC on two IBM Blades JS21 with AIX 5.3, both blades in the same chassis.

It seems that it is an Oracle requirement that the adapters names for public and private networks must be the same on each node. This means that if private IP is on adapter en0 on one node, it also must be on adapter en0 in the other node, and the same for public IP.

The problem is that both adapters en0 are on the same switch on both blades (and the same for en1), due to internal chassis network, and having both public IPs on the same switch is a single point of failure.

We've asked Oracle, and they've told us we cannot mismatch the adapters configuration, so we're thinking about a way of swapping the adapters names in one node.

Is there a way to change en0 to en1 and en1 to en0 from AIX in a blade JS21? We were thinking about rmdev both adapters, then running cfgmgr on the second, but we haven't seen the way to do this.

Thanks.

 

[khalidaaa]

I guess it should be rmdev -dl entX or you should go up for the parent device and use the R option to remove the ethernet from the parent!

I usually use odmget -q "name=entX" CuDv to locate the parent and then use the parent as the name attribute till i get into the root parent of them and finally use rmdev -Rdl devicename

after than cfgmgr

But i'm not sure whether this will give you the sequence you need! unless you physically remove the ethernet adapter that you want to be assigned the ent1 and cfgmgr while there is only one adapter (the one you want to assign ent0 for) and then you can add the adapter back with another cfgmgr.

The way to do it or whether this is doable or not (ie removing adapters) that i'm not sure about! but logically i guess that's how it should be done!

Regards,
Khalid

 

[MoreFeo]

The problem is that, as far as I know, in JS21 blades there is an integrated dual port ethernet, so I cannot physically remove them, nor I can rmdev only one adapter because removing it's parent would also remove the other port.

I was thinking about rmdev both adapters, create a dummy en0, cfgmgr and see if it creates en1 and en2, rmdev dummy en0 and en2, cfgmgr again and see if the second adapter picks up en0 as name. The problem with this is that in JS21 blades there is no serial console, the way to connect is thru Serial Over LAN, so I think I cannot rmdev both adapters at the same time.

Thanks anyway.

 

[khalidaaa]

I think the logic of what you are saying sound perfect! have you tried that?

REgards,
Khalid

 

[MoreFeo]

I will try this next time I go to this client, but I'm not sure it will work because of serial over lan.

I will tell you when I try.

 

[sbix]

This should work:
rmdev -dl en0
rmdev -l en1 (the missing "d" is not an error)
cfgmgr
rmdev -dl en1
cfgmgr

 

[MoreFeo]

sbix, when running the first cfgmgr, won't the old adapter en0 take again name en0?
I mean, there's no more en0 defined, so the first adapter detected by cfgmgr (the previous en0) will pick up again name en0, I think.

I need en0 to become en1 and en1 to become en0.

 

[p5wizard]

Only trick I know when both adapters have the same parent device is to *mess* around with odmget, rmdev, odmadd, cfgmgr

[tt]
step 1
------
logout

login via en1 address

for db in CuAt CuDv CuDep CuDvDr CuVPD
do
odmget ${db}|grep -Ewp 'en0|et0|ent0'
done > /tmp/odmadd.ent2

vi /tmp/odmadd.ent2
(replace all ent0 strings with ent2, en0 with en2 and et0 with et2, save and quit)

rmdev -dl en0
rmdev -dl et0
rmdev -dl ent0

odmadd </tmp/odmadd.ent2

You now have ent2/en2/et2 instead of ent0/en0/et0

define TCPIP stack on en2

step 2
------
logout

login via en2

rmdev -dl en1
rmdev -dl et1
rmdev -dl ent1

cfgmgr

configures ent0/en0/et0 (ex ent1/en1/et1)

define TCPIP stack on en0

step 3
------
logout

login via en0

rmdev -dl en2
rmdev -dl et2
rmdev -dl ent2

cfgmgr

configures ent1/en1/et1 (ex ent2/en2/et2 from step 1)

define TCPIP stack on en1
[/tt]

No warranty! Use at your own risk. But in any case, the manual change in ODM is only necessary during steps 1 and 2.


HTH,

p5wizard

 

[khalidaaa]

Nice but scary

 

[MoreFeo]

Well, playing with the ODM is a thing I knew could work, but it's a thing we don't want to try, not for now. I think we're going to wait and see if Oracle gives us a way to configure it on different adapters. If not, the client thinks he can risk having both public IPs on same switch and have a backup switch (already configured but not plugged).

 

[p5wizard]

Well I wouldn't do that kind of stuff either on a machine that's already in production, but if you can risk having to reinstall on a spare blade or if you still ned to roll out, why not.



HTH,

p5wizard

 

[DukeSSD]

Things to consider:

If you need a hack to make it work is it supported by either IBM or Oracle?

Is Oracle RAC supported on blades within the same blade centre, if so then how?

If you need to recover from mksysb then do you have to re-apply the hack, if so can you ensure it is documented in your recovery procedure?

It is all very well getting stuff like this to work but what happens when it all goes wrong while you are on holiday?

 

[sbix]

MoreFeo you're right.... obviously the sequence will be:
rmdev -l en0
cfgmgr
(en0 will became enx)
rmdev -dl en1
rmdev -dl en0
cfgmgr
(en1 will became en0)
rmdev -dl enx
cfgmgr
(enx will became en1)

 

[MoreFeo]

Thank you all for your comments, but I don't we'll finally do the hack.

Thanks anyway.

 

[DukeSSD]

MoreFo and sbix, you are wrong, this will not work.
rmdev -l en0
cfgmgr
en0 will became en0 again because the device characteristics are the same as they were.
You cannot do this without a bit of direct ODM manipulation.
HTH

 

[p5wizard]

You cannot do this without a bit of direct ODM manipulation.

At least not on a blade. If you have a machine with an unused PCI slot that you can temporarily put a spare ethernet card in, it is possible with rmdev/cfgmgr combinations if you know the parent device of that PCI slot...


HTH,

p5wizard

 

[DukeSSD]

Agreed p5.