Where is my dac? 2

[khalidaaa]

Hi Guros,

First of all i would like to say that i'm confused about these terms, dar and dac! I know that one of them is poing to our SAN controller! i'm not sure thought what do they mean!

I've recently added a new fiber channel to one of the clusters then run cfgmgr on the LPARs and i can see the fiber for now added:

# lsdev -C | grep fcs
fcs0        Available 03-08        FC Adapter
fcs1        Available 09-08        FC Adapter

The cluster used to have only one FC so by adding this i was trying to eliminate one of the single point of failure in the cluster.

But now i don't think that my newly added FC is being used by the SAN disks! here is what i did to discover that:

# lspv
hdisk0          00c5c1ebd9337eb2                    rootvg          active
hdisk1          00cf359f352e8582                    oravg           concurrent
hdisk2          00c5c1eb0ef90cd8                    oravg           concurrent
hdisk3          00c5c1ebcc9ededb                    oravg           concurrent

# lsdev -C -F parent -l hdisk2
dar0

# lsdev -C -F parent -l hdisk3
dar0

# lsdev -C | grep dar
dar0        Available              1742-900 (900) Disk Array Router
en0         Available 04-08        Standard Ethernet Network Interface
en1         Available 05-08        Standard Ethernet Network Interface

# lsdev -C | grep dac
dac0        Available 03-08-02     1742-900 (900) Disk Array Controller
dac1        Available 03-08-02     1742-900 (900) Disk Array Controller

# lsdev -C -F parent -l dac0
fscsi0

# lsdev -C -F parent -l dac1
fscsi0

# lsdev -C -F parent -l fscsi0
fcs0

# lsdev -C -F parent -l fscsi1
fcs1

So i guess my SAN hdisks are linked to fcs0 thru fscsi0 on dac0 only.

When i check my other clusters i get this:

# lsdev -C | grep fcs
fcs0        Available 07-08         FC Adapter
fcs1        Available 08-08         FC Adapter

# lspv
hdisk0          00c5c1ebd970edea                    rootvg          active
hdisk1          00c5c1ebe91077b1                    rootvg          active
hdisk2          00c5c1eb37584a50                    oravg           concurrent
hdisk3          00c5c1eb37584b90                    oravg           concurrent
hdisk4          00c5c1eb37584cd2                    oravg           concurrent
hdisk5          00c5c1eb0df2cddf                    oravg           concurrent
hdisk6          00c5c1eb525594d2                    oravg           concurrent

# lsdev -C -F parent -l hdisk2    
dar0

# lsdev -C -F parent -l hdisk3
dar0

# lsdev -C -F parent -l hdisk4 
dar0

# lsdev -C -F parent -l hdisk5
dar0

# lsdev -C -F parent -l hdisk6
dar0

# lsdev -C | grep dar
dar0        Available               1742-900 (900) Disk Array Router
en0         Available 0C-08         Standard Ethernet Network Interface
en1         Available 0D-08         Standard Ethernet Network Interface

# lsdev -C | grep dac
dac0        Available 07-08-02      1742-900 (900) Disk Array Controller
dac1        Available 08-08-02      1742-900 (900) Disk Array Controller
dac2        Available 07-08-02      1742-900 (900) Disk Array Controller
dac3        Available 08-08-02      1742-900 (900) Disk Array Controller

# lsdev -C -F parent -l dac0  
fscsi0

# lsdev -C -F parent -l dac1
fscsi1

# lsdev -C -F parent -l dac2
fscsi0

# lsdev -C -F parent -l dac3
fscsi1

# lsdev -C -F parent -l fscsi1
fcs1

how can i make the former cluster to be identical to the later?

Regards,
Khalid

 

[p5wizard]

IMHO it is either a SAN zoning issue on your switches or the SAN servers are not 'presenting' the LUNs to the server's newly added FC adapter.


HTH,

p5wizard

 

[DukeSSD]

I agree with p5wizard, you will need to configure the FastT 900 to allow access by the new fibre adapter wwn, and then zone your switch(es) to connect the dacs (disk array controllers) of the FastT 900 to the new adapter.

 

[khalidaaa]

Thanks p5wizard and DukeSSD for your comments.

But i'm sure i did add the fiber to the wwn and added that to the zone as well!

But for some reason i don't see the dac devices! Do i have to rmdev the current dac and dar devices and then cfgmgr?

I didn't do that option coz this will require downtime and i have to make sure that this work before arranging for the downtime!

Thanks for the help again

Khalid

 

[khalidaaa]

I just got this nice command from a friend:

# fget_config -vA

---dar0---

User array name = 'p5'
dac0 ACTIVE dac1 ACTIVE

Disk     DAC   LUN Logical Drive
utm             31
hdisk0   dac0    0 OPLP_hdisk0      
hdisk1   dac1    1 OPL01      
hdisk2   dac1    2 OPL02      
hdisk3   dac0    3 OPL_HB

It shows which dac is being used with which LUN on the SAN

he explained as well about the dar and the dac devices saying :"you will see one dar device for each storage partition on the DS4500. A storage partition is one host group or single host. The dac devices are the DS4500 controllers serving the storage partition"

But i only see two dac devices over here! not like the other cluster that i'm comparing this to!

# fget_config -vA

---dar0---

User array name = 'p5'
dac2 ACTIVE dac3 ACTIVE

Disk     DAC   LUN Logical Drive
utm             31
hdisk2   dac3    0 LS01       
hdisk3   dac3    1 LS02       
hdisk4   dac3    2 LS03       
hdisk5   dac3    3 LS04       
hdisk6   dac2    4 LS_HB

here i can see that dac3 and dac2 are being used! but i still have dac0 and dac1 available in case something happen in one of the controllers!

# lsdev -C | grep dac
dac0        Available 07-08-02      1742-900 (900) Disk Array Controller
dac1        Available 08-08-02      1742-900 (900) Disk Array Controller
dac2        Available 07-08-02      1742-900 (900) Disk Array Controller
dac3        Available 08-08-02      1742-900 (900) Disk Array Controller

Do i expect to get dac2 and dac3 in the former cluster?

He adviced me to change the preferred path on the SAN for the cluster but as this is a production one, i won't risk doing so now! i will wait after working hours.

Any more advices?

Regards,
Khalid

 

[khalidaaa]

Since its a cluster machine so i tried that on the standby node and i managed to change the preferred path successfully:


Before i change the path:

# fget_config -vA

---dar0---

User array name = 'p5'
dac0 ACTIVE dac1 ACTIVE

Disk     DAC   LUN Logical Drive
utm             31
hdisk0   dac1    0 OLS_hdisk0      
hdisk1   dac0    1 OPL01      
hdisk2   dac0    2 OPL02      
hdisk3   dac1    3 OPL_HB

after the change:

# fget_config -vA

---dar0---

User array name = 'p5'
dac0 ACTIVE dac1 ACTIVE

Disk     DAC   LUN Logical Drive
utm             31
hdisk0   dac0    0 OLS_hdisk0      
hdisk1   dac0    1 OPL01      
hdisk2   dac0    2 OPL02      
hdisk3   dac1    3 OPL_HB

Any conclusions?

Regards,
Khalid

 

[khalidaaa]

That's the best link ever on these topics:

http://www.redbooks.ibm.com/redbooks/pdfs/sg246363.pdf

The RDAC driver creates the following devices that represent the DS4000 storage subsystem
configuration:
 dar (disk array router): Represents the entire subsystem and storage partitions.
 dac (disk array controller devices): Represents a controller within the Storage Subsystem.
There are two dacs in the Storage Subsystem.
 hdisk: These devices represent individual LUNs on the array.
 utm: The universal transport mechanism (utm) device is used only with in-band
management configurations, as a communication channel between the SMagent and the
DS4000 Storage Server.

DAR
For a correct AIX host configuration, you should have DAR for every Storage Partition of every
DS4000 connected to the AIX host (Example 12-1).

Example 12-1 Storage server, storage partitions and dar
One DS4000 with one Storage Partition dar0
One DS4000 with two Storage Partition dar0,dar1
Two DS4000 with two Storage Partition dar0,dar1,dar2,dar3

You can verify the number of DAR configured on the system by typing the command:
# lsdev -C | grep dar

DAC
For a correct configuration, it is necessary to create an adequate zoning in such a way that
every DAR shows two DACs assigned.

Attention: More than two DAC on each DAR is not supported (this would decrease the
number of servers that you can connect to the DS4000 without additional benefit).

You can verify the number of DAC configured on the system by typing the command:
# lsdev -C | grep dac
Recreating the relationship between DAR and DAC
To recreate the relationships, you must first remove the DAR, DAC, and HBA definitions from
AIX. Proceed as follows:
Use the lsvg -o command to vary off all the volume groups attached to the HBA you want to
remove (make sure there is no activity on DS4000 disks).
1. Remove DAR with the following command:
# rmdev -dl dar 0 -R
hdisk1 delete
hdisk2 delete
dar0 delete
2. Then, remove the HBA with the following command:
# rmdev -dl fcs1 -R
dac0 delete
fscsi0 delete
fcnet0 delete
fcs0 delete
3. To reconfigure the HBA, DAC, DAR and hdisks, simply run the AIX configurator manager
using the following command:
# cfgmgr -S

I guess the extra dac0 and dac1 on the second cluster are not being used! so all I need to do (if i needed to arrange them is to follow the procedure above to delete them and then cfgmgr!

But i don't have to for now! I believe for now that what i did by adding the FC worked! I'm not sure if the fget_config -vA proves it though!

Any way, Thanks for the support guys

Regards,
Khalid

 

[khalidaaa]

I think you were both correct p5wizard and DukeSSd! There was a problem in the switch zoning!

I have two switches, one switch for each p5 rack, and my problem was that i have, by mistake, swapped the definition of one of the wwn that should be in rack1 switch with the other in rack2 switch!

The advice given to me that i should correct this only if the machines connected to these two lpars are down to avoid any problem!

What i'm not sure of, why am i still having those lpars running while i have a swapped wwns???????

regards,
Khalid

 

[p5wizard]

Well, the wwnn of the first FC cards assigned to the p5 lpars must be in the correct zone then?


HTH,

p5wizard

 

[khalidaaa]

But i can see that's the only two FCs that are swapped!

Any advices to make sure that they are correct!?

 

[khalidaaa]

p5wizard? where are you man? i was about to do the above but when you said what you said i'm having second thoughts right now!

Could you please help me making sure that what i'm going to do won't effect my production system?

Thanks

 

[p5wizard]

Khalidaaa,

FC zone modification is not something you want do "on the fly".

Make double sure the modification you are doing is correcting things, not worsening the situation. In your case, I think it is better to wait until a maintenance timeslot on the p5 LPARs involved.

The advice given to me that i should correct this only if the machines connected to these two lpars are down to avoid any problem!

Also, the procedure from the redbook you mentioned for correcting dar/dac configuration includes removing and rediscovering fcss/fscsis/dacs/dars/disks, so you would need to wait for a maintenance timeslot anyway.


HTH,

p5wizard

 

[khalidaaa]

Yeah there will be some changes to the application residing on those LPARs this Friday! so i Thought of doing this on friday while the machine is down!

But the strange thing that it is working for now!?! So i'm worried that it won't work when i do the change! coz as you said they might be in the correct place! but i'm sure they are swapped!

The swapped FCs in the zone, one for a clustered LPAR and the other is for a normal pilot test lpar. So What i thought of doing today is that i will remove the pilote test FC and put it in the right place and then i will start up the LPAR and see what happens!

But would that effect my other lpars configuration in any way? i don't think so!

 

[p5wizard]

Well, as long as you only modify stuff with the test LPAR's WWNNs, it should not affect other hosts in the zones you are modifying...

Other thought: We use one zone per host, with the SAN server's WWNNs and the host's WWNNs in a zone only. Yes, you have multiple zones in a zoneset, but it is very easy to find out what went wrong if a host doesn't "see" its LUNs...


HTH,

p5wizard

 

[khalidaaa]

Thanks p5wizard for your support

I just had a second thought! Now the port on the Switch is having the other FC HW address! So if i'm to change this i have to change both at the same time i beleive!

So i will delay this till Friday to have both LPARs down and to make a swap of the HW Addresses of both at the same time!

I will keep you informed with this issue.

Regards,
Khalid