Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
[Solved] "Fun" with Multipath, LVM and QLogic SANs
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Networking & Security
View previous topic :: View next topic  
Author Message
linear
Apprentice
Apprentice


Joined: 12 Aug 2004
Posts: 220

PostPosted: Tue Aug 21, 2018 7:47 pm    Post subject: [Solved] "Fun" with Multipath, LVM and QLogic SANs Reply with quote

Greetings,

I apologize for the length of this post, but as Inigo Montoya once said, "Let me explain... No, there is too much. Let me sum up."
I have been having an issue trying to get a server configured to properly address the partitions created on shares presented from a QLogic SAN.

The situation:
The server has two 8Gb Fibre Channel HBAs in it, configured and running. The SAN is presenting 5 shares to each HBA (even though the server seems to see it as 10 "drives" per HBA). So, when I do a multipath -l, I first got the following:
Code:
 ~ # multipath -l
360002ac0000000000000278300019b15 dm-1 3PARdata,VV
size=10T features='0' hwhandler='0' wp=rw
`-+- policy='service-time 0' prio=0 status=active
  |- 9:0:0:1  sdd 8:48  active undef running
  |- 9:0:1:1  sdi 8:128 active undef running
  |- 10:0:0:1 sdn 8:208 active undef running
  `- 10:0:1:1 sds 65:32 active undef running
360002ac0000000000000278600019b15 dm-4 3PARdata,VV
size=10T features='0' hwhandler='0' wp=rw
`-+- policy='service-time 0' prio=0 status=active
  |- 9:0:0:4  sdg 8:96  active undef running
  |- 9:0:1:4  sdl 8:176 active undef running
  |- 10:0:0:4 sdq 65:0  active undef running
  `- 10:0:1:4 sdv 65:80 active undef running
360002ac0000000000000278400019b15 dm-2 3PARdata,VV
size=10T features='0' hwhandler='0' wp=rw
`-+- policy='service-time 0' prio=0 status=active
  |- 9:0:0:2  sde 8:64  active undef running
  |- 9:0:1:2  sdj 8:144 active undef running
  |- 10:0:0:2 sdo 8:224 active undef running
  `- 10:0:1:2 sdt 65:48 active undef running
360002ac0000000000000278200019b15 dm-0 3PARdata,VV
size=10T features='0' hwhandler='0' wp=rw
`-+- policy='service-time 0' prio=0 status=active
  |- 9:0:0:0  sdc 8:32  active undef running
  |- 9:0:1:0  sdh 8:112 active undef running
  |- 10:0:0:0 sdm 8:192 active undef running
  `- 10:0:1:0 sdr 65:16 active undef running
360002ac0000000000000278500019b15 dm-3 3PARdata,VV
size=10T features='0' hwhandler='0' wp=rw
`-+- policy='service-time 0' prio=0 status=active
  |- 9:0:0:3  sdf 8:80  active undef running
  |- 9:0:1:3  sdk 8:160 active undef running
  |- 10:0:0:3 sdp 8:240 active undef running
  `- 10:0:1:3 sdu 65:64 active undef running
 ~ #


After finishing the multipath config, I get the following:
Code:
 ~ # multipath -l
rnssfcshr5 (360002ac0000000000000278600019b15) dm-4 3PARdata,VV
size=10T features='0' hwhandler='0' wp=rw
`-+- policy='service-time 0' prio=1 status=active
  |- 9:0:0:4  sdg 8:96  active ready running
  |- 9:0:1:4  sdl 8:176 active ready running
  |- 10:0:0:4 sdq 65:0  active ready running
  `- 10:0:1:4 sdv 65:80 active ready running
rnssfcshr4 (360002ac0000000000000278500019b15) dm-3 3PARdata,VV
size=10T features='0' hwhandler='0' wp=rw
`-+- policy='service-time 0' prio=1 status=active
  |- 9:0:0:3  sdf 8:80  active ready running
  |- 9:0:1:3  sdk 8:160 active ready running
  |- 10:0:0:3 sdp 8:240 active ready running
  `- 10:0:1:3 sdu 65:64 active ready running
rnssfcshr3 (360002ac0000000000000278400019b15) dm-2 3PARdata,VV
size=10T features='0' hwhandler='0' wp=rw
`-+- policy='service-time 0' prio=1 status=active
  |- 9:0:0:2  sde 8:64  active ready running
  |- 9:0:1:2  sdj 8:144 active ready running
  |- 10:0:0:2 sdo 8:224 active ready running
  `- 10:0:1:2 sdt 65:48 active ready running
rnssfcshr2 (360002ac0000000000000278300019b15) dm-1 3PARdata,VV
size=10T features='0' hwhandler='0' wp=rw
`-+- policy='service-time 0' prio=1 status=active
  |- 9:0:0:1  sdd 8:48  active ready running
  |- 9:0:1:1  sdi 8:128 active ready running
  |- 10:0:0:1 sdn 8:208 active ready running
  `- 10:0:1:1 sds 65:32 active ready running
rnssfcshr1 (360002ac0000000000000278200019b15) dm-0 3PARdata,VV
size=10T features='0' hwhandler='0' wp=rw
`-+- policy='service-time 0' prio=1 status=active
  |- 9:0:0:0  sdc 8:32  active ready running
  |- 9:0:1:0  sdh 8:112 active ready running
  |- 10:0:0:0 sdm 8:192 active ready running
  `- 10:0:1:0 sdr 65:16 active ready running
 ~ #


So, I went through and created partitions on each of the "shares", renaming them so I could be sure down the road to be able to discern which server each set of shares should be getting presented to...
Code:
 ~ # cat /sys/block/dm-0/queue/optimal_io_size
16777216
 ~ # cat /sys/block/dm-0/queue/minimum_io_size
16384
 ~ # cat /sys/block/dm-0/queue/physical_block_size
512
 ~ # cat /sys/block/dm-0/alignment_offset
0
 ~ # parted /dev/dm-0
GNU Parted 3.2
Using /dev/dm-0
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) print
Error: /dev/dm-0: unrecognised disk label
Model: Unknown (unknown)
Disk /dev/dm-0: 11.0TB
Sector size (logical/physical): 512B/512B
Partition Table: unknown
Disk Flags:
(parted) mklabel gpt
(parted) mkpart pri 32768s 100%
(parted) print
Model: Unknown (unknown)
Disk /dev/dm-0: 11.0TB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:

Number  Start   End     Size    File system  Name  Flags
 1      16.8MB  11.0TB  11.0TB               pri

(parted) name 1 rnsslvm01
(parted) set 1 lvm on
(parted) print
Model: Unknown (unknown)
Disk /dev/dm-0: 11.0TB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:

Number  Start   End     Size    File system  Name       Flags
 1      16.8MB  11.0TB  11.0TB               rnsslvm01  lvm

(parted) select /dev/dm-1
Using /dev/dm-1
(parted) mklabel gpt
(parted) mkpart pri 32768s 100%
(parted) name 1 rnsslvm02
(parted) set 1 lvm on
(parted) print
Model: Unknown (unknown)
Disk /dev/dm-1: 11.0TB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:

Number  Start   End     Size    File system  Name       Flags
 1      16.8MB  11.0TB  11.0TB               rnsslvm02  lvm

(parted) select /dev/dm-2
Using /dev/dm-2
(parted) mklabel gpt
(parted) mkpart pri 32768s 100%
(parted) name 1 rnsslvm03
(parted) set 1 lvm on
(parted) select /dev/dm-3
Using /dev/dm-3
(parted) mklabel gpt
(parted) mkpart pri 32768s 100%
(parted) name 1 rnsslvm04
(parted) set 1 lvm on
(parted) select /dev/dm-4
Using /dev/dm-4
(parted) mklabel gpt
(parted) mkpart pri 32768s 100%
(parted) name 1 rnsslvm05
(parted) set 1 lvm on
(parted) quit
Information: You may need to update /etc/fstab.

 ~ #


Now, the issue I am having is being able to address the partitions directly with pvcreate to put them together as a volume group. First I ran a pvscan -vvv to make sure I had all the names to address them by:
Code:
 ~ # pvscan -vvv
        DEGRADED MODE. Incomplete RAID LVs will be processed.
      Setting activation/monitoring to 1
        Processing: pvscan -vvv
        O_DIRECT will be used
      Setting global/locking_type to 1
      Setting global/wait_for_locks to 1
      File-based locking selected.
      Setting global/prioritise_write_locks to 1
      Setting global/locking_dir to /run/lock/lvm
      Locking /run/lock/lvm/P_global WB
        _do_flock /run/lock/lvm/P_global:aux WB
        _do_flock /run/lock/lvm/P_global WB
        _undo_flock /run/lock/lvm/P_global:aux
    Wiping cache of LVM-capable devices
        /dev/sdc: Added to device cache
        /dev/disk/by-id/scsi-360002ac0000000000000278200019b15: Added to device cache
        /dev/disk/by-id/wwn-0x60002ac0000000000000278200019b15: Aliased to /dev/disk/by-id/scsi-360002ac0000000000000278200019b15 in device cache
        /dev/disk/by-path/pci-0000:01:00.0-fc-0x22020002ac019b15-lun-0: Aliased to /dev/sdc in device cache
        /dev/sdc1: Added to device cache
        /dev/disk/by-id/scsi-360002ac0000000000000278200019b15-part1: Added to device cache
        /dev/disk/by-id/wwn-0x60002ac0000000000000278200019b15-part1: Aliased to /dev/disk/by-id/scsi-360002ac0000000000000278200019b15-part1 in device cache
        /dev/disk/by-partlabel/rnsslvm01: Aliased to /dev/disk/by-id/scsi-360002ac0000000000000278200019b15-part1 in device cache
        /dev/disk/by-partuuid/b2206bd0-e7f4-4f92-a62d-f97fa61c1233: Aliased to /dev/disk/by-id/scsi-360002ac0000000000000278200019b15-part1 in device cache
        /dev/disk/by-path/pci-0000:01:00.0-fc-0x22020002ac019b15-lun-0-part1: Aliased to /dev/sdc1 in device cache
        /dev/sdd: Added to device cache
        /dev/disk/by-id/scsi-360002ac0000000000000278300019b15: Added to device cache
        /dev/disk/by-id/wwn-0x60002ac0000000000000278300019b15: Aliased to /dev/disk/by-id/scsi-360002ac0000000000000278300019b15 in device cache
        /dev/disk/by-path/pci-0000:01:00.0-fc-0x22020002ac019b15-lun-1: Aliased to /dev/sdd in device cache
        /dev/sdd1: Added to device cache
        /dev/disk/by-id/scsi-360002ac0000000000000278300019b15-part1: Added to device cache
        /dev/disk/by-id/wwn-0x60002ac0000000000000278300019b15-part1: Aliased to /dev/disk/by-id/scsi-360002ac0000000000000278300019b15-part1 in device cache
        /dev/disk/by-partlabel/rnsslvm02: Aliased to /dev/disk/by-id/scsi-360002ac0000000000000278300019b15-part1 in device cache
        /dev/disk/by-partuuid/2347abf9-652f-41e6-b33a-7f83fc99e4cb: Aliased to /dev/disk/by-id/scsi-360002ac0000000000000278300019b15-part1 in device cache
        /dev/disk/by-path/pci-0000:01:00.0-fc-0x22020002ac019b15-lun-1-part1: Aliased to /dev/sdd1 in device cache
        /dev/sde: Added to device cache
        /dev/disk/by-id/scsi-360002ac0000000000000278400019b15: Added to device cache
        /dev/disk/by-id/wwn-0x60002ac0000000000000278400019b15: Aliased to /dev/disk/by-id/scsi-360002ac0000000000000278400019b15 in device cache
        /dev/disk/by-path/pci-0000:01:00.0-fc-0x22020002ac019b15-lun-2: Aliased to /dev/sde in device cache
        /dev/sde1: Added to device cache
        /dev/disk/by-id/scsi-360002ac0000000000000278400019b15-part1: Added to device cache
        /dev/disk/by-id/wwn-0x60002ac0000000000000278400019b15-part1: Aliased to /dev/disk/by-id/scsi-360002ac0000000000000278400019b15-part1 in device cache
        /dev/disk/by-partlabel/rnsslvm03: Aliased to /dev/disk/by-id/scsi-360002ac0000000000000278400019b15-part1 in device cache
        /dev/disk/by-partuuid/8e23807f-e754-44e8-bca3-f73079d5c147: Aliased to /dev/disk/by-id/scsi-360002ac0000000000000278400019b15-part1 in device cache
        /dev/disk/by-path/pci-0000:01:00.0-fc-0x22020002ac019b15-lun-2-part1: Aliased to /dev/sde1 in device cache
        /dev/sdf: Added to device cache
        /dev/disk/by-id/scsi-360002ac0000000000000278500019b15: Added to device cache
        /dev/disk/by-id/wwn-0x60002ac0000000000000278500019b15: Aliased to /dev/disk/by-id/scsi-360002ac0000000000000278500019b15 in device cache
        /dev/disk/by-path/pci-0000:01:00.0-fc-0x22020002ac019b15-lun-3: Aliased to /dev/sdf in device cache
        /dev/sdf1: Added to device cache
        /dev/disk/by-id/scsi-360002ac0000000000000278500019b15-part1: Added to device cache
        /dev/disk/by-id/wwn-0x60002ac0000000000000278500019b15-part1: Aliased to /dev/disk/by-id/scsi-360002ac0000000000000278500019b15-part1 in device cache
        /dev/disk/by-partlabel/rnsslvm04: Aliased to /dev/disk/by-id/scsi-360002ac0000000000000278500019b15-part1 in device cache
        /dev/disk/by-partuuid/ddf0bb9f-489b-4726-b06b-55a2c9512830: Aliased to /dev/disk/by-id/scsi-360002ac0000000000000278500019b15-part1 in device cache
        /dev/disk/by-path/pci-0000:01:00.0-fc-0x22020002ac019b15-lun-3-part1: Aliased to /dev/sdf1 in device cache
        /dev/sdg: Added to device cache
        /dev/disk/by-id/scsi-360002ac0000000000000278600019b15: Added to device cache
        /dev/disk/by-id/wwn-0x60002ac0000000000000278600019b15: Aliased to /dev/disk/by-id/scsi-360002ac0000000000000278600019b15 in device cache
        /dev/disk/by-path/pci-0000:01:00.0-fc-0x22020002ac019b15-lun-4: Aliased to /dev/sdg in device cache
        /dev/sdg1: Added to device cache
        /dev/disk/by-id/scsi-360002ac0000000000000278600019b15-part1: Added to device cache
        /dev/disk/by-id/wwn-0x60002ac0000000000000278600019b15-part1: Aliased to /dev/disk/by-id/scsi-360002ac0000000000000278600019b15-part1 in device cache
        /dev/disk/by-partlabel/rnsslvm05: Aliased to /dev/disk/by-id/scsi-360002ac0000000000000278600019b15-part1 in device cache
        /dev/disk/by-partuuid/67836ead-be6c-47e9-b1ef-8d7bf2a7518f: Aliased to /dev/disk/by-id/scsi-360002ac0000000000000278600019b15-part1 in device cache
        /dev/disk/by-path/pci-0000:01:00.0-fc-0x22020002ac019b15-lun-4-part1: Aliased to /dev/sdg1 in device cache
        /dev/sdh: Added to device cache
        /dev/disk/by-id/scsi-360002ac0000000000000278200019b15: Already in device cache
        /dev/disk/by-id/wwn-0x60002ac0000000000000278200019b15: Already in device cache
        /dev/disk/by-path/pci-0000:01:00.0-fc-0x23020002ac019b15-lun-0: Aliased to /dev/sdh in device cache
        /dev/sdh1: Added to device cache
        /dev/disk/by-id/scsi-360002ac0000000000000278200019b15-part1: Already in device cache
        /dev/disk/by-id/wwn-0x60002ac0000000000000278200019b15-part1: Already in device cache
        /dev/disk/by-partlabel/rnsslvm01: Already in device cache
        /dev/disk/by-partuuid/b2206bd0-e7f4-4f92-a62d-f97fa61c1233: Already in device cache
        /dev/disk/by-path/pci-0000:01:00.0-fc-0x23020002ac019b15-lun-0-part1: Aliased to /dev/sdh1 in device cache
        /dev/sdi: Added to device cache
        /dev/disk/by-id/scsi-360002ac0000000000000278300019b15: Already in device cache
        /dev/disk/by-id/wwn-0x60002ac0000000000000278300019b15: Already in device cache
        /dev/disk/by-path/pci-0000:01:00.0-fc-0x23020002ac019b15-lun-1: Aliased to /dev/sdi in device cache
        /dev/sdi1: Added to device cache
        /dev/disk/by-id/scsi-360002ac0000000000000278300019b15-part1: Already in device cache
        /dev/disk/by-id/wwn-0x60002ac0000000000000278300019b15-part1: Already in device cache
        /dev/disk/by-partlabel/rnsslvm02: Already in device cache
        /dev/disk/by-partuuid/2347abf9-652f-41e6-b33a-7f83fc99e4cb: Already in device cache
        /dev/disk/by-path/pci-0000:01:00.0-fc-0x23020002ac019b15-lun-1-part1: Aliased to /dev/sdi1 in device cache
        /dev/sdj: Added to device cache
        /dev/disk/by-id/scsi-360002ac0000000000000278400019b15: Already in device cache
        /dev/disk/by-id/wwn-0x60002ac0000000000000278400019b15: Already in device cache
        /dev/disk/by-path/pci-0000:01:00.0-fc-0x23020002ac019b15-lun-2: Aliased to /dev/sdj in device cache
        /dev/sdj1: Added to device cache
        /dev/disk/by-id/scsi-360002ac0000000000000278400019b15-part1: Already in device cache
        /dev/disk/by-id/wwn-0x60002ac0000000000000278400019b15-part1: Already in device cache
        /dev/disk/by-partlabel/rnsslvm03: Already in device cache
        /dev/disk/by-partuuid/8e23807f-e754-44e8-bca3-f73079d5c147: Already in device cache
        /dev/disk/by-path/pci-0000:01:00.0-fc-0x23020002ac019b15-lun-2-part1: Aliased to /dev/sdj1 in device cache
        /dev/sdk: Added to device cache
        /dev/disk/by-id/scsi-360002ac0000000000000278500019b15: Already in device cache
        /dev/disk/by-id/wwn-0x60002ac0000000000000278500019b15: Already in device cache
        /dev/disk/by-path/pci-0000:01:00.0-fc-0x23020002ac019b15-lun-3: Aliased to /dev/sdk in device cache
        /dev/sdk1: Added to device cache
        /dev/disk/by-id/scsi-360002ac0000000000000278500019b15-part1: Already in device cache
        /dev/disk/by-id/wwn-0x60002ac0000000000000278500019b15-part1: Already in device cache
        /dev/disk/by-partlabel/rnsslvm04: Already in device cache
        /dev/disk/by-partuuid/ddf0bb9f-489b-4726-b06b-55a2c9512830: Already in device cache
        /dev/disk/by-path/pci-0000:01:00.0-fc-0x23020002ac019b15-lun-3-part1: Aliased to /dev/sdk1 in device cache
        /dev/sdl: Added to device cache
        /dev/disk/by-id/scsi-360002ac0000000000000278600019b15: Already in device cache
        /dev/disk/by-id/wwn-0x60002ac0000000000000278600019b15: Already in device cache
        /dev/disk/by-path/pci-0000:01:00.0-fc-0x23020002ac019b15-lun-4: Aliased to /dev/sdl in device cache
        /dev/sdl1: Added to device cache
        /dev/disk/by-id/scsi-360002ac0000000000000278600019b15-part1: Already in device cache
        /dev/disk/by-id/wwn-0x60002ac0000000000000278600019b15-part1: Already in device cache
        /dev/disk/by-partlabel/rnsslvm05: Already in device cache
        /dev/disk/by-partuuid/67836ead-be6c-47e9-b1ef-8d7bf2a7518f: Already in device cache
        /dev/disk/by-path/pci-0000:01:00.0-fc-0x23020002ac019b15-lun-4-part1: Aliased to /dev/sdl1 in device cache
        /dev/sdb: Added to device cache
        /dev/disk/by-id/scsi-2adda004000d00000: Aliased to /dev/sdb in device cache
        /dev/disk/by-path/pci-0000:02:00.0-scsi-0:0:0:0: Aliased to /dev/sdb in device cache
        /dev/sdb1: Added to device cache
        /dev/disk/by-id/scsi-2adda004000d00000-part1: Aliased to /dev/sdb1 in device cache
        /dev/disk/by-partlabel/primary: Aliased to /dev/sdb1 in device cache
        /dev/disk/by-partuuid/34e3c26a-4b16-40de-9de5-0b0d18ade5ec: Aliased to /dev/sdb1 in device cache
        /dev/disk/by-path/pci-0000:02:00.0-scsi-0:0:0:0-part1: Aliased to /dev/sdb1 in device cache
        /dev/disk/by-uuid/c84cc120-9a19-44d7-b16e-13bf1a36059e: Aliased to /dev/sdb1 in device cache
        /dev/sr0: Added to device cache
        /dev/cdrom: Aliased to /dev/sr0 in device cache (preferred name)
        /dev/disk/by-id/ata-Slimtype_DVD_A_DS8ACSH_455749500531: Aliased to /dev/cdrom in device cache
        /dev/sda: Added to device cache
        /dev/disk/by-id/ata-WDC_WD10JPLX-00MBPT0_JS100Y6210YMZW: Aliased to /dev/sda in device cache
        /dev/disk/by-id/wwn-0x9712354333225930752x: Aliased to /dev/sda in device cache
        /dev/sda1: Added to device cache
        /dev/disk/by-id/ata-WDC_WD10JPLX-00MBPT0_JS100Y6210YMZW-part1: Aliased to /dev/sda1 in device cache
        /dev/disk/by-id/wwn-0x9712354333225930752x-part1: Aliased to /dev/sda1 in device cache
        /dev/disk/by-uuid/bd5a4fc0-b597-4ea9-8c84-a1045e95f42e: Aliased to /dev/sda1 in device cache
        /dev/sda2: Added to device cache
        /dev/disk/by-id/ata-WDC_WD10JPLX-00MBPT0_JS100Y6210YMZW-part2: Aliased to /dev/sda2 in device cache
        /dev/disk/by-id/wwn-0x9712354333225930752x-part2: Aliased to /dev/sda2 in device cache
        /dev/disk/by-uuid/2d50bc4d-bd4d-473c-bbab-6ed46ff8f4f6: Aliased to /dev/sda2 in device cache
        /dev/sda3: Added to device cache
        /dev/disk/by-id/ata-WDC_WD10JPLX-00MBPT0_JS100Y6210YMZW-part3: Aliased to /dev/sda3 in device cache
        /dev/disk/by-id/wwn-0x9712354333225930752x-part3: Aliased to /dev/sda3 in device cache
        /dev/disk/by-uuid/81bf4105-7959-499f-8386-e9b467005df7: Aliased to /dev/sda3 in device cache
        /dev/root: Aliased to /dev/sda3 in device cache
        /dev/sda4: Added to device cache
        /dev/disk/by-id/ata-WDC_WD10JPLX-00MBPT0_JS100Y6210YMZW-part4: Aliased to /dev/sda4 in device cache
        /dev/disk/by-id/wwn-0x9712354333225930752x-part4: Aliased to /dev/sda4 in device cache
        /dev/disk/by-uuid/8f91cc47-dab9-4aad-a0b8-b0d6e5746477: Aliased to /dev/sda4 in device cache
        /dev/sdm: Added to device cache
        /dev/disk/by-id/scsi-360002ac0000000000000278200019b15: Already in device cache
        /dev/disk/by-id/wwn-0x60002ac0000000000000278200019b15: Already in device cache
        /dev/disk/by-path/pci-0000:03:00.0-fc-0x22010002ac019b15-lun-0: Aliased to /dev/sdm in device cache
        /dev/sdm1: Added to device cache
        /dev/disk/by-id/scsi-360002ac0000000000000278200019b15-part1: Already in device cache
        /dev/disk/by-id/wwn-0x60002ac0000000000000278200019b15-part1: Already in device cache
        /dev/disk/by-partlabel/rnsslvm01: Already in device cache
        /dev/disk/by-partuuid/b2206bd0-e7f4-4f92-a62d-f97fa61c1233: Already in device cache
        /dev/disk/by-path/pci-0000:03:00.0-fc-0x22010002ac019b15-lun-0-part1: Aliased to /dev/sdm1 in device cache
        /dev/sdn: Added to device cache
        /dev/disk/by-id/scsi-360002ac0000000000000278300019b15: Already in device cache
        /dev/disk/by-id/wwn-0x60002ac0000000000000278300019b15: Already in device cache
        /dev/disk/by-path/pci-0000:03:00.0-fc-0x22010002ac019b15-lun-1: Aliased to /dev/sdn in device cache
        /dev/sdn1: Added to device cache
        /dev/disk/by-id/scsi-360002ac0000000000000278300019b15-part1: Already in device cache
        /dev/disk/by-id/wwn-0x60002ac0000000000000278300019b15-part1: Already in device cache
        /dev/disk/by-partlabel/rnsslvm02: Already in device cache
        /dev/disk/by-partuuid/2347abf9-652f-41e6-b33a-7f83fc99e4cb: Already in device cache
        /dev/disk/by-path/pci-0000:03:00.0-fc-0x22010002ac019b15-lun-1-part1: Aliased to /dev/sdn1 in device cache
        /dev/sdo: Added to device cache
        /dev/disk/by-id/scsi-360002ac0000000000000278400019b15: Already in device cache
        /dev/disk/by-id/wwn-0x60002ac0000000000000278400019b15: Already in device cache
        /dev/disk/by-path/pci-0000:03:00.0-fc-0x22010002ac019b15-lun-2: Aliased to /dev/sdo in device cache
        /dev/sdo1: Added to device cache
        /dev/disk/by-id/scsi-360002ac0000000000000278400019b15-part1: Already in device cache
        /dev/disk/by-id/wwn-0x60002ac0000000000000278400019b15-part1: Already in device cache
        /dev/disk/by-partlabel/rnsslvm03: Already in device cache
        /dev/disk/by-partuuid/8e23807f-e754-44e8-bca3-f73079d5c147: Already in device cache
        /dev/disk/by-path/pci-0000:03:00.0-fc-0x22010002ac019b15-lun-2-part1: Aliased to /dev/sdo1 in device cache
        /dev/sdp: Added to device cache
        /dev/disk/by-id/scsi-360002ac0000000000000278500019b15: Already in device cache
        /dev/disk/by-id/wwn-0x60002ac0000000000000278500019b15: Already in device cache
        /dev/disk/by-path/pci-0000:03:00.0-fc-0x22010002ac019b15-lun-3: Aliased to /dev/sdp in device cache
        /dev/sdp1: Added to device cache
        /dev/disk/by-id/scsi-360002ac0000000000000278500019b15-part1: Already in device cache
        /dev/disk/by-id/wwn-0x60002ac0000000000000278500019b15-part1: Already in device cache
        /dev/disk/by-partlabel/rnsslvm04: Already in device cache
        /dev/disk/by-partuuid/ddf0bb9f-489b-4726-b06b-55a2c9512830: Already in device cache
        /dev/disk/by-path/pci-0000:03:00.0-fc-0x22010002ac019b15-lun-3-part1: Aliased to /dev/sdp1 in device cache
        /dev/sdq: Added to device cache
        /dev/disk/by-id/scsi-360002ac0000000000000278600019b15: Already in device cache
        /dev/disk/by-id/wwn-0x60002ac0000000000000278600019b15: Already in device cache
        /dev/disk/by-path/pci-0000:03:00.0-fc-0x22010002ac019b15-lun-4: Aliased to /dev/sdq in device cache
        /dev/sdq1: Added to device cache
        /dev/disk/by-id/scsi-360002ac0000000000000278600019b15-part1: Already in device cache
        /dev/disk/by-id/wwn-0x60002ac0000000000000278600019b15-part1: Already in device cache
        /dev/disk/by-partlabel/rnsslvm05: Already in device cache
        /dev/disk/by-partuuid/67836ead-be6c-47e9-b1ef-8d7bf2a7518f: Already in device cache
        /dev/disk/by-path/pci-0000:03:00.0-fc-0x22010002ac019b15-lun-4-part1: Aliased to /dev/sdq1 in device cache
        /dev/sdr: Aliased to /dev/disk/by-id/scsi-360002ac0000000000000278200019b15 in device cache (preferred name)
        /dev/disk/by-id/scsi-360002ac0000000000000278200019b15: Already in device cache
        /dev/disk/by-id/wwn-0x60002ac0000000000000278200019b15: Already in device cache
        /dev/disk/by-path/pci-0000:03:00.0-fc-0x23010002ac019b15-lun-0: Aliased to /dev/sdr in device cache
        /dev/sdr1: Aliased to /dev/disk/by-id/scsi-360002ac0000000000000278200019b15-part1 in device cache (preferred name)
        /dev/disk/by-id/scsi-360002ac0000000000000278200019b15-part1: Already in device cache
        /dev/disk/by-id/wwn-0x60002ac0000000000000278200019b15-part1: Already in device cache
        /dev/disk/by-partlabel/rnsslvm01: Already in device cache
        /dev/disk/by-partuuid/b2206bd0-e7f4-4f92-a62d-f97fa61c1233: Already in device cache
        /dev/disk/by-path/pci-0000:03:00.0-fc-0x23010002ac019b15-lun-0-part1: Aliased to /dev/sdr1 in device cache
        /dev/sds: Aliased to /dev/disk/by-id/scsi-360002ac0000000000000278300019b15 in device cache (preferred name)
        /dev/disk/by-id/scsi-360002ac0000000000000278300019b15: Already in device cache
        /dev/disk/by-id/wwn-0x60002ac0000000000000278300019b15: Already in device cache
        /dev/disk/by-path/pci-0000:03:00.0-fc-0x23010002ac019b15-lun-1: Aliased to /dev/sds in device cache
        /dev/sds1: Aliased to /dev/disk/by-id/scsi-360002ac0000000000000278300019b15-part1 in device cache (preferred name)
        /dev/disk/by-id/scsi-360002ac0000000000000278300019b15-part1: Already in device cache
        /dev/disk/by-id/wwn-0x60002ac0000000000000278300019b15-part1: Already in device cache
        /dev/disk/by-partlabel/rnsslvm02: Already in device cache
        /dev/disk/by-partuuid/2347abf9-652f-41e6-b33a-7f83fc99e4cb: Already in device cache
        /dev/disk/by-path/pci-0000:03:00.0-fc-0x23010002ac019b15-lun-1-part1: Aliased to /dev/sds1 in device cache
        /dev/sdt: Aliased to /dev/disk/by-id/scsi-360002ac0000000000000278400019b15 in device cache (preferred name)
        /dev/disk/by-id/scsi-360002ac0000000000000278400019b15: Already in device cache
        /dev/disk/by-id/wwn-0x60002ac0000000000000278400019b15: Already in device cache
        /dev/disk/by-path/pci-0000:03:00.0-fc-0x23010002ac019b15-lun-2: Aliased to /dev/sdt in device cache
        /dev/sdt1: Aliased to /dev/disk/by-id/scsi-360002ac0000000000000278400019b15-part1 in device cache (preferred name)
        /dev/disk/by-id/scsi-360002ac0000000000000278400019b15-part1: Already in device cache
        /dev/disk/by-id/wwn-0x60002ac0000000000000278400019b15-part1: Already in device cache
        /dev/disk/by-partlabel/rnsslvm03: Already in device cache
        /dev/disk/by-partuuid/8e23807f-e754-44e8-bca3-f73079d5c147: Already in device cache
        /dev/disk/by-path/pci-0000:03:00.0-fc-0x23010002ac019b15-lun-2-part1: Aliased to /dev/sdt1 in device cache
        /dev/sdu: Aliased to /dev/disk/by-id/scsi-360002ac0000000000000278500019b15 in device cache (preferred name)
        /dev/disk/by-id/scsi-360002ac0000000000000278500019b15: Already in device cache
        /dev/disk/by-id/wwn-0x60002ac0000000000000278500019b15: Already in device cache
        /dev/disk/by-path/pci-0000:03:00.0-fc-0x23010002ac019b15-lun-3: Aliased to /dev/sdu in device cache
        /dev/sdu1: Aliased to /dev/disk/by-id/scsi-360002ac0000000000000278500019b15-part1 in device cache (preferred name)
        /dev/disk/by-id/scsi-360002ac0000000000000278500019b15-part1: Already in device cache
        /dev/disk/by-id/wwn-0x60002ac0000000000000278500019b15-part1: Already in device cache
        /dev/disk/by-partlabel/rnsslvm04: Already in device cache
        /dev/disk/by-partuuid/ddf0bb9f-489b-4726-b06b-55a2c9512830: Already in device cache
        /dev/disk/by-path/pci-0000:03:00.0-fc-0x23010002ac019b15-lun-3-part1: Aliased to /dev/sdu1 in device cache
        /dev/sdv: Aliased to /dev/disk/by-id/scsi-360002ac0000000000000278600019b15 in device cache (preferred name)
        /dev/disk/by-id/scsi-360002ac0000000000000278600019b15: Already in device cache
        /dev/disk/by-id/wwn-0x60002ac0000000000000278600019b15: Already in device cache
        /dev/disk/by-path/pci-0000:03:00.0-fc-0x23010002ac019b15-lun-4: Aliased to /dev/sdv in device cache
        /dev/sdv1: Aliased to /dev/disk/by-id/scsi-360002ac0000000000000278600019b15-part1 in device cache (preferred name)
        /dev/disk/by-id/scsi-360002ac0000000000000278600019b15-part1: Already in device cache
        /dev/disk/by-id/wwn-0x60002ac0000000000000278600019b15-part1: Already in device cache
        /dev/disk/by-partlabel/rnsslvm05: Already in device cache
        /dev/disk/by-partuuid/67836ead-be6c-47e9-b1ef-8d7bf2a7518f: Already in device cache
        /dev/disk/by-path/pci-0000:03:00.0-fc-0x23010002ac019b15-lun-4-part1: Aliased to /dev/sdv1 in device cache
        /dev/loop0: Added to device cache
        ...<snip>...
        /dev/ram9: Added to device cache
        /dev/dm-0: Added to device cache
        /dev/disk/by-id/dm-name-rnssfcshr1: Aliased to /dev/dm-0 in device cache (preferred name)
        /dev/disk/by-id/dm-uuid-mpath-360002ac0000000000000278200019b15: Aliased to /dev/disk/by-id/dm-name-rnssfcshr1 in device cache
        /dev/mapper/rnssfcshr1: Aliased to /dev/disk/by-id/dm-name-rnssfcshr1 in device cache (preferred name)
        /dev/dm-1: Added to device cache
        /dev/disk/by-id/dm-name-rnssfcshr2: Aliased to /dev/dm-1 in device cache (preferred name)
        /dev/disk/by-id/dm-uuid-mpath-360002ac0000000000000278300019b15: Aliased to /dev/disk/by-id/dm-name-rnssfcshr2 in device cache
        /dev/mapper/rnssfcshr2: Aliased to /dev/disk/by-id/dm-name-rnssfcshr2 in device cache (preferred name)
        /dev/dm-2: Added to device cache
        /dev/disk/by-id/dm-name-rnssfcshr3: Aliased to /dev/dm-2 in device cache (preferred name)
        /dev/disk/by-id/dm-uuid-mpath-360002ac0000000000000278400019b15: Aliased to /dev/disk/by-id/dm-name-rnssfcshr3 in device cache
        /dev/mapper/rnssfcshr3: Aliased to /dev/disk/by-id/dm-name-rnssfcshr3 in device cache (preferred name)
        /dev/dm-3: Added to device cache
        /dev/disk/by-id/dm-name-rnssfcshr4: Aliased to /dev/dm-3 in device cache (preferred name)
        /dev/disk/by-id/dm-uuid-mpath-360002ac0000000000000278500019b15: Aliased to /dev/diskx/by-id/dm-name-rnssfcshr4 in device cache
        /dev/mapper/rnssfcshr4: Aliased to /dev/disk/by-id/dm-name-rnssfcshr4 in device cache (preferred name)
        /dev/dm-4: Added to device cache
        /dev/disk/by-id/dm-name-rnssfcshr5: Aliased to /dev/dm-4 in device cache (preferred name)
        /dev/disk/by-id/dm-uuid-mpath-360002ac0000000000000278600019b15: Aliased to /dev/disk/by-id/dm-name-rnssfcshr5 in device cache
        /dev/mapper/rnssfcshr5: Aliased to /dev/disk/by-id/dm-name-rnssfcshr5 in device cache (preferred name)
    Wiping internal VG cache
        Metadata cache has no info for vgname: "#orphans_pool"
        Metadata cache has no info for vgname: "#orphans_pool"
        lvmcache: initialised VG #orphans_pool
        Metadata cache has no info for vgname: "#orphans_lvm2"
        Metadata cache has no info for vgname: "#orphans_lvm2"
        lvmcache: initialised VG #orphans_lvm2
        Asking lvmetad for complete list of known VGs
      Setting response to OK
      Setting response to OK
    Walking through all physical volumes
        Asking lvmetad for complete list of known PVs
      Setting response to OK
      Setting response to OK
      Unlocking /run/lock/lvm/P_global
        _undo_flock /run/lock/lvm/P_global
        Completed: pvscan -vvv
 ~ # pvcreate /dev/disk/by-partlabel/rnsslvm01 /dev/disk/by-partlabel/rnsslvm02 /dev/disk/by-partlabel/rnsslvm03 /dev/disk/by-partlabel/rnsslvm04 /dev/disk/by-partlabel/rnsslvm05
  Can't open /dev/disk/by-partlabel/rnsslvm01 exclusively.  Mounted filesystem?
  Can't open /dev/disk/by-partlabel/rnsslvm02 exclusively.  Mounted filesystem?
  Can't open /dev/disk/by-partlabel/rnsslvm03 exclusively.  Mounted filesystem?
  Can't open /dev/disk/by-partlabel/rnsslvm04 exclusively.  Mounted filesystem?
  Can't open /dev/disk/by-partlabel/rnsslvm05 exclusively.  Mounted filesystem?
 ~ # pvcreate /dev/disk/by-id/scsi-360002ac0000000000000278200019b15-part1 /dev/disk/by-id/scsi-360002ac0000000000000278300019b15-part1 /dev/disk/by-id/scsi-360002ac0000000000000278400019b15-part1 /dev/disk/by-id/scsi-360002ac0000000000000278500019b15-part1 /dev/disk/by-id/scsi-360002ac0000000000000278600019b15-part1
  Can't open /dev/disk/by-id/scsi-360002ac0000000000000278200019b15-part1 exclusively.  Mounted filesystem?
  Can't open /dev/disk/by-id/scsi-360002ac0000000000000278300019b15-part1 exclusively.  Mounted filesystem?
  Can't open /dev/disk/by-id/scsi-360002ac0000000000000278400019b15-part1 exclusively.  Mounted filesystem?
  Can't open /dev/disk/by-id/scsi-360002ac0000000000000278500019b15-part1 exclusively.  Mounted filesystem?
  Can't open /dev/disk/by-id/scsi-360002ac0000000000000278600019b15-part1 exclusively.  Mounted filesystem?
 ~ #


So, what am I missing to be able to address the partitions on the shares with pvcreate so I can put them together as a volume group?

TIA for your assistance.


Last edited by linear on Tue Oct 23, 2018 1:58 am; edited 1 time in total
Back to top
View user's profile Send private message
Cheyenne
n00b
n00b


Joined: 05 Jan 2005
Posts: 26

PostPosted: Tue Aug 21, 2018 10:40 pm    Post subject: You will need to blacklist the individual drives Reply with quote

I run several systems in a SAN environment with fibrechannel connections and multiple paths to the devices.

The problem is that LVM is doesn't recognize that the individual device components are the "same". You need to ensure that multipath and lvm are configured to handle what you want.


The way that devices are presented to the system in a multipathed environment is that each path will get it's own unique device, yet they all point to the same volume (WWN) that is created in the SAN. The multipath utility/package performs two main functions, 1st it recombines the devices back into a single device on the system, and 2nd it handles the "which device does the I/O go to" depending on the different algorithms used and how the physical SAN handles the multiple paths. Some SANS will use the multiple paths just for a failure, others allow I/Os to be presented on any path, etc. In addition the multipath package allows you to create an alias for the device.

My environment looks kind of like the following (bad ascii art..) -- two host adapters, each adapter has one connection to a switch, and the switches have two connections to the SAN. So each host adapter "sees" two devices -- and with 2 host adapters, overall the system sees 4 devices.

Code:

+-----+
| sys  |        +---+                                         
|      ha1 -----|sw |=== +---+
|      |        +---+    |san|
|      ha2 -----| sw|=== +---+
+-----+         +---+


I have a multipath.conf configured that creates aliases based off the specific WWIDs that the SAN uses to identify the individual LUNs.

Code:


blacklist {
       devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
# Local devices
       devnode "^(sda)$"
       devnode "^(sda)[0-9]*$"
       devnode "^(sdb)$"
       devnode "^(sdb)[0-9]*$"
}

defaults {
   polling_interval  30
   failback          immediate
   no_path_retry     5
   rr_min_io         100
   path_checker      tur
   user_friendly_names  yes
   dev_loss_tmo      7200
}

multipaths {
            multipath {
                # SAN SCSIid:98
                alias Sys1Data
                wwid 12305076802818eb70800000000000045
            }
            ... bunch more multipath stmts ...
}



Doing the following on my system multipath -ll Sys1Data shows:
Code:

Sys1Data (12305076802818eb70800000000000045) dm-87 IBM,2145
size=30G features='1 queue_if_no_path' hwhandler='0' wp=rw
|-+- policy='service-time 0' prio=50 status=active
| |- 10:0:3:98  sdko 66:448  active ready running
| `- 11:0:2:98  sdnf 71:272  active ready running
`-+- policy='service-time 0' prio=10 status=enabled
  |- 10:0:2:98  sddl 71:48   active ready running
  `- 11:0:3:98  sdrv 134:400 active ready running



What this is showing me are the 4 individual devices, two off of host adapter 10 and 2 off host adapter 11. The multipath daemon created a /dev/dm-87 that "combines" the 4 individual devices back into a single device, and has created an alias device /dev/mapper/Sys1Data.

So .. now comes the next part. LVM in your case. You need LVM to only look at either the device that multipath created -or- the alias. Here you need to configure LVM to ignore all the individual devices that are being presented by the host adapters.

In the /etc/lvm/lvm.conf file, look for a filter= statement and reject all the devices you want ignored.

Code:

   filter = ["a|^/dev/sd[ab][0-9]*", "r|^/dev/sd.*","r|^/dev/dm-.*","a|^/dev/mapper/.*"]

Breaking this apart, the first rule that matches for each device that LVM looks at.

  • accept any partitions on /dev/sda or /dev/sdb -- assume they are local
  • reject any /dev/sd.*
  • reject any /dev/dm-.*
  • accept any /dev/mapper/.*


This assumes you are using aliases. If you aren't using aliases, then accept the /dev/dm-.* pattern.
Back to top
View user's profile Send private message
Cheyenne
n00b
n00b


Joined: 05 Jan 2005
Posts: 26

PostPosted: Wed Aug 22, 2018 1:51 pm    Post subject: One more thing.. Reply with quote

One more thing, be aware that the scsi device discovery can end up assigning the same wwid to a different device id (e.g. on one boot up the device may show up as /dev/sdg and on the next boot up it might show up as /dev/sdm). It all depends on how the SAN responds when the system boots and if you have added any new luns. Also (not quite sure), but the dm-xx assignment may also change.

This is fine as long as whatever is using the devices can associate the device back into the context that it needs. For example LVM really doesn't care about what the /dev/... name is. LVM uses a UUID to identify the device that it "owns".
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Networking & Security All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum