1637688 – OCS 3.10 Multipath Adds ALUA which may not always be supported

Bug 1637688 - OCS 3.10 Multipath Adds ALUA which may not always be supported

Summary: OCS 3.10 Multipath Adds ALUA which may not always be supported

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat Gluster Storage
Classification:	Red Hat Storage
Component:	gluster-block
Sub Component:
Version:	cns-3.10
Hardware:	All
OS:	All
Priority:	unspecified
Severity:	urgent
Target Milestone:	---
Target Release:	OCS 3.11.1
Assignee:	Prasanna Kumar Kalever
QA Contact:	RamaKasturi
Docs Contact:
URL:
Whiteboard:
Depends On:	1638002 1638988
Blocks:	OCS-3.11.1-devel-triage-done 1644156
TreeView+	depends on / blocked

Reported:	2018-10-09 19:00 UTC by Matthew Robson
Modified:	2021-12-10 17:52 UTC (History)
CC List:	20 users (show)
Fixed In Version:	gluster-block-0.2.1-28.el7rhgs
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2019-02-07 03:38:29 UTC
Embargoed:
Dependent Products:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Red Hat Product Errata	RHBA-2019:0285	0	None	None	None	2019-02-07 03:38:49 UTC

Description Matthew Robson 2018-10-09 19:00:41 UTC

Description of problem:

OCS 3.10 changes multipath conf to use ALUA, but doesn't call out the change or ensure ensure customers backend storage supports it.

This may lead to failures in the upgrade.

Existing gluster-block devices created in CNS 3.9 failed with this change:

Oct 03 15:31:10 ociopf-t-322.dmz multipathd[108699]: sdf: alua not supported
Oct 03 15:31:10 ociopf-t-322.dmz kernel: sd 46:0:0:0: alua: not supported
Oct 03 15:31:10 ociopf-t-322.dmz kernel: sd 46:0:0:0: alua: not attached
Oct 03 15:31:10 ociopf-t-322.dmz multipathd[108699]: sdf: alua not supported
Oct 03 15:31:12 ociopf-t-322.dmz multipathd[108699]: sdg: alua not supported
Oct 03 15:31:12 ociopf-t-322.dmz multipathd[108699]: sdf: alua not supported
Oct 03 15:31:12 ociopf-t-322.dmz multipathd[108699]: sdg: alua not supported
Oct 03 15:31:12 ociopf-t-322.dmz kernel: sd 46:0:0:0: alua: not supported
Oct 03 15:31:12 ociopf-t-322.dmz kernel: sd 46:0:0:0: alua: not attached
Oct 03 15:31:13 ociopf-t-322.dmz multipathd[108699]: sdh: alua not supported
Oct 03 15:31:13 ociopf-t-322.dmz multipathd[108699]: sdf: alua not supported
Oct 03 15:31:13 ociopf-t-322.dmz multipathd[108699]: sdg: alua not supported
Oct 03 15:31:13 ociopf-t-322.dmz multipathd[108699]: sdh: alua not supported
Oct 03 15:31:13 ociopf-t-322.dmz kernel: sd 46:0:0:0: alua: not supported
Oct 03 15:31:13 ociopf-t-322.dmz kernel: sd 46:0:0:0: alua: not attached

Gluster Block Physical Volumes:

[root@server ~]# pvs | grep -E "vg_4e8106ce4a9c5278a4f231231ac3d915|vg_0aa5d6dad3331d73c048bc1d2dd575a0"
  /dev/sdd   vg_4e8106ce4a9c5278a4f231231ac3d915 lvm2 a--    99.87g   14.25g
  /dev/sdg   vg_0aa5d6dad3331d73c048bc1d2dd575a0 lvm2 a--   249.87g <132.11g

Multipath Devices:

[root@ociopf-t-322 ~]# multipath -ll
mpathh (36001405df12f77c20d14a089583d5536) dm-48 LIO-ORG ,TCMU device
size=2.0G features='1 queue_if_no_path' hwhandler='0' wp=rw
|-+- policy='round-robin 0' prio=1 status=active
| `- 54:0:0:0 sdg 8:96  active ready running
|-+- policy='round-robin 0' prio=1 status=enabled
| `- 57:0:0:0 sdj 8:144 active ready running
`-+- policy='round-robin 0' prio=1 status=enabled
  `- 58:0:0:0 sdk 8:160 active ready running
mpathg (3600140505d45aa0050948a6abe73e172) dm-49 LIO-ORG ,TCMU device
size=1.0G features='1 queue_if_no_path' hwhandler='0' wp=rw
|-+- policy='round-robin 0' prio=1 status=active
| `- 52:0:0:0 sdf 8:80  active ready running
|-+- policy='round-robin 0' prio=1 status=enabled
| `- 55:0:0:0 sdh 8:112 active ready running
`-+- policy='round-robin 0' prio=1 status=enabled
  `- 56:0:0:0 sdi 8:128 active ready running


sdg is from mpathh which is a new gluster block devices created post upgrade:

[root@server ~]# sg_rtpg -d /dev/sdg
Report target port groups:
  target port group id : 0x2 , Pref=0
    target port group asymmetric access state : 0x01 (active/non optimized)
    T_SUP : 1, O_SUP : 1, LBD_SUP : 0, U_SUP : 1, S_SUP : 1, AN_SUP : 1, AO_SUP : 1
    status code : 0x02 (target port asym. state changed by implicit lu behaviour)
    vendor unique status : 0x00
    target port count : 02
    Relative target port ids:
      0x02
      0x03
  target port group id : 0x1 , Pref=0
    target port group asymmetric access state : 0x00 (active/optimized)
    T_SUP : 1, O_SUP : 1, LBD_SUP : 0, U_SUP : 1, S_SUP : 1, AN_SUP : 1, AO_SUP : 1
    status code : 0x00 (no status available)
    vendor unique status : 0x00
    target port count : 01
    Relative target port ids:
      0x01

This is a gluster block device which existed pre-upgrade. It does not seem to have support:

[root@server ~]# sg_rtpg -d /dev/sdh
Report Target Port Groups command not supported


sh-4.2# grep alua_tg_pt_gp_name /etc/target/saveconfig.json
              "alua_tg_pt_gp_name": null,
              "alua_tg_pt_gp_name": null,
              "alua_tg_pt_gp_name": null,
              "alua_tg_pt_gp_name": null,
              "alua_tg_pt_gp_name": null,
              "alua_tg_pt_gp_name": null,
              "alua_tg_pt_gp_name": null,
              "alua_tg_pt_gp_name": null,
              "alua_tg_pt_gp_name": null,
              "alua_tg_pt_gp_name": null,
              "alua_tg_pt_gp_name": null,
              "alua_tg_pt_gp_name": null,
              "alua_tg_pt_gp_name": null,
              "alua_tg_pt_gp_name": null,
              "alua_tg_pt_gp_name": null,
              "alua_tg_pt_gp_name": "glfs_tg_pt_gp_ano",
              "alua_tg_pt_gp_name": "glfs_tg_pt_gp_ano",
              "alua_tg_pt_gp_name": "glfs_tg_pt_gp_ao",


Version-Release number of selected component (if applicable):

OCS 3.10


How reproducible:

Always


Steps to Reproduce:
1. Upgrade from CNS 3.9 with existing block volumes
2. Change multipath conf per 3.10 
3. If storage device doesn't support ALUA, it will fail
4. 

Actual results:

Can not apply updated multipath config

Expected results:

Should apply without issue and note the caveats.

Additional info:

Comment 27 RamaKasturi 2018-11-23 17:15:27 UTC

Verified the following in the build gluster-block-0.2.1-29.el7rhgs.x86_64.

created block volumes in cns3.9 where gluster-block version is gluster-block-0.2.1-14.1.el7rhgs.x86_64

sh-4.2# targetcli ls
o- / ......................................................................................................................... [...]
  o- backstores .............................................................................................................. [...]
  | o- block .................................................................................................. [Storage Objects: 0]
  | o- fileio ................................................................................................. [Storage Objects: 0]
  | o- pscsi .................................................................................................. [Storage Objects: 0]
  | o- ramdisk ................................................................................................ [Storage Objects: 0]
  | o- user:glfs .............................................................................................. [Storage Objects: 5]
  |   o- vol-39_glusterfs_blockclaim1_0d165c58-eda7-11e8-9f94-0a580a820002  [vol_fb52bfcbc97ebf63b5e540f7da250dd9.47.37/block-store/00dab599-d5c0-4dc7-bf6f-3241ac3558c9 (5.0GiB) activated]
  |   | o- alua ................................................................................................... [ALUA Groups: 1]
  |   |   o- default_tg_pt_gp ....................................................................... [ALUA state: Active/optimized]
  |   o- vol-39_glusterfs_blockclaim2_3cf680ce-eda8-11e8-9f94-0a580a820002  [vol_fb52bfcbc97ebf63b5e540f7da250dd9.47.37/block-store/df906384-cef4-4d2d-abc2-7ab74543afc3 (6.0GiB) activated]
  |   | o- alua ................................................................................................... [ALUA Groups: 1]
  |   |   o- default_tg_pt_gp ....................................................................... [ALUA state: Active/optimized]
  |   o- vol-39_glusterfs_blockclaim3_4c1e4cbf-eda8-11e8-9f94-0a580a820002  [vol_fb52bfcbc97ebf63b5e540f7da250dd9.47.37/block-store/bc0e2585-9b6a-4179-b06b-d30ffe6ea095 (7.0GiB) activated]
  |   | o- alua ................................................................................................... [ALUA Groups: 1]
  |   |   o- default_tg_pt_gp ....................................................................... [ALUA state: Active/optimized]
  |   o- vol-39_glusterfs_blockclaim4_54687b22-eda8-11e8-9f94-0a580a820002  [vol_fb52bfcbc97ebf63b5e540f7da250dd9.47.37/block-store/a97f81e6-6eb0-48d2-8c3f-820a2e2c98c9 (4.0GiB) activated]
  |   | o- alua ................................................................................................... [ALUA Groups: 1]
  |   |   o- default_tg_pt_gp ....................................................................... [ALUA state: Active/optimized]
  |   o- vol-39_glusterfs_blockclaim5_5eaadd88-eda8-11e8-9f94-0a580a820002  [vol_fb52bfcbc97ebf63b5e540f7da250dd9.47.37/block-store/94ab3b93-8dd3-4a38-8809-49dc6c895f42 (8.0GiB) activated]

glusterfs-storage-q5v4g
+++++++++++++++++++++++
o- / ......................................................................................................................... [...]
  o- backstores .............................................................................................................. [...]
  | o- block .................................................................................................. [Storage Objects: 0]
  | o- fileio ................................................................................................. [Storage Objects: 0]
  | o- pscsi .................................................................................................. [Storage Objects: 0]
  | o- ramdisk ................................................................................................ [Storage Objects: 0]
  | o- user:glfs .............................................................................................. [Storage Objects: 5]
  |   o- vol-39_glusterfs_blockclaim1_0d165c58-eda7-11e8-9f94-0a580a820002  [vol_fb52bfcbc97ebf63b5e540f7da250dd9.46.23/block-store/00dab599-d5c0-4dc7-bf6f-3241ac3558c9 (5.0GiB) activated]
  |   | o- alua ................................................................................................... [ALUA Groups: 1]
  |   |   o- default_tg_pt_gp ....................................................................... [ALUA state: Active/optimized]
  |   o- vol-39_glusterfs_blockclaim2_3cf680ce-eda8-11e8-9f94-0a580a820002  [vol_fb52bfcbc97ebf63b5e540f7da250dd9.46.23/block-store/df906384-cef4-4d2d-abc2-7ab74543afc3 (6.0GiB) activated]
  |   | o- alua ................................................................................................... [ALUA Groups: 1]
  |   |   o- default_tg_pt_gp ....................................................................... [ALUA state: Active/optimized]
  |   o- vol-39_glusterfs_blockclaim3_4c1e4cbf-eda8-11e8-9f94-0a580a820002  [vol_fb52bfcbc97ebf63b5e540f7da250dd9.46.23/block-store/bc0e2585-9b6a-4179-b06b-d30ffe6ea095 (7.0GiB) activated]
  |   | o- alua ................................................................................................... [ALUA Groups: 1]
  |   |   o- default_tg_pt_gp ....................................................................... [ALUA state: Active/optimized]
  |   o- vol-39_glusterfs_blockclaim4_54687b22-eda8-11e8-9f94-0a580a820002  [vol_fb52bfcbc97ebf63b5e540f7da250dd9.46.23/block-store/a97f81e6-6eb0-48d2-8c3f-820a2e2c98c9 (4.0GiB) activated]
  |   | o- alua ................................................................................................... [ALUA Groups: 1]
  |   |   o- default_tg_pt_gp ....................................................................... [ALUA state: Active/optimized]
  |   o- vol-39_glusterfs_blockclaim5_5eaadd88-eda8-11e8-9f94-0a580a820002  [vol_fb52bfcbc97ebf63b5e540f7da250dd9.46.23/block-store/94ab3b93-8dd3-4a38-8809-49dc6c895f42 (8.0GiB) activated]
  |     o- alua ................................................................................................... [ALUA Groups: 1]
  |       o- default_tg_pt_gp ....................................................................... [ALUA state: Active/optimized]

glusterfs-storage-qjfqr
+++++++++++++++++++++++
o- / ......................................................................................................................... [...]
  o- backstores .............................................................................................................. [...]
  | o- block .................................................................................................. [Storage Objects: 0]
  | o- fileio ................................................................................................. [Storage Objects: 0]
  | o- pscsi .................................................................................................. [Storage Objects: 0]
  | o- ramdisk ................................................................................................ [Storage Objects: 0]
  | o- user:glfs .............................................................................................. [Storage Objects: 5]
  |   o- vol-39_glusterfs_blockclaim1_0d165c58-eda7-11e8-9f94-0a580a820002  [vol_fb52bfcbc97ebf63b5e540f7da250dd9.47.80/block-store/00dab599-d5c0-4dc7-bf6f-3241ac3558c9 (5.0GiB) activated]
  |   | o- alua ................................................................................................... [ALUA Groups: 1]
  |   |   o- default_tg_pt_gp ....................................................................... [ALUA state: Active/optimized]
  |   o- vol-39_glusterfs_blockclaim2_3cf680ce-eda8-11e8-9f94-0a580a820002  [vol_fb52bfcbc97ebf63b5e540f7da250dd9.47.80/block-store/df906384-cef4-4d2d-abc2-7ab74543afc3 (6.0GiB) activated]
  |   | o- alua ................................................................................................... [ALUA Groups: 1]
  |   |   o- default_tg_pt_gp ....................................................................... [ALUA state: Active/optimized]
  |   o- vol-39_glusterfs_blockclaim3_4c1e4cbf-eda8-11e8-9f94-0a580a820002  [vol_fb52bfcbc97ebf63b5e540f7da250dd9.47.80/block-store/bc0e2585-9b6a-4179-b06b-d30ffe6ea095 (7.0GiB) activated]
  |   | o- alua ................................................................................................... [ALUA Groups: 1]
  |   |   o- default_tg_pt_gp ....................................................................... [ALUA state: Active/optimized]
  |   o- vol-39_glusterfs_blockclaim4_54687b22-eda8-11e8-9f94-0a580a820002  [vol_fb52bfcbc97ebf63b5e540f7da250dd9.47.80/block-store/a97f81e6-6eb0-48d2-8c3f-820a2e2c98c9 (4.0GiB) activated]
  |   | o- alua ................................................................................................... [ALUA Groups: 1]
  |   |   o- default_tg_pt_gp ....................................................................... [ALUA state: Active/optimized]
  |   o- vol-39_glusterfs_blockclaim5_5eaadd88-eda8-11e8-9f94-0a580a820002  [vol_fb52bfcbc97ebf63b5e540f7da250dd9.47.80/block-store/94ab3b93-8dd3-4a38-8809-49dc6c895f42 (8.0GiB) activated]
  |     o- alua ................................................................................................... [ALUA Groups: 1]
  |       o- default_tg_pt_gp ....................................................................... [ALUA state: Active/optimized]


[root@dhcp46-160 ~]# oc rsh glusterfs-storage-t788q
sh-4.2# cat /etc/target/saveconfig.json |grep iqn
      "wwn":"iqn.2016-12.org.gluster-block:00dab599-d5c0-4dc7-bf6f-3241ac3558c9"
      "wwn":"iqn.2016-12.org.gluster-block:df906384-cef4-4d2d-abc2-7ab74543afc3"
      "wwn":"iqn.2016-12.org.gluster-block:bc0e2585-9b6a-4179-b06b-d30ffe6ea095"
      "wwn":"iqn.2016-12.org.gluster-block:a97f81e6-6eb0-48d2-8c3f-820a2e2c98c9"
      "wwn":"iqn.2016-12.org.gluster-block:94ab3b93-8dd3-4a38-8809-49dc6c895f42"





After upgrade :
=======================
[root@dhcp46-160 ~]# oc rsh glusterfs-storage-qsrz5
sh-4.2# targetcli ls
o- / ......................................................................................................................... [...]
  o- backstores .............................................................................................................. [...]
  | o- block .................................................................................................. [Storage Objects: 0]
  | o- fileio ................................................................................................. [Storage Objects: 0]
  | o- pscsi .................................................................................................. [Storage Objects: 0]
  | o- ramdisk ................................................................................................ [Storage Objects: 0]
  | o- user:glfs .............................................................................................. [Storage Objects: 5]
  |   o- vol-39_glusterfs_blockclaim1_0d165c58-eda7-11e8-9f94-0a580a820002  [vol_fb52bfcbc97ebf63b5e540f7da250dd9.47.37/block-store/00dab599-d5c0-4dc7-bf6f-3241ac3558c9 (5.0GiB) activated]
  |   | o- alua ................................................................................................... [ALUA Groups: 3]
  |   |   o- default_tg_pt_gp ....................................................................... [ALUA state: Active/optimized]
  |   |   o- glfs_tg_pt_gp_ano .................................................................. [ALUA state: Active/non-optimized]
  |   |   o- glfs_tg_pt_gp_ao ....................................................................... [ALUA state: Active/optimized]
  |   o- vol-39_glusterfs_blockclaim2_3cf680ce-eda8-11e8-9f94-0a580a820002  [vol_fb52bfcbc97ebf63b5e540f7da250dd9.47.37/block-store/df906384-cef4-4d2d-abc2-7ab74543afc3 (6.0GiB) activated]
  |   | o- alua ................................................................................................... [ALUA Groups: 3]
  |   |   o- default_tg_pt_gp ....................................................................... [ALUA state: Active/optimized]
  |   |   o- glfs_tg_pt_gp_ano .................................................................. [ALUA state: Active/non-optimized]
  |   |   o- glfs_tg_pt_gp_ao ....................................................................... [ALUA state: Active/optimized]
  |   o- vol-39_glusterfs_blockclaim3_4c1e4cbf-eda8-11e8-9f94-0a580a820002  [vol_fb52bfcbc97ebf63b5e540f7da250dd9.47.37/block-store/bc0e2585-9b6a-4179-b06b-d30ffe6ea095 (7.0GiB) activated]
  |   | o- alua ................................................................................................... [ALUA Groups: 3]
  |   |   o- default_tg_pt_gp ....................................................................... [ALUA state: Active/optimized]
  |   |   o- glfs_tg_pt_gp_ano .................................................................. [ALUA state: Active/non-optimized]
  |   |   o- glfs_tg_pt_gp_ao ....................................................................... [ALUA state: Active/optimized]
  |   o- vol-39_glusterfs_blockclaim4_54687b22-eda8-11e8-9f94-0a580a820002  [vol_fb52bfcbc97ebf63b5e540f7da250dd9.47.37/block-store/a97f81e6-6eb0-48d2-8c3f-820a2e2c98c9 (4.0GiB) activated]
  |   | o- alua ................................................................................................... [ALUA Groups: 3]
  |   |   o- default_tg_pt_gp ....................................................................... [ALUA state: Active/optimized]
  |   |   o- glfs_tg_pt_gp_ano .................................................................. [ALUA state: Active/non-optimized]
  |   |   o- glfs_tg_pt_gp_ao ....................................................................... [ALUA state: Active/optimized]
  |   o- vol-39_glusterfs_blockclaim5_5eaadd88-eda8-11e8-9f94-0a580a820002  [vol_fb52bfcbc97ebf63b5e540f7da250dd9.47.37/block-store/94ab3b93-8dd3-4a38-8809-49dc6c895f42 (8.0GiB) activated]
  |     o- alua ................................................................................................... [ALUA Groups: 3]
  |       o- default_tg_pt_gp ....................................................................... [ALUA state: Active/optimized]
  |       o- glfs_tg_pt_gp_ano .................................................................. [ALUA state: Active/non-optimized]
  |       o- glfs_tg_pt_gp_ao ....................................................................... [ALUA state: Active/optimized]
  o- iscsi ............................................................................................................ [Targets: 5]
  | o- iqn.2016-12.org.gluster-block:00dab599-d5c0-4dc7-bf6f-3241ac3558c9 ................................................ [TPGs: 3]
  | | o- tpg1 ..................................................................................... [gen-acls, tpg-auth, 1-way auth]
  | | | o- acls .......................................................................................................... [ACLs: 0]
  | | | o- luns .......................................................................................................... [LUNs: 1]
  | | | | o- lun0 ...................... [user/vol-39_glusterfs_blockclaim1_0d165c58-eda7-11e8-9f94-0a580a820002 (glfs_tg_pt_gp_ao)]
  | | | o- portals .................................................................................................... [Portals: 1]
  | | |   o- 10.70.47.37:3260 ................................................................................................. [OK]
  | | o- tpg2 ........................................................................................................... [disabled]
  | | | o- acls .......................................................................................................... [ACLs: 0]
  | | | o- luns .......................................................................................................... [LUNs: 1]
  | | | | o- lun0 ..................... [user/vol-39_glusterfs_blockclaim1_0d165c58-eda7-11e8-9f94-0a580a820002 (glfs_tg_pt_gp_ano)]
  | | | o- portals .................................................................................................... [Portals: 1]
  | | |   o- 10.70.47.80:3260 ................................................................................................. [OK]
  | | o- tpg3 ........................................................................................................... [disabled]
  | |   o- acls .......................................................................................................... [ACLs: 0]
  | |   o- luns .......................................................................................................... [LUNs: 1]
  | |   | o- lun0 ..................... [user/vol-39_glusterfs_blockclaim1_0d165c58-eda7-11e8-9f94-0a580a820002 (glfs_tg_pt_gp_ano)]
  | |   o- portals .................................................................................................... [Portals: 1]
  | |     o- 10.70.46.23:3260 ................................................................................................. [OK]
  | o- iqn.2016-12.org.gluster-block:94ab3b93-8dd3-4a38-8809-49dc6c895f42 ................................................ [TPGs: 3]
  | | o- tpg1 ..................................................................................... [gen-acls, tpg-auth, 1-way auth]
  | | | o- acls .......................................................................................................... [ACLs: 0]
  | | | o- luns .......................................................................................................... [LUNs: 1]
  | | | | o- lun0 ...................... [user/vol-39_glusterfs_blockclaim5_5eaadd88-eda8-11e8-9f94-0a580a820002 (glfs_tg_pt_gp_ao)]
  | | | o- portals .................................................................................................... [Portals: 1]
  | | |   o- 10.70.47.37:3260 ................................................................................................. [OK]
  | | o- tpg2 ........................................................................................................... [disabled]
  | | | o- acls .......................................................................................................... [ACLs: 0]
  | | | o- luns .......................................................................................................... [LUNs: 1]
  | | | | o- lun0 ..................... [user/vol-39_glusterfs_blockclaim5_5eaadd88-eda8-11e8-9f94-0a580a820002 (glfs_tg_pt_gp_ano)]
  | | | o- portals .................................................................................................... [Portals: 1]
  | | |   o- 10.70.46.23:3260 ................................................................................................. [OK]
  | | o- tpg3 ........................................................................................................... [disabled]
  | |   o- acls .......................................................................................................... [ACLs: 0]
  | |   o- luns .......................................................................................................... [LUNs: 1]
  | |   | o- lun0 ..................... [user/vol-39_glusterfs_blockclaim5_5eaadd88-eda8-11e8-9f94-0a580a820002 (glfs_tg_pt_gp_ano)]
  | |   o- portals .................................................................................................... [Portals: 1]
  | |     o- 10.70.47.80:3260 ................................................................................................. [OK]
  | o- iqn.2016-12.org.gluster-block:a97f81e6-6eb0-48d2-8c3f-820a2e2c98c9 ................................................ [TPGs: 3]
  | | o- tpg1 ........................................................................................................... [disabled]
  | | | o- acls .......................................................................................................... [ACLs: 0]
  | | | o- luns .......................................................................................................... [LUNs: 1]
  | | | | o- lun0 ...................... [user/vol-39_glusterfs_blockclaim4_54687b22-eda8-11e8-9f94-0a580a820002 (glfs_tg_pt_gp_ao)]
  | | | o- portals .................................................................................................... [Portals: 1]
  | | |   o- 10.70.47.80:3260 ................................................................................................. [OK]
  | | o- tpg2 ..................................................................................... [gen-acls, tpg-auth, 1-way auth]
  | | | o- acls .......................................................................................................... [ACLs: 0]
  | | | o- luns .......................................................................................................... [LUNs: 1]
  | | | | o- lun0 ..................... [user/vol-39_glusterfs_blockclaim4_54687b22-eda8-11e8-9f94-0a580a820002 (glfs_tg_pt_gp_ano)]
  | | | o- portals .................................................................................................... [Portals: 1]
  | | |   o- 10.70.47.37:3260 ................................................................................................. [OK]
  | | o- tpg3 ........................................................................................................... [disabled]
  | |   o- acls .......................................................................................................... [ACLs: 0]
  | |   o- luns .......................................................................................................... [LUNs: 1]
  | |   | o- lun0 ..................... [user/vol-39_glusterfs_blockclaim4_54687b22-eda8-11e8-9f94-0a580a820002 (glfs_tg_pt_gp_ano)]
  | |   o- portals .................................................................................................... [Portals: 1]
  | |     o- 10.70.46.23:3260 ................................................................................................. [OK]
  | o- iqn.2016-12.org.gluster-block:bc0e2585-9b6a-4179-b06b-d30ffe6ea095 ................................................ [TPGs: 3]
  | | o- tpg1 ..................................................................................... [gen-acls, tpg-auth, 1-way auth]
  | | | o- acls .......................................................................................................... [ACLs: 0]
  | | | o- luns .......................................................................................................... [LUNs: 1]
  | | | | o- lun0 ..................... [user/vol-39_glusterfs_blockclaim3_4c1e4cbf-eda8-11e8-9f94-0a580a820002 (glfs_tg_pt_gp_ano)]
  | | | o- portals .................................................................................................... [Portals: 1]
  | | |   o- 10.70.47.37:3260 ................................................................................................. [OK]
  | | o- tpg2 ........................................................................................................... [disabled]
  | | | o- acls .......................................................................................................... [ACLs: 0]
  | | | o- luns .......................................................................................................... [LUNs: 1]
  | | | | o- lun0 ...................... [user/vol-39_glusterfs_blockclaim3_4c1e4cbf-eda8-11e8-9f94-0a580a820002 (glfs_tg_pt_gp_ao)]
  | | | o- portals .................................................................................................... [Portals: 1]
  | | |   o- 10.70.47.80:3260 ................................................................................................. [OK]
  | | o- tpg3 ........................................................................................................... [disabled]
  | |   o- acls .......................................................................................................... [ACLs: 0]
  | |   o- luns .......................................................................................................... [LUNs: 1]
  | |   | o- lun0 ..................... [user/vol-39_glusterfs_blockclaim3_4c1e4cbf-eda8-11e8-9f94-0a580a820002 (glfs_tg_pt_gp_ano)]
  | |   o- portals .................................................................................................... [Portals: 1]
  | |     o- 10.70.46.23:3260 ................................................................................................. [OK]
  | o- iqn.2016-12.org.gluster-block:df906384-cef4-4d2d-abc2-7ab74543afc3 ................................................ [TPGs: 3]
  |   o- tpg1 ..................................................................................... [gen-acls, tpg-auth, 1-way auth]
  |   | o- acls .......................................................................................................... [ACLs: 0]
  |   | o- luns .......................................................................................................... [LUNs: 1]
  |   | | o- lun0 ..................... [user/vol-39_glusterfs_blockclaim2_3cf680ce-eda8-11e8-9f94-0a580a820002 (glfs_tg_pt_gp_ano)]
  |   | o- portals .................................................................................................... [Portals: 1]
  |   |   o- 10.70.47.37:3260 ................................................................................................. [OK]
  |   o- tpg2 ........................................................................................................... [disabled]
  |   | o- acls .......................................................................................................... [ACLs: 0]
  |   | o- luns .......................................................................................................... [LUNs: 1]
  |   | | o- lun0 ...................... [user/vol-39_glusterfs_blockclaim2_3cf680ce-eda8-11e8-9f94-0a580a820002 (glfs_tg_pt_gp_ao)]
  |   | o- portals .................................................................................................... [Portals: 1]
  |   |   o- 10.70.46.23:3260 ................................................................................................. [OK]
  |   o- tpg3 ........................................................................................................... [disabled]
  |     o- acls .......................................................................................................... [ACLs: 0]
  |     o- luns .......................................................................................................... [LUNs: 1]
  |     | o- lun0 ..................... [user/vol-39_glusterfs_blockclaim2_3cf680ce-eda8-11e8-9f94-0a580a820002 (glfs_tg_pt_gp_ano)]
  |     o- portals .................................................................................................... [Portals: 1]
  |       o- 10.70.47.80:3260 ................................................................................................. [OK]
  o- loopback ......................................................................................................... [Targets: 0]


created new block volume with new sc :
==========================================

 o- vol-39_glusterfs_blockclaim7_89ce2133-ef3e-11e8-8cef-0a580a80004c  [vol_fb52bfcbc97ebf63b5e540f7da250dd9.47.37/block-store/c662d970-3c25-4606-9ce8-99406f93db1c (9.0GiB) activated]
  |     o- alua ................................................................................................... [ALUA Groups: 3]
  |       o- default_tg_pt_gp ....................................................................... [ALUA state: Active/optimized]
  |       o- glfs_tg_pt_gp_ano .................................................................. [ALUA state: Active/non-optimized]
  |       o- glfs_tg_pt_gp_ao ....................................................................... [ALUA state: Active/optimized]


PRIO_PATH set for new block volume created :
==================================================
[root@dhcp46-160 block-meta]# cat vol-39_glusterfs_blockclaim7_89ce2133-ef3e-11e8-8cef-0a580a80004c
VOLUME: vol_fb52bfcbc97ebf63b5e540f7da250dd9
GBID: c662d970-3c25-4606-9ce8-99406f93db1c
HA: 3
ENTRYCREATE: INPROGRESS
PRIOPATH: 10.70.46.222
SIZE: 9663676416
RINGBUFFER: 0
ENTRYCREATE: SUCCESS
PASSWORD: 4908c303-978d-48dc-b4bb-1eb12f4dbdb6
10.70.47.37: CONFIGINPROGRESS
10.70.46.222: CONFIGINPROGRESS
10.70.46.23: CONFIGINPROGRESS
10.70.46.222: CONFIGSUCCESS
10.70.46.222: AUTHENFORCED
10.70.47.37: CONFIGSUCCESS
10.70.47.37: AUTHENFORCED
10.70.46.23: CONFIGSUCCESS
10.70.46.23: AUTHENFORCED

sh-4.2# cat /etc/target/saveconfig.json |grep iqn
      "wwn": "iqn.2016-12.org.gluster-block:00dab599-d5c0-4dc7-bf6f-3241ac3558c9"
      "wwn": "iqn.2016-12.org.gluster-block:df906384-cef4-4d2d-abc2-7ab74543afc3"
      "wwn": "iqn.2016-12.org.gluster-block:bc0e2585-9b6a-4179-b06b-d30ffe6ea095"
      "wwn": "iqn.2016-12.org.gluster-block:a97f81e6-6eb0-48d2-8c3f-820a2e2c98c9"
      "wwn": "iqn.2016-12.org.gluster-block:94ab3b93-8dd3-4a38-8809-49dc6c895f42"
      "wwn": "iqn.2016-12.org.gluster-block:c662d970-3c25-4606-9ce8-99406f93db1c"

[root@dhcp46-160 ~]# cat /var/log/dmesg | grep -i alua
[    1.266705] alua: device handler registered


[root@dhcp46-160 ~]# cat /var/log/dmesg | grep 'alua not supported'
[root@dhcp46-160 ~]#

[root@dhcp46-160 log]# cat dmesg |grep 'failed in domap for addition of new path'
[root@dhcp46-160 log]# 


sh-4.2# systemctl cat tcmu-runner
# /usr/lib/systemd/system/tcmu-runner.service
[Unit]
Description=LIO Userspace-passthrough daemon
After=glusterd.service

[Service]
LimitNOFILE=1000000
Type=dbus
BusName=org.kernel.TCMUService1
KillMode=process
EnvironmentFile=-/etc/sysconfig/tcmu-runner-params
ExecStartPre=-/usr/libexec/gluster-block/upgrade_activities.sh
EnvironmentFile=-/etc/sysconfig/tcmu-runner-params
ExecStart=/usr/bin/tcmu-runner --tcmu-log-dir $TCMU_LOGDIR
TimeoutStartSec=600
sh-4.2# ip a|grep 10.70
    inet 10.70.46.23/22 brd 10.70.47.255 scope global noprefixroute dynamic ens192
sh-4.2# ls -l /etc/target/
total 80
drwxr-xr-x. 2 root root   252 Nov 23 16:40 backup
-rw-------. 1 root root 27812 Nov 23 16:40 saveconfig.json
-rw-------. 1 root root 50703 Nov 22 06:33 saveconfig.json-bak-2018-11-23-10:04:20
sh-4.2# ls -l /var/lib/gluster-block/gb_upgrade.status
-rw-r--r--. 1 root root 0 Nov 23 10:04 /var/lib/gluster-block/gb_upgrade.status
sh-4.2# date
Fri Nov 23 17:06:15 UTC 2018
sh-4.2#  systemctl restart tcmu-runner
sh-4.2# tail -f /var/log/glusterfs/gluster-block/gluster-block-bricks-start.log
[2018-11-23 06:08:51] INFO: All bricks for Block Hosting Volumes came online in 17 seconds
[2018-11-23 10:04:19] INFO: All bricks for Block Hosting Volumes came online in 54 seconds
[2018-11-23 10:24:29] INFO: All bricks for Block Hosting Volumes came online in 31 seconds
[2018-11-23 01:10:16] INFO: All bricks for Block Hosting Volumes came online in 18 seconds
[2018-11-23 05:06:40] INFO: All bricks for Block Hosting Volumes came online in 0 seconds


sh-4.2# tail -f  /var/log/glusterfs/gluster-block/gluster-block-upgrade-activities.log
[2018-11-23 06:08:51] INFO: started upgrade activities
[2018-11-23 06:08:51] ERROR: Gluster-blockd is not running
[2018-11-23 10:04:19] INFO: started upgrade activities
[2018-11-23 10:04:20] INFO: successfully completed the upgrade activities
[2018-11-23 10:24:29] INFO: skipping upgrade activities
[2018-11-23 01:10:16] INFO: skipping upgrade activities
[2018-11-23 05:06:40] INFO: skipping upgrade activities


Have not specifically verified the upgrade_activities.sh script because this has already been verified in the other bug https://bugzilla.redhat.com/show_bug.cgi?id=1638988#c7 and the fixed in version is same for BUG 1637688 and 1638988

Comment 28 RamaKasturi 2018-11-24 16:23:47 UTC

Hello Neha,

   I am planning to develop below cases for this bug and let me know if the test case list looks good or if you would like me to include anything else as part of this.

Verify if targetcli ls displays alua supported for old block volumes once upgraded from cns3.9 to cns3.11

Verify creating new block volumes on the upgraded setup  has the functionality of alua included

Verify restarting tcmu-runner does not trigger upgrade_activities.sh 

Verify pod restart does not trigger upgrade_activities.sh 

Validate upgrade_activities.sh script

Thanks
kasturi

Comment 29 Neha Berry 2019-01-31 11:23:09 UTC

(In reply to RamaKasturi from comment #28)
> Hello Neha,
> 
>    I am planning to develop below cases for this bug and let me know if the
> test case list looks good or if you would like me to include anything else
> as part of this.
> 
> Verify if targetcli ls displays alua supported for old block volumes once
> upgraded from cns3.9 to cns3.11
> 
> Verify creating new block volumes on the upgraded setup  has the
> functionality of alua included
> 
> Verify restarting tcmu-runner does not trigger upgrade_activities.sh 
> 
> Verify pod restart does not trigger upgrade_activities.sh 
> 
> Validate upgrade_activities.sh script
> 
> Thanks
> kasturi

Hi Kasturi,

these tests look good. But maybe we can confirm with Prasanna if any other coverage is required.

Also, for test#1 below:

 Verify if targetcli ls displays alua supported for old block volumes once
> upgraded from cns3.9 to cns3.11

we would need to edit the multipath.conf file to include "alua" once all pods are upgraded.

Regards,
neha

Comment 31 errata-xmlrpc 2019-02-07 03:38:29 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:0285

Note You need to log in before you can comment on or make changes to this bug.

akhakhar
bgoyal
bkunal
grodrigu
hchiramm
jarrpa
knarra
kramdoss
madam
nberry
pkarampu
pprakash
prasanna.kalever
rhs-bugs
rtalur
sankarshan
sarora
vbellur
vinug
xiubli