Bug 1314489 - [RHEV-H] iSCSI initiatorname changing back to default after reboot
[RHEV-H] iSCSI initiatorname changing back to default after reboot
Status: CLOSED ERRATA
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: ovirt-node (Show other bugs)
3.5.7
All Linux
high Severity high
: ovirt-4.0.0-beta
: ---
Assigned To: Fabian Deutsch
cshao
: OtherQA, ZStream
Depends On:
Blocks: 1322787
  Show dependency treegraph
 
Reported: 2016-03-03 13:16 EST by Javier Coscia
Modified: 2016-08-23 17:05 EDT (History)
13 users (show)

See Also:
Fixed In Version: rhev-hypervisor7-ng-3.6-20160429.0
Doc Type: Bug Fix
Doc Text:
Previously, the iSCSI initiator name was lost after reboot. Now, the iSCSI initiator name is persistent and is retained after reboot.
Story Points: ---
Clone Of:
: 1322787 (view as bug list)
Environment:
Last Closed: 2016-08-23 17:05:37 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: Node
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
ycui: testing_plan_complete?


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
oVirt gerrit 54616 master ABANDONED ovirt-early.service: restart iscsid after persistence 2016-03-17 10:10 EDT
oVirt gerrit 54723 None None None 2016-03-23 10:37 EDT
Red Hat Product Errata RHBA-2016:1702 normal SHIPPED_LIVE redhat-virtualization-host bug fix and enhancement update for RHV 4.0 2016-08-23 20:35:27 EDT

  None (edit)
Description Javier Coscia 2016-03-03 13:16:14 EST
Description of problem:

- After first reboot, RHEV-H will remain in non-operational state because it can't reach iSCSI LUNs.
- /etc/iscsi/initiatorname.iscsi and /config/etc/iscsi/initiatorname.iscsi have the correct/expected values
- iscsiadm -m session -P3 shows the default initiator name

Version-Release number of selected component (if applicable):

- Discovered in RHEV-H version 7.2 - 20160105.2

How reproducible:

- 100% in customer's environment

Steps to Reproduce:

1) install a new RHEV-H7.2
2) configure iscsi, Hypervisor state is UP.
3) put the Hypervisor into maintenance.
4) reboot.
5) when the Hypervisor is back up, it is in 'non-operational' state because there is no iscsi configuration.

Actual results:

- Hypervisor in non-operational state because can't reach iSCSI LUNs because is using a default initiatorname

Expected results:

- iSCSI sessions should use the initiatorname configured during installation and survive a reboot

Additional info:

- Storage is IBM Storwize V7000.
Comment 5 Javier Coscia 2016-03-04 07:01:07 EST
Here's a workaround the architect of the consulting team did:
===================================================================

As a workaround for being able to use the Hypervisor I did the following things:

1. Set Initiator Name on Setup

2. Fill the file /var/lib/stateless/writable/etc/iscsi/initiatorname.iscsi with the correct initiator info.
cat /etc/iscsi/initiatorname.iscsi > /var/lib/stateless/writable/etc/iscsi/initiatorname.iscsi

3. Persist the file:
persist /var/lib/stateless/writable/etc/iscsi/initiatorname.iscsi

4. Mod the vdsmd systemd unit adding the following:
ExecStartPost=/usr/bin/systemctl restart iscsi.service
ExecStartPost=/usr/bin/systemctl restart iscsid.service


cp /usr/lib/systemd/system/vdsmd.service /etc/systemd/system/vdsmd.service

cat << EOF > /etc/systemd/system/vdsmd.service
[Unit]
Description=Virtual Desktop Server Manager
Requires=multipathd.service libvirtd.service time-sync.target \
         iscsid.service rpcbind.service supervdsmd.service sanlock.service \
         vdsm-network.service
After=vdsm-network.service
Conflicts=libvirt-guests.service ksmtuned.service

[Service]
Type=simple
LimitCORE=infinity
EnvironmentFile=-/etc/sysconfig/vdsm
ExecStartPre=/usr/libexec/vdsm/vdsmd_init_common.sh --pre-start
ExecStart=/usr/share/vdsm/daemonAdapter -0 /dev/null -1 /dev/null -2 /dev/null "/usr/share/vdsm/vdsm"
ExecStartPost=/usr/bin/systemctl restart iscsi.service
ExecStartPost=/usr/bin/systemctl restart iscsid.service
ExecStopPost=/usr/libexec/vdsm/vdsmd_init_common.sh --post-stop
Restart=always
Nice=-20
User=vdsm
Group=kvm
PermissionsStartOnly=true
TimeoutStopSec=10

[Install]
WantedBy=multi-user.target
EOF

5. Persist the custom vdsmd systemd unit:

persist /etc/systemd/system/vdsmd.service

6. Reload systemd units

systemctl daemon-reload

7. Restart vdsmd unit
systemctl restart vdsmd.service

8. Check that the ExecStartPost, were successfully executed
systemctl status vdsmd.service


● vdsmd.service - Virtual Desktop Server Manager
   Loaded: loaded (/etc/systemd/system/vdsmd.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2016-03-03 22:58:09 UTC; 6min ago
  Process: 18510 ExecStartPost=/usr/bin/systemctl restart iscsid.service (code=exited, status=0/SUCCESS)
  Process: 18507 ExecStartPost=/usr/bin/systemctl restart iscsi.service (code=exited, status=0/SUCCESS)

For now with this steps we can reboot the hypervisor after being in maintenance and the connection to the storage were done successfully with the right Initiator.
Comment 6 Fabian Deutsch 2016-03-04 07:10:54 EST
Thanks.

That indicates that iscsi.service and/or iscsid.service come up before the persistence mechanism kicked in. We can probably solve it by running the persistence logic before the two services.
Comment 7 cshao 2016-03-07 04:19:14 EST
I can't reproduce this bug.

Test version:
RHEV-H version 7.2 - 20160105.2
ovirt-node-3.2.3-30.el7.noarch
rhevm-3.6.3.4-0.1.el6

Test steps:
1) install a new RHEV-H7.2
2) Set remote storage in TUI
3) register RHEV-H to RHEV-M.
4) configure iscsi, Hypervisor state is UP.
5) put the Hypervisor into maintenance.
6) reboot twice.

Test result:
RHEV-H can up after reboot, and the connection to the iscsi storage were done successfully with the right Initiator.

Hi jcoscia,

  Could you help me have a look what did I do wrong?

Thanks!
Comment 8 Adrian Andrade 2016-03-07 10:26:36 EST
Hey shaochen, 

Actually this is the Production Environment versions:
 
Red Hat Enterprise Virtualization Hypervisor release 7.2 (20160219.0.el7ev)
ovirt-node-3.2.3-31.el7.noarch 
rhevm-3.5.7-0.1.el6ev.noarch 

As you can see it defers a little, the other important thing here is that the SAN restricts the connections to an iSCSI Initiator Name, could you recheck the following output: 

iscsiadm -m session -P 1 | grep "Initiatorname" 

And compare it to:

cat /etc/iscsi/initiatorname.iscsi 

And also to:

cat /var/lib/stateless/writable/etc/iscsi/initiatorname.iscsi 

Thanks 

Adrian
Comment 9 Douglas Schilling Landgraf 2016-03-07 17:50:10 EST
(In reply to Adrian Andrade from comment #8)
> Hey shaochen, 
> 
> Actually this is the Production Environment versions:
>  
> Red Hat Enterprise Virtualization Hypervisor release 7.2 (20160219.0.el7ev)
> ovirt-node-3.2.3-31.el7.noarch 
> rhevm-3.5.7-0.1.el6ev.noarch 
> 
> As you can see it defers a little, the other important thing here is that
> the SAN restricts the connections to an iSCSI Initiator Name, could you
> recheck the following output: 
> 
> iscsiadm -m session -P 1 | grep "Initiatorname" 
> 
> And compare it to:
> 
> cat /etc/iscsi/initiatorname.iscsi 
> 
> And also to:
> 
> cat /var/lib/stateless/writable/etc/iscsi/initiatorname.iscsi 
> 
> Thanks 
> 
> Adrian

I cannot reproduce the report as well.

Test steps:
1) install a new RHEV-H7.2
2) Set remote storage in TUI
3) register RHEV-H to RHEV-M.
4) configure iscsi, Hypervisor state is UP.
5) put the Hypervisor into maintenance.
6) reboot 

After that, I activated my host and it became up and storage as well.

# cat /etc/redhat-release 
Red Hat Enterprise Virtualization Hypervisor release 7.2 (20160219.0.el7ev)
[root@localhost ~]# iscsiadm -m session -P 1 | grep "Initiatorname" 
		Iface Initiatorname: iqn.1992-04.com.emc:storage.storage.data

# cat /etc/iscsi/initiatorname.iscsi 
InitiatorName=iqn.1992-04.com.emc:storage.storage.data

# cat /var/lib/stateless/writable/etc/iscsi/initiatorname.iscsi 
InitiatorName=iqn.1994-05.com.redhat:5a1a12a62ae

I have tested multiple times so far, no reproducer. Shaochen let me know if you are able to reproduce.

Thanks!
Comment 10 cshao 2016-03-08 02:13:29 EST
(In reply to Adrian Andrade from comment #8)
> Hey shaochen, 
> 
> Actually this is the Production Environment versions:
>  
> Red Hat Enterprise Virtualization Hypervisor release 7.2 (20160219.0.el7ev)
> ovirt-node-3.2.3-31.el7.noarch 
> rhevm-3.5.7-0.1.el6ev.noarch 
> 
> As you can see it defers a little, the other important thing here is that
> the SAN restricts the connections to an iSCSI Initiator Name, could you
> recheck the following output: 
> 
> iscsiadm -m session -P 1 | grep "Initiatorname" 
> 
> And compare it to:
> 
> cat /etc/iscsi/initiatorname.iscsi 
> 
> And also to:
> 
> cat /var/lib/stateless/writable/etc/iscsi/initiatorname.iscsi 
> 
> Thanks 
> 
> Adrian



Still can't reproduce this issue with above build.

# cat /etc/redhat-release 
Red Hat Enterprise Virtualization Hypervisor release 7.2 (20160219.0.el7ev)
rhevm-3.5.8-0.1.el6ev
ovirt-node-3.2.3-31.el7.noarch


Before reboot:
# iscsiadm -m session -P 1 | grep "Initiatorname" 
		Iface Initiatorname: iqn.1994-05.com.redhat:rhevh-1
		Iface Initiatorname: iqn.1994-05.com.redhat:rhevh-1
		Iface Initiatorname: iqn.1994-05.com.redhat:rhevh-1

# cat /etc/iscsi/initiatorname.iscsi 
InitiatorName=iqn.1994-05.com.redhat:rhevh-1

# cat /var/lib/stateless/writable/etc/iscsi/initiatorname.iscsi 
InitiatorName=iqn.1994-05.com.redhat:5a1a12a62ae



After reboot:
# iscsiadm -m session -P 1 | grep "Initiatorname" 
		Iface Initiatorname: iqn.1994-05.com.redhat:rhevh-1
 
# cat /etc/iscsi/initiatorname.iscsi 
InitiatorName=iqn.1994-05.com.redhat:rhevh-1
 
# cat /var/lib/stateless/writable/etc/iscsi/initiatorname.iscsi 
InitiatorName=iqn.1994-05.com.redhat:5a1a12a62ae
Comment 20 Mike McCune 2016-03-28 18:30:10 EDT
This bug was accidentally moved from POST to MODIFIED via an error in automation, please see mmccune@redhat.com with any questions
Comment 29 cshao 2016-06-02 05:33:06 EDT
I am also did some test on rhev-hypervisor7-ng-3.6-20160518.0 build,

Test version:
rhev-hypervisor7-ng-3.6-20160518.0 
imgbased-0.6-0.1.el7ev.noarch

Test steps:
1) install RHEV-H-NG
2) Add RHEV-H to engine.
3) configure iscsi, Hypervisor state is UP.
4) put the Hypervisor into maintenance.
5) reboot.

Test result:
After step5, RHEV-H-ng can up after reboot, and the connection to the iscsi storage were done successfully with the right Initiator.

So the bug is fixed, change bug status to VERIFIED.
Comment 31 errata-xmlrpc 2016-08-23 17:05:37 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-1702.html

Note You need to log in before you can comment on or make changes to this bug.