Description of problem: - After first reboot, RHEV-H will remain in non-operational state because it can't reach iSCSI LUNs. - /etc/iscsi/initiatorname.iscsi and /config/etc/iscsi/initiatorname.iscsi have the correct/expected values - iscsiadm -m session -P3 shows the default initiator name Version-Release number of selected component (if applicable): - Discovered in RHEV-H version 7.2 - 20160105.2 How reproducible: - 100% in customer's environment Steps to Reproduce: 1) install a new RHEV-H7.2 2) configure iscsi, Hypervisor state is UP. 3) put the Hypervisor into maintenance. 4) reboot. 5) when the Hypervisor is back up, it is in 'non-operational' state because there is no iscsi configuration. Actual results: - Hypervisor in non-operational state because can't reach iSCSI LUNs because is using a default initiatorname Expected results: - iSCSI sessions should use the initiatorname configured during installation and survive a reboot Additional info: - Storage is IBM Storwize V7000.
Here's a workaround the architect of the consulting team did: =================================================================== As a workaround for being able to use the Hypervisor I did the following things: 1. Set Initiator Name on Setup 2. Fill the file /var/lib/stateless/writable/etc/iscsi/initiatorname.iscsi with the correct initiator info. cat /etc/iscsi/initiatorname.iscsi > /var/lib/stateless/writable/etc/iscsi/initiatorname.iscsi 3. Persist the file: persist /var/lib/stateless/writable/etc/iscsi/initiatorname.iscsi 4. Mod the vdsmd systemd unit adding the following: ExecStartPost=/usr/bin/systemctl restart iscsi.service ExecStartPost=/usr/bin/systemctl restart iscsid.service cp /usr/lib/systemd/system/vdsmd.service /etc/systemd/system/vdsmd.service cat << EOF > /etc/systemd/system/vdsmd.service [Unit] Description=Virtual Desktop Server Manager Requires=multipathd.service libvirtd.service time-sync.target \ iscsid.service rpcbind.service supervdsmd.service sanlock.service \ vdsm-network.service After=vdsm-network.service Conflicts=libvirt-guests.service ksmtuned.service [Service] Type=simple LimitCORE=infinity EnvironmentFile=-/etc/sysconfig/vdsm ExecStartPre=/usr/libexec/vdsm/vdsmd_init_common.sh --pre-start ExecStart=/usr/share/vdsm/daemonAdapter -0 /dev/null -1 /dev/null -2 /dev/null "/usr/share/vdsm/vdsm" ExecStartPost=/usr/bin/systemctl restart iscsi.service ExecStartPost=/usr/bin/systemctl restart iscsid.service ExecStopPost=/usr/libexec/vdsm/vdsmd_init_common.sh --post-stop Restart=always Nice=-20 User=vdsm Group=kvm PermissionsStartOnly=true TimeoutStopSec=10 [Install] WantedBy=multi-user.target EOF 5. Persist the custom vdsmd systemd unit: persist /etc/systemd/system/vdsmd.service 6. Reload systemd units systemctl daemon-reload 7. Restart vdsmd unit systemctl restart vdsmd.service 8. Check that the ExecStartPost, were successfully executed systemctl status vdsmd.service ● vdsmd.service - Virtual Desktop Server Manager Loaded: loaded (/etc/systemd/system/vdsmd.service; enabled; vendor preset: disabled) Active: active (running) since Thu 2016-03-03 22:58:09 UTC; 6min ago Process: 18510 ExecStartPost=/usr/bin/systemctl restart iscsid.service (code=exited, status=0/SUCCESS) Process: 18507 ExecStartPost=/usr/bin/systemctl restart iscsi.service (code=exited, status=0/SUCCESS) For now with this steps we can reboot the hypervisor after being in maintenance and the connection to the storage were done successfully with the right Initiator.
Thanks. That indicates that iscsi.service and/or iscsid.service come up before the persistence mechanism kicked in. We can probably solve it by running the persistence logic before the two services.
I can't reproduce this bug. Test version: RHEV-H version 7.2 - 20160105.2 ovirt-node-3.2.3-30.el7.noarch rhevm-3.6.3.4-0.1.el6 Test steps: 1) install a new RHEV-H7.2 2) Set remote storage in TUI 3) register RHEV-H to RHEV-M. 4) configure iscsi, Hypervisor state is UP. 5) put the Hypervisor into maintenance. 6) reboot twice. Test result: RHEV-H can up after reboot, and the connection to the iscsi storage were done successfully with the right Initiator. Hi jcoscia, Could you help me have a look what did I do wrong? Thanks!
Hey shaochen, Actually this is the Production Environment versions: Red Hat Enterprise Virtualization Hypervisor release 7.2 (20160219.0.el7ev) ovirt-node-3.2.3-31.el7.noarch rhevm-3.5.7-0.1.el6ev.noarch As you can see it defers a little, the other important thing here is that the SAN restricts the connections to an iSCSI Initiator Name, could you recheck the following output: iscsiadm -m session -P 1 | grep "Initiatorname" And compare it to: cat /etc/iscsi/initiatorname.iscsi And also to: cat /var/lib/stateless/writable/etc/iscsi/initiatorname.iscsi Thanks Adrian
(In reply to Adrian Andrade from comment #8) > Hey shaochen, > > Actually this is the Production Environment versions: > > Red Hat Enterprise Virtualization Hypervisor release 7.2 (20160219.0.el7ev) > ovirt-node-3.2.3-31.el7.noarch > rhevm-3.5.7-0.1.el6ev.noarch > > As you can see it defers a little, the other important thing here is that > the SAN restricts the connections to an iSCSI Initiator Name, could you > recheck the following output: > > iscsiadm -m session -P 1 | grep "Initiatorname" > > And compare it to: > > cat /etc/iscsi/initiatorname.iscsi > > And also to: > > cat /var/lib/stateless/writable/etc/iscsi/initiatorname.iscsi > > Thanks > > Adrian I cannot reproduce the report as well. Test steps: 1) install a new RHEV-H7.2 2) Set remote storage in TUI 3) register RHEV-H to RHEV-M. 4) configure iscsi, Hypervisor state is UP. 5) put the Hypervisor into maintenance. 6) reboot After that, I activated my host and it became up and storage as well. # cat /etc/redhat-release Red Hat Enterprise Virtualization Hypervisor release 7.2 (20160219.0.el7ev) [root@localhost ~]# iscsiadm -m session -P 1 | grep "Initiatorname" Iface Initiatorname: iqn.1992-04.com.emc:storage.storage.data # cat /etc/iscsi/initiatorname.iscsi InitiatorName=iqn.1992-04.com.emc:storage.storage.data # cat /var/lib/stateless/writable/etc/iscsi/initiatorname.iscsi InitiatorName=iqn.1994-05.com.redhat:5a1a12a62ae I have tested multiple times so far, no reproducer. Shaochen let me know if you are able to reproduce. Thanks!
(In reply to Adrian Andrade from comment #8) > Hey shaochen, > > Actually this is the Production Environment versions: > > Red Hat Enterprise Virtualization Hypervisor release 7.2 (20160219.0.el7ev) > ovirt-node-3.2.3-31.el7.noarch > rhevm-3.5.7-0.1.el6ev.noarch > > As you can see it defers a little, the other important thing here is that > the SAN restricts the connections to an iSCSI Initiator Name, could you > recheck the following output: > > iscsiadm -m session -P 1 | grep "Initiatorname" > > And compare it to: > > cat /etc/iscsi/initiatorname.iscsi > > And also to: > > cat /var/lib/stateless/writable/etc/iscsi/initiatorname.iscsi > > Thanks > > Adrian Still can't reproduce this issue with above build. # cat /etc/redhat-release Red Hat Enterprise Virtualization Hypervisor release 7.2 (20160219.0.el7ev) rhevm-3.5.8-0.1.el6ev ovirt-node-3.2.3-31.el7.noarch Before reboot: # iscsiadm -m session -P 1 | grep "Initiatorname" Iface Initiatorname: iqn.1994-05.com.redhat:rhevh-1 Iface Initiatorname: iqn.1994-05.com.redhat:rhevh-1 Iface Initiatorname: iqn.1994-05.com.redhat:rhevh-1 # cat /etc/iscsi/initiatorname.iscsi InitiatorName=iqn.1994-05.com.redhat:rhevh-1 # cat /var/lib/stateless/writable/etc/iscsi/initiatorname.iscsi InitiatorName=iqn.1994-05.com.redhat:5a1a12a62ae After reboot: # iscsiadm -m session -P 1 | grep "Initiatorname" Iface Initiatorname: iqn.1994-05.com.redhat:rhevh-1 # cat /etc/iscsi/initiatorname.iscsi InitiatorName=iqn.1994-05.com.redhat:rhevh-1 # cat /var/lib/stateless/writable/etc/iscsi/initiatorname.iscsi InitiatorName=iqn.1994-05.com.redhat:5a1a12a62ae
This bug was accidentally moved from POST to MODIFIED via an error in automation, please see mmccune with any questions
I am also did some test on rhev-hypervisor7-ng-3.6-20160518.0 build, Test version: rhev-hypervisor7-ng-3.6-20160518.0 imgbased-0.6-0.1.el7ev.noarch Test steps: 1) install RHEV-H-NG 2) Add RHEV-H to engine. 3) configure iscsi, Hypervisor state is UP. 4) put the Hypervisor into maintenance. 5) reboot. Test result: After step5, RHEV-H-ng can up after reboot, and the connection to the iscsi storage were done successfully with the right Initiator. So the bug is fixed, change bug status to VERIFIED.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHBA-2016-1702.html