Created attachment 1009976 [details]
attached sosreport logs for back version 6.5

Yaniv, could this be a side effect of the unified persistence?

RHEV-H 6.5 is using RHEV 3.4.z
RHEV-H 6.6 is using RHEV 3.5.z

IIUIC the once 3.5.z is booted, all network configuration get's migrated to the unified persistence, which includes removing the ifcfg files. If a user now boots back into RHEV-H 6.5/3.4.z, then no ifcfg files are around - and that vdsm does not understand the unified persistence.

Is this correct?

Haiyang, is this a dupe of bug 1194068?

(In reply to Fabian Deutsch from comment #3)
> Haiyang, is this a dupe of bug 1194068?

Seems it 's not duplicated of bug 1194068:
1. bug 1194068 talked about the network interfaces on RHEV hypervisors that have been upgraded are being set to ONBOOT=no from "ONBOOT=yes"
2. Also from https://bugzilla.redhat.com/show_bug.cgi?id=1206536#c2, we known that bug 1194068 has been fixed in rhev-hypervisor6-6.6-20150327.0, but we find bug 1208376 in rhev-hypervisor6-6.6-20150327.0 version

Thanks Haiyang.

But this looks very similar to bug 1209486.
Can you confirm this, Haiyang?

(In reply to Fabian Deutsch from comment #5)
> Thanks Haiyang.
> 
> But this looks very similar to bug 1209486.
> Can you confirm this, Haiyang?

seems it's not similar to bug 1209486.
If it's similar to bug 1209486, according to bug 1209486's Actual results,
For my test step 5 "login the new version 6.6 and check the notwork status", it should be there is no networks exists on host, no ifcfg files for those networks. host has no ip.

but the actual test results is:
After 5, Could found the network configuration files and it's status is up.

Bug 1208376 issue just happen on my test step 6 "Reboot the new version 6.6 and enter into backup 6.5 version and check the notwork status".

(In reply to haiyang,dong from comment #6)
> (In reply to Fabian Deutsch from comment #5)
> > Thanks Haiyang.
> > 
> > But this looks very similar to bug 1209486.
> > Can you confirm this, Haiyang?
> 
> seems it's not similar to bug 1209486.
> If it's similar to bug 1209486, according to bug 1209486's Actual results,
> For my test step 5 "login the new version 6.6 and check the notwork status",
> it should be there is no networks exists on host, no ifcfg files for those
> networks. host has no ip.
> 
> but the actual test results is:
> After 5, Could found the network configuration files and it's status is up.

Seems the root cause happen in here:
After step 5 "login the new version 6.6 and check the notwork status" , i could find the network configuration files in 
[root@dhcp-8-211 admin]# ls /etc/sysconfig/network-scripts/ifcfg-*
/etc/sysconfig/network-scripts/ifcfg-em1
/etc/sysconfig/network-scripts/ifcfg-lo
/etc/sysconfig/network-scripts/ifcfg-rhevm

but configuration files "ifcfg-em1/ifcfg-rhevm" missed in 
[root@dhcp-8-211 admin]# ls /config/etc/sysconfig/network-scripts/ifcfg-*
/config/etc/sysconfig/network-scripts/ifcfg-lo

it's the reason why after step6 "Reboot the new version 6.6 and enter into backup 6.5 version and check the notwork status", found network configuration files of the backup 6.5 version missed and it's status is down.

> 
> Bug 1208376 issue just happen on my test step 6 "Reboot the new version 6.6
> and enter into backup 6.5 version and check the notwork status".

(In reply to haiyang,dong from comment #0)
…
> RHEVH 6.5 20150115
^^ RHEV-H 3.4.z

> rhev-hypervisor6-6.6-20150327.0
^^ RHEV-H 3.5.1


In RHEV 3.5, vdsm will not persist the ifcfg anymore, this breaks the rollback to the old version, because 3.4.z does not know how to bring up the networks from the unified persistence.

I'd think about persisting the ifcfg files again, even for networks owned by vdsm.

Ido, maybe you have an insight if it makes sense to persist the ifcfg files. IMO it might be safer.

IMO it will be safer. It will be also make the system more predictable (similar to rhel) and the code simpler. It will also make our customers, QE and GSS less surprised. This is not a trivial change, but should be done soon.

In my opinion we should solve as much of this mess as possible while we still remember what's going on :) Dan, feel free to override any of these decisions.

*** Bug 1212714 has been marked as a duplicate of this bug. ***

Fabian,

Just to be sure, i'm going to test this report with latest rhev-h 6.7 and vdsm 4.16.23-1 and not with rhev-h 6.6 like described in original report, so please ACK this.

Scenario that will be tested here is:
Upgrade rhev-h 6.5 3.4.z >> rhev-h latest 6.7 3.5.4(vdsm 4.16.23-1)
verify network configurations are not broken and verify the network configurations on the 6.5 backup as well.

Michael, I can ack that you test it with the 6.7 build.

However, I can not ack the flow, as I'm not into this bug - but I'd recommend to also do a reboot of the machine afterwards, to ensure that the networks also come up right after reboot.

i am not sure step 6 in the original bug is possible anymore

I have tested all steps, except step 6, like Tolik wrote in comment 18 ^^, i can't back up to 6.5. No such option.

Upgrade from vdsm-4.14.13-2.el6ev >> vdsm-4.16.24-2.el6ev.x86_64
RHEV Hypervisor - 6.5 - 20140930.1.el6ev >> rhev-hypervisor6-6.7-20150804.0.el6ev

Host is up after reboots, all networks exists and up. Setup Networks configuration is done after second reboot and server rebooted for third time with success.

Fabian, is this can be considered as verified ?
Thanks,

IMO it is not yet verified.

In the setup of comment 19 - you just need to boot back into the initially installed rhevh (RHEV Hypervisor - 6.5 - 20140930.1.el6ev)

It is important that the networks will come up then as well.

How do we boot from the backup?

1. Reboot the host
2. After the BIOS an "countdown" appears, hit <Tab>
3. The grub menu appears, select the backup entry

Verified on https://brewweb.devel.redhat.com/taskinfo?taskID=9670665