Bug 1457889 - Host is Non-Operational and missing networks after vdsm upgrade
Summary: Host is Non-Operational and missing networks after vdsm upgrade
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: vdsm
Classification: oVirt
Component: Core
Version: 4.20.0
Hardware: x86_64
OS: Linux
high
high
Target Milestone: ovirt-4.2.0
: ---
Assignee: Edward Haas
QA Contact: Michael Burman
URL:
Whiteboard:
Depends On:
Blocks: 1448837
TreeView+ depends on / blocked
 
Reported: 2017-06-01 13:09 UTC by Michael Burman
Modified: 2019-03-04 16:27 UTC (History)
2 users (show)

Fixed In Version:
Clone Of:
Environment:
Last Closed: 2017-12-20 11:36:30 UTC
oVirt Team: Network
Embargoed:
edwardh: needinfo-
rule-engine: ovirt-4.2+


Attachments (Terms of Use)
vdsm logs (786.46 KB, application/x-gzip)
2017-06-01 13:09 UTC, Michael Burman
no flags Details
ovirtmgmt is gone after upgrade vdsm and then reboot (153.01 KB, application/x-gzip)
2017-06-26 08:13 UTC, Michael Burman
no flags Details


Links
System ID Private Priority Status Summary Last Updated
oVirt gerrit 77785 0 master MERGED spec: Handle VDSM services restart on upgrade to posttrans 2020-08-12 08:08:14 UTC
oVirt gerrit 78663 0 master MERGED net: Unified configuration existance can have only nets or bonds 2020-08-12 08:08:14 UTC

Description Michael Burman 2017-06-01 13:09:19 UTC
Description of problem:
Host is Non-Operational and missing networks after vdsm upgrade.

After vdsm upgrade, 4.1.3 > 4.2, the networks are missing and host is non-operational in the engine. 

Version-Release number of selected component (if applicable):
vdsm-4.20.0-958.gita877434.el7.centos.x86_64

Looks like vdsm-network.service is not restarted after the upgrade(vdsmd and supervdsmd restarted, which is weird..)

vdsm-client report no networks. 

 "networks": {}, 

as well in return network_caps

'networks': {}}

How reproducible:
100%

Steps to Reproduce:
1. Install host with vdsm 4.19.16 (it doesn't matter in which rhv engine)
2. Optional - You may attach some additional networks to the host
3. Upgrade to latest vdsm 4.2 -  vdsm-4.20.0-958.gita877434

Actual results:
Networks are missing. ovirtmgmt network is missing and host become Non-operational. Looks like vdsm-network.service wasn't restarted during the upgrade. 

Expected results:
Should work as expected. upgrade should work.

Additional info:
The upgrade.log is missing the relevant update info for the networks.
If running manually - /usr/bin/vdsm-tool --vvverbose upgrade-networks
The upgrade succeeded.

Comment 1 Michael Burman 2017-06-01 13:09:41 UTC
Created attachment 1284137 [details]
vdsm logs

Comment 2 Michael Burman 2017-06-26 06:13:09 UTC
Are you sure this should be ON_QA?

Comment 3 Michael Burman 2017-06-26 08:06:44 UTC
* Although the upgrade flow seems to be ok now and host is operational after vdsm upgrade, after reboot the host is down. Because ovirtmgmt network is gone and it missing ifcfg-ovirtmgmt file.

- I have checked few times if the ovirtmgmt was persisted as it should be before upgrade and reboot, but it was gone after upgrade+reboot. 
I don't think this should be verified. Will attach log once manage to bring host up. 

- Upgraded from vdsm-4.19.20-1.el7ev.x86_64 > vdsm-4.20.1-67.git4721d69.el7.centos.x86_64
 
[root@orchid-vds2 yum.repos.d]# tree /var/lib/vdsm/persistence/netconf/
/var/lib/vdsm/persistence/netconf/
├── bonds
│   └── bond0
└── nets
    ├── n41
    └── ovirtmgmt

2 directories, 3 files
[root@orchid-vds2 yum.repos.d]# tree /var/lib/vdsm/staging/netconf/
/var/lib/vdsm/staging/netconf/
├── bonds
│   └── bond0
└── nets
    ├── n41
    └── ovirtmgmt

2 directories, 3 files
[root@orchid-vds2 yum.repos.d]# rpm -q vdsm
vdsm-4.20.1-67.git4721d69.el7.centos.x86_64
[root@orchid-vds2 yum.repos.d]# reboot -f

Comment 4 Michael Burman 2017-06-26 08:12:12 UTC
After upgrade and reboot:

[root@orchid-vds2 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: enp4s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 00:1a:64:7a:94:62 brd ff:ff:ff:ff:ff:ff
    inet 10.35.128.15/24 brd 10.35.128.255 scope global dynamic enp4s0
       valid_lft 43180sec preferred_lft 43180sec
3: enp6s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master n41 state UP qlen 1000
    link/ether 00:1a:64:7a:94:64 brd ff:ff:ff:ff:ff:ff
4: ens1f0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master bond0 state UP qlen 1000
    link/ether 00:15:17:3d:cd:ce brd ff:ff:ff:ff:ff:ff
5: ens1f1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master bond0 state UP qlen 1000
    link/ether 00:15:17:3d:cd:ce brd ff:ff:ff:ff:ff:ff
6: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP qlen 1000
    link/ether 00:15:17:3d:cd:ce brd ff:ff:ff:ff:ff:ff
8: n41: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP qlen 1000
    link/ether 00:1a:64:7a:94:64 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::21a:64ff:fe7a:9464/64 scope link 
       valid_lft forever preferred_lft forever
9: ;vdsmdummy;: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
    link/ether 5e:2a:96:e5:91:51 brd ff:ff:ff:ff:ff:ff
[root@orchid-vds2 ~]# tree /var/lib/vdsm/staging/
/var/lib/vdsm/staging/
└── netconf
    ├── bonds
    │   └── bond0
    └── nets
        ├── n41
        └── ovirtmgmt

3 directories, 3 files
[root@orchid-vds2 ~]# tree /var/lib/vdsm/persistence/
/var/lib/vdsm/persistence/
├── netconf -> /var/lib/vdsm/persistence/netconf.Pe435ce8
└── netconf.Pe435ce8
    ├── bonds
    │   └── bond0
    └── nets
        ├── n41
        └── ovirtmgmt

4 directories, 3 files

[root@orchid-vds2 ~]# cat /etc/sysconfig/network-scripts/ifcfg-ovirtmgmt
cat: /etc/sysconfig/network-scripts/ifcfg-ovirtmgmt: No such file or directory

ovirtmgmt is missing...

Comment 5 Michael Burman 2017-06-26 08:13:51 UTC
Created attachment 1291894 [details]
ovirtmgmt is gone after upgrade vdsm and then reboot

Comment 6 Michael Burman 2017-06-26 08:15:55 UTC
In the bottom line, now the host is non-operational after upgrade and reboot.
I think this should be back to assigned

Comment 7 Michael Burman 2017-06-26 13:36:17 UTC
Decided with Edy to keep it on this bug as it's an upgrade issue.

"ipv6gateway": "fe80:52:0:2380::fe" was generated and added after vdsm upgrade and on reboot, callback failed as vdsm expecting to have some ip address if ipv6 gateway is set. 

Edy working on a patch to handle this issue

Comment 8 Red Hat Bugzilla Rules Engine 2017-06-26 13:36:22 UTC
Target release should be placed once a package build is known to fix a issue. Since this bug is not modified, the target version has been reset. Please use target milestone to plan a fix for a oVirt release.

Comment 9 Michael Burman 2017-07-05 08:59:26 UTC
Tested and verified on - vdsm-4.20.1-120.git28558d7.el7.centos.x86_64
-Upgrade flow vdsm-4.19.20-1.el7ev > vdsm-4.20.1-120.git28558d7.el7.centos.x86_64
Host operational after vdsm upgrade and after reboot as well.

Comment 10 Sandro Bonazzola 2017-12-20 11:36:30 UTC
This bugzilla is included in oVirt 4.2.0 release, published on Dec 20th 2017.

Since the problem described in this bug report should be
resolved in oVirt 4.2.0 release, published on Dec 20th 2017, it has been closed with a resolution of CURRENT RELEASE.

If the solution does not work for you, please open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.