Bug 1457889
| Summary: | Host is Non-Operational and missing networks after vdsm upgrade | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Product: | [oVirt] vdsm | Reporter: | Michael Burman <mburman> | ||||||
| Component: | Core | Assignee: | Edward Haas <edwardh> | ||||||
| Status: | CLOSED CURRENTRELEASE | QA Contact: | Michael Burman <mburman> | ||||||
| Severity: | high | Docs Contact: | |||||||
| Priority: | high | ||||||||
| Version: | 4.20.0 | CC: | bugs, danken | ||||||
| Target Milestone: | ovirt-4.2.0 | Flags: | edwardh:
needinfo-
rule-engine: ovirt-4.2+ |
||||||
| Target Release: | --- | ||||||||
| Hardware: | x86_64 | ||||||||
| OS: | Linux | ||||||||
| Whiteboard: | |||||||||
| Fixed In Version: | Doc Type: | No Doc Update | |||||||
| Doc Text: | Story Points: | --- | |||||||
| Clone Of: | Environment: | ||||||||
| Last Closed: | 2017-12-20 11:36:30 UTC | Type: | Bug | ||||||
| Regression: | --- | Mount Type: | --- | ||||||
| Documentation: | --- | CRM: | |||||||
| Verified Versions: | Category: | --- | |||||||
| oVirt Team: | Network | RHEL 7.3 requirements from Atomic Host: | |||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||
| Embargoed: | |||||||||
| Bug Depends On: | |||||||||
| Bug Blocks: | 1448837 | ||||||||
| Attachments: |
|
||||||||
Created attachment 1284137 [details]
vdsm logs
Are you sure this should be ON_QA? * Although the upgrade flow seems to be ok now and host is operational after vdsm upgrade, after reboot the host is down. Because ovirtmgmt network is gone and it missing ifcfg-ovirtmgmt file.
- I have checked few times if the ovirtmgmt was persisted as it should be before upgrade and reboot, but it was gone after upgrade+reboot.
I don't think this should be verified. Will attach log once manage to bring host up.
- Upgraded from vdsm-4.19.20-1.el7ev.x86_64 > vdsm-4.20.1-67.git4721d69.el7.centos.x86_64
[root@orchid-vds2 yum.repos.d]# tree /var/lib/vdsm/persistence/netconf/
/var/lib/vdsm/persistence/netconf/
├── bonds
│ └── bond0
└── nets
├── n41
└── ovirtmgmt
2 directories, 3 files
[root@orchid-vds2 yum.repos.d]# tree /var/lib/vdsm/staging/netconf/
/var/lib/vdsm/staging/netconf/
├── bonds
│ └── bond0
└── nets
├── n41
└── ovirtmgmt
2 directories, 3 files
[root@orchid-vds2 yum.repos.d]# rpm -q vdsm
vdsm-4.20.1-67.git4721d69.el7.centos.x86_64
[root@orchid-vds2 yum.repos.d]# reboot -f
After upgrade and reboot:
[root@orchid-vds2 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: enp4s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
link/ether 00:1a:64:7a:94:62 brd ff:ff:ff:ff:ff:ff
inet 10.35.128.15/24 brd 10.35.128.255 scope global dynamic enp4s0
valid_lft 43180sec preferred_lft 43180sec
3: enp6s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master n41 state UP qlen 1000
link/ether 00:1a:64:7a:94:64 brd ff:ff:ff:ff:ff:ff
4: ens1f0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master bond0 state UP qlen 1000
link/ether 00:15:17:3d:cd:ce brd ff:ff:ff:ff:ff:ff
5: ens1f1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master bond0 state UP qlen 1000
link/ether 00:15:17:3d:cd:ce brd ff:ff:ff:ff:ff:ff
6: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP qlen 1000
link/ether 00:15:17:3d:cd:ce brd ff:ff:ff:ff:ff:ff
8: n41: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP qlen 1000
link/ether 00:1a:64:7a:94:64 brd ff:ff:ff:ff:ff:ff
inet6 fe80::21a:64ff:fe7a:9464/64 scope link
valid_lft forever preferred_lft forever
9: ;vdsmdummy;: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
link/ether 5e:2a:96:e5:91:51 brd ff:ff:ff:ff:ff:ff
[root@orchid-vds2 ~]# tree /var/lib/vdsm/staging/
/var/lib/vdsm/staging/
└── netconf
├── bonds
│ └── bond0
└── nets
├── n41
└── ovirtmgmt
3 directories, 3 files
[root@orchid-vds2 ~]# tree /var/lib/vdsm/persistence/
/var/lib/vdsm/persistence/
├── netconf -> /var/lib/vdsm/persistence/netconf.Pe435ce8
└── netconf.Pe435ce8
├── bonds
│ └── bond0
└── nets
├── n41
└── ovirtmgmt
4 directories, 3 files
[root@orchid-vds2 ~]# cat /etc/sysconfig/network-scripts/ifcfg-ovirtmgmt
cat: /etc/sysconfig/network-scripts/ifcfg-ovirtmgmt: No such file or directory
ovirtmgmt is missing...
Created attachment 1291894 [details]
ovirtmgmt is gone after upgrade vdsm and then reboot
In the bottom line, now the host is non-operational after upgrade and reboot. I think this should be back to assigned Decided with Edy to keep it on this bug as it's an upgrade issue. "ipv6gateway": "fe80:52:0:2380::fe" was generated and added after vdsm upgrade and on reboot, callback failed as vdsm expecting to have some ip address if ipv6 gateway is set. Edy working on a patch to handle this issue Target release should be placed once a package build is known to fix a issue. Since this bug is not modified, the target version has been reset. Please use target milestone to plan a fix for a oVirt release. Tested and verified on - vdsm-4.20.1-120.git28558d7.el7.centos.x86_64 -Upgrade flow vdsm-4.19.20-1.el7ev > vdsm-4.20.1-120.git28558d7.el7.centos.x86_64 Host operational after vdsm upgrade and after reboot as well. This bugzilla is included in oVirt 4.2.0 release, published on Dec 20th 2017. Since the problem described in this bug report should be resolved in oVirt 4.2.0 release, published on Dec 20th 2017, it has been closed with a resolution of CURRENT RELEASE. If the solution does not work for you, please open a new bug report. |
Description of problem: Host is Non-Operational and missing networks after vdsm upgrade. After vdsm upgrade, 4.1.3 > 4.2, the networks are missing and host is non-operational in the engine. Version-Release number of selected component (if applicable): vdsm-4.20.0-958.gita877434.el7.centos.x86_64 Looks like vdsm-network.service is not restarted after the upgrade(vdsmd and supervdsmd restarted, which is weird..) vdsm-client report no networks. "networks": {}, as well in return network_caps 'networks': {}} How reproducible: 100% Steps to Reproduce: 1. Install host with vdsm 4.19.16 (it doesn't matter in which rhv engine) 2. Optional - You may attach some additional networks to the host 3. Upgrade to latest vdsm 4.2 - vdsm-4.20.0-958.gita877434 Actual results: Networks are missing. ovirtmgmt network is missing and host become Non-operational. Looks like vdsm-network.service wasn't restarted during the upgrade. Expected results: Should work as expected. upgrade should work. Additional info: The upgrade.log is missing the relevant update info for the networks. If running manually - /usr/bin/vdsm-tool --vvverbose upgrade-networks The upgrade succeeded.