Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1602047

Summary: vdsm-tool upgrade-networks fails with KeyError: 'defaultRoute'
Product: Red Hat Enterprise Virtualization Manager Reporter: Miguel Martin <mmartinv>
Component: vdsmAssignee: Edward Haas <edwardh>
Status: CLOSED ERRATA QA Contact: Jan Zmeskal <jzmeskal>
Severity: high Docs Contact:
Priority: high    
Version: 4.2.0CC: danken, edwardh, gchakkar, lsurette, lsvaty, mburman, myakove, obockows, srevivo, ycui
Target Milestone: ovirt-4.3.0Keywords: ZStream
Target Release: 4.3.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: v4.30.3 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1607860 (view as bug list) Environment:
Last Closed: 2019-05-08 12:36:02 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Network RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1607860    

Description Miguel Martin 2018-07-17 16:59:29 UTC 
Description of problem:

After upgrading the RHEL based hypervisor the vdsm-network service fails to start with:
~~~
Jul 17 10:52:40 systemd: Starting Virtual Desktop Server Manager network restoration...                                                                                                                     
Jul 17 10:52:41 vdsm-tool: Traceback (most recent call last):                                                                                                                                               
Jul 17 10:52:41 vdsm-tool: File "/usr/bin/vdsm-tool", line 219, in main                                                                                                                                     
Jul 17 10:52:41 vdsm-tool: return tool_command[cmd]["command"](*args)                                                                                                                                       
Jul 17 10:52:41 vdsm-tool: File "/usr/lib/python2.7/site-packages/vdsm/tool/network.py", line 55, in upgrade_networks                                                                                       
Jul 17 10:52:41 vdsm-tool: netupgrade.upgrade()                                                                                                                                                             
Jul 17 10:52:41 vdsm-tool: File "/usr/lib/python2.7/site-packages/vdsm/network/netupgrade.py", line 67, in upgrade                                                                                          
Jul 17 10:52:41 vdsm-tool: _upgrade_unified_configuration(rconfig)                                                                                                                                          
Jul 17 10:52:41 vdsm-tool: File "/usr/lib/python2.7/site-packages/vdsm/network/netupgrade.py", line 108, in _upgrade_unified_configuration                                                                  
Jul 17 10:52:41 vdsm-tool: canonicalize_networks(config.networks)                                                                                                                                           
Jul 17 10:52:41 vdsm-tool: File "/usr/lib/python2.7/site-packages/vdsm/network/canonicalize.py", line 43, in canonicalize_networks                                                                          
Jul 17 10:52:41 vdsm-tool: _canonicalize_ip_default_route(nets)                                                                                                                                             
Jul 17 10:52:41 vdsm-tool: File "/usr/lib/python2.7/site-packages/vdsm/network/canonicalize.py", line 205, in _canonicalize_ip_default_route                                                                
Jul 17 10:52:41 vdsm-tool: existing_net_with_default_route = _net_with_default_route_from_config()                                                                                                          
Jul 17 10:52:41 vdsm-tool: File "/usr/lib/python2.7/site-packages/vdsm/network/canonicalize.py", line 217, in _net_with_default_route_from_config                                                           
Jul 17 10:52:41 vdsm-tool: if attrs['defaultRoute']:                                                                                                                                                        
Jul 17 10:52:41 vdsm-tool: KeyError: 'defaultRoute'                                                                                                                                                         
Jul 17 10:52:41 systemd: vdsm-network.service: control process exited, code=exited status=1                                                                                                                 
Jul 17 10:52:41 systemd: Failed to start Virtual Desktop Server Manager network restoration
~~~

Version-Release number of selected component (if applicable):

vdsm-4.20.32-1.el7ev.x86_64

How reproducible:
Always

Steps to Reproduce:
- Upgrade vdsm to 4.20.32
- At least one network without 'defaultRoute' key defined in '/var/lib/vdsm/persistence/netconf/nets' directory.
- Restart vdsmd service

Actual results:

The vdsmd service fails to start because vdsm-network dependency failed to start

Expected results:

The vdsmd and vdsm-network services restart correctly

Additional info:

The corresponding code is:
~~~
215 def _net_with_default_route_from_config():                                                                                                                                                              
216     for net, attrs in six.iteritems(RunningConfig().networks):                                                                                                                                          
217         if attrs['defaultRoute']:                                                                                                                                                                       
218     ┆   ┆   return net, attrs                                                                                                                                                                           
219     return None     
~~~

I think the following patch should fix the issue:
~~~
diff --git a/lib/vdsm/network/canonicalize.py b/lib/vdsm/network/canonicalize.py
index 674f98e53..91ebce77e 100644
--- a/lib/vdsm/network/canonicalize.py
+++ b/lib/vdsm/network/canonicalize.py
@@ -214,7 +214,7 @@ def _canonicalize_ip_default_route(nets):
 
 def _net_with_default_route_from_config():
     for net, attrs in six.iteritems(RunningConfig().networks):
-        if attrs['defaultRoute']:
+        if attrs.get('defaultRoute'):
             return net, attrs
     return None
~~~
Comment 2 RHV bug bot 2018-07-24 12:27:20 UTC 
WARN: Bug status wasn't changed from MODIFIED to ON_QA due to the following reason:

[Found non-acked flags: '{'rhevm-4.2.z': '?'}', ]

For more info please contact: rhv-devops: Bug status wasn't changed from MODIFIED to ON_QA due to the following reason:

[Found non-acked flags: '{'rhevm-4.2.z': '?'}', ]

For more info please contact: rhv-devops
Comment 4 Lukas Svaty 2018-09-05 05:44:17 UTC 
Hi Edward, I believe this one should be on 4.3.0 as we had a clone for 4.2.5 - bug#1607860. And 4.2.6 is already out (with this patch since 4.2.5)
Comment 7 Jan Zmeskal 2019-01-18 16:01:41 UTC 
Verified on vdsm package version 4.30.6-1 using this network:
{
    "ipv6autoconf": true, 
    "nameservers": [], 
    "nic": "eth1", 
    "mtu": 1500, 
    "switch": "legacy", 
    "dhcpv6": false, 
    "stp": false, 
    "bridged": true, 
    "bootproto": "dhcp"
}
and running "systemctl restart vdsmd".
Comment 9 errata-xmlrpc 2019-05-08 12:36:02 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:1077