Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1607860

Summary: [downstream clone - 4.2.5] vdsm-tool upgrade-networks fails with KeyError: 'defaultRoute'
Product: Red Hat Enterprise Virtualization Manager Reporter: RHV bug bot <rhv-bugzilla-bot>
Component: vdsmAssignee: Edward Haas <edwardh>
Status: CLOSED ERRATA QA Contact: Jiri Belka <jbelka>
Severity: high Docs Contact:
Priority: high    
Version: unspecifiedCC: danken, edwardh, eheftman, gchakkar, lsurette, lsvaty, mburman, myakove, srevivo, ycui, ylavi
Target Milestone: ovirt-4.2.5Keywords: ZStream
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1602047 Environment:
Last Closed: 2018-07-31 17:50:12 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Network RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1602047    
Bug Blocks:    

Description RHV bug bot 2018-07-24 12:40:23 UTC
+++ This bug is a downstream clone. The original bug is: +++
+++   bug 1602047 +++
======================================================================

Description of problem:

After upgrading the RHEL based hypervisor the vdsm-network service fails to start with:
~~~
Jul 17 10:52:40 systemd: Starting Virtual Desktop Server Manager network restoration...                                                                                                                     
Jul 17 10:52:41 vdsm-tool: Traceback (most recent call last):                                                                                                                                               
Jul 17 10:52:41 vdsm-tool: File "/usr/bin/vdsm-tool", line 219, in main                                                                                                                                     
Jul 17 10:52:41 vdsm-tool: return tool_command[cmd]["command"](*args)                                                                                                                                       
Jul 17 10:52:41 vdsm-tool: File "/usr/lib/python2.7/site-packages/vdsm/tool/network.py", line 55, in upgrade_networks                                                                                       
Jul 17 10:52:41 vdsm-tool: netupgrade.upgrade()                                                                                                                                                             
Jul 17 10:52:41 vdsm-tool: File "/usr/lib/python2.7/site-packages/vdsm/network/netupgrade.py", line 67, in upgrade                                                                                          
Jul 17 10:52:41 vdsm-tool: _upgrade_unified_configuration(rconfig)                                                                                                                                          
Jul 17 10:52:41 vdsm-tool: File "/usr/lib/python2.7/site-packages/vdsm/network/netupgrade.py", line 108, in _upgrade_unified_configuration                                                                  
Jul 17 10:52:41 vdsm-tool: canonicalize_networks(config.networks)                                                                                                                                           
Jul 17 10:52:41 vdsm-tool: File "/usr/lib/python2.7/site-packages/vdsm/network/canonicalize.py", line 43, in canonicalize_networks                                                                          
Jul 17 10:52:41 vdsm-tool: _canonicalize_ip_default_route(nets)                                                                                                                                             
Jul 17 10:52:41 vdsm-tool: File "/usr/lib/python2.7/site-packages/vdsm/network/canonicalize.py", line 205, in _canonicalize_ip_default_route                                                                
Jul 17 10:52:41 vdsm-tool: existing_net_with_default_route = _net_with_default_route_from_config()                                                                                                          
Jul 17 10:52:41 vdsm-tool: File "/usr/lib/python2.7/site-packages/vdsm/network/canonicalize.py", line 217, in _net_with_default_route_from_config                                                           
Jul 17 10:52:41 vdsm-tool: if attrs['defaultRoute']:                                                                                                                                                        
Jul 17 10:52:41 vdsm-tool: KeyError: 'defaultRoute'                                                                                                                                                         
Jul 17 10:52:41 systemd: vdsm-network.service: control process exited, code=exited status=1                                                                                                                 
Jul 17 10:52:41 systemd: Failed to start Virtual Desktop Server Manager network restoration
~~~

Version-Release number of selected component (if applicable):

vdsm-4.20.32-1.el7ev.x86_64

How reproducible:
Always

Steps to Reproduce:
- Upgrade vdsm to 4.20.32
- At least one network without 'defaultRoute' key defined in '/var/lib/vdsm/persistence/netconf/nets' directory.
- Restart vdsmd service

Actual results:

The vdsmd service fails to start because vdsm-network dependency failed to start

Expected results:

The vdsmd and vdsm-network services restart correctly

Additional info:

The corresponding code is:
~~~
215 def _net_with_default_route_from_config():                                                                                                                                                              
216     for net, attrs in six.iteritems(RunningConfig().networks):                                                                                                                                          
217         if attrs['defaultRoute']:                                                                                                                                                                       
218     ┆   ┆   return net, attrs                                                                                                                                                                           
219     return None     
~~~

I think the following patch should fix the issue:
~~~
diff --git a/lib/vdsm/network/canonicalize.py b/lib/vdsm/network/canonicalize.py
index 674f98e53..91ebce77e 100644
--- a/lib/vdsm/network/canonicalize.py
+++ b/lib/vdsm/network/canonicalize.py
@@ -214,7 +214,7 @@ def _canonicalize_ip_default_route(nets):
 
 def _net_with_default_route_from_config():
     for net, attrs in six.iteritems(RunningConfig().networks):
-        if attrs['defaultRoute']:
+        if attrs.get('defaultRoute'):
             return net, attrs
     return None
~~~

(Originally by Miguel Martin Villamuelas)

Comment 3 RHV bug bot 2018-07-24 12:40:33 UTC
WARN: Bug status wasn't changed from MODIFIED to ON_QA due to the following reason:

[Found non-acked flags: '{'rhevm-4.2.z': '?'}', ]

For more info please contact: rhv-devops: Bug status wasn't changed from MODIFIED to ON_QA due to the following reason:

[Found non-acked flags: '{'rhevm-4.2.z': '?'}', ]

For more info please contact: rhv-devops

(Originally by rhv-bugzilla-bot)

Comment 6 Jiri Belka 2018-07-27 09:59:28 UTC
ok, vdsm-4.20.35-1.el7ev.x86_64


# grep default /var/lib/vdsm/persistence/netconf/nets/isolated; echo $?
1

# rpm -q vdsm ; grep default /var/lib/vdsm/persistence/netconf/nets/isolated; echo $?
vdsm-4.20.35-1.el7ev.x86_64
    "defaultRoute": false, 
0

Comment 8 errata-xmlrpc 2018-07-31 17:50:12 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2018:2319