Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1518587

Summary: Supervdsm log - ERROR - 'Failed source route addition:' on every dhclient lease renew
Product: [oVirt] vdsm Reporter: Michael Burman <mburman>
Component: CoreAssignee: Edward Haas <edwardh>
Status: CLOSED CURRENTRELEASE QA Contact: Michael Burman <mburman>
Severity: low Docs Contact:
Priority: medium    
Version: 4.20.8CC: bugs, danken, edwardh, mburman
Target Milestone: ovirt-4.2.1Flags: rule-engine: ovirt-4.2+
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: v4.20.9-9-gf10278730 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-02-12 11:54:02 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Network RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
supervdsm log
none
supervdsm after fix none

Description Michael Burman 2017-11-29 09:26:56 UTC
Description of problem:
Supervdsm log - ERROR - 'Failed source route addition:' on every dhclient lease renew  

Each new dhclient lease, error is shown is the supervdsm log - 

MainProcess|dhclient-monitor::DEBUG::2017-11-29 09:39:56,213::sourceroute::190::root::(add) Adding source route for device ovirtmgmt
MainProcess|dhclient-monitor::DEBUG::2017-11-29 09:39:56,213::cmdutils::150::root::(exec_cmd) /sbin/ip -4 route add 0.0.0.0/0 via 10.35.128.254 dev ovirtmgmt table 170098706 (cwd None)
MainProcess|dhclient-monitor::DEBUG::2017-11-29 09:39:56,221::cmdutils::158::root::(exec_cmd) FAILED: <err> = 'RTNETLINK answers: File exists\n'; <rc> = 2
MainProcess|dhclient-monitor::ERROR::2017-11-29 09:39:56,221::sourceroute::199::root::(add) Failed source route addition: ("IPRouteData(to='0.0.0.0/0' via='10.35.128.254' src=None family=4 device='ovirtmgmt' table='170098706')", 'RTNETLINK answers: File exists')


Version-Release number of selected component (if applicable):
vdsm-4.20.8-53.gitc3edfc0.el7.centos.x86_64

How reproducible:
100%
every new dhclient lease(every 4-5 hours)

Steps to Reproduce:
1. Run host for few hours and check the supervdsm log

Actual results:
Error each new lease

Comment 1 Michael Burman 2017-11-29 09:29:48 UTC
Created attachment 1360241 [details]
supervdsm log

Comment 2 Michael Burman 2017-12-10 09:00:28 UTC
Still see those on vdsm-4.20.9-25.git5bd5e94.el7.centos.x86_64 

MainProcess|dhclient-monitor::DEBUG::2017-12-10 05:24:13,353::cmdutils::158::root::(exec_cmd) FAILED: <err> = 'RTNETLINK answers: File exists\n'; <rc> = 2
MainProcess|dhclient-monitor::DEBUG::2017-12-10 05:24:13,353::sourceroute::196::root::(add) Route already exists, addition failed,: ("IPRouteData(to='0.0.0.0/0' via='10.35.128.x' src=None family=4 device='ovirtmgmt' table='170098706')", 'RTNETLINK answers: File exists')

Comment 3 Michael Burman 2017-12-10 09:02:04 UTC
Created attachment 1365514 [details]
supervdsm after fix

Comment 4 Edward Haas 2017-12-10 10:11:39 UTC
(In reply to Michael Burman from comment #2)
> Still see those on vdsm-4.20.9-25.git5bd5e94.el7.centos.x86_64 
> 
> MainProcess|dhclient-monitor::DEBUG::2017-12-10
> 05:24:13,353::cmdutils::158::root::(exec_cmd) FAILED: <err> = 'RTNETLINK
> answers: File exists\n'; <rc> = 2
> MainProcess|dhclient-monitor::DEBUG::2017-12-10
> 05:24:13,353::sourceroute::196::root::(add) Route already exists, addition
> failed,: ("IPRouteData(to='0.0.0.0/0' via='10.35.128.x' src=None family=4
> device='ovirtmgmt' table='170098706')", 'RTNETLINK answers: File exists')

It is now a debug message (was an error) and it specifies why it failed ("Route already exists").
This is the intended behaviour.

Comment 5 Michael Burman 2017-12-10 10:16:54 UTC
(In reply to Edward Haas from comment #4)
> (In reply to Michael Burman from comment #2)
> > Still see those on vdsm-4.20.9-25.git5bd5e94.el7.centos.x86_64 
> > 
> > MainProcess|dhclient-monitor::DEBUG::2017-12-10
> > 05:24:13,353::cmdutils::158::root::(exec_cmd) FAILED: <err> = 'RTNETLINK
> > answers: File exists\n'; <rc> = 2
> > MainProcess|dhclient-monitor::DEBUG::2017-12-10
> > 05:24:13,353::sourceroute::196::root::(add) Route already exists, addition
> > failed,: ("IPRouteData(to='0.0.0.0/0' via='10.35.128.x' src=None family=4
> > device='ovirtmgmt' table='170098706')", 'RTNETLINK answers: File exists')
> 
> It is now a debug message (was an error) and it specifies why it failed
> ("Route already exists").
> This is the intended behaviour.

I see, but 
1) It is spamming the log
2) I thought we going to remove this messages, because the route is exists, but dhclient trying to add new one and failing. It's going to spam the log on every new lease request from dhclient. is that expected?

Comment 6 Edward Haas 2017-12-10 10:33:44 UTC
(In reply to Michael Burman from comment #5)
> > It is now a debug message (was an error) and it specifies why it failed
> > ("Route already exists").
> > This is the intended behaviour.
> 
> I see, but 
> 1) It is spamming the log
> 2) I thought we going to remove this messages, because the route is exists,
> but dhclient trying to add new one and failing. It's going to spam the log
> on every new lease request from dhclient. is that expected?

We are not checking if something exists and only then add it, that is not safe (between the questioning and the adding the status can change).
Therefore, we always add and if it fails we check the error and report.

I think that the debug message is needed, because we may find ourself in cases where we expect one thing but something else is happening (like a wrong route set).
As this is occurring once per lease, I think it is fine. In most setups the lease is between a few hours to a few days.

Comment 7 Michael Burman 2017-12-10 11:19:51 UTC
I see, fine with me.

Comment 8 Michael Burman 2017-12-10 11:22:02 UTC
Base on comment6#, this bug is verified on - vdsm-4.20.9-27.gitb8f880e.el7.centos.x86_64

Comment 9 Sandro Bonazzola 2018-02-12 11:54:02 UTC
This bugzilla is included in oVirt 4.2.1 release, published on Feb 12th 2018.

Since the problem described in this bug report should be
resolved in oVirt 4.2.1 release, it has been closed with a resolution of CURRENT RELEASE.

If the solution does not work for you, please open a new bug report.