Bug 1305465

Summary: Unble to persist the 'ip route' in RHEV-H
Product: Red Hat Enterprise Virtualization Manager Reporter: Sarvesh Pandit <sapandit>
Component: rhev-hypervisorAssignee: Fabian Deutsch <fdeutsch>
Status: CLOSED WORKSFORME QA Contact: wanghui <huiwa>
Severity: high Docs Contact:
Priority: high    
Version: 3.5.3CC: cshao, danken, fdeutsch, gklein, huiwa, huzhao, leiwang, lsurette, pablo.iranzo, pstehlik, pzhukov, sapandit, weiwang, yaniwang, ycui, yeylon, ykaul
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-03-23 13:48:25 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Node RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Sarvesh Pandit 2016-02-08 10:40:21 UTC
Description of problem:
Able to persist the 'ip route' in  RHEV-H

Version-Release number of selected component (if applicable):
Red Hat Enterprise Virtualization Hypervisor release 7.1 (20150603.0.el7ev)
rhevm-3.5.3.1-1.4.el6ev.noarch

How reproducible:
100%

Steps to Reproduce:
1. SSH to hypervisor

2. Check current "ip route" 

# ip route show
default via 10.XX.XXX.XXX dev rhevm 
10.XX.XXX.0/24 dev rhevm  proto kernel  scope link  src 10.XX.XXX.XXX

3. To enable the fix to disable TCP delayed ACK:

# /sbin/ip route change default via 10.XX.XXX.XXXdev rhevm quickack 1
# /sbin/ip route change 10.XX.XXX.0/24/24 dev rhevm  proto kernel  scope link  src 10.XX.XXX.XXX  quickack 1

4. Check again "ip route" 

# ip route show
default via 10.XX.XXX.XXX dev rhevm quickack 1
10.XX.XXX.0/24 dev rhevm  proto kernel  scope link  src 10.XX.XXX.XXX quickack 1

5. Persist the changes:

# persist /etc/sysconfig/network-scripts/route-rhevm

6. Reboot the hypervisor

7. Check the "route-rhevm"

# cat /etc/sysconfig/network-scripts/route-rhevm 
/sbin/ip route change default via 10.xx.xxx.xxx dev rhevm quickack 1
/sbin/ip route change 10.xx.xxx.0/24 dev rhevm  proto kernel  scope link  src 10.xx.xxx.xxx  quickack 1

8. Check "ip route"

# ip route show
default via 10.XX.XXX.XXX dev rhevm 
10.XX.XXX.0/24 dev rhevm  proto kernel  scope link  src 10.XX.XXX.XXX

Actual results:
disable TCP delayed ACK i.e. changes are not persisted even though the "route-rhevm" persisted

Expected results:
Changes should be persisted across reboot.

Additional info:

Comment 3 Fabian Deutsch 2016-02-08 11:26:15 UTC
Persistence can just be applied to configurations files, changes to the runtime (in-kernel) configuration is not persisted.
Calling the ip utility will only change the runtime.

Thus persisting the ifcfg file after performing some ip calls did not have any effect after a reboot.

If this tuning using ip is done often, the user can consider to open an RFE against vdsm to allow seeting this through RHEV-M.

For the time beeing, the necessary ip calls can be written to /etc/rc.d/rc.local and will then be performed during the boot of RHEV-H.

To persist the rc.local script:

1. Write /etc/rc.d/rc.local
2. chmod a+x /etc/rc.d/rc.local
3. persist /etc/rc.d/rc.local

After reboot the rc.local script will be executed.

Please let me know if this solution is viable to the customer.

Comment 12 Fabian Deutsch 2016-02-17 19:30:03 UTC
It could be that we are seeing a race here, a race between the script which is setting up the routes and vdsm. Because vdsm will be taking care of the networking.

Dan, is there a way to configure custom routes as described in comment 0 with vdsm?

Comment 13 Dan Kenigsberg 2016-02-18 07:56:46 UTC
Vdsm is expected to rewrite route-rhevm; it cannot be safely persisted without the use of an after_network_setup hook (which is not at all hard to write).

Sarvesh, would you please read bug 1301879 and explain why your own route is needed? I suspect this bug is a dup of rfe 1301879.

Comment 14 Pavel Zhukov 2016-02-18 10:45:34 UTC
(In reply to Sarvesh Pandit from comment #0)
> # cat /etc/sysconfig/network-scripts/route-rhevm 
> /sbin/ip route change default via 10.xx.xxx.xxx dev rhevm quickack 1
> /sbin/ip route change 10.xx.xxx.0/24 dev rhevm  proto kernel  scope link 
> src 10.xx.xxx.xxx  quickack 1
are you sure the syntax is correct? If I remember correctly you cannot put ip call inside because it's called from the parser [See 1]
Can you please check documentation.

[1]

handle_ip_file() {
    local f t type= file=$1 proto="-4"
    f=${file##*/}
    t=${f%%-*}
    type=${t%%6}
    if [ "$type" != "$t" ]; then
        proto="-6"
    fi
    { cat "$file" ; echo ; } | while read line; do
        if [[ ! "$line" =~ $MATCH ]]; then
            /sbin/ip $proto $type add $line   <==== !!!!!!
        fi
    done
}

Comment 15 Pavel Zhukov 2016-02-18 10:47:38 UTC
(In reply to Dan Kenigsberg from comment #13)
> Vdsm is expected to rewrite route-rhevm; it cannot be safely persisted
> without the use of an after_network_setup hook (which is not at all hard to
> write).
> 
> Sarvesh, would you please read bug 1301879 and explain why your own route is
> needed? I suspect this bug is a dup of rfe 1301879.

Hi Dan, 

I'm sorry for dummy question.
Doesn't vdsm call ifup after generating of ifcfg-rhevm file? If so the route-XXX approach should work because ifup will call ifup-routes (see my previous comment).
Thank you in advance!

Comment 16 Dan Kenigsberg 2016-02-18 12:42:24 UTC
Pavel, the problem is that vdsm would overwrite route-rhevm and rule-rhevm. If the customer would like to change the default routing (the one set by http://www.ovirt.org/Features/Multiple_Gateways ) they have to place a hook script in /usr/libexec/vdsm/after_network_setup which can modify route-rhevm (and source it) or call the needed `ip route` commands explicitly.

This can be implemented based on the example of https://gerrit.ovirt.org/#/c/29738/5/vdsm_hooks/extra_ipv4_addrs/extra_ipv4_addrs.py

Comment 18 Fabian Deutsch 2016-02-22 13:55:49 UTC
Is there anything else to discuss, or can I close this issue?

Comment 20 Fabian Deutsch 2016-02-22 19:06:51 UTC
Sarvesh, to my understanding setting of multiple routes is handled in bug 1301879 and according to comment 13 there is a workaround by using a hook.

Comment 21 Dan Kenigsberg 2016-02-22 20:34:25 UTC
Sarvesh, with rhev-3.6 you can also add a hook that modifies route-* files, where you could add "quickack 1" [untested] to the route defined there by Vdsm.
See bug 965929.

Comment 22 Fabian Deutsch 2016-03-08 09:33:06 UTC
Sarvesh, are these suggestions enough to solve your issue?

Comment 23 wanghui 2016-03-14 08:17:02 UTC
Hi Sarvesh Pandit,

I have tested this follow you steps but I can not found the /etc/sysconfig/network-scripts/route-rhevm file to persist. So I need to confirm with you that where can I find this file?

Test version:
Red Hat Enterprise Virtualization Hypervisor release 7.1 (20150603.0.el7ev)
Red Hat Enterprise Virtualization Manager Version: 3.5.8-0.1.el6ev

Test steps:
1. SSH to hypervisor
2. Check current "ip route" 
3. To enable the fix to disable TCP delayed ACK:
4. Check again "ip route" 

5. Persist the changes:   
# persist /etc/sysconfig/network-scripts/route-rhevm   -- can not find it

Thanks!
Hui Wang

Comment 25 Fabian Deutsch 2016-03-23 10:27:05 UTC
Sarvesh, do you need anything else?

Comment 27 Fabian Deutsch 2016-03-23 13:48:25 UTC
Thanks. Closing according to comment 26.