Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1532825

Summary: NeutronDpdkSocketMemory Needs Both Single and Double Quotes
Product: Red Hat OpenStack Reporter: Sean Lee <slee>
Component: documentationAssignee: Sandra McCann <samccann>
Status: CLOSED CURRENTRELEASE QA Contact: RHOS Documentation Team <rhos-docs>
Severity: low Docs Contact:
Priority: unspecified    
Version: 10.0 (Newton)CC: akaris, mburns, samccann, skramaja, slee, srevivo, yrachman
Target Milestone: asyncKeywords: Triaged, ZStream
Target Release: 10.0 (Newton)   
Hardware: All   
OS: All   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-06-28 15:35:25 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1570832    

Description Sean Lee 2018-01-09 20:58:59 UTC
Description of problem:

For DPDK deployment, NeutronDpdkSocketMemory parameter needs both single quotes and double quotes (single quotes inside double quotes), e.g.

NeutronDpdkSocketMemory: "'4096,4096'"

Without the single quotes, the two numbers would be concatenated during deployment, DPDK_OPTIONS in /etc/sysconfig/openvswitch would be incorrectly set with --socket-mem 40964096.

However, in both installation and update guides, NeutronDpdkSocketMemory is incorrectly set without single quotes inside double quotes, e.g.:

NeutronDpdkSocketMemory: "4096,4096"

https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/10/single/upgrading_red_hat_openstack_platform/index#sect-Pre_Update-NFV_Configuration
https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/10/single/network_functions_virtualization_configuration_guide/index#modify_link_linkend_ap_ovsdpdk_vxlan_network_environment_network_environment_yaml_link


How reproducible:

Every time.


Steps to Reproduce:

Deploy DPDK compute node with NeutronDpdkSocketMemory: "4096,4096" (no single quotes)


Actual results:

DPDK_OPTIONS in /etc/sysconfig/openvswitch set with --socket-mem 40964096.


Expected results:

DPDK_OPTIONS in /etc/sysconfig/openvswitch should be set with --socket-mem 4096,4096.


Additional info:

https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/10/single/upgrading_red_hat_openstack_platform/index#sect-Pre_Update-NFV_Configuration
https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/10/single/network_functions_virtualization_configuration_guide/index#modify_link_linkend_ap_ovsdpdk_vxlan_network_environment_network_environment_yaml_link

Comment 1 Saravanan KR 2018-01-10 06:43:39 UTC
I assume that this is used with ovs2.6 version in OSP10. In which, the DPDK_OPTIONS from /etc/sysconfig/openvswitch is not read by ovs, instead ovs expects it to be configured in db (using ovs-vsctl command). All the these configurations are done by first-boot.yaml script (available in the documentation). Could you try with the first-boot script and update if it is working? IMO, this is not a bug.

Comment 2 Andreas Karis 2018-01-10 14:19:41 UTC
Indeed, the correct verification command is:
~~~
[root@overcloud-compute-0 ~]# ovs-vsctl list Open_vSwitch | grep other_config
other_config        : {dpdk-init="true", dpdk-lcore-mask="00300003", dpdk-socket-mem="4096,4096", pmd-cpu-mask="4407c4407c"}
~~~

Comment 3 Sean Lee 2018-01-10 17:35:43 UTC
The update post-install script reads from /etc/sysconfig/openvswitch.

From https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/10/single/upgrading_red_hat_openstack_platform/index#ap-rhosp10-ovs-2-6-update-post-install:

        function ovs_update_with_dpdk_special_case() {
            # Read the /etc/sysconfig/openvswitch file for DPDK_OPTIONS
            DPDK_OPTIONS=""
            while read line; do
                if [[  $line == "DPDK_OPTIONS"* ]]; then
                    DPDK_OPTIONS=$line
                fi
            done < /etc/sysconfig/openvswitch
            if [[ -n $DPDK_OPTIONS ]]; then
                # Parse -l and --socket-mem parameter and set it accordingly
                DPDK_OPTIONS=$(echo $DPDK_OPTIONS | cut -d '"' -f 2)
                #echo $DPDK_OPTIONS
                PARAM_ARR=($DPDK_OPTIONS)
                if [[ ${#PARAM_ARR[@]} -ge 6 ]]; then
                    if [[ ${PARAM_ARR[0]} == "-l" ]]; then
                        LCORE_LIST=${PARAM_ARR[1]}
                    fi
                    if [[ ${PARAM_ARR[4]} == "--socket-mem" ]]; then
                        SOCKET_MEMORY='"'${PARAM_ARR[5]}'"'
                    fi
                fi
                LCORE_MASK='"'$(get_core_mask $LCORE_LIST)'"'
                #echo "LCORE_LIST=$LCORE_LIST"
                echo "SOCKET_MEMORY=$SOCKET_MEMORY"
                echo "LCORE_MASK=$LCORE_MASK"
                if [[ $(ovs-vsctl get Open_vSwitch . other_config:dpdk-init) != '"true"' ]]; then
                  ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-init=true
                fi
                if [[ $(ovs-vsctl get Open_vSwitch . other_config:dpdk-socket-mem) != $SOCKET_MEMORY ]]; then
                  ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-socket-mem=$SOCKET_MEMORY
                fi
                if [[ $(ovs-vsctl get Open_vSwitch . other_config:dpdk-lcore-mask) != $LCORE_MASK ]]; then
                  ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-lcore-mask=$LCORE_MASK
                fi
            fi
        }

When the script runs, ovsdb-server would panic:

Jan  4 11:27:47 xxxxxxxx-dpdk ovs-ctl: EAL: Not enough memory available on socket 0! Requested: 40964096MB, available: 102400MB
Jan  4 11:27:47 xxxxxxxx-dpdk ovs-vswitchd[11113]: EAL: Not enough memory available on socket 0! Requested: 40964096MB, available: 102400MB
Jan  4 11:27:47 xxxxxxxx-dpdk ovs-ctl: PANIC in rte_eal_init():
Jan  4 11:27:47 xxxxxxxx-dpdk ovs-ctl: Cannot init memory
Jan  4 11:27:47 xxxxxxxx-dpdk ovs-ctl: 9: [ovs-vswitchd(+0x2f3d5) [0x56233924c3d5]]
Jan  4 11:27:47 xxxxxxxx-dpdk ovs-ctl: 8: [/lib64/libc.so.6(__libc_start_main+0xf5) [0x7f40a6ca8c05]]
Jan  4 11:27:47 xxxxxxxx-dpdk ovs-ctl: 7: [ovs-vswitchd(+0x2e65d) [0x56233924b65d]]
Jan  4 11:27:47 xxxxxxxx-dpdk ovs-ctl: 6: [ovs-vswitchd(+0x10fc32) [0x56233932cc32]]
Jan  4 11:27:47 xxxxxxxx-dpdk ovs-ctl: 5: [ovs-vswitchd(+0x223a7c) [0x562339440a7c]]
Jan  4 11:27:47 xxxxxxxx-dpdk ovs-ctl: 4: [ovs-vswitchd(+0x223526) [0x562339440526]]
Jan  4 11:27:47 xxxxxxxx-dpdk ovs-ctl: 3: [ovs-vswitchd(+0xb22ba) [0x5623392cf2ba]]
Jan  4 11:27:47 xxxxxxxx-dpdk ovs-ctl: 2: [ovs-vswitchd(+0x28a18) [0x562339245a18]]
Jan  4 11:27:47 xxxxxxxx-dpdk ovs-ctl: 1: [ovs-vswitchd(+0xb92bd) [0x5623392d62bd]]
Jan  4 11:27:47 xxxxxxxx-dpdk ovs-vswitchd[11113]: PANIC in rte_eal_init():
Jan  4 11:27:47 xxxxxxxx-dpdk ovs-vswitchd[11113]: Cannot init memory
Jan  4 11:27:47 xxxxxxxx-dpdk ovs-vswitchd[11113]: 9: [ovs-vswitchd(+0x2f3d5) [0x56233924c3d5]]
Jan  4 11:27:47 xxxxxxxx-dpdk ovs-vswitchd[11113]: 8: [/lib64/libc.so.6(__libc_start_main+0xf5) [0x7f40a6ca8c05]]
Jan  4 11:27:47 xxxxxxxx-dpdk ovs-vswitchd[11113]: 7: [ovs-vswitchd(+0x2e65d) [0x56233924b65d]]
Jan  4 11:27:47 xxxxxxxx-dpdk ovs-vswitchd[11113]: 6: [ovs-vswitchd(+0x10fc32) [0x56233932cc32]]
Jan  4 11:27:47 xxxxxxxx-dpdk ovs-vswitchd[11113]: 5: [ovs-vswitchd(+0x223a7c) [0x562339440a7c]]
Jan  4 11:27:47 xxxxxxxx-dpdk ovs-vswitchd[11113]: 4: [ovs-vswitchd(+0x223526) [0x562339440526]]
Jan  4 11:27:47 xxxxxxxx-dpdk ovs-vswitchd[11113]: 3: [ovs-vswitchd(+0xb22ba) [0x5623392cf2ba]]
Jan  4 11:27:47 xxxxxxxx-dpdk ovs-vswitchd[11113]: 2: [ovs-vswitchd(+0x28a18) [0x562339245a18]]
Jan  4 11:27:47 xxxxxxxx-dpdk ovs-vswitchd[11113]: 1: [ovs-vswitchd(+0xb92bd) [0x5623392d62bd]]
Jan  4 11:27:47 xxxxxxxx-dpdk ovs-vswitchd: ovs|00002|daemon_unix|ERR|fork child died before signaling startup (killed (Aborted))
Jan  4 11:27:47 xxxxxxxx-dpdk ovs-vswitchd: ovs|00003|daemon_unix|EMER|could not detach from foreground session

Comment 5 Saravanan KR 2018-01-22 09:16:19 UTC
There is a difference in the post-install required for minor update based on ovs versions.

OSP10 Minor update from OvS2.5 to OvS2.6 --> Above mentioned post-install.yaml script is correct
OSP10 Minor update within Ovs2.6 (like, 2.6.1-10 to 2.6.1-14) --> The script to translate the DPDK_OPTIONS is not required.

I suggest QE to validate this scenario before we update the doc. I remember QE has a CI test cases to validate minor updates in OSP10. Lets hear from them whether they face similar issues.

Comment 6 Yariv 2018-02-20 21:38:33 UTC
That is Correct 

See the following patch for RHOS 10 Update/Upgrades
https://code.engineering.redhat.com/gerrit/#/c/130127/

For RHOS 10 Minor Updates OVS 2.6 the following params must be double single quote

NeutronDpdkSocketMemory: "'1024,1024'"
HostCpusList: "'0,16,8,24'"
NeutronDpdkCoreList: "'1,17,9,25'"

Other wise Update success complete, VM could not scheduled on OVS

Comment 7 Yariv 2018-02-20 21:43:57 UTC
Could you close this BZ once it is documented