Bug 1396605

Partner has hit this issue, but the work-around listed in the upstream did not work for him and he got a new message. Can you please take a look and offer guidance?

Created attachment 1237886 [details]
/var/log/messages file from compute node

Created attachment 1237887 [details]
ovs-vswitchd.log from compute node

Created attachment 1237889 [details]
ovsdb-server.log from compute node

@surya

I ran into another issue after I found the first workaround that I forgot to include here.  By default, OVS doesn't seem to be running with the "--dpdk" flag and other required parameters (even though it seems to be properly set to do so in the OOO templates).  That's why you see these messages repeated in your ovs-vswitchd.log:

...
2017-01-06T12:03:14.291Z|00070|netdev|WARN|could not create netdev dpdk0 of unknown type dpdk
...
2017-01-06T12:03:14.291Z|00072|netdev|WARN|could not create netdev dpdk1 of unknown type dpdk
...

Check to see if this is the case with the partner's setup.  What I had to do was to run OVS manually with the missing DPDK flags because setting DPDK_OPTIONS in OOO templates (NeutronDpdkCoreList, NeutronDpdkMemoryChannels, NeutronDpdkSocketMemory and NeutronDpdkDriverType) did not work properly.  Once I had OVS running with DPDK enabled, os-collect-config was able to add the ports when it looped and tried to configure networking again.

OVS DPDK BONDING [ ACTIVE BACKUP ] 

ovs-vswitchd --dpdk -c 0x1 -n 4 --socket-mem 2048,2048 \ -- unix:$DB_SOCK --pidfile --detach


NOTE : This is completely manual work Around, it is working. 

NOTE : We are completely doing the automated deployment and definitely looking for patch to help us in the automation. 

We require PATCH for OVS DPDK BONDING for AUTOMATED DEPLOYMENT

(In reply to Andrew Bays from comment #7)
> @surya
> 
> I ran into another issue after I found the first workaround that I forgot to
> include here.  By default, OVS doesn't seem to be running with the "--dpdk"
> flag and other required parameters (even though it seems to be properly set
> to do so in the OOO templates).  That's why you see these messages repeated
> in your ovs-vswitchd.log:
> 
> ...
> 2017-01-06T12:03:14.291Z|00070|netdev|WARN|could not create netdev dpdk0 of
> unknown type dpdk
> ...
> 2017-01-06T12:03:14.291Z|00072|netdev|WARN|could not create netdev dpdk1 of
> unknown type dpdk
> ...
> 
> Check to see if this is the case with the partner's setup.  What I had to do
> was to run OVS manually with the missing DPDK flags because setting
> DPDK_OPTIONS in OOO templates (NeutronDpdkCoreList,
> NeutronDpdkMemoryChannels, NeutronDpdkSocketMemory and
> NeutronDpdkDriverType) did not work properly.  Once I had OVS running with
> DPDK enabled, os-collect-config was able to add the ports when it looped and
> tried to configure networking again.
Please refer
https://bugzilla.redhat.com/show_bug.cgi?id=1366356

patch submitted -  https://review.openstack.org/#/c/417805/ 
moving to QA

@(In reply to Sanjay Upadhyay from comment #11)
> patch submitted -  https://review.openstack.org/#/c/417805/ 
> moving to QA

Does this patch take care of the workaround mentioned in comment #7 and #8?

Hi Sanjay, does the patch you submitted include the work-arounds for those mentioned in comments 7 and 8?

(In reply to Sean Merrow from comment #13)
> Hi Sanjay, does the patch you submitted include the work-arounds for those
> mentioned in comments 7 and 8?

some parts of the workaround are not required, ie the restarting of ovs-vswitchd. THe ovsswitchd is being started by the installer on step4 of the installation.

Thanks Sanjay for the workaround. We are able to deploy a compute node with OVS dpdkbond in active-stndby mode

Changed back to On_DEV
Patch is not merged yet.. 10z released on January 25th
Please put change status Back, once the patch received +2

(In reply to Yariv from comment #17)
> Changed back to On_DEV
> Patch is not merged yet.. 10z released on January 25th
> Please put change status Back, once the patch received +2

https://review.openstack.org/#/c/417805 is merged now, placing the bug on_qa again.

sanjay, i believe this patch is merged. Can you create a new bz and start the downporting asap.

Yariv,
We will start the backport, but let us know if with this change we can get past this issue.

Regards
Vijay.

moving back to POST as patch is not in a build yet, but has been proposed/merged upstream

*** Bug 1416421 has been marked as a duplicate of this bug. ***

Thanks @Jschlueter.

This is now packaged, and I am moving the bug to QA for testing and validating. I have run a test and this is fixed.

I have verified DPDK + bonding deployment with "os-net-config-5.1.0-1.el7ost.noarch".

os-net-config-5.1.0-1.el7ost.noarch released

Hi,

Sorry for reopening the bugzilla.

I tried the newest 10.0.1 image from official site but the os-net-config version is still os-net-config-5.0.0-5.el7ost.noarch. Where can I get images which contain os-net-config-5.1.0-1 ?

Best Regards,
Chen

Hi Sanjay,

Can you guide me where I can get image with os-net-config-5.1.0-1 ?

Best Regards,
Chen

[root@rhelv73 ~]# yum list os-net-config
Loaded plugins: search-disabled-repos
Available Packages
os-net-config.noarch      5.1.0-1.el7ost     rhel-7-server-openstack-10-rpms


Can you please tally from the info given above with your setup. I am guessing its a simple case of correcting to correct repolist. It should have rhel-7-server-openstack-10-rpms.

Hi Sanjay,

I am using director to deploy the overcloud nodes. My understanding is that it should be the os-net-config inside the overcloud image who configures the ovs-dpdk bond, not the packages in the official repository. I understand new os-net-config is available in the official repo but the os-net-config inside the overcloud image is too old to configure the ovs-dpdk bond. 

Or do I need to update the os-net-config in undercloud side ? Please correct me if I'm wrong.

Best Regards,
Chen

Hi Chen,

you are right, the overcloud image is having older os-net-config. -

Eyal, can you check on this?

[stack@rhelv73 ~]$ sudo yum list rhosp-director-images
Loaded plugins: search-disabled-repos
Installed Packages
rhosp-director-images.noarch  10.0-20170201.1.el7ost    @rhel-7-server-openstack-10-rpms
[root@rhelv73 ~]# LIBGUESTFS_BACKEND=direct guestmount -a ./overcloud-full.qcow2 -m /dev/sda /mnt
[root@rhelv73 mnt]# logout
[stack@rhelv73 ~]$ sudo chroot /mnt
[root@rhelv73 /]# cd
[root@rhelv73 ~]# rpm -q os-net-config
os-net-config-5.0.0-5.el7ost.noarch

^^^ overcloud image in the latest package has the old os-net-config, which would lead to a failed deploy.

Hi Sanjay,

This case currently affects a serials of huawei PoC project so it is in a high priority from Red Hat side.

Best Regards,
Chen

(In reply to Sanjay Upadhyay from comment #32)
> Hi Chen,
> 
> you are right, the overcloud image is having older os-net-config. -
> 
> Eyal, can you check on this?
> 
> [stack@rhelv73 ~]$ sudo yum list rhosp-director-images
> Loaded plugins: search-disabled-repos
> Installed Packages
> rhosp-director-images.noarch  10.0-20170201.1.el7ost   
> @rhel-7-server-openstack-10-rpms
> [root@rhelv73 ~]# LIBGUESTFS_BACKEND=direct guestmount -a
> ./overcloud-full.qcow2 -m /dev/sda /mnt
> [root@rhelv73 mnt]# logout
> [stack@rhelv73 ~]$ sudo chroot /mnt
> [root@rhelv73 /]# cd
> [root@rhelv73 ~]# rpm -q os-net-config
> os-net-config-5.0.0-5.el7ost.noarch
> 
> ^^^ overcloud image in the latest package has the old os-net-config, which
> would lead to a failed deploy.

Hi,

Not in my case, both have the same version..

[root@controller-0 ~]# rpm -qa | grep os-net-config
os-net-config-5.1.0-1.el7ost.noarch

Hi Eyal,

Which image version are you using ?

Hi Chen,

[root@panther05 ~]# rpm -qa | grep rhosp
rhosp-director-images-ipa-10.0-20170214.1.el7ost.noarch
rhosp-director-images-10.0-20170214.1.el7ost.noarch
[root@panther05 ~]# yum list rhosp-director-images
Loaded plugins: langpacks, product-id, search-disabled-repos, subscription-manager
Installed Packages
rhosp-director-images.noarch       10.0-20170214.1.el7ost        @rhelosp-10.0-puddle

Any further information?

Hi Eyal. Sanjay and Jon,

The customer confirmed that 10.0-20170228.1 from puddle solved the bonding issue but it is not practical to let the customer use our internal images. Do you have any idea when the new images will be released ?

Best Regards,
Chen

(In reply to Chen from comment #37)
> Hi Eyal. Sanjay and Jon,
> 
> The customer confirmed that 10.0-20170228.1 from puddle solved the bonding
> issue but it is not practical to let the customer use our internal images.
> Do you have any idea when the new images will be released ?
> 
> Best Regards,
> Chen

Should be shipping today.

This is already shipped live, if you have additional issues here please open a new issue please open a new bug for the issue.