Red Hat Bugzilla – Bug 1496584
no permissions on namespaces "RTNETLINK answers: Invalid argument"
Last modified: 2018-10-25 15:13:06 EDT
no permissions on namespaces "RTNETLINK answers: Invalid argument" Environment: openstack-tripleo-heat-templates-7.0.1-0.20170925173113.afbe64a.el7ost.noarch openstack-puppet-modules-11.0.0-0.20170828113154.el7ost.noarch instack-undercloud-7.4.1-0.20170925172804.el7ost.noarch python-neutron-11.0.1-0.20170923193224.5b0191f.el7ost.noarch python-neutron-lbaas-11.0.2-0.20170906215706.c43b9a7.el7ost.noarch python-neutronclient-6.5.0-0.20170814170137.355983d.el7ost.noarch openstack-neutron-ml2-11.0.1-0.20170923193224.5b0191f.el7ost.noarch openstack-neutron-linuxbridge-11.0.1-0.20170923193224.5b0191f.el7ost.noarch puppet-neutron-11.3.1-0.20170923020712.889da59.el7ost.noarch openstack-neutron-common-11.0.1-0.20170923193224.5b0191f.el7ost.noarch openstack-neutron-11.0.1-0.20170923193224.5b0191f.el7ost.noarch openstack-neutron-sriov-nic-agent-11.0.1-0.20170923193224.5b0191f.el7ost.noarch openstack-neutron-metering-agent-11.0.1-0.20170923193224.5b0191f.el7ost.noarch python-neutron-lib-1.9.1-0.20170821170222.0ef54c3.el7ost.noarch openstack-neutron-lbaas-11.0.2-0.20170906215706.c43b9a7.el7ost.noarch openstack-neutron-openvswitch-11.0.1-0.20170923193224.5b0191f.el7ost.noarch Steps to reproduce: 1) Deploy overcloud with: openstack overcloud deploy --templates \ --libvirt-type kvm \ -e /home/stack/templates/nodes_data.yaml \ -r /home/stack/templates/roles_data.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-ansible.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \ -e /home/stack/virt/network/network-environment.yaml \ -e /home/stack/rhos12.yam 2) Create some network in overcloud Try to access the namespace for the network. Result: [root@overcloud-networker-0 ~]# ip netns RTNETLINK answers: Invalid argument RTNETLINK answers: Invalid argument RTNETLINK answers: Invalid argument qrouter-ff6b2c1d-ec95-4e52-9bb7-f0c3815f8ea1 RTNETLINK answers: Invalid argument qdhcp-8a6a0a64-c79d-4e62-9f8c-a5376a4ea2fb [root@overcloud-networker-0 ~]# ip netns exec qrouter-ff6b2c1d-ec95-4e52-9bb7-f0c3815f8ea1 ip -o a RTNETLINK answers: Invalid argument RTNETLINK answers: Invalid argument setting the network namespace "qrouter-ff6b2c1d-ec95-4e52-9bb7-f0c3815f8ea1" failed: Invalid argument [root@overcloud-networker-0 ~]# ls -la /var/run/netns/ total 0 drwxr-xr-x. 2 root root 80 Sep 27 20:38 . drwxr-xr-x. 52 root root 1540 Sep 27 20:38 .. ----------. 1 root root 0 Sep 27 20:38 qdhcp-8a6a0a64-c79d-4e62-9f8c-a5376a4ea2fb ----------. 1 root root 0 Sep 27 20:38 qrouter-ff6b2c1d-ec95-4e52-9bb7-f0c3815f8ea1 The issue reproduces.
I see neutron containers on networker: [root@overcloud-networker-0 ~]# docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 8991186bdca7 192.168.24.1:8787/rhosp12/openstack-cron-docker:2017-09-27.3 "kolla_start" About an hour ago Up About an hour logrotate_crond 80078e7cc93d 192.168.24.1:8787/rhosp12/openstack-neutron-openvswitch-agent-docker:2017-09-27.3 "kolla_start" About an hour ago Up About an hour (healthy) neutron_ovs_agent de2068ef75b6 192.168.24.1:8787/rhosp12/openstack-neutron-l3-agent-docker:2017-09-27.3 "kolla_start" About an hour ago Up About an hour (healthy) neutron_l3_agent 727419e067e2 192.168.24.1:8787/rhosp12/openstack-neutron-metadata-agent-docker:2017-09-27.3 "kolla_start" About an hour ago Up About an hour (healthy) neutron_metadata_agent 50998c2b12fa 192.168.24.1:8787/rhosp12/openstack-neutron-server-docker:2017-09-27.3 "kolla_start" About an hour ago Up About an hour neutron_api 53039d35f8cd 192.168.24.1:8787/rhosp12/openstack-neutron-dhcp-agent-docker:2017-09-27.3 "kolla_start" About an hour ago Up About an hour (healthy) neutron_dhcp
(overcloud) [stack@undercloud-0 ~]$ grep neutron /usr/share/openstack-tripleo-heat-templates/environments/docker.yaml OS::TripleO::Docker::NeutronMl2PluginBase: ../puppet/services/neutron-plugin-ml2.yaml OS::TripleO::Services::ComputeNeutronOvsAgent: ../docker/services/neutron-ovs-agent.yaml OS::TripleO::Services::NeutronServer: ../docker/services/neutron-api.yaml OS::TripleO::Services::NeutronApi: ../docker/services/neutron-api.yaml OS::TripleO::Services::NeutronCorePlugin: ../docker/services/neutron-plugin-ml2.yaml OS::TripleO::Services::NeutronMetadataAgent: ../docker/services/neutron-metadata.yaml OS::TripleO::Services::NeutronOvsAgent: ../docker/services/neutron-ovs-agent.yaml OS::TripleO::Services::NeutronDhcpAgent: ../docker/services/neutron-dhcp.yaml OS::TripleO::Services::NeutronL3Agent: ../docker/services/neutron-l3.yaml
Just checked the latest code on the remotes/rhos/rhos-12.0-patches branch of t-h-t. It looks to me that we still have the neutron services running on baremetal there via the following commit: commit 9f63530b2499aea5c23321cd38d5850567abe03c Author: Dan Prince <dprince@redhat.com> Date: Fri Jul 7 14:07:10 2017 -0400 [Downstream only] - Disable Neutron containers for OSP12 In order to preserve the partner interfaces for OSP 12 we are disabling Neutron containerization for OSP12. ---- Is it possible there is something that is editing the /usr/share/openstack-tripleo-heat-templates/environments/docker.yaml post install? Or maybe an older version of the RPM got pulled in somehow before this patch was applied to the OSP12 branch.
The RPM didn't get changed: (overcloud) [stack@undercloud-0 dd]$ rpm -qf /usr/share/openstack-tripleo-heat-templates/environments/docker.yaml openstack-tripleo-heat-templates-7.0.1-0.20170925173113.afbe64a.el7ost.noarch (overcloud) [stack@undercloud-0 dd]$ rpm -V openstack-tripleo-heat-templates-7.0.1-0.20170925173113.afbe64a.el7ost.noarch (overcloud) [stack@undercloud-0 dd]$
This is a doc issue, assigned to Brian to write a release note.
To clarify you need to run commands in the namespace from the relevant agent, we'll document this and check with the kernel team if there's anything further we can do.
So this issue here is that with containers, you need to be in the container to exec into the namespace. Here is a potential release note: "When neutron services are containerized, trying to run commands in a network namespace might fail with: # ip netns exec qrouter... RTNETLINK answers: Invalid argument In order to run a command inside a network namespace, you must do it from the neutron container that created the namespace. For example, the l3-agent creates network namespace for routers, so the command would need to change to: # docker exec neutron_l3_agent ip netns exec qrouter... Similarly with network namespaces beginning with 'qdhcp' you would need to exec from the 'neutron_dhcp' container."
Doc text is done, I don't believe there is anything else to fix for this bug.