Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1496584

Summary: no permissions on namespaces "RTNETLINK answers: Invalid argument"
Product: Red Hat OpenStack Reporter: Alexander Chuzhoy <sasha>
Component: openstack-neutronAssignee: Brian Haley <bhaley>
Status: CLOSED ERRATA QA Contact: Federico Ressi <fressi>
Severity: high Docs Contact:
Priority: high    
Version: 12.0 (Pike)CC: amuller, bfournie, bhaley, ccamposr, chrisw, dprince, jlibosva, jschluet, lmarsh, mlammon, njohnston, ragiman, sclewis, srevivo, tvignaud
Target Milestone: z3Keywords: Documentation, TestOnly, Triaged, ZStream
Target Release: 13.0 (Queens)Flags: bhaley: needinfo-
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-neutron-12.0.3-5.el7ost Doc Type: Release Note
Doc Text:
When neutron services are containerized, trying to run commands in a network namespace might fail with the following error: # ip netns exec qrouter... RTNETLINK answers: Invalid argument To run a command inside a network namespace, you must do it from the neutron container that created the namespace. For example, the l3-agent creates network namespace for routers, so the command should be: # docker exec neutron_l3_agent ip netns exec qrouter... Similarly, with network namespaces beginning with 'qdhcp' you would need to exec from the 'neutron_dhcp' container.
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-05-09 16:00:01 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Alexander Chuzhoy 2017-09-27 20:59:04 UTC
no permissions on namespaces "RTNETLINK answers: Invalid argument"

Environment:
openstack-tripleo-heat-templates-7.0.1-0.20170925173113.afbe64a.el7ost.noarch
openstack-puppet-modules-11.0.0-0.20170828113154.el7ost.noarch
instack-undercloud-7.4.1-0.20170925172804.el7ost.noarch


python-neutron-11.0.1-0.20170923193224.5b0191f.el7ost.noarch
python-neutron-lbaas-11.0.2-0.20170906215706.c43b9a7.el7ost.noarch
python-neutronclient-6.5.0-0.20170814170137.355983d.el7ost.noarch
openstack-neutron-ml2-11.0.1-0.20170923193224.5b0191f.el7ost.noarch
openstack-neutron-linuxbridge-11.0.1-0.20170923193224.5b0191f.el7ost.noarch
puppet-neutron-11.3.1-0.20170923020712.889da59.el7ost.noarch
openstack-neutron-common-11.0.1-0.20170923193224.5b0191f.el7ost.noarch
openstack-neutron-11.0.1-0.20170923193224.5b0191f.el7ost.noarch
openstack-neutron-sriov-nic-agent-11.0.1-0.20170923193224.5b0191f.el7ost.noarch
openstack-neutron-metering-agent-11.0.1-0.20170923193224.5b0191f.el7ost.noarch
python-neutron-lib-1.9.1-0.20170821170222.0ef54c3.el7ost.noarch
openstack-neutron-lbaas-11.0.2-0.20170906215706.c43b9a7.el7ost.noarch
openstack-neutron-openvswitch-11.0.1-0.20170923193224.5b0191f.el7ost.noarch

Steps to reproduce:
1)
Deploy overcloud with:
openstack overcloud deploy --templates \
--libvirt-type kvm \
-e /home/stack/templates/nodes_data.yaml \
-r /home/stack/templates/roles_data.yaml \
-e  /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-ansible.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
-e /home/stack/virt/network/network-environment.yaml \
-e /home/stack/rhos12.yam


2) Create some network in overcloud

Try to access the namespace for the network.


Result:
[root@overcloud-networker-0 ~]# ip netns
RTNETLINK answers: Invalid argument
RTNETLINK answers: Invalid argument
RTNETLINK answers: Invalid argument
qrouter-ff6b2c1d-ec95-4e52-9bb7-f0c3815f8ea1
RTNETLINK answers: Invalid argument
qdhcp-8a6a0a64-c79d-4e62-9f8c-a5376a4ea2fb


[root@overcloud-networker-0 ~]# ip netns exec qrouter-ff6b2c1d-ec95-4e52-9bb7-f0c3815f8ea1 ip -o a
RTNETLINK answers: Invalid argument
RTNETLINK answers: Invalid argument
setting the network namespace "qrouter-ff6b2c1d-ec95-4e52-9bb7-f0c3815f8ea1" failed: Invalid argument


[root@overcloud-networker-0 ~]# ls -la /var/run/netns/
total 0
drwxr-xr-x.  2 root root   80 Sep 27 20:38 .
drwxr-xr-x. 52 root root 1540 Sep 27 20:38 ..
----------.  1 root root    0 Sep 27 20:38 qdhcp-8a6a0a64-c79d-4e62-9f8c-a5376a4ea2fb
----------.  1 root root    0 Sep 27 20:38 qrouter-ff6b2c1d-ec95-4e52-9bb7-f0c3815f8ea1



The issue reproduces.

Comment 1 Alexander Chuzhoy 2017-09-27 21:52:46 UTC
I see neutron containers on networker:
[root@overcloud-networker-0 ~]# docker ps
CONTAINER ID        IMAGE                                                                               COMMAND             CREATED             STATUS                       PORTS               NAMES
8991186bdca7        192.168.24.1:8787/rhosp12/openstack-cron-docker:2017-09-27.3                        "kolla_start"       About an hour ago   Up About an hour                                 logrotate_crond
80078e7cc93d        192.168.24.1:8787/rhosp12/openstack-neutron-openvswitch-agent-docker:2017-09-27.3   "kolla_start"       About an hour ago   Up About an hour (healthy)                       neutron_ovs_agent
de2068ef75b6        192.168.24.1:8787/rhosp12/openstack-neutron-l3-agent-docker:2017-09-27.3            "kolla_start"       About an hour ago   Up About an hour (healthy)                       neutron_l3_agent
727419e067e2        192.168.24.1:8787/rhosp12/openstack-neutron-metadata-agent-docker:2017-09-27.3      "kolla_start"       About an hour ago   Up About an hour (healthy)                       neutron_metadata_agent
50998c2b12fa        192.168.24.1:8787/rhosp12/openstack-neutron-server-docker:2017-09-27.3              "kolla_start"       About an hour ago   Up About an hour                                 neutron_api
53039d35f8cd        192.168.24.1:8787/rhosp12/openstack-neutron-dhcp-agent-docker:2017-09-27.3          "kolla_start"       About an hour ago   Up About an hour (healthy)                       neutron_dhcp

Comment 2 Alexander Chuzhoy 2017-09-27 22:07:29 UTC
(overcloud) [stack@undercloud-0 ~]$ grep neutron /usr/share/openstack-tripleo-heat-templates/environments/docker.yaml
  OS::TripleO::Docker::NeutronMl2PluginBase: ../puppet/services/neutron-plugin-ml2.yaml
  OS::TripleO::Services::ComputeNeutronOvsAgent: ../docker/services/neutron-ovs-agent.yaml
  OS::TripleO::Services::NeutronServer: ../docker/services/neutron-api.yaml
  OS::TripleO::Services::NeutronApi: ../docker/services/neutron-api.yaml
  OS::TripleO::Services::NeutronCorePlugin: ../docker/services/neutron-plugin-ml2.yaml
  OS::TripleO::Services::NeutronMetadataAgent: ../docker/services/neutron-metadata.yaml
  OS::TripleO::Services::NeutronOvsAgent: ../docker/services/neutron-ovs-agent.yaml
  OS::TripleO::Services::NeutronDhcpAgent: ../docker/services/neutron-dhcp.yaml
  OS::TripleO::Services::NeutronL3Agent: ../docker/services/neutron-l3.yaml

Comment 3 Dan Prince 2017-09-28 01:50:51 UTC
Just checked the latest code on the remotes/rhos/rhos-12.0-patches branch of t-h-t. It looks to me that we still have the neutron services running on baremetal there via the following commit:

commit 9f63530b2499aea5c23321cd38d5850567abe03c
Author: Dan Prince <dprince>
Date:   Fri Jul 7 14:07:10 2017 -0400

    [Downstream only] - Disable Neutron containers for OSP12
    
    In order to preserve the partner interfaces for OSP 12 we are disabling
    Neutron containerization for OSP12.

----

Is it possible there is something that is editing the /usr/share/openstack-tripleo-heat-templates/environments/docker.yaml post install? Or maybe an older version of the RPM got pulled in somehow before this patch was applied to the OSP12 branch.

Comment 4 Alexander Chuzhoy 2017-09-28 13:01:20 UTC
The RPM didn't get changed:

(overcloud) [stack@undercloud-0 dd]$ rpm -qf /usr/share/openstack-tripleo-heat-templates/environments/docker.yaml
openstack-tripleo-heat-templates-7.0.1-0.20170925173113.afbe64a.el7ost.noarch
(overcloud) [stack@undercloud-0 dd]$ rpm -V openstack-tripleo-heat-templates-7.0.1-0.20170925173113.afbe64a.el7ost.noarch
(overcloud) [stack@undercloud-0 dd]$

Comment 6 Assaf Muller 2017-10-02 13:13:00 UTC
This is a doc issue, assigned to Brian to write a release note.

Comment 7 Assaf Muller 2017-10-02 13:13:39 UTC
To clarify you need to run commands in the namespace from the relevant agent, we'll document this and check with the kernel team if there's anything further we can do.

Comment 11 Brian Haley 2018-06-26 17:02:39 UTC
So this issue here is that with containers, you need to be in the container to exec into the namespace.  Here is a potential release note:

"When neutron services are containerized, trying to run commands in a network namespace might fail with:

# ip netns exec qrouter...
RTNETLINK answers: Invalid argument

In order to run a command inside a network namespace, you must do it from the neutron container that created the namespace.  For example, the l3-agent creates network namespace for routers, so the command would need to change to:

# docker exec neutron_l3_agent ip netns exec qrouter...

Similarly with network namespaces beginning with 'qdhcp' you would need to exec from the 'neutron_dhcp' container."

Comment 14 Brian Haley 2018-09-10 23:18:29 UTC
Doc text is done, I don't believe there is anything else to fix for this bug.

Comment 21 Nate Johnston 2019-05-09 16:00:01 UTC
This fix was shipped in a Pike z-stream release in October 2018.  Closing.