Red Hat Bugzilla – Bug 1589849
[OVN] Stopping metadata agent on a compute node will leave running/new instances without metadata
Last modified: 2018-10-31 09:29:58 EDT
Description of problem: When OVN metadata agent (controlplane) is stopped, any VM being restarted on that compute node won't be able to access the metadata service (dataplane) as all the haproxy processes will go away with the agent container. How reproducible: 100% Steps to Reproduce: 1. Spawn a VM on a compute node 2. Stop the agent in that compute node 3. Restart the VM Actual results: The VM will fail to fetch metadata at boot. Expected results: Access to metadata should still be possible. Additional info: We need to apply the same approach as we do in other neutron containers of creating a sidecar container for haproxy instances which remains running after stopping the agent.
Patches merged upstream
For clarification and helping QA to verify this BZ, this is what we expect on a compute node hosting a VM: [heat-admin@compute-1 ~]$ sudo docker ps | grep metadata-agent 3d4b63f13a8e 192.168.24.1:8787/rhosp14/openstack-neutron-metadata-agent-ovn:2018-10-01.1 "ip netns exec ovn..." 19 minutes ago Up 19 minutes neutron-haproxy-ovnmeta-2f7d8747-bce6-4afd-8f20-cff29e531ff4 aea63c8cb334 192.168.24.1:8787/rhosp14/openstack-neutron-metadata-agent-ovn:2018-10-01.1 "kolla_start" 15 hours ago Up 15 hours (healthy) ovn_metadata_agent The docker container running the OVN metadata agent plus a sidecar container running the haproxy instance. If we switch the VM off, that sidecar container is expected to go away: (overcloud) [stack@undercloud-0 ~]$ openstack server stop cirros1 [heat-admin@compute-1 ~]$ sudo docker ps | grep metadata-agent aea63c8cb334 192.168.24.1:8787/rhosp14/openstack-neutron-metadata-agent-ovn:2018-10-01.1 "kolla_start" 16 hours ago Up 16 hours (healthy) ovn_metadata_agent [heat-admin@compute-1 ~]$ ps -aef | grep haproxy [heat-admin@compute-1 ~]$
*** Bug 1633594 has been marked as a duplicate of this bug. ***
The issue is still exists: After stopping the container, there is still haproxy sidecar container but it is not serving the metadata )[root@compute-0 /]# ps -aef | grep haproxy neutron 111054 111034 0 Oct29 ? 00:00:00 /usr/sbin/haproxy -Ds -f /var/lib/neutron/ovn-metadata-proxy/056ca522-7e19-44c0-985c-c9599872ceb1.conf neutron 111075 111054 0 Oct29 ? 00:00:05 /usr/sbin/haproxy -Ds -f /var/lib/neutron/ovn-metadata-proxy/056ca522-7e19-44c0-985c-c9599872ceb1.conf root 863630 863578 0 07:54 ? 00:00:00 grep --color=auto haproxy ()[root@compute-0 /]# ls /var/lib/neutron/ovn-metadata-proxy/056ca522-7e19-44c0-985c-c9599872ceb1.conf /var/lib/neutron/ovn-metadata-proxy/056ca522-7e19-44c0-985c-c9599872ceb1.conf Starting Execute cloud user/final scripts... [ 63.407748] cloud-init[979]: Cloud-init v. 0.7.9 running 'modules:final' at Wed, 31 Oct 2018 07:14:15 +0000. Up 63.25 seconds. [ 63.521554] cloud-init[979]: Cloud-init v. 0.7.9 finished at Wed, 31 Oct 2018 07:14:15 +0000. Datasource DataSourceOpenStack [net,ver=2]. Up 63.50 seconds [ OK ] Started Execute cloud user/final scripts. [ OK ] Reached target Multi-User System. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Red Hat Enterprise Linux Server 7.5 (Maipo) Kernel 3.10.0-860.el7.x86_64 on an x86_64 net-64-1-vm-1 login: root Password: Last login: Wed Oct 31 03:11:44 on ttyS0 [root@net-64-1-vm-1 ~]# curl http://169.254.169.254/ <html><body><h1>503 Service Unavailable</h1> No server is available to handle this request. re-opened on: openstack-tripleo-heat-templates-8.0.7-4.el7ost.noarch OSP13z3 2018-10-24.1
re-open
(In reply to Eran Kuris from comment #21) > The issue is still exists: > > After stopping the container, there is still haproxy sidecar container > but it is not serving the metadata > )[root@compute-0 /]# ps -aef | grep haproxy > neutron 111054 111034 0 Oct29 ? 00:00:00 /usr/sbin/haproxy -Ds -f > /var/lib/neutron/ovn-metadata-proxy/056ca522-7e19-44c0-985c-c9599872ceb1.conf > neutron 111075 111054 0 Oct29 ? 00:00:05 /usr/sbin/haproxy -Ds -f > /var/lib/neutron/ovn-metadata-proxy/056ca522-7e19-44c0-985c-c9599872ceb1.conf > root 863630 863578 0 07:54 ? 00:00:00 grep --color=auto haproxy > ()[root@compute-0 /]# ls > /var/lib/neutron/ovn-metadata-proxy/056ca522-7e19-44c0-985c-c9599872ceb1.conf > /var/lib/neutron/ovn-metadata-proxy/056ca522-7e19-44c0-985c-c9599872ceb1.conf > > Can you show the sidecar containers through docker ps and not ps? Like: [heat-admin@compute-1 ~]$ sudo docker ps | grep metadata-agent 3d4b63f13a8e 192.168.24.1:8787/rhosp14/openstack-neutron-metadata-agent-ovn:2018-10-01.1 "ip netns exec ovn..." 19 minutes ago Up 19 minutes neutron-haproxy-ovnmeta-2f7d8747-bce6-4afd-8f20-cff29e531ff4 aea63c8cb334 192.168.24.1:8787/rhosp14/openstack-neutron-metadata-agent-ovn:2018-10-01.1 "kolla_start" 15 hours ago Up 15 hours (healthy) ovn_metadata_agent
(In reply to Daniel Alvarez Sanchez from comment #23) > (In reply to Eran Kuris from comment #21) > > The issue is still exists: > > > > After stopping the container, there is still haproxy sidecar container > > but it is not serving the metadata > > )[root@compute-0 /]# ps -aef | grep haproxy > > neutron 111054 111034 0 Oct29 ? 00:00:00 /usr/sbin/haproxy -Ds -f > > /var/lib/neutron/ovn-metadata-proxy/056ca522-7e19-44c0-985c-c9599872ceb1.conf > > neutron 111075 111054 0 Oct29 ? 00:00:05 /usr/sbin/haproxy -Ds -f > > /var/lib/neutron/ovn-metadata-proxy/056ca522-7e19-44c0-985c-c9599872ceb1.conf > > root 863630 863578 0 07:54 ? 00:00:00 grep --color=auto haproxy > > ()[root@compute-0 /]# ls > > /var/lib/neutron/ovn-metadata-proxy/056ca522-7e19-44c0-985c-c9599872ceb1.conf > > /var/lib/neutron/ovn-metadata-proxy/056ca522-7e19-44c0-985c-c9599872ceb1.conf > > > > > > Can you show the sidecar containers through docker ps and not ps? Like: > > [heat-admin@compute-1 ~]$ sudo docker ps | grep metadata-agent > 3d4b63f13a8e > 192.168.24.1:8787/rhosp14/openstack-neutron-metadata-agent-ovn:2018-10-01.1 > "ip netns exec ovn..." 19 minutes ago Up 19 minutes > neutron-haproxy-ovnmeta-2f7d8747-bce6-4afd-8f20-cff29e531ff4 > aea63c8cb334 > 192.168.24.1:8787/rhosp14/openstack-neutron-metadata-agent-ovn:2018-10-01.1 > "kolla_start" 15 hours ago Up 15 hours (healthy) > ovn_metadata_agent yes, I can : root@compute-0 ~]# docker ps -a | grep meta 194feb06385f 192.168.24.1:8787/rhosp13/openstack-neutron-metadata-agent-ovn:2018-10-24.1 "ip netns exec ovn..." 42 hours ago Up 42 hours neutron-haproxy-ovnmeta-056ca522-7e19-44c0-985c-c9599872ceb1 788d1b1310b9 192.168.24.1:8787/rhosp13/openstack-neutron-metadata-agent-ovn:2018-10-24.1 "kolla_start" 44 hours ago Exited (0) 5 seconds ago ovn_metadata_agent 8c5a2aa9780b 192.168.24.1:8787/rhosp13/openstack-neutron-metadata-agent-ovn:2018-10-24.1 "/docker_puppet_ap..." 44 hours ago Exited (0) 44 hours ago setup_ovs_manager fad957a1a538 192.168.24.1:8787/rhosp13/openstack-neutron-metadata-agent-ovn:2018-10-24.1 "/docker_puppet_ap..." 45 hours ago Exited (0) 45 hours ago create_haproxy_wrapper