Bug 1729764

Summary: [OSP15- OVN]Metadata service does not work on TLS deployment
Product: Red Hat OpenStack Reporter: Eran Kuris <ekuris>
Component: python-networking-ovnAssignee: Maciej Józefczyk <mjozefcz>
Status: CLOSED ERRATA QA Contact: Eran Kuris <ekuris>
Severity: high Docs Contact:
Priority: high    
Version: 15.0 (Stein)CC: amuller, apevec, atragler, jlibosva, jschluet, lhh, majopela, mjozefcz, scohen
Target Milestone: rcKeywords: AutomationBlocker, Triaged
Target Release: 15.0 (Stein)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: python-networking-ovn-6.0.1-0.20190813160454.89e84c9.el8ost Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-09-21 11:23:52 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Eran Kuris 2019-07-14 11:21:43 UTC 
Description of problem:
All scenarios tests are failing in OVN TLS jibs.
It looks like metadata service does not work
WARN: /etc/rc3.d/S10-load-modules failed
Initializing random number generator... [    5.022762] random: dd urandom read with 24 bits of entropy available
done.
Starting acpid: OK
Starting network...
udhcpc (v1.23.2) started
Sending discover...
Sending select for 10.100.0.8...
Lease of 10.100.0.8 obtained, lease time 43200
route: SIOCADDRT: File exists
WARN: failed: route add -net "0.0.0.0/0" gw "10.100.0.1"
checking http://169.254.169.254/2009-04-04/instance-id
failed 1/20: up 5.19. request failed
failed 2/20: up 7.24. request failed
failed 3/20: up 9.27. request failed
failed 4/20: up 11.30. request failed
failed 5/20: up 13.33. request failed
failed 6/20: up 15.36. request failed
failed 7/20: up 17.39. request failed
failed 8/20: up 19.42. request failed
failed 9/20: up 21.46. request failed
failed 10/20: up 23.49. request failed
failed 11/20: up 25.51. request failed
failed 12/20: up 27.55. request failed
failed 13/20: up 29.58. request failed
failed 14/20: up 31.61. request failed
failed 15/20: up 33.63. request failed
failed 16/20: up 35.67. request failed
failed 17/20: up 37.70. request failed
failed 18/20: up 39.73. request failed
failed 19/20: up 41.76. request failed
failed 20/20: up 43.79. request failed
failed to read iid from metadata. tried 20
failed to get instance-id of datasource
Top of dropbear init script
Starting dropbear sshd: failed to get instance-id of datasource
OK
GROWROOT: CHANGED: partition=1 start=18432 old: size=71647 end=90079 new: size=2078687,end=2097119
=== system information ===
Platform: Red Hat OpenStack Compute
Container: none
Arch: x86_64
CPU(s): 1 @ 2199.996 MHz
Cores/Sockets/Threads: 1/1/1
Virt-type: 
RAM Size: 46MB
Disks:
NAME  MAJ:MIN       SIZE LABEL         MOUNTPOINT
vda   253:0   1073741824               
vda1  253:1   1064287744 cirros-rootfs /
vda15 253:15     8388608               
=== sshd host keys ===
-----BEGIN SSH HOST KEY KEYS-----
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCCr+eK9Vn09GbbnTDuTznYtIocW0q2zldLOcD+MFISHgZSEKhpJRVDt/47fuAtClMUXiK61tkjpfOjo0Ry9GWHDkE2Vdkjkn4sdvvmQascGYpdJ/bCbi65jPhPoUE1XTwfJn/w0CfLSn5LSWtFDSQXuUX6m9cLhcqV4s1CRVDWE/9KZ2vWLHQHseXGWi8FK2XMUNzR5L/XCJmNu8mkA030695Iu8UrQU6d8DUwc13/BIFy4QjlzR76t/7y+iHr67wPzA22hO6ZtiS7sDQxXeIT1gTWhP5QPg1ja2jAIsfWqculunzIgZYHneWnFgC2LvVhg5TPkG0IeIcokLj+O3OL root@cirros
ssh-dss AAAAB3NzaC1kc3MAAACBAN+Z5ZEPv4oqHACb9AY3Z7KlmWR6X2ajOym3wRsev+ekf56lTfqXhfGkMswNPdb1jik1ruIO9bmpR8TXScOp/2cYpTwWgu4sldOvFhaNjwe80C1o0pDVK4H2RsQJ9VYijjklQVXMOJcERaCsQSuVU3S+hjdgiu4qhpGb83y8wY9/AAAAFQC1QcEv2SPyd2GU8gezSxFuUZul+QAAAIB8F+zwUQJ2UL0cxZl74mwvX8Xp/nzvOyQFH0Gx75+FKuPie9s1jZskMRWPBOWwOn+AKxlqYDFA8M1loS/etv/Amnkt7VZiljTH7HLa7dwOBq3TsKa5Qtgl/Kn+eD+QQnG14bBbdWHHqdP5I8/zjn7EOoGb3FoIVW9KZawcEnYb2AAAAIEAlYFKHLEXa8gBvFMBC70ZNTFP5DH/pcq0g00wzWf5vELAwbHmTRTSp1IP6Dzdti1SuqUjAXmkK+5d5/G0626QVZGMns1YeykBKYJQQlqER/a0GLCzRxN316inFnezT9CWjDHla/uhnkPh0AeVCjqIHJ32i7WsxBkDQJaWhAfEBOg= root@cirros
-----END SSH HOST KEY KEYS-----
=== network info ===
if-info: lo,up,127.0.0.1,8,,
if-info: eth0,up,10.100.0.8,28,fe80::f816:3eff:fec3:eaf8/64,
ip-route:default via 10.100.0.1 dev eth0 
ip-route:10.100.0.0/28 dev eth0  src 10.100.0.8 
ip-route:169.254.169.254 via 10.100.0.2 dev eth0 
ip-route6:fe80::/64 dev eth0  metric 256 
ip-route6:unreachable default dev lo  metric -1  error -101
ip-route6:ff00::/8 dev eth0  metric 256 
ip-route6:unreachable default dev lo  metric -1  error -101
=== datasource: None None ===
=== cirros: current=0.4.0 uptime=46.62 ===
  ____               ____  ____
 / __/ __ ____ ____ / __ \/ __/
/ /__ / // __// __// /_/ /\ \ 
\___//_//_/  /_/   \____/___/ 
   http://cirros-cloud.net

/dev/root resized successfully [took 0.41s]

login as 'cirros' user. default password: 'gocubsgo'. use 'sudo' for root.
cirros login: [  212.801369] random: nonblocking pool is initialized

(overcloud) [stack@undercloud-0 etc]$ ssh -i  /tmp/key2   root.0.244
Warning: Permanently added '10.0.0.244' (ECDSA) to the list of known hosts.
root.0.244's password: 

(overcloud) [stack@undercloud-0 etc]$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master ovs-system state UP group default qlen 1000
    link/ether 52:54:00:21:4d:b6 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::5054:ff:fe21:4db6/64 scope link 
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 52:54:00:ca:f8:bf brd ff:ff:ff:ff:ff:ff
    inet 172.16.0.30/24 brd 172.16.0.255 scope global dynamic noprefixroute eth1
       valid_lft 3182sec preferred_lft 3182sec
    inet6 fe80::5054:ff:feca:f8bf/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 52:54:00:3d:27:d6 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.83/24 brd 10.0.0.255 scope global dynamic noprefixroute eth2
       valid_lft 3594sec preferred_lft 3594sec
    inet6 fe80::5054:ff:fe3d:27d6/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
5: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether e6:3c:aa:38:d0:81 brd ff:ff:ff:ff:ff:ff
6: br-ctlplane: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether 52:54:00:21:4d:b6 brd ff:ff:ff:ff:ff:ff
    inet 192.168.24.1/24 brd 192.168.24.255 scope global br-ctlplane
       valid_lft forever preferred_lft forever
    inet 192.168.24.3/32 scope global br-ctlplane
       valid_lft forever preferred_lft forever
    inet 192.168.24.2/32 scope global br-ctlplane
       valid_lft forever preferred_lft forever
    inet6 fe80::5054:ff:fe21:4db6/64 scope link 
       valid_lft forever preferred_lft forever
7: br-int: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether f2:00:6c:1c:2d:43 brd ff:ff:ff:ff:ff:ff
(overcloud) [stack@undercloud-0 etc]$ ssh -i  /tmp/key2   cirros.0.244
Warning: Permanently added '10.0.0.244' (ECDSA) to the list of known hosts.
cirros.0.244's password: 
$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1442 qdisc pfifo_fast qlen 1000
    link/ether fa:16:3e:c3:ea:f8 brd ff:ff:ff:ff:ff:ff
    inet 10.100.0.8/28 brd 10.100.0.15 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fec3:eaf8/64 scope link 
       valid_lft forever preferred_lft forever
$ route 
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         10.100.0.1      0.0.0.0         UG    0      0        0 eth0
10.100.0.0      *               255.255.255.240 U     0      0        0 eth0
169.254.169.254 10.100.0.2      255.255.255.255 UGH   0      0        0 eth0
$ ip r
default via 10.100.0.1 dev eth0 
10.100.0.0/28 dev eth0  src 10.100.0.8 
169.254.169.254 via 10.100.0.2 dev eth0 
$ curl http://169.254.169.254/2009-04-04/instance-id
<html>
 <head>
  <title>500 Internal Server Error</title>
 </head>
 <body>
  <h1>500 Internal Server Error</h1>
  An unknown error has occurred. Please try your request again.<br /><br 



 </body>
</html>$ 
 
$ curl http://169.254.169.254/

 </body>
</html>$ 
$ curl http://169.254.169.254/
<html>
 <head>
  <title>500 Internal Server Error</title>
 </head>
 <body>
  <h1>500 Internal Server Error</h1>
  An unknown error has occurred. Please try your request again.<br /><br />

The status of the agent is up : 
(.vevn) (overcloud) [stack@undercloud-0 tempest]$ openstack network agent list
+--------------------------------------+------------------------------+---------------------------+-------------------+-------+-------+-------------------------------+
| ID                                   | Agent Type                   | Host                      | Availability Zone | Alive | State | Binary                        |
+--------------------------------------+------------------------------+---------------------------+-------------------+-------+-------+-------------------------------+
| 62925450-1feb-4341-89f7-3a47f27f0b71 | OVN Controller agent         | compute-1.redhat.local    | n/a               | :-)   | UP    | ovn-controller                |
| a4881130-8f58-481e-a7f7-c1aab5d9ce94 | OVN Metadata agent           | compute-1.redhat.local    | n/a               | :-)   | UP    | networking-ovn-metadata-agent |
| 600dba3f-6e24-45c0-bd0a-2ab3c16e2805 | OVN Controller agent         | compute-0.redhat.local    | n/a               | :-)   | UP    | ovn-controller                |
| 84315d5f-719d-4859-b09c-f5329b48e9c4 | OVN Metadata agent           | compute-0.redhat.local    | n/a               | :-)   | UP    | networking-ovn-metadata-agent |
| 731603be-92d1-4744-8c0e-2fe59dccfcca | OVN Controller Gateway agent | controller-0.redhat.local | n/a               | :-)   | UP    | ovn-controller                |
| a250901f-506e-4151-8d8e-538713a97906 | OVN Controller Gateway agent | controller-1.redhat.local | n/a               | :-)   | UP    | ovn-controller                |
| 183b8a53-d91f-4e4d-a391-cd44139a13d5 | OVN Controller Gateway agent | controller-2.redhat.local | n/a               | :-)   | UP    | ovn-controller                |
+--------------------------------------+------------------------------+--------


Version-Release number of selected component (if applicable):
OSP15 core_puddle: RHOS_TRUNK-15.0-RHEL-8-20190708.n.2

How reproducible:
100%

Steps to Reproduce:
1.run ovn tls job: https://rhos-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/DFG/view/network/view/networking-ovn/job/DFG-network-networking-ovn-15_director-rhel-virthost-3cont_2comp_1ipa-ipv4-geneve-tls/
2. when a scenarion test is running connect to the VM with ssh and check the metadata service curl http://169.254.169.254/ 

3.

Actual results:


Expected results:


Additional info:
Comment 1 Eran Kuris 2019-07-14 11:50:03 UTC 
Regression from OSP14
Comment 9 Maciej Józefczyk 2019-07-30 13:05:23 UTC 
Merged upstream.
Comment 15 Eran Kuris 2019-08-29 06:04:34 UTC 
Fixed verified on  CI run. all the scenario tests passed.
https://rhos-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/DFG/view/network/view/networking-ovn/job/DFG-network-networking-ovn-15_director-rhel-virthost-3cont_2comp_1ipa-ipv4-geneve-tls/63/

core_puddle: RHOS_TRUNK-15.0-RHEL-8-20190827.n.0
Comment 20 errata-xmlrpc 2019-09-21 11:23:52 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2019:2811