Description of problem: All scenarios tests are failing in OVN TLS jibs. It looks like metadata service does not work WARN: /etc/rc3.d/S10-load-modules failed Initializing random number generator... [ 5.022762] random: dd urandom read with 24 bits of entropy available done. Starting acpid: OK Starting network... udhcpc (v1.23.2) started Sending discover... Sending select for 10.100.0.8... Lease of 10.100.0.8 obtained, lease time 43200 route: SIOCADDRT: File exists WARN: failed: route add -net "0.0.0.0/0" gw "10.100.0.1" checking http://169.254.169.254/2009-04-04/instance-id failed 1/20: up 5.19. request failed failed 2/20: up 7.24. request failed failed 3/20: up 9.27. request failed failed 4/20: up 11.30. request failed failed 5/20: up 13.33. request failed failed 6/20: up 15.36. request failed failed 7/20: up 17.39. request failed failed 8/20: up 19.42. request failed failed 9/20: up 21.46. request failed failed 10/20: up 23.49. request failed failed 11/20: up 25.51. request failed failed 12/20: up 27.55. request failed failed 13/20: up 29.58. request failed failed 14/20: up 31.61. request failed failed 15/20: up 33.63. request failed failed 16/20: up 35.67. request failed failed 17/20: up 37.70. request failed failed 18/20: up 39.73. request failed failed 19/20: up 41.76. request failed failed 20/20: up 43.79. request failed failed to read iid from metadata. tried 20 failed to get instance-id of datasource Top of dropbear init script Starting dropbear sshd: failed to get instance-id of datasource OK GROWROOT: CHANGED: partition=1 start=18432 old: size=71647 end=90079 new: size=2078687,end=2097119 === system information === Platform: Red Hat OpenStack Compute Container: none Arch: x86_64 CPU(s): 1 @ 2199.996 MHz Cores/Sockets/Threads: 1/1/1 Virt-type: RAM Size: 46MB Disks: NAME MAJ:MIN SIZE LABEL MOUNTPOINT vda 253:0 1073741824 vda1 253:1 1064287744 cirros-rootfs / vda15 253:15 8388608 === sshd host keys === -----BEGIN SSH HOST KEY KEYS----- ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCCr+eK9Vn09GbbnTDuTznYtIocW0q2zldLOcD+MFISHgZSEKhpJRVDt/47fuAtClMUXiK61tkjpfOjo0Ry9GWHDkE2Vdkjkn4sdvvmQascGYpdJ/bCbi65jPhPoUE1XTwfJn/w0CfLSn5LSWtFDSQXuUX6m9cLhcqV4s1CRVDWE/9KZ2vWLHQHseXGWi8FK2XMUNzR5L/XCJmNu8mkA030695Iu8UrQU6d8DUwc13/BIFy4QjlzR76t/7y+iHr67wPzA22hO6ZtiS7sDQxXeIT1gTWhP5QPg1ja2jAIsfWqculunzIgZYHneWnFgC2LvVhg5TPkG0IeIcokLj+O3OL root@cirros ssh-dss AAAAB3NzaC1kc3MAAACBAN+Z5ZEPv4oqHACb9AY3Z7KlmWR6X2ajOym3wRsev+ekf56lTfqXhfGkMswNPdb1jik1ruIO9bmpR8TXScOp/2cYpTwWgu4sldOvFhaNjwe80C1o0pDVK4H2RsQJ9VYijjklQVXMOJcERaCsQSuVU3S+hjdgiu4qhpGb83y8wY9/AAAAFQC1QcEv2SPyd2GU8gezSxFuUZul+QAAAIB8F+zwUQJ2UL0cxZl74mwvX8Xp/nzvOyQFH0Gx75+FKuPie9s1jZskMRWPBOWwOn+AKxlqYDFA8M1loS/etv/Amnkt7VZiljTH7HLa7dwOBq3TsKa5Qtgl/Kn+eD+QQnG14bBbdWHHqdP5I8/zjn7EOoGb3FoIVW9KZawcEnYb2AAAAIEAlYFKHLEXa8gBvFMBC70ZNTFP5DH/pcq0g00wzWf5vELAwbHmTRTSp1IP6Dzdti1SuqUjAXmkK+5d5/G0626QVZGMns1YeykBKYJQQlqER/a0GLCzRxN316inFnezT9CWjDHla/uhnkPh0AeVCjqIHJ32i7WsxBkDQJaWhAfEBOg= root@cirros -----END SSH HOST KEY KEYS----- === network info === if-info: lo,up,127.0.0.1,8,, if-info: eth0,up,10.100.0.8,28,fe80::f816:3eff:fec3:eaf8/64, ip-route:default via 10.100.0.1 dev eth0 ip-route:10.100.0.0/28 dev eth0 src 10.100.0.8 ip-route:169.254.169.254 via 10.100.0.2 dev eth0 ip-route6:fe80::/64 dev eth0 metric 256 ip-route6:unreachable default dev lo metric -1 error -101 ip-route6:ff00::/8 dev eth0 metric 256 ip-route6:unreachable default dev lo metric -1 error -101 === datasource: None None === === cirros: current=0.4.0 uptime=46.62 === ____ ____ ____ / __/ __ ____ ____ / __ \/ __/ / /__ / // __// __// /_/ /\ \ \___//_//_/ /_/ \____/___/ http://cirros-cloud.net /dev/root resized successfully [took 0.41s] login as 'cirros' user. default password: 'gocubsgo'. use 'sudo' for root. cirros login: [ 212.801369] random: nonblocking pool is initialized (overcloud) [stack@undercloud-0 etc]$ ssh -i /tmp/key2 root.0.244 Warning: Permanently added '10.0.0.244' (ECDSA) to the list of known hosts. root.0.244's password: (overcloud) [stack@undercloud-0 etc]$ ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master ovs-system state UP group default qlen 1000 link/ether 52:54:00:21:4d:b6 brd ff:ff:ff:ff:ff:ff inet6 fe80::5054:ff:fe21:4db6/64 scope link valid_lft forever preferred_lft forever 3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000 link/ether 52:54:00:ca:f8:bf brd ff:ff:ff:ff:ff:ff inet 172.16.0.30/24 brd 172.16.0.255 scope global dynamic noprefixroute eth1 valid_lft 3182sec preferred_lft 3182sec inet6 fe80::5054:ff:feca:f8bf/64 scope link noprefixroute valid_lft forever preferred_lft forever 4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000 link/ether 52:54:00:3d:27:d6 brd ff:ff:ff:ff:ff:ff inet 10.0.0.83/24 brd 10.0.0.255 scope global dynamic noprefixroute eth2 valid_lft 3594sec preferred_lft 3594sec inet6 fe80::5054:ff:fe3d:27d6/64 scope link noprefixroute valid_lft forever preferred_lft forever 5: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000 link/ether e6:3c:aa:38:d0:81 brd ff:ff:ff:ff:ff:ff 6: br-ctlplane: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000 link/ether 52:54:00:21:4d:b6 brd ff:ff:ff:ff:ff:ff inet 192.168.24.1/24 brd 192.168.24.255 scope global br-ctlplane valid_lft forever preferred_lft forever inet 192.168.24.3/32 scope global br-ctlplane valid_lft forever preferred_lft forever inet 192.168.24.2/32 scope global br-ctlplane valid_lft forever preferred_lft forever inet6 fe80::5054:ff:fe21:4db6/64 scope link valid_lft forever preferred_lft forever 7: br-int: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000 link/ether f2:00:6c:1c:2d:43 brd ff:ff:ff:ff:ff:ff (overcloud) [stack@undercloud-0 etc]$ ssh -i /tmp/key2 cirros.0.244 Warning: Permanently added '10.0.0.244' (ECDSA) to the list of known hosts. cirros.0.244's password: $ ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1442 qdisc pfifo_fast qlen 1000 link/ether fa:16:3e:c3:ea:f8 brd ff:ff:ff:ff:ff:ff inet 10.100.0.8/28 brd 10.100.0.15 scope global eth0 valid_lft forever preferred_lft forever inet6 fe80::f816:3eff:fec3:eaf8/64 scope link valid_lft forever preferred_lft forever $ route Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface default 10.100.0.1 0.0.0.0 UG 0 0 0 eth0 10.100.0.0 * 255.255.255.240 U 0 0 0 eth0 169.254.169.254 10.100.0.2 255.255.255.255 UGH 0 0 0 eth0 $ ip r default via 10.100.0.1 dev eth0 10.100.0.0/28 dev eth0 src 10.100.0.8 169.254.169.254 via 10.100.0.2 dev eth0 $ curl http://169.254.169.254/2009-04-04/instance-id <html> <head> <title>500 Internal Server Error</title> </head> <body> <h1>500 Internal Server Error</h1> An unknown error has occurred. Please try your request again.<br /><br </body> </html>$ $ curl http://169.254.169.254/ </body> </html>$ $ curl http://169.254.169.254/ <html> <head> <title>500 Internal Server Error</title> </head> <body> <h1>500 Internal Server Error</h1> An unknown error has occurred. Please try your request again.<br /><br /> The status of the agent is up : (.vevn) (overcloud) [stack@undercloud-0 tempest]$ openstack network agent list +--------------------------------------+------------------------------+---------------------------+-------------------+-------+-------+-------------------------------+ | ID | Agent Type | Host | Availability Zone | Alive | State | Binary | +--------------------------------------+------------------------------+---------------------------+-------------------+-------+-------+-------------------------------+ | 62925450-1feb-4341-89f7-3a47f27f0b71 | OVN Controller agent | compute-1.redhat.local | n/a | :-) | UP | ovn-controller | | a4881130-8f58-481e-a7f7-c1aab5d9ce94 | OVN Metadata agent | compute-1.redhat.local | n/a | :-) | UP | networking-ovn-metadata-agent | | 600dba3f-6e24-45c0-bd0a-2ab3c16e2805 | OVN Controller agent | compute-0.redhat.local | n/a | :-) | UP | ovn-controller | | 84315d5f-719d-4859-b09c-f5329b48e9c4 | OVN Metadata agent | compute-0.redhat.local | n/a | :-) | UP | networking-ovn-metadata-agent | | 731603be-92d1-4744-8c0e-2fe59dccfcca | OVN Controller Gateway agent | controller-0.redhat.local | n/a | :-) | UP | ovn-controller | | a250901f-506e-4151-8d8e-538713a97906 | OVN Controller Gateway agent | controller-1.redhat.local | n/a | :-) | UP | ovn-controller | | 183b8a53-d91f-4e4d-a391-cd44139a13d5 | OVN Controller Gateway agent | controller-2.redhat.local | n/a | :-) | UP | ovn-controller | +--------------------------------------+------------------------------+-------- Version-Release number of selected component (if applicable): OSP15 core_puddle: RHOS_TRUNK-15.0-RHEL-8-20190708.n.2 How reproducible: 100% Steps to Reproduce: 1.run ovn tls job: https://rhos-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/DFG/view/network/view/networking-ovn/job/DFG-network-networking-ovn-15_director-rhel-virthost-3cont_2comp_1ipa-ipv4-geneve-tls/ 2. when a scenarion test is running connect to the VM with ssh and check the metadata service curl http://169.254.169.254/ 3. Actual results: Expected results: Additional info:
Regression from OSP14
Merged upstream.
Fixed verified on CI run. all the scenario tests passed. https://rhos-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/DFG/view/network/view/networking-ovn/job/DFG-network-networking-ovn-15_director-rhel-virthost-3cont_2comp_1ipa-ipv4-geneve-tls/63/ core_puddle: RHOS_TRUNK-15.0-RHEL-8-20190827.n.0
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2019:2811