Bug 1158020
| Summary: | interface VLAN is not configured on host to be used by external network | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Ofer Blaut <oblaut> | ||||||
| Component: | rhel-osp-installer | Assignee: | Brad P. Crochet <brad> | ||||||
| Status: | CLOSED ERRATA | QA Contact: | Alexander Chuzhoy <sasha> | ||||||
| Severity: | high | Docs Contact: | |||||||
| Priority: | unspecified | ||||||||
| Version: | 5.0 (RHEL 6) | CC: | aberezin, mburns, oblaut, rhos-maint, sasha, sclewis, sseago, yeylon | ||||||
| Target Milestone: | ga | Keywords: | TestOnly | ||||||
| Target Release: | Installer | ||||||||
| Hardware: | Unspecified | ||||||||
| OS: | Unspecified | ||||||||
| Whiteboard: | |||||||||
| Fixed In Version: | Doc Type: | Bug Fix | |||||||
| Doc Text: | Story Points: | --- | |||||||
| Clone Of: | Environment: | ||||||||
| Last Closed: | 2015-02-09 15:17:39 UTC | Type: | Bug | ||||||
| Regression: | --- | Mount Type: | --- | ||||||
| Documentation: | --- | CRM: | |||||||
| Verified Versions: | Category: | --- | |||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||
| Embargoed: | |||||||||
| Attachments: |
|
||||||||
Can you attach the ifcfg file? /etc/sysconfig/network-scripts/ifcfg-enp4s0f1.195 [root@mac441ea173385f ~]# cat /etc/sysconfig/network-scripts/ifcfg-enp4s0f
ifcfg-enp4s0f0 ifcfg-enp4s0f1 ifcfg-enp4s0f1.189 ifcfg-enp4s0f1.195
[root@mac441ea173385f ~]# cat /etc/sysconfig/network-scripts/ifcfg-enp4s0f1.195
DEVICE=enp4s0f1.195
DEVICETYPE=ovs
TYPE=OVSPort
OVS_BRIDGE=br-ex
ONBOOT=yes
BOOTPROTO=none
[root@mac441ea173385f ~]# ifconfig
enp4s0f0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet6 fe80::29c:2ff:feb0:af40 prefixlen 64 scopeid 0x20<link>
ether 00:9c:02:b0:af:40 txqueuelen 1000 (Ethernet)
RX packets 982843 bytes 62750496 (59.8 MiB)
RX errors 0 dropped 331 overruns 0 frame 0
TX packets 1536 bytes 130852 (127.7 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
enp4s0f1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet6 fe80::29c:2ff:feb0:af44 prefixlen 64 scopeid 0x20<link>
ether 00:9c:02:b0:af:44 txqueuelen 1000 (Ethernet)
RX packets 8689 bytes 571828 (558.4 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 729 bytes 36382 (35.5 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
enp4s0f1.189: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.35.174.22 netmask 255.255.255.0 broadcast 10.35.174.255
inet6 fe80::29c:2ff:feb0:af44 prefixlen 64 scopeid 0x20<link>
ether 00:9c:02:b0:af:44 txqueuelen 0 (Ethernet)
RX packets 4594 bytes 214916 (209.8 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 714 bytes 30996 (30.2 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
enp5s0f0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet6 fe80::461e:a1ff:fe73:385e prefixlen 64 scopeid 0x20<link>
ether 44:1e:a1:73:38:5e txqueuelen 1000 (Ethernet)
RX packets 1939189 bytes 158308456 (150.9 MiB)
RX errors 0 dropped 6138 overruns 0 frame 0
TX packets 1543 bytes 131798 (128.7 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
device memory 0xfbe60000-fbe80000
enp5s0f1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.0.4 netmask 255.255.255.0 broadcast 192.168.0.255
inet6 fe80::461e:a1ff:fe73:385f prefixlen 64 scopeid 0x20<link>
ether 44:1e:a1:73:38:5f txqueuelen 1000 (Ethernet)
RX packets 27366780 bytes 3091180667 (2.8 GiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 26999008 bytes 3826130927 (3.5 GiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
device memory 0xfbee0000-fbf00000
int-br-ex: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet6 fe80::ec33:daff:fe50:cc74 prefixlen 64 scopeid 0x20<link>
ether ee:33:da:50:cc:74 txqueuelen 1000 (Ethernet)
RX packets 8 bytes 648 (648.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 8 bytes 648 (648.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 0 (Local Loopback)
RX packets 13305857 bytes 1440101796 (1.3 GiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 13305857 bytes 1440101796 (1.3 GiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
phy-br-ex: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet6 fe80::6c8d:8bff:fe8b:4ceb prefixlen 64 scopeid 0x20<link>
ether 6e:8d:8b:8b:4c:eb txqueuelen 1000 (Ethernet)
RX packets 8 bytes 648 (648.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 8 bytes 648 (648.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
[root@mac441ea173385f ~]#
Created attachment 951767 [details]
Adding debug logs
Adding debug logs
more logs : [root@mac441ea173366b ~]# grep enp4s0f1.195 /var/log/messages Oct 29 19:35:37 mac441ea173366b network: Bringing up interface enp4s0f1.195: Oct 29 19:35:37 mac441ea173366b dhclient[1630]: DHCPDISCOVER on enp4s0f1.195 to 255.255.255.255 port 67 interval 7 (xid=0x706441c5) Oct 29 19:35:44 mac441ea173366b dhclient[1630]: DHCPDISCOVER on enp4s0f1.195 to 255.255.255.255 port 67 interval 12 (xid=0x706441c5) Oct 29 19:35:56 mac441ea173366b dhclient[1630]: DHCPDISCOVER on enp4s0f1.195 to 255.255.255.255 port 67 interval 17 (xid=0x706441c5) Oct 29 19:36:13 mac441ea173366b dhclient[1630]: DHCPDISCOVER on enp4s0f1.195 to 255.255.255.255 port 67 interval 13 (xid=0x706441c5) Oct 29 19:36:26 mac441ea173366b dhclient[1630]: DHCPDISCOVER on enp4s0f1.195 to 255.255.255.255 port 67 interval 12 (xid=0x706441c5) Oct 29 19:36:38 mac441ea173366b network: Determining IP information for enp4s0f1.195... failed. Oct 29 19:45:11 mac441ea173366b start-puppet-agent: Debug: Executing '/usr/bin/ovs-vsctl add-port br-ex enp4s0f1.195' Oct 29 19:45:11 mac441ea173366b ovs-vsctl: ovs|00001|vsctl|INFO|Called as /usr/bin/ovs-vsctl add-port br-ex enp4s0f1.195 Oct 29 19:45:11 mac441ea173366b kernel: device enp4s0f1.195 entered promiscuous mode Oct 29 19:45:11 mac441ea173366b start-puppet-agent: Debug: Executing '/usr/sbin/ip addr show enp4s0f1.195' Oct 29 19:45:12 mac441ea173366b kernel: device enp4s0f1.195 left promiscuous mode Oct 29 19:45:12 mac441ea173366b start-puppet-agent: Debug: Executing '/usr/sbin/ifdown enp4s0f1.195' Oct 29 19:45:12 mac441ea173366b ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl -t 10 -- --if-exists del-port br-ex enp4s0f1.195 Oct 29 19:45:12 mac441ea173366b start-puppet-agent: Error: Execution of '/usr/sbin/ifdown enp4s0f1.195' returned 1: Oct 29 19:45:12 mac441ea173366b puppet-agent[4373]: Execution of '/usr/sbin/ifdown enp4s0f1.195' returned 1: Oct 29 19:45:12 mac441ea173366b puppet-agent[4373]: (/Stage[main]/Neutron::Agents::Ovs/Neutron::Plugins::Ovs::Port[br-ex:enp4s0f1.195]/Vs_port[enp4s0f1.195]/ensure) change from absent to present failed: Execution of '/usr/sbin/ifdown enp4s0f1.195' returned 1: Oct 29 19:45:12 mac441ea173366b start-puppet-agent: Error: /Stage[main]/Neutron::Agents::Ovs/Neutron::Plugins::Ovs::Port[br-ex:enp4s0f1.195]/Vs_port[enp4s0f1.195]/ensure: change from absent to present failed: Execution of '/usr/sbin/ifdown enp4s0f1.195' returned 1: Oct 29 20:00:39 mac441ea173366b start-puppet-agent: Notice: /Stage[main]/Neutron::Agents::Ovs/Service[neutron-plugin-ovs-service]: Dependency Vs_port[enp4s0f1.195] has failures: true Oct 29 20:00:39 mac441ea173366b puppet-agent[4373]: (/Stage[main]/Neutron::Agents::Ovs/Service[neutron-plugin-ovs-service]) Dependency Vs_port[enp4s0f1.195] has failures: true Oct 29 20:08:38 mac441ea173366b start-puppet-agent: Debug: /Stage[main]/Vswitch::Ovs/Service[openvswitch]/before: requires Vs_port[enp4s0f1.195] Oct 29 20:08:40 mac441ea173366b kernel: device enp4s0f1.195 entered promiscuous mode Oct 29 20:08:41 mac441ea173366b kernel: device enp4s0f1.195 left promiscuous mode Oct 29 20:08:41 mac441ea173366b kernel: device enp4s0f1.195 entered promiscuous mode Oct 29 21:08:44 mac441ea173366b start-puppet-agent: Debug: /Stage[main]/Vswitch::Ovs/Service[openvswitch]/before: requires Vs_port[enp4s0f1.195] Oct 29 21:28:58 mac441ea173366b ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl -t 10 -- --if-exists del-port br-ex enp4s0f1.195 Oct 29 21:28:58 mac441ea173366b kernel: device enp4s0f1.195 left promiscuous mode Oct 29 21:29:37 mac441ea173366b ovs-vsctl: ovs|00001|vsctl|INFO|Called as /usr/bin/ovs-vsctl add-port br-ex enp4s0f1.195 Oct 29 21:29:37 mac441ea173366b kernel: device enp4s0f1.195 entered promiscuous mode Oct 29 21:29:37 mac441ea173366b kernel: device enp4s0f1.195 left promiscuous mode Oct 29 21:29:37 mac441ea173366b ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl -t 10 -- --if-exists del-port br-ex enp4s0f1.195 Oct 29 21:29:37 mac441ea173366b ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl -t 10 -- --may-exist add-port br-ex enp4s0f1.195 Oct 29 21:29:37 mac441ea173366b kernel: device enp4s0f1.195 entered promiscuous mode Oct 29 21:29:37 mac441ea173366b puppet-agent[6862]: (/Stage[main]/Neutron::Agents::Ovs/Neutron::Plugins::Ovs::Port[br-ex:enp4s0f1.195]/Vs_port[enp4s0f1.195]/ensure) created Oct 29 21:34:00 mac441ea173366b ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl -t 10 -- --if-exists del-port br-ex enp4s0f1.195 Oct 29 21:34:00 mac441ea173366b kernel: device enp4s0f1.195 left promiscuous mode Oct 29 21:34:05 mac441ea173366b ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl -t 10 -- --if-exists del-port br-ex enp4s0f1.195 Oct 29 21:34:09 mac441ea173366b ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl -t 10 -- --if-exists del-port br-ex enp4s0f1.195 Oct 29 21:38:43 mac441ea173366b start-puppet-agent: Debug: /Stage[main]/Vswitch::Ovs/Service[openvswitch]/before: requires Vs_port[enp4s0f1.195] Oct 29 21:38:46 mac441ea173366b kernel: device enp4s0f1.195 entered promiscuous mode Oct 29 21:38:46 mac441ea173366b kernel: device enp4s0f1.195 left promiscuous mode Oct 29 21:38:46 mac441ea173366b kernel: device enp4s0f1.195 entered promiscuous mode Oct 29 21:39:12 mac441ea173366b ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl -t 10 Ofer, Did the deployment fail? Also, do you know if the interface gets configured on subsequent run of puppet agent? So the root cause here seems to be that puppet begins before the tenant interface is fully up and running with an IP address listed in system facts. Initially we thought this could be resolved by using a static IP address, but since foreman isn't where we look this up, that doesn't really change anything here. Essentially the usual progression is that this fails on the first puppet run and succeeds on the next one, so in that case no further user action is needed as a workaround. In some cases the second run fails too, but the third succeeds, but after two failures, the deployment gets paused, so the workaround is to resume the deployment. Eventually, once we have puppetssh integrated, this problem will go away, since we will be able to decouple provisioning from puppet runs, so we can delay running puppet until the host is fully operational with all interfaces up with IP addresses. Moving this to ON_QA now that we have puppetssh available Verified:
Environment:
ruby193-rubygem-staypuft-0.5.12-1.el7ost.noarch
openstack-puppet-modules-2014.2.8-1.el7ost.noarch
ruby193-rubygem-foreman_openstack_simplify-0.0.6-8.el7ost.noarch
openstack-foreman-installer-3.0.10-2.el7ost.noarch
rhel-osp-installer-0.5.5-1.el7ost.noarch
rhel-osp-installer-client-0.5.5-1.el7ost.noarch
eno2 is where the external role was dragged/dropped.
[root@mac000af77993d2 ~]# ifconfig eno2.185
eno2.185: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.19.105.6 netmask 255.255.255.0 broadcast 10.19.105.255
inet6 2620:52:0:1369:ca1f:66ff:fec7:dce0 prefixlen 64 scopeid 0x0<global>
inet6 fe80::ca1f:66ff:fec7:dce0 prefixlen 64 scopeid 0x20<link>
ether c8:1f:66:c7:dc:e0 txqueuelen 0 (Ethernet)
RX packets 73 bytes 4482 (4.3 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 41 bytes 3434 (3.3 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
[root@mac000af77993d2 ~]# ping -c1 10.19.105.254
PING 10.19.105.254 (10.19.105.254) 56(84) bytes of data.
64 bytes from 10.19.105.254: icmp_seq=1 ttl=64 time=2.36 ms
--- 10.19.105.254 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 2.363/2.363/2.363/0.000 ms
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHBA-2015-0156.html |
Created attachment 951348 [details] ifconfig Description of problem: interface VLAN is not configured on host to be used by external network attached output (enp4s0f1.195 ) should have been configured Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1.install staypuft with external network to be used on vlan 2.check interface vlan is configured on host 3. Actual results: interface vlan is not configured Expected results: interface vlan should be configured Additional info: