Bug 1364281

Summary:

Network not brought up correctly since 20160714.n.0 nightlies

Product:

[Fedora] Fedora

Reporter:

Adam Williamson (Red Hat non-Fedora) <awilliam>

Component:

anaconda

Assignee:

Anaconda Maintenance Team <anaconda-maint-list>

Status:

CLOSED DUPLICATE

QA Contact:

Fedora Extras Quality Assurance <extras-qa>

Severity:

urgent

Docs Contact:

Priority:

unspecified

Version:

rawhide

CC:

anaconda-maint-list, g.kaviyarasu, jonathan, vanmeeuwen+fedora

Target Milestone:

---

Target Release:

---

Hardware:

x86_64

OS:

Linux

Whiteboard:

Fixed In Version:

Doc Type:

If docs needed, set a value

Doc Text:

Story Points:

---

Clone Of:

Environment:

Last Closed:

2016-08-05 15:09:47 UTC

Type:

Bug

Regression:

---

Mount Type:

---

Documentation:

---

CRM:

Verified Versions:

Category:

---

oVirt Team:

---

RHEL 7.3 requirements from Atomic Host:

Cloudforms Team:

---

Target Upstream Version:

Embargoed:

Attachments:

Description	Flags
journal from boot of Fedora-Server-dvd-x86_64-Rawhide-20160708.n.0.iso on KVM with libvirt NAT networking	none
journal from boot of Fedora-Server-dvd-x86_64-Rawhide-20160715.n.0.iso on KVM with libvirt NAT networking	none
`nmcli conn show ens3` output from boot of Fedora-Server-dvd-x86_64-Rawhide-20160708.n.0.iso on KVM with libvirt NAT networking	none
`nmcli conn show ens3` output from boot of Fedora-Server-dvd-x86_64-Rawhide-20160715.n.0.iso on KVM with libvirt NAT networking	none

Description Adam Williamson (Red Hat non-Fedora) 2016-08-04 23:49:43 UTC

In several configurations it seems that the network connection is not correctly brought up by anaconda, since Fedora-Rawhide-20160714.n.0.

It's easy to observe this bug, if you're affected: just boot any installer image with a working ethernet connection and either try to ping 8.8.8.8 (it won't work) or go to the NETWORK & HOST NAME spoke (it'll show 'Connected', but no 'Speed', 'Default Route' or 'DNS'). If you use a netinst image, it'll show 'Error setting up base repository' at the hub, also.

I'm observing this effect in three different configurations here:

1. KVM with standard libvirt NAT networking
2. KVM with networking via a bridge
3. Bare metal system (on the same network)

If I boot a 20160708.n.0 image, the network works fine. If I boot a 20160715.n.0 image (I don't have a 20160714.n.0 image lying around locally and they've been garbage-collected from https://kojipkgs.fedoraproject.org/compose/rawhide ) the bug appears.

I believe it's also what's causing the install_kickstart_nfs openQA tests to fail; they started failing exactly with the 20160714.n.0 compose (which is why I identify that as the first broken compose). You can see this at https://openqa.stg.fedoraproject.org/tests/30327?limit_previous=20#previous .

Other openQA tests are not affected, though; the kickstart_nfs test uses a somewhat different network configuration from the default openQA config, suggesting the default openQA config (which uses qemu's default networking, IIRC) is not affected but the network config used by the kickstart_nfs test (which is more or less a bridged configuration set up via openvswitch) is affected.

There were no images for the composes between 20160708.n.0 and 20160714.n.0 so we can't pin down precisely where the breakage occurred between those two composes, but there are some network-related changes in anaconda-25.20-1.fc25 which landed on 20160709. I've tried reverting each of those and haven't yet found that any reversion avoids the bug. It's possible the bug isn't in anaconda, but if not anaconda, I don't know where; NetworkManager was not changed in the affected time frame.

Comment 1 Adam Williamson (Red Hat non-Fedora) 2016-08-05 00:06:40 UTC

The issue does seem to be in the installer environment somehow, though - live images do not seem to be affected.

Comment 2 Adam Williamson (Red Hat non-Fedora) 2016-08-05 00:24:54 UTC

Created attachment 1187681 [details]
journal from boot of Fedora-Server-dvd-x86_64-Rawhide-20160708.n.0.iso on KVM with libvirt NAT networking

Attaching journals from 20160708.n.0 (working case) and 20160715.n.0 (broken case) on the same VM. The NetworkManager messages seem to diverge, at least, at the end. In the 20160708.n.0 case, we see this:

Aug 05 00:07:48 localhost NetworkManager[1573]: <info>  [1470355668.1997] manager: NetworkManager state is now CONNECTED_LOCAL
Aug 05 00:07:48 localhost NetworkManager[1573]: <debug> [1470355668.1999] active-connection[0x55a3fe6f34f0]: check-master-ready: not signalling (state activated, no master)
Aug 05 00:07:48 localhost NetworkManager[1573]: <debug> [1470355668.2001] device[0x55a3fe71e4e0] (ens3): remove_pending_action (2): 'activation::0x55a3fe6f34f0'
Aug 05 00:07:48 localhost NetworkManager[1573]: <debug> [1470355668.2003] dns-mgr: (device_state_changed): queueing DNS updates (1)
Aug 05 00:07:48 localhost NetworkManager[1573]: <debug> [1470355668.2006] dns-mgr: (update_routing_and_dns): queueing DNS updates (2)
Aug 05 00:07:48 localhost NetworkManager[1573]: <info>  [1470355668.2007] manager: NetworkManager state is now CONNECTED_GLOBAL
Aug 05 00:07:48 localhost NetworkManager[1573]: <info>  [1470355668.2009] policy: set 'Wired connection 1' (ens3) as default for IPv4 routing and DNS
Aug 05 00:07:48 localhost NetworkManager[1573]: <debug> [1470355668.2010] manager: PrimaryConnection now Wired connection 1
Aug 05 00:07:48 localhost NetworkManager[1573]: <debug> [1470355668.2014] dns-mgr: (update_routing_and_dns): DNS configuration changed
Aug 05 00:07:48 localhost NetworkManager[1573]: <debug> [1470355668.2026] dns-mgr: (update_routing_and_dns): no DNS changes to commit (1)
Aug 05 00:07:48 localhost NetworkManager[1573]: <debug> [1470355668.2027] dns-mgr: (device_state_changed): DNS configuration changed
Aug 05 00:07:48 localhost NetworkManager[1573]: <debug> [1470355668.2027] dns-mgr: (device_state_changed): committing DNS changes (0)
Aug 05 00:07:48 localhost NetworkManager[1573]: <debug> [1470355668.2027] dns-mgr: update-dns: updating resolv.conf
Aug 05 00:07:48 localhost NetworkManager[1573]: <debug> [1470355668.2033] policy: restarting reverse-lookup thread for address 192.168.122.176
Aug 05 00:07:48 localhost NetworkManager[1573]: <info>  [1470355668.2034] device (ens3): Activation: successful, device activated.

but in the 20160715 case, only this:

Aug 05 00:13:23 localhost NetworkManager[1597]: <info>  [1470356003.3241] manager: NetworkManager state is now CONNECTED_LOCAL
Aug 05 00:13:23 localhost NetworkManager[1597]: <debug> [1470356003.3246] active-connection[0x55fbc96974f0]: check-master-ready: not signalling (state activated, no master)
Aug 05 00:13:23 localhost NetworkManager[1597]: <debug> [1470356003.3247] device[0x55fbc96c25f0] (ens3): remove_pending_action (2): 'activation::0x55fbc96974f0'
Aug 05 00:13:23 localhost NetworkManager[1597]: <debug> [1470356003.3248] dns-mgr: (device_state_changed): queueing DNS updates (1)
Aug 05 00:13:23 localhost NetworkManager[1597]: <debug> [1470356003.3249] dns-mgr: (update_routing_and_dns): queueing DNS updates (2)
Aug 05 00:13:23 localhost NetworkManager[1597]: <debug> [1470356003.3249] dns-mgr: (update_routing_and_dns): DNS configuration did not change
Aug 05 00:13:23 localhost NetworkManager[1597]: <debug> [1470356003.3249] dns-mgr: (update_routing_and_dns): no DNS changes to commit (1)
Aug 05 00:13:23 localhost NetworkManager[1597]: <debug> [1470356003.3249] dns-mgr: (device_state_changed): DNS configuration did not change
Aug 05 00:13:23 localhost NetworkManager[1597]: <debug> [1470356003.3249] dns-mgr: (device_state_changed): no DNS changes to commit (0)
Aug 05 00:13:23 localhost NetworkManager[1597]: <info>  [1470356003.3250] device (ens3): Activation: successful, device activated.

Comment 3 Adam Williamson (Red Hat non-Fedora) 2016-08-05 00:25:30 UTC

Created attachment 1187682 [details]
journal from boot of Fedora-Server-dvd-x86_64-Rawhide-20160715.n.0.iso on KVM with libvirt NAT networking

Comment 4 Adam Williamson (Red Hat non-Fedora) 2016-08-05 00:42:04 UTC

Created attachment 1187696 [details]
`nmcli conn show ens3` output from boot of Fedora-Server-dvd-x86_64-Rawhide-20160708.n.0.iso on KVM with libvirt NAT networking

Comment 5 Adam Williamson (Red Hat non-Fedora) 2016-08-05 00:44:04 UTC

Created attachment 1187697 [details]
`nmcli conn show ens3` output from boot of Fedora-Server-dvd-x86_64-Rawhide-20160715.n.0.iso on KVM with libvirt NAT networking

Here's the output from `nmcli conn show ens3` for both images. Note the difference in GENERAL.DEFAULT as well as all the DHCP4.OPTION values.

Also for the record just running 'dhclient ens3' brings up the network on the affected image. I've not yet figured out what nmcli commands or NetworkManager service changes will cause the connection to come up properly, though.

Comment 6 Adam Williamson (Red Hat non-Fedora) 2016-08-05 15:09:47 UTC


*** This bug has been marked as a duplicate of bug 1358416 ***