First off, maybe this is a problem that's going away when some systemd extended ecosystem is going to occupy the area in question, in which case I suppose this problem will be resolved on its own eventually. So, in the morning, I was to start the work day, however, the web browser and consequently common network problem diagnostics indicated domain names cannot be resolved. From here, quick check of resolve.conf and indeed -- it contained stale entries. I recalled I might have low space on the root partition and indeed -- that was the case. In hindsight, I was dissapointed that no such pin-pointing of the cause was mentioned in the logs (considering logging was still going on, since journal goes normally volatile-persistent hybrid storage model, IIRC). That really makes the experience of the (power) user really suboptimal in this case. Moreover, at least one colleage confirmed she observed a similar (mysterious) issue with the same root cause in the past. Of course, primary request here is: - make NetworkManager (dhcpd?) holler loudly when it cannot change resolve.conf on its own, so as to get as immediate attention by the investigators of the problem as possible but we could go a bit deeper, indeed: - /etc/resolve.conf can be just a symlink to some /run-rooted location (which would obviate this problem in the first place!) Thanks for considering the current unpleasantly nonstraightforward ability to diagnose the problem right away. FTR. I am using nmtui to control the NetworkManager back-end, if there would be any chance to propagate said obstacle there (nice to have) in addition to logs themselves (assumed core of the resolution). Thanks for considering. $ rpm -q NetworkManager dhcp-client > NetworkManager-1.22.0-0.2.fc32.x86_64 > dhcp-client-4.4.1-20.fc32.x86_64
> In hindsight, I was dissapointed that no such pin-pointing of the cause was mentioned in the logs When having a problem with logs, it seems pretty important to actually attach the full logs. > but we could go a bit deeper, indeed: > > - /etc/resolve.conf can be just a symlink to some > /run-rooted location (which would obviate this problem > in the first place!) Do `ln -snf /var/run/NetworkManager/resolv.conf /etc/resolv.conf` and NM won't touch that file anymore. Or link it to anywhere you like. There are various ways how to configure handling of /etc/resolv.conf. There is not one answer. It depends on your preference. See `man NetworkManager.conf` for "main.dns" and "main.rc-manager" settings. Future Fedora versions may enable systemd-resolved by default (https://pagure.io/fedora-workstation/issue/123#comment-625579), where NetworkManager wouldn't directly write /etc/resolv.conf either. But you can enable that today for your machine already.
> When having a problem with logs, it seems pretty important to actually > attach the full logs. Something like this? > journalctl -b -u NetworkManager -u systemd-networkd -S ... -U ... ? No need to share a full bin of irrelevant records, I think.
> Something like this? > > > journalctl -b -u NetworkManager -u systemd-networkd -S ... -U ... Yes.
Ok, slightly annotated log message of relevance below. It comes from this command: > journalctl -b --no-hostname -u NetworkManager -u systemd-networkd \ > -u systemd-suspend -u sysstat-collect \ > -S "2020-03-02 11:00:00 CET" -U "2020-03-02 11:35:00 CET" \ > | cut "-d " -f3- \ > | sed -e 's|:00:[:[:xdigit:]]*|:00SCRAMBLED|' \ > -e 's|10\.[.[:digit:]]*|10.SCRAMBLED|g' -e 's|Guest ([^)]*|Guest (SCRAMBLED|' (protect native device MAC addresses and particulars of internal addressing, sysstat-collect unit file originated messages actually pin-point both the problem at hand and where it got removed) [situation: laptop suspended, brought into a new network environment (WiFI to be used instead of cable connection) & resumed] > 11:05:06 systemd-sleep[3305161]: System resumed. > 11:05:06 systemd[1]: Starting system activity accounting tool... > 11:05:06 sa1[3305213]: Cannot write data to system activity file: No space left on device ^ oh well, it pin-points the situation right away, but this bug is not about not being able to deduce that fact from unrelated components, but rather from the components directly responsible for the connection maintenance > 11:05:06 systemd[1]: systemd-suspend.service: Succeeded. > 11:05:06 systemd[1]: Started Suspend. > 11:05:06 systemd[1]: sysstat-collect.service: Main process exited, code=exited, status=2/INVALIDARGUMENT > 11:05:06 systemd[1]: sysstat-collect.service: Failed with result 'exit-code'. > 11:05:06 systemd[1]: Failed to start system activity accounting tool. > 11:05:06 NetworkManager[1198]: <info> [1583143506.3256] manager: sleep: wake requested (sleeping: yes enabled: yes) > 11:05:06 NetworkManager[1198]: <info> [1583143506.3260] device (enp0s31f6): state change: activated -> unmanaged (reason 'sleeping', sys-iface-state: 'managed') > 11:05:06 NetworkManager[1198]: <info> [1583143506.3263] dhcp4 (enp0s31f6): canceled DHCP transaction > 11:05:06 NetworkManager[1198]: <info> [1583143506.3263] dhcp4 (enp0s31f6): state changed bound -> done > 11:05:06 systemd-networkd[1199]: lo: Reset carrier > 11:05:06 systemd-networkd[1199]: enp0s31f6: Reset carrier > 11:05:06 NetworkManager[1198]: <info> [1583143506.3506] manager: NetworkManager state is now CONNECTED_GLOBAL > 11:05:06 systemd-networkd[1199]: enp0s31f6: Link DOWN > 11:05:06 systemd-networkd[1199]: enp0s31f6: Lost carrier > 11:05:06 NetworkManager[1198]: <info> [1583143506.4442] manager: NetworkManager state is now CONNECTED_LOCAL > 11:05:06 NetworkManager[1198]: <info> [1583143506.4449] device (wlp3s0): state change: disconnected -> unmanaged (reason 'sleeping', sys-iface-state: 'managed') > 11:05:06 systemd-networkd[1199]: wlp3s0: Link DOWN > 11:05:06 NetworkManager[1198]: <info> [1583143506.4572] device (wlp3s0): set-hw-addr: reset MAC address to 44:85:00SCRAMBLED (unmanage) > 11:05:06 systemd-networkd[1199]: wlp3s0: Link UP > 11:05:06 NetworkManager[1198]: <info> [1583143506.4620] device (enp0s31f6): state change: unmanaged -> unavailable (reason 'managed', sys-iface-state: 'managed') > 11:05:06 systemd-networkd[1199]: enp0s31f6: Link UP > 11:05:06 NetworkManager[1198]: <info> [1583143506.6460] device (wlp3s0): state change: unmanaged -> unavailable (reason 'managed', sys-iface-state: 'managed') > 11:05:06 systemd-networkd[1199]: wlp3s0: Link DOWN > 11:05:06 NetworkManager[1198]: <info> [1583143506.7032] device (wlp3s0): set-hw-addr: set MAC address to F6:A8:03:7C:72:0F (scanning) > 11:05:06 systemd-networkd[1199]: wlp3s0: Link UP > 11:05:06 NetworkManager[1198]: <info> [1583143506.9820] device (p2p-dev-wlp3s0): state change: unmanaged -> unavailable (reason 'managed', sys-iface-state: 'managed') > 11:05:07 NetworkManager[1198]: <info> [1583143507.0167] sup-iface[0x56067f3d9cf0,wlp3s0]: supports 5 scan SSIDs > 11:05:07 NetworkManager[1198]: <info> [1583143507.0179] device (wlp3s0): supplicant interface state: starting -> ready > 11:05:07 NetworkManager[1198]: <info> [1583143507.0180] device (p2p-dev-wlp3s0): state change: unavailable -> disconnected (reason 'supplicant-available', sys-iface-state: 'managed') > 11:05:07 NetworkManager[1198]: <info> [1583143507.0185] device (wlp3s0): state change: unavailable -> disconnected (reason 'supplicant-available', sys-iface-state: 'managed') > 11:05:10 NetworkManager[1198]: <info> [1583143510.SCRAMBLED] policy: auto-activating connection 'Red Hat Guest' (b5130fe3-9d50-4c7b-8de7-e3d79ace9e3a) > 11:05:10 NetworkManager[1198]: <info> [1583143510.SCRAMBLED] device (wlp3s0): Activation: starting connection 'Red Hat Guest' (b5130fe3-9d50-4c7b-8de7-e3d79ace9e3a) > 11:05:10 NetworkManager[1198]: <info> [1583143510.SCRAMBLED] device (wlp3s0): state change: disconnected -> prepare (reason 'none', sys-iface-state: 'managed') > 11:05:10 NetworkManager[1198]: <info> [1583143510.SCRAMBLED] manager: NetworkManager state is now CONNECTING > 11:05:10 systemd-networkd[1199]: wlp3s0: Link DOWN > 11:05:10 NetworkManager[1198]: <info> [1583143510.SCRAMBLED] device (wlp3s0): set-hw-addr: reset MAC address to 44:85:00SCRAMBLED (preserve) > 11:05:10 systemd-networkd[1199]: wlp3s0: Link UP > 11:05:10 NetworkManager[1198]: <info> [1583143510.SCRAMBLED] device (wlp3s0): state change: prepare -> config (reason 'none', sys-iface-state: 'managed') > 11:05:10 NetworkManager[1198]: <info> [1583143510.SCRAMBLED] device (wlp3s0): Activation: (wifi) access point 'Red Hat Guest' has security, but secrets are required. > 11:05:10 NetworkManager[1198]: <info> [1583143510.SCRAMBLED] device (wlp3s0): state change: config -> need-auth (reason 'none', sys-iface-state: 'managed') > 11:05:10 NetworkManager[1198]: <info> [1583143510.SCRAMBLED] device (wlp3s0): supplicant interface state: ready -> disabled > 11:05:10 NetworkManager[1198]: <info> [1583143510.SCRAMBLED] device (p2p-dev-wlp3s0): supplicant management interface state: ready -> disabled > 11:05:10 NetworkManager[1198]: <info> [1583143510.SCRAMBLED] device (wlp3s0): state change: need-auth -> prepare (reason 'none', sys-iface-state: 'managed') > 11:05:10 NetworkManager[1198]: <info> [1583143510.SCRAMBLED] device (wlp3s0): state change: prepare -> config (reason 'none', sys-iface-state: 'managed') > 11:05:10 NetworkManager[1198]: <info> [1583143510.SCRAMBLED] device (wlp3s0): Activation: (wifi) connection 'Red Hat Guest' has security, and secrets exist. No new secrets needed. > 11:05:10 NetworkManager[1198]: <info> [1583143510.SCRAMBLED] Config: added 'ssid' value 'Red Hat Guest' > 11:05:10 NetworkManager[1198]: <info> [1583143510.SCRAMBLED] Config: added 'scan_ssid' value '1' > 11:05:10 NetworkManager[1198]: <info> [1583143510.SCRAMBLED] Config: added 'bgscan' value 'simple:30:-70:86400' > 11:05:10 NetworkManager[1198]: <info> [1583143510.SCRAMBLED] Config: added 'key_mgmt' value 'WPA-PSK WPA-PSK-SHA256 FT-PSK' > 11:05:10 NetworkManager[1198]: <info> [1583143510.SCRAMBLED] Config: added 'auth_alg' value 'OPEN' > 11:05:10 NetworkManager[1198]: <info> [1583143510.SCRAMBLED] Config: added 'psk' value '<hidden>' > 11:05:10 NetworkManager[1198]: <info> [1583143510.SCRAMBLED] device (wlp3s0): supplicant interface state: disabled -> inactive > 11:05:10 NetworkManager[1198]: <info> [1583143510.SCRAMBLED] device (p2p-dev-wlp3s0): supplicant management interface state: disabled -> inactive > 11:05:10 NetworkManager[1198]: <info> [1583143510.SCRAMBLED] device (wlp3s0): supplicant interface state: inactive -> scanning > 11:05:10 NetworkManager[1198]: <info> [1583143510.SCRAMBLED] device (p2p-dev-wlp3s0): supplicant management interface state: inactive -> scanning > 11:05:11 NetworkManager[1198]: <info> [1583143511.3859] device (wlp3s0): supplicant interface state: scanning -> authenticating > 11:05:11 NetworkManager[1198]: <info> [1583143511.3859] device (p2p-dev-wlp3s0): supplicant management interface state: scanning -> authenticating > 11:05:11 NetworkManager[1198]: <info> [1583143511.3947] device (wlp3s0): supplicant interface state: authenticating -> associating > 11:05:11 NetworkManager[1198]: <info> [1583143511.3948] device (p2p-dev-wlp3s0): supplicant management interface state: authenticating -> associating > 11:05:11 NetworkManager[1198]: <info> [1583143511.4110] device (wlp3s0): supplicant interface state: associating -> 4-way handshake > 11:05:11 NetworkManager[1198]: <info> [1583143511.4110] device (p2p-dev-wlp3s0): supplicant management interface state: associating -> 4-way handshake > 11:05:11 systemd-networkd[1199]: wlp3s0: Gained carrier > 11:05:11 systemd-networkd[1199]: wlp3s0: Connected WiFi access point: Red Hat Guest (SCRAMBLED) > 11:05:11 NetworkManager[1198]: <info> [1583143511.4617] device (wlp3s0): supplicant interface state: 4-way handshake -> completed > 11:05:11 NetworkManager[1198]: <info> [1583143511.4617] device (wlp3s0): Activation: (wifi) Stage 2 of 5 (Device Configure) successful. Connected to wireless network "Red Hat Guest" > 11:05:11 NetworkManager[1198]: <info> [1583143511.4619] device (p2p-dev-wlp3s0): supplicant management interface state: 4-way handshake -> completed > 11:05:11 NetworkManager[1198]: <info> [1583143511.4692] device (wlp3s0): state change: config -> ip-config (reason 'none', sys-iface-state: 'managed') > 11:05:11 NetworkManager[1198]: <info> [1583143511.4696] dhcp4 (wlp3s0): activation: beginning transaction (timeout in 45 seconds) > 11:05:12 NetworkManager[1198]: <info> [1583143512.5420] dhcp4 (wlp3s0): option dhcp_lease_time => '3600' > 11:05:12 NetworkManager[1198]: <info> [1583143512.5420] dhcp4 (wlp3s0): option domain_name => 'redhat.com' > 11:05:12 NetworkManager[1198]: <info> [1583143512.5421] dhcp4 (wlp3s0): option domain_name_servers => '10.SCRAMBLED 10.SCRAMBLED 10.SCRAMBLED' > 11:05:12 NetworkManager[1198]: <info> [1583143512.5421] dhcp4 (wlp3s0): option expiry => '1583147111' > 11:05:12 NetworkManager[1198]: <info> [1583143512.5421] dhcp4 (wlp3s0): option ip_address => '10.SCRAMBLED' > 11:05:12 NetworkManager[1198]: <info> [1583143512.5421] dhcp4 (wlp3s0): option ntp_servers => '10.SCRAMBLED' > 11:05:12 NetworkManager[1198]: <info> [1583143512.5421] dhcp4 (wlp3s0): option requested_broadcast_address => '1' > 11:05:12 NetworkManager[1198]: <info> [1583143512.5421] dhcp4 (wlp3s0): option requested_dhcp_server_identifier => '1' > 11:05:12 NetworkManager[1198]: <info> [1583143512.5421] dhcp4 (wlp3s0): option requested_domain_name => '1' > 11:05:12 NetworkManager[1198]: <info> [1583143512.5422] dhcp4 (wlp3s0): option requested_domain_name_servers => '1' > 11:05:12 NetworkManager[1198]: <info> [1583143512.5422] dhcp4 (wlp3s0): option requested_domain_search => '1' > 11:05:12 NetworkManager[1198]: <info> [1583143512.5422] dhcp4 (wlp3s0): option requested_host_name => '1' > 11:05:12 NetworkManager[1198]: <info> [1583143512.5422] dhcp4 (wlp3s0): option requested_interface_mtu => '1' > 11:05:12 NetworkManager[1198]: <info> [1583143512.5422] dhcp4 (wlp3s0): option requested_ms_classless_static_routes => '1' > 11:05:12 NetworkManager[1198]: <info> [1583143512.5422] dhcp4 (wlp3s0): option requested_nis_domain => '1' > 11:05:12 NetworkManager[1198]: <info> [1583143512.5422] dhcp4 (wlp3s0): option requested_nis_servers => '1' > 11:05:12 NetworkManager[1198]: <info> [1583143512.5422] dhcp4 (wlp3s0): option requested_ntp_servers => '1' > 11:05:12 NetworkManager[1198]: <info> [1583143512.5423] dhcp4 (wlp3s0): option requested_rfc3442_classless_static_routes => '1' > 11:05:12 NetworkManager[1198]: <info> [1583143512.5423] dhcp4 (wlp3s0): option requested_root_path => '1' > 11:05:12 NetworkManager[1198]: <info> [1583143512.5423] dhcp4 (wlp3s0): option requested_routers => '1' > 11:05:12 NetworkManager[1198]: <info> [1583143512.5423] dhcp4 (wlp3s0): option requested_static_routes => '1' > 11:05:12 NetworkManager[1198]: <info> [1583143512.5423] dhcp4 (wlp3s0): option requested_subnet_mask => '1' > 11:05:12 NetworkManager[1198]: <info> [1583143512.5423] dhcp4 (wlp3s0): option requested_time_offset => '1' > 11:05:12 NetworkManager[1198]: <info> [1583143512.5423] dhcp4 (wlp3s0): option requested_wpad => '1' > 11:05:12 NetworkManager[1198]: <info> [1583143512.5423] dhcp4 (wlp3s0): option routers => '10.SCRAMBLED' > 11:05:12 NetworkManager[1198]: <info> [1583143512.5423] dhcp4 (wlp3s0): option subnet_mask => '255.255.252.0' > 11:05:12 NetworkManager[1198]: <info> [1583143512.5424] dhcp4 (wlp3s0): state changed unknown -> bound > 11:05:12 NetworkManager[1198]: <info> [1583143512.5452] device (wlp3s0): state change: ip-config -> ip-check (reason 'none', sys-iface-state: 'managed') > 11:05:12 NetworkManager[1198]: <info> [1583143512.5477] device (wlp3s0): state change: ip-check -> secondaries (reason 'none', sys-iface-state: 'managed') > 11:05:12 NetworkManager[1198]: <info> [1583143512.5480] device (wlp3s0): state change: secondaries -> activated (reason 'none', sys-iface-state: 'managed') > 11:05:12 NetworkManager[1198]: <info> [1583143512.5489] manager: NetworkManager state is now CONNECTED_LOCAL > 11:05:12 NetworkManager[1198]: <info> [1583143512.5514] manager: NetworkManager state is now CONNECTED_SITE > 11:05:12 NetworkManager[1198]: <info> [1583143512.5515] policy: set 'Red Hat Guest' (wlp3s0) as default for IPv4 routing and DNS > 11:05:12 NetworkManager[1198]: <info> [1583143512.5528] device (wlp3s0): Activation: successful, device activated. > 11:05:12 NetworkManager[1198]: <info> [1583143512.5536] manager: NetworkManager state is now CONNECTED_GLOBAL ^ at this point, I think NetworkManager assumed resolv.conf was successfully written, but it was _not_, which would be discovered if either of these happened: o return value from fopen/open + fwrite/write (or equivalents) was actually checked and propagated - this is the proper resolution, I think, not sure if there are any artificially imposed obstacles to this (limitation of the APIs, etc.) o localized behavioural self-test was run (does the data in the file resembles what shall have been written?) o full round-trip behavioural self-test was run (can we actually resolve "." server, for instance?) - this is actually pretty tricky if no previous suggested measure was run -- if my home router previously set the DNS to be 8.8.8.8, it would still be functional and would not reflect user's wish to use what the current DHCP response wanted to set, rendering internal LAN enablement broken, for instance - I don't think DNS would ever be subject to "captive portal" barrier -- it would perhaps hijack the resolution in a sense, but would still keep anything resolvable? > 11:05:13 systemd-networkd[1199]: wlp3s0: Gained IPv6LL > 11:10:20 systemd[1]: Starting system activity accounting tool... > 11:10:20 sa1[3305463]: Cannot write data to system activity file: No space left on device > 11:10:20 systemd[1]: sysstat-collect.service: Main process exited, code=exited, status=2/INVALIDARGUMENT > 11:10:20 systemd[1]: sysstat-collect.service: Failed with result 'exit-code'. > 11:10:20 systemd[1]: Failed to start system activity accounting tool. > 11:15:56 NetworkManager[1198]: <info> [1583144156.7084] device (wlp3s0): state change: activated -> deactivating (reason 'user-requested', sys-iface-state: 'managed') > 11:15:56 NetworkManager[1198]: <info> [1583144156.7121] manager: NetworkManager state is now CONNECTED_LOCAL > 11:15:56 NetworkManager[1198]: <info> [1583144156.7134] audit: op="connection-deactivate" uuid="b5130fe3-9d50-4c7b-8de7-e3d79ace9e3a" name="Red Hat Guest" pid=3305771 uid=0 result="success" ^ me trying to reconnect, if it helps > 11:15:56 systemd-networkd[1199]: wlp3s0: Lost carrier > 11:15:56 NetworkManager[1198]: <warn> [1583144156.8182] sup-iface[0x56067f3d9cf0,wlp3s0]: connection disconnected (reason -3) > 11:15:56 NetworkManager[1198]: <info> [1583144156.8185] device (wlp3s0): state change: deactivating -> disconnected (reason 'user-requested', sys-iface-state: 'managed') > 11:15:56 NetworkManager[1198]: <info> [1583144156.8380] dhcp4 (wlp3s0): canceled DHCP transaction > 11:15:56 NetworkManager[1198]: <info> [1583144156.8380] dhcp4 (wlp3s0): state changed bound -> done > 11:15:56 systemd-networkd[1199]: wlp3s0: Link DOWN > 11:15:56 NetworkManager[1198]: <info> [1583144156.8591] device (wlp3s0): set-hw-addr: set MAC address to BA:83:28:D8:CA:03 (scanning) > 11:15:56 systemd-networkd[1199]: wlp3s0: Link UP > 11:15:56 NetworkManager[1198]: <info> [1583144156.8686] device (wlp3s0): supplicant interface state: completed -> disconnected > 11:15:56 NetworkManager[1198]: <info> [1583144156.8687] device (p2p-dev-wlp3s0): supplicant management interface state: completed -> disconnected > 11:15:56 NetworkManager[1198]: <info> [1583144156.8691] device (wlp3s0): supplicant interface state: disconnected -> disabled > 11:15:56 NetworkManager[1198]: <info> [1583144156.8692] device (p2p-dev-wlp3s0): supplicant management interface state: disconnected -> disabled > 11:15:56 NetworkManager[1198]: <info> [1583144156.8787] device (wlp3s0): supplicant interface state: disabled -> disconnected > 11:15:56 NetworkManager[1198]: <info> [1583144156.8788] device (p2p-dev-wlp3s0): supplicant management interface state: disabled -> disconnected > 11:16:03 NetworkManager[1198]: <info> [1583144163.8953] agent-manager: agent[ff50dac3568553b9,:1.16556/nmtui/0]: agent registered > 11:16:03 NetworkManager[1198]: <info> [1583144163.8987] device (wlp3s0): Activation: starting connection 'Red Hat Guest' (b5130fe3-9d50-4c7b-8de7-e3d79ace9e3a) > 11:16:03 NetworkManager[1198]: <info> [1583144163.8990] audit: op="connection-activate" uuid="b5130fe3-9d50-4c7b-8de7-e3d79ace9e3a" name="Red Hat Guest" pid=3305771 uid=0 result="success" ^ see the previous remark > 11:16:03 NetworkManager[1198]: <info> [1583144163.8994] device (wlp3s0): state change: disconnected -> prepare (reason 'none', sys-iface-state: 'managed') > 11:16:03 NetworkManager[1198]: <info> [1583144163.9009] manager: NetworkManager state is now CONNECTING > 11:16:03 systemd-networkd[1199]: wlp3s0: Link DOWN > 11:16:03 NetworkManager[1198]: <info> [1583144163.9287] device (wlp3s0): set-hw-addr: reset MAC address to 44:85:00SCRAMBLED (preserve) > 11:16:03 systemd-networkd[1199]: wlp3s0: Link UP > 11:16:03 NetworkManager[1198]: <info> [1583144163.9348] device (wlp3s0): state change: prepare -> config (reason 'none', sys-iface-state: 'managed') > 11:16:03 NetworkManager[1198]: <info> [1583144163.9359] device (wlp3s0): Activation: (wifi) access point 'Red Hat Guest' has security, but secrets are required. > 11:16:03 NetworkManager[1198]: <info> [1583144163.9360] device (wlp3s0): state change: config -> need-auth (reason 'none', sys-iface-state: 'managed') > 11:16:03 NetworkManager[1198]: <info> [1583144163.9436] device (wlp3s0): supplicant interface state: disconnected -> disabled > 11:16:03 NetworkManager[1198]: <info> [1583144163.9437] device (p2p-dev-wlp3s0): supplicant management interface state: disconnected -> disabled > 11:16:03 NetworkManager[1198]: <info> [1583144163.9456] device (wlp3s0): state change: need-auth -> prepare (reason 'none', sys-iface-state: 'managed') > 11:16:03 NetworkManager[1198]: <info> [1583144163.9461] device (wlp3s0): state change: prepare -> config (reason 'none', sys-iface-state: 'managed') > 11:16:03 NetworkManager[1198]: <info> [1583144163.9465] device (wlp3s0): Activation: (wifi) connection 'Red Hat Guest' has security, and secrets exist. No new secrets needed. > 11:16:03 NetworkManager[1198]: <info> [1583144163.9465] Config: added 'ssid' value 'Red Hat Guest' > 11:16:03 NetworkManager[1198]: <info> [1583144163.9466] Config: added 'scan_ssid' value '1' > 11:16:03 NetworkManager[1198]: <info> [1583144163.9466] Config: added 'bgscan' value 'simple:30:-70:86400' > 11:16:03 NetworkManager[1198]: <info> [1583144163.9466] Config: added 'key_mgmt' value 'WPA-PSK WPA-PSK-SHA256 FT-PSK' > 11:16:03 NetworkManager[1198]: <info> [1583144163.9466] Config: added 'auth_alg' value 'OPEN' > 11:16:03 NetworkManager[1198]: <info> [1583144163.9466] Config: added 'psk' value '<hidden>' > 11:16:03 NetworkManager[1198]: <info> [1583144163.9539] device (wlp3s0): supplicant interface state: disabled -> disconnected > 11:16:03 NetworkManager[1198]: <info> [1583144163.9540] device (p2p-dev-wlp3s0): supplicant management interface state: disabled -> disconnected > 11:16:03 NetworkManager[1198]: <info> [1583144163.9639] device (wlp3s0): supplicant interface state: disconnected -> scanning > 11:16:03 NetworkManager[1198]: <info> [1583144163.9640] device (p2p-dev-wlp3s0): supplicant management interface state: disconnected -> scanning > 11:16:04 NetworkManager[1198]: <info> [1583144164.7531] device (wlp3s0): supplicant interface state: scanning -> authenticating > 11:16:04 NetworkManager[1198]: <info> [1583144164.7532] device (p2p-dev-wlp3s0): supplicant management interface state: scanning -> authenticating > 11:16:04 NetworkManager[1198]: <info> [1583144164.7786] device (wlp3s0): supplicant interface state: authenticating -> associating > 11:16:04 NetworkManager[1198]: <info> [1583144164.7787] device (p2p-dev-wlp3s0): supplicant management interface state: authenticating -> associating > 11:16:04 NetworkManager[1198]: <info> [1583144164.7791] device (wlp3s0): supplicant interface state: associating -> associated > 11:16:04 NetworkManager[1198]: <info> [1583144164.7791] device (p2p-dev-wlp3s0): supplicant management interface state: associating -> associated > 11:16:04 systemd-networkd[1199]: wlp3s0: Gained carrier > 11:16:04 systemd-networkd[1199]: wlp3s0: Connected WiFi access point: Red Hat Guest (SCRAMBLED) > 11:16:04 NetworkManager[1198]: <info> [1583144164.7894] device (wlp3s0): supplicant interface state: associated -> completed > 11:16:04 NetworkManager[1198]: <info> [1583144164.7895] device (wlp3s0): Activation: (wifi) Stage 2 of 5 (Device Configure) successful. Connected to wireless network "Red Hat Guest" > 11:16:04 NetworkManager[1198]: <info> [1583144164.7896] device (p2p-dev-wlp3s0): supplicant management interface state: associated -> completed > 11:16:04 NetworkManager[1198]: <info> [1583144164.7960] device (wlp3s0): state change: config -> ip-config (reason 'none', sys-iface-state: 'managed') > 11:16:04 NetworkManager[1198]: <info> [1583144164.7963] dhcp4 (wlp3s0): activation: beginning transaction (timeout in 45 seconds) > 11:16:05 NetworkManager[1198]: <info> [1583144165.8534] dhcp4 (wlp3s0): option dhcp_lease_time => '7200' > 11:16:05 NetworkManager[1198]: <info> [1583144165.8535] dhcp4 (wlp3s0): option domain_name => 'redhat.com' > 11:16:05 NetworkManager[1198]: <info> [1583144165.8536] dhcp4 (wlp3s0): option domain_name_servers => '10.SCRAMBLED 10.SCRAMBLED 10.SCRAMBLED' > 11:16:05 NetworkManager[1198]: <info> [1583144165.8536] dhcp4 (wlp3s0): option expiry => '1583151364' > 11:16:05 NetworkManager[1198]: <info> [1583144165.8537] dhcp4 (wlp3s0): option ip_address => '10.SCRAMBLED' > 11:16:05 NetworkManager[1198]: <info> [1583144165.8537] dhcp4 (wlp3s0): option ntp_servers => '10.SCRAMBLED' > 11:16:05 NetworkManager[1198]: <info> [1583144165.8537] dhcp4 (wlp3s0): option requested_broadcast_address => '1' > 11:16:05 NetworkManager[1198]: <info> [1583144165.8537] dhcp4 (wlp3s0): option requested_dhcp_server_identifier => '1' > 11:16:05 NetworkManager[1198]: <info> [1583144165.8538] dhcp4 (wlp3s0): option requested_domain_name => '1' > 11:16:05 NetworkManager[1198]: <info> [1583144165.8538] dhcp4 (wlp3s0): option requested_domain_name_servers => '1' > 11:16:05 NetworkManager[1198]: <info> [1583144165.8539] dhcp4 (wlp3s0): option requested_domain_search => '1' > 11:16:05 NetworkManager[1198]: <info> [1583144165.8539] dhcp4 (wlp3s0): option requested_host_name => '1' > 11:16:05 NetworkManager[1198]: <info> [1583144165.8539] dhcp4 (wlp3s0): option requested_interface_mtu => '1' > 11:16:05 NetworkManager[1198]: <info> [1583144165.8540] dhcp4 (wlp3s0): option requested_ms_classless_static_routes => '1' > 11:16:05 NetworkManager[1198]: <info> [1583144165.8540] dhcp4 (wlp3s0): option requested_nis_domain => '1' > 11:16:05 NetworkManager[1198]: <info> [1583144165.8540] dhcp4 (wlp3s0): option requested_nis_servers => '1' > 11:16:05 NetworkManager[1198]: <info> [1583144165.8540] dhcp4 (wlp3s0): option requested_ntp_servers => '1' > 11:16:05 NetworkManager[1198]: <info> [1583144165.8541] dhcp4 (wlp3s0): option requested_rfc3442_classless_static_routes => '1' > 11:16:05 NetworkManager[1198]: <info> [1583144165.8541] dhcp4 (wlp3s0): option requested_root_path => '1' > 11:16:05 NetworkManager[1198]: <info> [1583144165.8541] dhcp4 (wlp3s0): option requested_routers => '1' > 11:16:05 NetworkManager[1198]: <info> [1583144165.8542] dhcp4 (wlp3s0): option requested_static_routes => '1' > 11:16:05 NetworkManager[1198]: <info> [1583144165.8542] dhcp4 (wlp3s0): option requested_subnet_mask => '1' > 11:16:05 NetworkManager[1198]: <info> [1583144165.8542] dhcp4 (wlp3s0): option requested_time_offset => '1' > 11:16:05 NetworkManager[1198]: <info> [1583144165.8542] dhcp4 (wlp3s0): option requested_wpad => '1' > 11:16:05 NetworkManager[1198]: <info> [1583144165.8543] dhcp4 (wlp3s0): option routers => '10.SCRAMBLED' > 11:16:05 NetworkManager[1198]: <info> [1583144165.8543] dhcp4 (wlp3s0): option subnet_mask => '255.255.252.0' > 11:16:05 NetworkManager[1198]: <info> [1583144165.8543] dhcp4 (wlp3s0): state changed unknown -> bound > 11:16:05 NetworkManager[1198]: <info> [1583144165.8597] device (wlp3s0): state change: ip-config -> ip-check (reason 'none', sys-iface-state: 'managed') > 11:16:05 NetworkManager[1198]: <info> [1583144165.8646] device (wlp3s0): state change: ip-check -> secondaries (reason 'none', sys-iface-state: 'managed') > 11:16:05 NetworkManager[1198]: <info> [1583144165.8654] device (wlp3s0): state change: secondaries -> activated (reason 'none', sys-iface-state: 'managed') > 11:16:05 NetworkManager[1198]: <info> [1583144165.8669] manager: NetworkManager state is now CONNECTED_LOCAL > 11:16:05 NetworkManager[1198]: <info> [1583144165.8724] manager: NetworkManager state is now CONNECTED_SITE > 11:16:05 NetworkManager[1198]: <info> [1583144165.8728] policy: set 'Red Hat Guest' (wlp3s0) as default for IPv4 routing and DNS > 11:16:05 NetworkManager[1198]: <info> [1583144165.8759] device (wlp3s0): Activation: successful, device activated. > 11:16:05 NetworkManager[1198]: <info> [1583144165.8776] manager: NetworkManager state is now CONNECTED_GLOBAL > 11:16:06 systemd-networkd[1199]: wlp3s0: Gained IPv6LL > 11:19:06 NetworkManager[1198]: <info> [1583144346.6817] device (wlp3s0): state change: activated -> deactivating (reason 'user-requested', sys-iface-state: 'managed') > 11:19:06 NetworkManager[1198]: <info> [1583144346.6860] manager: NetworkManager state is now CONNECTED_LOCAL > 11:19:06 NetworkManager[1198]: <info> [1583144346.6871] audit: op="connection-deactivate" uuid="b5130fe3-9d50-4c7b-8de7-e3d79ace9e3a" name="Red Hat Guest" pid=3305771 uid=0 result="success" > 11:19:06 systemd-networkd[1199]: wlp3s0: Lost carrier > 11:19:06 NetworkManager[1198]: <warn> [1583144346.7850] sup-iface[0x56067f3d9cf0,wlp3s0]: connection disconnected (reason -3) > 11:19:06 NetworkManager[1198]: <info> [1583144346.7863] device (wlp3s0): supplicant interface state: completed -> disconnected > 11:19:06 NetworkManager[1198]: <info> [1583144346.7863] device (p2p-dev-wlp3s0): supplicant management interface state: completed -> disconnected > 11:19:06 NetworkManager[1198]: <info> [1583144346.7865] device (wlp3s0): state change: deactivating -> disconnected (reason 'user-requested', sys-iface-state: 'managed') > 11:19:06 NetworkManager[1198]: <info> [1583144346.8059] dhcp4 (wlp3s0): canceled DHCP transaction > 11:19:06 NetworkManager[1198]: <info> [1583144346.8060] dhcp4 (wlp3s0): state changed bound -> done > 11:19:06 systemd-networkd[1199]: wlp3s0: Link DOWN > 11:19:06 NetworkManager[1198]: <info> [1583144346.8253] device (wlp3s0): set-hw-addr: set MAC address to DA:0D:4A:E2:1B:22 (scanning) > 11:19:06 systemd-networkd[1199]: wlp3s0: Link UP > 11:19:06 NetworkManager[1198]: <info> [1583144346.8405] device (wlp3s0): supplicant interface state: disconnected -> disabled > 11:19:06 NetworkManager[1198]: <info> [1583144346.8406] device (p2p-dev-wlp3s0): supplicant management interface state: disconnected -> disabled > 11:19:06 NetworkManager[1198]: <info> [1583144346.8547] device (wlp3s0): supplicant interface state: disabled -> disconnected > 11:19:06 NetworkManager[1198]: <info> [1583144346.8549] device (p2p-dev-wlp3s0): supplicant management interface state: disabled -> disconnected > 11:19:33 NetworkManager[1198]: <info> [1583144373.7504] agent-manager: agent[90be93161dc3f69c,:1.16556/nmtui/0]: agent registered > 11:19:33 NetworkManager[1198]: <info> [1583144373.7521] device (wlp3s0): Activation: starting connection 'Red Hat Guest' (b5130fe3-9d50-4c7b-8de7-e3d79ace9e3a) > 11:19:33 NetworkManager[1198]: <info> [1583144373.7522] audit: op="connection-activate" uuid="b5130fe3-9d50-4c7b-8de7-e3d79ace9e3a" name="Red Hat Guest" pid=3305771 uid=0 result="success" > 11:19:33 NetworkManager[1198]: <info> [1583144373.7524] device (wlp3s0): state change: disconnected -> prepare (reason 'none', sys-iface-state: 'managed') > 11:19:33 NetworkManager[1198]: <info> [1583144373.7531] manager: NetworkManager state is now CONNECTING > 11:19:33 systemd-networkd[1199]: wlp3s0: Link DOWN > 11:19:33 NetworkManager[1198]: <info> [1583144373.7651] device (wlp3s0): set-hw-addr: reset MAC address to 44:85:00SCRAMBLED (preserve) > 11:19:33 systemd-networkd[1199]: wlp3s0: Link UP > 11:19:33 NetworkManager[1198]: <info> [1583144373.7705] device (wlp3s0): state change: prepare -> config (reason 'none', sys-iface-state: 'managed') > 11:19:33 NetworkManager[1198]: <info> [1583144373.7711] device (wlp3s0): Activation: (wifi) access point 'Red Hat Guest' has security, but secrets are required. > 11:19:33 NetworkManager[1198]: <info> [1583144373.7712] device (wlp3s0): state change: config -> need-auth (reason 'none', sys-iface-state: 'managed') > 11:19:33 NetworkManager[1198]: <info> [1583144373.7754] device (wlp3s0): supplicant interface state: disconnected -> disabled > 11:19:33 NetworkManager[1198]: <info> [1583144373.7755] device (p2p-dev-wlp3s0): supplicant management interface state: disconnected -> disabled > 11:19:33 NetworkManager[1198]: <info> [1583144373.7773] device (wlp3s0): state change: need-auth -> prepare (reason 'none', sys-iface-state: 'managed') > 11:19:33 NetworkManager[1198]: <info> [1583144373.7778] device (wlp3s0): state change: prepare -> config (reason 'none', sys-iface-state: 'managed') > 11:19:33 NetworkManager[1198]: <info> [1583144373.7784] device (wlp3s0): Activation: (wifi) connection 'Red Hat Guest' has security, and secrets exist. No new secrets needed. > 11:19:33 NetworkManager[1198]: <info> [1583144373.7784] Config: added 'ssid' value 'Red Hat Guest' > 11:19:33 NetworkManager[1198]: <info> [1583144373.7784] Config: added 'scan_ssid' value '1' > 11:19:33 NetworkManager[1198]: <info> [1583144373.7784] Config: added 'bgscan' value 'simple:30:-70:86400' > 11:19:33 NetworkManager[1198]: <info> [1583144373.7785] Config: added 'key_mgmt' value 'WPA-PSK WPA-PSK-SHA256 FT-PSK' > 11:19:33 NetworkManager[1198]: <info> [1583144373.7785] Config: added 'auth_alg' value 'OPEN' > 11:19:33 NetworkManager[1198]: <info> [1583144373.7785] Config: added 'psk' value '<hidden>' > 11:19:33 NetworkManager[1198]: <info> [1583144373.7883] device (wlp3s0): supplicant interface state: disabled -> disconnected > 11:19:33 NetworkManager[1198]: <info> [1583144373.7883] device (p2p-dev-wlp3s0): supplicant management interface state: disabled -> disconnected > 11:19:33 NetworkManager[1198]: <info> [1583144373.7982] device (wlp3s0): supplicant interface state: disconnected -> scanning > 11:19:33 NetworkManager[1198]: <info> [1583144373.7982] device (p2p-dev-wlp3s0): supplicant management interface state: disconnected -> scanning > 11:19:34 NetworkManager[1198]: <info> [1583144374.4700] device (wlp3s0): supplicant interface state: scanning -> authenticating > 11:19:34 NetworkManager[1198]: <info> [1583144374.4701] device (p2p-dev-wlp3s0): supplicant management interface state: scanning -> authenticating > 11:19:34 systemd-networkd[1199]: wlp3s0: Gained carrier > 11:19:34 NetworkManager[1198]: <info> [1583144374.4914] device (wlp3s0): supplicant interface state: authenticating -> associating > 11:19:34 NetworkManager[1198]: <info> [1583144374.4915] device (p2p-dev-wlp3s0): supplicant management interface state: authenticating -> associating > 11:19:34 NetworkManager[1198]: <info> [1583144374.4920] device (wlp3s0): supplicant interface state: associating -> completed > 11:19:34 NetworkManager[1198]: <info> [1583144374.4921] device (wlp3s0): Activation: (wifi) Stage 2 of 5 (Device Configure) successful. Connected to wireless network "Red Hat Guest" > 11:19:34 NetworkManager[1198]: <info> [1583144374.4923] device (p2p-dev-wlp3s0): supplicant management interface state: associating -> completed > 11:19:34 NetworkManager[1198]: <info> [1583144374.5047] device (wlp3s0): state change: config -> ip-config (reason 'none', sys-iface-state: 'managed') > 11:19:34 NetworkManager[1198]: <info> [1583144374.5052] dhcp4 (wlp3s0): activation: beginning transaction (timeout in 45 seconds) > 11:19:35 NetworkManager[1198]: <info> [1583144375.5682] dhcp4 (wlp3s0): option dhcp_lease_time => '7200' > 11:19:35 NetworkManager[1198]: <info> [1583144375.5683] dhcp4 (wlp3s0): option domain_name => 'redhat.com' > 11:19:35 NetworkManager[1198]: <info> [1583144375.5683] dhcp4 (wlp3s0): option domain_name_servers => '10.SCRAMBLED 10.SCRAMBLED 10.SCRAMBLED' > 11:19:35 NetworkManager[1198]: <info> [1583144375.5683] dhcp4 (wlp3s0): option expiry => '1583151574' > 11:19:35 NetworkManager[1198]: <info> [1583144375.5684] dhcp4 (wlp3s0): option ip_address => '10.SCRAMBLED' > 11:19:35 NetworkManager[1198]: <info> [1583144375.5684] dhcp4 (wlp3s0): option ntp_servers => '10.SCRAMBLED' > 11:19:35 NetworkManager[1198]: <info> [1583144375.5684] dhcp4 (wlp3s0): option requested_broadcast_address => '1' > 11:19:35 NetworkManager[1198]: <info> [1583144375.5685] dhcp4 (wlp3s0): option requested_dhcp_server_identifier => '1' > 11:19:35 NetworkManager[1198]: <info> [1583144375.5685] dhcp4 (wlp3s0): option requested_domain_name => '1' > 11:19:35 NetworkManager[1198]: <info> [1583144375.5685] dhcp4 (wlp3s0): option requested_domain_name_servers => '1' > 11:19:35 NetworkManager[1198]: <info> [1583144375.5686] dhcp4 (wlp3s0): option requested_domain_search => '1' > 11:19:35 NetworkManager[1198]: <info> [1583144375.5686] dhcp4 (wlp3s0): option requested_host_name => '1' > 11:19:35 NetworkManager[1198]: <info> [1583144375.5686] dhcp4 (wlp3s0): option requested_interface_mtu => '1' > 11:19:35 NetworkManager[1198]: <info> [1583144375.5686] dhcp4 (wlp3s0): option requested_ms_classless_static_routes => '1' > 11:19:35 NetworkManager[1198]: <info> [1583144375.5687] dhcp4 (wlp3s0): option requested_nis_domain => '1' > 11:19:35 NetworkManager[1198]: <info> [1583144375.5687] dhcp4 (wlp3s0): option requested_nis_servers => '1' > 11:19:35 NetworkManager[1198]: <info> [1583144375.5687] dhcp4 (wlp3s0): option requested_ntp_servers => '1' > 11:19:35 NetworkManager[1198]: <info> [1583144375.5688] dhcp4 (wlp3s0): option requested_rfc3442_classless_static_routes => '1' > 11:19:35 NetworkManager[1198]: <info> [1583144375.5688] dhcp4 (wlp3s0): option requested_root_path => '1' > 11:19:35 NetworkManager[1198]: <info> [1583144375.5689] dhcp4 (wlp3s0): option requested_routers => '1' > 11:19:35 NetworkManager[1198]: <info> [1583144375.5689] dhcp4 (wlp3s0): option requested_static_routes => '1' > 11:19:35 NetworkManager[1198]: <info> [1583144375.5689] dhcp4 (wlp3s0): option requested_subnet_mask => '1' > 11:19:35 NetworkManager[1198]: <info> [1583144375.5690] dhcp4 (wlp3s0): option requested_time_offset => '1' > 11:19:35 NetworkManager[1198]: <info> [1583144375.5690] dhcp4 (wlp3s0): option requested_wpad => '1' > 11:19:35 NetworkManager[1198]: <info> [1583144375.5690] dhcp4 (wlp3s0): option routers => '10.SCRAMBLED' > 11:19:35 NetworkManager[1198]: <info> [1583144375.5691] dhcp4 (wlp3s0): option subnet_mask => '255.255.252.0' > 11:19:35 NetworkManager[1198]: <info> [1583144375.5691] dhcp4 (wlp3s0): state changed unknown -> bound > 11:19:35 NetworkManager[1198]: <info> [1583144375.5748] device (wlp3s0): state change: ip-config -> ip-check (reason 'none', sys-iface-state: 'managed') > 11:19:35 NetworkManager[1198]: <info> [1583144375.6364] device (wlp3s0): state change: ip-check -> secondaries (reason 'none', sys-iface-state: 'managed') > 11:19:35 NetworkManager[1198]: <info> [1583144375.6370] device (wlp3s0): state change: secondaries -> activated (reason 'none', sys-iface-state: 'managed') > 11:19:35 NetworkManager[1198]: <info> [1583144375.6381] manager: NetworkManager state is now CONNECTED_LOCAL > 11:19:35 NetworkManager[1198]: <info> [1583144375.6421] manager: NetworkManager state is now CONNECTED_SITE > 11:19:35 NetworkManager[1198]: <info> [1583144375.6423] policy: set 'Red Hat Guest' (wlp3s0) as default for IPv4 routing and DNS > 11:19:35 NetworkManager[1198]: <info> [1583144375.6441] device (wlp3s0): Activation: successful, device activated. > 11:19:35 NetworkManager[1198]: <info> [1583144375.6453] manager: NetworkManager state is now CONNECTED_GLOBAL > 11:19:36 systemd-networkd[1199]: wlp3s0: Gained IPv6LL > 11:20:15 systemd[1]: Starting system activity accounting tool... > 11:20:15 sa1[3306050]: Cannot write data to system activity file: No space left on device > 11:20:15 systemd[1]: sysstat-collect.service: Main process exited, code=exited, status=2/INVALIDARGUMENT > 11:20:15 systemd[1]: sysstat-collect.service: Failed with result 'exit-code'. > 11:20:15 systemd[1]: Failed to start system activity accounting tool. ^ apparently, disk full scenario was still ongoing, and at this point, I think I checked /etc/resolv.conf, observed references to nameservers of my home ISP, realized something fishy is going on and luckily very shortly after, realized that the root mount point is almost full, which explained that stale entry from here, it was easy, make more free space -> ... > 11:23:13 NetworkManager[1198]: <info> [1583144593.5389] device (wlp3s0): state change: activated -> deactivating (reason 'user-requested', sys-iface-state: 'managed') > 11:23:13 NetworkManager[1198]: <info> [1583144593.5432] manager: NetworkManager state is now CONNECTED_LOCAL > 11:23:13 NetworkManager[1198]: <info> [1583144593.5446] audit: op="connection-deactivate" uuid="b5130fe3-9d50-4c7b-8de7-e3d79ace9e3a" name="Red Hat Guest" pid=3306261 uid=0 result="success" ^ ... -> disconnect -> ... > 11:23:13 systemd-networkd[1199]: wlp3s0: Lost carrier > 11:23:13 NetworkManager[1198]: <warn> [1583144593.6306] sup-iface[0x56067f3d9cf0,wlp3s0]: connection disconnected (reason -3) > 11:23:13 NetworkManager[1198]: <info> [1583144593.6309] device (wlp3s0): supplicant interface state: completed -> disconnected > 11:23:13 NetworkManager[1198]: <info> [1583144593.6309] device (p2p-dev-wlp3s0): supplicant management interface state: completed -> disconnected > 11:23:13 NetworkManager[1198]: <info> [1583144593.6313] device (wlp3s0): state change: deactivating -> disconnected (reason 'user-requested', sys-iface-state: 'managed') > 11:23:13 NetworkManager[1198]: <info> [1583144593.6500] dhcp4 (wlp3s0): canceled DHCP transaction > 11:23:13 NetworkManager[1198]: <info> [1583144593.6500] dhcp4 (wlp3s0): state changed bound -> done > 11:23:13 systemd-networkd[1199]: wlp3s0: Link DOWN > 11:23:13 NetworkManager[1198]: <info> [1583144593.6684] device (wlp3s0): set-hw-addr: set MAC address to 4E:DA:8D:01:82:93 (scanning) > 11:23:13 systemd-networkd[1199]: wlp3s0: Link UP > 11:23:13 NetworkManager[1198]: <info> [1583144593.6895] device (wlp3s0): supplicant interface state: disconnected -> disabled > 11:23:13 NetworkManager[1198]: <info> [1583144593.6895] device (p2p-dev-wlp3s0): supplicant management interface state: disconnected -> disabled > 11:23:13 NetworkManager[1198]: <info> [1583144593.7326] device (wlp3s0): supplicant interface state: disabled -> disconnected > 11:23:13 NetworkManager[1198]: <info> [1583144593.7327] device (p2p-dev-wlp3s0): supplicant management interface state: disabled -> disconnected > 11:23:16 NetworkManager[1198]: <info> [1583144596.9888] agent-manager: agent[5bc57a54029f33cc,:1.16561/nmtui/0]: agent registered > 11:23:16 NetworkManager[1198]: <info> [1583144596.9921] device (wlp3s0): Activation: starting connection 'Red Hat Guest' (b5130fe3-9d50-4c7b-8de7-e3d79ace9e3a) > 11:23:16 NetworkManager[1198]: <info> [1583144596.9923] audit: op="connection-activate" uuid="b5130fe3-9d50-4c7b-8de7-e3d79ace9e3a" name="Red Hat Guest" pid=3306261 uid=0 result="success" ^ ... -> reconnect > 11:23:16 NetworkManager[1198]: <info> [1583144596.9927] device (wlp3s0): state change: disconnected -> prepare (reason 'none', sys-iface-state: 'managed') > 11:23:16 NetworkManager[1198]: <info> [1583144596.9938] manager: NetworkManager state is now CONNECTING > 11:23:16 systemd-networkd[1199]: wlp3s0: Link DOWN > 11:23:17 NetworkManager[1198]: <info> [1583144597.0216] device (wlp3s0): set-hw-addr: reset MAC address to 44:85:00SCRAMBLED (preserve) > 11:23:17 systemd-networkd[1199]: wlp3s0: Link UP > 11:23:17 NetworkManager[1198]: <info> [1583144597.0281] device (wlp3s0): state change: prepare -> config (reason 'none', sys-iface-state: 'managed') > 11:23:17 NetworkManager[1198]: <info> [1583144597.0292] device (wlp3s0): Activation: (wifi) access point 'Red Hat Guest' has security, but secrets are required. > 11:23:17 NetworkManager[1198]: <info> [1583144597.0293] device (wlp3s0): state change: config -> need-auth (reason 'none', sys-iface-state: 'managed') > 11:23:17 NetworkManager[1198]: <info> [1583144597.0337] device (wlp3s0): supplicant interface state: disconnected -> disabled > 11:23:17 NetworkManager[1198]: <info> [1583144597.0338] device (p2p-dev-wlp3s0): supplicant management interface state: disconnected -> disabled > 11:23:17 NetworkManager[1198]: <info> [1583144597.0365] device (wlp3s0): state change: need-auth -> prepare (reason 'none', sys-iface-state: 'managed') > 11:23:17 NetworkManager[1198]: <info> [1583144597.0372] device (wlp3s0): state change: prepare -> config (reason 'none', sys-iface-state: 'managed') > 11:23:17 NetworkManager[1198]: <info> [1583144597.0381] device (wlp3s0): Activation: (wifi) connection 'Red Hat Guest' has security, and secrets exist. No new secrets needed. > 11:23:17 NetworkManager[1198]: <info> [1583144597.0381] Config: added 'ssid' value 'Red Hat Guest' > 11:23:17 NetworkManager[1198]: <info> [1583144597.0382] Config: added 'scan_ssid' value '1' > 11:23:17 NetworkManager[1198]: <info> [1583144597.0382] Config: added 'bgscan' value 'simple:30:-70:86400' > 11:23:17 NetworkManager[1198]: <info> [1583144597.0382] Config: added 'key_mgmt' value 'WPA-PSK WPA-PSK-SHA256 FT-PSK' > 11:23:17 NetworkManager[1198]: <info> [1583144597.0382] Config: added 'auth_alg' value 'OPEN' > 11:23:17 NetworkManager[1198]: <info> [1583144597.0382] Config: added 'psk' value '<hidden>' > 11:23:17 NetworkManager[1198]: <info> [1583144597.0509] device (wlp3s0): supplicant interface state: disabled -> disconnected > 11:23:17 NetworkManager[1198]: <info> [1583144597.0510] device (p2p-dev-wlp3s0): supplicant management interface state: disabled -> disconnected > 11:23:17 NetworkManager[1198]: <info> [1583144597.0624] device (wlp3s0): supplicant interface state: disconnected -> scanning > 11:23:17 NetworkManager[1198]: <info> [1583144597.0625] device (p2p-dev-wlp3s0): supplicant management interface state: disconnected -> scanning > 11:23:17 NetworkManager[1198]: <info> [1583144597.6190] device (wlp3s0): supplicant interface state: scanning -> authenticating > 11:23:17 NetworkManager[1198]: <info> [1583144597.6191] device (p2p-dev-wlp3s0): supplicant management interface state: scanning -> authenticating > 11:23:17 systemd-networkd[1199]: wlp3s0: Gained carrier > 11:23:17 NetworkManager[1198]: <info> [1583144597.6389] device (wlp3s0): supplicant interface state: authenticating -> associating > 11:23:17 NetworkManager[1198]: <info> [1583144597.6390] device (p2p-dev-wlp3s0): supplicant management interface state: authenticating -> associating > 11:23:17 NetworkManager[1198]: <info> [1583144597.6412] device (wlp3s0): supplicant interface state: associating -> completed > 11:23:17 NetworkManager[1198]: <info> [1583144597.6413] device (wlp3s0): Activation: (wifi) Stage 2 of 5 (Device Configure) successful. Connected to wireless network "Red Hat Guest" > 11:23:17 NetworkManager[1198]: <info> [1583144597.6414] device (p2p-dev-wlp3s0): supplicant management interface state: associating -> completed > 11:23:17 NetworkManager[1198]: <info> [1583144597.6526] device (wlp3s0): state change: config -> ip-config (reason 'none', sys-iface-state: 'managed') > 11:23:17 NetworkManager[1198]: <info> [1583144597.6532] dhcp4 (wlp3s0): activation: beginning transaction (timeout in 45 seconds) > 11:23:18 NetworkManager[1198]: <info> [1583144598.7217] dhcp4 (wlp3s0): option dhcp_lease_time => '7200' > 11:23:18 NetworkManager[1198]: <info> [1583144598.7218] dhcp4 (wlp3s0): option domain_name => 'redhat.com' > 11:23:18 NetworkManager[1198]: <info> [1583144598.7218] dhcp4 (wlp3s0): option domain_name_servers => '10.SCRAMBLED 10.SCRAMBLED 10.SCRAMBLED' > 11:23:18 NetworkManager[1198]: <info> [1583144598.7218] dhcp4 (wlp3s0): option expiry => '1583151797' > 11:23:18 NetworkManager[1198]: <info> [1583144598.7218] dhcp4 (wlp3s0): option ip_address => '10.SCRAMBLED' > 11:23:18 NetworkManager[1198]: <info> [1583144598.7219] dhcp4 (wlp3s0): option ntp_servers => '10.SCRAMBLED' > 11:23:18 NetworkManager[1198]: <info> [1583144598.7219] dhcp4 (wlp3s0): option requested_broadcast_address => '1' > 11:23:18 NetworkManager[1198]: <info> [1583144598.7219] dhcp4 (wlp3s0): option requested_dhcp_server_identifier => '1' > 11:23:18 NetworkManager[1198]: <info> [1583144598.7219] dhcp4 (wlp3s0): option requested_domain_name => '1' > 11:23:18 NetworkManager[1198]: <info> [1583144598.7219] dhcp4 (wlp3s0): option requested_domain_name_servers => '1' > 11:23:18 NetworkManager[1198]: <info> [1583144598.7219] dhcp4 (wlp3s0): option requested_domain_search => '1' > 11:23:18 NetworkManager[1198]: <info> [1583144598.7220] dhcp4 (wlp3s0): option requested_host_name => '1' > 11:23:18 NetworkManager[1198]: <info> [1583144598.7220] dhcp4 (wlp3s0): option requested_interface_mtu => '1' > 11:23:18 NetworkManager[1198]: <info> [1583144598.7220] dhcp4 (wlp3s0): option requested_ms_classless_static_routes => '1' > 11:23:18 NetworkManager[1198]: <info> [1583144598.7220] dhcp4 (wlp3s0): option requested_nis_domain => '1' > 11:23:18 NetworkManager[1198]: <info> [1583144598.7220] dhcp4 (wlp3s0): option requested_nis_servers => '1' > 11:23:18 NetworkManager[1198]: <info> [1583144598.7221] dhcp4 (wlp3s0): option requested_ntp_servers => '1' > 11:23:18 NetworkManager[1198]: <info> [1583144598.7221] dhcp4 (wlp3s0): option requested_rfc3442_classless_static_routes => '1' > 11:23:18 NetworkManager[1198]: <info> [1583144598.7221] dhcp4 (wlp3s0): option requested_root_path => '1' > 11:23:18 NetworkManager[1198]: <info> [1583144598.7221] dhcp4 (wlp3s0): option requested_routers => '1' > 11:23:18 NetworkManager[1198]: <info> [1583144598.7221] dhcp4 (wlp3s0): option requested_static_routes => '1' > 11:23:18 NetworkManager[1198]: <info> [1583144598.7221] dhcp4 (wlp3s0): option requested_subnet_mask => '1' > 11:23:18 NetworkManager[1198]: <info> [1583144598.7222] dhcp4 (wlp3s0): option requested_time_offset => '1' > 11:23:18 NetworkManager[1198]: <info> [1583144598.7222] dhcp4 (wlp3s0): option requested_wpad => '1' > 11:23:18 NetworkManager[1198]: <info> [1583144598.7222] dhcp4 (wlp3s0): option routers => '10.SCRAMBLED' > 11:23:18 NetworkManager[1198]: <info> [1583144598.7222] dhcp4 (wlp3s0): option subnet_mask => '255.255.252.0' > 11:23:18 NetworkManager[1198]: <info> [1583144598.7222] dhcp4 (wlp3s0): state changed unknown -> bound > 11:23:18 NetworkManager[1198]: <info> [1583144598.7255] device (wlp3s0): state change: ip-config -> ip-check (reason 'none', sys-iface-state: 'managed') > 11:23:18 NetworkManager[1198]: <info> [1583144598.7285] device (wlp3s0): state change: ip-check -> secondaries (reason 'none', sys-iface-state: 'managed') > 11:23:18 NetworkManager[1198]: <info> [1583144598.7289] device (wlp3s0): state change: secondaries -> activated (reason 'none', sys-iface-state: 'managed') > 11:23:18 NetworkManager[1198]: <info> [1583144598.7296] manager: NetworkManager state is now CONNECTED_LOCAL > 11:23:18 NetworkManager[1198]: <info> [1583144598.7328] manager: NetworkManager state is now CONNECTED_SITE > 11:23:18 NetworkManager[1198]: <info> [1583144598.7330] policy: set 'Red Hat Guest' (wlp3s0) as default for IPv4 routing and DNS > 11:23:18 NetworkManager[1198]: <info> [1583144598.7430] device (wlp3s0): Activation: successful, device activated. > 11:23:18 NetworkManager[1198]: <info> [1583144598.7439] manager: NetworkManager state is now CONNECTED_GLOBAL > 11:23:19 systemd-networkd[1199]: wlp3s0: Gained IPv6LL > 11:30:12 systemd[1]: Starting system activity accounting tool... > 11:30:12 systemd[1]: sysstat-collect.service: Succeeded. > 11:30:12 systemd[1]: Started system activity accounting tool. ^ ta-da, validates that indeed, disk wasn't full at this point As you can see, there was no explicit complaint -- or even failure -- reported about this peculiar situation. I am not sure which component does the resolve.conf file modification, would suspect dhcpd. Perhaps, it shall better indicate it could not fulfill its job end-to-end?
(and thanks for understanding my s/resolv.conf/resolve.conf/ typos, too bad it sounds the same in my inner hint helper)
True to be told, there was also: > 11:05:12 nm-dispatcher[3305319]: /etc/dhcp/dhclient.d/chrony.sh: line 9: /var/lib/dhclient/chrony.servers.wlp3s0: No space left on device > 11:16:05 nm-dispatcher[3305800]: /etc/dhcp/dhclient.d/chrony.sh: line 9: /var/lib/dhclient/chrony.servers.wlp3s0: No space left on device > 11:19:35 nm-dispatcher[3305972]: /etc/dhcp/dhclient.d/chrony.sh: line 9: /var/lib/dhclient/chrony.servers.wlp3s0: No space left on device I should have used -u 'NetworkManager*' to capture also NetworkManager-dispatcher.service. Still, time synchronization related handling != setting resolv.conf.
Interesting pointer in NetworkManager.conf(5): https://github.com/NetworkManager/NetworkManager/blob/3719c0fe84847dc52490832f54f1d4aaed3bace1/man/NetworkManager.conf.xml#L348-L382 FTR: # NetworkManager --print-config > # NetworkManager configuration: /etc/NetworkManager/NetworkManager.conf > > [main] > # plugins=ifcfg-rh > # rc-manager=symlink > # auth-polkit=true > # dhcp=internal > > [logging] > # backend=journal > # audit=false > > # no-auto-default file "/var/lib/NetworkManager/no-auto-default.state" # ls -l /etc/resolv.conf > -rw-r--r--. 1 root root 289 Mar 4 11:02 /etc/resolv.conf So it appears it would be sufficient if some automatic power would just preconfigure /etc/resolv.conf to be the symlink to /run/NetworkManager/resolv.conf. Then, this problem would not, out-of-the-box (system should behave reasonably from get go, shouldn't?), exist anymore, to begin with. Correct? I may be overlooking another can of worms, though. Of course, in case of lacking operation memory (limiting tmpfs growth), you already have a bigger problem. Note, I am not sure what's the situation with the freshly installed system -- may be guilty of editing resolv.conf by hand some years back (turning it back from symlink accidentally) and in that case, it would be handy to have something like symlink-override mode that would not hesitate to enforce symlink scheme. Just an idea.
> So it appears it would be sufficient if some automatic power would just preconfigure /etc/resolv.conf to be the symlink to /run/NetworkManager/resolv.conf. Yes. I meant that earlier with "Do `ln -snf /var/run/NetworkManager/resolv.conf /etc/resolv.conf` and NM won't touch that file anymore.". The admin (or some setup tool of the distribution) indicates the preference by setting the symlink. That "automatic power" is never NetworkManager but the admin. > be handy to have something like symlink-override mode that would not hesitate to enforce symlink scheme. Not sure what you mean. - there is "rc-manager=file" setting in NetworkManager.conf which always writes to the file (following symlinks). Opt in to that, if you want that behaviour. - with "rc-manager=symlink" (the default), NM will rewrite /etc/resolv.conf, if (and only if) it is a file. If resolv.conf is a symlink, NM leaves it alone. I think this mode is the best by default, because the admin can make /etc/resolv.conf a file or a symlink, depending on what she wants. With https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/commit/324f67956aee7f5408da99ac4f10616d356bf4d5 , there should be a message logged about failure to write /etc/resolv.conf. Which should solve this bug.
Vaguely, looks about right, thanks! Looks some verbose messaging interface to deliver warnings collected along such activation and presented in nmtui or graphical tools/widgets is not currently possible, or is it? Any plans? Re "educated opt-in" configuration ... indeed, would be nice if people "were afforted the luxury" of knowing important components in bigger detail, but it is not that realistic, I am afraid, especially if networking normally JustWorks(TM), also courtesy of NetworkManager, but perhaps this is something distro might do (the symlinking so as to enforce low-space-proof procedure) ... perhaps via tmpfiles.d(5)? We shall see if there's any change of the landscape around systemd.
> Looks some verbose messaging interface to deliver warnings collected along such activation and presented in nmtui or graphical tools/widgets is not currently possible, or is it? Any plans? No plans. Patch welcome. But it's not easy, because you might have any number of clients. How would a client know whether a notification is relevant and how to handle it best? Then, you need to solve translation/i18n too. > We shall see if there's any change of the landscape around systemd. https://pagure.io/fedora-workstation/issue/123#comment-625579 It's not a change "around systemd". It's merely to use systemd-resolved in Fedora (by default), which solves these problems.
> No plans. Patch welcome. With absence of actual info-message flow infrastructure I solicited pointers about, it would be really a non-local change to wire it in especially with multiple parties involved, just as you point out, i.e., non-starter for rather a bystander like me. (Otherwise, sure I'd try to express what I would have appreciated to see at the time of triggering the problem. Maybe there will be a heavier argument for such tell-user infra in the future -- my use case doesn't qualify for any prioritizing I figure.) > But it's not easy, because you might have any number of clients. > How would a client know whether a notification is relevant and > how to handle it best? If there's a protocol delving beyond exit statuses or other sort of flat or even binary values, like XML/json/binary structs (DBus structured messages, or Protobuffers and friends), then it might be natural to have request-response (asynchronous if somehow pairable later on), and presumably only the requestor would be interested in those extra data collected when the request was worked on. But sorry for mentioning (likely) obvious, I know that doesn't help if the architecture is radically different. > https://pagure.io/fedora-workstation/issue/123#comment-625579 Thanks for the reference. Also, beside low-space scenarios, it'd help with some paths (like /etc) being intentionally locked down (in fact, arguments can be made that potentially highly dynamic and non-durable configurations, like DNS to query depending what AP/router you happen to be connected to, do not naturally belong to that persistence-needed, static hierarchy). * * * I think this bug can be closed then. Thanks again. If anything else in the future feels like hazarding silent malfunction in low-disk scenario (like missing a write), verbose logging is what makes this subtle difference between pulling one's hair out and timely containing the root cause.
This bug appears to have been reported against 'rawhide' during the Fedora 33 development cycle. Changing version to 33.
This message is a reminder that Fedora 33 is nearing its end of life. Fedora will stop maintaining and issuing updates for Fedora 33 on 2021-11-30. It is Fedora's policy to close all bug reports from releases that are no longer maintained. At that time this bug will be closed as EOL if it remains open with a Fedora 'version' of '33'. Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, simply change the 'version' to a later Fedora version. Thank you for reporting this issue and we are sorry that we were not able to fix it before Fedora 33 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora, you are encouraged change the 'version' to a later Fedora version prior this bug is closed as described in the policy above. Although we aim to fix as many bugs as possible during every release's lifetime, sometimes those efforts are overtaken by events. Often a more recent Fedora release includes newer upstream software that fixes bugs or makes them obsolete.
Fedora 33 changed to end-of-life (EOL) status on 2021-11-30. Fedora 33 is no longer maintained, which means that it will not receive any further security or bug fix updates. As a result we are closing this bug. If you can reproduce this bug against a currently maintained version of Fedora please feel free to reopen this bug against that version. If you are unable to reopen this bug, please file a new report against the current release. If you experience problems, please add a comment to this bug. Thank you for reporting this bug and we are sorry it could not be fixed.