Bug 1285974
| Summary: | Network crashes when updating systemd-208-20.el7.x86_64 and initscripts-9.49.24-1.el7.x86_64 to systemd-219-19.el7.x86_64 and initscripts-9.49.30-1.el7.x86_64 | ||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 7 | Reporter: | Ricardo Schmidt <ricardo.xmit> | ||||||||||||||
| Component: | glib2 | Assignee: | Colin Walters <walters> | ||||||||||||||
| Status: | CLOSED WONTFIX | QA Contact: | Desktop QE <desktop-qa-list> | ||||||||||||||
| Severity: | urgent | Docs Contact: | |||||||||||||||
| Priority: | urgent | ||||||||||||||||
| Version: | 7.0 | CC: | aloughla, bgalvani, dcbw, lnykryn, lrintel, msekleta, ptalbert, ricardo.xmit, rkhan, systemd-maint-list, thaller, udev-maint-list | ||||||||||||||
| Target Milestone: | rc | ||||||||||||||||
| Target Release: | --- | ||||||||||||||||
| Hardware: | x86_64 | ||||||||||||||||
| OS: | Linux | ||||||||||||||||
| Whiteboard: | |||||||||||||||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |||||||||||||||
| Doc Text: | Story Points: | --- | |||||||||||||||
| Clone Of: | Environment: | ||||||||||||||||
| Last Closed: | 2020-12-15 07:38:49 UTC | Type: | Bug | ||||||||||||||
| Regression: | --- | Mount Type: | --- | ||||||||||||||
| Documentation: | --- | CRM: | |||||||||||||||
| Verified Versions: | Category: | --- | |||||||||||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||||||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||||||||||
| Embargoed: | |||||||||||||||||
| Bug Depends On: | |||||||||||||||||
| Bug Blocks: | 1203710, 1420851 | ||||||||||||||||
| Attachments: |
|
||||||||||||||||
Created attachment 1099581 [details]
print screen showing box crashed
Can you please append logs from that machine? Yes, the logs are attached. Some more info: - The box lost network connection after 1 hour from the creation of it. The box becomes available again after rebooting it. - I reproduced this problem every time I had a clean Redhat 7.0 created from the official RedHat ami-83e08db9 + yum update systemd initscript. - The AMI is from the region ap-southeast-2 (Sydney) - I am sending 2 logs: one is system log created by aws and the other one is the /var/log/messages You can notice in /var/log/messages that after updating systemd, the box start giving some errors: Nov 26 23:56:34 localhost systemd-logind: Failed to mount per-user tmpfs directory /run/user/1000: Permission denied Nov 26 23:56:34 localhost systemd-logind: Failed to open /dev/shm: Permission denied Nov 26 23:56:34 localhost systemd-logind: Failed to open /dev/mqueue: Permission denied Nov 27 00:01:01 localhost systemd-logind: Failed to mount per-user tmpfs directory /run/user/0: Permission denied Created attachment 1100319 [details]
var/log/messages
Created attachment 1100320 [details]
aws-system-log
From what I've seen in the logs I'd say that we probably have two separate issues here. First, network connection to instance is lost because dhclient spawned by NetworkManager dies from some reason during package update and can't renew dhcp lease. IIRC, this issue is fixed in NetworkManager as shipped with RHEL-7.1. Thus, before upgrading systemd and initscripts please upgrade NetworkManager first. As for permissions denied errors, those maybe caused by SELinux. Also update SELinux policy to the latest version before updating systemd. Hi Michal, Thanks for the help with this problem. In our last interaction, you suggested me to update networkmanager (dracut-network) and selinux before updating systemd. The problem is dracut-network has a dependency with systemd, it means, systemd gets updated when we do a yum update dracut-network. Anyway, on 2/12/2015 I created a clean redhat 7.0 using the official ami from redhat (ami-83e08db9), I updated dracut-network (and yum updated systemd too due the dependency) and then the server crashed after 10 minutes (it lost network connection). Today I tried to do the same process as I did on 2/12/2015 and the box didn't crash. Checking the version of the packages I realised that Redhat did release an update for dracut on 3/12/2015 - https://rhn.redhat.com/errata/RHBA-2015-2543.html The update fixes the previous bug: Updated dracut packages that fix one bug are now available for Red Hat Enterprise Linux 7. The dracut packages include an event-driven initial RAM file system (initramfs) generator infrastructure based on the udev device manager. The virtual file system, initramfs, is loaded together with the kernel at boot time and initializes the system, so it can read and boot from the root partition. This update fixes the following bug: * Previously, a dracut script in the initial RAM file system (initramfs) tried to stop the udev daemon by using the "udevadm control" command. The udev daemon then exited, but the systemd service has a policy to restart the daemon, which could, as a consequence, cause the system not to boot on rare occasions. To fix this bug, this update removes the code that attempted to stop the udev daemon from the dracut script. (BZ#1285686) Users of dracut are advised to upgrade to these updated packages, which fix this bug. https://rhn.redhat.com/errata/RHBA-2015-2543.html So I guess life is beautiful again. Thanks for the support, Ricardo Schmidt Hi guys, Don't consider my last update: the server crashed after 1 hour from the creation time. What I did: - created new ec2 server using redhat 7.0 (official ami from redhat ami-83e08db9, Sydney Region). - update selinux policy package (version selinux-policy-3.13.1-60.el7.noarch) - update dracut (version dracut-033-360.el7_2.x86_64) dracut has dependency with systemd, it means systemd got updated to a new version systemd-219-19.el7.x86_64 I am attaching the /var/log/messages of this box. Life still dark and sad :( Created attachment 1105382 [details]
new /var/log/messages
IIRC, in initramfs networking is handled by bash scripts and NetworkManager is not running there. Again, like I said in comment #6. Please explicitly update NetworkManager first and then update rest of the packages. Hi Michal, Thanks for your answer. The suggestion of updating NetworkManager doesn't work as this package has dependencies with systemd and other packages. To show that, I am attaching a new log from a box I created today with redhat 7.0. The only thing I did in this box was update NetworkManager. The box lost connection after 1 hour from the creating time. Created attachment 1108154 [details]
/var/log/messages file from a redhat 7.0
The is a log from a redhat 7.0. The only thing I did in this box was update networkManager.
Seems like we are still facing the same issue. In log I see these messages Dec 20 18:13:12 localhost NetworkManager[321]: <info> (eth0): DHCPv4 client pid 423 exited with status -1 Dec 20 18:13:12 localhost NetworkManager[321]: <warn> DHCP client died abnormally DHCP client died for some reason and NetworkManager didn't respawn it, thus are was no one to ask for renewal of DHCP lease hence network connection dropped after some time. Doesn't look like systemd bug to me. (In reply to Ricardo Schmidt from comment #12) > Created attachment 1108154 [details] > /var/log/messages file from a redhat 7.0 > > The is a log from a redhat 7.0. The only thing I did in this box was update > networkManager. DHCP works fine first (beginning of the log), but after the update, the packages seems to be broken: Dec 20 22:34:53 ip-172-31-19-46 systemd: Starting Network Manager... Dec 20 22:34:53 ip-172-31-19-46 NetworkManager: /usr/sbin/NetworkManager: symbol lookup error: /lib64/libgudev-1.0.so.0: undefined symbol: g_type_class_adjust_private_offset Dec 20 22:34:53 ip-172-31-19-46 systemd: NetworkManager.service: main process exited, code=exited, status=127/n/a Dec 20 22:34:53 ip-172-31-19-46 systemd: Failed to start Network Manager. (In reply to Jirka Klimes from comment #14) > (In reply to Ricardo Schmidt from comment #12) > > Created attachment 1108154 [details] > > /var/log/messages file from a redhat 7.0 > > > > The is a log from a redhat 7.0. The only thing I did in this box was update > > networkManager. > > DHCP works fine first (beginning of the log), but after the update, the > packages seems to be broken: > Dec 20 22:34:53 ip-172-31-19-46 systemd: Starting Network Manager... > Dec 20 22:34:53 ip-172-31-19-46 NetworkManager: /usr/sbin/NetworkManager: > symbol lookup error: /lib64/libgudev-1.0.so.0: undefined symbol: > g_type_class_adjust_private_offset I confirm this, an upgrade of glib2 is also required, probably there are some missing dependencies. Even after upgrading glib2, dhclient dies here while trying to write to stderr: write(2, "DHCPREQUEST on ens4 to 2.2.2.11 "..., 56) = -1 EPIPE (Broken pipe) --- SIGPIPE {si_signo=SIGPIPE, si_code=SI_USER, si_pid=14211, si_uid=0} --- +++ killed by SIGPIPE +++ I don't know how the upgrade of systemd is supposed to work, but could this be related to the restart of systemd/journald? This was fixed in RHEL 7.1 starting from NetworkManager-0.9.9.1-35.git20140326.4dba720.el7 (see bug 1136836). What do these commands give you? rpm -qv glib2 rpm -qv libgudev1 It appears the problem is that libgudev needs a different version of glib2 than you have installed; one that includes g_type_class_adjust_private_offset. g_type_class_adjust_private_offset was added to glib in the 7.1 glib-2.40 rebase, so likely a new libgudev1 was pulled in due to updates that was built against rhel 7.1's glib2, but the required glib2 is not available on the system. Over to udev; perhaps libgudev needs updated Requires: glib2 >= 2.40, though you'd think RPM would handle this somehow... It seems to me more like the glib problem. They obviously changed ABI but did not change the SONAME. *** Bug 1298329 has been marked as a duplicate of this bug. *** After evaluating this issue, there are no plans to address it further or fix it in an upcoming release. Therefore, it is being closed. If plans change such that this issue will be fixed in an upcoming release, then the bug can be reopened. |
Created attachment 1099580 [details] print screen with failed box Description of problem: Create an EC2 server using the office redhat AMI from REDHAT: RHEL-7.0_HVM_GA-20141017-x86_64-1-Hourly2-GP2 (ami-83e08db9). Update the version of systemd and initscripts. After 1 hour, the network becomes unreachable and AWS reports the box as Status Check Failed. Version-Release number of selected component (if applicable): Red Hat Enterprise Linux Server release 7.0 (Maipo) Linux ip-10-0-130-158.sfjhl.net 3.10.0-123.8.1.el7.x86_64 #1 SMP Mon Aug 11 13:37:49 EDT 2014 x86_64 x86_64 x86_64 GNU/Linux How reproducible: Create an ec2 box with the office redhat ami (redhat 7.0). AMI is ami-83e08db9. Steps to Reproduce: 1. yum update systemd initscripts (it updates from systemd-208-20.el7.x86_64 and initscripts-9.49.24-1.el7.x86_64 to systemd-219-19.el7.x86_64 and initscripts-9.49.30-1.el7.x86_64) 2. Wait 1 hour. The box will crash after it. Actual results: Expected results: Additional info: RedHat configuration: Red Hat Enterprise Linux Server release 7.0 (Maipo) Linux ip-10-0-130-158.sfjhl.net 3.10.0-123.8.1.el7.x86_64 #1 SMP Mon Aug 11 13:37:49 EDT 2014 x86_64 x86_64 x86_64 GNU/Linux Default packages: systemd-libs-208-11.el7_0.2.x86_64 systemd-208-11.el7_0.2.x86_64 systemd-sysv-208-11.el7_0.2.x86_64 initscripts-9.49.17-1.el7_0.1.x86_64 After yum update: systemd-libs-219-19.el7.x86_64 initscripts-9.49.30-1.el7.x86_64 systemd-sysv-219-19.el7.x86_64 systemd-219-19.el7.x86_64 ----------------- $ ssh 10.0.130.158 -l ec2-user The authenticity of host '10.0.130.158 (10.0.130.158)' can't be established. ECDSA key fingerprint is 2b:03:ac:fc:88:a2:58:fb:79:ff:20:86:36:92:13:ba. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added '10.0.130.158' (ECDSA) to the list of known hosts. [ec2-user@ip-10-0-130-158 ~]$ sudo su [root@ip-10-0-130-158 ec2-user]# uname -a Linux ip-10-0-130-158.sfjhl.net 3.10.0-123.8.1.el7.x86_64 #1 SMP Mon Aug 11 13:37:49 EDT 2014 x86_64 x86_64 x86_64 GNU/Linux [root@ip-10-0-130-158 ec2-user]# cat /etc/redhat-release Red Hat Enterprise Linux Server release 7.0 (Maipo) [root@ip-10-0-130-158 ec2-user]# rpm -qa | egrep 'systemd|initscript' systemd-libs-208-11.el7_0.2.x86_64 systemd-208-11.el7_0.2.x86_64 systemd-sysv-208-11.el7_0.2.x86_64 initscripts-9.49.17-1.el7_0.1.x86_64 [root@ip-10-0-130-158 ec2-user]# [root@ip-10-0-130-158 ec2-user]# yum update systemd initscript Loaded plugins: amazon-id, rhui-lb rhui-REGION-client-config-server-7 | 2.9 kB 00:00:00 rhui-REGION-rhel-server-releases | 4.0 kB 00:00:00 rhui-REGION-rhel-server-rh-common | 3.8 kB 00:00:00 (1/7): rhui-REGION-client-config-server-7/x86_64/primary_db | 5.4 kB 00:00:05 (2/7): rhui-REGION-rhel-server-releases/7Server/x86_64/group | 873 kB 00:00:05 (3/7): rhui-REGION-rhel-server-rh-common/7Server/x86_64/group | 104 B 00:00:05 (4/7): rhui-REGION-rhel-server-rh-common/7Server/x86_64/primary_db | 89 kB 00:00:00 (5/7): rhui-REGION-rhel-server-releases/7Server/x86_64/updateinfo | 2.8 MB 00:00:05 (6/7): rhui-REGION-rhel-server-releases/7Server/x86_64/primary_db | 20 MB 00:00:07 (7/7): rhui-REGION-rhel-server-rh-common/7Server/x86_64/updateinfo | 47 kB 00:00:05 No Match for argument: initscript No package initscript available. Resolving Dependencies --> Running transaction check ---> Package systemd.x86_64 0:208-11.el7_0.2 will be updated --> Processing Dependency: systemd = 208-11.el7_0.2 for package: systemd-sysv-208-11.el7_0.2.x86_64 --> Processing Dependency: systemd = 208-11.el7_0.2 for package: libgudev1-208-11.el7_0.2.x86_64 ---> Package systemd.x86_64 0:219-19.el7 will be an update --> Processing Dependency: systemd-libs = 219-19.el7 for package: systemd-219-19.el7.x86_64 --> Processing Dependency: kmod >= 18-4 for package: systemd-219-19.el7.x86_64 --> Running transaction check ---> Package kmod.x86_64 0:14-9.el7 will be updated ---> Package kmod.x86_64 0:20-5.el7 will be an update ---> Package libgudev1.x86_64 0:208-11.el7_0.2 will be updated ---> Package libgudev1.x86_64 0:219-19.el7 will be an update ---> Package systemd-libs.x86_64 0:208-11.el7_0.2 will be updated ---> Package systemd-libs.x86_64 0:219-19.el7 will be an update ---> Package systemd-sysv.x86_64 0:208-11.el7_0.2 will be updated ---> Package systemd-sysv.x86_64 0:219-19.el7 will be an update --> Processing Conflict: systemd-219-19.el7.x86_64 conflicts dracut < 033-243 --> Restarting Dependency Resolution with new changes. --> Running transaction check ---> Package dracut.x86_64 0:033-161.el7 will be updated --> Processing Dependency: dracut = 033-161.el7 for package: dracut-config-generic-033-161.el7.x86_64 --> Processing Dependency: dracut = 033-161.el7 for package: dracut-network-033-161.el7.x86_64 --> Processing Dependency: dracut = 033-161.el7 for package: dracut-config-rescue-033-161.el7.x86_64 ---> Package dracut.x86_64 0:033-359.el7 will be an update --> Running transaction check ---> Package dracut-config-generic.x86_64 0:033-161.el7 will be updated ---> Package dracut-config-generic.x86_64 0:033-359.el7 will be an update ---> Package dracut-config-rescue.x86_64 0:033-161.el7 will be updated ---> Package dracut-config-rescue.x86_64 0:033-359.el7 will be an update ---> Package dracut-network.x86_64 0:033-161.el7 will be updated ---> Package dracut-network.x86_64 0:033-359.el7 will be an update --> Processing Conflict: systemd-219-19.el7.x86_64 conflicts initscripts < 9.49.28-1 --> Restarting Dependency Resolution with new changes. --> Running transaction check ---> Package initscripts.x86_64 0:9.49.17-1.el7_0.1 will be updated ---> Package initscripts.x86_64 0:9.49.30-1.el7 will be an update --> Finished Dependency Resolution Dependencies Resolved =============================================================================================================================================================================== Package Arch Version Repository Size =============================================================================================================================================================================== Updating: dracut x86_64 033-359.el7 rhui-REGION-rhel-server-releases 311 k initscripts x86_64 9.49.30-1.el7 rhui-REGION-rhel-server-releases 429 k systemd x86_64 219-19.el7 rhui-REGION-rhel-server-releases 5.1 M Updating for dependencies: dracut-config-generic x86_64 033-359.el7 rhui-REGION-rhel-server-releases 48 k dracut-config-rescue x86_64 033-359.el7 rhui-REGION-rhel-server-releases 49 k dracut-network x86_64 033-359.el7 rhui-REGION-rhel-server-releases 90 k kmod x86_64 20-5.el7 rhui-REGION-rhel-server-releases 114 k libgudev1 x86_64 219-19.el7 rhui-REGION-rhel-server-releases 64 k systemd-libs x86_64 219-19.el7 rhui-REGION-rhel-server-releases 356 k systemd-sysv x86_64 219-19.el7 rhui-REGION-rhel-server-releases 52 k Transaction Summary =============================================================================================================================================================================== Upgrade 3 Packages (+7 Dependent packages) Total download size: 6.6 M Is this ok [y/d/N]: y Downloading packages: Delta RPMs disabled because /usr/bin/applydeltarpm not installed. (1/10): dracut-config-rescue-033-359.el7.x86_64.rpm | 49 kB 00:00:05 (2/10): dracut-033-359.el7.x86_64.rpm | 311 kB 00:00:05 (3/10): dracut-config-generic-033-359.el7.x86_64.rpm | 48 kB 00:00:05 (4/10): dracut-network-033-359.el7.x86_64.rpm | 90 kB 00:00:00 (5/10): initscripts-9.49.30-1.el7.x86_64.rpm | 429 kB 00:00:00 (6/10): kmod-20-5.el7.x86_64.rpm | 114 kB 00:00:00 (7/10): libgudev1-219-19.el7.x86_64.rpm | 64 kB 00:00:00 (8/10): systemd-libs-219-19.el7.x86_64.rpm | 356 kB 00:00:00 (9/10): systemd-sysv-219-19.el7.x86_64.rpm | 52 kB 00:00:00 (10/10): systemd-219-19.el7.x86_64.rpm | 5.1 MB 00:00:00 ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Total 1.1 MB/s | 6.6 MB 00:00:06 Running transaction check Running transaction test Transaction test succeeded Running transaction Updating : systemd-libs-219-19.el7.x86_64 1/20 Updating : systemd-219-19.el7.x86_64 2/20 warning: /etc/systemd/logind.conf created as /etc/systemd/logind.conf.rpmnew Updating : dracut-033-359.el7.x86_64 3/20 Updating : kmod-20-5.el7.x86_64 4/20 Updating : initscripts-9.49.30-1.el7.x86_64 5/20 Updating : dracut-network-033-359.el7.x86_64 6/20 Updating : dracut-config-rescue-033-359.el7.x86_64 7/20 Updating : dracut-config-generic-033-359.el7.x86_64 8/20 Updating : systemd-sysv-219-19.el7.x86_64 9/20 Updating : libgudev1-219-19.el7.x86_64 10/20 Cleanup : libgudev1-208-11.el7_0.2.x86_64 11/20 Cleanup : initscripts-9.49.17-1.el7_0.1.x86_64 12/20 Cleanup : dracut-config-generic-033-161.el7.x86_64 13/20 Cleanup : dracut-config-rescue-033-161.el7.x86_64 14/20 Cleanup : systemd-sysv-208-11.el7_0.2.x86_64 15/20 Cleanup : dracut-network-033-161.el7.x86_64 16/20 Cleanup : dracut-033-161.el7.x86_64 17/20 Cleanup : systemd-208-11.el7_0.2.x86_64 18/20 Cleanup : kmod-14-9.el7.x86_64 19/20 Cleanup : systemd-libs-208-11.el7_0.2.x86_64 20/20 Verifying : initscripts-9.49.30-1.el7.x86_64 1/20 Verifying : dracut-network-033-359.el7.x86_64 2/20 Verifying : kmod-20-5.el7.x86_64 3/20 Verifying : systemd-sysv-219-19.el7.x86_64 4/20 Verifying : libgudev1-219-19.el7.x86_64 5/20 Verifying : systemd-219-19.el7.x86_64 6/20 Verifying : dracut-config-rescue-033-359.el7.x86_64 7/20 Verifying : dracut-033-359.el7.x86_64 8/20 Verifying : dracut-config-generic-033-359.el7.x86_64 9/20 Verifying : systemd-libs-219-19.el7.x86_64 10/20 Verifying : dracut-config-rescue-033-161.el7.x86_64 11/20 Verifying : dracut-config-generic-033-161.el7.x86_64 12/20 Verifying : systemd-208-11.el7_0.2.x86_64 13/20 Verifying : systemd-sysv-208-11.el7_0.2.x86_64 14/20 Verifying : kmod-14-9.el7.x86_64 15/20 Verifying : libgudev1-208-11.el7_0.2.x86_64 16/20 Verifying : initscripts-9.49.17-1.el7_0.1.x86_64 17/20 Verifying : systemd-libs-208-11.el7_0.2.x86_64 18/20 Verifying : dracut-033-161.el7.x86_64 19/20 Verifying : dracut-network-033-161.el7.x86_64 20/20 Updated: dracut.x86_64 0:033-359.el7 initscripts.x86_64 0:9.49.30-1.el7 systemd.x86_64 0:219-19.el7 Dependency Updated: dracut-config-generic.x86_64 0:033-359.el7 dracut-config-rescue.x86_64 0:033-359.el7 dracut-network.x86_64 0:033-359.el7 kmod.x86_64 0:20-5.el7 libgudev1.x86_64 0:219-19.el7 systemd-libs.x86_64 0:219-19.el7 systemd-sysv.x86_64 0:219-19.el7 Complete! [root@ip-10-0-130-158 ec2-user]#