Bug 1404435

Summary: "yum update" on fresh installation of RHEL 7.0 or 7.1 PPC64 (updating to 7.3) results in various yum errors
Product: Red Hat Enterprise Linux 7 Reporter: jcastran
Component: glibcAssignee: Florian Weimer <fweimer>
Status: CLOSED ERRATA QA Contact: Sergey Kolosov <skolosov>
Severity: high Docs Contact:
Priority: high    
Version: 7.0CC: ashankar, bgollahe, cdany2000, fweimer, james.antill, jcastran, jreznik, kmrstiko, mcermak, mdomonko, mnewsome, nobody+bgollahe, pfrankli, skolosov
Target Milestone: rc   
Target Release: ---   
Hardware: ppc64   
OS: Linux   
Whiteboard:
Fixed In Version: glibc-2.17-165.el7 Doc Type: Bug Fix
Doc Text:
Cause: During updates from earlier versions of Red Hat Enterprise Linux for POWER, the system glibc installationis temporarily in an inconsistent state. Consequence: RPM scriptlets fail. Fix: The power8 AT_PLATFORM directory is removed early during the update. Result: The glibc installation remains during the update, and RPM scriptlets run successfully. Harmless warnings will be printed during the update; see <https://access.redhat.com/articles/2931341>.
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-08-01 18:09:25 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1390370    
Attachments:
Description Flags
PPC64 7.0 and 7.1 yum update results.txt none

Description jcastran 2016-12-13 20:50:21 UTC
Created attachment 1231330 [details]
PPC64 7.0 and 7.1 yum update results.txt

Description of problem:
This is specifically for PPC64 (I have not had the time to test ppc64le). With a fresh installation of RHEL 7.0 or 7.1 PPC64, then registering with RHSM and running "yum update", the system will have various packages fail installation. 

Version-Release number of selected component (if applicable):
RHEL 7.0 PPC64
RHEL 7.1 PPC64

How reproducible:
every time

Steps to Reproduce:
1. Install RHEL 7.0/1 PPC64
2. Register
3. yum update

Actual results:
    System has various packages that error out with post or pre scriptlet warnings. End result is attached showing packages that installed, updated, or failed installation.

Expected results:
   System updates to 7.3 without issue.

Additional info:

Comment 5 Florian Weimer 2017-01-09 21:24:04 UTC
I can reproduce this.

The old glibc package had this:

/lib64/libc.so.6
/lib64/power8/libc.so.6

The new glibc package has just this:

/lib64/libc.so.6

The dynamic linker will prefer the “power8” platform directory with the old libc.so.6 file, which is incompatible with the new /lib64/ld64.so.1 file (which replaces the old file once the new glibc package installed, so it takes effect much earlier).

This will only happen on POWER8 machines.  On POWER7 machines, the directory is always ignored.

I think this is a fundamental incompatibility between RPM, ldconfig/ld.so and the way platform directories work.

Comment 6 Carlos O'Donell 2017-01-10 03:57:37 UTC
(In reply to Florian Weimer from comment #5)
> I can reproduce this.
> 
> The old glibc package had this:
> 
> /lib64/libc.so.6
> /lib64/power8/libc.so.6
> 
> The new glibc package has just this:
> 
> /lib64/libc.so.6
> 
> The dynamic linker will prefer the “power8” platform directory with the old
> libc.so.6 file, which is incompatible with the new /lib64/ld64.so.1 file
> (which replaces the old file once the new glibc package installed, so it
> takes effect much earlier).
> 
> This will only happen on POWER8 machines.  On POWER7 machines, the directory
> is always ignored.
> 
> I think this is a fundamental incompatibility between RPM, ldconfig/ld.so
> and the way platform directories work.

Interesting, I had not considered this failure scenario. It effectively means that platform directories can never be removed if you expect in-place upgrades to work robustly.

The easiest solution is to keep /lib64/power8 and just make it a symlink to the /lib64 versions. We can't change the dynamic linker behaviour because the point of the platform directories is to search them first.

Any reason why such a solution wouldn't solve this problem?

Comment 9 Florian Weimer 2017-01-10 12:20:05 UTC
(In reply to Carlos O'Donell from comment #6)
> Interesting, I had not considered this failure scenario. It effectively
> means that platform directories can never be removed if you expect in-place
> upgrades to work robustly.

Right.

> The easiest solution is to keep /lib64/power8 and just make it a symlink to
> the /lib64 versions. We can't change the dynamic linker behaviour because
> the point of the platform directories is to search them first.
> 
> Any reason why such a solution wouldn't solve this problem?

ldconfig might to something unexpected with the symbolic link.

This problem is *not* new.  There is code in releng/glibc_post_upgrade.c which deletes libraries in platform directories.  We just need to add the “power8” subdirectory there, I think.

This code is very old, and not adjusting it caused quite similar issues in the past:

* Tue May  3 2005 Jakub Jelinek <jakub> 2.3.5-5
…
- clean up /lib64/tls instead of /lib/tls on x86-64, s390x and
  ppc64 in glibc_post_upgrade (#156656)

* Fri Apr  5 2002 Jakub Jelinek <jakub> 2.2.5-31
…
- remove in glibc_post_upgrade on i386 all /lib/i686/libc-*.so,
  /lib/i686/libm-*.so and /lib/i686/libpthread-*.so, not just current
  version (#61633)

It's certainly hackish, but it's apparently quite reliable, so I'd suggest to stick with that instead of coming up with a new solution.

Comment 12 Florian Weimer 2017-01-10 12:53:50 UTC
Untested workaround:

rpm -ql glibc | grep /power8/ | xargs rm

After this command, the system will use the base versions of the libraries, but will continue working until the glibc update is concluded.

Comment 13 Sergey Kolosov 2017-01-10 15:50:27 UTC
> Untested workaround:

> rpm -ql glibc | grep /power8/ | xargs rm

> After this command, the system will use the base versions of the libraries, but > will continue working until the glibc update is concluded.

Tested on "POWER8" platforms, workaround works for both versions(7.0 and 7.1)

Comment 15 Florian Weimer 2017-02-10 18:41:36 UTC
I implemented the usual fix, but error messages remain during an upgrade:

  Updating   : glibc-devel-2.17-163.el7.bz1228114.1.ppc64                         6/11 
  Cleanup    : glibc-devel-2.17-78.el7.ppc64                                      7/11 
  Cleanup    : glibc-2.17-78.el7                                                  8/11 
warning: file /lib/power8/libthread_db.so.1: remove failed: No such file or directory
warning: file /lib/power8/libthread_db-1.0.so: remove failed: No such file or directory
warning: file /lib/power8/librt.so.1: remove failed: No such file or directory
warning: file /lib/power8/librt-2.17.so: remove failed: No such file or directory
warning: file /lib/power8/libpthread.so.0: remove failed: No such file or directory
warning: file /lib/power8/libpthread-2.17.so: remove failed: No such file or directory
warning: file /lib/power8/libm.so.6: remove failed: No such file or directory
warning: file /lib/power8/libm-2.17.so: remove failed: No such file or directory
warning: file /lib/power8/libc.so.6: remove failed: No such file or directory
warning: file /lib/power8/libc-2.17.so: remove failed: No such file or directory
  Cleanup    : glibc-headers-2.17-78.el7.ppc64                                    9/11 
  Cleanup    : glibc-2.17-78.el7                                                 10/11 
warning: file /lib64/power8/libthread_db.so.1: remove failed: No such file or directory
warning: file /lib64/power8/libthread_db-1.0.so: remove failed: No such file or directory
warning: file /lib64/power8/librt.so.1: remove failed: No such file or directory
warning: file /lib64/power8/librt-2.17.so: remove failed: No such file or directory
warning: file /lib64/power8/libpthread.so.0: remove failed: No such file or directory
warning: file /lib64/power8/libpthread-2.17.so: remove failed: No such file or directory
warning: file /lib64/power8/libm.so.6: remove failed: No such file or directory
warning: file /lib64/power8/libm-2.17.so: remove failed: No such file or directory
warning: file /lib64/power8/libc.so.6: remove failed: No such file or directory
warning: file /lib64/power8/libc-2.17.so: remove failed: No such file or directory
  Cleanup    : glibc-common-2.17-78.el7.ppc64                                    11/11 

The messages are harmless (they are just warnings), but I don't like them.  But on the other hand, I do not see any other solution here.

Comment 20 errata-xmlrpc 2017-08-01 18:09:25 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2017:1916