Bug 1182297

Summary: nss-softokn prevents dracut from building the initrd
Product: Red Hat Enterprise Linux 6 Reporter: Fabian Deutsch <fdeutsch>
Component: nss-softoknAssignee: Elio Maldonado Batiz <emaldona>
Status: CLOSED CURRENTRELEASE QA Contact: BaseOS QE Security Team <qe-baseos-security>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 6.6CC: adevolder, bmcclain, chorn, deryni, dougsland, ekeck, harald, jjennings, jrieden, kkartikeya, ksrot, martin.wilck, moorereason, pasteur, rawson4, redhat, rrelyea, salmy, shamino, sinosuse, tlavigne, toracat, vcojot, vincent, ycui
Target Milestone: rcKeywords: ZStream
Target Release: 6.6   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: nss-softokn-3.14.3-22.el6 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1182725 (view as bug list) Environment:
Last Closed: 2015-10-22 09:34:15 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1164308, 1182725    

Description Fabian Deutsch 2015-01-14 19:55:18 UTC
Description of problem:
brew fails to create a livecd because nss-softokn is missing some file. See the trace below

Version-Release number of selected component (if applicable):
nss-softokn-3.14.3-19.el6_6

How reproducible:
Always

Steps to Reproduce:
1.
2.
3.

Actual results:
initrd was not build

Expected results:
initrd can be build

Additional info:
nss-softokn-3.14.3-20.el6_6 is already available and fixes this issue.

The failed build:
http://download.devel.redhat.com/brewroot/work/tasks/2664/8532664/root.log

DEBUG util.py:257:  
  Installing: ql2100-firmware              ##################### [492/492]E: Failed to install /usr/lib64/libfreebl3.chk
DEBUG util.py:257:  mkinitrd failed
DEBUG util.py:257:  warning: %posttrans(kernel-2.6.32-504.3.3.el6.x86_64) scriptlet failed, exit status 1
DEBUG util.py:257:  error reading information on service mcelog: No such file or directory
DEBUG util.py:257:   
DEBUG util.py:257:  Traceback (most recent call last):
DEBUG util.py:257:    File "/usr/bin/livecd-creator", line 150, in <module>
DEBUG util.py:257:      sys.exit(main())
DEBUG util.py:257:    File "/usr/bin/livecd-creator", line 134, in main
DEBUG util.py:257:      creator.configure()
DEBUG util.py:257:    File "/usr/lib/python2.6/site-packages/imgcreate/creator.py", line 740, in configure
DEBUG util.py:257:      self._create_bootconfig()
DEBUG util.py:257:    File "/usr/lib/python2.6/site-packages/imgcreate/live.py", line 215, in _create_bootconfig
DEBUG util.py:257:      self._configure_bootloader(self.__ensure_isodir())
DEBUG util.py:257:    File "/usr/lib/python2.6/site-packages/imgcreate/live.py", line 711, in _configure_bootloader
DEBUG util.py:257:      self._configure_syslinux_bootloader(isodir)
DEBUG util.py:257:    File "/usr/lib/python2.6/site-packages/imgcreate/live.py", line 611, in _configure_syslinux_bootloader
DEBUG util.py:257:      cfg += self.__get_image_stanzas(isodir)
DEBUG util.py:257:    File "/usr/lib/python2.6/site-packages/imgcreate/live.py", line 510, in __get_image_stanzas
DEBUG util.py:257:      (is_xen, isDracut) = self.__copy_kernel_and_initramfs(isodir, version, index)
DEBUG util.py:257:    File "/usr/lib/python2.6/site-packages/imgcreate/live.py", line 432, in __copy_kernel_and_initramfs
DEBUG util.py:257:      isodir + "/isolinux/initrd" + index + ".img")
DEBUG util.py:257:    File "/usr/lib64/python2.6/shutil.py", line 50, in copyfile
DEBUG util.py:257:      with open(src, 'rb') as fsrc:
DEBUG util.py:257:  IOError: [Errno 2] No such file or directory: '/var/tmp/imgcreate-ed985P/install_root/boot/initrd-2.6.32-504.3.3.el6.x86_64.img'

Comment 3 Bob Relyea 2015-01-14 21:37:40 UTC
This has been fixed in the -20 build in the spec file by generating a .chk file for libfreebl3.so and including it in the package.

Comment 7 Vincent S. Cojot 2015-01-20 17:39:49 UTC
The package still isn't released on RHN.
I worked around the issue by doing this:

# dracut -f
E: Failed to install /usr/lib64/libfreebl3.chk

# cd /usr/lib64/

# /usr/lib64/nss/unsupported-tools/shlibsign -v -i libfreebl3.so
moduleSpec configdir='' certPrefix='' keyPrefix='' secmod='' flags=noCertDB, noModDB
Generate a DSA key pair ...
Library File: ../../lib64/libfreebl3.so 12776 bytes
Check File: ../../lib64/libfreebl3.chk
Link: libfreebl3.chk
  hash: 32 bytes
[....]

This allowed dracut to run..

Comment 8 Akemi Yagi 2015-01-21 09:20:21 UTC
Please make Red Hat Knowledge Base (Solution) 1323013 accessible (public).

Comment 9 Martin Wilck 2015-01-28 09:18:44 UTC
Two weeeks have passed since Fabian wrote "nss-softokn-3.14.3-20.el6_6 is already available and fixes this issue.", but that package is still not available on RHN and I couldn't find a KB article, either.

Is there a workaround for the meantime, e.g. downgrading or uninstalling nss-softokn?

Comment 10 Christian Horn 2015-01-28 09:44:48 UTC
https://access.redhat.com/solutions/1323013 is now public.

nss-softokn-3.14.3-20.el6_6 is an internal build from the tree aimed at 6.7. 

In bz1182725 the 6.6.z fix is on its way.  We have hit an issue there the last days and the bz is again in ASSIGNED.

Engineering: any ideas for workarounds are welcome.

Comment 11 jared jennings 2015-01-28 23:04:36 UTC
bz1186549 has the same cause and solution, although it doesn't involve a live CD.

Comment 12 David Charlap 2015-01-29 16:07:37 UTC
This problem has just become critical as of the last rounds of updates.  It is now creating systems that won't boot.

On January 14, nss-softokn-3.14.3-19.el6_6 was installed via a "yum update".

On January 28, the kernel updated to version 2.6.32-504.8.1.el6.  As a part of the kernel update, dracut failed (because of the above bug.)  There was, therefore, no initramfs file created.  The new lines added to /boot/grub/grub.conf therefore had no initrd= line.  As you might expect, when trying to boot the new kernel, I got the pleasure of seeing the following:

    VFS: Cannot open root device "UUID=...." or unknown-block(0,0)
    Please append a correct "root=" boot option;...
    Kernel panic - not syncing: VFS: Unable to moutn root fs ...
    ...

I could boot to an older kernel, but (since no packages updated since then), "yum update" did nothing and there was no other obvious fix for the problem.

Fortunately, I noticed the missing initrd= lines in grub and the missing initramfs file, and a Google search led me here, where I found the workaround in comment 7 (above) which worked.  I was able to manually create the initramfs, add the initrd= line to grub and boot into the new kernel.

I'm sure I'm not the only customer to encounter this.  I think your cleanup work has just gotten more complicated than just pushing out an updated version of nss-softokn, since many of those customers are not going to be able to manually fix the damage.  I suspect many will simply conclude that the 2.6.32-504.8.1.el6 kernel is simply defective and will be stuck using an older kernel until the next kernel is released.

Comment 13 Vincent S. Cojot 2015-01-29 16:15:53 UTC
I ran into this a few days ago too. I'm sure this will brick some customer's machines (please get the darn fix out to RHN).

I worked around this issue by doing this:
# rpm -e dracut-fips-004-356.el6.noarch dracut-fips-aesni-004-356.el6.noarch
# dracut -f
# reboot

This disables some fips so I don't think this is an acceptable workaround.

My 2c,

Vincent

Comment 14 Saihua Shi 2015-01-30 02:04:39 UTC
Hi David,

   I think workaround from @Vincent is better.

   Personally I downgrade those sfotokn* packages to 3.14.3-18 manually, and comment out in grub excluded line.  As -19 pkg is not a serious security fix, I'm waiting for the -20 version.

   FYI.

-suse

http://www.shisaihua.com/dont-update-nss-softokn-freebl-3-14-3-19-el6_6-x86_64/


(In reply to David Charlap from comment #12)
> This problem has just become critical as of the last rounds of updates.  It
> is now creating systems that won't boot.
> 
> On January 14, nss-softokn-3.14.3-19.el6_6 was installed via a "yum update".
> 
> On January 28, the kernel updated to version 2.6.32-504.8.1.el6.  As a part
> of the kernel update, dracut failed (because of the above bug.)  There was,
> therefore, no initramfs file created.  The new lines added to
> /boot/grub/grub.conf therefore had no initrd= line.  As you might expect,
> when trying to boot the new kernel, I got the pleasure of seeing the
> following:
> 
>     VFS: Cannot open root device "UUID=...." or unknown-block(0,0)
>     Please append a correct "root=" boot option;...
>     Kernel panic - not syncing: VFS: Unable to moutn root fs ...
>     ...
> 
> I could boot to an older kernel, but (since no packages updated since then),
> "yum update" did nothing and there was no other obvious fix for the problem.
> 
> Fortunately, I noticed the missing initrd= lines in grub and the missing
> initramfs file, and a Google search led me here, where I found the
> workaround in comment 7 (above) which worked.  I was able to manually create
> the initramfs, add the initrd= line to grub and boot into the new kernel.
> 
> I'm sure I'm not the only customer to encounter this.  I think your cleanup
> work has just gotten more complicated than just pushing out an updated
> version of nss-softokn, since many of those customers are not going to be
> able to manually fix the damage.  I suspect many will simply conclude that
> the 2.6.32-504.8.1.el6 kernel is simply defective and will be stuck using an
> older kernel until the next kernel is released.

Comment 15 Vincent S. Cojot 2015-01-30 02:57:36 UTC
FFI, just saw that -22 was released:

vcs15:   nss-softokn.i686 0:3.14.3-22.el6_6
vcs15:   nss-softokn.x86_64 0:3.14.3-22.el6_6
vcs15:   nss-softokn-devel.i686 0:3.14.3-22.el6_6
vcs15:   nss-softokn-devel.x86_64 0:3.14.3-22.el6_6
vcs15:   nss-softokn-freebl.i686 0:3.14.3-22.el6_6
vcs15:   nss-softokn-freebl.x86_64 0:3.14.3-22.el6_6
vcs15:   nss-softokn-freebl-devel.i686 0:3.14.3-22.el6_6
vcs15:   nss-softokn-freebl-devel.x86_64 0:3.14.3-22.el6_6

I do not know if it solves that issue but I think I am going to keep dracut-fips disabled.. :)

Vincent

Comment 16 Christian Horn 2015-01-30 08:33:22 UTC
The 6.6.z errata for this issue has been released; https://rhn.redhat.com/errata/RHBA-2015-0110.html .

This bugzilla is for the fix in the next minor version.

Comment 17 Harald Hoyer 2015-03-03 09:31:37 UTC
what about libsoftokn3.so and libsoftokn3.chk in /usr/share/dracut/modules.d/05nss-softokn/install ??

Comment 18 Christian Horn 2015-07-24 10:14:19 UTC
Hm..
- this bz got not CLOSED ERRATA, state is MODIFIED
- all ACKs are set
- we have "fixed in: nss-softokn-3.14.3-22.el6" which is what is shipped in 6.7GA
- changelog of nss-softokn-3.14.3-22.el6_6.x86_64 does not mention the bz though

Comment 19 Elio Maldonado Batiz 2015-10-09 18:22:37 UTC
*** Bug 1186549 has been marked as a duplicate of this bug. ***