Bug 736318

We are no longer using mkinitrd, instead we are using dracut now. But this seems to be a problem with pre-generated initrd's moving this over to the installer, I'm sure they know which component this bug really belongs too :)

It's not corrupted, it's two concatenated cpio images:

  [xz-compressed dracut initramfs][uncompressed cpio with /squashfs.img]

Your xz tools don't know how to deal with this. xz 5.1.1alpha and higher can do:

( xz -dc --single-stream > initramfs.cpio && cat > install.cpio ) < initrd.img

It would really nice if xz was updated to support --single-stream. Right now there are no packages supporting it and lsinitrd fails no matter what I do...

For the curious, the squashfs.img of F16 alpha can be extracted with

(dd of=/dev/null bs=1 skip=$((0x2F3844)) count=0; cat > install.cpio) < /boot/initrd.img

Inside it you can find the ext4 image rootfs.img.

Moving to component xz, see comment 3.

People on fedora 15 might want to study the install environment due to the RAID issues that have been open since fedora 9 or so.
xz 5.1 on fedora 15 is therefor also welcomed.

(In reply to comment #2)
> It's not corrupted, it's two concatenated cpio images:
> 
>   [xz-compressed dracut initramfs][uncompressed cpio with /squashfs.img]
> 
> Your xz tools don't know how to deal with this. xz 5.1.1alpha and higher can
> do:
> 
> ( xz -dc --single-stream > initramfs.cpio && cat > install.cpio ) < initrd.img

It doesn't work for me. The second file ("install.cpio") is not recognized as a cpio archive, or as a direct squashfs image.

(In reply to comment #2)

> ( xz -dc --single-stream > initramfs.cpio && cat > install.cpio ) < initrd.img

I tried with xz-5.1.1alpha. xz seems to read in blocks of 4K, and with --single-stream it simply ignores the garbage in the most recently read block. However that won't help with the creation of "install.cpio", the leading parts of that are already wasted.

The initrd I'm trying to extract has md5 7055fca88953a498814ba0a3091f2ec4, size 134,748,640. When xz stops reading it in the quoted command, still 130,775,520 bytes are left to read for "cat". So the first xz stream should supposedly take 3,973,120 bytes. However trying to decompress that leading chunk without --single-stream alrady reports a corrupt file.

In reality the xz stream is only 3,970,528 bytes long, 2,592 bytes shorter than found with the above (binary search salutes you). "tail -c +$((3970528+1)) initrd.img >install.cpio" yields the second (uncompressed) cpio with the squashfs.img inside it.

- I'm not particularly entertained that I have to disassemble an initrd image manually with an upstream-only decompressor,
- so that I can have a chance at figuring out why F16 anaconda/grubby fails to install grub when run in a Xen PV guest:
  grubby fatal error: unable to find a suitable template
  grubby: doing this would leave no kernel entries. Not writing out new config.
(only getting to that error message took half a day on its own)
- so that I can test the backport I'm actually interested in with F16 (bug 742880).

(In reply to comment #8)
> So the first xz stream should supposedly take
> 3,973,120 bytes.

= 970 * 4096

While this bug is pending we cannot even look at the presented f16 initrd ourselves because someone decided  to use a version of xz that is not even in rawhide. What project organ is responsible for that?
Did they notice the impact they made? Did the evaluate the impact? Did they learn?

So effectively that decision is blocking the community effort to fix the RAID-related preupgrade issues. (e.g. https://bugzilla.redhat.com/show_bug.cgi?id=504826)

xz-5.1.1-1alpha.fc16 has been submitted as an update for Fedora 16.
https://admin.fedoraproject.org/updates/xz-5.1.1-1alpha.fc16

XZ is now updated to 5.1.1alpha in rawhide and this update for f16 is created:

https://admin.fedoraproject.org/updates/xz-5.1.1-1alpha.fc16

Please test it and give it positive karma if you want to see it in F16 stable.
(I've been using xz-5.1.1alpha for a couple of months not seeing any problems so far)

Here is changelog snippet:

5.1.1alpha (2011-04-12)

    * All fixes from 5.0.2

    * liblzma fixes that will also be included in 5.0.3:

        - A memory leak was fixed.

        - lzma_stream_buffer_encode() no longer creates an empty .xz
          Block if encoding an empty buffer. Such an empty Block with
          LZMA2 data would trigger a bug in 5.0.1 and older (see the
          first bullet point in 5.0.2 notes). When releasing 5.0.2,
          I thought that no encoder creates this kind of files but
          I was wrong.

        - Validate function arguments better in a few functions. Most
          importantly, specifying an unsupported integrity check to
          lzma_stream_buffer_encode() no longer creates a corrupt .xz
          file. Probably no application tries to do that, so this
          shouldn't be a big problem in practice.

        - Document that lzma_block_buffer_encode(),
          lzma_easy_buffer_encode(), lzma_stream_encoder(), and
          lzma_stream_buffer_encode() may return LZMA_UNSUPPORTED_CHECK.

        - The return values of the _memusage() functions are now
          documented better.

    * Support for multithreaded compression was added using the simplest
      method, which splits the input data into blocks and compresses
      them independently. Other methods will be added in the future.
      The current method has room for improvement, e.g. it is possible
      to reduce the memory usage.

    * Added the options --single-stream and --block-size=SIZE to xz.

    * xzdiff and xzgrep now support .lzo files if lzop is installed.
      The .tzo suffix is also recognized as a shorthand for .tar.lzo.

    * Support for short 8.3 filenames under DOS was added to xz. It is
      experimental and may change before it gets into a stable release.

Package xz-5.1.1-1alpha.fc16:
* should fix your issue,
* was pushed to the Fedora 16 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing xz-5.1.1-1alpha.fc16'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2011-14444
then log in and leave karma (feedback).

udo, this update has nothing to do with the way the anaconda initramfs is put together.  According to Laszlo, in fact, even --single-stream is actually not enough to separate the pieces of the initramfs.

For the F16 alpha, see comment 4.  ghex2 can help you figuring out a similar incantation for newer images.

Paolo? Has nothing to do?
I am looking for a way to have a look into the initrd for Fedora 16 because of the RAID issues, the /boot space issue etc.
If I still cannot unpack the initrd from Fedora 15, then what use is the update?
Of course I can open a separate bug but what would that help?
I just need a build of the rpm for f15 and a way to unpack the initrd on f15.

Has nothing todo with the way lorax puts together the initramfs, because lorax is _not_ using a newer xz.  It's just that newer xz provides (if Laszlo is wrong and Will is right) a way to deal with it easily.

The separate bug would have to be against xz (also upstream), I guess.

Have you seen comment 4?

Sure, but how would that work on any random rawhide/f16 initrd?
I need a general working method for unpacking.

Hello Udo,

the upstream repo for lorax is git://git.fedorahosted.org/git/lorax.git. (Don't ask how I found it, I don't know.)

"src/pylorax/installtree.py", method make_squashfs_runtime() says:

    def make_squashfs_runtime(self, runtime, kernel, type, args):
        """This is a little complicated, but dracut wants to find a squashfs
        image named "squashfs.img" which contains a filesystem image named
        "LiveOS/rootfs.img".
        Placing squashfs.img inside a cpio image and concatenating that
        with the existing initramfs.img will make squashfs.img appear inside
        initramfs at boot time.
        """

So, we have to look at dracut. The upstream repo for dracut is at git://dracut.git.sourceforge.net/gitroot/dracut/dracut. (Because... I have no idea.) ... I tried to look at dracut, and found nothing.

I think the decompression and the saving of the exact end of the xz stream happens in the kernel, "init/initramfs.c", unpack_to_rootfs(). It calls unxz() via a function pointer.

static unsigned my_inptr;   /* index of next byte to be processed in inbuf */

			res = decompress(buf, len, NULL, flush_buffer, NULL,
				   &my_inptr, error);


"lib/decompress_unxz.c":

/*
 * This function implements the API defined in <linux/decompress/generic.h>.
 *
 * This wrapper will automatically choose single-call or multi-call mode
 * of the native XZ decoder API. The single-call mode can be used only when
 * both input and output buffers are available as a single chunk, i.e. when
 * fill() and flush() won't be used.
 */
STATIC int INIT unxz(unsigned char *in, int in_size,
		     int (*fill)(void *dest, unsigned int size),
		     int (*flush)(void *src, unsigned int size),
		     unsigned char *out, int *in_used,
		     void (*error)(char *x))

(This one seems to return successfully when it hits XZ_STREAM_END; it doesn't check for trailing data.)


"include/linux/decompress/generic.h":

typedef int (*decompress_fn) (unsigned char *inbuf, int len,
			      int(*fill)(void*, unsigned int),
			      int(*flush)(void*, unsigned int),
			      unsigned char *outbuf,
			      int *posp,
			      void(*error)(char *x));

/* [...]
 *posp    - if non-null, input position (number of bytes read) will be
 *	  returned here
*/

Then in unpack_to_rootfs() the loop iterates to the next compressed image, using the value of "my_inptr".

So, unless there's already a command line utility that returns the exact byte position of the xz stream's end, someone will have to write it. The current decompression happens by direct use of the XZ API (... well, the kernel-internal version thereof).

I wonder though if passing --force to xz -dc would have any effect.

Thanks for explaining. It looks like this method of packing stuff might not yet be quite userfriendly, from reading the comment #18.

I just want to download a F16/rawhide initrd, unpack it and explore the install.img to see what I can do to circumvent the raid and /boot issues sustained by Fedora.

Package xz-5.1.1-1alpha.fc16, perl-Compress-Raw-Lzma-2.037-3.fc16:
* should fix your issue,
* was pushed to the Fedora 16 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing xz-5.1.1-1alpha.fc16 perl-Compress-Raw-Lzma-2.037-3.fc16'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2011-14444
then log in and leave karma (feedback).

xz-5.1.1-1alpha.fc16, perl-Compress-Raw-Lzma-2.037-3.fc16 has been pushed to the Fedora 16 stable repository.  If problems still persist, please make note of it in this bug report.

So how can I decompress the initrd on F15 and research the install.img therein?

(In reply to comment #23)
> So how can I decompress the initrd on F15 and research the install.img therein?

I on FC16 and trying to decompress its initrd with no success.

I am in same state as in comment 7.

> ( xz -dc --single-stream > initramfs.cpio && cat > install.cpio ) < initrd.img

It doesn't work for me. The second file ("install.cpio") is not recognized as a
cpio archive, or as a direct squashfs image.

Does anyone have the correct options to extract the install.cpio part of initrd.img?

Jeevan, use the latest xz version available in git, the one supplied in fc16 does not handle the streams properly (see http://git.tukaani.org/?p=xz.git;a=commit;h=7a480e485938884ef3021b48c3b0b9f9699dc9b6). With this patch you will get the install.cpio.

these concatenated initrds are quite hell to do anything with.

Is there a planning for the fix?
I mean: why use stuff to develop that is not distributed in a working form?
Why close this issue?
Why even accept FC16 in this state?

Jaroslav,

Following your link I did following
( xz -dc --single-stream > initramfs.cpio; xz -dc --single-stream > install.cpio) < initrd.img

It yields "xz: (stdin): File format not recognized".

I even tried it using the FC17/Rawhide XZ
xz-libs-5.1.1-1alpha.fc17.x86_64
xz-5.1.1-1alpha.fc17.x86_64

As mentioned in comment 8, FC16 initrd is a concatenation of "compressed initramfs.cpio" and "non-compressed install.cpio" streams. 

XZ cannot find the correct start of install.cpio in initrd.

Jeevan, use the example shown in comment 2 (i.e. no second xz but use cat instead, that was just what the author of the patch tested). The second part is indeed an uncompressed cpio archive.

Jaroslav, I originally did use the command in comment 2. I had not worked comment 24.

However following worked

Steps that worked for me to extract initrd on FC16

Start with following command
( xz -dc --single-stream > initramfs.cpio && cat > install.cpio ) < initrd.img

However this only yields initramfs.cpio (xz uncompresses it, so you can use it directly).

The install.cpio is extracted from incorrect location so its unusable.
[root@FC16-64-VM tmp]# file install.cpio
install.cpio: data
It should have read "ASCII cpio archive"

Refer to Comment 8 by Lazlo Ersek where it says xz is jumping about 2592 bytes.

Now you have to figure out how much it jumped in your case. The way to do this is as follows

1) First take a look at XZ format definition
http://www.tukaani.org/xz/xz-file-format.txt
XZ stream starts with ".7zXZ" and end with "YZ". This information will be used in next steps.

2) Get a hexdump of entire initrd.img
hexdump -C initrd.img > initrd.hex

3) Get a hexdump of incorrect install.cpio
hexdump -C install.cpio > install.hex

4) Look at the first line in install.cpio's hexdump
00000000  44 5d 53 ed b3 03 a9 04  62 ca 65 47 73 e8 89 1b  |D]S.....b.eGs...|
Note the ASCII string "b.eGs"

5) Find the pattern ".b.eGs" in initrd.img's hexdump (i.e. initrd.hex)
vi initrd.hex
search "\.b\.eGs"
It occurs at 0x0003b8000 (in my case)

6) Now start moving up in initrd.hex. At 0x0003b7f60 you see the XZ stream start marker ".7zXZ". You would think that's the right point to start, but no, you have to keep going up until you find end of squashfs.
You see that at 
003b78e0     30 30 73 71 75 61 73 68  66 73 2e 69 6d 67 00 00  |00squashfs.img..|

7) Now you need to find the start of this SquashFS. The way to find this is look for previous XZ streams end marker which is "YZ". You find that at 
003b7800  00 01 59 5a 30 37 30 37  30 31 30 30 45 35 34 32  |..YZ07070100E542|

8) So your install.cpio starts after this "YZ"
install.cpio starts at 3897349

tail -c +3897349 initrd.img > install.cpio
[root@FC16-64-VM tmp]# file install.cpio
install.cpio: ASCII cpio archive (SVR4 with no CRC)

As you can see this is quite painful, I wish there were instructions from the team that built initrd on this.

this is very painful, I stopped hunting the xz headers in the file and got new update from git which fixes the shift. That way I was able to get both the initramfs + install.cpio via the command in comment 2. 

I was using netboot.iso from FC16 iso directory. If you used something else, it may still be wrong, I have no idea :). Let me point out one more time that you have to download latest GIT version (i.e. NO alpha, NO RPM, but git clone http://git.tukaani.org/xz.git with autoconf, configure (I used --disable-shared to get static xz binary), make,...).

I'd also like to see initramdisk that is not composed like two concatenated images :( It was quite surprising for me to find that I have to use hex editor, latest pre-alpha version of xz and study the xz file format just to be able to make one small change in the supplied initrd.

Thanks Jaroslav.

Using the xz built from source got from git works.
It could successfully get initrd opened up.

Surprisingly the hand built xz's version and the FC17 alpha xz version is shown same

xz built from source
[root@FC16-64-VM xz]# ./xz -V
xz (XZ Utils) 5.1.1alpha
liblzma 5.1.1alpha

FC17 xz
[root@FC16-64-VM xz]# xz -V
xz (XZ Utils) 5.1.1alpha
liblzma 5.1.1alpha


But only the one built from source works.

Overall xz on FC16 is broken and so is the one in current iteration of FC17.

Shouldn't this be reopened since we haven't fixed the reporter's problem in a shipping update?

As I understand the git log, we're shipping version 5.1.1alpha which was tagged on 2011-04-12.

The patch needed to fix this problem is in commit 7a480e485938884ef3021b48c3b0b9f9699dc9b6 which shipped on 2011-05-01.  Presumably those constructing the f16 initramfs were using a git pull later than that and didn't notice.

To meet the Fedora Objective of being self-building and self-hosting (implies self-debugging) we should be shipping at least an xz-5.1.1-2.20110501git7a480e48 .

That's right Bill, let's reopen this :)

Why are the two images concatenated together anyway?  This sounds like a terrible idea to me.  If two images are necessary, there should be two separate files.

This message is a reminder that Fedora 16 is nearing its end of life.
Approximately 4 (four) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 16. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as WONTFIX if it remains open with a Fedora 
'version' of '16'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version prior to Fedora 16's end of life.

Bug Reporter: Thank you for reporting this issue and we are sorry that 
we may not be able to fix it before Fedora 16 is end of life. If you 
would still like to see this bug fixed and are able to reproduce it 
against a later version of Fedora, you are encouraged to click on 
"Clone This Bug" and open it against that version of Fedora.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

The process we are following is described here: 
http://fedoraproject.org/wiki/BugZappers/HouseKeeping

Fedora 16 changed to end-of-life (EOL) status on 2013-02-12. Fedora 16 is 
no longer maintained, which means that it will not receive any further 
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of 
Fedora please feel free to reopen this bug against that version.

Thank you for reporting this bug and we are sorry it could not be fixed.