I recently used cpio to create and restore an archive of my root filesystem. When I restored, the files were effectively "scrambled" - many files were incorrectly hard-linked. This particularly affected several classes of files: - many .pyc/.pyo files - much of the yumdb consisted of hard links to incorrect files - many files under /usr/src/kernels were hard links to Python bytecode files A few other random files were incorrectly hard-linked as well. The archive was created with cpio-2.11-12.fc18.x86_64. I initially extracted it using the version of cpio included in Finnix; after recovering my system to a usable state, I re-extracted the archive in a scratch directory using Fedora's cpio (same version as used for creation) and confirmed that the same symptoms appear. Given the symptoms, it appears to me like something in either the archive creation or restoration code confused an inode lookup table used to identify hard links.
Thanks for report, are there any special options used for archive creation/restoration? Some special filesystem? Are you able to reproduce it even with some older version of cpio, or you just spotted the issue recently? Are you able to reproduce the same in smaller scale? IOW, do you have some smaller reproducer? TIA.
No special options - just cpio -o -0, as I recall. I've only done large backups/restores like this with cpio recently, so I don't have any data on previous versions. I do not yet have a smaller-scale reproducer. Also, I no longer have the file system from which the CPIO archive was created, so I cannot inspect it for anomalies. I will try this week to reproduce on a smaller scale, and to inspect the CPIO archive itself to see what insight it might shed on the problem.
Hi Michael, thanks again for the report. I can confirm this, you don't have to waste a time with finding a reproducer - here is one: $ echo -e '/usr/src/kernels/3.8.5-201.fc18.x86_64/include/uapi/linux/lp.h\n/usr/lib64/pygtk/2.0/demos/panes.pyc' \ | cpio -o > /tmp/test.cpio $ cd /tmp && cat test.cpio | cpio -i -d --no-absolute-filenames I am going to try to find the reason and fix it. I still haven't looked at the details but it seems that (even if other 'scpio' and 'bsdcpio' are unable to unpack correctly) the file contents are stored OK in the archive. So there could exist some workaround to get your data back. I will look at it. I would suggest (before we get it fixed) not to use the default ('bin') archive format — use rather ustar (-H ustar) if any because it is not affected, has more possibilities and should more portable. I would also encourage you to use 'tar' for your backups as it supports 'pax' archive type and is able to store extended attributes, etc. Pavel
Scratch my reproducer please, sorry for that. I was curious what has these files (on my computer) in common — and it is this: inode(file1) % (2^16) == inode(file2) % (2^16) The problem: cpio 'bin' header format has defined only 2B size for inode number (max is 65536). So the stored inode number is inumber & 0xFF. This problem was taken into account several times before — here in Red Hat bugzilla and even upstream. Some time before, it was easier to detect, because the -Wtruncate was enabled by default [1]. There was proposed some unaccepted not-good enough improvement [2], probably a little bit better solution has 'star' which re-creates and stores new inode IDs from the number 1. But even this solution is *not* good enough these days according to present `df -i` output: $ df -i / Filesystem Inodes IUsed IFree IUse% Mounted on /dev/mapper/my-root 3276800 507791 2769009 16% / Simply - 'bin' archive has small header to be used as the right format for whole system backup and no worth to create patch can exist. So I guess this bug as-is is WONTFIX candidate. Probably some very verbose man page edit (info edit) would be useful (I am going to let this open) even if it is already mentioned - see the 'newc' format comment. ===================================================== Workaround (how to restore files from broken archive) ===================================================== Firsly, restoration is not automatic (and costs a lot of time) but it is not [DATA LOSS]. You have to find file you wasn't able to restore, put it's name into 'LIST' file and run: cat archive.bin.cpio | cpio -i -d -E FILE This will bypass the hardlink lookup... if the 'LIST' contains only files with _distinct_ stored inodes. The lucky feature of 'bin' archive format is that the hardlinked files are always completely dumped into archive... ------- Some other relevant links [3], [4], [5]. Pavel Some links are here [1], [2], [3], [4], [5] .. [1] http://lists.gnu.org/archive/html/bug-cpio/2004-09/msg00003.html [2] http://www.mail-archive.com/bug-cpio@gnu.org/msg00426.html [3] https://bugzilla.redhat.com/show_bug.cgi?id=23329 [4] https://www.redhat.com/archives/redhat-list/1999-February/msg00366.html [5] http://docs.freebsd.org/cgi/getmsg.cgi?fetch=345092+0+archive/1996/freebsd-questions/19961222.freebsd-questions
And the 'again', the 'tar --posix ...' command is definitelly better for whole system backup. > Some links are here [1], [2], [3], [4], [5] .. sorry for this big forgotten typo .. :) Pavel
This message is a reminder that Fedora 18 is nearing its end of life. Approximately 4 (four) weeks from now Fedora will stop maintaining and issuing updates for Fedora 18. It is Fedora's policy to close all bug reports from releases that are no longer maintained. At that time this bug will be closed as WONTFIX if it remains open with a Fedora 'version' of '18'. Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, simply change the 'version' to a later Fedora version prior to Fedora 18's end of life. Thank you for reporting this issue and we are sorry that we may not be able to fix it before Fedora 18 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora, you are encouraged change the 'version' to a later Fedora version prior to Fedora 18's end of life. Although we aim to fix as many bugs as possible during every release's lifetime, sometimes those efforts are overtaken by events. Often a more recent Fedora release includes newer upstream software that fixes bugs or makes them obsolete.
Fix for f20: https://lists.fedoraproject.org/pipermail/scm-commits/Week-of-Mon-20140407/1220842.html
cpio-2.11-25.fc20 has been submitted as an update for Fedora 20. https://admin.fedoraproject.org/updates/cpio-2.11-25.fc20
Package cpio-2.11-25.fc20: * should fix your issue, * was pushed to the Fedora 20 testing repository, * should be available at your local mirror within two days. Update it with: # su -c 'yum update --enablerepo=updates-testing cpio-2.11-25.fc20' as soon as you are able to. Please go to the following url: https://admin.fedoraproject.org/updates/FEDORA-2014-4921/cpio-2.11-25.fc20 then log in and leave karma (feedback).
cpio-2.11-25.fc20 has been pushed to the Fedora 20 stable repository. If problems still persist, please make note of it in this bug report.