Bug 125646 - files restored from dump are corrupt
files restored from dump are corrupt
Status: CLOSED ERRATA
Product: Fedora
Classification: Fedora
Component: dump (Show other bugs)
2
i686 Linux
medium Severity high
: ---
: ---
Assigned To: Jindrich Novy
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2004-06-09 13:03 EDT by Aleksandar Milivojevic
Modified: 2013-07-02 19:00 EDT (History)
3 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2004-08-12 03:10:16 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Aleksandar Milivojevic 2004-06-09 13:03:27 EDT
From Bugzilla Helper:
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.6) Gecko/20040510

Description of problem:
When dumping only set of files (not entire partition), in same cases
files in dump archive are corrupted.  My colegue Don Tiessen
discovered this with dump-0.4b27-3 on Red Hat 7.3, and I confirmed and
did some additinal testing with dump-0.4b33-3 on Fedora Core 2.

The corruption is not reproducible always.  Sometimes (but only
sometimes) everything goes fine.  Sometimes some files are corrupted.

What I did was to set up directory /root/test, and copied current set
of official RPM updates for Fedora Core 2 as test files.  Than I did:

   # cd /root
   # dump -0 -f test.dump test
   # restore -ivf test1.dump
   # mv root root1
   # cd root1/test
   # for a in *; do diff $a /root/test/$a; done

I repeated this test three times (changing 1 to 2 and 3 in above
commands).  First and third time I ended up with some files corrupted,
and second time everything was correctly restored.  Dump and restore
did not print any error messages.

The results of one run:

Binary files ipsec-tools-0.2.5-2.i386.rpm and
/root/test/ipsec-tools-0.2.5-2.i386.rpm differ
Binary files kdelibs-3.2.2-6.i386.rpm and
/root/test/kdelibs-3.2.2-6.i386.rpm differ
Binary files kdelibs-devel-3.2.2-6.i386.rpm and
/root/test/kdelibs-devel-3.2.2-6.i386.rpm differ
Binary files subversion-perl-1.0.2-2.1.i386.rpm and
/root/test/subversion-perl-1.0.2-2.1.i386.rpm differ

The results of another run:

Binary files cups-1.1.20-11.1.i386.rpm and
/root/test/cups-1.1.20-11.1.i386.rpm differ
Binary files php-ldap-4.3.6-5.i386.rpm and
/root/test/php-ldap-4.3.6-5.i386.rpm differ
Binary files php-pear-4.3.6-5.i386.rpm and
/root/test/php-pear-4.3.6-5.i386.rpm differ
Binary files subversion-1.0.2-2.1.i386.rpm and
/root/test/subversion-1.0.2-2.1.i386.rpm differ

The corrupted files have same size as original files.

Than I did two additional rounds of tests.

In second round of tests, I used dump file on different partition
(original files on hda1, dump on hda6).  In this case I was not able
to reproduce the problem (but it might be that I wasn't trying hard
enough).

In third round of test, I dumped entire partition.  Again, I was not
able to reproduce the problem (but it might be that I wasn't trying
hard enough).

I'm reporting the bug with high severity, since it might result in
loss of valuable data (we all trust utilities such as dump or tar to
work reliably).

Version-Release number of selected component (if applicable):
dump-0.4b33-3

How reproducible:
Sometimes

Steps to Reproduce:
1. dump -0 -f dump_file dir
2. restore -ivf dump_file
3. diff between original and restored files
    

Actual Results:  The restored files were different than original files
(corrupted).

Expected Results:  The restored files should be the same as original
files.

Additional info:
Comment 1 Jeff Johnson 2004-06-12 17:55:00 EDT
Your trust is misplaced if using tar and/or dump to back up
active systems.

dump has never promised reliable archives on active unix
systems, and the problem is worse on linux because of
the lack of a character device, output from raw devices is
subject to change while cached by the kernel.

I'm pretty sure that's what the problem is, but I
will upgrade to -b36 and double check that the problem
exists there as well.
Comment 2 Aleksandar Milivojevic 2004-06-15 19:43:35 EDT
I know all about the dangers of backing up files on active file
system.  However, I expect that files that are not active (no process
have them open for writing) should be backed up fine.  The files that
I was trying to archive using dump were untouched by any process on
the system for long enough time that the kernel flushed all pages to
the disk for sure.

Anyhow, in case that (parts) of the file are still in cache and not
commited to disk, shouldn't the kernel return information from the cache?
Comment 3 Stelian Pop 2004-07-06 05:47:43 EDT
Jeff: you are probably correct, this looks very much like an active
filesystem problem.

Aleksandar: dump bypasses the kernel cache when it access the raw
disk, so if the kernel hasn't flushed the data dump will not see it.
Even worse, if the kernel has flushed only part of the data (for
example, flushed the inode metadata but not the data blocks), dump can
see an invalid filesystem.

Most of the time dumping a mounted filesystem works just fine, but
there is no guarantee, and you'd better run restore -C to verify it. 

On the other side, dumping a unmounted filesystem (or a filesystem
snapshot created by LVM/EVMS) is 100% guaranteed to be valid.

Stelian.
Comment 4 Aleksandar Milivojevic 2004-07-06 09:38:45 EDT
Than I guess we can mark this as NOTABUG (I'll leave it to Jeff to
decide)?

Stelian, if I understood you correctly, if I do something like:

   raidhotremove
   dump
   raidhotadd

Linux kernel will flush all data to the disk prior of it being removed
from the RAID device?  Is that documented and garanteed behaviour?  Or
you had something else in mind?
Comment 5 Stelian Pop 2004-07-07 11:54:51 EDT
Jeff: I released 0.4b37 today, which fixes a filesystem offset
calculation which could also lead to read errors or data corruption.
Make sure you package 0.4b37 if you decide to upgrade.

Aleksandar: no need for raidhotremove. I was talking about:
  umount /dev/whatever
  dump 0f /dev/tape /dev/whatever
  mount /dev/whatever


Comment 6 Aleksandar Milivojevic 2004-07-28 10:07:58 EDT
Stelian: The workaround doesn't work in my case (or I would be doing
it in a first place, and there wouldn't be this bug report).  It would
be nice if /dev/whatever was unmountable.  However it isn't.  So the
backup must be done on mounted file system.  No way around it. 
Booting single user or from CD is not an option either (for starters,
no physical access to console, not to mention other restrictions). 
Never mind, on Linux this obviously can't be done in a safe and simple
way with minimum (application) downtime like on Solaris by use of
lockfs -fw;metaoffline;lockfs -u;ufsdump;metaonline (which would only
sync the differences if any after metaonline which takes seconds to
complete, not resync entire meta device like raidhotadd witch takes
loooong time for large volumes).  Time to stop typing, I'm going too
much off topic anyway...
Comment 7 Stelian Pop 2004-07-28 12:28:04 EDT
I would consider that if 'raidhotremove' doesn't flush the data on the
hardware device then you have a kernel bug.

Anyway, you may want to look at LVM (or EVMS) snapshots, this may be
the only way for you to run dump without problems.

Just in case, please give a try to 0b4b37 too (you can download the
.src.rpm or binary rpm from dump.sf.net), a corruption fix is included
and who knows, you may be hitting exactly this bug...

Stelian.
Comment 8 Jindrich Novy 2004-08-11 07:39:00 EDT
Alex, have you tested the dump 0.4b37? Does the restore bug occur
again? I doubt that even Stellian tried hard to fix the problem, dump
cannot be safely used to mounted (and even not idle filesystems)
without the EVMS kernel patches. If you agree, I'll close this bug as
NOTABUG.

Jindrich
Comment 9 Aleksandar Milivojevic 2004-08-11 10:12:31 EDT
I've downloaded and installed 0.4b37.  The fix Stellian made fixed
this bug too (or maybe it was the same bug).  If you agree, it can be
closed as CURRENTRELEASE or NEXTRELEASE (whichever is appropriate).
Comment 10 Tim Powers 2005-05-19 09:37:14 EDT
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on the solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHBA-2005-439.html

Note You need to log in before you can comment on or make changes to this bug.