Bug 1303456
| Summary: | dd byte count report does not correlate with df byte count report | ||
|---|---|---|---|
| Product: | [Fedora] Fedora | Reporter: | Bob Gustafson <bobgus> |
| Component: | coreutils | Assignee: | Ondrej Vasik <ovasik> |
| Status: | CLOSED UPSTREAM | QA Contact: | Fedora Extras Quality Assurance <extras-qa> |
| Severity: | medium | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 23 | CC: | admiller, bobgus, kdudka, kzak, ovasik, p, ricky.tigg, twaugh |
| Target Milestone: | --- | ||
| Target Release: | --- | ||
| Hardware: | x86_64 | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | Bug Fix | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2016-02-01 16:50:28 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
|
Description
Bob Gustafson
2016-01-31 21:45:52 UTC
It is Raid 1, so dd is IMHO correct with the statement of the double copy byte size. dd just reports reads/writes byte counts provided by kernel... I'm not sure what causes the home->homen copy difference, strace output might be helpful. If you want to have this seen by wider audience, please send this info to coreutils , asking for the root cause of this behaviour - as I think the same behaviour will be in upstream coreutils. I most probably won't get to reproducing this anytime soon. df shows sizes of mounted file systems, which may have different (lower) sizes than the block devices they reside on. For the actual size of the block devices, which are logical volumes in your case, you should use lvdisplay instead. OK, I sent it out to bug-coreutils (In reply to Kamil Dudka from comment #2) > For the actual size of the > block devices, which are logical volumes in your case, you should use > lvdisplay instead. lvdisplay does give more or less the same information as found iin the Size column of the df display, but the actual amount of data in the allocated space is given only loosely by the Current LE number. (In reply to Bob Gustafson from comment #4) > But the actual amount of data in the allocated > space is given only loosely by the Current LE number. It looks like all of the space in Size has been allocated and is reflected in the number of 4k extents. No correlation to actual data written. This bug report is totally confusing for me. The sentence in summary is something that is not supposed to hold -- sizes of mounted file systems will almost never match sizes of their underlying block devices if you copy just contents of the block device. Please show us: - exactly one dd operation for which you think the reported size is wrong (the exact command line and its output) - the exact sizes of the source and destination block devices for that operation (note that df/du is unusable for this) (In reply to Kamil Dudka from comment #6) > This bug report is totally confusing for me. > > Please show us: > > - exactly one dd operation for which you think the reported size is wrong > (the exact command line and its output) > > - the exact sizes of the source and destination block devices for that > operation (note that df/du is unusable for this) Give me a command that you would like me to test and I will do it. The system is lightly used and it should be quite close to the state tested originally. (In reply to Bob Gustafson from comment #7) > Give me a command that you would like me to test and I will do it. The > system is lightly used and it should be quite close to the state tested > originally. You first need to tell us what kind of problem you are trying to report. If the only problem is that "dd byte count report does not correlate with df byte count report", then we can safely close this as NOTABUG. There are two invocations of dd in comment #0 -- which of them do you think reported wrong byte count? How are you confirming that the byte count reported by dd is wrong? (In reply to Kamil Dudka from comment #8) > (In reply to Bob Gustafson from comment #7) > There are two invocations of dd in comment #0 -- which of them do you think > reported wrong byte count? > > How are you confirming that the byte count reported by dd is wrong? I am confused by the reporting of 'bytes copied' in this invocation of dd: [root@hoho8-chidig-com user1]# dd if=/dev/fedora/home of=/dev/fedora23/home 1814978560+0 records in 1814978560+0 records out 929269022720 bytes (929 GB) copied, 32437.8 s, 28.6 MB/s When I mount the target partition (/dev/fedora23/home) and look to see the number of bytes as reported by df I don't see 929 GB. I see only 57 GB. This is a big difference. Can you explain the discrepancy? [root@hoho8-chidig-com user1]# mount /dev/fedora23/home /mnt/homen [root@hoho8-chidig-com user1]# du -h /mnt/homen | tail -1 57G /mnt/homen All these numbers were reported in the original presentation. I just now repeated the 'root' dd copy. [root@hoho8-chidig-com user1]# date Mon Feb 1 10:05:58 CST 2016 [root@hoho8-chidig-com user1]# dd if=/dev/fedora/root of=/dev/fedora23/root 104857600+0 records in 104857600+0 records out 53687091200 bytes (54 GB) copied, 1392.33 s, 38.6 MB/s [root@hoho8-chidig-com user1]# [root@hoho8-chidig-com user1]# date Mon Feb 1 10:29:34 CST 2016 Compared with the original copy - the bytes copied are almost exactly the same. The speed is a lot higher though. I did not delete everything in the target partition before starting this copy, but that shouldn't matter with a dd copy, yes? dd transfers data from the input file to the output file (both of them being block devices in your case). The data is transferred at a low level, where file systems do not exist. dd does not (and cannot) resize the file system for you. If you transfer file system data by dd between block devices of different size, you need to update the file system _metadata_ to reflect the actual size of the underlying block device. This can be done by file system-specific utilities (e.g. resize2fs for ext2/ext3/ext4). Note that using file system-specific dump/restore utilities to transfer the data is usually a better choice. I'm disappointed that you were not able to explain why dd reports 929 GB of data transferred - and only 57 GB shows up in the target partition. Bytes are bytes, independent of block sizes.
Ondrej Vasik did a good job (at least to me) of explaining why dd reports 54 G and the target disk ('root') showed only 27 G (see Comment #1). But this explanation does not easily extend to the dd copy of 'home'.
The reason I was using 'dd' was that I had previously used 'cpio' and cpio was failing because it 'ran out of space'. If cpio is using dd to do the actual transfers and gets a report of greater than 50 GB when copying the /root segment (where dd reports it transferred 54 GB, then you have a case where the inaccurate reporting of the bytes copied is a problem.
I think you are a bit premature in your proclamation of NOTABUG.
(In reply to Bob Gustafson from comment #12) > I'm disappointed that you were not able to explain why dd reports 929 GB of > data transferred - and only 57 GB shows up in the target partition. I have put a lot of effort to explain that. I am sorry it was not successful. > Bytes are bytes, independent of block sizes. Size of a block device is one thing and size of a file system is another thing. You do not seem to understand why the sizes may differ from each other and how to deal with that. It is explained in comment #11. (In reply to Kamil Dudka from comment #13) > (In reply to Bob Gustafson from comment #12) > > Bytes are bytes, independent of block sizes. > > Size of a block device is one thing and size of a file system is another > thing. You do not seem to understand why the sizes may differ from each > other and how to deal with that. It is explained in comment #11. In Comment #11, you are talking about block sizes and resizing disks. I am not doing that at all. I'm doing a simple copy from one disk partition to *another partition* on a *different disk*. dd is a dirt simple data copy program. **As one of its outputs, it reports 'bytes copied'.** I am trying to understand why dd is reporting such a large value for bytes copied - see my Comment #9 (In reply to Bob Gustafson from comment #14) > In Comment #11, you are talking about block sizes Nope. You are confusing the term "block size" with "size of block device". > and resizing disks. Nope. It was about resizing a file system. > I am not doing that at all. And that is exactly the cause of your troubles as I understand it ;-) You have been talking about blocks and filesystems and I have been consistently talking about bytes - particularly the byte count reported by dd. Why not just pass this bug report on to someone else. And take off the CLOSED and NOTABUG status flags. For reference, there is a response to the identical bug report on the upstream mailing list: http://lists.gnu.org/archive/html/bug-coreutils/2016-02/msg00004.html (In reply to Kamil Dudka from comment #17) > upstream mailing list: > > http://lists.gnu.org/archive/html/bug-coreutils/2016-02/msg00004.html Yes, I did receive an email from Bernard Voelker and I replied yesterday morning. I agreed that there can be small differences between the different reporting utilities, but: these differences are not off by a factor of 929/57=16+ in the case of copying the 'home' partition. My reply should appear on that upstream thread sooner or later. *** Bug 1596183 has been marked as a duplicate of this bug. *** |