Description of problem: After install os on the image encrypted by LUKS, fail to dump the metadata of the image using human format. But it could be successful using json format. It will be strange that human format and json format work differently. As the human format is the default, So I think we should support it. Version-Release number of selected component (if applicable): kernel-3.10.0-806.el7.x86_64 qemu-kvm-rhev-2.10.0-11.el7 How reproducible: 100% Steps to Reproduce: 1. create an image encrypted with luks # qemu-img create --object secret,id=sec0,data=backing -f qcow2 -o encrypt.format=luks,encrypt.key-secret=sec0 base.qcow2 20G Formatting 'base.qcow2', fmt=qcow2 size=21474836480 encrypt.format=luks encrypt.key-secret=sec0 cluster_size=65536 lazy_refcounts=off refcount_bits=16 2. Dump the metadata # qemu-img map --object secret,id=sec0,data=backing --image-opts driver=qcow2,encrypt.key-secret=sec0,file.filename=base.qcow2 Offset Length Mapped to File # qemu-img map --object secret,id=sec0,data=backing --image-opts driver=qcow2,encrypt.key-secret=sec0,file.filename=base.qcow2 --output=json [{ "start": 0, "length": 21474836480, "depth": 0, "zero": true, "data": false}] 3. Install os on the image. 4. Dump the metadata ---> failed to dump with human format # qemu-img map --object secret,id=sec0,data=backing --image-opts driver=qcow2,encrypt.key-secret=sec0,file.filename=base.qcow2 Offset Length Mapped to File qemu-img: File contains external, encrypted or compressed clusters. # qemu-img map --object secret,id=sec0,data=backing --image-opts driver=qcow2,encrypt.key-secret=sec0,file.filename=base.qcow2 --output=json [{ "start": 0, "length": 65536, "depth": 0, "zero": false, "data": true}, { "start": 65536, "length": 983040, "depth": 0, "zero": true, "data": false}, { "start": 1048576, "length": 131792896, "depth": 0, "zero": false, "data": true}, { "start": 132841472, "length": 136642560, "depth": 0, "zero": true, "data": false}, { "start": 269484032, "length": 3342336, "depth": 0, "zero": false, "data": true}, { "start": 272826368, "length": 265093120, "depth": 0, "zero": true, "data": false}, { "start": 537919488, "length": 13172736, "depth": 0, "zero": false, "data": true}, { "start": 551092224, "length": 255262720, "depth": 0, "zero": true, "data": false}, { "start": 806354944, "length": 131072, "depth": 0, "zero": false, "data": true}, { "start": 806486016, "length": 268173312, "depth": 0, "zero": true, "data": false}, { "start": 1074659328, "length": 1245184, "depth": 0, "zero": false, "data": true}, { "start": 1075904512, "length": 2147418112, "depth": 0, "zero": true, "data": false}, { "start": 3223322624, "length": 381878272, "depth": 0, "zero": false, "data": true}, { "start": 3605200896, "length": 4180475904, "depth": 0, "zero": true, "data": false}, { "start": 7785676800, "length": 155254784, "depth": 0, "zero": false, "data": true}, { "start": 7940931584, "length": 25296896, "depth": 0, "zero": true, "data": false}, { "start": 7966228480, "length": 75300864, "depth": 0, "zero": false, "data": true}, { "start": 8041529344, "length": 1900544, "depth": 0, "zero": true, "data": false}, { "start": 8043429888, "length": 327680, "depth": 0, "zero": false, "data": true}, { "start": 8043757568, "length": 131072, "depth": 0, "zero": true, "data": false}, { "start": 8043888640, "length": 327680, "depth": 0, "zero": false, "data": true}, { "start": 8044216320, "length": 131072, "depth": 0, "zero": true, "data": false}, { "start": 8044347392, "length": 327680, "depth": 0, "zero": false, "data": true}, { "start": 8044675072, "length": 131072, "depth": 0, "zero": true, "data": false}, { "start": 8044806144, "length": 458752, "depth": 0, "zero": false, "data": true}, { "start": 8045264896, "length": 65536, "depth": 0, "zero": true, "data": false}, { "start": 8045330432, "length": 2228224, "depth": 0, "zero": false, "data": true}, { "start": 8047558656, "length": 4300472320, "depth": 0, "zero": true, "data": false}, { "start": 12348030976, "length": 202506240, "depth": 0, "zero": false, "data": true}, { "start": 12550537216, "length": 1900544, "depth": 0, "zero": true, "data": false}, { "start": 12552437760, "length": 2686976, "depth": 0, "zero": false, "data": true}, { "start": 12555124736, "length": 1900544, "depth": 0, "zero": true, "data": false}, { "start": 12557025280, "length": 25296896, "depth": 0, "zero": false, "data": true}, { "start": 12582322176, "length": 4328062976, "depth": 0, "zero": true, "data": false}, { "start": 16910385152, "length": 304218112, "depth": 0, "zero": false, "data": true}, { "start": 17214603264, "length": 4258004992, "depth": 0, "zero": true, "data": false}, { "start": 21472608256, "length": 131072, "depth": 0, "zero": false, "data": true}, { "start": 21472739328, "length": 2031616, "depth": 0, "zero": true, "data": false}, { "start": 21474770944, "length": 65536, "depth": 0, "zero": false, "data": true}] Actual results: Fail to dump metadata using human format Expected results: It should be ok using both human and json format Additional info: The issue doesn't occur on the image without encrypted. 1. Create image # qemu-img create -f qcow2 rhel75.qcow2 20G Formatting 'rhel75.qcow2', fmt=qcow2 size=21474836480 cluster_size=65536 lazy_refcounts=off refcount_bits=16 2. Dump the metadata # qemu-img map rhel75.qcow2 Offset Length Mapped to File 3. Install os 4. Dump the metadata # qemu-img map rhel75.qcow2 Offset Length Mapped to File 0 0x10000 0x50000 rhel75.qcow2 0x100000 0x80000 0x80000 rhel75.qcow2 0x180000 0x70000 0x110000 rhel75.qcow2 0x1f0000 0x10000 0x100000 rhel75.qcow2 0x200000 0x10000 0x25a70000 rhel75.qcow2 0x210000 0x90000 0x251f0000 rhel75.qcow2 0x2a0000 0x70000 0x25990000 rhel75.qcow2
(In reply to Ping Li from comment #0) > Description of problem: > After install os on the image encrypted by LUKS, fail to dump the metadata > of the image using human format. But it could be successful using json > format. It will be strange that human format and json format work > differently. As the human format is the default, So I think we should > support it. The two outputs are different in what they report. The JSON output intentionally conveys more information than the human output. The human output is not supposed to be machine-parsed, so we could theoretically change it to improve the amount of information it outputs, but I'm not sure if that is a wise idea, or if we should just mark this as won't fix. > Steps to Reproduce: > 1. create an image encrypted with luks > # qemu-img create --object secret,id=sec0,data=backing -f qcow2 -o > encrypt.format=luks,encrypt.key-secret=sec0 base.qcow2 20G > Formatting 'base.qcow2', fmt=qcow2 size=21474836480 encrypt.format=luks > encrypt.key-secret=sec0 cluster_size=65536 lazy_refcounts=off > refcount_bits=16 > > 2. Dump the metadata > # qemu-img map --object secret,id=sec0,data=backing --image-opts > driver=qcow2,encrypt.key-secret=sec0,file.filename=base.qcow2 > Offset Length Mapped to File > > # qemu-img map --object secret,id=sec0,data=backing --image-opts > driver=qcow2,encrypt.key-secret=sec0,file.filename=base.qcow2 --output=json > [{ "start": 0, "length": 21474836480, "depth": 0, "zero": true, "data": > false}] At this point, the difference matches non-encrypted images: the JSON output describes holes, but the human output only describes literal mappings (of which there are none). > > 3. Install os on the image. > > 4. Dump the metadata ---> failed to dump with human format > # qemu-img map --object secret,id=sec0,data=backing --image-opts > driver=qcow2,encrypt.key-secret=sec0,file.filename=base.qcow2 > Offset Length Mapped to File > qemu-img: File contains external, encrypted or compressed clusters. Here, the output matches reality. There are encrypted clusters, so we CAN'T tell you what the offsets are mapped to (because reading at that offset will see only the encrypted data, although the human-readable format shows only the offsets that you can directly read without having to decompress or decrypt). > > # qemu-img map --object secret,id=sec0,data=backing --image-opts > driver=qcow2,encrypt.key-secret=sec0,file.filename=base.qcow2 --output=json > [{ "start": 0, "length": 65536, "depth": 0, "zero": false, "data": true}, And the JSON output indeed displays more information, because it is describing as much as possible, rather than just the places that can be directly read without decompression or decryption. > Actual results: > Fail to dump metadata using human format > > Expected results: > It should be ok using both human and json format > > Additional info: > The issue doesn't occur on the image without encrypted. It also occurs on unencrypted images that use compression. We can raise the question upstream whether the human output should be more verbose, but it may just be that if you want full information, you have to use the JSON output.
This issue still exists in RHEL-8.0.1-20190325.n.0-ppc64le kernel-4.18.0-80.el8.ppc64le qemu-kvm-3.1.0-20.module+el8+2888+cdc893a8 How reproducible: 100% Steps to Reproduce: 1. create an image encrypted with luks # qemu-img create --object secret,id=sec0,data=backing -f qcow2 -o encrypt.format=luks,encrypt.key-secret=sec0 base.qcow2 20G Formatting 'base.qcow2', fmt=qcow2 size=21474836480 encrypt.format=luks encrypt.key-secret=sec0 cluster_size=65536 lazy_refcounts=off refcount_bits=16 2. Install os on the image. 3. Dump the metadata ---> failed to dump with human format # qemu-img map --object secret,id=sec0,data=backing --image-opts driver=qcow2,encrypt.key-secret=sec0,file.filename=base.qcow2 Offset Length Mapped to File qemu-img: File contains external, encrypted or compressed clusters. # qemu-img map --object secret,id=sec0,data=backing --image-opts driver=qcow2,encrypt.key-secret=sec0,file.filename=base.qcow2 --output=json [{ "start": 0, "length": 65536, "depth": 0, "zero": false, "data": true}, { "start": 65536, "length": 983040, "depth": 0, "zero": true, "data": false}, { "start": 1048576, "length": 177602560, "depth": 0, "zero": false, "data": true}, { "start": 178651136, "length": 95027200, "depth": 0, "zero": true, "data": false}, { "start": 273678336, "length": 2686976, "depth": 0, "zero": false, "data": true}, { "start": 276365312, "length": 265748480, "depth": 0, "zero": true, "data": false}, { "start": 542113792, "length": 10616832, "depth": 0, "zero": false, "data": true}, { "start": 552730624, "length": 257818624, "depth": 0, "zero": true, "data": false}, { "start": 810549248, "length": 327680, "depth": 0, "zero": false, "data": true}, { "start": 810876928, "length": 267976704, "depth": 0, "zero": true, "data": false}, { "start": 1078853632, "length": 1245184, "depth": 0, "zero": false, "data": true}, { "start": 1080098816, "length": 2147418112, "depth": 0, "zero": true, "data": false}, { "start": 3227516928, "length": 153223168, "depth": 0, "zero": false, "data": true}, { "start": 3380740096, "length": 65536, "depth": 0, "zero": true, "data": false}, { "start": 3380805632, "length": 63373312, "depth": 0, "zero": false, "data": true}, { "start": 3444178944, "length": 65536, "depth": 0, "zero": true, "data": false}, { "start": 3444244480, "length": 173015040, "depth": 0, "zero": false, "data": true}, { "start": 3617259520, "length": 4171563008, "depth": 0, "zero": true, "data": false}, { "start": 7788822528, "length": 195952640, "depth": 0, "zero": false, "data": true}, { "start": 7984775168, "length": 3211264, "depth": 0, "zero": true, "data": false}, { "start": 7987986432, "length": 1638400, "depth": 0, "zero": false, "data": true}, { "start": 7989624832, "length": 393216, "depth": 0, "zero": true, "data": false}, { "start": 7990018048, "length": 81788928, "depth": 0, "zero": false, "data": true}, { "start": 8071806976, "length": 4278321152, "depth": 0, "zero": true, "data": false}, { "start": 12350128128, "length": 168230912, "depth": 0, "zero": false, "data": true}, { "start": 12518359040, "length": 6422528, "depth": 0, "zero": true, "data": false}, { "start": 12524781568, "length": 54394880, "depth": 0, "zero": false, "data": true}, { "start": 12579176448, "length": 4332257280, "depth": 0, "zero": true, "data": false}, { "start": 16911433728, "length": 544997376, "depth": 0, "zero": false, "data": true}, { "start": 17456431104, "length": 4016177152, "depth": 0, "zero": true, "data": false}, { "start": 21472608256, "length": 131072, "depth": 0, "zero": false, "data": true}, { "start": 21472739328, "length": 2031616, "depth": 0, "zero": true, "data": false}, { "start": 21474770944, "length": 65536, "depth": 0, "zero": false, "data": true}]
Closing this as wontfix per comment #3; upstream made the intentional decision that the human-readable version only shows direct mappings (and compressed/encrypted images don't have direct mappings) and is constrained to that format due to the choice of column headers implying that the data is already processed from the raw layer into the format matching the columns; while the JSON output shows everything at a lower layer and is not constrained by formatting.