Bug 1523459 - RFE: support to dump metadata of the image encrypted by LUKS using default human format
Summary: RFE: support to dump metadata of the image encrypted by LUKS using default hu...
Keywords:
Status: CLOSED WONTFIX
Alias: None
Product: Red Hat Enterprise Linux Advanced Virtualization
Classification: Red Hat
Component: qemu-kvm
Version: ---
Hardware: Unspecified
OS: Unspecified
low
low
Target Milestone: rc
: ---
Assignee: Maxim Levitsky
QA Contact: Tingting Mao
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-12-08 03:34 UTC by Ping Li
Modified: 2019-07-31 19:40 UTC (History)
12 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-07-31 19:40:13 UTC
Type: Feature Request
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description Ping Li 2017-12-08 03:34:31 UTC
Description of problem:
After install os on the image encrypted by LUKS, fail to dump the metadata of the image using human format. But it could be successful using json format. It will be strange that human format and json format work differently. As the human format is the default, So I think we should support it.

Version-Release number of selected component (if applicable):
kernel-3.10.0-806.el7.x86_64
qemu-kvm-rhev-2.10.0-11.el7

How reproducible:
100%

Steps to Reproduce:
1. create an image encrypted with luks
# qemu-img create --object secret,id=sec0,data=backing -f qcow2 -o encrypt.format=luks,encrypt.key-secret=sec0 base.qcow2 20G
Formatting 'base.qcow2', fmt=qcow2 size=21474836480 encrypt.format=luks encrypt.key-secret=sec0 cluster_size=65536 lazy_refcounts=off refcount_bits=16
     
2. Dump the metadata
# qemu-img map --object secret,id=sec0,data=backing --image-opts driver=qcow2,encrypt.key-secret=sec0,file.filename=base.qcow2
Offset          Length          Mapped to       File
     
# qemu-img map --object secret,id=sec0,data=backing --image-opts driver=qcow2,encrypt.key-secret=sec0,file.filename=base.qcow2 --output=json
[{ "start": 0, "length": 21474836480, "depth": 0, "zero": true, "data": false}]
     
3. Install os on the image.

4. Dump the metadata ---> failed to dump with human format
# qemu-img map --object secret,id=sec0,data=backing --image-opts driver=qcow2,encrypt.key-secret=sec0,file.filename=base.qcow2
Offset          Length          Mapped to       File
qemu-img: File contains external, encrypted or compressed clusters.

# qemu-img map --object secret,id=sec0,data=backing --image-opts driver=qcow2,encrypt.key-secret=sec0,file.filename=base.qcow2 --output=json
[{ "start": 0, "length": 65536, "depth": 0, "zero": false, "data": true},
{ "start": 65536, "length": 983040, "depth": 0, "zero": true, "data": false},
{ "start": 1048576, "length": 131792896, "depth": 0, "zero": false, "data": true},
{ "start": 132841472, "length": 136642560, "depth": 0, "zero": true, "data": false},
{ "start": 269484032, "length": 3342336, "depth": 0, "zero": false, "data": true},
{ "start": 272826368, "length": 265093120, "depth": 0, "zero": true, "data": false},
{ "start": 537919488, "length": 13172736, "depth": 0, "zero": false, "data": true},
{ "start": 551092224, "length": 255262720, "depth": 0, "zero": true, "data": false},
{ "start": 806354944, "length": 131072, "depth": 0, "zero": false, "data": true},
{ "start": 806486016, "length": 268173312, "depth": 0, "zero": true, "data": false},
{ "start": 1074659328, "length": 1245184, "depth": 0, "zero": false, "data": true},
{ "start": 1075904512, "length": 2147418112, "depth": 0, "zero": true, "data": false},
{ "start": 3223322624, "length": 381878272, "depth": 0, "zero": false, "data": true},
{ "start": 3605200896, "length": 4180475904, "depth": 0, "zero": true, "data": false},
{ "start": 7785676800, "length": 155254784, "depth": 0, "zero": false, "data": true},
{ "start": 7940931584, "length": 25296896, "depth": 0, "zero": true, "data": false},
{ "start": 7966228480, "length": 75300864, "depth": 0, "zero": false, "data": true},
{ "start": 8041529344, "length": 1900544, "depth": 0, "zero": true, "data": false},
{ "start": 8043429888, "length": 327680, "depth": 0, "zero": false, "data": true},
{ "start": 8043757568, "length": 131072, "depth": 0, "zero": true, "data": false},
{ "start": 8043888640, "length": 327680, "depth": 0, "zero": false, "data": true},
{ "start": 8044216320, "length": 131072, "depth": 0, "zero": true, "data": false},
{ "start": 8044347392, "length": 327680, "depth": 0, "zero": false, "data": true},
{ "start": 8044675072, "length": 131072, "depth": 0, "zero": true, "data": false},
{ "start": 8044806144, "length": 458752, "depth": 0, "zero": false, "data": true},
{ "start": 8045264896, "length": 65536, "depth": 0, "zero": true, "data": false},
{ "start": 8045330432, "length": 2228224, "depth": 0, "zero": false, "data": true},
{ "start": 8047558656, "length": 4300472320, "depth": 0, "zero": true, "data": false},
{ "start": 12348030976, "length": 202506240, "depth": 0, "zero": false, "data": true},
{ "start": 12550537216, "length": 1900544, "depth": 0, "zero": true, "data": false},
{ "start": 12552437760, "length": 2686976, "depth": 0, "zero": false, "data": true},
{ "start": 12555124736, "length": 1900544, "depth": 0, "zero": true, "data": false},
{ "start": 12557025280, "length": 25296896, "depth": 0, "zero": false, "data": true},
{ "start": 12582322176, "length": 4328062976, "depth": 0, "zero": true, "data": false},
{ "start": 16910385152, "length": 304218112, "depth": 0, "zero": false, "data": true},
{ "start": 17214603264, "length": 4258004992, "depth": 0, "zero": true, "data": false},
{ "start": 21472608256, "length": 131072, "depth": 0, "zero": false, "data": true},
{ "start": 21472739328, "length": 2031616, "depth": 0, "zero": true, "data": false},
{ "start": 21474770944, "length": 65536, "depth": 0, "zero": false, "data": true}]

Actual results:
Fail to dump metadata using human format

Expected results:
It should be ok using both human and json format

Additional info:
The issue doesn't occur on the image without encrypted.

1. Create image
# qemu-img create -f qcow2 rhel75.qcow2 20G
Formatting 'rhel75.qcow2', fmt=qcow2 size=21474836480 cluster_size=65536 lazy_refcounts=off refcount_bits=16

2. Dump the metadata
# qemu-img map rhel75.qcow2 
Offset          Length          Mapped to       File

3. Install os

4. Dump the metadata
# qemu-img map rhel75.qcow2 
Offset          Length          Mapped to       File
0               0x10000         0x50000         rhel75.qcow2
0x100000        0x80000         0x80000         rhel75.qcow2
0x180000        0x70000         0x110000        rhel75.qcow2
0x1f0000        0x10000         0x100000        rhel75.qcow2
0x200000        0x10000         0x25a70000      rhel75.qcow2
0x210000        0x90000         0x251f0000      rhel75.qcow2
0x2a0000        0x70000         0x25990000      rhel75.qcow2

Comment 3 Eric Blake 2018-01-02 18:59:19 UTC
(In reply to Ping Li from comment #0)
> Description of problem:
> After install os on the image encrypted by LUKS, fail to dump the metadata
> of the image using human format. But it could be successful using json
> format. It will be strange that human format and json format work
> differently. As the human format is the default, So I think we should
> support it.

The two outputs are different in what they report.  The JSON output intentionally conveys more information than the human output.  The human output is not supposed to be machine-parsed, so we could theoretically change it to improve the amount of information it outputs, but I'm not sure if that is a wise idea, or if we should just mark this as won't fix.


> Steps to Reproduce:
> 1. create an image encrypted with luks
> # qemu-img create --object secret,id=sec0,data=backing -f qcow2 -o
> encrypt.format=luks,encrypt.key-secret=sec0 base.qcow2 20G
> Formatting 'base.qcow2', fmt=qcow2 size=21474836480 encrypt.format=luks
> encrypt.key-secret=sec0 cluster_size=65536 lazy_refcounts=off
> refcount_bits=16
>      
> 2. Dump the metadata
> # qemu-img map --object secret,id=sec0,data=backing --image-opts
> driver=qcow2,encrypt.key-secret=sec0,file.filename=base.qcow2
> Offset          Length          Mapped to       File
>      
> # qemu-img map --object secret,id=sec0,data=backing --image-opts
> driver=qcow2,encrypt.key-secret=sec0,file.filename=base.qcow2 --output=json
> [{ "start": 0, "length": 21474836480, "depth": 0, "zero": true, "data":
> false}]

At this point, the difference matches non-encrypted images: the JSON output describes holes, but the human output only describes literal mappings (of which there are none).

>      
> 3. Install os on the image.
> 
> 4. Dump the metadata ---> failed to dump with human format
> # qemu-img map --object secret,id=sec0,data=backing --image-opts
> driver=qcow2,encrypt.key-secret=sec0,file.filename=base.qcow2
> Offset          Length          Mapped to       File
> qemu-img: File contains external, encrypted or compressed clusters.

Here, the output matches reality.  There are encrypted clusters, so we CAN'T tell you what the offsets are mapped to (because reading at that offset will see only the encrypted data, although the human-readable format shows only the offsets that you can directly read without having to decompress or decrypt).

> 
> # qemu-img map --object secret,id=sec0,data=backing --image-opts
> driver=qcow2,encrypt.key-secret=sec0,file.filename=base.qcow2 --output=json
> [{ "start": 0, "length": 65536, "depth": 0, "zero": false, "data": true},

And the JSON output indeed displays more information, because it is describing as much as possible, rather than just the places that can be directly read without decompression or decryption.

> Actual results:
> Fail to dump metadata using human format
> 
> Expected results:
> It should be ok using both human and json format
> 
> Additional info:
> The issue doesn't occur on the image without encrypted.

It also occurs on unencrypted images that use compression.  We can raise the question upstream whether the human output should be more verbose, but it may just be that if you want full information, you have to use the JSON output.

Comment 4 Zhenyu Zhang 2019-03-27 06:12:40 UTC
This issue still exists in RHEL-8.0.1-20190325.n.0-ppc64le
kernel-4.18.0-80.el8.ppc64le
qemu-kvm-3.1.0-20.module+el8+2888+cdc893a8

How reproducible:
100%

Steps to Reproduce:
1. create an image encrypted with luks
# qemu-img create --object secret,id=sec0,data=backing -f qcow2 -o encrypt.format=luks,encrypt.key-secret=sec0 base.qcow2 20G 
Formatting 'base.qcow2', fmt=qcow2 size=21474836480 encrypt.format=luks encrypt.key-secret=sec0 cluster_size=65536 lazy_refcounts=off refcount_bits=16

2. Install os on the image.

3. Dump the metadata ---> failed to dump with human format
# qemu-img map --object secret,id=sec0,data=backing --image-opts driver=qcow2,encrypt.key-secret=sec0,file.filename=base.qcow2
Offset          Length          Mapped to       File
qemu-img: File contains external, encrypted or compressed clusters.

# qemu-img map --object secret,id=sec0,data=backing --image-opts driver=qcow2,encrypt.key-secret=sec0,file.filename=base.qcow2 --output=json
[{ "start": 0, "length": 65536, "depth": 0, "zero": false, "data": true},
{ "start": 65536, "length": 983040, "depth": 0, "zero": true, "data": false},
{ "start": 1048576, "length": 177602560, "depth": 0, "zero": false, "data": true},
{ "start": 178651136, "length": 95027200, "depth": 0, "zero": true, "data": false},
{ "start": 273678336, "length": 2686976, "depth": 0, "zero": false, "data": true},
{ "start": 276365312, "length": 265748480, "depth": 0, "zero": true, "data": false},
{ "start": 542113792, "length": 10616832, "depth": 0, "zero": false, "data": true},
{ "start": 552730624, "length": 257818624, "depth": 0, "zero": true, "data": false},
{ "start": 810549248, "length": 327680, "depth": 0, "zero": false, "data": true},
{ "start": 810876928, "length": 267976704, "depth": 0, "zero": true, "data": false},
{ "start": 1078853632, "length": 1245184, "depth": 0, "zero": false, "data": true},
{ "start": 1080098816, "length": 2147418112, "depth": 0, "zero": true, "data": false},
{ "start": 3227516928, "length": 153223168, "depth": 0, "zero": false, "data": true},
{ "start": 3380740096, "length": 65536, "depth": 0, "zero": true, "data": false},
{ "start": 3380805632, "length": 63373312, "depth": 0, "zero": false, "data": true},
{ "start": 3444178944, "length": 65536, "depth": 0, "zero": true, "data": false},
{ "start": 3444244480, "length": 173015040, "depth": 0, "zero": false, "data": true},
{ "start": 3617259520, "length": 4171563008, "depth": 0, "zero": true, "data": false},
{ "start": 7788822528, "length": 195952640, "depth": 0, "zero": false, "data": true},
{ "start": 7984775168, "length": 3211264, "depth": 0, "zero": true, "data": false},
{ "start": 7987986432, "length": 1638400, "depth": 0, "zero": false, "data": true},
{ "start": 7989624832, "length": 393216, "depth": 0, "zero": true, "data": false},
{ "start": 7990018048, "length": 81788928, "depth": 0, "zero": false, "data": true},
{ "start": 8071806976, "length": 4278321152, "depth": 0, "zero": true, "data": false},
{ "start": 12350128128, "length": 168230912, "depth": 0, "zero": false, "data": true},
{ "start": 12518359040, "length": 6422528, "depth": 0, "zero": true, "data": false},
{ "start": 12524781568, "length": 54394880, "depth": 0, "zero": false, "data": true},
{ "start": 12579176448, "length": 4332257280, "depth": 0, "zero": true, "data": false},
{ "start": 16911433728, "length": 544997376, "depth": 0, "zero": false, "data": true},
{ "start": 17456431104, "length": 4016177152, "depth": 0, "zero": true, "data": false},
{ "start": 21472608256, "length": 131072, "depth": 0, "zero": false, "data": true},
{ "start": 21472739328, "length": 2031616, "depth": 0, "zero": true, "data": false},
{ "start": 21474770944, "length": 65536, "depth": 0, "zero": false, "data": true}]

Comment 5 Eric Blake 2019-07-31 19:40:13 UTC
Closing this as wontfix per comment #3; upstream made the intentional decision that the human-readable version only shows direct mappings (and compressed/encrypted images don't have direct mappings) and is constrained to that format due to the choice of column headers implying that the data is already processed from the raw layer into the format matching the columns; while the JSON output shows everything at a lower layer and is not constrained by formatting.


Note You need to log in before you can comment on or make changes to this bug.