Bug 857838
| Summary: | VM payload should accept arbitrary binary input | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Virtualization Manager | Reporter: | Geert Jansen <gjansen> |
| Component: | ovirt-engine-restapi | Assignee: | Nobody <nobody> |
| Status: | CLOSED NOTABUG | QA Contact: | Tareq Alayan <talayan> |
| Severity: | high | Docs Contact: | |
| Priority: | high | ||
| Version: | 3.1.0 | CC: | acathrow, bugzilla, ecohen, iheim, jkt, jrankin, michal.skrivanek, mpastern, Rhev-m-bugs, shavivi |
| Target Milestone: | --- | Keywords: | Reopened |
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | virt | ||
| Fixed In Version: | Doc Type: | Bug Fix | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2013-07-11 21:24:27 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
This is not a FutureFeature. We should accept any content in the payload. Michael - what's the scope of effort to fix this? michael/shahar - don't we support base64 encoded content? (In reply to comment #2) > We should accept any content in the payload. According to http://wiki.ovirt.org/wiki/Features/VMPayload : "The payload data will be encoded using base64 encoding", so we don't accept *any* type - unless we have some encoding mechanism. QE conditional NACK on requirements. > > Michael - what's the scope of effort to fix this? Hi Geert,
I had no problem passing multilined <content> using [1], i think the problem
is in the client you have used.
[1]
<vm>
<payloads>
<payload type="floppy">
<file name="payload.txt">
<content>
content here1
content here2
content here3
</content>
</file>
</payload>
</payloads>
</vm>
(In reply to comment #4) > (In reply to comment #2) > > We should accept any content in the payload. > > According to http://wiki.ovirt.org/wiki/Features/VMPayload : "The payload > data will be encoded using base64 encoding", so we don't accept *any* type - > unless we have some encoding mechanism. if there any "encoding mechanism" it's resides in BE, (api is agnostic to data formats and /get on payload returns plain text), Shahar? > > QE conditional NACK on requirements. > > > > > Michael - what's the scope of effort to fix this? The VmPayload current design is support only text files. The engine is getting a plain text and encode it to Base64 before storing the data in DB. Geert, I think Michael is right, you can try use cdata tag as your content wrapper: http://www.w3schools.com/xml/xml_cdata.asp please re-open if can be reproduced per above comments. I tried CDATA but it gave a RESTeasy error. But in any case, you cannot pass arbitrary binary data in CDATA so the point is moot. Why is this closed by the way? It was already moved to rhevm-future and if it is closed we would not be able to find it anymore. bugs with needinfo for too long are closed insufficent data asking to re-open if can be reproduced. michael, per your comment 5, can you please put a sample of updating multi line content and getting it back as multiline? thanks, Itamar (In reply to comment #12) > michael, per your comment 5, can you please put a sample of updating multi > line content and getting it back as multiline? > > thanks, > Itamar * using FF REST extension vm before ========= GET http://localhost:8080/api/vms/f409c734-65f2-4d7c-b25b-3e285d8d5a76 <vm id="f409c734-65f2-4d7c-b25b-3e285d8d5a76" href="/api/vms/f409c734-65f2-4d7c-b25b-3e285d8d5a76"> <name>iscsi_desktop</name> <type>desktop</type> <status> <state>down</state> </status> ... </vm> request ======= PUT http://localhost:8080/api/vms/f409c734-65f2-4d7c-b25b-3e285d8d5a76 <vm> <payloads> <payload type="floppy"> <file name="payload.txt"> <content> content here1 content here2 content here3 </content> </file> </payload> </payloads> </vm> vm after ======== GET http://localhost:8080/api/vms/f409c734-65f2-4d7c-b25b-3e285d8d5a76 <?xml version="1.0" encoding="UTF-8" standalone="yes"?> <vm id="f409c734-65f2-4d7c-b25b-3e285d8d5a76" href="/api/vms/f409c734-65f2-4d7c-b25b-3e285d8d5a76"> <name>iscsi_desktop</name> <type>desktop</type> <status> <state>down</state> </status> ... <payloads> <payload type="floppy"> <file name="payload.txt"> <content> content here1 content here2 content here3 </content> </file> </payload> </payloads> ... </vm> geert - please review comment 13 Regarding comment 13, multi-line input may work now, but this BZ is about accepting *binary* input for the vmpayload feature. Even using CDATA, you cannot pass binary input. (In reply to comment #15) > Regarding comment 13, multi-line input may work now, but this BZ is about > accepting *binary* input for the vmpayload feature. > > Even using CDATA, you cannot pass binary input. WORKS-FOR-ME (Geert, i think the problem is in your client): request: ======= PUT http://localhost:8080/api/vms/f409c734-65f2-4d7c-b25b-3e285d8d5a76 <vm> <payloads> <payload type="floppy"> <file name="payload.txt"> <content> <![CDATA[ ^?ELF^B^A^A^@^@^@^@^@^@^@^@^@^B^@>^@^A^@^@^@<D0>^_@^@^@^@^@^@@^@^@^@^@^@^@^@<U+0620>^@^@^@^@^@^@^@^@^@^@@^@8^@^H^@@^@^_^@^^^@^F^@^@^@^E^@^@^@@^@^@^@^@^@^@^@@^@ @^@^@^@^@^@@^@@^@^@^@^@^@<C0>^A^@^@^@^@^@^@<C0>^A^@^@^@^@^@^@^H^@^@^@^@^@^@^@^C^@^@^@^D^@^@^@^@^B^@^@^@^@^@^@^@^B@^@^@^@^@^@^@^B@^@^@^@^@^@^\^@^@^@^@^@^@^@^\^@ ^@^@^@^@^@^@^A^@^@^@^@^@^@^@^A^@^@^@^E^@^@^@^@^@^@^@^@^@^@^@^@^@@^@^@^@^@^@^@^@@^@^@^@^@^@><8D>^@^@^@^@^@^@><8D>^@^@^@^@^@^@^@^@ ^@^@^@^@^@^A^@^@^@^F^@^@^@^@ <90>^@^@^@^@^@^@^@<90>`^@^@^@^@^@^@<90>`^@^@^@^@^@<E0>^F^@^@^@^@^@^@<98>8^A^@^@^@^@^@^@^@ ^@^@^@^@^@^B^@^@^@^F^@^@^@(<90>^@^@^@^@^@^@(<90>`^@^@^@^@^@(<90>`^@^@ ^@^@^@<A0>^A^@^@^@^@^@^@<A0>^A^@^@^@^@^@^@^H^@^@^@^@^@^@^@^D^@^@^@^D^@^@^@^\^B^@^@^@^@^@^@^\^B@^@^@^@^@^@^\^B@^@^@^@^@^@D^@^@^@^@^@^@^@D^@^@^@^@^@^@^@^D^@^@^@ ^@^@^@^@P<E5>td^D^@^@^@<E8><82>^@^@^@^@^@^@<E8><82>@^@^@^@^@^@<E8><82>@^@^@^@^@^@<E4>^@^@^@^@^@^@^@<E4>^@^@^@^@^@^@^@^D^@^@^@^@^@^@^@Q<E5>td^F^@^@^@^@^@^@^@^@ ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^H^@^@^@^@^@^@^@/lib64/ld-linux-x86-64.so.2^@^D^@^@^@^P^@^@^@^A^@^@^@GNU^@^@^@^@^@^B^@^@ ^@^F^@^@^@^R^@^@^@^D^@^@^@^T^@^@^@^C^@^@^@GNU^@o<B8>ղH<92> <DE>n<8A><B1><83><93>,ش[<80><B9>H%^@^@^@@^@^@^@^D^@^@^@^H^@^@^@^B<A1><A0><A0><D1>d^@<82>^Ha^D^P<84> ^Y`<E2><81>^@#<C4>Z<80>^\<A6><80>^@^M`<82>^@)^C@^@^@^@^@^@^@^@A^@^@^@^@^@^@^@B^@^@^@D^@^@^@^@^@^@^@F^@^@^@G^@^@^@I^@^@^@J^@^@^@^@^@^@^@^@^@^@^@M^@^@^@^@^@^@^@O ^@^@^@Q^@^@^@S^@^@^@T^@^@^@^@^@^@^@U^@^@^@V^@^@^@^@^@^@^@W^@^@^@^@^@^@^@^@^@^@^@X^@^@^@Z^@^@^@\^@^@^@^^@^@^@_^@^@^@`^@^@^@b^@^@^@d^@^@^@e^@^@^@f^@^@^@^@^@^@^@ <9F><9F><F7><96><DD><F9>j^X&A<88><95><E7><9E>(;|<E8>a<E9><F9>0s*w<FF>:^\*O4<8C>?<A7><88>^K<D1>^S.^K<BA><D5><ED><81>~<A2><96>2<87><B2><88>^K8<9F><A3>^O<91>P<F0> q<BE>^Z<99><CC>M<88>s<B3><A4>t<A4><EB>^]^W^R<82>C<F9><AB>1<91><80><9A>|<F5><F3>Q<90><A3>G^N<C6>O<A7><C9>w^T^O<89><F8><DF><E1>-^@l^R<87>¡濰 *s^P^_<88><9E>|]<B5> <9D>}-^M^_^^<C6>R^_^\<B3><FD>q<D3>^\g^R^O<B9><AF><88>^K/<A4><9E>|c+<B5><DD>^??^ES^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ ^A^@^@^R^@^@^@^@^@^@^@^@^@^@ ^@^@^@^@^@^@^@^@^@'^B^@^@^R^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@<CA>^A^@^@^R^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@<FC>^@^@^@^R^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ ^@^@^@d^A^@^@^R^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^Z^A^@^@^R^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@<D0>^A^@^@^R^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^N^@^@^@ ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^]^@^@^@ ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@<DE>^@^@^@^R^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@U^B^@^@^R^@^@^@^@^@^@^@ ^@^@^@^@^@^@^@^@^@^@^@^@<83>^@^@^@^R^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@v^@^@^@^R^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^C^B^@^@^R^@^@^@^@^@^@^@^@^@^@^@^@^@^@ ^@^@^@^@^@<F4>^@^@^@^R^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@X^A^@^@^R^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@<9B>^@^@^@^R^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ ]]> </content> </file> </payload> </payloads> </vm> response: ======== GET http://localhost:8080/api/vms/f409c734-65f2-4d7c-b25b-3e285d8d5a76 <?xml version="1.0" encoding="UTF-8" standalone="yes"?> <vm id="f409c734-65f2-4d7c-b25b-3e285d8d5a76" href="/api/vms/f409c734-65f2-4d7c-b25b-3e285d8d5a76"> <name>iscsi_desktop</name> <type>desktop</type> <status> <state>down</state> </status> <payloads> <payload type="floppy"> <file name="payload.txt"> <content> ^?ELF^B^A^A^@^@^@^@^@^@^@^@^@^B^@>^@^A^@^@^@<D0>^_@^@^@^@^@^@@^@^@^@^@^@^@^@<U+0620>^@^@^@^@^@^@^@^@^@^@@^@8^@^H^@@^@^_^@^^^@^F^@^@^@^E^@^@^@@^@^@^@^@^@^@^@@^@ @^@^@^@^@^@@^@@^@^@^@^@^@<C0>^A^@^@^@^@^@^@<C0>^A^@^@^@^@^@^@^H^@^@^@^@^@^@^@^C^@^@^@^D^@^@^@^@^B^@^@^@^@^@^@^@^B@^@^@^@^@^@^@^B@^@^@^@^@^@^\^@^@^@^@^@^@^@^\^@ ^@^@^@^@^@^@^A^@^@^@^@^@^@^@^A^@^@^@^E^@^@^@^@^@^@^@^@^@^@^@^@^@@^@^@^@^@^@^@^@@^@^@^@^@^@><8D>^@^@^@^@^@^@><8D>^@^@^@^@^@^@^@^@ ^@^@^@^@^@^A^@^@^@^F^@^@^@^@ <90>^@^@^@^@^@^@^@<90>`^@^@^@^@^@^@<90>`^@^@^@^@^@<E0>^F^@^@^@^@^@^@<98>8^A^@^@^@^@^@^@^@ ^@^@^@^@^@^B^@^@^@^F^@^@^@(<90>^@^@^@^@^@^@(<90>`^@^@^@^@^@(<90>`^@^@ ^@^@^@<A0>^A^@^@^@^@^@^@<A0>^A^@^@^@^@^@^@^H^@^@^@^@^@^@^@^D^@^@^@^D^@^@^@^\^B^@^@^@^@^@^@^\^B@^@^@^@^@^@^\^B@^@^@^@^@^@D^@^@^@^@^@^@^@D^@^@^@^@^@^@^@^D^@^@^@ ^@^@^@^@P<E5>td^D^@^@^@<E8><82>^@^@^@^@^@^@<E8><82>@^@^@^@^@^@<E8><82>@^@^@^@^@^@<E4>^@^@^@^@^@^@^@<E4>^@^@^@^@^@^@^@^D^@^@^@^@^@^@^@Q<E5>td^F^@^@^@^@^@^@^@^@ ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^H^@^@^@^@^@^@^@/lib64/ld-linux-x86-64.so.2^@^D^@^@^@^P^@^@^@^A^@^@^@GNU^@^@^@^@^@^B^@^@ ^@^F^@^@^@^R^@^@^@^D^@^@^@^T^@^@^@^C^@^@^@GNU^@o<B8>H<92> <DE>n<8A><B1><83><93>,[<80><B9>H%^@^@^@@^@^@^@^D^@^@^@^H^@^@^@^B<A1><A0><A0><D1>d^@<82>^Ha^D^P<84> ^Y`<E2><81>^@#<C4>Z<80>^\<A6><80>^@^M`<82>^@)^C@^@^@^@^@^@^@^@A^@^@^@^@^@^@^@B^@^@^@D^@^@^@^@^@^@^@F^@^@^@G^@^@^@I^@^@^@J^@^@^@^@^@^@^@^@^@^@^@M^@^@^@^@^@^@^@O ^@^@^@Q^@^@^@S^@^@^@T^@^@^@^@^@^@^@U^@^@^@V^@^@^@^@^@^@^@W^@^@^@^@^@^@^@^@^@^@^@X^@^@^@Z^@^@^@\^@^@^@^^@^@^@_^@^@^@`^@^@^@b^@^@^@d^@^@^@e^@^@^@f^@^@^@^@^@^@^@ <9F><9F><F7><96><DD><F9>j^X&A<88><95><E7><9E>(;|<E8>a<E9><F9>0s*w<FF>:^\*O4<8C>?<A7><88>^K<D1>^S.^K<BA><D5><ED><81>~<A2><96>2<87><B2><88>^K8<9F><A3>^O<91>P<F0> q<BE>^Z<99><CC>M<88>s<B3><A4>t<A4><EB>^]^W^R<82>C<F9><AB>1<91><80><9A>|<F5><F3>Q<90><A3>G^N<C6>O<A7><C9>w^T^O<89><F8><DF><E1>-^@l^R<87> *s^P^_<88><9E>|]<B5> <9D>}-^M^_^^<C6>R^_^\<B3><FD>q<D3>^\g^R^O<B9><AF><88>^K/<A4><9E>|c+<B5><DD>^??^ES^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ ^A^@^@^R^@^@^@^@^@^@^@^@^@^@ ^@^@^@^@^@^@^@^@^@'^B^@^@^R^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@<CA>^A^@^@^R^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@<FC>^@^@^@^R^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ ^@^@^@d^A^@^@^R^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^Z^A^@^@^R^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@<D0>^A^@^@^R^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^N^@^@^@ ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^]^@^@^@ ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@<DE>^@^@^@^R^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@U^B^@^@^R^@^@^@^@^@^@^@ ^@^@^@^@^@^@^@^@^@^@^@^@<83>^@^@^@^R^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@v^@^@^@^R^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^C^B^@^@^R^@^@^@^@^@^@^@^@^@^@^@^@^@^@ ^@^@^@^@^@<F4>^@^@^@^R^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@X^A^@^@^R^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@<9B>^@^@^@^R^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ </content> </file> </payload> </payloads> ... </vm> Michael, please try to upload a file that contains the characters ']]>'. This will not work. Also, the *output* of the API above, will very likely be invalid XML, unless RESTeasy inserts CDATA on output too. (In reply to comment #17) > Michael, please try to upload a file that contains the characters ']]>'. > This will not work. indeed, - it's explicitly forbidden by spec., workaround will be allowing file upload via dedicated endpoint rather than using xml element for this. also it means that we won't be able displaying it. i guess for now we can state that we support only text files and binary is a TP. > > Also, the *output* of the API above, will very likely be invalid XML, unless > RESTeasy inserts CDATA on output too. this output fetched from my server. Can we do this: add a encoding="base64" attribute to <content>, and if set, base64-decode the contents? Should be a few lines of Java code. (In reply to comment #19) > Can we do this: add a encoding="base64" attribute to <content>, and if set, > base64-decode the contents? Should be a few lines of Java code. we can, but it's not so user friendly, now when i'm thinking about this, we modelled entire feature incorrectly, it was suppose to be sub-collection/resource of vm, this way we would have /api/vms/xxx/payloads endpoint for uploads (also it adds tone of garbage to vm representation on GET): /api/vms/xxx/payloads <payloads> <payload type="floppy" id=yyy> <file name="payload.txt"> <content> ... </content> </file> </payload> </payloads> /api/vms/xxx/payloads/yyy <payload type="floppy" id=yyy> <file name="payload.txt"> <content> ... </content> </file> </payload> if we want changing it, the time is now, or we will have to carry this design for next few versions, andy/simon/itamar/geert? Your proposed changes make sense. However, this does not remove the need for base64 encoded <content>. (In reply to comment #21) > Your proposed changes make sense. However, this does not remove the need for > base64 encoded <content>. why? you will be able uploading binary file as-is, without converting it to text and/or encoding. If we keep the logic, then the "payloads" collection will contain <payload> resources that itself are XML and contain a <file> child element with a <content> grandchild element. Or do you want a "content" sub-resource below the payload resource, i.e "/api/vms/xxx/payloads/yyy/contents"? geert - can you explain the use case for binary input? you do remember it is limited to 16KB? it is not intended for binary upload in general? michael - we don't have an id per payload (other than the file id maybe)? (In reply to comment #24) > geert - can you explain the use case for binary input? you do remember it is > limited to 16KB? it is not intended for binary upload in general? > > michael - we don't have an id per payload (other than the file id maybe)? not a problem, i can create UUID from the file name (it's unique - thanks to file system) (In reply to comment #23) > If we keep the logic, then the "payloads" collection will contain <payload> > resources that itself are XML and contain a <file> child element with a > <content> grandchild element. > > Or do you want a "content" sub-resource below the payload resource, i.e > "/api/vms/xxx/payloads/yyy/contents"? i planned [1], but your proposal even better, this way we can restrict viewing content nicely, i guess the question now "if user should see payloads?" or it available for admins only and we can stay with [1]. [1] /api/vms/xxx/payloads/yyy <payload type="floppy" id=yyy> <file name="payload.txt"> <content> ... </content> </file> </payload> If you do your proposal [1] then you still need to encode the <contents> section, right? I am for your proposal by the way, i do not want to introduce a separate "contents" resource. Regarding comment 24: yes i realize this is limited to small files. But we still need binary input. |
Currently the VM payload feature is restricted in the input it accepts. I haven't tested the full range of characters that it might accept, but so far my testing has found this: * Newlines are not accepted. * XML unsafe characters like '<' and '>' are not accepted as they result in the input XML being not well formed and rejected at the RESTeasy level. As a design goal, we should make sure we support a wide array of applications with VM payload. This means that we should have as little limitations on the payload data as possible. I propose that the API accepts the VM payload in an encoded format that is safe to embed in XML. The most common format seems to be base64. At some point before the payload is provided to the VM, should be decoded. This decoding can be done where ever makes sense (API, engine, VDSM). The goal is that a user is able to provide a binary blob as payload containing the full range of byte values (0 .. 255) up to the length limit for VM payloads. Below i've added an example that shows that the API currently does not accept multi-line input: # cat update.xml <vm> <payloads> <payload type="floppy"> <file name="payload.txt"> <content>line one line two line three </content> </file> </payload> </payloads> </vm> Using this cURL command: # curl -k --user admin@internal:xxxx --request PUT --data @update.xml --header "Content-Type: application/xml" https://localhost:443/api/vms/{id} What i get back is this: <payloads> <payload type="floppy"> <file name="payload.txt"> <content>line oneline twoline three</content> </file> </payload> </payloads> So the content is now all on 1 line. And when i mount the floppy in the guest, i see the same thing (newlines removed).