Created attachment 1163829 [details] Screenshot of VM Analysis History Description of problem: Cloudforms SmartState Analysis produces an error when an OpenStack instance is booted from a Cinder volume. The reported error is: "Unable to mount filesystem. Reason:[Image: EvmSnapshot (00cc8ca8-a06e-4fd0-a278-c77678bc9e76) is empty]" Version-Release number of selected component (if applicable): 5.5.3.4.20160407153134_b3e2a83 How reproducible: 1. Create an OpenStack instance booted from a Cinder volume a. Log-in to Horizon b. Select an appropriate project c. Select "Volumes" d. Select "Create a Volume" e. Under "Volume Source" f. Under "Use image as a source" select an appropriate image g. Select "Instances" h. Select "Launch Instance" i. Under "Instance boot source" select "Boot from volume" j. Under "Volume" select the volume created in (c)-(f) 2. Log-in to the Cloudforms appliance, and select the instance created in (1) (perform a "Refresh Relationships and Power States" for the provider if required to discover the instance) 3. Select "Configuration" -> "Perform SmartState Analysis" for the instance Actual results: Cloudforms is unable to analyse the instance, and reports an error: "Unable to mount filesystem. Reason:[Image: EvmSnapshot (00cc8ca8-a06e-4fd0-a278-c77678bc9e76) is empty]" Expected results: Cloudforms performs the SmartState Analysis and reports back results as it would for an OpenStack instance booted from ephemeral disk Additional info: Attached screenshot of Cloudforms error reported under instance "Analysis history"
Step 1.c should state: Under "Volume Source" select "Image"
Created attachment 1166358 [details] evm.log containing the errors
Rich, We discussed about this bugzilla in mahwah office last week. scanning Cinder volumes is critical for the customer and the Red Hat Consulting wants to know how long this might take to resolve so we can manage customer expectations and adjust the project deliverables if necessary. I resetting the severity of this bz back to urgent as this is very important functionality for my customer, My Findings : what happens when a smart state analysis is executed for an OpenStack Instance. 1. Instance metadata is collected(API) 2. Snapshot created with name "EvmSnapshot" . (API) 3. Instance root disk is copied to cfme appliance (API) (Need to verify this) 4. Snapshot is removed 5. Image is read by smartproxy to fetch the insights 6. Image destroyed. . --- If the Nova instance you snapshotted was using a Cinder volume as its root disk, The image created from the “nova image-create” command is not a snapshot of the root disk, but rather contains metadata about the actual snapshot. nova image-show 2f623b9a-685d-4af4-b537-77e6c85840ef | OS-EXT-IMG-SIZE:size | 0 | metadata block_device_mapping | [{"guest_format": null, "boot_index": 0, "no_device": null, "image_id": null, "snapshot_id": "a9c9d41c-6efa-47d8-8720-731a64dc8ad9", "device_name": "/dev/vda", "disk_bus": "virtio", "volume_size": 1, "source_type": "snapshot", "device_type": "disk", "volume_id": null, "destination_type": "volume", "delete_on_termination": false}] | | metadata root_device_name | /dev/vda In the “block_device_mapping” property there is a “snapshot_id”, this id corresponds to the Cinder volume snapshot, which is a snapshot of the root disk for that instance. # cinder snapshot-show a9c9d41c-6efa-47d8-8720-731a64dc8ad9 +--------------------------------------------+--------------------------------------+ | Property | Value | +--------------------------------------------+--------------------------------------+ | created_at | 2016-06-02T12:01:41.000000 | | description | | | id | a9c9d41c-6efa-47d8-8720-731a64dc8ad9 | | metadata | {} | | name | snapshot for snap1 | | os-extended-snapshot-attributes:progress | 100% | | os-extended-snapshot-attributes:project_id | 8b21e8f95ba6410db8cb5b4e62f5de0d | | size | 1 | | status | available | | volume_id | 1c722231-0e76-436f-ab29-9c3531d021b2 | +--------------------------------------------+--------------------------------------+ So snapshot image size always remain 0. I think our cloudforms code has verification logic that first checks the size of snapshot image. If size is zero throws error.
It's hard to say how long this will take, given it'll take some research to determine if it's even possible. Currently, all of our snapshot and data acquisition is accomplished through Glance. We do a GET on the image (snapshot) reading the image data. There's even an extension to the Glance API to read the image data given offset and length that we intend to use to improve performance. Now, we're saying for Cinder based images, Glance GET doesn't work, and we have to go through Cinder to read the data. Currently, I'm not sure if Cinder has a comparable method to retrieve the volume data, and if it does, will it let us read given an offset and length. Even if the API methods are available, we then have to ensure that they're supported by the Fog gem. Even then, we'll need to create volumes from which instances can be booted. As I recall, that wasn't a simple task - that was some time ago, it could be easier now. Ultimately, I think this is a Glance issue. If the purpose of Glance is to be a general abstraction for interacting with images, the semantics of the GET operation shouldn't change based on how the image data is stored.
Upon further investigation, it appears very difficult to accomplish this with the current state of the Glance and Cinder APIs. As described in previous comments, when a request is made to read the data for a Cinder-backed image, no data is returned; instead, the ID of the backing Cinder snapshot is returned in the metadata. We can create a volume based on this snapshot, but there doesn't appear to be a way to read the volume's data via the Cinder API. It may be possible to create a backup of the volume to object store, then read the data via the Swift API. However, this would be time consuming, and would consume object store storage, which may not be available. From a Glance perspective, this would probably be treated as an RFE. It would most likely require support from Cinder, which doesn't appear to exist currently. Then, before we could employ it, we'd have to wait for the release and installation of the new Glance and Cinder APIs.
This should be fixed by https://github.com/ManageIQ/manageiq-smartstate/pull/21
New commit detected on ManageIQ/manageiq-smartstate/master: https://github.com/ManageIQ/manageiq-smartstate/commit/bcce73b223026c4b80b1610ee03d626a83397974 commit bcce73b223026c4b80b1610ee03d626a83397974 Author: Tzu-Mainn Chen <tzumainn> AuthorDate: Tue Jul 18 16:06:36 2017 +0200 Commit: Tzu-Mainn Chen <tzumainn> CommitDate: Tue Aug 1 16:55:10 2017 +0200 Add alternate path to allow SSA of instances booted from volumes https://bugzilla.redhat.com/show_bug.cgi?id=1341867 .../MiqOpenStackVm/MiqOpenStackCommon.rb | 60 +++++++++++++++++++--- .../MiqOpenStackVm/MiqOpenStackInstance.rb | 4 ++ 2 files changed, 58 insertions(+), 6 deletions(-)
Verified ======== 5.9.0.4
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2018:0380