Bug 1551273 - [RHV] Smart state analysis task succeed, however the packages are not collected for the CFME VM.
Summary: [RHV] Smart state analysis task succeed, however the packages are not collect...
Alias: None
Product: Red Hat CloudForms Management Engine
Classification: Red Hat
Component: Providers
Version: 5.9.0
Hardware: Unspecified
OS: Unspecified
Target Milestone: GA
: 5.10.0
Assignee: Hui Song
QA Contact: Angelina Nikiforova
Whiteboard: rhev:SmartStateAnalysis
Depends On:
TreeView+ depends on / blocked
Reported: 2018-03-04 08:57 UTC by Ilanit Stein
Modified: 2019-08-01 12:55 UTC (History)
12 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Last Closed: 2019-02-07 23:01:17 UTC
Category: Bug
Cloudforms Team: RHEVM
Target Upstream Version:
istein: needinfo-
istein: needinfo-

Attachments (Terms of Use)
evm.log (512.97 KB, application/x-gzip)
2018-03-04 08:58 UTC, Ilanit Stein
no flags Details
automation.log (87.94 KB, application/x-gzip)
2018-03-04 08:58 UTC, Ilanit Stein
no flags Details
production.log (97.60 KB, application/x-gzip)
2018-03-04 08:59 UTC, Ilanit Stein
no flags Details
rhevm.log (26.21 KB, application/x-gzip)
2018-03-04 08:59 UTC, Ilanit Stein
no flags Details
evm_2.log (306.64 KB, application/x-gzip)
2018-03-11 14:53 UTC, Ilanit Stein
no flags Details
automation_2.log (26.48 KB, application/x-gzip)
2018-03-11 14:53 UTC, Ilanit Stein
no flags Details

System ID Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2019:0212 None None None 2019-02-07 23:01:24 UTC

Description Ilanit Stein 2018-03-04 08:57:35 UTC
Description of problem:
For CFME- (RHEL-7.4 based) VM, that all it's disks are on ISCSI storage,
SmartState analysis task ends up successfully,
however the packages and the files fields has content 0.
Other fields (Users, Groups, Init Processes) are filled OK. 

Version-Release number of selected component (if applicable):

Expected results:
Info on packages and files

Additional info:
Regarding the files collection, the *.conf files were marked for collection, under CFME->Configuration.

Comment 3 Ilanit Stein 2018-03-04 08:58:31 UTC
Created attachment 1403697 [details]

Comment 4 Ilanit Stein 2018-03-04 08:58:55 UTC
Created attachment 1403698 [details]

Comment 5 Ilanit Stein 2018-03-04 08:59:19 UTC
Created attachment 1403699 [details]

Comment 6 Ilanit Stein 2018-03-04 08:59:47 UTC
Created attachment 1403700 [details]

Comment 7 Moti Asayag 2018-03-04 18:29:28 UTC
By examine the log, there seems to be connectivity issue with the storage domain:

This is the creation of the scan job:
[----] I, [2018-03-04T02:14:59.348639 #13076:d2b130]  INFO -- : Job created: guid: [ab6803b5-f9ab-4354-8cb9-955ef1cabd5f], userid: [admin], name: [Scan from Vm cfme-], target class: [VmOrTemplate], target id: [5], process type: [VmScan], server id: [], zone: [default]

right after there is a reported event from the engine:

[----] I, [2018-03-04T02:15:03.457230 #13029:d2b130]  INFO -- : MIQ(ManageIQ::Providers::Redhat::InfraManager::EventCatcher::Runner#queue_event) EMS [istein-rhv41.scl.lab.tlv.redhat.com] as [admin@internal] Caught event [VDS_STORAGES_CONNECTION_FAILED]
[----] I, [2018-03-04T02:15:03.502888 #13029:d2b130]  INFO -- : MIQ(MiqQueue.put) Message id: [309],  id: [], Zone: [default], Role: [event], Server: [], Ident: [ems], Target id: [1], Instance id: [], Task id: [], Command: [EmsEvent.add], Timeout: [600], Priority: [100], State: [ready], Deliver On: [], Data: [], Args: [{:event_type=>"VDS_STORAGES_CONNECTION_FAILED", :source=>"RHEVM", :message=>"Failed to connect Host puma44.scl.lab.tlv.redhat.com to the Storage Domains export_domain.", :timestamp=>Sun, 04 Mar 2018 02:15:01 -0500, :username=>nil, :full_data=>#<OvirtSDK4::Event:0x00000001b82b98 @href="/ovirt-engine/api/events/2281388", @comment=nil, @description="Failed to connect Host puma44.scl.lab.tlv.redhat.com to the Storage Domains export_domain.",

And continuing with the scanning job:
[ready], Deliver On: [], Data: [], Args: ["ab6803b5-f9ab-4354-8cb9-955ef1cabd5f", "Initializing scan"]
[----] I, [2018-03-04T02:15:37.386661 #13137:d2b130]  INFO -- : Q-task_id([job_dispatcher]) MIQExtract using config file: [/rhev/data-center/c2aba05a-113b-494c-9be4-c1add73d694b/mastersd/master/vms/ac4df678-e86b-4595-a296-f0c38ece40b2/ac4df678-e86b-4595-a296-f0c38ece40b2.ovf]  settings: [{"ems"=>{"ems"=>{:hostname=>"istein-rhv41.scl.lab.tlv.redhat.com", :ipaddress=>nil, :username=>"admin@internal", :password=>"********", :class_name=>"ManageIQ::Providers::Redhat::InfraManager"}, "host"=>{:hostname=>"puma44.scl.lab.tlv.redhat.com", :ipaddress=>"", :username=>nil, :password=>nil, :class_name=>"ManageIQ::Providers::Redhat::InfraManager::Host"}, "connect_to"=>"ems", "connect"=>true}, "snapshot"=>{"use_existing"=>false, "description"=>nil, "create_free_percent"=>100, "remove_free_percent"=>100, "name"=>nil}, "vmScanProfiles"=>[], "permissions"=>{"group"=>36}}]
[----] I, [2018-03-04T02:15:37.565430 #13137:d2b130]  INFO -- : Q-task_id([job_dispatcher]) MIQ(DiskProbe-getDiskMod) Disk probe detected [RhevmDisk-QcowDisk] for [/dev/25162aa6-249c-4abe-a1b0-73946b338946/c56fc405-bca7-4f92-befe-aa9d557cbb14]
[----] E, [2018-03-04T02:15:37.579845 #13137:d2b130] ERROR -- : Q-task_id([job_dispatcher]) Couldn't open disk file: /dev/25162aa6-249c-4abe-a1b0-73946b338946/c56fc405-bca7-4f92-befe-aa9d557cbb14
[----] E, [2018-03-04T02:15:37.579944 #13137:d2b130] ERROR -- : Q-task_id([job_dispatcher]) No such file or directory @ rb_file_s_stat - /dev/25162aa6-249c-4abe-a1b0-73946b338946/c56fc405-bca7-4f92-befe-aa9d557cbb14

[root@puma44 ~]# ll /dev/25162aa6-249c-4abe-a1b0-73946b338946/c56fc405-bca7-4f92-befe-aa9d557cbb14
lrwxrwxrwx. 1 root root 7 Mar  4 10:04 /dev/25162aa6-249c-4abe-a1b0-73946b338946/c56fc405-bca7-4f92-befe-aa9d557cbb14 -> ../dm-9
[root@puma44 ~]# ll ../dm-9
ls: cannot access ../dm-9: No such file or directory

So it seems like a specific issue with RHV storage.
Can we re-test and get an access to the env when it fails ?

Comment 8 Ilanit Stein 2018-03-05 09:53:43 UTC
env details was provided in private.

Comment 9 Ilanit Stein 2018-03-05 12:09:48 UTC
This bug do not exist on CFME-

Comment 10 Ilanit Stein 2018-03-05 12:13:22 UTC

Is this bug RHV specific, or is it seen on other providers too? 


Comment 12 Ilanit Stein 2018-03-11 14:17:45 UTC
From further investigation, done with masayag, we have the following fining:

For CFME VM with disks on NFS/ISCSI storage, when defined as evm server,
and Smart State analysis (SSA) is run on it, the SSA task ends up successfully, however the packages number is 0.
Running SSA on another (down) VM on, residing in the same RHV env, end successfully, and the packages are collected.

There for this bug seem to be CFME VM specific.

Note that in all SSA tests the files were also not collected, but this is another issue, still under investigation.

Comment 14 Ilanit Stein 2018-03-11 14:53:08 UTC
Created attachment 1406895 [details]

Comment 15 Ilanit Stein 2018-03-11 14:53:41 UTC
Created attachment 1406896 [details]

Comment 16 Ilanit Stein 2018-03-11 14:58:15 UTC

Would you please take a look at this Smart State analysis "CFME VM" specific package collection issue?
(I can supply reproduce environment if required) 


Comment 17 Ilanit Stein 2018-03-15 06:56:37 UTC
Moving the need info to Hui, who started to look into this.

Comment 18 Ilanit Stein 2018-03-19 10:41:56 UTC
This bug is about packages details is not reported, further to running SmartState Analysis, specifically on A CFME appliance (with disks from any storage type).

According to lavenel, SmartState Analysis for the CFME appliance is not recommended/supported.

Therefore, turning this bug into a doc bug, in order to document it.

Comment 19 Ilanit Stein 2018-03-19 16:08:20 UTC
Additional information - This VM SSA problem, of packages not collected happen on every CFME VM, not only the one that serve as the appliance.

Are you aware of this?
Seem there is something specific in the CFME VMs, that fail the collection.

Comment 20 Ilanit Stein 2018-03-22 12:29:15 UTC
Moving back from Docs back to Providers, based on comment 19.

Comment 21 Hui Song 2018-03-22 13:48:23 UTC
I will work on this issue. Assigned to myself.

Comment 28 CFME Bot 2018-05-14 19:58:00 UTC
New commit detected on ManageIQ/manageiq-smartstate/master:

commit f14783da05e5868dae3b3a8f388b816ae099499c
Author:     hsong-rh <hsong@redhat.com>
AuthorDate: Mon May 14 09:26:14 2018 -0400
Commit:     hsong-rh <hsong@redhat.com>
CommitDate: Mon May 14 09:26:14 2018 -0400

    Fix to remove extra leading spaces in /etc/fstab during SSA


 lib/fs/modules/LinuxMount.rb | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

Comment 30 Ilanit Stein 2018-05-21 10:43:42 UTC

In which version, can this fix be included please?


Comment 35 Ilanit Stein 2018-09-16 11:12:33 UTC
Verified on CFME-

Smart state analysis CFME VM with ISCSI disk ended up with Packages & Files collection.

Comment 36 errata-xmlrpc 2019-02-07 23:01:17 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.