Bug 1825148

Summary: CFME applaince stopped working after performing SSA on vsphere65 vm
Product: Red Hat CloudForms Management Engine Reporter: Niyaz Akhtar Ansari <nansari>
Component: SmartState AnalysisAssignee: Jerry Keselman <jkeselma>
Status: CLOSED DUPLICATE QA Contact: Niyaz Akhtar Ansari <nansari>
Severity: high Docs Contact: Red Hat CloudForms Documentation <cloudforms-docs>
Priority: high    
Version: 5.11.5CC: dmetzger, obarenbo
Target Milestone: GA   
Target Release: 5.11.6   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-04-21 12:53:24 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: Bug
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: CFME Core Target Upstream Version:
Embargoed:
Attachments:
Description Flags
centos issue none

Description Niyaz Akhtar Ansari 2020-04-17 07:59:30 UTC 
Description of problem:


Version-Release number of selected component (if applicable):
Version: 5.11.5.1

How reproducible:
100&

Steps to Reproduce:
1. Add Vsphere65-nested provider
2. Install vmware-vix-disklib-6.5.0-1.el7.x86_64
3. Start smart proxy
4. Copy the sample profile as the default and make true to some entry content 
5. Add credentials to host 
6. Perform SSA on vm. ( I did ssa on centos vm)
7. wait to complete the SSA 
7. See the evm log



Actual results:
After performing the SSA my appliance stopped working and it through the error on ui. 

Expected results:

It should work


Additional info:
evm log

[----] E, [2020-04-17T03:11:30.950182 #6777:2b28c57b65c4] ERROR -- : MIQ(MiqServer#validate_worker) Worker [MiqEmsRefreshCoreWorker] with ID: [47], PID: [13425], GUID: [cc6462e4-07b6-438e-a18e-7c465a81fda5] has not responded in 121.187901697 seconds, restarting worker
[----] E, [2020-04-17T03:14:05.721399 #6777:2b28c57b65c4] ERROR -- : MIQ(MiqServer#validate_worker) Worker [ManageIQ::Providers::Vmware::InfraManager::EventCatcher] with ID: [45], PID: [13401], GUID: [ded7e755-2431-4038-b7da-c128ab700d10] has not responded in 162.111513473 seconds, restarting worker
[----] E, [2020-04-17T03:14:07.384176 #6777:2b28c57b65c4] ERROR -- : MIQ(MiqServer#validate_worker) Worker [MiqUiWorker] with ID: [42], PID: [7390], GUID: [6a84258a-802c-45f8-9c22-ecacc7cbeac2] has not responded in 140.500458397 seconds, restarting worker
[----] E, [2020-04-17T03:14:20.659387 #6777:2b28c57b65c4] ERROR -- : MIQ(MiqServer#validate_worker) Worker [MiqVimBrokerWorker] with ID: [48], PID: [13439], GUID: [143eb583-319a-4ad7-8ba6-a158776db627] has not responded in 174.821802747 seconds, restarting worker
[----] E, [2020-04-17T03:14:21.016098 #6777:2b28c57b65c4] ERROR -- : MIQ(MiqServer#validate_worker) Worker [MiqWebServiceWorker] with ID: [43], PID: [7434], GUID: [17daea6c-6a5c-45f1-a45d-a0b01d8a0bbb] has not responded in 135.349454827 seconds, restarting worker
[----] E, [2020-04-17T03:14:31.765169 #6777:2b28c57b65c4] ERROR -- : MIQ(MiqServer#validate_worker) Worker [MiqRemoteConsoleWorker] with ID: [44], PID: [7443], GUID: [d1bd84f3-9ca7-403a-81e2-2a9aa694f5d0] has not responded in 150.737987564 seconds, restarting worker
[----] E, [2020-04-17T03:15:23.564384 #6777:2b28c57b65c4] ERROR -- : MIQ(MiqServer#validate_worker) Worker [MiqEventHandler] with ID: [34], PID: [7316], GUID: [47cf8d3e-956c-426a-8720-d92e4dbda962] has not responded in 624.047829554 seconds, restarting worker
[----] E, [2020-04-17T03:15:24.080131 #6777:2b28c57b65c4] ERROR -- : MIQ(MiqServer#validate_worker) Worker [MiqReportingWorker] with ID: [40], PID: [7368], GUID: [251558f5-61ca-4d4c-92df-618424e2a99a] has not responded in 182.096201503 seconds, restarting worker
[----] E, [2020-04-17T03:15:24.121736 #6777:2b28c57b65c4] ERROR -- : MIQ(MiqServer#validate_worker) Worker [MiqReportingWorker] with ID: [39], PID: [7361], GUID: [4ae5c94e-84a4-4856-895e-e3af22730e10] has not responded in 182.761045491 seconds, restarting worker
[----] E, [2020-04-17T03:15:24.197949 #6777:2b28c57b65c4] ERROR -- : MIQ(MiqServer#validate_worker) Worker [MiqScheduleWorker] with ID: [41], PID: [7381], GUID: [887a65ee-dd42-4bb7-b1ef-872c9d1d6305] has not responded in 161.277248506 seconds, restarting worker
[----] E, [2020-04-17T03:15:37.723020 #6777:2b28c57b65c4] ERROR -- : MIQ(MiqServer#start_algorithm_used_swap_percent_lt_value) Not allowing worker [MiqReportingWorker] to start since system memory usage has exceeded 60% of swap: Total: [9659478016], Used: [5876391936]
[----] E, [2020-04-17T03:15:37.790891 #6777:2b28c57b65c4] ERROR -- : MIQ(MiqServer#start_algorithm_used_swap_percent_lt_value) Not allowing worker [MiqScheduleWorker] to start since system memory usage has exceeded 60% of swap: Total: [9659478016], Used: [5882871808]
[----] E, [2020-04-17T03:16:23.974495 #6777:2b28c57b65c4] ERROR -- : MIQ(MiqServer#start_algorithm_used_swap_percent_lt_value) Not allowing worker [MiqReportingWorker] to start since system memory usage has exceeded 60% of swap: Total: [9659478016], Used: [6899396608]
[----] E, [2020-04-17T03:16:23.979799 #6777:2b28c57b65c4] ERROR -- : MIQ(MiqServer#start_algorithm_used_swap_percent_lt_value) Not allowing worker [MiqScheduleWorker] to start since system memory usage has exceeded 60% of swap: Total: [9659478016], Used: [6899396608]
[----] E, [2020-04-17T03:16:33.150109 #6777:2b28c57b65c4] ERROR -- : MIQ(MiqServer#validate_worker) Worker [MiqSmartProxyWorker] with ID: [50], PID: [14474], GUID: [090889c4-d471-499c-af1a-60786b9afc8a] has not responded in 156.975523339 seconds, restarting worker
[----] E, [2020-04-17T03:16:44.688234 #6777:2b28c57b65c4] ERROR -- : MIQ(MiqServer#start_algorithm_used_swap_percent_lt_value) Not allowing worker [MiqReportingWorker] to start since system memory usage has exceeded 60% of swap: Total: [9659478016], Used: [7481962496]
[----] E, [2020-04-17T03:16:44.693381 #6777:2b28c57b65c4] ERROR -- : MIQ(MiqServer#start_algorithm_used_swap_percent_lt_value) Not allowing worker [MiqScheduleWorker] to start since system memory usage has exceeded 60% of swap: Total: [9659478016], Used: [7481729024]
[----] E, [2020-04-17T03:16:44.699758 #6777:2b28c57b65c4] ERROR -- : MIQ(MiqServer#start_algorithm_used_swap_percent_lt_value) Not allowing worker [MiqSmartProxyWorker] to start since system memory usage has exceeded 60% of swap: Total: [9659478016], Used: [7481757696]

[----] E, [2020-04-17T03:17:03.587247 #6777:2b28c57b65c4] ERROR -- : MIQ(MiqServer#start_algorithm_used_swap_percent_lt_value) Not allowing worker [MiqReportingWorker] to start since system memory usage has exceeded 60% of swap: Total: [9659478016], Used: [7509741568]
[----] E, [2020-04-17T03:17:03.591971 #6777:2b28c57b65c4] ERROR -- : MIQ(MiqServer#start_algorithm_used_swap_percent_lt_value) Not allowing worker [MiqScheduleWorker] to start since system memory usage has exceeded 60% of swap: Total: [9659478016], Used: [7509737472]
[----] E, [2020-04-17T03:17:03.597870 #6777:2b28c57b65c4] ERROR -- : MIQ(MiqServer#start_algorithm_used_swap_percent_lt_value) Not allowing worker [MiqSmartProxyWorker] to start since system memory usage has exceeded 60% of swap: Total: [9659478016], Used: [7509737472]







[----] E, [2020-04-17T03:17:21.275099 #6777:2b28c57b65c4] ERROR -- : MIQ(MiqServer#start_algorithm_used_swap_percent_lt_value) Not allowing worker [MiqReportingWorker] to start since system memory usage has exceeded 60% of swap: Total: [9659478016], Used: [7602581504]
[----] E, [2020-04-17T03:17:21.281392 #6777:2b28c57b65c4] ERROR -- : MIQ(MiqServer#start_algorithm_used_swap_percent_lt_value) Not allowing worker [MiqScheduleWorker] to start since system memory usage has exceeded 60% of swap: Total: [9659478016], Used: [7602962432]
[----] E, [2020-04-17T03:17:21.289562 #6777:2b28c57b65c4] ERROR -- : MIQ(MiqServer#start_algorithm_used_swap_percent_lt_value) Not allowing worker [MiqSmartProxyWorker] to start since system memory usage has exceeded 60% of swap: Total: [9659478016], Used: [7603298304]
[----] E, [2020-04-17T03:17:38.986503 #6777:2b28c57b65c4] ERROR -- : MIQ(MiqServer#start_algorithm_used_swap_percent_lt_value) Not allowing worker [MiqReportingWorker] to start since system memory usage has exceeded 60% of swap: Total: [9659478016], Used: [7728984064]
[----] E, [2020-04-17T03:17:38.991109 #6777:2b28c57b65c4] ERROR -- : MIQ(MiqServer#start_algorithm_used_swap_percent_lt_value) Not allowing worker [MiqScheduleWorker] to start since system memory usage has exceeded 60% of swap: Total: [9659478016], Used: [7728984064]
[----] E, [2020-04-17T03:17:38.997413 #6777:2b28c57b65c4] ERROR -- : MIQ(MiqServer#start_algorithm_used_swap_percent_lt_value) Not allowing worker [MiqSmartProxyWorker] to start since system memory usage has exceeded 60% of swap: Total: [9659478016], Used: [7728984064]
[----] E, [2020-04-17T03:17:51.337000 #15301:2b28c5f26174] ERROR -- : MIQ(MiqSmartProxyWorker::Runner) ID [51] PID [15301] GUID [04a91589-051e-4ff8-b9c6-f64b472cb29c] Error in before_exit: Target thread must not be current thread
[----] E, [2020-04-17T03:17:56.101265 #6777:2b28c57b65c4] ERROR -- : MIQ(MiqServer#start_algorithm_used_swap_percent_lt_value) Not allowing worker [MiqReportingWorker] to start since system memory usage has exceeded 60% of swap: Total: [9659478016], Used: [7749677056]
[----] E, [2020-04-17T03:17:56.106751 #6777:2b28c57b65c4] ERROR -- : MIQ(MiqServer#start_algorithm_used_swap_percent_lt_value) Not allowing worker [MiqScheduleWorker] to start since system memory usage has exceeded 60% of swap: Total: [9659478016], Used: [7749677056]
[----] E, [2020-04-17T03:17:56.114268 #6777:2b28c57b65c4] ERROR -- : MIQ(MiqServer#start_algorithm_used_swap_percent_lt_value) Not allowing worker [MiqSmartProxyWorker] to start since system memory usage has exceeded 60% of swap: Total: [9659478016], Used: [7749677056]
[----] E, [2020-04-17T03:17:57.618683 #6777:2b28c57b65c4] ERROR -- : M
Comment 4 Niyaz Akhtar Ansari 2020-04-17 10:28:40 UTC 
Created attachment 1679625 [details]
centos issue

The issue is with centos vm.  I tired on vsphere67-nested as well and faced same issue.

after Performing the ssa on centos vm, the request seems to be stuck with "Scanning system" message on Task Page
Comment 9 Jerry Keselman 2020-04-20 16:38:32 UTC 
With my fix applied to this appliance, I can successfully run Smartstate against vm nansari_ssa.  Can you see if you can reproduce this issue against any other VMs on this appliance?  If not I am going to close this BZ as a duplicate of https://bugzilla.redhat.com/show_bug.cgi?id=1824259
Comment 10 Niyaz Akhtar Ansari 2020-04-21 11:41:16 UTC 
looks good to me for others VMs
Comment 11 Jerry Keselman 2020-04-21 12:53:24 UTC 

*** This bug has been marked as a duplicate of bug 1824259 ***