Bug 1568875 - evm failing to start after upgrade from 5.6
Summary: evm failing to start after upgrade from 5.6
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat CloudForms Management Engine
Classification: Red Hat
Component: Documentation
Version: 5.9.0
Hardware: Unspecified
OS: Unspecified
high
urgent
Target Milestone: GA
: 5.9.2
Assignee: Chris Budzilowicz
QA Contact: Suyog Sainkar
URL:
Whiteboard: black:upgrade:migration
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-04-18 11:04 UTC by luke couzens
Modified: 2018-05-01 18:51 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-05-01 18:51:58 UTC
Category: ---
Cloudforms Team: ---
Target Upstream Version:


Attachments (Terms of Use)

Description luke couzens 2018-04-18 11:04:32 UTC
Document URL: 

https://access.redhat.com/documentation/en-us/red_hat_cloudforms/4.6/html/migrating_to_red_hat_cloudforms_4.6/index

https://doc-stage.usersys.redhat.com/documentation/en-us/red_hat_cloudforms/4.5/html-single/migrating_to_red_hat_cloudforms_4.5/

https://access.redhat.com/documentation/en-us/red_hat_cloudforms/4.2/html/migrating_to_red_hat_cloudforms_4.2/index

Section Number and Name: 3.6. Migrating from CFME 5.6 to 5.9

Describe the issue: It seems with the recent ruby update to 2.3.6 there is some additional change that affects the evm process starting up, due to a df command run by evm which hangs indefinitely due to package/location changes.

● evmserverd.service - EVM server daemon
   Loaded: loaded (/usr/lib/systemd/system/evmserverd.service; enabled; vendor preset: disabled)
   Active: active (running) since Wed 2018-04-18 06:46:40 EDT; 6min ago
  Process: 49233 ExecStart=/bin/sh -c /bin/evmserver.sh start (code=exited, status=0/SUCCESS)
 Main PID: 49258 (ruby)
   CGroup: /system.slice/evmserverd.service
           ├─49258 MIQ Server
           └─50712 df --all --local --human-readable --print-type


Suggestions for improvement: Simply rebooting the appliance fixes the location/package change issues and allows evm to start.

I suggest we add an appliance reboot step to all current migration procedures to overcome this issue. So to be sure lets add a step to reboot the appliance after starting evmserverd in each migration section in all the above docs. 
So all these migration sections: 
migrating_to_red_hat_cloudforms_4.2     5.5/5.6 - 5.7 
migrating_to_red_hat_cloudforms_4.5     5.6/5.7 - 5.8 
migrating_to_red_hat_cloudforms_4.6     5.5/5.6/5.7/5.8 - 5.9 
this should then cover any CFME version a customer may be running which may not have the latest ruby update yet.

Additional information: 
I have tested 5.6 to 5.7.4.3, 5.8.4.1 and 5.9.2.2 which have the latest ruby 2.3.6 all these see the same failing evm result and are fixed by an appliance reboot.

Comment 2 Nick Carboni 2018-04-18 14:50:38 UTC
This looks like this issue [1]. I was able to reproduce the same strace on an appliance with the issue. When we update we would need to reboot to start using the new version of systemd to get the fix applied to the running services.

[1] https://bugzilla.redhat.com/show_bug.cgi?id=1498318

Comment 4 luke couzens 2018-04-24 07:59:07 UTC
Hi Chris,

This looks good to me, rebooting both VMDB and non-VMDB is correct.

Thanks,
Luke


Note You need to log in before you can comment on or make changes to this bug.