Bug 1672537

* This bug might be related to v2v (VMware to RHV), since this CFME was used for running it (successfully).

* Tried to restart evmserverd service, and now I can't connect to the CFME UI:

[root@cloudforms vmdb]# systemctl restart evmserverd
Job for evmserverd.service failed because the control process exited with error code. See "systemctl status evmserverd.service" and "journalctl -xe" for details.

[root@cloudforms log]# systemctl status evmserverd.service
● evmserverd.service - EVM server daemon
   Loaded: loaded (/usr/lib/systemd/system/evmserverd.service; enabled; vendor preset: disabled)
   Active: activating (start) since Tue 2019-02-05 04:49:02 EST; 914ms ago
 Main PID: 15432 (code=exited, status=0/SUCCESS);         : 35815 (evmserver.sh)
   CGroup: /system.slice/evmserverd.service
           ├─17243 /usr/lib64/erlang/erts-8.3.5.4/bin/epmd -daemon
           ├─35815 /bin/bash /bin/evmserver.sh start
           └─35820 /opt/rh/cfme-gemset/bin/rake evm:start

Feb 05 04:49:02 cloudforms.rhev.openstack.engineering.redhat.com systemd[1]: Starting EVM server daemon...
[root@cloudforms log]# journalctl -xe
Feb 05 04:49:27 cloudforms.rhev.openstack.engineering.redhat.com sh[35860]: /opt/rh/cfme-gemset/gems/activesupport-5.0.7.1/lib/active_support/de
Feb 05 04:49:27 cloudforms.rhev.openstack.engineering.redhat.com sh[35860]: /opt/rh/cfme-gemset/gems/activesupport-5.0.7.1/lib/active_support/de
Feb 05 04:49:27 cloudforms.rhev.openstack.engineering.redhat.com sh[35860]: /opt/rh/cfme-gemset/gems/activesupport-5.0.7.1/lib/active_support/de
Feb 05 04:49:27 cloudforms.rhev.openstack.engineering.redhat.com sh[35860]: /opt/rh/cfme-gemset/gems/activesupport-5.0.7.1/lib/active_support/de
Feb 05 04:49:27 cloudforms.rhev.openstack.engineering.redhat.com sh[35860]: /opt/rh/cfme-gemset/gems/activesupport-5.0.7.1/lib/active_support/de
Feb 05 04:49:27 cloudforms.rhev.openstack.engineering.redhat.com sh[35860]: /opt/rh/cfme-gemset/gems/railties-5.0.7.1/lib/rails/application.rb:3
Feb 05 04:49:27 cloudforms.rhev.openstack.engineering.redhat.com sh[35860]: /opt/rh/cfme-gemset/gems/railties-5.0.7.1/lib/rails/application.rb:4
Feb 05 04:49:27 cloudforms.rhev.openstack.engineering.redhat.com sh[35860]: /opt/rh/cfme-gemset/gems/rake-12.3.2/exe/rake:27:in `<top (required)
Feb 05 04:49:27 cloudforms.rhev.openstack.engineering.redhat.com sh[35860]: /opt/rh/cfme-gemset/gems/bundler-1.15.4/lib/bundler/cli/exec.rb:74:i
Feb 05 04:49:27 cloudforms.rhev.openstack.engineering.redhat.com sh[35860]: /opt/rh/cfme-gemset/gems/bundler-1.15.4/lib/bundler/cli/exec.rb:74:i
Feb 05 04:49:27 cloudforms.rhev.openstack.engineering.redhat.com sh[35860]: /opt/rh/cfme-gemset/gems/bundler-1.15.4/lib/bundler/cli/exec.rb:27:i
Feb 05 04:49:27 cloudforms.rhev.openstack.engineering.redhat.com sh[35860]: /opt/rh/cfme-gemset/gems/bundler-1.15.4/lib/bundler/cli.rb:362:in `e
Feb 05 04:49:27 cloudforms.rhev.openstack.engineering.redhat.com sh[35860]: /opt/rh/cfme-gemset/gems/bundler-1.15.4/lib/bundler/vendor/thor/lib/
Feb 05 04:49:27 cloudforms.rhev.openstack.engineering.redhat.com sh[35860]: /opt/rh/cfme-gemset/gems/bundler-1.15.4/lib/bundler/vendor/thor/lib/
Feb 05 04:49:27 cloudforms.rhev.openstack.engineering.redhat.com sh[35860]: /opt/rh/cfme-gemset/gems/bundler-1.15.4/lib/bundler/vendor/thor/lib/
Feb 05 04:49:27 cloudforms.rhev.openstack.engineering.redhat.com sh[35860]: /opt/rh/cfme-gemset/gems/bundler-1.15.4/lib/bundler/cli.rb:22:in `di
Feb 05 04:49:27 cloudforms.rhev.openstack.engineering.redhat.com sh[35860]: /opt/rh/cfme-gemset/gems/bundler-1.15.4/lib/bundler/vendor/thor/lib/
Feb 05 04:49:27 cloudforms.rhev.openstack.engineering.redhat.com sh[35860]: /opt/rh/cfme-gemset/gems/bundler-1.15.4/lib/bundler/cli.rb:13:in `st
Feb 05 04:49:27 cloudforms.rhev.openstack.engineering.redhat.com sh[35860]: /opt/rh/cfme-gemset/gems/bundler-1.15.4/exe/bundle:30:in `block in <
Feb 05 04:49:27 cloudforms.rhev.openstack.engineering.redhat.com sh[35860]: /opt/rh/cfme-gemset/gems/bundler-1.15.4/lib/bundler/friendly_errors.
Feb 05 04:49:27 cloudforms.rhev.openstack.engineering.redhat.com systemd[1]: evmserverd.service: control process exited, code=exited status=1
Feb 05 04:49:27 cloudforms.rhev.openstack.engineering.redhat.com sh[35860]: /opt/rh/cfme-gemset/gems/bundler-1.15.4/exe/bundle:22:in `<top (requ
Feb 05 04:49:27 cloudforms.rhev.openstack.engineering.redhat.com sh[35860]: /opt/rh/cfme-gemset/bin/bundle:23:in `load'
Feb 05 04:49:27 cloudforms.rhev.openstack.engineering.redhat.com sh[35860]: /opt/rh/cfme-gemset/bin/bundle:23:in `<main>'
Feb 05 04:49:27 cloudforms.rhev.openstack.engineering.redhat.com sh[35860]: Tasks: TOP => evm:start => environment
Feb 05 04:49:27 cloudforms.rhev.openstack.engineering.redhat.com sh[35860]: (See full trace by running task with --trace)
Feb 05 04:49:27 cloudforms.rhev.openstack.engineering.redhat.com systemd[1]: Failed to start EVM server daemon.
-- Subject: Unit evmserverd.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit evmserverd.service has failed.
-- 
-- The result is failed.
Feb 05 04:49:27 cloudforms.rhev.openstack.engineering.redhat.com systemd[1]: Unit evmserverd.service entered failed state.
Feb 05 04:49:27 cloudforms.rhev.openstack.engineering.redhat.com systemd[1]: evmserverd.service failed.
Feb 05 04:49:27 cloudforms.rhev.openstack.engineering.redhat.com systemd[1]: evmserverd.service holdoff time over, scheduling restart.
Feb 05 04:49:27 cloudforms.rhev.openstack.engineering.redhat.com systemd[1]: Stopped EVM server daemon.
-- Subject: Unit evmserverd.service has finished shutting down
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit evmserverd.service has finished shutting down.
Feb 05 04:49:27 cloudforms.rhev.openstack.engineering.redhat.com systemd[1]: Starting EVM server daemon...
-- Subject: Unit evmserverd.service has begun start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit evmserverd.service has begun starting up.

Ilanit, this error usually happens when you restore a database with passwords but don't copy over the v2_key from the vmdb/certs directory.

I'm not sure what was the process followed to create this bug.  Can you review the v2_key to see if that could be the problem?

Thanks Joe.

Database restore was not done.
I don't know what caused this problem.
but there are 2 things bothering me:
1. I didn't do anything that is not considered as normal usage of the system.
2. I encountered it more than once, on different CFME versions (also latest)

How can I review the v2_key please?

Hi Ilanit.

The error I'm talking about is:
can not decrypt v2_key encrypted string (MiqPassword::MiqPasswordError)

If you're getting that error and the evmserverd process will not start, you need to check your v2_key.  If you are restoring a db or connecting to another db, you need to use the v2_key from the system that created the passwords.

The v2_key is  an encryption key found in vmdb/certs/v2_key.  We generate this on new installations and passwords that are stored in the database are encrypted using this key.  Therefore, you must copy the v2_key when you export/restore a database.

Are you using a brand new database? Is it possible someone connected to your database and stored a password using a different key?

Joe,

Thanks for explaining that.

I am not aware of any db connection trials.

The CFME with the v2_key issue no longer exist.

I will update the bug, once I'll hit it again.

Closing for now, let's reopen when we have more information.

I had this failure again.

I think this is happening after I run oVirt.manageiq CFME deploy on RHV, without removing the CFME VM first.
I suspect that something done in the database init part may cause this.

Normally, in testing the CFME is deployed each time from new.
But in this case it is not.

Reopenning this bug, as it is reproducing quite often, and we now have some direction.

I can provide env if required.

Now I am certain that this is happening when I run Ansible automation for CFME deploy, without removing the CFME VM.
The problem can be reproduced.

This is the step in the oVirt.manageiq role that causes the failure:

TASK [oVirt.manageiq : Initialize CloudForms] **********************************
changed: [localhost -> 10.12.69.26]

> after I run oVirt.manageiq CFME deploy on RHV, without removing the CFME VM first

Thanks Ilanit.  I'm unfamiliar with the RHV tems you're using here.  Can explain what you mean by CFME deploy on RHV without removing the CFME VM first?  Are you importing the downloaded appliance image onto RHV storage, creating a vm from it and starting it? Or are you using some automation to deploy the CFME?  If so, where is this automation?  Where is the source code for the ansible/automate scripts that deploy this? What do you mean by removing the CFME Vm first?  Is this new appliance being deployed to connect to the existing CFME vm's database?  Can you compare the hostname/ip addresses of the two CFME appliances and compare them to their config/database.yml files?  

Thanks!

This is the ansible role I use:

Ansible role to create ManageIQ or CloudForms virtual machine from qcow image on RHV": 
https://github.com/oVirt/ovirt-ansible-manageiq
(It is delivered within RHV).

This is the flow that generates the bug:

1. Run this role. As a result a new CFME VM is created on RHV.
   It's database is initiated, and rails console works OK.
2. Do not delete the CFME VM.
3. Run the role again.
   The rails console fail on v2_key error.
 
 
We run automation for v2v flow testing, and the CFME VM deploy is the first step.
If we run this flow for the second time, without removing the CFME VM beforehand,
it will create the v2_key error. 

I shall send the failing CFME by email.

I'm really unfamiliar with ovirt-ansible-manageiq.  Can you try something?

1) After your step 1, copy off the vmdb/certs/v2_key
2) After your step 3, verify if you get a v2_key decrypt error.  If you do, backup your existing vmdb/certs/v2_key.  Then, copy over the v2_key from step 1 and see if that v2_key encounters the same v2_key error or if it works.

If it works in step 2, then the initialization of the ovirt-ansible-manageiq appliance db/v2_key has a bug.  I'm not sure how to get that fixed.  I can reassign it to a different group perhaps.

Thanks Joe,

Step 2 works.

So there is a bug in the initialization of the ovirt-ansible-manageiq appliance db/v2_key.
Comment #12 describe the bug flow.
Comment #13 describe the bug work around. 

Moving bug to Ondra - the Ansible playbook owner.

Ilanit, could you please describe your goal? Why do want to deploy more ManageIQ/CFME setups within single oVirt/RHV setup?

The main purpose of ovirt.manageiq role is to deploy ManageIQ/CFME as a VM within oVirt/RHV setup and configure oVirt/RHV provider within ManageIQ/CFME VM to be able to communicate with oVirt/RHV setup.

Martin,

This is not about deploy more CFME VMs on same RHV.

The bug is that if the oVirt.manageiq role is run twice, 
On the first run it 1.deploys a CFME VM on RHV, 2.configure it data base, and then 3. adds this CFME as a provider to the RHV.
On a second run, in case the CFME VM was not removed, it SHOULD identify all these 3 steps, were already run, and skips them.
however, what it is actually doing is initializing the data base, in such way that the v2v_key changes.
This causes the Rails Console run fail, as it cannot access the database.

A second run of the role should not harm the database access.

In IMS testing (RHV->VMware), 
We have a flow, that deploys CFME on RHV (using ovirt.manageiq role), and then does thing on CFME.
To save run time, the flow was run more than once, __without__ removing the CFME VM, in between the runs.
But then, when I needed Rails console access, it failed.

Verified in ovirt-ansible-manageiq-1.1.15-0.1.master.20190515131718.el7.noarch with ManageIQ
ansible-2.8.0-0.5.b1.el7ae.noarch

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2019:1564