Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1557735

Summary: VDSM is dead after upgrade to vdsm-4.20.22-1.el7ev.x86_64
Product: [oVirt] vdsm Reporter: Michael Burman <mburman>
Component: CoreAssignee: Martin Perina <mperina>
Status: CLOSED CURRENTRELEASE QA Contact: Michael Burman <mburman>
Severity: high Docs Contact:
Priority: unspecified    
Version: 4.20.19CC: bugs, fromani, mburman, mperina, nsoffer
Target Milestone: ovirt-4.2.3Keywords: Regression, Reopened
Target Release: ---Flags: rule-engine: ovirt-4.2+
rule-engine: blocker+
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-05-10 06:28:44 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Infra RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
Logs
none
Logs none

Description Michael Burman 2018-03-18 07:39:52 UTC 
Description of problem:
vdsm is dead after upgrade to vdsm-4.20.22-1.el7ev.x86_64

After upgrade to latest d/s vdsm build, vdsm is dead - 

Mar 18 09:01:47 zeus-vds1 systemd: Starting Virtual Desktop Server Manager...
Mar 18 09:01:47 zeus-vds1 vdsmd_init_common.sh: vdsm: Running mkdirs
Mar 18 09:01:47 zeus-vds1 vdsmd_init_common.sh: vdsm: Running configure_coredump
Mar 18 09:01:47 zeus-vds1 vdsmd_init_common.sh: vdsm: Running configure_vdsm_logs
Mar 18 09:01:47 zeus-vds1 vdsmd_init_common.sh: vdsm: Running wait_for_network
Mar 18 09:01:47 zeus-vds1 vdsmd_init_common.sh: vdsm: Running run_init_hooks
Mar 18 09:01:47 zeus-vds1 vdsmd_init_common.sh: vdsm: Running check_is_configured
Mar 18 09:01:48 zeus-vds1 vdsmd_init_common.sh: Error:
Mar 18 09:01:48 zeus-vds1 vdsmd_init_common.sh: One of the modules is not configured to work with VDSM.
Mar 18 09:01:48 zeus-vds1 vdsmd_init_common.sh: To configure the module use the following:
Mar 18 09:01:48 zeus-vds1 vdsmd_init_common.sh: 'vdsm-tool configure [--module module-name]'.
Mar 18 09:01:48 zeus-vds1 vdsmd_init_common.sh: If all modules are not configured try to use:
Mar 18 09:01:48 zeus-vds1 vdsmd_init_common.sh: 'vdsm-tool configure --force'
Mar 18 09:01:48 zeus-vds1 vdsmd_init_common.sh: (The force flag will stop the module's service and start it
Mar 18 09:01:48 zeus-vds1 vdsmd_init_common.sh: afterwards automatically to load the new configuration.)
Mar 18 09:01:48 zeus-vds1 vdsmd_init_common.sh: abrt is already configured for vdsm
Mar 18 09:01:48 zeus-vds1 vdsmd_init_common.sh: lvm is configured for vdsm
Mar 18 09:01:48 zeus-vds1 vdsmd_init_common.sh: libvirt is already configured for vdsm
Mar 18 09:01:48 zeus-vds1 vdsmd_init_common.sh: Current revision of multipath.conf detected, preserving
Mar 18 09:01:48 zeus-vds1 vdsmd_init_common.sh: schema should be configuredModules schema are not configured
Mar 18 09:01:48 zeus-vds1 vdsmd_init_common.sh: vdsm: stopped during execute check_is_configured task (task returned with error code 1).
Mar 18 09:01:48 zeus-vds1 systemd: vdsmd.service: control process exited, code=exited status=1
Mar 18 09:01:48 zeus-vds1 systemd: Failed to start Virtual Desktop Server Manager.
Mar 18 09:01:48 zeus-vds1 systemd: Dependency failed for MOM instance configured for VDSM purposes.
Mar 18 09:01:48 zeus-vds1 systemd: Job mom-vdsm.service/start failed with result 'dependency'.
Mar 18 09:01:48 zeus-vds1 systemd: Unit vdsmd.service entered failed state.
Mar 18 09:01:48 zeus-vds1 systemd: vdsmd.service failed.
Mar 18 09:01:48 zeus-vds1 systemd: Cannot add dependency job for unit lvm2-lvmetad.socket, ignoring: Invalid request descriptor
Mar 18 09:01:49 zeus-vds1 systemd: vdsmd.service holdoff time over, scheduling restart.
Mar 18 09:01:49 zeus-vds1 systemd: Cannot add dependency job for unit lvm2-lvmetad.socket, ignoring: Unit is masked.


Version-Release number of selected component (if applicable):
vdsm-4.20.22-1.el7ev.x86_64

How reproducible:
65% from vdsm 4.20.20 > 4.20.22
From 7 servers i have updated 4 have failed to start

Steps to Reproduce:
1. Set host to maintenance and update to latest vdsm 4.20.22 

Actual results:
vdsmd is dead

Expected results:
vdsmd must be alive after upgrade/update
Comment 1 Michael Burman 2018-03-18 07:40:16 UTC 
Created attachment 1409415 [details]
Logs
Comment 2 Michael Burman 2018-03-18 08:00:56 UTC 
Bug reproduce around 90% , Almost all my hosts(10) failed to come up after vdsm update, vdsm is dead.
Tested 3 different scenarios -
- Host in maintenance during vdsm upgrade(officially recommended)
- Host is UP in RHV - (this is the way i do updates)
- Host with rhel 7.4 as well

In all cases vdsmd is dead after update and cannot be activated in RHV, stay in non-responsive state
Comment 3 Dan Kenigsberg 2018-03-20 10:30:59 UTC 
Why is this placed on Network? Isn't it a general Infra thingy? Any more info regarding

Mar 18 09:01:48 zeus-vds1 vdsmd_init_common.sh: schema should be configuredModules schema are not configured
Mar 18 09:01:48 zeus-vds1 vdsmd_init_common.sh: vdsm: stopped during execute check_is_configured task (task returned with error code 1).
Comment 4 Michael Burman 2018-03-20 11:39:49 UTC 
(In reply to Dan Kenigsberg from comment #3)
> Why is this placed on Network? Isn't it a general Infra thingy? Any more
> info regarding
> 
> Mar 18 09:01:48 zeus-vds1 vdsmd_init_common.sh: schema should be
> configuredModules schema are not configured
> Mar 18 09:01:48 zeus-vds1 vdsmd_init_common.sh: vdsm: stopped during execute
> check_is_configured task (task returned with error code 1).

I wasn't sure if it's ours or Infra.
Comment 5 Red Hat Bugzilla Rules Engine 2018-03-21 09:19:40 UTC 
This bug report has Keywords: Regression or TestBlocker.
Since no regressions or test blockers are allowed between releases, it is also being identified as a blocker for this release. Please resolve ASAP.
Comment 6 Yaniv Bronhaim 2018-03-21 10:26:27 UTC 
did you use yum upgrade or you removed and reinstalled vdsm? using yum upgrade should perform for you "vdsm-tool configure" as part of the rpm installation (unless we have a bug in recognizing the upgrade which seems alright to me), if you removed and then install, you should run it manually
Comment 7 Yaniv Bronhaim 2018-03-21 11:22:57 UTC 
In the following after installing v4.20.20 and starting vdsmd I upgraded to 4.20.22

... snip

# vdsm_install
Loaded plugins: fastestmirror
Repository extras is listed more than once in the configuration
Repository centosplus is listed more than once in the configuration
Repository centos-opstools-release is listed more than once in the configuration
Examining /root/rpmbuild/RPMS/noarch/vdsm-jsonrpc-4.20.22-1.el7.centos.noarch.rpm: vdsm-jsonrpc-4.20.22-1.el7.centos.noarch
Marking /root/rpmbuild/RPMS/noarch/vdsm-jsonrpc-4.20.22-1.el7.centos.noarch.rpm as an update to vdsm-jsonrpc-4.20.20-1.el7.centos.noarch
Examining /root/rpmbuild/RPMS/noarch/vdsm-python-4.20.22-1.el7.centos.noarch.rpm: vdsm-python-4.20.22-1.el7.centos.noarch
Marking /root/rpmbuild/RPMS/noarch/vdsm-python-4.20.22-1.el7.centos.noarch.rpm as an update to vdsm-python-4.20.20-1.el7.centos.noarch
Examining /root/rpmbuild/RPMS/noarch/vdsm-common-4.20.22-1.el7.centos.noarch.rpm: vdsm-common-4.20.22-1.el7.centos.noarch
Marking /root/rpmbuild/RPMS/noarch/vdsm-common-4.20.22-1.el7.centos.noarch.rpm as an update to vdsm-common-4.20.20-1.el7.centos.noarch
Examining /root/rpmbuild/RPMS/noarch/vdsm-api-4.20.22-1.el7.centos.noarch.rpm: vdsm-api-4.20.22-1.el7.centos.noarch
Marking /root/rpmbuild/RPMS/noarch/vdsm-api-4.20.22-1.el7.centos.noarch.rpm as an update to vdsm-api-4.20.20-1.el7.centos.noarch
Examining /root/rpmbuild/RPMS/noarch/vdsm-hook-vmfex-dev-4.20.22-1.el7.centos.noarch.rpm: vdsm-hook-vmfex-dev-4.20.22-1.el7.centos.noarch
Marking /root/rpmbuild/RPMS/noarch/vdsm-hook-vmfex-dev-4.20.22-1.el7.centos.noarch.rpm as an update to vdsm-hook-vmfex-dev-4.20.20-1.el7.centos.noarch
Examining /root/rpmbuild/RPMS/noarch/vdsm-yajsonrpc-4.20.22-1.el7.centos.noarch.rpm: vdsm-yajsonrpc-4.20.22-1.el7.centos.noarch
Marking /root/rpmbuild/RPMS/noarch/vdsm-yajsonrpc-4.20.22-1.el7.centos.noarch.rpm as an update to vdsm-yajsonrpc-4.20.20-1.el7.centos.noarch
Examining /root/rpmbuild/RPMS/noarch/vdsm-http-4.20.22-1.el7.centos.noarch.rpm: vdsm-http-4.20.22-1.el7.centos.noarch
Marking /root/rpmbuild/RPMS/noarch/vdsm-http-4.20.22-1.el7.centos.noarch.rpm as an update to vdsm-http-4.20.20-1.el7.centos.noarch
Examining /root/rpmbuild/RPMS/noarch/vdsm-tests-4.20.22-1.el7.centos.noarch.rpm: vdsm-tests-4.20.22-1.el7.centos.noarch
Marking /root/rpmbuild/RPMS/noarch/vdsm-tests-4.20.22-1.el7.centos.noarch.rpm as an update to vdsm-tests-4.20.20-1.el7.centos.noarch
Examining /root/rpmbuild/RPMS/noarch/vdsm-client-4.20.22-1.el7.centos.noarch.rpm: vdsm-client-4.20.22-1.el7.centos.noarch
Marking /root/rpmbuild/RPMS/noarch/vdsm-client-4.20.22-1.el7.centos.noarch.rpm as an update to vdsm-client-4.20.20-1.el7.centos.noarch
Examining /root/rpmbuild/RPMS/x86_64/vdsm-network-4.20.22-1.el7.centos.x86_64.rpm: vdsm-network-4.20.22-1.el7.centos.x86_64
Marking /root/rpmbuild/RPMS/x86_64/vdsm-network-4.20.22-1.el7.centos.x86_64.rpm as an update to vdsm-network-4.20.20-1.el7.centos.x86_64
Examining /root/rpmbuild/RPMS/x86_64/vdsm-4.20.22-1.el7.centos.x86_64.rpm: vdsm-4.20.22-1.el7.centos.x86_64
Marking /root/rpmbuild/RPMS/x86_64/vdsm-4.20.22-1.el7.centos.x86_64.rpm as an update to vdsm-4.20.20-1.el7.centos.x86_64
Resolving Dependencies
--> Running transaction check
---> Package vdsm.x86_64 0:4.20.20-1.el7.centos will be updated
---> Package vdsm.x86_64 0:4.20.22-1.el7.centos will be an update
---> Package vdsm-api.noarch 0:4.20.20-1.el7.centos will be updated
---> Package vdsm-api.noarch 0:4.20.22-1.el7.centos will be an update
---> Package vdsm-client.noarch 0:4.20.20-1.el7.centos will be updated
---> Package vdsm-client.noarch 0:4.20.22-1.el7.centos will be an update
---> Package vdsm-common.noarch 0:4.20.20-1.el7.centos will be updated
---> Package vdsm-common.noarch 0:4.20.22-1.el7.centos will be an update
---> Package vdsm-hook-vmfex-dev.noarch 0:4.20.20-1.el7.centos will be updated
---> Package vdsm-hook-vmfex-dev.noarch 0:4.20.22-1.el7.centos will be an update
---> Package vdsm-http.noarch 0:4.20.20-1.el7.centos will be updated
---> Package vdsm-http.noarch 0:4.20.22-1.el7.centos will be an update
---> Package vdsm-jsonrpc.noarch 0:4.20.20-1.el7.centos will be updated
---> Package vdsm-jsonrpc.noarch 0:4.20.22-1.el7.centos will be an update
---> Package vdsm-network.x86_64 0:4.20.20-1.el7.centos will be updated
---> Package vdsm-network.x86_64 0:4.20.22-1.el7.centos will be an update
---> Package vdsm-python.noarch 0:4.20.20-1.el7.centos will be updated
---> Package vdsm-python.noarch 0:4.20.22-1.el7.centos will be an update
---> Package vdsm-tests.noarch 0:4.20.20-1.el7.centos will be updated
---> Package vdsm-tests.noarch 0:4.20.22-1.el7.centos will be an update
---> Package vdsm-yajsonrpc.noarch 0:4.20.20-1.el7.centos will be updated
---> Package vdsm-yajsonrpc.noarch 0:4.20.22-1.el7.centos will be an update
--> Finished Dependency Resolution

Dependencies Resolved

=========================================================================================================================================================================================
 Package                                 Arch                       Version                                   Repository                                                            Size
=========================================================================================================================================================================================
Updating:
 vdsm                                    x86_64                     4.20.22-1.el7.centos                      /vdsm-4.20.22-1.el7.centos.x86_64                                    186 k
 vdsm-api                                noarch                     4.20.22-1.el7.centos                      /vdsm-api-4.20.22-1.el7.centos.noarch                                794 k
 vdsm-client                             noarch                     4.20.22-1.el7.centos                      /vdsm-client-4.20.22-1.el7.centos.noarch                              65 k
 vdsm-common                             noarch                     4.20.22-1.el7.centos                      /vdsm-common-4.20.22-1.el7.centos.noarch                             445 k
 vdsm-hook-vmfex-dev                     noarch                     4.20.22-1.el7.centos                      /vdsm-hook-vmfex-dev-4.20.22-1.el7.centos.noarch                      21 k
 vdsm-http                               noarch                     4.20.22-1.el7.centos                      /vdsm-http-4.20.22-1.el7.centos.noarch                                23 k
 vdsm-jsonrpc                            noarch                     4.20.22-1.el7.centos                      /vdsm-jsonrpc-4.20.22-1.el7.centos.noarch                             83 k
 vdsm-network                            x86_64                     4.20.22-1.el7.centos                      /vdsm-network-4.20.22-1.el7.centos.x86_64                            1.1 M
 vdsm-python                             noarch                     4.20.22-1.el7.centos                      /vdsm-python-4.20.22-1.el7.centos.noarch                             4.9 M
 vdsm-tests                              noarch                     4.20.22-1.el7.centos                      /vdsm-tests-4.20.22-1.el7.centos.noarch                              5.5 M
 vdsm-yajsonrpc                          noarch                     4.20.22-1.el7.centos                      /vdsm-yajsonrpc-4.20.22-1.el7.centos.noarch                          126 k

Transaction Summary
=========================================================================================================================================================================================
Upgrade  11 Packages

Total size: 13 M
Downloading packages:
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Updating   : vdsm-common-4.20.22-1.el7.centos.noarch                                                                                                                              1/22 
  Updating   : vdsm-api-4.20.22-1.el7.centos.noarch                                                                                                                                 2/22 
  Updating   : vdsm-network-4.20.22-1.el7.centos.x86_64                                                                                                                             3/22 
  Updating   : vdsm-python-4.20.22-1.el7.centos.noarch                                                                                                                              4/22 
  Updating   : vdsm-yajsonrpc-4.20.22-1.el7.centos.noarch                                                                                                                           5/22 
  Updating   : vdsm-jsonrpc-4.20.22-1.el7.centos.noarch                                                                                                                             6/22 
  Updating   : vdsm-http-4.20.22-1.el7.centos.noarch                                                                                                                                7/22 
  Updating   : vdsm-hook-vmfex-dev-4.20.22-1.el7.centos.noarch                                                                                                                      8/22 
  Updating   : vdsm-4.20.22-1.el7.centos.x86_64                                                                                                                                     9/22 
  Updating   : vdsm-tests-4.20.22-1.el7.centos.noarch                                                                                                                              10/22 
  Updating   : vdsm-client-4.20.22-1.el7.centos.noarch                                                                                                                             11/22 
  Cleanup    : vdsm-tests-4.20.20-1.el7.centos.noarch                                                                                                                              12/22 
  Cleanup    : vdsm-hook-vmfex-dev-4.20.20-1.el7.centos.noarch                                                                                                                     13/22 
  Cleanup    : vdsm-4.20.20-1.el7.centos.x86_64                                                                                                                                    14/22 
  Cleanup    : vdsm-jsonrpc-4.20.20-1.el7.centos.noarch                                                                                                                            15/22 
  Cleanup    : vdsm-client-4.20.20-1.el7.centos.noarch                                                                                                                             16/22 
  Cleanup    : vdsm-http-4.20.20-1.el7.centos.noarch                                                                                                                               17/22 
  Cleanup    : vdsm-python-4.20.20-1.el7.centos.noarch                                                                                                                             18/22 
  Cleanup    : vdsm-network-4.20.20-1.el7.centos.x86_64                                                                                                                            19/22 
  Cleanup    : vdsm-common-4.20.20-1.el7.centos.noarch                                                                                                                             20/22 
  Cleanup    : vdsm-api-4.20.20-1.el7.centos.noarch                                                                                                                                21/22 
  Cleanup    : vdsm-yajsonrpc-4.20.20-1.el7.centos.noarch                                                                                                                          22/22 
  Verifying  : vdsm-jsonrpc-4.20.22-1.el7.centos.noarch                                                                                                                             1/22 
  Verifying  : vdsm-4.20.22-1.el7.centos.x86_64                                                                                                                                     2/22 
  Verifying  : vdsm-client-4.20.22-1.el7.centos.noarch                                                                                                                              3/22 
  Verifying  : vdsm-http-4.20.22-1.el7.centos.noarch                                                                                                                                4/22 
  Verifying  : vdsm-api-4.20.22-1.el7.centos.noarch                                                                                                                                 5/22 
  Verifying  : vdsm-tests-4.20.22-1.el7.centos.noarch                                                                                                                               6/22 
  Verifying  : vdsm-network-4.20.22-1.el7.centos.x86_64                                                                                                                             7/22 
  Verifying  : vdsm-yajsonrpc-4.20.22-1.el7.centos.noarch                                                                                                                           8/22 
  Verifying  : vdsm-python-4.20.22-1.el7.centos.noarch                                                                                                                              9/22 
  Verifying  : vdsm-common-4.20.22-1.el7.centos.noarch                                                                                                                             10/22 
  Verifying  : vdsm-hook-vmfex-dev-4.20.22-1.el7.centos.noarch                                                                                                                     11/22 
  Verifying  : vdsm-yajsonrpc-4.20.20-1.el7.centos.noarch                                                                                                                          12/22 
  Verifying  : vdsm-4.20.20-1.el7.centos.x86_64                                                                                                                                    13/22 
  Verifying  : vdsm-python-4.20.20-1.el7.centos.noarch                                                                                                                             14/22 
  Verifying  : vdsm-client-4.20.20-1.el7.centos.noarch                                                                                                                             15/22 
  Verifying  : vdsm-common-4.20.20-1.el7.centos.noarch                                                                                                                             16/22 
  Verifying  : vdsm-api-4.20.20-1.el7.centos.noarch                                                                                                                                17/22 
  Verifying  : vdsm-jsonrpc-4.20.20-1.el7.centos.noarch                                                                                                                            18/22 
  Verifying  : vdsm-tests-4.20.20-1.el7.centos.noarch                                                                                                                              19/22 
  Verifying  : vdsm-http-4.20.20-1.el7.centos.noarch                                                                                                                               20/22 
  Verifying  : vdsm-network-4.20.20-1.el7.centos.x86_64                                                                                                                            21/22 
  Verifying  : vdsm-hook-vmfex-dev-4.20.20-1.el7.centos.noarch                                                                                                                     22/22 

Updated:
  vdsm.x86_64 0:4.20.22-1.el7.centos                 vdsm-api.noarch 0:4.20.22-1.el7.centos    vdsm-client.noarch 0:4.20.22-1.el7.centos     vdsm-common.noarch 0:4.20.22-1.el7.centos  
  vdsm-hook-vmfex-dev.noarch 0:4.20.22-1.el7.centos  vdsm-http.noarch 0:4.20.22-1.el7.centos   vdsm-jsonrpc.noarch 0:4.20.22-1.el7.centos    vdsm-network.x86_64 0:4.20.22-1.el7.centos 
  vdsm-python.noarch 0:4.20.22-1.el7.centos          vdsm-tests.noarch 0:4.20.22-1.el7.centos  vdsm-yajsonrpc.noarch 0:4.20.22-1.el7.centos 

Complete!

Checking configuration status...

abrt is already configured for vdsm
lvm is configured for vdsm
libvirt is already configured for vdsm
SUCCESS: ssl configured to true. No conflicts
Current revision of multipath.conf detected, preserving
schema is already configured
Running configure...
Reconfiguration of abrt is done.
Reconfiguration of passwd is done.
Reconfiguration of libvirt is done.

Done configuring modules to VDSM.
[root@reserved-0-241 vdsm]# service vdsmd status
Redirecting to /bin/systemctl status vdsmd.service
● vdsmd.service - Virtual Desktop Server Manager
   Loaded: loaded (/usr/lib/systemd/system/vdsmd.service; enabled; vendor preset: enabled)
   Active: active (running) since Wed 2018-03-21 13:19:09 IST; 31s ago
  Process: 10366 ExecStopPost=/usr/libexec/vdsm/vdsmd_init_common.sh --post-stop (code=exited, status=0/SUCCESS)
  Process: 10637 ExecStartPre=/usr/libexec/vdsm/vdsmd_init_common.sh --pre-start (code=exited, status=0/SUCCESS)
 Main PID: 10824 (vdsmd)
   CGroup: /system.slice/vdsmd.service
           ├─10824 /usr/bin/python2 /usr/share/vdsm/vdsmd
           ├─10922 /usr/libexec/ioprocess --read-pipe-fd 49 --write-pipe-fd 48 --max-threads 10 --max-queued-requests 10
           ├─10929 /usr/libexec/ioprocess --read-pipe-fd 54 --write-pipe-fd 51 --max-threads 10 --max-queued-requests 10
           ├─10937 /usr/libexec/ioprocess --read-pipe-fd 57 --write-pipe-fd 54 --max-threads 10 --max-queued-requests 10
           ├─10978 /usr/libexec/ioprocess --read-pipe-fd 72 --write-pipe-fd 71 --max-threads 10 --max-queued-requests 10
           └─10999 /usr/libexec/ioprocess --read-pipe-fd 85 --write-pipe-fd 84 --max-threads 10 --max-queued-requests 10

Mar 21 13:19:10 reserved-0-241.tlv.redhat.com sudo[10920]: PAM adding faulty module: /usr/lib64/security/pam_krb5.so
Mar 21 13:19:11 reserved-0-241.tlv.redhat.com vdsm[10824]: WARN Not ready yet, ignoring event '|virt|VM_status|29b896da-79cf-464d-9504-ce6e14e71b45' args={'29b896da-79cf-464d-9504-ce...
Mar 21 13:19:11 reserved-0-241.tlv.redhat.com vdsm[10824]: WARN Not ready yet, ignoring event '|virt|VM_status|2fbdbf7f-b3d3-46dd-b9d7-e6f80608c581' args={'2fbdbf7f-b3d3-46dd-b9d7-e6...
Mar 21 13:19:11 reserved-0-241.tlv.redhat.com vdsm[10824]: WARN MOM not available.
Mar 21 13:19:11 reserved-0-241.tlv.redhat.com vdsm[10824]: WARN MOM not available, KSM stats will be missing.
Mar 21 13:19:14 reserved-0-241.tlv.redhat.com sudo[10985]: PAM unable to dlopen(/usr/lib64/security/pam_krb5.so): /usr/lib64/security/pam_krb5.so: cannot open shared object... directory
Mar 21 13:19:14 reserved-0-241.tlv.redhat.com sudo[10985]: PAM adding faulty module: /usr/lib64/security/pam_krb5.so
Mar 21 13:19:14 reserved-0-241.tlv.redhat.com sudo[10995]: PAM unable to dlopen(/usr/lib64/security/pam_krb5.so): /usr/lib64/security/pam_krb5.so: cannot open shared object... directory
Mar 21 13:19:14 reserved-0-241.tlv.redhat.com sudo[10995]: PAM adding faulty module: /usr/lib64/security/pam_krb5.so
Mar 21 13:19:14 reserved-0-241.tlv.redhat.com vdsm[10824]: WARN MOM not available, Policy could not be set.
Hint: Some lines were ellipsized, use -l to show in full.


vdsm up and running. the schema module is already configured if 4.20.20 ran.

If this still doesn't work for you please submit the output of 
# ll /var/cache/vdsm/schema/vdsm*
before and after the upgrade

Thanks.
Comment 8 Michael Burman 2018-03-21 12:40:38 UTC 
(In reply to Yaniv Bronhaim from comment #6)
> did you use yum upgrade or you removed and reinstalled vdsm? using yum
> upgrade should perform for you "vdsm-tool configure" as part of the rpm
> installation (unless we have a bug in recognizing the upgrade which seems
> alright to me), if you removed and then install, you should run it manually

I used yum update only)
Comment 9 Yaniv Bronhaim 2018-03-22 09:38:44 UTC 
We can't manage to reproduce it again - although it appeared in more than one env. I tried few upgrade flows that worked as expected (4.1>latest, 4.2>latest, 4.20.20>4.20.23), we call the schema re-creation after the upgrade. Please re-open immediately in case it appears again
Comment 10 Michael Burman 2018-03-25 14:05:08 UTC 
(In reply to Yaniv Bronhaim from comment #9)
> We can't manage to reproduce it again - although it appeared in more than
> one env. I tried few upgrade flows that worked as expected (4.1>latest,
> 4.2>latest, 4.20.20>4.20.23), we call the schema re-creation after the
> upgrade. Please re-open immediately in case it appears again

W ehave a reprodcution - 

Mar 25 15:53:36 orchid-vds2 vdsmd_init_common.sh: schema should be configuredModules schema are not configured
Mar 25 15:53:36 orchid-vds2 vdsmd_init_common.sh: vdsm: stopped during execute check_is_configured task (task returned with error code 1).
Mar 25 15:53:36 orchid-vds2 systemd: vdsmd.service: control process exited, code=exited status=1

upgraded vdsm from 4.20.20 >> 4.20.23
I sent you the details in the mail.
Please take a look ASAP, keeping the host for you.
Comment 11 Yaniv Bronhaim 2018-03-26 08:12:49 UTC 
I suspect that something restarts vdsm service during the upgrade before we get to the %post install of the spec

Those prints come when the spec gets to the part that configure and start the service - if the service was up before.

Mar 25 15:54:16 orchid-vds2 systemd: Stopped Auxiliary vdsm service for running helper functions as root.
Mar 25 15:54:16 orchid-vds2 systemd: Stopped Virtual Desktop Server Manager network restoration.
Mar 25 15:54:16 orchid-vds2 systemd: Stopping Virtual Desktop Server Manager network restoration...
Mar 25 15:54:20 orchid-vds2 systemd: Stopping Virtualization daemon...
Mar 25 15:54:20 orchid-vds2 systemd: Stopped Virtualization daemon.
Mar 25 15:54:20 orchid-vds2 saslpasswd2: error deleting entry from sasldb: BDB0073 DB_NOTFOUND: No matching key/data pair found
Mar 25 15:54:20 orchid-vds2 saslpasswd2: error deleting entry from sasldb: BDB0073 DB_NOTFOUND: No matching key/data pair found
Mar 25 15:54:20 orchid-vds2 saslpasswd2: error deleting entry from sasldb: BDB0073 DB_NOTFOUND: No matching key/data pair found
Mar 25 15:54:21 orchid-vds2 systemd: Cannot add dependency job for unit lvm2-lvmetad.socket, ignoring: Unit is masked.
Mar 25 15:54:21 orchid-vds2 systemd: Starting Virtualization daemon...
Mar 25 15:54:21 orchid-vds2 systemd: Started Virtualization daemon.


But before it fails to start - it seems like something just after the upgrade stop and start vdsm service


see the following-
upgrade finished here:
Mar 25 15:52:44 orchid-vds2 yum[32280]: Updated: python-perf-3.10.0-862.el7.x86_64

Maybe this ?
Mar 25 15:52:47 orchid-vds2 systemd: Starting oVirt ImageIO Daemon...
?
Mar 25 15:52:48 orchid-vds2 systemd: Started Gofer Agent..
?
Mar 25 15:52:53 orchid-vds2 ovn-ctl: Starting ovn-controller [  OK  ] 
?
Mar 25 15:53:06 orchid-vds2 systemd: Starting Kernel Samepage Merging...
( I don't have those in my env)
then 
Mar 25 15:53:25 orchid-vds2 systemd: Stopping Virtualization daemon...
Mar 25 15:53:25 orchid-vds2 systemd: Starting Virtualization daemon...


Then many fails, and only in Mar 25 15:54:16 we see that we get to the %post part of the spec - vdsm already down, so we left only with configured vdsm without trying to start it.
between 15:52:44 to 15:54:16 its 1:30min, at my env its 00:03:32 - 00:04:01 


Did we commit something that restarts vdsmd probably due to new systemd dependencies after upgrade?
Comment 12 Yaniv Bronhaim 2018-03-26 13:46:01 UTC 
I sent 2 choices to fix this issue:
https://gerrit.ovirt.org/#/c/89446/ - checking vdsm status before starting the upgrade. here we stop the service during the upgrade - I think this approach might be the best if indeed restart to vdsmd appears more than only in the %post part during the upgrade
https://gerrit.ovirt.org/#/c/89441/ - create the schema during installation and removes it from configurator
Comment 13 Nir Soffer 2018-03-26 18:42:49 UTC 
(In reply to Yaniv Bronhaim from comment #11)
> Did we commit something that restarts vdsmd probably due to new systemd
> dependencies after upgrade?

I don't know adout such change, please check with Dan and Francesco.
Comment 14 Francesco Romani 2018-03-27 10:14:38 UTC 
(In reply to Nir Soffer from comment #13)
> (In reply to Yaniv Bronhaim from comment #11)
> > Did we commit something that restarts vdsmd probably due to new systemd
> > dependencies after upgrade?
> 
> I don't know adout such change, please check with Dan and Francesco.

Quick update:
As far as I know, we didn't. Will review the bigger picture later.
Comment 15 Martin Perina 2018-04-04 12:45:49 UTC 
Back to POST, we don't have all patches merged yet
Comment 16 Martin Perina 2018-04-04 18:13:50 UTC 
After offline discussion moving back to MODIFIED, https://gerrit.ovirt.org/89446 is not required to fix this upgrade issue
Comment 19 Michael Burman 2018-04-17 10:40:55 UTC 
Verified on - vdsm-4.20.25-1.el7ev.x86_64
Comment 21 Sandro Bonazzola 2018-05-10 06:28:44 UTC 
This bugzilla is included in oVirt 4.2.3 release, published on May 4th 2018.

Since the problem described in this bug report should be
resolved in oVirt 4.2.3 release, it has been closed with a resolution of CURRENT RELEASE.

If the solution does not work for you, please open a new bug report.