Description of problem: vdsm is dead after upgrade to vdsm-4.20.22-1.el7ev.x86_64 After upgrade to latest d/s vdsm build, vdsm is dead - Mar 18 09:01:47 zeus-vds1 systemd: Starting Virtual Desktop Server Manager... Mar 18 09:01:47 zeus-vds1 vdsmd_init_common.sh: vdsm: Running mkdirs Mar 18 09:01:47 zeus-vds1 vdsmd_init_common.sh: vdsm: Running configure_coredump Mar 18 09:01:47 zeus-vds1 vdsmd_init_common.sh: vdsm: Running configure_vdsm_logs Mar 18 09:01:47 zeus-vds1 vdsmd_init_common.sh: vdsm: Running wait_for_network Mar 18 09:01:47 zeus-vds1 vdsmd_init_common.sh: vdsm: Running run_init_hooks Mar 18 09:01:47 zeus-vds1 vdsmd_init_common.sh: vdsm: Running check_is_configured Mar 18 09:01:48 zeus-vds1 vdsmd_init_common.sh: Error: Mar 18 09:01:48 zeus-vds1 vdsmd_init_common.sh: One of the modules is not configured to work with VDSM. Mar 18 09:01:48 zeus-vds1 vdsmd_init_common.sh: To configure the module use the following: Mar 18 09:01:48 zeus-vds1 vdsmd_init_common.sh: 'vdsm-tool configure [--module module-name]'. Mar 18 09:01:48 zeus-vds1 vdsmd_init_common.sh: If all modules are not configured try to use: Mar 18 09:01:48 zeus-vds1 vdsmd_init_common.sh: 'vdsm-tool configure --force' Mar 18 09:01:48 zeus-vds1 vdsmd_init_common.sh: (The force flag will stop the module's service and start it Mar 18 09:01:48 zeus-vds1 vdsmd_init_common.sh: afterwards automatically to load the new configuration.) Mar 18 09:01:48 zeus-vds1 vdsmd_init_common.sh: abrt is already configured for vdsm Mar 18 09:01:48 zeus-vds1 vdsmd_init_common.sh: lvm is configured for vdsm Mar 18 09:01:48 zeus-vds1 vdsmd_init_common.sh: libvirt is already configured for vdsm Mar 18 09:01:48 zeus-vds1 vdsmd_init_common.sh: Current revision of multipath.conf detected, preserving Mar 18 09:01:48 zeus-vds1 vdsmd_init_common.sh: schema should be configuredModules schema are not configured Mar 18 09:01:48 zeus-vds1 vdsmd_init_common.sh: vdsm: stopped during execute check_is_configured task (task returned with error code 1). Mar 18 09:01:48 zeus-vds1 systemd: vdsmd.service: control process exited, code=exited status=1 Mar 18 09:01:48 zeus-vds1 systemd: Failed to start Virtual Desktop Server Manager. Mar 18 09:01:48 zeus-vds1 systemd: Dependency failed for MOM instance configured for VDSM purposes. Mar 18 09:01:48 zeus-vds1 systemd: Job mom-vdsm.service/start failed with result 'dependency'. Mar 18 09:01:48 zeus-vds1 systemd: Unit vdsmd.service entered failed state. Mar 18 09:01:48 zeus-vds1 systemd: vdsmd.service failed. Mar 18 09:01:48 zeus-vds1 systemd: Cannot add dependency job for unit lvm2-lvmetad.socket, ignoring: Invalid request descriptor Mar 18 09:01:49 zeus-vds1 systemd: vdsmd.service holdoff time over, scheduling restart. Mar 18 09:01:49 zeus-vds1 systemd: Cannot add dependency job for unit lvm2-lvmetad.socket, ignoring: Unit is masked. Version-Release number of selected component (if applicable): vdsm-4.20.22-1.el7ev.x86_64 How reproducible: 65% from vdsm 4.20.20 > 4.20.22 From 7 servers i have updated 4 have failed to start Steps to Reproduce: 1. Set host to maintenance and update to latest vdsm 4.20.22 Actual results: vdsmd is dead Expected results: vdsmd must be alive after upgrade/update
Created attachment 1409415 [details] Logs
Bug reproduce around 90% , Almost all my hosts(10) failed to come up after vdsm update, vdsm is dead. Tested 3 different scenarios - - Host in maintenance during vdsm upgrade(officially recommended) - Host is UP in RHV - (this is the way i do updates) - Host with rhel 7.4 as well In all cases vdsmd is dead after update and cannot be activated in RHV, stay in non-responsive state
Why is this placed on Network? Isn't it a general Infra thingy? Any more info regarding Mar 18 09:01:48 zeus-vds1 vdsmd_init_common.sh: schema should be configuredModules schema are not configured Mar 18 09:01:48 zeus-vds1 vdsmd_init_common.sh: vdsm: stopped during execute check_is_configured task (task returned with error code 1).
(In reply to Dan Kenigsberg from comment #3) > Why is this placed on Network? Isn't it a general Infra thingy? Any more > info regarding > > Mar 18 09:01:48 zeus-vds1 vdsmd_init_common.sh: schema should be > configuredModules schema are not configured > Mar 18 09:01:48 zeus-vds1 vdsmd_init_common.sh: vdsm: stopped during execute > check_is_configured task (task returned with error code 1). I wasn't sure if it's ours or Infra.
This bug report has Keywords: Regression or TestBlocker. Since no regressions or test blockers are allowed between releases, it is also being identified as a blocker for this release. Please resolve ASAP.
did you use yum upgrade or you removed and reinstalled vdsm? using yum upgrade should perform for you "vdsm-tool configure" as part of the rpm installation (unless we have a bug in recognizing the upgrade which seems alright to me), if you removed and then install, you should run it manually
In the following after installing v4.20.20 and starting vdsmd I upgraded to 4.20.22 ... snip # vdsm_install Loaded plugins: fastestmirror Repository extras is listed more than once in the configuration Repository centosplus is listed more than once in the configuration Repository centos-opstools-release is listed more than once in the configuration Examining /root/rpmbuild/RPMS/noarch/vdsm-jsonrpc-4.20.22-1.el7.centos.noarch.rpm: vdsm-jsonrpc-4.20.22-1.el7.centos.noarch Marking /root/rpmbuild/RPMS/noarch/vdsm-jsonrpc-4.20.22-1.el7.centos.noarch.rpm as an update to vdsm-jsonrpc-4.20.20-1.el7.centos.noarch Examining /root/rpmbuild/RPMS/noarch/vdsm-python-4.20.22-1.el7.centos.noarch.rpm: vdsm-python-4.20.22-1.el7.centos.noarch Marking /root/rpmbuild/RPMS/noarch/vdsm-python-4.20.22-1.el7.centos.noarch.rpm as an update to vdsm-python-4.20.20-1.el7.centos.noarch Examining /root/rpmbuild/RPMS/noarch/vdsm-common-4.20.22-1.el7.centos.noarch.rpm: vdsm-common-4.20.22-1.el7.centos.noarch Marking /root/rpmbuild/RPMS/noarch/vdsm-common-4.20.22-1.el7.centos.noarch.rpm as an update to vdsm-common-4.20.20-1.el7.centos.noarch Examining /root/rpmbuild/RPMS/noarch/vdsm-api-4.20.22-1.el7.centos.noarch.rpm: vdsm-api-4.20.22-1.el7.centos.noarch Marking /root/rpmbuild/RPMS/noarch/vdsm-api-4.20.22-1.el7.centos.noarch.rpm as an update to vdsm-api-4.20.20-1.el7.centos.noarch Examining /root/rpmbuild/RPMS/noarch/vdsm-hook-vmfex-dev-4.20.22-1.el7.centos.noarch.rpm: vdsm-hook-vmfex-dev-4.20.22-1.el7.centos.noarch Marking /root/rpmbuild/RPMS/noarch/vdsm-hook-vmfex-dev-4.20.22-1.el7.centos.noarch.rpm as an update to vdsm-hook-vmfex-dev-4.20.20-1.el7.centos.noarch Examining /root/rpmbuild/RPMS/noarch/vdsm-yajsonrpc-4.20.22-1.el7.centos.noarch.rpm: vdsm-yajsonrpc-4.20.22-1.el7.centos.noarch Marking /root/rpmbuild/RPMS/noarch/vdsm-yajsonrpc-4.20.22-1.el7.centos.noarch.rpm as an update to vdsm-yajsonrpc-4.20.20-1.el7.centos.noarch Examining /root/rpmbuild/RPMS/noarch/vdsm-http-4.20.22-1.el7.centos.noarch.rpm: vdsm-http-4.20.22-1.el7.centos.noarch Marking /root/rpmbuild/RPMS/noarch/vdsm-http-4.20.22-1.el7.centos.noarch.rpm as an update to vdsm-http-4.20.20-1.el7.centos.noarch Examining /root/rpmbuild/RPMS/noarch/vdsm-tests-4.20.22-1.el7.centos.noarch.rpm: vdsm-tests-4.20.22-1.el7.centos.noarch Marking /root/rpmbuild/RPMS/noarch/vdsm-tests-4.20.22-1.el7.centos.noarch.rpm as an update to vdsm-tests-4.20.20-1.el7.centos.noarch Examining /root/rpmbuild/RPMS/noarch/vdsm-client-4.20.22-1.el7.centos.noarch.rpm: vdsm-client-4.20.22-1.el7.centos.noarch Marking /root/rpmbuild/RPMS/noarch/vdsm-client-4.20.22-1.el7.centos.noarch.rpm as an update to vdsm-client-4.20.20-1.el7.centos.noarch Examining /root/rpmbuild/RPMS/x86_64/vdsm-network-4.20.22-1.el7.centos.x86_64.rpm: vdsm-network-4.20.22-1.el7.centos.x86_64 Marking /root/rpmbuild/RPMS/x86_64/vdsm-network-4.20.22-1.el7.centos.x86_64.rpm as an update to vdsm-network-4.20.20-1.el7.centos.x86_64 Examining /root/rpmbuild/RPMS/x86_64/vdsm-4.20.22-1.el7.centos.x86_64.rpm: vdsm-4.20.22-1.el7.centos.x86_64 Marking /root/rpmbuild/RPMS/x86_64/vdsm-4.20.22-1.el7.centos.x86_64.rpm as an update to vdsm-4.20.20-1.el7.centos.x86_64 Resolving Dependencies --> Running transaction check ---> Package vdsm.x86_64 0:4.20.20-1.el7.centos will be updated ---> Package vdsm.x86_64 0:4.20.22-1.el7.centos will be an update ---> Package vdsm-api.noarch 0:4.20.20-1.el7.centos will be updated ---> Package vdsm-api.noarch 0:4.20.22-1.el7.centos will be an update ---> Package vdsm-client.noarch 0:4.20.20-1.el7.centos will be updated ---> Package vdsm-client.noarch 0:4.20.22-1.el7.centos will be an update ---> Package vdsm-common.noarch 0:4.20.20-1.el7.centos will be updated ---> Package vdsm-common.noarch 0:4.20.22-1.el7.centos will be an update ---> Package vdsm-hook-vmfex-dev.noarch 0:4.20.20-1.el7.centos will be updated ---> Package vdsm-hook-vmfex-dev.noarch 0:4.20.22-1.el7.centos will be an update ---> Package vdsm-http.noarch 0:4.20.20-1.el7.centos will be updated ---> Package vdsm-http.noarch 0:4.20.22-1.el7.centos will be an update ---> Package vdsm-jsonrpc.noarch 0:4.20.20-1.el7.centos will be updated ---> Package vdsm-jsonrpc.noarch 0:4.20.22-1.el7.centos will be an update ---> Package vdsm-network.x86_64 0:4.20.20-1.el7.centos will be updated ---> Package vdsm-network.x86_64 0:4.20.22-1.el7.centos will be an update ---> Package vdsm-python.noarch 0:4.20.20-1.el7.centos will be updated ---> Package vdsm-python.noarch 0:4.20.22-1.el7.centos will be an update ---> Package vdsm-tests.noarch 0:4.20.20-1.el7.centos will be updated ---> Package vdsm-tests.noarch 0:4.20.22-1.el7.centos will be an update ---> Package vdsm-yajsonrpc.noarch 0:4.20.20-1.el7.centos will be updated ---> Package vdsm-yajsonrpc.noarch 0:4.20.22-1.el7.centos will be an update --> Finished Dependency Resolution Dependencies Resolved ========================================================================================================================================================================================= Package Arch Version Repository Size ========================================================================================================================================================================================= Updating: vdsm x86_64 4.20.22-1.el7.centos /vdsm-4.20.22-1.el7.centos.x86_64 186 k vdsm-api noarch 4.20.22-1.el7.centos /vdsm-api-4.20.22-1.el7.centos.noarch 794 k vdsm-client noarch 4.20.22-1.el7.centos /vdsm-client-4.20.22-1.el7.centos.noarch 65 k vdsm-common noarch 4.20.22-1.el7.centos /vdsm-common-4.20.22-1.el7.centos.noarch 445 k vdsm-hook-vmfex-dev noarch 4.20.22-1.el7.centos /vdsm-hook-vmfex-dev-4.20.22-1.el7.centos.noarch 21 k vdsm-http noarch 4.20.22-1.el7.centos /vdsm-http-4.20.22-1.el7.centos.noarch 23 k vdsm-jsonrpc noarch 4.20.22-1.el7.centos /vdsm-jsonrpc-4.20.22-1.el7.centos.noarch 83 k vdsm-network x86_64 4.20.22-1.el7.centos /vdsm-network-4.20.22-1.el7.centos.x86_64 1.1 M vdsm-python noarch 4.20.22-1.el7.centos /vdsm-python-4.20.22-1.el7.centos.noarch 4.9 M vdsm-tests noarch 4.20.22-1.el7.centos /vdsm-tests-4.20.22-1.el7.centos.noarch 5.5 M vdsm-yajsonrpc noarch 4.20.22-1.el7.centos /vdsm-yajsonrpc-4.20.22-1.el7.centos.noarch 126 k Transaction Summary ========================================================================================================================================================================================= Upgrade 11 Packages Total size: 13 M Downloading packages: Running transaction check Running transaction test Transaction test succeeded Running transaction Updating : vdsm-common-4.20.22-1.el7.centos.noarch 1/22 Updating : vdsm-api-4.20.22-1.el7.centos.noarch 2/22 Updating : vdsm-network-4.20.22-1.el7.centos.x86_64 3/22 Updating : vdsm-python-4.20.22-1.el7.centos.noarch 4/22 Updating : vdsm-yajsonrpc-4.20.22-1.el7.centos.noarch 5/22 Updating : vdsm-jsonrpc-4.20.22-1.el7.centos.noarch 6/22 Updating : vdsm-http-4.20.22-1.el7.centos.noarch 7/22 Updating : vdsm-hook-vmfex-dev-4.20.22-1.el7.centos.noarch 8/22 Updating : vdsm-4.20.22-1.el7.centos.x86_64 9/22 Updating : vdsm-tests-4.20.22-1.el7.centos.noarch 10/22 Updating : vdsm-client-4.20.22-1.el7.centos.noarch 11/22 Cleanup : vdsm-tests-4.20.20-1.el7.centos.noarch 12/22 Cleanup : vdsm-hook-vmfex-dev-4.20.20-1.el7.centos.noarch 13/22 Cleanup : vdsm-4.20.20-1.el7.centos.x86_64 14/22 Cleanup : vdsm-jsonrpc-4.20.20-1.el7.centos.noarch 15/22 Cleanup : vdsm-client-4.20.20-1.el7.centos.noarch 16/22 Cleanup : vdsm-http-4.20.20-1.el7.centos.noarch 17/22 Cleanup : vdsm-python-4.20.20-1.el7.centos.noarch 18/22 Cleanup : vdsm-network-4.20.20-1.el7.centos.x86_64 19/22 Cleanup : vdsm-common-4.20.20-1.el7.centos.noarch 20/22 Cleanup : vdsm-api-4.20.20-1.el7.centos.noarch 21/22 Cleanup : vdsm-yajsonrpc-4.20.20-1.el7.centos.noarch 22/22 Verifying : vdsm-jsonrpc-4.20.22-1.el7.centos.noarch 1/22 Verifying : vdsm-4.20.22-1.el7.centos.x86_64 2/22 Verifying : vdsm-client-4.20.22-1.el7.centos.noarch 3/22 Verifying : vdsm-http-4.20.22-1.el7.centos.noarch 4/22 Verifying : vdsm-api-4.20.22-1.el7.centos.noarch 5/22 Verifying : vdsm-tests-4.20.22-1.el7.centos.noarch 6/22 Verifying : vdsm-network-4.20.22-1.el7.centos.x86_64 7/22 Verifying : vdsm-yajsonrpc-4.20.22-1.el7.centos.noarch 8/22 Verifying : vdsm-python-4.20.22-1.el7.centos.noarch 9/22 Verifying : vdsm-common-4.20.22-1.el7.centos.noarch 10/22 Verifying : vdsm-hook-vmfex-dev-4.20.22-1.el7.centos.noarch 11/22 Verifying : vdsm-yajsonrpc-4.20.20-1.el7.centos.noarch 12/22 Verifying : vdsm-4.20.20-1.el7.centos.x86_64 13/22 Verifying : vdsm-python-4.20.20-1.el7.centos.noarch 14/22 Verifying : vdsm-client-4.20.20-1.el7.centos.noarch 15/22 Verifying : vdsm-common-4.20.20-1.el7.centos.noarch 16/22 Verifying : vdsm-api-4.20.20-1.el7.centos.noarch 17/22 Verifying : vdsm-jsonrpc-4.20.20-1.el7.centos.noarch 18/22 Verifying : vdsm-tests-4.20.20-1.el7.centos.noarch 19/22 Verifying : vdsm-http-4.20.20-1.el7.centos.noarch 20/22 Verifying : vdsm-network-4.20.20-1.el7.centos.x86_64 21/22 Verifying : vdsm-hook-vmfex-dev-4.20.20-1.el7.centos.noarch 22/22 Updated: vdsm.x86_64 0:4.20.22-1.el7.centos vdsm-api.noarch 0:4.20.22-1.el7.centos vdsm-client.noarch 0:4.20.22-1.el7.centos vdsm-common.noarch 0:4.20.22-1.el7.centos vdsm-hook-vmfex-dev.noarch 0:4.20.22-1.el7.centos vdsm-http.noarch 0:4.20.22-1.el7.centos vdsm-jsonrpc.noarch 0:4.20.22-1.el7.centos vdsm-network.x86_64 0:4.20.22-1.el7.centos vdsm-python.noarch 0:4.20.22-1.el7.centos vdsm-tests.noarch 0:4.20.22-1.el7.centos vdsm-yajsonrpc.noarch 0:4.20.22-1.el7.centos Complete! Checking configuration status... abrt is already configured for vdsm lvm is configured for vdsm libvirt is already configured for vdsm SUCCESS: ssl configured to true. No conflicts Current revision of multipath.conf detected, preserving schema is already configured Running configure... Reconfiguration of abrt is done. Reconfiguration of passwd is done. Reconfiguration of libvirt is done. Done configuring modules to VDSM. [root@reserved-0-241 vdsm]# service vdsmd status Redirecting to /bin/systemctl status vdsmd.service ● vdsmd.service - Virtual Desktop Server Manager Loaded: loaded (/usr/lib/systemd/system/vdsmd.service; enabled; vendor preset: enabled) Active: active (running) since Wed 2018-03-21 13:19:09 IST; 31s ago Process: 10366 ExecStopPost=/usr/libexec/vdsm/vdsmd_init_common.sh --post-stop (code=exited, status=0/SUCCESS) Process: 10637 ExecStartPre=/usr/libexec/vdsm/vdsmd_init_common.sh --pre-start (code=exited, status=0/SUCCESS) Main PID: 10824 (vdsmd) CGroup: /system.slice/vdsmd.service ├─10824 /usr/bin/python2 /usr/share/vdsm/vdsmd ├─10922 /usr/libexec/ioprocess --read-pipe-fd 49 --write-pipe-fd 48 --max-threads 10 --max-queued-requests 10 ├─10929 /usr/libexec/ioprocess --read-pipe-fd 54 --write-pipe-fd 51 --max-threads 10 --max-queued-requests 10 ├─10937 /usr/libexec/ioprocess --read-pipe-fd 57 --write-pipe-fd 54 --max-threads 10 --max-queued-requests 10 ├─10978 /usr/libexec/ioprocess --read-pipe-fd 72 --write-pipe-fd 71 --max-threads 10 --max-queued-requests 10 └─10999 /usr/libexec/ioprocess --read-pipe-fd 85 --write-pipe-fd 84 --max-threads 10 --max-queued-requests 10 Mar 21 13:19:10 reserved-0-241.tlv.redhat.com sudo[10920]: PAM adding faulty module: /usr/lib64/security/pam_krb5.so Mar 21 13:19:11 reserved-0-241.tlv.redhat.com vdsm[10824]: WARN Not ready yet, ignoring event '|virt|VM_status|29b896da-79cf-464d-9504-ce6e14e71b45' args={'29b896da-79cf-464d-9504-ce... Mar 21 13:19:11 reserved-0-241.tlv.redhat.com vdsm[10824]: WARN Not ready yet, ignoring event '|virt|VM_status|2fbdbf7f-b3d3-46dd-b9d7-e6f80608c581' args={'2fbdbf7f-b3d3-46dd-b9d7-e6... Mar 21 13:19:11 reserved-0-241.tlv.redhat.com vdsm[10824]: WARN MOM not available. Mar 21 13:19:11 reserved-0-241.tlv.redhat.com vdsm[10824]: WARN MOM not available, KSM stats will be missing. Mar 21 13:19:14 reserved-0-241.tlv.redhat.com sudo[10985]: PAM unable to dlopen(/usr/lib64/security/pam_krb5.so): /usr/lib64/security/pam_krb5.so: cannot open shared object... directory Mar 21 13:19:14 reserved-0-241.tlv.redhat.com sudo[10985]: PAM adding faulty module: /usr/lib64/security/pam_krb5.so Mar 21 13:19:14 reserved-0-241.tlv.redhat.com sudo[10995]: PAM unable to dlopen(/usr/lib64/security/pam_krb5.so): /usr/lib64/security/pam_krb5.so: cannot open shared object... directory Mar 21 13:19:14 reserved-0-241.tlv.redhat.com sudo[10995]: PAM adding faulty module: /usr/lib64/security/pam_krb5.so Mar 21 13:19:14 reserved-0-241.tlv.redhat.com vdsm[10824]: WARN MOM not available, Policy could not be set. Hint: Some lines were ellipsized, use -l to show in full. vdsm up and running. the schema module is already configured if 4.20.20 ran. If this still doesn't work for you please submit the output of # ll /var/cache/vdsm/schema/vdsm* before and after the upgrade Thanks.
(In reply to Yaniv Bronhaim from comment #6) > did you use yum upgrade or you removed and reinstalled vdsm? using yum > upgrade should perform for you "vdsm-tool configure" as part of the rpm > installation (unless we have a bug in recognizing the upgrade which seems > alright to me), if you removed and then install, you should run it manually I used yum update only)
We can't manage to reproduce it again - although it appeared in more than one env. I tried few upgrade flows that worked as expected (4.1>latest, 4.2>latest, 4.20.20>4.20.23), we call the schema re-creation after the upgrade. Please re-open immediately in case it appears again
(In reply to Yaniv Bronhaim from comment #9) > We can't manage to reproduce it again - although it appeared in more than > one env. I tried few upgrade flows that worked as expected (4.1>latest, > 4.2>latest, 4.20.20>4.20.23), we call the schema re-creation after the > upgrade. Please re-open immediately in case it appears again W ehave a reprodcution - Mar 25 15:53:36 orchid-vds2 vdsmd_init_common.sh: schema should be configuredModules schema are not configured Mar 25 15:53:36 orchid-vds2 vdsmd_init_common.sh: vdsm: stopped during execute check_is_configured task (task returned with error code 1). Mar 25 15:53:36 orchid-vds2 systemd: vdsmd.service: control process exited, code=exited status=1 upgraded vdsm from 4.20.20 >> 4.20.23 I sent you the details in the mail. Please take a look ASAP, keeping the host for you.
I suspect that something restarts vdsm service during the upgrade before we get to the %post install of the spec Those prints come when the spec gets to the part that configure and start the service - if the service was up before. Mar 25 15:54:16 orchid-vds2 systemd: Stopped Auxiliary vdsm service for running helper functions as root. Mar 25 15:54:16 orchid-vds2 systemd: Stopped Virtual Desktop Server Manager network restoration. Mar 25 15:54:16 orchid-vds2 systemd: Stopping Virtual Desktop Server Manager network restoration... Mar 25 15:54:20 orchid-vds2 systemd: Stopping Virtualization daemon... Mar 25 15:54:20 orchid-vds2 systemd: Stopped Virtualization daemon. Mar 25 15:54:20 orchid-vds2 saslpasswd2: error deleting entry from sasldb: BDB0073 DB_NOTFOUND: No matching key/data pair found Mar 25 15:54:20 orchid-vds2 saslpasswd2: error deleting entry from sasldb: BDB0073 DB_NOTFOUND: No matching key/data pair found Mar 25 15:54:20 orchid-vds2 saslpasswd2: error deleting entry from sasldb: BDB0073 DB_NOTFOUND: No matching key/data pair found Mar 25 15:54:21 orchid-vds2 systemd: Cannot add dependency job for unit lvm2-lvmetad.socket, ignoring: Unit is masked. Mar 25 15:54:21 orchid-vds2 systemd: Starting Virtualization daemon... Mar 25 15:54:21 orchid-vds2 systemd: Started Virtualization daemon. But before it fails to start - it seems like something just after the upgrade stop and start vdsm service see the following- upgrade finished here: Mar 25 15:52:44 orchid-vds2 yum[32280]: Updated: python-perf-3.10.0-862.el7.x86_64 Maybe this ? Mar 25 15:52:47 orchid-vds2 systemd: Starting oVirt ImageIO Daemon... ? Mar 25 15:52:48 orchid-vds2 systemd: Started Gofer Agent.. ? Mar 25 15:52:53 orchid-vds2 ovn-ctl: Starting ovn-controller [ OK ] ? Mar 25 15:53:06 orchid-vds2 systemd: Starting Kernel Samepage Merging... ( I don't have those in my env) then Mar 25 15:53:25 orchid-vds2 systemd: Stopping Virtualization daemon... Mar 25 15:53:25 orchid-vds2 systemd: Starting Virtualization daemon... Then many fails, and only in Mar 25 15:54:16 we see that we get to the %post part of the spec - vdsm already down, so we left only with configured vdsm without trying to start it. between 15:52:44 to 15:54:16 its 1:30min, at my env its 00:03:32 - 00:04:01 Did we commit something that restarts vdsmd probably due to new systemd dependencies after upgrade?
I sent 2 choices to fix this issue: https://gerrit.ovirt.org/#/c/89446/ - checking vdsm status before starting the upgrade. here we stop the service during the upgrade - I think this approach might be the best if indeed restart to vdsmd appears more than only in the %post part during the upgrade https://gerrit.ovirt.org/#/c/89441/ - create the schema during installation and removes it from configurator
(In reply to Yaniv Bronhaim from comment #11) > Did we commit something that restarts vdsmd probably due to new systemd > dependencies after upgrade? I don't know adout such change, please check with Dan and Francesco.
(In reply to Nir Soffer from comment #13) > (In reply to Yaniv Bronhaim from comment #11) > > Did we commit something that restarts vdsmd probably due to new systemd > > dependencies after upgrade? > > I don't know adout such change, please check with Dan and Francesco. Quick update: As far as I know, we didn't. Will review the bigger picture later.
Back to POST, we don't have all patches merged yet
After offline discussion moving back to MODIFIED, https://gerrit.ovirt.org/89446 is not required to fix this upgrade issue
Verified on - vdsm-4.20.25-1.el7ev.x86_64
This bugzilla is included in oVirt 4.2.3 release, published on May 4th 2018. Since the problem described in this bug report should be resolved in oVirt 4.2.3 release, it has been closed with a resolution of CURRENT RELEASE. If the solution does not work for you, please open a new bug report.