Description of problem: When upgrading a RHV-H node from UI , spm is stopped and due to this trace backs are seen in vdsm logs. Thread-557::DEBUG::2016-09-30 18:20:25,064::resourceManager::661::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_57ee3a08-004b-02 7b-0395-0000000001d6', Clearing records. Thread-557::ERROR::2016-09-30 18:20:25,064::utils::375::Storage.StoragePool::(wrapper) Unhandled exception Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 372, in wrapper return f(*a, **kw) File "/usr/lib/python2.7/site-packages/vdsm/concurrent.py", line 177, in run return func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/securable.py", line 78, in wrapper return method(self, *args, **kwargs) File "/usr/share/vdsm/storage/sp.py", line 207, in _upgradePoolDomain self._finalizePoolUpgradeIfNeeded() File "/usr/lib/python2.7/site-packages/vdsm/storage/securable.py", line 76, in wrapper raise SecureError("Secured object is not in safe state") SecureError: Secured object is not in safe state b38e7a14-f880-4259-a7dd-3994bae2dbc2::DEBUG::2016-09-30 18:20:25,065::__init__::398::IOProcessClient::(_startCommunication) Communication thread for client ioprocess-7 started ioprocess communication (22325)::INFO::2016-09-30 18:20:25,067::__init__::447::IOProcess::(_processLogs) Starting ioprocess ioprocess communication (22325)::INFO::2016-09-30 18:20:25,067::__init__::447::IOProcess::(_processLogs) Starting ioprocess Version-Release number of selected component (if applicable): vdsm-4.18.13-1.el7ev.x86_64 How reproducible: Always Steps to Reproduce: 1. Upgrade a RHV-H node from UI 2. 3. Actual results: Once the upgrade is done, i see that tracebacks mentioned in description are seen in vdsm log. Expected results: vdsm log should not throw any trace backs Additional info:
These message means that the SPM is stopped when a domain upgrade thread finish. This may be expected in a multithreaded environment, or may reveal a bug in thread synchronization. RamaKasturi, can you attach complete vdsm log showing the entire flow since the host was activated?
This is not related to node, I see it also locally on RHEL 7.3. I think this is related to a fix we did in 4.0 for stuck upgrades threads.
*** This bug has been marked as a duplicate of bug 1400534 ***