Bug 1552565
| Summary: | VDSM fails to accept jsonrpc connections due to bad permission on /var/cache/vdsm/schema/vdsm* | ||
|---|---|---|---|
| Product: | [oVirt] vdsm | Reporter: | Simone Tiraboschi <stirabos> |
| Component: | Core | Assignee: | Yaniv Bronhaim <ybronhei> |
| Status: | CLOSED CURRENTRELEASE | QA Contact: | SATHEESARAN <sasundar> |
| Severity: | high | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 4.20.15 | CC: | bshetty, bugs, lsvaty, mperina, pkliczew, sasundar, sbonazzo, stirabos |
| Target Milestone: | ovirt-4.2.2 | Flags: | rule-engine:
ovirt-4.2?
sasundar: planning_ack? rule-engine: devel_ack+ sasundar: testing_ack+ |
| Target Release: | 4.20.23 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | v4.20.23 | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2018-04-18 12:25:04 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | Infra | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
Please provide both client and vdsm logs. Bipin, are you able to reproduce it? Do we feel it's going to make it into 4.2.2? If not, please defer. (In reply to Yaniv Kaul from comment #3) > Do we feel it's going to make it into 4.2.2? If not, please defer. I can't answer for this, being handled by Piotr. redirecting needinfo on him. I am missing some information asked in comment #2. If we want to have it fix in 4.2.2 please change priority for it and I will handle. Otherwise I would defer it. Sorry, wrong bug (In reply to Simone Tiraboschi from comment #2) > Bipin, are you able to reproduce it? Simone, we will check this out. We hit this problem initially in our setup Tested with vdsm-4.20.23 and these files are accessible for read # ls -lsah /var/cache/vdsm/schema/* 48K -r--r--r--. 1 root root 45K Mar 21 16:15 /var/cache/vdsm/schema/vdsm-api-gluster.pickle 252K -r--r--r--. 1 root root 250K Mar 21 16:15 /var/cache/vdsm/schema/vdsm-api.pickle 8.0K -r--r--r--. 1 root root 6.5K Mar 21 16:15 /var/cache/vdsm/schema/vdsm-events.pickle Marking this bug as VERIFIED This bugzilla is included in oVirt 4.2.2 release, published on March 28th 2018. Since the problem described in this bug report should be resolved in oVirt 4.2.2 release, it has been closed with a resolution of CURRENT RELEASE. If the solution does not work for you, please open a new bug report. |
Description of problem: VDSM fails to accept jsonrpc connections and if fails with: 2018-03-07 14:32:44,170+0530 INFO (MainThread) [vdsm.api] FINISH prepareForShutdown return=None from=internal, task_id=06629140-da75-4899-9848-5ad7d243a115 (api:52) 2018-03-07 14:32:44,171+0530 ERROR (MainThread) [vds] Exception raised (vdsmd:158) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/vdsmd.py", line 156, in run serve_clients(log) File "/usr/lib/python2.7/site-packages/vdsm/vdsmd.py", line 103, in serve_clients cif = clientIF.getInstance(irs, log, scheduler) File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 250, in getInstance cls._instance = clientIF(irs, log, scheduler) File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 144, in __init__ self._prepareJSONRPCServer() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 307, in _prepareJSONRPCServer bridge = Bridge.DynamicBridge() File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 67, in __init__ self._schema = vdsmapi.Schema(paths, api_strict_mode) File "/usr/lib/python2.7/site-packages/vdsm/api/vdsmapi.py", line 217, in __init__ raise SchemaNotFound("Unable to find API schema file") SchemaNotFound: Unable to find API schema file 2018-03-07 14:32:44,172+0530 INFO (MainThread) [vds] Stopping threads (vdsmd:160) 2018-03-07 14:32:44,172+0530 INFO (MainThread) [vds] Exiting (vdsmd:171) 2018-03-07 14:32:44,200+0530 ERROR (hsm/init) [storage.HSM] FINISH thread <Thread(hsm/init, started daemon 139793912547072)> failed (concurrent:201) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/common/concurrent.py", line 194, in run ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 406, in storageRefresh sdCache.refreshStorage() File "/usr/lib/python2.7/site-packages/vdsm/storage/misc.py", line 404, in helper return sm(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/misc.py", line 389, in __call__ self.__lastResult = self.__func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 82, in refreshStorage multipath.rescan() File "/usr/lib/python2.7/site-packages/vdsm/storage/multipath.py", line 83, in rescan udevadm.settle(timeout) File "/usr/lib/python2.7/site-packages/vdsm/common/udevadm.py", line 50, in settle _run_command(args) File "/usr/lib/python2.7/site-packages/vdsm/common/udevadm.py", line 105, in _run_command rc, out, err = commands.execCmd(cmd, raw=True) File "/usr/lib/python2.7/site-packages/vdsm/common/commands.py", line 80, in execCmd (out, err) = p.communicate(data) File "/usr/lib64/python2.7/contextlib.py", line 24, in __exit__ self.gen.next() File "/usr/lib/python2.7/site-packages/vdsm/common/commands.py", line 372, in terminating terminate(proc) TypeError: 'NoneType' object is not callable The source of the issue is that: [root@rhsqa-grafton7 ~]# ls -l /var/cache/vdsm/schema/vdsm* -rw-------. 1 root root 44856 2 mar 02.34 /var/cache/vdsm/schema/vdsm-api-gluster.pickle -rw-------. 1 root root 255129 2 mar 02.34 /var/cache/vdsm/schema/vdsm-api.pickle -rw-------. 1 root root 6607 2 mar 02.34 /var/cache/vdsm/schema/vdsm-events.pickle are not readable by vdsm user resulting in a IOError: [Errno 13] Permission denied: '/var/cache/vdsm/schema/vdsm-api-gluster.pickle' which is hidden by 'SchemaNotFound: Unable to find API schema file'. chmod 644 /var/cache/vdsm/schema/vdsm* seams enough to avoid that. Version-Release number of selected component (if applicable): How reproducible: Seen once on HC env, not sure if 100% reproducible. Steps to Reproduce: 1. Try deploying HC on 4.2 2. 3. Actual results: VDSM does not accept jsonrpc connections due to 'SchemaNotFound: Unable to find API schema file'. [root@rhsqa-grafton7 ~]# ls -l /var/cache/vdsm/schema/vdsm* -rw-------. 1 root root 44856 2 mar 02.34 /var/cache/vdsm/schema/vdsm-api-gluster.pickle -rw-------. 1 root root 255129 2 mar 02.34 /var/cache/vdsm/schema/vdsm-api.pickle -rw-------. 1 root root 6607 2 mar 02.34 /var/cache/vdsm/schema/vdsm-events.pickle are not readable by vdsm user Expected results: VDSM correctly starts and accepts jsonrpc connections Additional info: