Bug 1969302
| Summary: | /bin/podman ps --format {{.Names}} exit code 125 | ||
|---|---|---|---|
| Product: | [Red Hat Storage] Red Hat Ceph Storage | Reporter: | Juan Miguel Olmo <jolmomar> |
| Component: | Cephadm | Assignee: | Juan Miguel Olmo <jolmomar> |
| Status: | CLOSED ERRATA | QA Contact: | Vasishta <vashastr> |
| Severity: | urgent | Docs Contact: | Karen Norteman <knortema> |
| Priority: | high | ||
| Version: | 5.0 | CC: | kdreyer, pnataraj, sewagner, tserlin, vereddy |
| Target Milestone: | --- | Keywords: | Regression |
| Target Release: | 5.0 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | ceph-16.2.0-65.el8cp | Doc Type: | No Doc Update |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2021-08-30 08:31:02 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
|
Comment 1
Sebastian Wagner
2021-06-08 11:22:54 UTC
@Juan, Issue is not seen after redeploy of a services. below output for reference. Hence, moving the issue to Verified state after discussing with you on the behaviour noticed.
Traceback (most recent call last):
File "/lib/python3.6/site-packages/cherrypy/process/wspbus.py", line 230, in publish
output.append(listener(*args, **kwargs))
File "/lib/python3.6/site-packages/cherrypy/_cpserver.py", line 180, in start
super(Server, self).start()
File "/lib/python3.6/site-packages/cherrypy/process/servers.py", line 184, in start
self.wait()
File "/lib/python3.6/site-packages/cherrypy/process/servers.py", line 246, in wait
raise self.interrupt
File "/lib64/python3.6/threading.py", line 916, in _bootstrap_inner
self.run()
File "/lib64/python3.6/threading.py", line 864, in run
self._target(*self._args, **self._kwargs)
File "/lib/python3.6/site-packages/cherrypy/process/servers.py", line 225, in _start_http_thread
self.httpserver.start()
File "/lib/python3.6/site-packages/cheroot/server.py", line 1836, in start
self.prepare()
File "/lib/python3.6/site-packages/cheroot/server.py", line 1791, in prepare
raise socket.error(msg)
OSError: No socket could be created -- (('::', 2, 0, 0): [Errno 13] Permission denied)
debug 2021-06-14T10:12:50.203+0000 7f1e3fb25700 0 [prometheus ERROR cherrypy.error] [14/Jun/2021:10:12:50] ENGINE Shutting down due to error in start listener:
Traceback (most recent call last):
File "/lib/python3.6/site-packages/cherrypy/process/wspbus.py", line 268, in start
self.publish('start')
File "/lib/python3.6/site-packages/cherrypy/process/wspbus.py", line 248, in publish
raise exc
cherrypy.process.wspbus.ChannelFailures: OSError("No socket could be created -- (('::', 2, 0, 0): [Errno 13] Permission denied)",)
[14/Jun/2021:10:12:50] ENGINE Bus STOPPING
[14/Jun/2021:10:12:50] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 2)) already shut down
[14/Jun/2021:10:12:50] ENGINE Bus STOPPED
[14/Jun/2021:10:12:50] ENGINE Bus EXITING
[14/Jun/2021:10:12:50] ENGINE Bus EXITED
debug 2021-06-14T10:12:50.203+0000 7f1e3fb25700 -1 log_channel(cluster) log [ERR] : Unhandled exception from module 'prometheus' while running on mgr.ceph-adm10.pzhfuh: OSError("No socket could be created -- (('::', 2, 0, 0): [Errno 13] Permission denied)",)
debug 2021-06-14T10:12:50.203+0000 7f1e3fb25700 -1 prometheus.serve:
debug 2021-06-14T10:12:50.203+0000 7f1e3fb25700 -1 Traceback (most recent call last):
File "/usr/share/ceph/mgr/prometheus/module.py", line 1418, in serve
cherrypy.engine.start()
File "/lib/python3.6/site-packages/cherrypy/process/wspbus.py", line 283, in start
raise e_info
File "/lib/python3.6/site-packages/cherrypy/process/wspbus.py", line 268, in start
self.publish('start')
File "/lib/python3.6/site-packages/cherrypy/process/wspbus.py", line 248, in publish
raise exc
cherrypy.process.wspbus.ChannelFailures: OSError("No socket could be created -- (('::', 2, 0, 0): [Errno 13] Permission denied)",)
debug 2021-06-14T10:15:04.764+0000 7f1e56e90700 1 mgr handle_mgr_map Activating!
debug 2021-06-14T10:15:04.765+0000 7f1e56e90700 1 mgr handle_mgr_map I am now activating
debug 2021-06-14T10:15:04.770+0000 7f1e59e96700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2]
debug 2021-06-14T10:15:04.772+0000 7f1e5a697700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2]
debug 2021-06-14T10:15:04.777+0000 7f1e5ae98700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2]
but however i saw the above prometheus error
[ceph: root@ceph-adm10 /]# ceph orch ls
NAME RUNNING REFRESHED AGE PLACEMENT IMAGE ID
alertmanager 1/1 11m ago 3w count:1 a952062e05ca
crash 2/3 11m ago 3w * 7c956aac1349
grafana 1/1 11m ago 3w count:1 mix
mds.test 2/2 2m ago 2w ceph-adm10;ceph-adm11;ceph-adm12;count:2 7c956aac1349
mgr 2/3 11m ago 3w ceph-adm10;ceph-adm11;ceph-adm12 7c956aac1349
mon 2/3 11m ago 3w ceph-adm10;ceph-adm11;ceph-adm12 7c956aac1349
node-exporter 2/3 11m ago 3w * 8ff23136b0fe
osd.all-available-devices 7/11 11m ago 3w * 7c956aac1349
osd.dashboard-admin-1621424607412 0/3 - - * <unknown>
prometheus 1/1 11m ago 3w count:1 mix
[ceph: root@ceph-adm10 /]# ceph -s
cluster:
id: 81a4597a-b711-11eb-8cb8-001a4a000740
health: HEALTH_ERR
Module 'prometheus' has failed: OSError("No socket could be created -- (('::', 2, 0, 0): [Errno 13] Permission denied)",)
services:
mon: 3 daemons, quorum ceph-adm10,ceph-adm11,ceph-adm12 (age 3m)
mgr: ceph-adm11.xwzphg(active, since 5m), standbys: ceph-adm12.wsnngf
mds: 1/1 daemons up, 1 standby
osd: 11 osds: 11 up (since 10d), 11 in (since 2w)
data:
volumes: 1/1 healthy
pools: 5 pools, 129 pgs
objects: 43 objects, 5.8 KiB
usage: 267 MiB used, 165 GiB / 165 GiB avail
pgs: 129 active+clean
@Juan, This needs to be verified with exact scenario where the issue is hit. Will move back to ON QA to re verify @Juan, This needs to be verified with exact scenario where the issue is hit. Will move back to ON QA to re verify Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Red Hat Ceph Storage 5.0 bug fix and enhancement), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2021:3294 |