Bug 1784801
| Summary: | [RFE] [ceph-container] - ceph-medic support for containerized cluster | ||||||
|---|---|---|---|---|---|---|---|
| Product: | [Red Hat Storage] Red Hat Ceph Storage | Reporter: | Vasishta <vashastr> | ||||
| Component: | Ceph-Medic | Assignee: | Zack Cerza <zcerza> | ||||
| Status: | CLOSED CURRENTRELEASE | QA Contact: | Vasishta <vashastr> | ||||
| Severity: | high | Docs Contact: | |||||
| Priority: | urgent | ||||||
| Version: | 4.0 | CC: | gmeno, hyelloji, sunnagar, tchandra, tserlin, vereddy, zcerza | ||||
| Target Milestone: | --- | Keywords: | FutureFeature | ||||
| Target Release: | 4.2 | ||||||
| Hardware: | Unspecified | ||||||
| OS: | Unspecified | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | ceph-medic-1.0.8-1.el8cp, ceph-medic-1.0.8-1.el7cp | Doc Type: | If docs needed, set a value | ||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2021-06-02 10:08:34 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Bug Depends On: | |||||||
| Bug Blocks: | 1709475 | ||||||
| Attachments: |
|
||||||
Not sure where/what in the output indicates that there is no support for containers. Could you help me understand what portion is not working? If you have deployed with ansible, can you try to add the following in the ceph-medic configuration file (~/.cephmedic.conf) >[global] > >deployment_type = container Hi Alfredo,
Thanks for the info. I'm getting below error message, let me know if I'm missing anything.
(I have RHCEPH containerized cluster on RHEL 7.7) (RHCEPH containers are of RHEL 8 (they don't have ceph-medic package))
# ceph-medic check
Host: magna110 connection: [connecting ] Traceback (most recent call last):
File "/usr/bin/ceph-medic", line 6, in <module>
main.Medic()
File "/usr/lib/python2.7/site-packages/ceph_medic/main.py", line 45, in __init__
self.main(argv)
File "/usr/lib/python2.7/site-packages/ceph_medic/decorators.py", line 46, in newfunc
return f(*a, **kw)
File "/usr/lib/python2.7/site-packages/ceph_medic/main.py", line 145, in main
parser.dispatch()
File "/usr/lib/python2.7/site-packages/tambo/dispatcher.py", line 21, in dispatch
result = instance.main()
File "/usr/lib/python2.7/site-packages/ceph_medic/check.py", line 82, in main
collector.collect()
File "/usr/lib/python2.7/site-packages/ceph_medic/collector.py", line 178, in collect
conn = get_connection(node['host'])
File "/usr/lib/python2.7/site-packages/ceph_medic/connection.py", line 64, in get_connection
raise error
UnboundLocalError: local variable 'conn' referenced before assignment
How was Ceph deployed? There is support via ceph-ansible deployment and its hosts file. Can you include the .cephmedic.conf file you are using? (In reply to Alfredo Deza from comment #7) Hi Alfredo, > How was Ceph deployed? There is support via ceph-ansible deployment and its > hosts file. Sorry, I could understand Do you mean that ceph-medic can be configured using ceph-ansible ? Can you please help me with hints to work on this ? I installed ceph-medic manually on ceph-ansible node and changed deployment_type to container manually in conf file > Can you include the .cephmedic.conf file you are using? cephmedic conf - # cat .cephmedic.conf| egrep -v ^# | grep -v ^$ [global] --cluster = ceph --verbosity = info --log-path = . deployment_type = container [check] [baremetal] [kubernetes] namespace = rook-ceph (In reply to Vasishta from comment #8) > Do you mean that ceph-medic can be configured using ceph-ansible ? Can you > please help me with hints to work on this ? No, I mean that ceph-medic uses the hosts file that is used by ceph-ansible. If you are deploying your containers with ceph-ansible and you have a hosts file, that needs to be used so that ceph-medic understands where to look. Can you include the hosts file that you are using? If you aren't using one... then that would be one of the problems here After going through the code it seems there is no support in ceph-medic to run in containers. The underlying connection library does support connecting to containers though... the missing link is a way for ceph-medic to retrieve the container IDs of the remote hosts it is connecting to. Currently, there is support for Kubernetes and Openshift. I estimate that the work to do to get this supported will take about two weeks at the most. Support for docker/podman clusters was merged last week, and I just released v1.0.6 which includes those changes. (In reply to Zack Cerza from comment #13) > Support for docker/podman clusters was merged last week, and I just released > v1.0.6 which includes those changes. Nice - please provide devel-ack. Does it work on RHEL 8? (In reply to Yaniv Kaul from comment #19) > Does it work on RHEL 8? @Yaniv There should not be a problem in RHEL8, but I'd love to hear from QE on that as well. The issue is mostly because of bizarre differences between docker and podman. I believe I've tracked this down, and the fix is going to come in two parts: One for remoto, and the other of course for ceph-medic. I've attached the remoto PR, and the ceph-medic one will be filed ASAP after I get a response on remoto. Moving out of 4.1 -- no longer a blocker Zack, Would you please confirm the upstream release of ceph-medic that contains this fix? Upstream release was waiting on me while I figured out how serious the bug found by Vasishta is, and whether we needed fixes for ceph-medic or remoto (or other things). After a whole lot of debugging, I began to suspect a regression in podman itself. Sure enough: https://bugzilla.redhat.com/show_bug.cgi?id=1807379 I'll get a release out by tomorrow. Regional internet outages slowed this down quite a bit, but here's 1.0.8 with everything we need https://github.com/ceph/ceph-medic/releases/tag/v1.0.8 Created attachment 1698811 [details]
ceph-medic validation log
It looks like it works on RHEL 7.8 and doesn't on RHEL 8.x? I'd open a separate specific bug for the scenario (instead of this one). The remaining issue was with podman having regressed, breaking anything that needed to pass content via stdin. It's been fixed, now, and we should have everything we need. Sunil, could you retest please? |
Description of problem: ceph-medic needs to be supported for containerized cluster from RHCS 4.x Version-Release number of selected component (if applicable): 1.0.5 How reproducible: Always(1/1) Steps to Reproduce: 1. Configure containerized cluster 2. Install ceph-medic on ansible node 3. run ceph-medic check Actual results: ======================= Starting remote check session ======================== Version: 1.0.5 Cluster Name: "ceph" Connection: ssh Total hosts: [13] OSDs: 6 MONs: 4 Clients: 0 MDSs: 1 RGWs: 1 MGRs: 1 ================================================================================ ---------- managers ---------- magna110 ECOM2: ceph executable was not found in common paths when running `which` WCOM1: collocated OSDs with MONs running: /var/run/ceph/ceph-osd.1.asok,/var/run/ceph/ceph-osd.2.asok,/var/run/ceph/ceph-osd.0.asok ECOM4: /var/lib/ceph has invalid ownership: 167:167, should be ceph:ceph -------------mdss------------- magna118 ECOM2: ceph executable was not found in common paths when running `which` ECOM4: /var/lib/ceph has invalid ownership: 167:167, should be ceph:ceph ------- rados gateways ------- magna117 ECOM2: ceph executable was not found in common paths when running `which` ECOM4: /var/lib/ceph has invalid ownership: 167:167, should be ceph:ceph ------------ osds ------------ magna115 ECOM2: ceph executable was not found in common paths when running `which` WCOM1: collocated OSDs with MONs running: /var/run/ceph/ceph-osd.9.asok,/var/run/ceph/ceph-osd.17.asok,/var/run/ceph/ceph-osd.12.asok,/var/run/ceph/ceph-osd.15.asok,/var/run/ceph/ceph-osd.18.asok,/var/run/ceph/ceph-osd.20.asok,/var/run/ceph/ceph-osd.14.asok,/var/run/ceph/ceph-osd.19.asok,/var/run/ceph/ceph-osd.16.asok,/var/run/ceph/ceph-osd.10.asok,/var/run/ceph/ceph-osd.11.asok,/var/run/ceph/ceph-osd.13.asok ECOM4: /var/lib/ceph has invalid ownership: 167:167, should be ceph:ceph magna116 ECOM2: ceph executable was not found in common paths when running `which` ECOM4: /var/lib/ceph has invalid ownership: 167:167, should be ceph:ceph magna117 ECOM2: ceph executable was not found in common paths when running `which` ECOM4: /var/lib/ceph has invalid ownership: 167:167, should be ceph:ceph magna110 ECOM2: ceph executable was not found in common paths when running `which` WCOM1: collocated OSDs with MONs running: /var/run/ceph/ceph-osd.1.asok,/var/run/ceph/ceph-osd.2.asok,/var/run/ceph/ceph-osd.0.asok ECOM4: /var/lib/ceph has invalid ownership: 167:167, should be ceph:ceph magna118 ECOM2: ceph executable was not found in common paths when running `which` ECOM4: /var/lib/ceph has invalid ownership: 167:167, should be ceph:ceph magna109 ECOM2: ceph executable was not found in common paths when running `which` WCOM1: collocated OSDs with MONs running: /var/run/ceph/ceph-osd.21.asok,/var/run/ceph/ceph-osd.22.asok ECOM4: /var/lib/ceph has invalid ownership: 167:167, should be ceph:ceph ------------ mons ------------ magna115 ECOM2: ceph executable was not found in common paths when running `which` WCOM1: collocated OSDs with MONs running: /var/run/ceph/ceph-osd.9.asok,/var/run/ceph/ceph-osd.17.asok,/var/run/ceph/ceph-osd.12.asok,/var/run/ceph/ceph-osd.15.asok,/var/run/ceph/ceph-osd.18.asok,/var/run/ceph/ceph-osd.20.asok,/var/run/ceph/ceph-osd.14.asok,/var/run/ceph/ceph-osd.19.asok,/var/run/ceph/ceph-osd.16.asok,/var/run/ceph/ceph-osd.10.asok,/var/run/ceph/ceph-osd.11.asok,/var/run/ceph/ceph-osd.13.asok ECOM4: /var/lib/ceph has invalid ownership: 167:167, should be ceph:ceph WMON2: collocated OSDs found: ceph-16,ceph-9,ceph-11,ceph-10,ceph-13,ceph-12,ceph-20,ceph-14,ceph-17,ceph-18,ceph-19,ceph-15 WMON4: Number of MONs is not an odd number: 4 magna110 ECOM2: ceph executable was not found in common paths when running `which` WCOM1: collocated OSDs with MONs running: /var/run/ceph/ceph-osd.1.asok,/var/run/ceph/ceph-osd.2.asok,/var/run/ceph/ceph-osd.0.asok ECOM4: /var/lib/ceph has invalid ownership: 167:167, should be ceph:ceph WMON2: collocated OSDs found: ceph-1,ceph-0,ceph-2 WMON4: Number of MONs is not an odd number: 4 magna071 ECOM1: /etc/ceph/ceph.conf does not exist ECOM2: ceph executable was not found in common paths when running `which` ECOM8: 'fsid' is missing in the ceph configuration ECOM3: /var/lib/ceph could not be parsed: [Errno 2] No such file or directory: '/var/lib/ceph' WMON4: Number of MONs is not an odd number: 4 magna109 ECOM2: ceph executable was not found in common paths when running `which` WCOM1: collocated OSDs with MONs running: /var/run/ceph/ceph-osd.21.asok,/var/run/ceph/ceph-osd.22.asok ECOM4: /var/lib/ceph has invalid ownership: 167:167, should be ceph:ceph WMON2: collocated OSDs found: ceph-21,ceph-22 WMON4: Number of MONs is not an odd number: 4 ---------- cluster ----------- ceph cluster 157 passed, 28 errors, 14 warnings, 38 internal errors, on 13 hosts While running checks, ceph-medic had 38 unhandled errors, please look at the configured log file and report the issue along with the traceback.