Description of problem: This is part of a osp 17.1 deployment with ceph 6, the following error is blocking the grafana container from starting: Deploy daemon grafana.overcloud-controller-1 ... Verifying port 3100 ... Cannot bind to IP :: port 3100: [Errno 98] Address already in use ERROR: TCP Port(s) '3100' required for grafana already in use The in use address is haproxy on a different interface The config looks good. From "ceph orch ls --export" --- service_type: grafana service_name: grafana placement: hosts: - overcloud-controller-0 - overcloud-controller-1 - overcloud-controller-2 networks: - 2001:db8:1:9::/64 - 2001:db8:1:c::/64 - 2001:db8:1:b::/64 - 2001:db8:1:a::/64 - 2001:db8:1:d::/64 - 2001:db8:1:8::/64 spec: port: 3100 --- If I understand correctly, the "networks" option should limit binding to interfaces contained there. Here is overcloud-controller-0 interface information showing a valid interface for binding. overcloud-controller-0]$ grep 2001:db8:1 ip_addr 16: vlan123 inet6 2001:db8:1:8::b5/64 scope global \ valid_lft forever preferred_lft forever It should only bind to 2001:db8:1:8::b5:3100 This seems to also impact other services such as prometheus & alertmanager but likely the same issue. I'll provide more details and logs in private comments. Version-Release number of selected component (if applicable): cephadm-17.2.6-70.el9cp.noarch ceph 6 deployment How reproducible: this environment Steps to Reproduce: 1. see notes above Actual results: grafana daemon attempting to bind to all interfaces and failing. Expected results: specific interface based on networks configuration. Additional info: In private comments.
*** Bug 2236231 has been marked as a duplicate of this bug. ***
I don't see the parameter "only_bind_port_on_networks" in image quay.io/ceph/ceph:v18.2.1. I would like test it by introducing the parameter in tripleo. Can you share the container image i can use to test it ?
(In reply to Manoj Katari from comment #20) > I don't see the parameter "only_bind_port_on_networks" in image > quay.io/ceph/ceph:v18.2.1. I would like test it by introducing the parameter > in tripleo. > > Can you share the container image i can use to test it ? I haven't done a quincy backport yet as there were a lot of merge conflicts to deal with there. If you're okay with reef though, the regular upstream CI image quay.ceph.io/ceph-ci/ceph:reef should have it.
i tried the image you suggested but i still don't the support for the parameter [root@controller-0 ~]# podman run --rm --net=host -v /etc/ceph:/etc/ceph:z -v /var/lib/ceph/:/var/lib/ceph/:z -v /var/log/ceph/:/var/log/ceph/:z -v /home/ceph-admin/specs/grafana:/home/ceph-admin/specs/grafana:z --entrypoint=ceph quay.ceph.io/ceph-ci/ceph:reef -n client.admin -k /etc/ceph/ceph.client.admin.keyring --cluster ceph orch apply --in-file /home/ceph-admin/specs/grafana Error EINVAL: ServiceSpec: __init__() got an unexpected keyword argument 'only_bind_port_on_networks'
(In reply to Manoj Katari from comment #22) > i tried the image you suggested but i still don't the support for the > parameter > > [root@controller-0 ~]# podman run --rm --net=host -v /etc/ceph:/etc/ceph:z > -v /var/lib/ceph/:/var/lib/ceph/:z -v /var/log/ceph/:/var/log/ceph/:z -v > /home/ceph-admin/specs/grafana:/home/ceph-admin/specs/grafana:z > --entrypoint=ceph quay.ceph.io/ceph-ci/ceph:reef -n client.admin -k > /etc/ceph/ceph.client.admin.keyring --cluster ceph orch apply --in-file > /home/ceph-admin/specs/grafana > Error EINVAL: ServiceSpec: __init__() got an unexpected keyword argument > 'only_bind_port_on_networks' maybe just a formatting thing in the spec? I think the setting has to go into the `spec` section for example. I was able to make use of it with the recommended image --- [root@vm-00 ~]# podman images REPOSITORY TAG IMAGE ID CREATED SIZE quay.ceph.io/ceph-ci/ceph reef bb2c5eac26a4 2 days ago 1.37 GB quay.io/ceph/ceph-grafana 9.4.7 954c08fa6188 4 months ago 647 MB quay.io/prometheus/prometheus v2.43.0 a07b618ecd1d 12 months ago 235 MB quay.io/prometheus/alertmanager v0.25.0 c8568f914cd2 15 months ago 66.5 MB quay.io/prometheus/node-exporter v1.5.0 0da6a335fe13 16 months ago 23.9 MB [root@vm-00 ~]# [root@vm-00 ~]# cephadm shell Inferring fsid 1fd068c6-f820-11ee-b857-5254002b59e2 Inferring config /var/lib/ceph/1fd068c6-f820-11ee-b857-5254002b59e2/mon.vm-00/config Using ceph image with id 'bb2c5eac26a4' and tag 'reef' created on 2024-04-09 00:48:11 +0000 UTC quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3 [ceph: root@vm-00 /]# [ceph: root@vm-00 /]# [ceph: root@vm-00 /]# ceph orch ls grafana --export > grafana.yaml [ceph: root@vm-00 /]# [ceph: root@vm-00 /]# vi grafana.yaml [ceph: root@vm-00 /]# [ceph: root@vm-00 /]# cat grafana.yaml service_type: grafana service_name: grafana placement: count: 1 networks: 192.168.122.0/24 spec: anonymous_access: true protocol: https only_bind_port_on_networks: true [ceph: root@vm-00 /]# [ceph: root@vm-00 /]# ceph orch apply -i grafana.yaml Scheduled grafana update... [ceph: root@vm-00 /]# [ceph: root@vm-00 /]# ceph orch ls grafana --export service_type: grafana service_name: grafana placement: count: 1 networks: - 192.168.122.0/24 spec: anonymous_access: true only_bind_port_on_networks: true protocol: https
Thanks Adam, it was my mistake earlier. I could see the new parameter is supported for both grafana and prometheus. [ceph: root@controller-0 /]# cat /mnt/prometheus --- networks: - 172.17.3.0/24 placement: hosts: - controller-0 - controller-1 - controller-2 service_id: prometheus service_name: prometheus service_type: prometheus spec: only_bind_port_on_networks: true [ceph: root@controller-0 /]# ceph orch apply -i /mnt/prometheus Scheduled prometheus update... [ceph: root@controller-0 /] I opened a BZ 2274719 to add it to alertmanager also.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Critical: Red Hat Ceph Storage 7.1 security, enhancements, and bug fix update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2024:3925