Bug 2233659

Summary:	cephadm trys to bind grafana daemon to all (::) interfaces when valid networks list is provided.
Product:	[Red Hat Storage] Red Hat Ceph Storage	Reporter:	Matt Flusche <mflusche>
Component:	Cephadm	Assignee:	Adam King <adking>
Status:	CLOSED ERRATA	QA Contact:	Mohit Bisht <mobisht>
Severity:	high	Docs Contact:	Akash Raj <akraj>
Priority:	unspecified
Version:	6.0	CC:	adking, akraj, bmclaren, cephqe-warriors, c-harsh.kotak, fpantano, gfidente, mcaldeir, mkatari, mobisht, saraut, tserlin, vereddy
Target Milestone:	---	Flags:	mobisht: needinfo+
Target Release:	7.1
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:	ceph-18.2.1-87.el9cp	Doc Type:	Enhancement
Doc Text:	.Grafana now binds to an IP within a specific network on a host, rather that always binding to 0.0.0.0 With this enhancement, using a Grafana specification file that includes both the networks' section with the network that Grafana binds to an IP on, and `only_bind_port_on_networks: true` included in the "spec" section of the specification, Cephadm configures the Grafana daemon to bind to an IP within that network rather than 0.0.0.0. This enables users to use the same port that Grafana uses for another service but on a different IP on the host. If it is a specification update that does not cause them all to be moved, `ceph orch redeploy grafana` can be run to pick up the changes to the settings. Grafana specification file: ---- service_type: grafana service_name: grafana placement: count: 1 networks: 192.168.122.0/24 spec: anonymous_access: true protocol: https only_bind_port_on_networks: true ----	Story Points:	---
Clone Of:
Clones:	2246440 (view as bug list)		Environment:
Last Closed:	2024-06-13 14:20:51 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	2160010, 2236231, 2246440, 2254553, 2267614, 2298578, 2298579, 2350124

Description Matt Flusche 2023-08-22 20:25:14 UTC

Description of problem:


This is part of a osp 17.1 deployment with ceph 6,  the following error is blocking the grafana container from starting:

Deploy daemon grafana.overcloud-controller-1 ...
Verifying port 3100 ...
Cannot bind to IP :: port 3100: [Errno 98] Address already in use
ERROR: TCP Port(s) '3100' required for grafana already in use

The in use address is haproxy on a different interface


The config looks good.  From "ceph orch ls --export"

---
service_type: grafana
service_name: grafana
placement:
  hosts:
  - overcloud-controller-0
  - overcloud-controller-1
  - overcloud-controller-2
networks:
- 2001:db8:1:9::/64
- 2001:db8:1:c::/64
- 2001:db8:1:b::/64
- 2001:db8:1:a::/64
- 2001:db8:1:d::/64
- 2001:db8:1:8::/64
spec:
  port: 3100
---

If I understand correctly, the "networks" option should limit binding to interfaces contained there.

Here is overcloud-controller-0 interface information showing a valid interface for binding.

overcloud-controller-0]$ grep 2001:db8:1 ip_addr 
16: vlan123    inet6 2001:db8:1:8::b5/64 scope global \       valid_lft forever preferred_lft forever

It should only bind to 2001:db8:1:8::b5:3100

This seems to also impact other services such as prometheus & alertmanager but likely the same issue.

I'll provide more details and logs in private comments.


Version-Release number of selected component (if applicable):

cephadm-17.2.6-70.el9cp.noarch
ceph 6 deployment

How reproducible:
this environment


Steps to Reproduce:
1. see notes above

Actual results:

grafana daemon attempting to bind to all interfaces and failing.


Expected results:

specific interface based on networks configuration.

Additional info:

In private comments.

Comment 5 Francesco Pantano 2023-10-16 06:51:28 UTC

*** Bug 2236231 has been marked as a duplicate of this bug. ***

Comment 20 Manoj Katari 2024-04-11 14:03:27 UTC

I don't see the parameter "only_bind_port_on_networks" in image quay.io/ceph/ceph:v18.2.1. I would like test it by introducing the parameter in tripleo.

Can you share the container image i can use to test it ?

Comment 21 Adam King 2024-04-11 14:29:43 UTC

(In reply to Manoj Katari from comment #20)
> I don't see the parameter "only_bind_port_on_networks" in image
> quay.io/ceph/ceph:v18.2.1. I would like test it by introducing the parameter
> in tripleo.
> 
> Can you share the container image i can use to test it ?

I haven't done a quincy backport yet as there were a lot of merge conflicts to deal with there. If you're okay with reef though, the regular upstream CI image quay.ceph.io/ceph-ci/ceph:reef should have it.

Comment 22 Manoj Katari 2024-04-11 15:15:06 UTC

i tried the image you suggested but i still don't the support for the parameter

[root@controller-0 ~]# podman run --rm --net=host -v /etc/ceph:/etc/ceph:z -v /var/lib/ceph/:/var/lib/ceph/:z -v /var/log/ceph/:/var/log/ceph/:z -v /home/ceph-admin/specs/grafana:/home/ceph-admin/specs/grafana:z --entrypoint=ceph quay.ceph.io/ceph-ci/ceph:reef -n client.admin -k /etc/ceph/ceph.client.admin.keyring --cluster ceph orch apply --in-file /home/ceph-admin/specs/grafana
Error EINVAL: ServiceSpec: __init__() got an unexpected keyword argument 'only_bind_port_on_networks'

Comment 23 Adam King 2024-04-11 17:26:14 UTC

(In reply to Manoj Katari from comment #22)
> i tried the image you suggested but i still don't the support for the
> parameter
> 
> [root@controller-0 ~]# podman run --rm --net=host -v /etc/ceph:/etc/ceph:z
> -v /var/lib/ceph/:/var/lib/ceph/:z -v /var/log/ceph/:/var/log/ceph/:z -v
> /home/ceph-admin/specs/grafana:/home/ceph-admin/specs/grafana:z
> --entrypoint=ceph quay.ceph.io/ceph-ci/ceph:reef -n client.admin -k
> /etc/ceph/ceph.client.admin.keyring --cluster ceph orch apply --in-file
> /home/ceph-admin/specs/grafana
> Error EINVAL: ServiceSpec: __init__() got an unexpected keyword argument
> 'only_bind_port_on_networks'

maybe just a formatting thing in the spec? I think the setting has to go into the `spec` section for example. I was able to make use of it with the recommended image

---

[root@vm-00 ~]# podman images
REPOSITORY                        TAG         IMAGE ID      CREATED        SIZE
quay.ceph.io/ceph-ci/ceph         reef        bb2c5eac26a4  2 days ago     1.37 GB
quay.io/ceph/ceph-grafana         9.4.7       954c08fa6188  4 months ago   647 MB
quay.io/prometheus/prometheus     v2.43.0     a07b618ecd1d  12 months ago  235 MB
quay.io/prometheus/alertmanager   v0.25.0     c8568f914cd2  15 months ago  66.5 MB
quay.io/prometheus/node-exporter  v1.5.0      0da6a335fe13  16 months ago  23.9 MB
[root@vm-00 ~]# 
[root@vm-00 ~]# cephadm shell
Inferring fsid 1fd068c6-f820-11ee-b857-5254002b59e2
Inferring config /var/lib/ceph/1fd068c6-f820-11ee-b857-5254002b59e2/mon.vm-00/config
Using ceph image with id 'bb2c5eac26a4' and tag 'reef' created on 2024-04-09 00:48:11 +0000 UTC
quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3
[ceph: root@vm-00 /]# 
[ceph: root@vm-00 /]# 
[ceph: root@vm-00 /]# ceph orch ls grafana --export > grafana.yaml
[ceph: root@vm-00 /]# 
[ceph: root@vm-00 /]# vi grafana.yaml 
[ceph: root@vm-00 /]# 
[ceph: root@vm-00 /]# cat grafana.yaml 
service_type: grafana
service_name: grafana
placement:
  count: 1
networks:
  192.168.122.0/24
spec:
  anonymous_access: true
  protocol: https
  only_bind_port_on_networks: true
[ceph: root@vm-00 /]# 
[ceph: root@vm-00 /]# ceph orch apply -i grafana.yaml 
Scheduled grafana update...
[ceph: root@vm-00 /]# 
[ceph: root@vm-00 /]# ceph orch ls grafana --export               
service_type: grafana
service_name: grafana
placement:
  count: 1
networks:
- 192.168.122.0/24
spec:
  anonymous_access: true
  only_bind_port_on_networks: true
  protocol: https

Comment 24 Manoj Katari 2024-04-12 12:35:55 UTC

Thanks Adam, it was my mistake earlier. I could see the new parameter is supported for both grafana and prometheus.

[ceph: root@controller-0 /]# cat /mnt/prometheus                                                                                                                                                                                             
---                                                                                              
networks:                                                                                        
- 172.17.3.0/24                                                                                  
placement:                                                                                          
  hosts:                                                                                        
  - controller-0                                                                                                                                             
  - controller-1                                                                                                                                             
  - controller-2                                                                                                                                             
service_id: prometheus                                                                                                                                                                                                                       
service_name: prometheus                                                                                                                                                                                                                     
service_type: prometheus                                                                                                                                                                                                                     
spec:                                                                                                                                                                                                                                        
  only_bind_port_on_networks: true                                                                                                                                                                                                              
[ceph: root@controller-0 /]# ceph orch apply -i /mnt/prometheus                                                                                                                                                                              
Scheduled prometheus update...                                                                                                                                                                                                               
[ceph: root@controller-0 /]



I opened a BZ 2274719 to add it to alertmanager also.

Comment 27 errata-xmlrpc 2024-06-13 14:20:51 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Critical: Red Hat Ceph Storage 7.1 security, enhancements, and bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2024:3925