Bug 2014500

Summary: beast not working on IPv6 failed to parse endpoint=fd00:fd00:fd00:3000::397:808
Product: [Red Hat Storage] Red Hat Ceph Storage Reporter: John Fulton <johfulto>
Component: CephadmAssignee: Adam King <adking>
Status: CLOSED ERRATA QA Contact: Sunil Kumar Nagaraju <sunnagar>
Severity: high Docs Contact: Ranjini M N <rmandyam>
Priority: high    
Version: 5.0CC: adking, agunn, aoconnor, cbodley, ceph-eng-bugs, fpantano, gfidente, gsitlani, kbader, mbenjamin, rlepaksh, rmandyam, sunnagar, tserlin, vereddy
Target Milestone: ---   
Target Release: 5.1   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: ceph-16.2.6-41.el8cp Doc Type: Enhancement
Doc Text:
.`cephadm` now supports IPv6 networks for Ceph Object Gateway deployment With this release, `cephadm` supports specifying an IPv6 network for Ceph Object Gateway specifications. An example of a service configuration file for deploying Ceph Object Gateway is: .Example ---- service_type: rgw service_id: rgw placement: count: 3 networks: - fd00:fd00:3000::/64 ----
 Story Points: ---
Clone Of: Environment:
Last Closed: 2022-04-04 10:22:04 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1820257, 2031073    

Description John Fulton 2021-10-15 12:10:40 UTC 
Description of problem:

Beast as deployed by cephadm is not starting with IPv6:

Oct 15 11:47:43 controller-0 conmon[477168]: debug 2021-10-15T11:47:43.232+0000 7febbcaf1480  0 starting handler: beast                                                                      
Oct 15 11:47:43 controller-0 conmon[477168]: debug 2021-10-15T11:47:43.234+0000 7febbcaf1480 -1 failed to parse endpoint=fd00:fd00:fd00:3000::397:8080                                       
Oct 15 11:47:43 controller-0 conmon[477168]: debug 2021-10-15T11:47:43.234+0000 7febbcaf1480 -1 ERROR: failed initializing frontend

Similar to BZ 1644840

Version-Release number of selected component (if applicable): 

- ceph-16.2.0-130.el8cp (FIV of BZ 2002639)
- ceph container 5-14

How reproducible: deterministic

Steps to Reproduce:
1. Deploy with cephadm and observe working mon,mgr,osd
2. Apply spec [1] for RGW 
3. Observe in `ceph orch ls` that the RGW service is not running 
4. Try to start RGW w/ systemctl start [2]
5. Observe error in journalctl logs [3]

Actual results:

 [ceph: root@controller-0 /]# ceph orch ls
NAME                     RUNNING  REFRESHED  AGE  PLACEMENT                              
crash                        7/7  7m ago     14h  *                                      
mgr                          3/3  7m ago     14h  controller-0;controller-1;controller-2  
mon                          3/3  7m ago     14h  controller-0;controller-1;controller-2  
osd.default_drive_group      4/8  7m ago     14h  ceph-0;ceph-1;ceph-2;ceph-3            
rgw.rgw                      0/3  7m ago     14h  controller-0;controller-1;controller-2  
[ceph: root@controller-0 /]#

Expected results:
rgw.rgw                      3/3  7m ago     14h  controller-0;controller-1;controller-2  


Additional info:

As per https://tracker.ceph.com/issues/52867 we needed to set an additional setting for OSD to work on IPv6. Is there a similar setting required for RGW? As per 1644840, I'd think this bug is fixed and but maybe I'm missing config option?


Footnotes:

[1] Spec for RGW
"""
---
service_type: rgw
service_id: rgw
service_name: rgw.rgw
placement:
  hosts:
  - controller-0
  - controller-1
  - controller-2
networks:
- fd00:fd00:fd00:3000::/64
spec:
  rgw_frontend_port: 8080
  rgw_realm: default
  rgw_zone: default
"""

[2] [root@controller-0 ceph]# systemctl start ceph-8552069d-fddd-4d77-81e5-9e07cf282f81.controller-0.glbqfs.service

[3] [root@controller-0 ceph]# journalctl -xef -u ceph-8552069d-fddd-4d77-81e5-9e07cf282f81.controller-0.glbqfs.service
...
Oct 15 11:47:43 controller-0 conmon[477168]: debug 2021-10-15T11:47:43.050+0000 7febbcaf1480  0 framework: beast                                                                             
Oct 15 11:47:43 controller-0 conmon[477168]: debug 2021-10-15T11:47:43.050+0000 7febbcaf1480  0 framework conf key: endpoint, val: fd00:fd00:fd00:3000::397:8080                             
Oct 15 11:47:43 controller-0 conmon[477168]: debug 2021-10-15T11:47:43.050+0000 7febbcaf1480  1 radosgw_Main not setting numa affinity                                                       
Oct 15 11:47:43 controller-0 conmon[477168]: debug 2021-10-15T11:47:43.227+0000 7febbcaf1480  0 framework: beast                                                                             
Oct 15 11:47:43 controller-0 conmon[477168]: debug 2021-10-15T11:47:43.227+0000 7febbcaf1480  0 framework conf key: ssl_certificate, val: config://rgw/cert/$realm/$zone.crt                 
Oct 15 11:47:43 controller-0 conmon[477168]: debug 2021-10-15T11:47:43.227+0000 7febbcaf1480  0 framework conf key: ssl_private_key, val: config://rgw/cert/$realm/$zone.key                 
Oct 15 11:47:43 controller-0 conmon[477168]: debug 2021-10-15T11:47:43.227+0000 7feb7eed3700  0 RGWReshardLock::lock failed to acquire lock on reshard.0000000001 ret=-16                    
Oct 15 11:47:43 controller-0 conmon[477168]: debug 2021-10-15T11:47:43.232+0000 7febbcaf1480  0 starting handler: beast                                                                      
Oct 15 11:47:43 controller-0 conmon[477168]: debug 2021-10-15T11:47:43.234+0000 7febbcaf1480 -1 failed to parse endpoint=fd00:fd00:fd00:3000::397:8080                                       
Oct 15 11:47:43 controller-0 conmon[477168]: debug 2021-10-15T11:47:43.234+0000 7febbcaf1480 -1 ERROR: failed initializing frontend                                                          
Oct 15 11:47:43 controller-0 podman[477332]: 2021-10-15 11:47:43.378281038 +0000 UTC m=+0.101282259 container died e5fbe91a7d53a857457556e600b2356aef642442808df45f19ef6d42ff9d7667 (image=und
ercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhceph@sha256:55cb1de88341300daa1ab6d59e4897edc733a3f90162149c21f18abe49ed87c7, name=ceph-8552069d-fddd-4d77-81e5-9e07cf282f81-rgw-rgw-controller
-0-glbqfs)
Comment 1 RHEL Program Management 2021-10-15 12:10:47 UTC 
Please specify the severity of this bug. Severity is defined here:
https://bugzilla.redhat.com/page.cgi?id=fields.html#bug_severity.
Comment 2 Casey Bodley 2021-10-15 13:36:27 UTC 
that parser expects v6 addrs to be in square brackets. have you tried [fd00:fd00:fd00:3000::397:8080]?
Comment 4 Sebastian Wagner 2021-10-20 13:42:21 UTC 
https://github.com/ceph/ceph/pull/43602
Comment 12 Sebastian Wagner 2021-12-23 09:32:11 UTC 
Could you dump the config for the RGW frontend?
Comment 25 errata-xmlrpc 2022-04-04 10:22:04 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: Red Hat Ceph Storage 5.1 Security, Enhancement, and Bug Fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:1174