Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
This project is now read‑only. Starting Monday, February 2, please use https://ibm-ceph.atlassian.net/ for all bug tracking management.

Bug 2240179

Summary: create_listener always successful with wrong inputs ends up in OMAP
Product: [Red Hat Storage] Red Hat Ceph Storage Reporter: Sunil Kumar Nagaraju <sunnagar>
Component: NVMeOFAssignee: Aviv Caro <acaro>
Status: CLOSED COMPLETED QA Contact: Manohar Murthy <mmurthy>
Severity: urgent Docs Contact: Rivka Pollack <rpollack>
Priority: unspecified    
Version: 7.0CC: acaro, akraj, cephqe-warriors, mmurthy, tserlin, vereddy
Target Milestone: ---   
Target Release: 7.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2023-12-13 15:23:56 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 2237662    

Description Sunil Kumar Nagaraju 2023-09-22 09:51:06 UTC
Description of problem:

create_listener always successful with wrong inputs ends up in OMAP

>>>Here, the gateway name doesn't belongs to this gateway service.
>>>and another negative input was invalid gateway name was provided to cli, which ended up as entries in omap object. 
>>>
>>># podman run quay.io/ceph/nvmeof-cli:0.0.3 --server-address 10.8.130.11 --server-port 5500 create_listener -n nqn.2016-06.io.spdk:cnode1 -g client.nvmeof.rbd.cali009.nzkcff -a 10.8.130.11 -s 4420 
>>>INFO:__main__:Created nqn.2016-06.io.spdk:cnode1 listener: True
>>>
>>>INFO:control.grpc:Received request to create client.nvmeof.rbd.cali009.nzkcff TCP listener for nqn.2016-06.io.spdk:cnode1 at 10.8.130.11:4420.
>>>DEBUG:control.state:omap_key generated: listener_nqn.2016-06.io.spdk:cnode1_client.nvmeof.rbd.cali009.nzkcff_TCP_10.8.130.11_4420
>>>
>>>
>>>---------------------------------------
>>>No listener entries found, since it is created at all 
>>># podman run quay.io/ceph/nvmeof-cli:0.0.3 --server-address 10.8.130.11 --server-port 5500 get_subsystems
>>>INFO:__main__:Get subsystems:
>>>[
>>>    {
>>>        "nqn": "nqn.2014-08.org.nvmexpress.discovery",
>>>        "subtype": "Discovery",
>>>        "listen_addresses": [],
>>>        "allow_any_host": true,
>>>        "hosts": []
>>>    },
>>>    {
>>>        "nqn": "nqn.2016-06.io.spdk:cnode1",
>>>        "subtype": "NVMe",
>>>        "listen_addresses": [],
>>>        "allow_any_host": false,
>>>        "hosts": [],
>>>        "serial_number": "1",
>>>        "model_number": "SPDK bdev Controller",
>>>        "max_namespaces": 32,
>>>        "min_cntlid": 1,
>>>        "max_cntlid": 65519,
>>>        "namespaces": [
>>>            {
>>>                "nsid": 1,
>>>                "bdev_name": "bdev1",
>>>                "name": "bdev1",
>>>                "nguid": "BBB4C5EF469A466BBFA20D8CE352717F",
>>>                "uuid": "bbb4c5ef-469a-466b-bfa2-0d8ce352717f"
>>>            }
>>>        ]
>>>    }
>>>
>>>--------------------------------------------------------
>>>listener entries at OMAP is created, without having listener entity created at the subsystem.
>>># rados -p rbd1 listomapvals nvmeof.state
>>>bdev_bdev1
>>>value (154 bytes) :
>>>00000000  7b 0a 20 20 22 62 64 65  76 5f 6e 61 6d 65 22 3a  |{.  "bdev_name":|
>>>00000010  20 22 62 64 65 76 31 22  2c 0a 20 20 22 72 62 64  | "bdev1",.  "rbd|
>>>00000020  5f 70 6f 6f 6c 5f 6e 61  6d 65 22 3a 20 22 72 62  |_pool_name": "rb|
>>>00000030  64 22 2c 0a 20 20 22 72  62 64 5f 69 6d 61 67 65  |d",.  "rbd_image|
>>>00000040  5f 6e 61 6d 65 22 3a 20  22 69 6d 61 67 65 31 22  |_name": "image1"|
>>>00000050  2c 0a 20 20 22 62 6c 6f  63 6b 5f 73 69 7a 65 22  |,.  "block_size"|
>>>00000060  3a 20 34 30 39 36 2c 0a  20 20 22 75 75 69 64 22  |: 4096,.  "uuid"|
>>>00000070  3a 20 22 62 62 62 34 63  35 65 66 2d 34 36 39 61  |: "bbb4c5ef-469a|
>>>00000080  2d 34 36 36 62 2d 62 66  61 32 2d 30 64 38 63 65  |-466b-bfa2-0d8ce|
>>>00000090  33 35 32 37 31 37 66 22  0a 7d                    |352717f".}|
>>>0000009a
>>>
>>>listener_nqn.2016-06.io.spdk:cnode1_client.nvmeof.rbd.cali009.nzkcff_TCP_10.8.130.11_4420
>>>value (182 bytes) :
>>>00000000  7b 0a 20 20 22 6e 71 6e  22 3a 20 22 6e 71 6e 2e  |{.  "nqn": "nqn.|
>>>00000010  32 30 31 36 2d 30 36 2e  69 6f 2e 73 70 64 6b 3a  |2016-06.io.spdk:|
>>>00000020  63 6e 6f 64 65 31 22 2c  0a 20 20 22 67 61 74 65  |cnode1",.  "gate|
>>>00000030  77 61 79 5f 6e 61 6d 65  22 3a 20 22 63 6c 69 65  |way_name": "clie|
>>>00000040  6e 74 2e 6e 76 6d 65 6f  66 2e 72 62 64 2e 63 61  |nt.nvmeof.rbd.ca|
>>>00000050  6c 69 30 30 39 2e 6e 7a  6b 63 66 66 22 2c 0a 20  |li009.nzkcff",. |
>>>00000060  20 22 74 72 74 79 70 65  22 3a 20 22 54 43 50 22  | "trtype": "TCP"|
>>>00000070  2c 0a 20 20 22 61 64 72  66 61 6d 22 3a 20 22 69  |,.  "adrfam": "i|
>>>00000080  70 76 34 22 2c 0a 20 20  22 74 72 61 64 64 72 22  |pv4",.  "traddr"|
>>>00000090  3a 20 22 31 30 2e 38 2e  31 33 30 2e 31 31 22 2c  |: "10.8.130.11",|
>>>000000a0  0a 20 20 22 74 72 73 76  63 69 64 22 3a 20 22 34  |.  "trsvcid": "4|
>>>000000b0  34 32 30 22 0a 7d                                 |420".}|
>>>000000b6
>>>
>>>listener_nqn.2016-06.io.spdk:cnode1_nvmeof.rbd.cali009.nzkcff_TCP_10.8.130.11_4420
>>>value (175 bytes) :
>>>00000000  7b 0a 20 20 22 6e 71 6e  22 3a 20 22 6e 71 6e 2e  |{.  "nqn": "nqn.|
>>>00000010  32 30 31 36 2d 30 36 2e  69 6f 2e 73 70 64 6b 3a  |2016-06.io.spdk:|
>>>00000020  63 6e 6f 64 65 31 22 2c  0a 20 20 22 67 61 74 65  |cnode1",.  "gate|
>>>00000030  77 61 79 5f 6e 61 6d 65  22 3a 20 22 6e 76 6d 65  |way_name": "nvme|
>>>00000040  6f 66 2e 72 62 64 2e 63  61 6c 69 30 30 39 2e 6e  |of.rbd.cali009.n|
>>>00000050  7a 6b 63 66 66 22 2c 0a  20 20 22 74 72 74 79 70  |zkcff",.  "trtyp|
>>>00000060  65 22 3a 20 22 54 43 50  22 2c 0a 20 20 22 61 64  |e": "TCP",.  "ad|
>>>00000070  72 66 61 6d 22 3a 20 22  69 70 76 34 22 2c 0a 20  |rfam": "ipv4",. |
>>>00000080  20 22 74 72 61 64 64 72  22 3a 20 22 31 30 2e 38  | "traddr": "10.8|
>>>00000090  2e 31 33 30 2e 31 31 22  2c 0a 20 20 22 74 72 73  |.130.11",.  "trs|
>>>000000a0  76 63 69 64 22 3a 20 22  34 34 32 30 22 0a 7d     |vcid": "4420".}|
>>>000000af
>>>
>>>

Version-Release number of selected component (if applicable):
# ceph versions
{
    "mon": {
        "ceph version 18.0.0-6249-g70973ad6 (70973ad678e6322356dcc6ff69b79e4315a1d0c7) reef (dev)": 3
    },
    "mgr": {
        "ceph version 18.0.0-6249-g70973ad6 (70973ad678e6322356dcc6ff69b79e4315a1d0c7) reef (dev)": 2
    },
    "osd": {
        "ceph version 18.0.0-6249-g70973ad6 (70973ad678e6322356dcc6ff69b79e4315a1d0c7) reef (dev)": 11
    },
    "overall": {
        "ceph version 18.0.0-6249-g70973ad6 (70973ad678e6322356dcc6ff69b79e4315a1d0c7) reef (dev)": 16
    }
}



How reproducible: always


Additional info: https://github.com/ceph/ceph-nvmeof/issues/240

Comment 1 Aviv Caro 2023-09-26 15:13:35 UTC
Sunil, I don't think this is urgent for the TP. We do have a fix for that in upstream, it should be included in 0.0.4. Meanwhile I suggest to reduce severity to medium.

Comment 2 Aviv Caro 2023-10-04 08:27:03 UTC
Fixed in 0.0.4.

Comment 3 Sunil Kumar Nagaraju 2023-10-05 12:50:14 UTC
Unable to add listener if gateway address is wrong,

# podman run registry-proxy.engineering.redhat.com/rh-osbs/ceph-nvmeof-cli:0.0.3-1 --server-address 10.0.209.201 --server-port 5500 create_listener  --subnqn nqn.2016-06.io.spdk:cnode1 --trsvcid 5002 --gateway-name client.nvmeof.rbd.ceph-2sunilkumar-nhs96b-node6.egpoy --traddr 10.0.209.201
usage: python3 -m control.cli [-h] [--server-address SERVER_ADDRESS]
                              [--server-port SERVER_PORT]
                              [--client-key CLIENT_KEY]
                              [--client-cert CLIENT_CERT]
                              [--server-cert SERVER_CERT]
                              {create_bdev,delete_bdev,create_subsystem,delete_subsystem,add_namespace,remove_namespace,add_host,remove_host,create_listener,delete_listener,get_subsystems}
                              ...
python3 -m control.cli: error: create_listener failed: code=StatusCode.INTERNAL message=Gateway name must match current gateway (client.nvmeof.rbd.ceph-2sunilkumar-nhs96b-node6.egpoyz)

Comment 9 Akash Raj 2023-10-11 09:12:23 UTC
Hi Aviv.

Please confirm if this needs to be added to 7.0 release notes. If so, please provide the doc type and text.

Thanks.

Comment 10 errata-xmlrpc 2023-12-13 15:23:56 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Red Hat Ceph Storage 7.0 Bug Fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2023:7780

Comment 11 Red Hat Bugzilla 2024-05-24 04:25:07 UTC
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 120 days