Bug 1962492

I have added the /var/log/message in the link: https://drive.google.com/drive/folders/1J1Z1VDIj9HfZ7A9DTo5VU2ZpV-_zdYvZ?usp=sharing


tcmu-runner log files: 
[root@magna032 log]# podman logs 69273f266cd9
log file path now is '/var/log/tcmu-runner.log'

But no such file exits in the path. 


I deployed as follows:(old deployment)

1. I created an iscsi pool and enabled the rbd application
2. I labelled a couple of nodes with iscsi, then 
3. [ceph: root@maint-1 /]# ceph orch apply iscsi --pool iscsi --api_user admin --api_password admin --placement="label:iscsi"
Scheduled iscsi.iscsi update...


I have not used iscsi.yaml config here. But even after using the iscsi.yaml config with trusted_ip_list (on a different cluster seeing the same issue)

Hi Xiubo Li, Thanks for sharing the steps to find the above required logs. 

1. firewalld status: It is disabled in both the gateway nodes

[root@magna031 ~]# systemctl status firewalld
● firewalld.service - firewalld - dynamic firewall daemon
   Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled)
   Active: inactive (dead)
     Docs: man:firewalld(1)
[root@magna032 ~]# systemctl status firewalld
● firewalld.service - firewalld - dynamic firewall daemon
   Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled)
   Active: inactive (dead)
     Docs: man:firewalld(1)

2. tcmu-runner.log

on magna031 gateway node: 

sh-4.4# cat tcmu-runner.log 
2021-05-17 10:06:21.866 7:tcmu-runner [CRIT] main:1302: Starting...
2021-05-17 10:06:21.881 7:tcmu-runner [INFO] tcmur_register_handler:92: Handler rbd is registered
2021-05-17 10:06:21.956 7:tcmu-runner [INFO] tcmu_rbd_open:1162 rbd/rbd.disk_1: address: {10.8.128.31:0/3050073637}
2021-05-20 08:10:07.951 7:cmdproc-uio0 [INFO] alua_implicit_transition:581 rbd/rbd.disk_1: Starting read lock acquisition operation.
2021-05-20 08:10:07.952 7:ework-thread [INFO] tcmu_acquire_dev_lock:486 rbd/rbd.disk_1: Read lock acquisition successful


sh-4.4# cat tcmu-runner.log 
2021-05-17 10:06:56.026 7:tcmu-runner [CRIT] main:1302: Starting...
2021-05-17 10:06:56.041 7:tcmu-runner [INFO] tcmur_register_handler:92: Handler rbd is registered
2021-05-17 10:06:56.099 7:tcmu-runner [INFO] tcmu_rbd_open:1162 rbd/rbd.disk_1: address: {10.8.128.32:0/1572013376}
2021-05-19 06:50:27.534 7:cmdproc-uio0 [INFO] alua_implicit_transition:581 rbd/rbd.disk_1: Starting read lock acquisition operation.
2021-05-19 06:50:27.535 7:ework-thread [INFO] tcmu_acquire_dev_lock:486 rbd/rbd.disk_1: Read lock acquisition successful

3. rbd-target-gw logs - Folder is empty

sh-4.4# ls
ceph  dnf.librepo.log  dnf.log	dnf.rpm.log  ganesha  hawkey.log  maillog  messages  rbd-target-api  rbd-target-gw  rhsm  secure  spooler  tcmu-runner.log
sh-4.4# cd rbd-target-gw/
sh-4.4# ls  //Empty 


4. /etc/iscsi-gateway.cfg

I dont see the iscsi-gateway.cfg file in the container/node. 

5. I have attached the rbd-target-api logs

6. contents of /var/log 
sh-4.4# ls
ceph  dnf.librepo.log  dnf.log	dnf.rpm.log  ganesha  hawkey.log  maillog  messages  rbd-target-api  rbd-target-gw  rhsm  secure  spooler  tcmu-runner.log

Created attachment 1786270 [details]
rbd-target-api logs

Working as expected from QA with mixed ip's.

[ceph: root@magna007 ~]# cat iscsi.yaml 
service_type: iscsi
service_id: iscsi
placement:
  hosts:
  - magna007
  - magna010
spec:
  pool: iscsi_pool
  trusted_ip_list: "10.8.128.7,10.8.128.10,2620:52:0:880:225:90ff:fefc:2538,2620:52:0:880:225:90ff:fefc:252c"
  api_user: admin
  api_password: admin
  api_secure: false

/iscsi-targets> ls
o- iscsi-targets ................................................................................. [DiscoveryAuth: None, Targets: 1]
  o- iqn.2003-01.com.redhat.iscsi-gw:ceph-igw ............................................................ [Auth: None, Gateways: 2]
    o- disks ............................................................................................................ [Disks: 1]
    | o- rbd/disk_1 ...................................................................................... [Owner: magna007, Lun: 0]
    o- gateways .............................................................................................. [Up: 2/2, Portals: 2]
    | o- magna007 ................................................................................................ [10.8.128.7 (UP)]
    | o- magna010 ............................................................................................... [10.8.128.10 (UP)]
    o- host-groups .................................................................................................... [Groups : 0]
    o- hosts ......................................................................................... [Auth: ACL_ENABLED, Hosts: 1]
      o- iqn.1994-05.com.redhat:rh7-client .................................................. [LOGGED-IN, Auth: CHAP, Disks: 1(50G)]
        o- lun 0 ................................................................................ [rbd/disk_1(50G), Owner: magna007]
/iscsi-targets>


[root@magna108 ubuntu]# multipath -ll
.
.
.
3600140574bd1036b1d24cd6a13c11702 dm-6 LIO-ORG,TCMU device
size=50G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
|-+- policy='queue-length 0' prio=50 status=active
| `- 7:0:0:0 sdf 8:80 active ready running
`-+- policy='queue-length 0' prio=10 status=enabled
  `- 6:0:0:0 sde 8:64 active ready running
[root@magna108 ubuntu]#

[root@magna108 ubuntu]# fdisk -l
.
.
.
Disk /dev/mapper/ceph--14b17026--899d--4499--84d7--470abe3546f5-osd--block--a5ab9580--7b19--49bc--8d36--d2dad7cf9872: 931.5 GiB, 1000203091968 bytes, 1953521664 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/sde: 50 GiB, 53687091200 bytes, 104857600 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 524288 bytes


Disk /dev/sdf: 50 GiB, 53687091200 bytes, 104857600 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 524288 bytes


Disk /dev/mapper/3600140574bd1036b1d24cd6a13c11702: 50 GiB, 53687091200 bytes, 104857600 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 524288 bytes
[root@magna108 ubuntu]#

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Red Hat Ceph Storage 5.0 bug fix and enhancement), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:3294