Description of problem: When testing the CDC with nvme-stas I am seeing the following log messages which appear to repeat indefinitely: Jun 14 08:10:08 rhel-storage-09.storage.lab.eng.bos.redhat.com stafd[1040]: (tcp, 172.18.220.70, 8009, nqn.1988-11.com.dell:SFSS:2:20230413041432mR, enp10s0f1np1) | nvme1 - Registration error. Status:0xffffffff - Interrupted system call. Jun 14 08:10:08 rhel-storage-09.storage.lab.eng.bos.redhat.com kernel: nvme nvme1: queue 0: timeout cid 0x9012 type 4 opcode 0x21 (Unknown) Jun 14 08:10:08 rhel-storage-09.storage.lab.eng.bos.redhat.com kernel: nvme nvme1: starting error recovery Jun 14 08:10:08 rhel-storage-09.storage.lab.eng.bos.redhat.com kernel: nvme nvme0: queue 0: timeout cid 0xe012 type 4 opcode 0x21 (Unknown) Jun 14 08:10:08 rhel-storage-09.storage.lab.eng.bos.redhat.com kernel: nvme nvme0: starting error recovery Jun 14 08:10:08 rhel-storage-09.storage.lab.eng.bos.redhat.com stafd[1040]: (tcp, 172.18.210.70, 8009, nqn.1988-11.com.dell:SFSS:1:20230413041432mR, enp10s0f0np0) | nvme0 - Registration error. Status:0xffffffff - Interrupted system call. Jun 14 08:10:08 rhel-storage-09.storage.lab.eng.bos.redhat.com stafd[1040]: (tcp, 172.18.210.70, 8009, nqn.1988-11.com.dell:SFSS:1:20230413041432mR, enp10s0f0np0) | nvme0 - Failed to retrieve log pages. RuntimeError: Discover failed (0) Jun 14 08:10:08 rhel-storage-09.storage.lab.eng.bos.redhat.com kernel: nvme nvme1: Reconnecting in 10 seconds... Jun 14 08:10:08 rhel-storage-09.storage.lab.eng.bos.redhat.com kernel: nvme nvme0: Reconnecting in 10 seconds... Jun 14 08:10:08 rhel-storage-09.storage.lab.eng.bos.redhat.com stafd[1040]: (tcp, 172.18.220.70, 8009, nqn.1988-11.com.dell:SFSS:2:20230413041432mR, enp10s0f1np1) | nvme1 - Failed to retrieve log pages. RuntimeError: Discover failed (0) Jun 14 08:10:18 rhel-storage-09.storage.lab.eng.bos.redhat.com kernel: nvme nvme0: queue_size 128 > ctrl sqsize 31, clamping down Jun 14 08:10:18 rhel-storage-09.storage.lab.eng.bos.redhat.com kernel: nvme nvme0: Successfully reconnected (1 attempt) Jun 14 08:10:18 rhel-storage-09.storage.lab.eng.bos.redhat.com kernel: nvme nvme1: queue_size 128 > ctrl sqsize 31, clamping down Jun 14 08:10:18 rhel-storage-09.storage.lab.eng.bos.redhat.com kernel: nvme nvme1: Successfully reconnected (1 attempt) Jun 14 08:10:18 rhel-storage-09.storage.lab.eng.bos.redhat.com stafd[1040]: (tcp, 172.18.210.70, 8009, nqn.1988-11.com.dell:SFSS:1:20230413041432mR, enp10s0f0np0) | nvme0 - Received discovery log pages (num records=2). Jun 14 08:10:18 rhel-storage-09.storage.lab.eng.bos.redhat.com stafd[1040]: (tcp, 172.18.220.70, 8009, nqn.1988-11.com.dell:SFSS:2:20230413041432mR, enp10s0f1np1) | nvme1 - Received discovery log pages (num records=2). Jun 14 08:10:26 rhel-storage-09.storage.lab.eng.bos.redhat.com stafd[1040]: (tcp, 172.18.210.70, 8009, nqn.1988-11.com.dell:SFSS:1:20230413041432mR, enp10s0f0np0) | nvme0 - Received AEN: 0x70f002 Jun 14 08:10:26 rhel-storage-09.storage.lab.eng.bos.redhat.com stafd[1040]: (tcp, 172.18.210.70, 8009, nqn.1988-11.com.dell:SFSS:1:20230413041432mR, enp10s0f0np0) | nvme0 - Received discovery log pages (num records=2). Jun 14 08:10:26 rhel-storage-09.storage.lab.eng.bos.redhat.com stafd[1040]: (tcp, 172.18.220.70, 8009, nqn.1988-11.com.dell:SFSS:2:20230413041432mR, enp10s0f1np1) | nvme1 - Received AEN: 0x70f002 Jun 14 08:10:26 rhel-storage-09.storage.lab.eng.bos.redhat.com stafd[1040]: (tcp, 172.18.220.70, 8009, nqn.1988-11.com.dell:SFSS:2:20230413041432mR, enp10s0f1np1) | nvme1 - Received discovery log pages (num records=2). Jun 14 08:10:28 rhel-storage-09.storage.lab.eng.bos.redhat.com stafd[1040]: (tcp, 172.18.220.70, 8009, nqn.1988-11.com.dell:SFSS:2:20230413041432mR, enp10s0f1np1) | nvme1 - Received discovery log pages (num records=2). Jun 14 08:10:28 rhel-storage-09.storage.lab.eng.bos.redhat.com stafd[1040]: (tcp, 172.18.210.70, 8009, nqn.1988-11.com.dell:SFSS:1:20230413041432mR, enp10s0f0np0) | nvme0 - Received discovery log pages (num records=2). The nvme namespaces appear to be available: [root@rhel-storage-09 log]# nvme list Node Generic SN Model Namespace Usage Format FW Rev --------------------- --------------------- -------------------- ---------------------------------------- ---------- -------------------------- ---------------- -------- /dev/nvme2n9 /dev/ng2n9 18S1BW2 dellemc-powerstore 0xf2 503.32 MB / 1.07 GB 512 B + 0 B 3.0.0.0 /dev/nvme2n8 /dev/ng2n8 18S1BW2 dellemc-powerstore 0xf1 503.32 MB / 1.07 GB 512 B + 0 B 3.0.0.0 /dev/nvme2n7 /dev/ng2n7 18S1BW2 dellemc-powerstore 0xf0 503.32 MB / 1.07 GB 512 B + 0 B 3.0.0.0 /dev/nvme2n6 /dev/ng2n6 18S1BW2 dellemc-powerstore 0xef 503.32 MB / 1.07 GB 512 B + 0 B 3.0.0.0 /dev/nvme2n5 /dev/ng2n5 18S1BW2 dellemc-powerstore 0xee 503.32 MB / 1.07 GB 512 B + 0 B 3.0.0.0 /dev/nvme2n4 /dev/ng2n4 18S1BW2 dellemc-powerstore 0xed 503.32 MB / 1.07 GB 512 B + 0 B 3.0.0.0 /dev/nvme2n32 /dev/ng2n32 18S1BW2 dellemc-powerstore 0x109 503.32 MB / 1.07 GB 512 B + 0 B 3.0.0.0 /dev/nvme2n31 /dev/ng2n31 18S1BW2 dellemc-powerstore 0x108 503.32 MB / 1.07 GB 512 B + 0 B 3.0.0.0 /dev/nvme2n30 /dev/ng2n30 18S1BW2 dellemc-powerstore 0x107 503.32 MB / 1.07 GB 512 B + 0 B 3.0.0.0 /dev/nvme2n3 /dev/ng2n3 18S1BW2 dellemc-powerstore 0xec 503.32 MB / 1.07 GB 512 B + 0 B 3.0.0.0 /dev/nvme2n29 /dev/ng2n29 18S1BW2 dellemc-powerstore 0x106 503.32 MB / 1.07 GB 512 B + 0 B 3.0.0.0 /dev/nvme2n28 /dev/ng2n28 18S1BW2 dellemc-powerstore 0x105 503.32 MB / 1.07 GB 512 B + 0 B 3.0.0.0 /dev/nvme2n27 /dev/ng2n27 18S1BW2 dellemc-powerstore 0x104 503.32 MB / 1.07 GB 512 B + 0 B 3.0.0.0 /dev/nvme2n26 /dev/ng2n26 18S1BW2 dellemc-powerstore 0x103 503.32 MB / 1.07 GB 512 B + 0 B 3.0.0.0 /dev/nvme2n25 /dev/ng2n25 18S1BW2 dellemc-powerstore 0x102 503.32 MB / 1.07 GB 512 B + 0 B 3.0.0.0 /dev/nvme2n24 /dev/ng2n24 18S1BW2 dellemc-powerstore 0x101 503.32 MB / 1.07 GB 512 B + 0 B 3.0.0.0 /dev/nvme2n23 /dev/ng2n23 18S1BW2 dellemc-powerstore 0x100 503.32 MB / 1.07 GB 512 B + 0 B 3.0.0.0 /dev/nvme2n22 /dev/ng2n22 18S1BW2 dellemc-powerstore 0xff 503.32 MB / 1.07 GB 512 B + 0 B 3.0.0.0 /dev/nvme2n21 /dev/ng2n21 18S1BW2 dellemc-powerstore 0xfe 503.32 MB / 1.07 GB 512 B + 0 B 3.0.0.0 /dev/nvme2n20 /dev/ng2n20 18S1BW2 dellemc-powerstore 0xfd 503.32 MB / 1.07 GB 512 B + 0 B 3.0.0.0 /dev/nvme2n2 /dev/ng2n2 18S1BW2 dellemc-powerstore 0xe 16.11 GB / 1.10 TB 512 B + 0 B 3.0.0.0 /dev/nvme2n19 /dev/ng2n19 18S1BW2 dellemc-powerstore 0xfc 503.32 MB / 1.07 GB 512 B + 0 B 3.0.0.0 /dev/nvme2n18 /dev/ng2n18 18S1BW2 dellemc-powerstore 0xfb 503.32 MB / 1.07 GB 512 B + 0 B 3.0.0.0 /dev/nvme2n17 /dev/ng2n17 18S1BW2 dellemc-powerstore 0xfa 503.32 MB / 1.07 GB 512 B + 0 B 3.0.0.0 /dev/nvme2n16 /dev/ng2n16 18S1BW2 dellemc-powerstore 0xf9 503.32 MB / 1.07 GB 512 B + 0 B 3.0.0.0 /dev/nvme2n15 /dev/ng2n15 18S1BW2 dellemc-powerstore 0xf8 503.32 MB / 1.07 GB 512 B + 0 B 3.0.0.0 /dev/nvme2n14 /dev/ng2n14 18S1BW2 dellemc-powerstore 0xf7 503.32 MB / 1.07 GB 512 B + 0 B 3.0.0.0 /dev/nvme2n13 /dev/ng2n13 18S1BW2 dellemc-powerstore 0xf6 503.32 MB / 1.07 GB 512 B + 0 B 3.0.0.0 /dev/nvme2n12 /dev/ng2n12 18S1BW2 dellemc-powerstore 0xf5 503.32 MB / 1.07 GB 512 B + 0 B 3.0.0.0 /dev/nvme2n11 /dev/ng2n11 18S1BW2 dellemc-powerstore 0xf4 503.32 MB / 1.07 GB 512 B + 0 B 3.0.0.0 /dev/nvme2n10 /dev/ng2n10 18S1BW2 dellemc-powerstore 0xf3 951.22 MB / 1.07 GB 512 B + 0 B 3.0.0.0 /dev/nvme2n1 /dev/ng2n1 18S1BW2 dellemc-powerstore 0x7 1.08 GB / 536.87 GB 512 B + 0 B 3.0.0 # nvme list-subsys nvme-subsys2 - NQN=nqn.1988-11.com.dell:powerstore:00:88b402df2d762AA7AF94 \ +- nvme2 tcp traddr=172.18.220.61,trsvcid=4420,host_iface=enp10s0f1np1,src_addr=172.18.220.36 live +- nvme3 tcp traddr=172.18.220.60,trsvcid=4420,host_iface=enp10s0f1np1,src_addr=172.18.220.36 live +- nvme4 tcp traddr=172.18.210.61,trsvcid=4420,host_iface=enp10s0f0np0,src_addr=172.18.210.35 live +- nvme5 tcp traddr=172.18.210.60,trsvcid=4420,host_iface=enp10s0f0np0,src_addr=172.18.210.35 live nvme-subsys1 - NQN=nqn.1988-11.com.dell:SFSS:2:20230413041432mR \ +- nvme1 tcp traddr=172.18.220.70,trsvcid=8009,host_iface=enp10s0f1np1,src_addr=172.18.220.36 live nvme-subsys0 - NQN=nqn.1988-11.com.dell:SFSS:1:20230413041432mR \ +- nvme0 tcp traddr=172.18.210.70,trsvcid=8009,host_iface=enp10s0f0np0,src_addr=172.18.210.35 live Version-Release number of selected component (if applicable): # rpm -qa nvme-stas nvme-stas-2.2.1-2.el9.noarch # uname -r 5.14.0-324.el9.x86_64 How reproducible: Often