Bug 2215005

Summary: [DELL SFSS] nvme-stas - Registration error. Status:0xffffffff - Interrupted system call
Product: Red Hat Enterprise Linux 9 Reporter: Marco Patalano <mpatalan>
Component: nvme-stasAssignee: Maurizio Lombardi <mlombard>
Status: NEW --- QA Contact: Zhang Yi <yizhan>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 9.3CC: kasmith
Target Milestone: rc   
Target Release: 9.3   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 2186842, 2186843    

Description Marco Patalano 2023-06-14 12:18:55 UTC 
Description of problem: When testing the CDC with nvme-stas I am seeing the following log messages which appear to repeat indefinitely:

Jun 14 08:10:08 rhel-storage-09.storage.lab.eng.bos.redhat.com stafd[1040]: (tcp, 172.18.220.70, 8009, nqn.1988-11.com.dell:SFSS:2:20230413041432mR, enp10s0f1np1) | nvme1 - Registration error. Status:0xffffffff - Interrupted system call.
Jun 14 08:10:08 rhel-storage-09.storage.lab.eng.bos.redhat.com kernel: nvme nvme1: queue 0: timeout cid 0x9012 type 4 opcode 0x21 (Unknown)
Jun 14 08:10:08 rhel-storage-09.storage.lab.eng.bos.redhat.com kernel: nvme nvme1: starting error recovery
Jun 14 08:10:08 rhel-storage-09.storage.lab.eng.bos.redhat.com kernel: nvme nvme0: queue 0: timeout cid 0xe012 type 4 opcode 0x21 (Unknown)
Jun 14 08:10:08 rhel-storage-09.storage.lab.eng.bos.redhat.com kernel: nvme nvme0: starting error recovery
Jun 14 08:10:08 rhel-storage-09.storage.lab.eng.bos.redhat.com stafd[1040]: (tcp, 172.18.210.70, 8009, nqn.1988-11.com.dell:SFSS:1:20230413041432mR, enp10s0f0np0) | nvme0 - Registration error. Status:0xffffffff - Interrupted system call.
Jun 14 08:10:08 rhel-storage-09.storage.lab.eng.bos.redhat.com stafd[1040]: (tcp, 172.18.210.70, 8009, nqn.1988-11.com.dell:SFSS:1:20230413041432mR, enp10s0f0np0) | nvme0 - Failed to retrieve log pages. RuntimeError: Discover failed (0)
Jun 14 08:10:08 rhel-storage-09.storage.lab.eng.bos.redhat.com kernel: nvme nvme1: Reconnecting in 10 seconds...
Jun 14 08:10:08 rhel-storage-09.storage.lab.eng.bos.redhat.com kernel: nvme nvme0: Reconnecting in 10 seconds...
Jun 14 08:10:08 rhel-storage-09.storage.lab.eng.bos.redhat.com stafd[1040]: (tcp, 172.18.220.70, 8009, nqn.1988-11.com.dell:SFSS:2:20230413041432mR, enp10s0f1np1) | nvme1 - Failed to retrieve log pages. RuntimeError: Discover failed (0)
Jun 14 08:10:18 rhel-storage-09.storage.lab.eng.bos.redhat.com kernel: nvme nvme0: queue_size 128 > ctrl sqsize 31, clamping down
Jun 14 08:10:18 rhel-storage-09.storage.lab.eng.bos.redhat.com kernel: nvme nvme0: Successfully reconnected (1 attempt)
Jun 14 08:10:18 rhel-storage-09.storage.lab.eng.bos.redhat.com kernel: nvme nvme1: queue_size 128 > ctrl sqsize 31, clamping down
Jun 14 08:10:18 rhel-storage-09.storage.lab.eng.bos.redhat.com kernel: nvme nvme1: Successfully reconnected (1 attempt)
Jun 14 08:10:18 rhel-storage-09.storage.lab.eng.bos.redhat.com stafd[1040]: (tcp, 172.18.210.70, 8009, nqn.1988-11.com.dell:SFSS:1:20230413041432mR, enp10s0f0np0) | nvme0 - Received discovery log pages (num records=2).
Jun 14 08:10:18 rhel-storage-09.storage.lab.eng.bos.redhat.com stafd[1040]: (tcp, 172.18.220.70, 8009, nqn.1988-11.com.dell:SFSS:2:20230413041432mR, enp10s0f1np1) | nvme1 - Received discovery log pages (num records=2).
Jun 14 08:10:26 rhel-storage-09.storage.lab.eng.bos.redhat.com stafd[1040]: (tcp, 172.18.210.70, 8009, nqn.1988-11.com.dell:SFSS:1:20230413041432mR, enp10s0f0np0) | nvme0 - Received AEN: 0x70f002
Jun 14 08:10:26 rhel-storage-09.storage.lab.eng.bos.redhat.com stafd[1040]: (tcp, 172.18.210.70, 8009, nqn.1988-11.com.dell:SFSS:1:20230413041432mR, enp10s0f0np0) | nvme0 - Received discovery log pages (num records=2).
Jun 14 08:10:26 rhel-storage-09.storage.lab.eng.bos.redhat.com stafd[1040]: (tcp, 172.18.220.70, 8009, nqn.1988-11.com.dell:SFSS:2:20230413041432mR, enp10s0f1np1) | nvme1 - Received AEN: 0x70f002
Jun 14 08:10:26 rhel-storage-09.storage.lab.eng.bos.redhat.com stafd[1040]: (tcp, 172.18.220.70, 8009, nqn.1988-11.com.dell:SFSS:2:20230413041432mR, enp10s0f1np1) | nvme1 - Received discovery log pages (num records=2).
Jun 14 08:10:28 rhel-storage-09.storage.lab.eng.bos.redhat.com stafd[1040]: (tcp, 172.18.220.70, 8009, nqn.1988-11.com.dell:SFSS:2:20230413041432mR, enp10s0f1np1) | nvme1 - Received discovery log pages (num records=2).
Jun 14 08:10:28 rhel-storage-09.storage.lab.eng.bos.redhat.com stafd[1040]: (tcp, 172.18.210.70, 8009, nqn.1988-11.com.dell:SFSS:1:20230413041432mR, enp10s0f0np0) | nvme0 - Received discovery log pages (num records=2).

The nvme namespaces appear to be available:

[root@rhel-storage-09 log]# nvme list
Node                  Generic               SN                   Model                                    Namespace  Usage                      Format           FW Rev  
--------------------- --------------------- -------------------- ---------------------------------------- ---------- -------------------------- ---------------- --------
/dev/nvme2n9          /dev/ng2n9            18S1BW2              dellemc-powerstore                       0xf2       503.32  MB /   1.07  GB    512   B +  0 B   3.0.0.0 
/dev/nvme2n8          /dev/ng2n8            18S1BW2              dellemc-powerstore                       0xf1       503.32  MB /   1.07  GB    512   B +  0 B   3.0.0.0 
/dev/nvme2n7          /dev/ng2n7            18S1BW2              dellemc-powerstore                       0xf0       503.32  MB /   1.07  GB    512   B +  0 B   3.0.0.0 
/dev/nvme2n6          /dev/ng2n6            18S1BW2              dellemc-powerstore                       0xef       503.32  MB /   1.07  GB    512   B +  0 B   3.0.0.0 
/dev/nvme2n5          /dev/ng2n5            18S1BW2              dellemc-powerstore                       0xee       503.32  MB /   1.07  GB    512   B +  0 B   3.0.0.0 
/dev/nvme2n4          /dev/ng2n4            18S1BW2              dellemc-powerstore                       0xed       503.32  MB /   1.07  GB    512   B +  0 B   3.0.0.0 
/dev/nvme2n32         /dev/ng2n32           18S1BW2              dellemc-powerstore                       0x109      503.32  MB /   1.07  GB    512   B +  0 B   3.0.0.0 
/dev/nvme2n31         /dev/ng2n31           18S1BW2              dellemc-powerstore                       0x108      503.32  MB /   1.07  GB    512   B +  0 B   3.0.0.0 
/dev/nvme2n30         /dev/ng2n30           18S1BW2              dellemc-powerstore                       0x107      503.32  MB /   1.07  GB    512   B +  0 B   3.0.0.0 
/dev/nvme2n3          /dev/ng2n3            18S1BW2              dellemc-powerstore                       0xec       503.32  MB /   1.07  GB    512   B +  0 B   3.0.0.0 
/dev/nvme2n29         /dev/ng2n29           18S1BW2              dellemc-powerstore                       0x106      503.32  MB /   1.07  GB    512   B +  0 B   3.0.0.0 
/dev/nvme2n28         /dev/ng2n28           18S1BW2              dellemc-powerstore                       0x105      503.32  MB /   1.07  GB    512   B +  0 B   3.0.0.0 
/dev/nvme2n27         /dev/ng2n27           18S1BW2              dellemc-powerstore                       0x104      503.32  MB /   1.07  GB    512   B +  0 B   3.0.0.0 
/dev/nvme2n26         /dev/ng2n26           18S1BW2              dellemc-powerstore                       0x103      503.32  MB /   1.07  GB    512   B +  0 B   3.0.0.0 
/dev/nvme2n25         /dev/ng2n25           18S1BW2              dellemc-powerstore                       0x102      503.32  MB /   1.07  GB    512   B +  0 B   3.0.0.0 
/dev/nvme2n24         /dev/ng2n24           18S1BW2              dellemc-powerstore                       0x101      503.32  MB /   1.07  GB    512   B +  0 B   3.0.0.0 
/dev/nvme2n23         /dev/ng2n23           18S1BW2              dellemc-powerstore                       0x100      503.32  MB /   1.07  GB    512   B +  0 B   3.0.0.0 
/dev/nvme2n22         /dev/ng2n22           18S1BW2              dellemc-powerstore                       0xff       503.32  MB /   1.07  GB    512   B +  0 B   3.0.0.0 
/dev/nvme2n21         /dev/ng2n21           18S1BW2              dellemc-powerstore                       0xfe       503.32  MB /   1.07  GB    512   B +  0 B   3.0.0.0 
/dev/nvme2n20         /dev/ng2n20           18S1BW2              dellemc-powerstore                       0xfd       503.32  MB /   1.07  GB    512   B +  0 B   3.0.0.0 
/dev/nvme2n2          /dev/ng2n2            18S1BW2              dellemc-powerstore                       0xe         16.11  GB /   1.10  TB    512   B +  0 B   3.0.0.0 
/dev/nvme2n19         /dev/ng2n19           18S1BW2              dellemc-powerstore                       0xfc       503.32  MB /   1.07  GB    512   B +  0 B   3.0.0.0 
/dev/nvme2n18         /dev/ng2n18           18S1BW2              dellemc-powerstore                       0xfb       503.32  MB /   1.07  GB    512   B +  0 B   3.0.0.0 
/dev/nvme2n17         /dev/ng2n17           18S1BW2              dellemc-powerstore                       0xfa       503.32  MB /   1.07  GB    512   B +  0 B   3.0.0.0 
/dev/nvme2n16         /dev/ng2n16           18S1BW2              dellemc-powerstore                       0xf9       503.32  MB /   1.07  GB    512   B +  0 B   3.0.0.0 
/dev/nvme2n15         /dev/ng2n15           18S1BW2              dellemc-powerstore                       0xf8       503.32  MB /   1.07  GB    512   B +  0 B   3.0.0.0 
/dev/nvme2n14         /dev/ng2n14           18S1BW2              dellemc-powerstore                       0xf7       503.32  MB /   1.07  GB    512   B +  0 B   3.0.0.0 
/dev/nvme2n13         /dev/ng2n13           18S1BW2              dellemc-powerstore                       0xf6       503.32  MB /   1.07  GB    512   B +  0 B   3.0.0.0 
/dev/nvme2n12         /dev/ng2n12           18S1BW2              dellemc-powerstore                       0xf5       503.32  MB /   1.07  GB    512   B +  0 B   3.0.0.0 
/dev/nvme2n11         /dev/ng2n11           18S1BW2              dellemc-powerstore                       0xf4       503.32  MB /   1.07  GB    512   B +  0 B   3.0.0.0 
/dev/nvme2n10         /dev/ng2n10           18S1BW2              dellemc-powerstore                       0xf3       951.22  MB /   1.07  GB    512   B +  0 B   3.0.0.0 
/dev/nvme2n1          /dev/ng2n1            18S1BW2              dellemc-powerstore                       0x7          1.08  GB / 536.87  GB    512   B +  0 B   3.0.0

# nvme list-subsys
nvme-subsys2 - NQN=nqn.1988-11.com.dell:powerstore:00:88b402df2d762AA7AF94
\
 +- nvme2 tcp traddr=172.18.220.61,trsvcid=4420,host_iface=enp10s0f1np1,src_addr=172.18.220.36 live 
 +- nvme3 tcp traddr=172.18.220.60,trsvcid=4420,host_iface=enp10s0f1np1,src_addr=172.18.220.36 live 
 +- nvme4 tcp traddr=172.18.210.61,trsvcid=4420,host_iface=enp10s0f0np0,src_addr=172.18.210.35 live 
 +- nvme5 tcp traddr=172.18.210.60,trsvcid=4420,host_iface=enp10s0f0np0,src_addr=172.18.210.35 live 
nvme-subsys1 - NQN=nqn.1988-11.com.dell:SFSS:2:20230413041432mR
\
 +- nvme1 tcp traddr=172.18.220.70,trsvcid=8009,host_iface=enp10s0f1np1,src_addr=172.18.220.36 live
nvme-subsys0 - NQN=nqn.1988-11.com.dell:SFSS:1:20230413041432mR
\
 +- nvme0 tcp traddr=172.18.210.70,trsvcid=8009,host_iface=enp10s0f0np0,src_addr=172.18.210.35 live 

Version-Release number of selected component (if applicable):
# rpm -qa nvme-stas
nvme-stas-2.2.1-2.el9.noarch

# uname -r
5.14.0-324.el9.x86_64

How reproducible: Often