Created attachment 1714271 [details]
Related messages extraced from the systemd journal

1. Please describe the problem:

With a 5.8 kernel, every 4-5 boots iscsid crashes and isci devices fail to come up, which hangs the boot in my case because of dependencies.

This is really an upstream kernel bug that I have reported a couple of times on lkml, here: 
    https://lkml.org/lkml/2020/7/28/1085
and here:
    https://lkml.org/lkml/2020/8/31/459

It got no traction from the commit (1b66d253610c7 ("bpf: Add get{peer, sock}name attach types for sock_addr") author or the iscsi maintainers, maybe someone in Fedora can help nudge the process forward.

The iscsi_sw_tcp_conn_get_param function has this:

                spin_lock_bh(&conn->session->frwd_lock);
                if (!tcp_sw_conn || !tcp_sw_conn->sock) {
                        spin_unlock_bh(&conn->session->frwd_lock);
                        return -ENOTCONN;
                }
                if (param == ISCSI_PARAM_LOCAL_PORT)
                        rc = kernel_getsockname(tcp_sw_conn->sock,
                                                (struct sockaddr *)&addr);
                else
                        rc = kernel_getpeername(tcp_sw_conn->sock,
                                                (struct sockaddr *)&addr);
                spin_unlock_bh(&conn->session->frwd_lock);

.. so it's calling kernel_getpeername while holding a spinlock.   Commit 1b66d253610c7 introduces BPF_CGROUP_RUN_SA_PROG_LOCK() within getpeername, which does lock_sock(sk); .... release_sock(sk);, and will try to acquire a mutex.  My guess is that when the mutex is not contended this works out fine, but when it is and there's a need to sleep, the "scheduling while atomic" occurs.

While that commit introduces the issue, calling into something like kernel_getpeername while holding a spinlock seems a bit optimistic, so maybe this needs fixing in the iscsi caller.

2. What is the Version-Release number of the kernel:

5.8.6-201.fc32.x86_64

3. Did it work previously in Fedora? If so, what kernel version did the issue
   *first* appear?  Old kernels are available for download at
   https://koji.fedoraproject.org/koji/packageinfo?packageID=8 :

This is a kernel upstream bug that appeared with the 5.8 kernel.

4. Can you reproduce this issue? If so, please provide the steps to reproduce
   the issue below:

Yes, it reproduces once out of every 4-5 boots, on several systems.  It has occurred with 5.8.6-201.fc32.x86_64 and also several times with custom kernels based on very recent mainline 5.9-rc kernels.  As far as I can tell the problem still exists in the mainline kernel.

5. Does this problem occur with the latest Rawhide kernel? To install the
   Rawhide kernel, run ``sudo dnf install fedora-repos-rawhide`` followed by
   ``sudo dnf update --enablerepo=rawhide kernel``:

Not tested, but I assume the problem is present there as well.

6. Are you running any modules that not shipped with directly Fedora's kernel?:

I sometimes do, but this was reproduced without it.

7. Please attach the kernel logs. You can get the complete kernel log
   for a boot with ``journalctl --no-hostname -k > dmesg.txt``. If the
   issue occurred on a previous boot, use the journalctl ``-b`` flag.

Related journal entries attached.