Created attachment 1506550 [details]
sssd logs

This is literally the bind(2) syscall failing, this debug message:
(Fri Nov 16 14:20:57 2018) [sssd[nss]] [create_pipe_fd] (0x0010): Unable to bind on socket '/var/lib/sss/pipes/nss'

corresponds to:
 848 
 849     if (bind(fd, (struct sockaddr *)&addr, sizeof(addr)) == -1) {
 850         DEBUG(SSSDBG_FATAL_FAILURE,
 851               "Unable to bind on socket '%s'\n", sock_name);
 852         ret = EIO;
 853         goto done;
 854     }

At the very least, we should print errno to know why bind failed.

btw Scott, did the system maybe use socket activation for the nss responder (systemctl status sssd-nss.socket) ?

I don't think so.  I just booted the VM back up and it's not running:

[root@rhel8-2 ~]# systemctl status sssd-nss.socket
● sssd-nss.socket - SSSD NSS Service responder socket
   Loaded: loaded (/usr/lib/systemd/system/sssd-nss.socket; disabled; vendor preset: disabled)
   Active: inactive (dead)
     Docs: man:sssd.conf(5)
   Listen: /var/lib/sss/pipes/nss (Stream)

I don't know if it's related but, I was seeing the failure several times so I added a 5 second sleep after cache cleanup rm and before sssd start.

Also, I manually ran the following one time:

sss_cache -E
sssctl cache-remove


After those steps, I reran my tests and did not see this issue again.

(In reply to Scott Poore from comment #3)
> I don't think so.  I just booted the VM back up and it's not running:
> 

Is it a local VM or a VM in beaker? I'm asking if it was possible to poke around..

If not, is the reproducer as simple as 'restart sssd in a loop' ?

In the meantime, I opened https://github.com/SSSD/sssd/pull/697 which should at least help with debugging.

> [root@rhel8-2 ~]# systemctl status sssd-nss.socket
> ● sssd-nss.socket - SSSD NSS Service responder socket
>    Loaded: loaded (/usr/lib/systemd/system/sssd-nss.socket; disabled; vendor
> preset: disabled)
>    Active: inactive (dead)
>      Docs: man:sssd.conf(5)
>    Listen: /var/lib/sss/pipes/nss (Stream)
> 

OK, then the socket is disable and stopped, so it's not the socket activation causing issues.

> I don't know if it's related but, I was seeing the failure several times so
> I added a 5 second sleep after cache cleanup rm and before sssd start.
> 
> Also, I manually ran the following one time:
> 
> sss_cache -E
> sssctl cache-remove
> 
> 
> After those steps, I reran my tests and did not see this issue again.

Local VM.  I can set it up so you can login though if needed.   restarting sssd in a loop might reproduce the issue.  The test automation would have restarted it many times before I hit this issue.

Let me know if you want me to setup env for you to take a look

Hi Scott.

can you still reproduce the issue? I tried to restart SSSD in a loop for a long time but wasn't able to see a startup failure.

bye,
Sumit

I have not seen the issue again.  Maybe corrupt cache somewhere else?   Or fluke or fixed as a side effect of another bugfix.

This is the version I am trying now.

sssd-2.0.0-43.el8.x86_64


So, if you don't have any ideas about what could have done it, I guess we can close this.   I am running a loop but, I'm not expecting it to find anything.  It's already run through 30+ restarts and no issues.

(In reply to Scott Poore from comment #7)
> I have not seen the issue again.  Maybe corrupt cache somewhere else?   Or
> fluke or fixed as a side effect of another bugfix.
> 
> This is the version I am trying now.
> 
> sssd-2.0.0-43.el8.x86_64
> 
> 
> So, if you don't have any ideas about what could have done it, I guess we
> can close this.   I am running a loop but, I'm not expecting it to find
> anything.  It's already run through 30+ restarts and no issues.

Hi Scott,

thanks for checking again. Imo there is no real issue here. Maybe in the run where you have seen the issue the restart was "too" fast and the kernel side of the socket wasn't properly cleaned up or something similar.

I think we can use the ticket to implement Jakub's suggestion and add errno and the related error message to the log messages. But I'm not sure if this is really relevant for RHEL. So as an alternative we can just clone this ticket to upstream and close it here?

bye,
Sumit

Sounds like a plan to me.

Since this can't even be tested, I'm just going to close the bug as UPSTREAM.

The commit that added the better logging is 75696ddc84ab08c8c885dacc7796ebc8afc429ec