Bug 1879851

Summary: sssd fails to start in CentOS 8: Lower version of database is expected
Product: Red Hat Enterprise Linux 8 Reporter: Martin Pitt <mpitt>
Component: sssdAssignee: Pavel Březina <pbrezina>
Status: CLOSED CANTFIX QA Contact: sssd-qe <sssd-qe>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: CentOS StreamCC: bstinson, carl, grajaiya, jhrozek, jwboyer, lslebodn, mzidek, pbrezina, sbose, tscherf
Target Milestone: rc   
Target Release: 8.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-09-22 07:05:44 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Martin Pitt 2020-09-17 07:32:18 UTC
Description of problem: In a fresh install of current CentOS 8 stream [1], sssd fails to start:

# journalctl -u sssd

Sep 17 03:24:58 localhost.localdomain systemd[1]: Starting System Security Services Daemon...
Sep 17 03:24:59 localhost.localdomain sssd[780]: Starting up
Sep 17 03:24:59 localhost.localdomain sssd[780]: Lower version of database is expected!
Sep 17 03:24:59 localhost.localdomain sssd[780]: Removing cache files in /var/lib/sss/db should fix the issue, but note that removing cache files will also remove all of your cached credentials.
Sep 17 03:24:59 localhost.localdomain systemd[1]: sssd.service: Main process exited, code=exited, status=3/NOTIMPLEMENTED
Sep 17 03:24:59 localhost.localdomain systemd[1]: sssd.service: Failed with result 'exit-code'.
Sep 17 03:24:59 localhost.localdomain systemd[1]: Failed to start System Security Services Daemon.

Command line has something similar:

# sss_cache  -E
(Thu Sep 17 03:27:58:570627 2020) [sss_cache] [sysdb_domain_cache_connect] (0x0010): DB version too new [0.22], expected [0.21] for domain implicit_files!
Lower version of database is expected!
Removing cache files in /var/lib/sss/db should fix the issue, but note that removing cache files will also remove all of your cached credentials.
Could not open available domains

Despite this being a fresh and unconfigured install, I tried rm -f /var/lib/sss/db/*, and that does seem to help.


Version-Release number of selected component (if applicable): 
sssd-2.2.3-20.el8.x86_64


How reproducible: Always


Steps to Reproduce:
1. Install current CentOS 8 Stream [1]
2. Boot

Actual results: sssd.service fails to start


Expected results: sssd starts successfully


Additional info:

[1] https://mirrors.mit.edu/centos/8-stream/isos/x86_64/CentOS-Stream-8-x86_64-20200910-dvd1.iso

Comment 1 Pavel Březina 2020-09-17 12:59:59 UTC
By any chance did you downgrade SSSD after installation?

This seems like there was an SSSD cache present before SSSD was actually run and the cache was generated by a newer SSSD (0.22 is first in RHEL 8.3, I think). Is it possible to get list of files that are installed from the iso?

SSSD does not create cache files upon installation, we just own the directory in the spec file. The cache file is created only when SSSD starts and since removing the cache fixed the problem, you can see that it was created correctly.

Comment 2 Sumit Bose 2020-09-17 13:29:46 UTC
Hi,

as Pavel said, where did you get the version 'sssd-2.2.3-20.el8.x86_64' from. On the iso there is 'sssd-2.3.0-8.el8.x86_64' which has version '0.22'. I installed a VM from the iso and it is working fine.

bye,
Sumit

Comment 3 Martin Pitt 2020-09-17 16:23:59 UTC
The image build log is here: https://logs.cockpit-project.org/logs/image-refresh-1230-20200916-163636/log 

This just seems to come from BaseOS:


sssd                                     x86_64  2.2.3-20.el8                                     BaseOS      94 k

and then later, when it configures the libvirt package, it runs into the DB mismatch:

  Running scriptlet: libvirt-daemon-4.5.0-42.module_el8.2.0+320+13f86   236/451 
(Wed Sep 16 12:51:14:763193 2020) [sss_cache] [sysdb_domain_cache_connect] (0x0010): DB version too new [0.22], expected [0.21] for domain implicit_files!
Lower version of database is expected!
Removing cache files in /var/lib/sss/db should fix the issue, but note that removing cache files will also remove all of your cached credentials.
Could not open available domains
groupadd: sss_cache exited with status 71

[root@m1 ~]# yum repolist
repo id                                              repo name
AppStream                                            CentOS-8 - AppStream
BaseOS                                               CentOS-8 - Base
extras                                               CentOS-8 - Extras


[root@m1 ~]# yum info sssd
CentOS-8 - AppStream                                                                1.5 MB/s | 5.8 MB     00:03    
CentOS-8 - Base                                                                     976 kB/s | 2.2 MB     00:02    
CentOS-8 - Extras                                                                    10 kB/s | 7.9 kB     00:00    
Installed Packages
Name         : sssd
Version      : 2.2.3
Release      : 20.el8
Architecture : x86_64
Size         : 34 k
Source       : sssd-2.2.3-20.el8.src.rpm
Repository   : @System
From repo    : BaseOS
Summary      : System Security Services Daemon
URL          : https://pagure.io/SSSD/sssd/
License      : GPLv3+
Description  : Provides a set of daemons to manage access to remote directories and
             : authentication mechanisms. It provides an NSS and PAM interface toward
             : the system and a plug-gable back-end system to connect to multiple different
             : account sources. It is also the basis to provide client auditing and policy
             : services for projects like FreeIPA.
             : 
             : The sssd sub-package is a meta-package that contains the daemon as well as all
             : the existing back ends.

I see nothing weird going on in the repositories.. That's exactly the version on http://mirror.centos.org/centos-8/8/BaseOS/x86_64/os/Packages/ .

Comment 4 Sumit Bose 2020-09-18 05:27:37 UTC
(In reply to Martin Pitt from comment #3)
> The image build log is here:
> https://logs.cockpit-project.org/logs/image-refresh-1230-20200916-163636/log 
> 
> This just seems to come from BaseOS:
> 
> 
> sssd                                     x86_64  2.2.3-20.el8               
> BaseOS      94 k
> 
> and then later, when it configures the libvirt package, it runs into the DB
> mismatch:
> 
>   Running scriptlet: libvirt-daemon-4.5.0-42.module_el8.2.0+320+13f86  
> 236/451 
> (Wed Sep 16 12:51:14:763193 2020) [sss_cache] [sysdb_domain_cache_connect]
> (0x0010): DB version too new [0.22], expected [0.21] for domain
> implicit_files!
> Lower version of database is expected!
> Removing cache files in /var/lib/sss/db should fix the issue, but note that
> removing cache files will also remove all of your cached credentials.
> Could not open available domains
> groupadd: sss_cache exited with status 71
> 
> [root@m1 ~]# yum repolist
> repo id                                              repo name
> AppStream                                            CentOS-8 - AppStream
> BaseOS                                               CentOS-8 - Base
> extras                                               CentOS-8 - Extras
> 
> 
> [root@m1 ~]# yum info sssd
> CentOS-8 - AppStream                                                        
> 1.5 MB/s | 5.8 MB     00:03    
> CentOS-8 - Base                                                             
> 976 kB/s | 2.2 MB     00:02    
> CentOS-8 - Extras                                                           
> 10 kB/s | 7.9 kB     00:00    
> Installed Packages
> Name         : sssd
> Version      : 2.2.3
> Release      : 20.el8
> Architecture : x86_64
> Size         : 34 k
> Source       : sssd-2.2.3-20.el8.src.rpm
> Repository   : @System
> From repo    : BaseOS
> Summary      : System Security Services Daemon
> URL          : https://pagure.io/SSSD/sssd/
> License      : GPLv3+
> Description  : Provides a set of daemons to manage access to remote
> directories and
>              : authentication mechanisms. It provides an NSS and PAM
> interface toward
>              : the system and a plug-gable back-end system to connect to
> multiple different
>              : account sources. It is also the basis to provide client
> auditing and policy
>              : services for projects like FreeIPA.
>              : 
>              : The sssd sub-package is a meta-package that contains the
> daemon as well as all
>              : the existing back ends.
> 
> I see nothing weird going on in the repositories.. That's exactly the
> version on http://mirror.centos.org/centos-8/8/BaseOS/x86_64/os/Packages/ .

Hi,

yes, but if you install with the centos8-stream iso shouldn't you use the stream repo from http://mirror.centos.org/centos/8-stream/BaseOS/x86_64/os/Packages/ ?

bye,
Sumit

Comment 5 Martin Pitt 2020-09-22 07:05:44 UTC
@Sumit: Interesting point!

# yum repoinfo BaseOS
Repo-id            : BaseOS
Repo-name          : CentOS-8 - Base
Repo-status        : enabled
Repo-revision      : 8.2.2004
Repo-distro-tags      : [cpe:/o:centos:centos:8]:  , 8, C, O, S, e, n, t
Repo-updated       : Fr 18 Sep 2020 12:28:04 EDT
Repo-pkgs          : 1.675
Repo-available-pkgs: 1.673
Repo-size          : 1.0 G
Repo-mirrors       : http://mirrorlist.centos.org/?release=8&arch=x86_64&repo=BaseOS&infra=stock
Repo-baseurl       : http://centos.intergenia.de/8.2.2004/BaseOS/x86_64/os/ (9 more)

and http://centos.intergenia.de/8.2.2004/BaseOS/x86_64/os/Packages/ indeed has the newer sssd. So it seems installing the -stream dvd configures non-stream yum repos by default, and that stream has *older* versions than the "regular" CentOS 8. I still don't understand where in that story sssd's version went backwards, as currently there's clearly the newer version installed. Anyway, we've had so many problems with that DVD image, and we don't really want to use it either (it's just unbearably huge), I'll try again to build our test image from scratch with virt-install, or use the cloud images.

This is most probably not a bug in sssd then, but just some really broken CentOS 8 stream DVD. Sorry for the noise then!

Comment 6 Martin Pitt 2020-09-22 08:47:17 UTC
FTR, virt-install with centos-8-stream repos is still broken (centos-8 works). So I moved over to https://cloud.centos.org/centos/8-stream/x86_64/images/ , and even though that's an ancient image (from January), yum updating it works fine. That also has the correct repos:

# yum repolist
repo id                                      repo name
appstream                                    CentOS Stream 8 - AppStream
baseos                                       CentOS Stream 8 - BaseOS
extras                                       CentOS Stream 8 - Extras

# rpm -q sssd
sssd-2.3.0-9.el8.x86_64

So what remains is that C8-Stream has older package versions than C8, which is somewhat besides the point?

Comment 7 Sumit Bose 2020-09-22 13:12:21 UTC
(In reply to Martin Pitt from comment #6)
> FTR, virt-install with centos-8-stream repos is still broken (centos-8
> works). So I moved over to
> https://cloud.centos.org/centos/8-stream/x86_64/images/ , and even though
> that's an ancient image (from January), yum updating it works fine. That
> also has the correct repos:
> 
> # yum repolist
> repo id                                      repo name
> appstream                                    CentOS Stream 8 - AppStream
> baseos                                       CentOS Stream 8 - BaseOS
> extras                                       CentOS Stream 8 - Extras
> 
> # rpm -q sssd
> sssd-2.3.0-9.el8.x86_64
> 
> So what remains is that C8-Stream has older package versions than C8, which
> is somewhat besides the point?

Hi,

it is the other way round:

C8-Stream: sssd-2.3.0-9.el8.x86_64
C8:        sssd-2.2.3-20.el8.x86_64

So C8-Stream has the newer version. The C8-Stream DVD worked find for me and installed the proper repo. Do you, by chance, have an ansible configuration or similar which replaces the C8-Stream repo with something else?

I agree, this is most probably not related to SSSD, thanks for closing the ticket.

bye,
Sumit

Comment 8 Martin Pitt 2020-09-22 13:33:03 UTC
> it is the other way round:

Argh, of course -- I just looked at the -9 vs. 20, sorry.

There's no ansible or any funny configuration going on, BTW. It's more or less just virt-install.