Bug 1565774

Summary:	After updating to RHEL 7.5 failing to clear the sssd cache
Product:	Red Hat Enterprise Linux 7	Reporter:	aheverle
Component:	sssd	Assignee:	Jakub Hrozek <jhrozek>
Status:	CLOSED ERRATA	QA Contact:	sssd-qe <sssd-qe>
Severity:	high	Docs Contact:
Priority:	high
Version:	7.5	CC:	abokovoy, aheverle, daniele, dconsoli, fidencio, gparente, grajaiya, jhrozek, jstephen, lslebodn, mkosek, mtenheuv, mupadhye, myka.rein, mzidek, pbrezina, sgoveas, tscherf, vvasilev
Target Milestone:	rc	Keywords:	ZStream
Target Release:	---
Hardware:	All
OS:	Linux
Whiteboard:
Fixed In Version:	sssd-1.16.0-21.el7	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:
Clones:	1579780 (view as bug list)		Environment:
Last Closed:	2018-10-30 10:42:26 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1579780

Description aheverle 2018-04-10 18:20:57 UTC

Description of problem:
After updating to RHEL 7.5 sssd throws an error when attempting to clear its cache via "sss_cache -E".  

How reproducible:
Everytime

Steps to Reproduce:
1. sss_cache -E

Actual results:
[root@server ~]# sss_cache -E
(Tue Apr 10 14:04:53:698618 2018) [sss_cache] [sysdb_domain_cache_connect] (0x0010): DB version too old [0.18], expected [0.20] for domain example.com!
Higher version of database is expected!
In order to upgrade the database, you must run SSSD.
Removing cache files in /var/lib/sss/db should fix the issue, but note that removing cache files will also remove all of your cached credentials.
Could not open available domains

Expected results:
For the cache to clear

Comment 2 Jakub Hrozek 2018-04-10 19:00:53 UTC

The error message has one way to fix this right there:
In order to upgrade the database, you must run SSSD.

So, just running SSSD should upgrade its internal database. Is the SSSD service up? Why didn't it restart during the upgrade?

Comment 3 aheverle 2018-04-10 19:50:57 UTC

(In reply to Jakub Hrozek from comment #2)
> The error message has one way to fix this right there:
> In order to upgrade the database, you must run SSSD.
> 
> So, just running SSSD should upgrade its internal database. Is the SSSD
> service up? Why didn't it restart during the upgrade?

After restarting sssd and clear the cache, the error went away.

It was replicated on another vm.  SSSD was already running.

Comment 4 Jakub Hrozek 2018-04-11 15:16:03 UTC

Our QE colleagues are telling me that we have an upgrade test which passed and I haven't heard (so far) about any other issues from other customers. Was there anything in the syslog or sssd logs that looked like the upgrade didn't run correctly?

Comment 9 Jakub Hrozek 2018-04-18 12:53:47 UTC

To test:
 - one way to test would be to do exactly what the customer did, e.g. upgrade from 7.4 to 7.4 and run sss_cache. This would only work between 7.4 and 7.5 or in general in between any two updates where we upgraded the cache (which we don't do all the time)
 - record the PID of the SSSD process before the update, compare with PID after the update. You can use the JSON output of systemctl -ojson status to get the PID

Comment 11 Jakub Hrozek 2018-04-18 12:57:20 UTC

Created attachment 1423565 [details]
A proposed patch

Comment 23 Madhuri 2018-08-08 07:00:17 UTC

verified with
sssd-1.16.2-11.el7

Verificatios steps:
1. Configure sssd, check authentication and lookup
=====================================
DEBUG - RUN getent passwd foo0
DEBUG - foo0:*:1211:1211:foo0 User:/home/foo0:/bin/bash
DEBUG - Exit code: 0

DEBUG - RUN getent passwd foo499
DEBUG - foo499:*:1710:1710:foo499 User:/home/foo499:/bin/bash
DEBUG - Exit code: 0

WRITE /tmp/qe_pytest_expect_file9O0XEAFM0D
DEBUG - line 2: /root/multihost_tests/env.sh: No such file or directory
DEBUG - spawn ssh -o NumberOfPasswordPrompts=1 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -l foo0 localhost whoami
DEBUG - Warning: Permanently added 'localhost' (ECDSA) to the list of known hosts.
DEBUG - foo499@localhost's password:
DEBUG - /usr/bin/id: cannot find name for group ID 1710
DEBUG - foo499
DEBUG - Exit code: 3

2. Update sssd
========================================
RUN yum -y update sssd

DEBUG -   Updating   : libsss_idmap-1.16.2-11.el7.x86_64                           1/56
DEBUG -   Updating   : libsss_nss_idmap-1.16.2-11.el7.x86_64                       2/56
DEBUG -   Updating   : libipa_hbac-1.16.2-11.el7.x86_64                            3/56
DEBUG -   Updating   : python-sssdconfig-1.16.2-11.el7.noarch                      4/56
DEBUG -   Updating   : sssd-client-1.16.2-11.el7.x86_64                            5/56
DEBUG -   Updating   : libsss_autofs-1.16.2-11.el7.x86_64                          6/56
DEBUG -   Updating   : libsss_sudo-1.16.2-11.el7.x86_64                            7/56
DEBUG -   Updating   : sssd-common-1.16.2-11.el7.x86_64                            8/56
DEBUG -   Updating   : sssd-krb5-common-1.16.2-11.el7.x86_64                       9/56
DEBUG -   Updating   : sssd-common-pac-1.16.2-11.el7.x86_64                       10/56
DEBUG -   Updating   : sssd-ad-1.16.2-11.el7.x86_64                               11/56
DEBUG -   Updating   : sssd-ipa-1.16.2-11.el7.x86_64                              12/56
DEBUG -   Updating   : sssd-ldap-1.16.2-11.el7.x86_64                             13/56
DEBUG -   Updating   : sssd-krb5-1.16.2-11.el7.x86_64                             14/56
DEBUG -   Updating   : python-sss-1.16.2-11.el7.x86_64                            15/56
DEBUG -   Updating   : sssd-proxy-1.16.2-11.el7.x86_64                            16/56
DEBUG -   Updating   : sssd-dbus-1.16.2-11.el7.x86_64                             17/56
DEBUG -   Updating   : libsss_simpleifp-1.16.2-11.el7.x86_64                      18/56
DEBUG -   Updating   : sssd-tools-1.16.2-11.el7.x86_64                            19/56
DEBUG -   Updating   : libsss_simpleifp-devel-1.16.2-11.el7.x86_64                20/56
DEBUG -   Updating   : sssd-1.16.2-11.el7.x86_64                                  21/56

DEBUG - Dependency Updated:
DEBUG -   libipa_hbac.x86_64 0:1.16.2-11.el7
DEBUG -   sssd-ad.x86_64 0:1.16.2-11.el7
DEBUG -   sssd-client.x86_64 0:1.16.2-11.el7
DEBUG -   sssd-common.x86_64 0:1.16.2-11.el7
DEBUG -   sssd-common-pac.x86_64 0:1.16.2-11.el7
DEBUG -   sssd-dbus.x86_64 0:1.16.2-11.el7
DEBUG -   sssd-ipa.x86_64 0:1.16.2-11.el7

3. Check sssd version
====================================
DEBUG - RUN rpm -q sssd
DEBUG - sssd-1.16.2-11.el7.x86_64
DEBUG - Exit code: 0

4. Check pid of sssd
=====================================
INFO - Upgrade successful. Moving ahead to test SSSD...
INFO - Test environment setup complete.
INFO - RUN pidof sssd
DEBUG - RUN pidof sssd
DEBUG - 12269
DEBUG - Exit code: 0

5. Clear the cache using #sss_cache -E
=========================================
INFO - RUN sss_cache -E
DEBUG - RUN sss_cache -E
DEBUG - Exit code: 0

6. Check user lookup
================================
DEBUG - RUN getent passwd foo999
DEBUG - foo999:*:2210:2210:foo999 User:/home/foo999:/bin/bash
DEBUG - Exit code: 0

Comment 25 errata-xmlrpc 2018-10-30 10:42:26 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:3158