Bug 1007969

Summary:	sss tools do not have an option to remove the sssd database
Product:	Red Hat Enterprise Linux 7	Reporter:	Patrik Kis <pkis>
Component:	sssd	Assignee:	SSSD Maintainers <sssd-maint>
Status:	CLOSED ERRATA	QA Contact:	Steeve Goveas <sgoveas>
Severity:	low	Docs Contact:	Marc Muehlfeld <mmuehlfe>
Priority:	low
Version:	7.3	CC:	dlavu, dpal, ebenes, grajaiya, jgalipea, jhrozek, ksrot, lslebodn, mkosek, mzidek, pbrezina, pkis, sgoveas, stefw
Target Milestone:	rc
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	sssd-1.14.0-0.2.beta1.el7	Doc Type:	Enhancement
Doc Text:	New sssctl option remove-cache This update adds the "remove-cache" option to the "sssctl" utility. The option removes the local System Security Services Daemon's (SSSD) database contents, and restarts the sssd service. This enables the administrator to start from a clean state with SSSD and avoid the need to manually remove cache files.	Story Points:	---
Clone Of:		Environment:
Last Closed:	2016-11-04 07:09:45 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Patrik Kis 2013-09-13 16:03:51 UTC

Description of problem:
realmd should clean sssd database after leave. The database is located in /var/lib/sss/db/cache_$domain.ldb.

Version-Release number of selected component (if applicable):
realmd-0.14.6-1.el7

How reproducible:
always

Steps to Reproduce:
1. Join via sssd
2. leave
3. check the content of /var/lib/sss/db/

Additional info:
So far only one case was found where this matter and the case is not really typical for production environment so this has low priority.

The case is like this:
1/ join to a server e.g. ipa.baseos.qe
2/ login as a remote user e.g. amy.qe
3/ leave
4/ join again but to another server that by the way server the same doman ipa.baseos.qe, but it is a completely independent server
5/ login as amy.qe does not work unless the have the same ID on booth servers

Comment 1 Stef Walter 2013-09-13 16:13:53 UTC

We do run sss_cache. Could you provide verbose 'realm leave' output? 

Also, how are you examining the contents of the cache?

Comment 2 Patrik Kis 2013-09-17 07:16:51 UTC

I hope the test below make the case clearer. I'm cc-ing also jhrozek.


0 [root@rhel7 ~ ]# cat /etc/resolv.conf 
# Generated by NetworkManager
nameserver 10.34.37.24
0 [root@rhel7 ~ ]# ls -l /var/lib/sss/db/
total 0
0 [root@rhel7 ~ ]# realm -v join ipa.baseos.qe
 * Resolving: _ldap._tcp.ipa.baseos.qe
 * Performing LDAP DSE lookup on: 10.34.37.24
 * Successfully discovered: ipa.baseos.qe
Password for admin: 
 * Required files: /usr/sbin/ipa-client-install, /usr/sbin/oddjobd, /usr/libexec/oddjob/mkhomedir, /usr/sbin/sssd
 * LANG=C /usr/sbin/ipa-client-install --domain ipa.baseos.qe --realm IPA.BASEOS.QE --mkhomedir --enable-dns-updates --unattended --force-join --principal admin -W --force-ntpd
Discovery was successful!
Hostname: rhel7.pkis.net
Realm: IPA.BASEOS.QE
DNS Domain: ipa.baseos.qe
IPA Server: sec-ipa1.ipa.baseos.qe
BaseDN: dc=ipa,dc=baseos,dc=qe
Synchronizing time with KDC...
Successfully retrieved CA cert
    Subject:     CN=Certificate Authority,O=IPA.BASEOS.QE
    Issuer:      CN=Certificate Authority,O=IPA.BASEOS.QE
    Valid From:  Tue Jul 23 12:18:48 2013 UTC
    Valid Until: Sat Jul 23 12:18:48 2033 UTC

Enrolled in IPA realm IPA.BASEOS.QE
Created /etc/ipa/default.conf
Configured /etc/sssd/sssd.conf
Configured /etc/krb5.conf for IPA realm IPA.BASEOS.QE
Hostname (rhel7.pkis.net) not found in DNS
Failed to update DNS records.
Adding SSH public key from /etc/ssh/ssh_host_dsa_key.pub
Adding SSH public key from /etc/ssh/ssh_host_rsa_key.pub
host_mod: Unknown option: no_members
Failed to upload host SSH public keys.
SSSD enabled
Configured /etc/openldap/ldap.conf
NTP enabled
Configured /etc/ssh/ssh_config
Configured /etc/ssh/sshd_config

Client configuration complete.
 * /usr/bin/systemctl enable sssd.service
 * /usr/bin/systemctl restart sssd.service
 * /usr/bin/sh -c /usr/sbin/authconfig --update --enablesssd --enablesssdauth --enablemkhomedir --nostart && /usr/bin/systemctl enable oddjobd.service
 * Successfully enrolled machine in realm
0 [root@rhel7 ~ ]# 
0 [root@rhel7 ~ ]# ls -l /var/lib/sss/db/
total 3772
-rw-------. 1 root root 1286144 Sep 17 09:01 cache_ipa.baseos.qe.ldb
-rw-------. 1 root root    1121 Sep 17 09:01 ccache_IPA.BASEOS.QE
-rw-------. 1 root root 1286144 Sep 17 09:01 config.ldb
-rw-------. 1 root root 1286144 Sep 17 09:01 sssd.ldb
0 [root@rhel7 ~ ]# 
0 [root@rhel7 ~ ]# getent amy.qe
Unknown database: amy.qe
Try `getent --help' or `getent --usage' for more information.
1 [root@rhel7 ~ ]# getent passwd amy.qe
amy.qe:*:1365200005:1365200005:Amy Amy:/home/amy:/bin/sh
0 [root@rhel7 ~ ]# ssh amy.qe@localhost
The authenticity of host 'localhost (<no hostip for proxy command>)' can't be established.
RSA key fingerprint is c5:2e:97:10:26:7a:6d:f5:9e:a7:44:92:4f:0b:d7:a8.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'localhost' (RSA) to the list of known hosts.
amy.qe@localhost's password: 
Creating home directory for amy.qe.
-sh-4.2$ exit
logout
Connection to localhost closed.
0 [root@rhel7 ~ ]# 
0 [root@rhel7 ~ ]# ls -l /tmp/krb5cc_1365200005_zJcbJ6 
-rw-------. 1 amy.qe amy.qe 530 Sep 17 09:03 /tmp/krb5cc_1365200005_zJcbJ6
0 [root@rhel7 ~ ]# 
0 [root@rhel7 ~ ]# realm -v leave ipa.baseos.qe
 * LANG=C /usr/sbin/ipa-client-install --uninstall --unattended
Unenrolling client from IPA server
Removing Kerberos service principals from /etc/krb5.keytab
Disabling client Kerberos and LDAP configurations
Restoring client configuration files
nscd daemon is not installed, skip configuration
nslcd daemon is not installed, skip configuration
Client uninstall complete.
 * Removing entries from keytab for realm
 * /usr/sbin/sss_cache --users --groups --netgroups --services --autofs-maps
 * Removing domain configuration from sssd.conf
 * /usr/sbin/authconfig --update --disablesssdauth --nostart
 * /usr/bin/systemctl disable sssd.service
rm '/etc/systemd/system/multi-user.target.wants/sssd.service'
 * /usr/bin/systemctl stop sssd.service
 * Successfully unenrolled machine from realm
0 [root@rhel7 ~ ]# 
0 [root@rhel7 ~ ]# ls -l /var/lib/sss/db/
total 4088
-rw-------. 1 root root 1609728 Sep 17 09:03 cache_ipa.baseos.qe.ldb
-rw-------. 1 root root    1121 Sep 17 09:01 ccache_IPA.BASEOS.QE
-rw-------. 1 root root 1286144 Sep 17 09:01 config.ldb
-rw-------. 1 root root 1286144 Sep 17 09:01 sssd.ldb
0 [root@rhel7 ~ ]# 

// Now I change the IPA server

0 [root@rhel7 ~ ]# echo 'nameserver 10.34.24.252' >/etc/resolv.conf 
0 [root@rhel7 ~ ]# 
0 [root@rhel7 ~ ]# 
0 [root@rhel7 ~ ]# realm -v join ipa.baseos.qe
 * Resolving: _ldap._tcp.ipa.baseos.qe
 * Performing LDAP DSE lookup on: 10.34.24.252
 * Successfully discovered: ipa.baseos.qe
Password for admin: 
 * Required files: /usr/sbin/ipa-client-install, /usr/sbin/oddjobd, /usr/libexec/oddjob/mkhomedir, /usr/sbin/sssd
 * LANG=C /usr/sbin/ipa-client-install --domain ipa.baseos.qe --realm IPA.BASEOS.QE --mkhomedir --enable-dns-updates --unattended --force-join --principal admin -W --force-ntpd
Discovery was successful!
Hostname: rhel7.pkis.net
Realm: IPA.BASEOS.QE
DNS Domain: ipa.baseos.qe
IPA Server: server.ipa.baseos.qe
BaseDN: dc=ipa,dc=baseos,dc=qe
Synchronizing time with KDC...
Successfully retrieved CA cert
    Subject:     CN=Certificate Authority,O=IPA.BASEOS.QE
    Issuer:      CN=Certificate Authority,O=IPA.BASEOS.QE
    Valid From:  Tue Apr 30 14:33:21 2013 UTC
    Valid Until: Sat Apr 30 14:33:21 2033 UTC

Enrolled in IPA realm IPA.BASEOS.QE
Created /etc/ipa/default.conf
Configured /etc/sssd/sssd.conf
Configured /etc/krb5.conf for IPA realm IPA.BASEOS.QE
Hostname (rhel7.pkis.net) not found in DNS
Failed to update DNS records.
Adding SSH public key from /etc/ssh/ssh_host_dsa_key.pub
Adding SSH public key from /etc/ssh/ssh_host_rsa_key.pub
host_mod: Unknown option: no_members
Failed to upload host SSH public keys.
SSSD enabled
Configured /etc/openldap/ldap.conf
NTP enabled
Configured /etc/ssh/ssh_config
Configured /etc/ssh/sshd_config

Client configuration complete.
 * /usr/bin/systemctl enable sssd.service
 * /usr/bin/systemctl restart sssd.service
 * /usr/bin/sh -c /usr/sbin/authconfig --update --enablesssd --enablesssdauth --enablemkhomedir --nostart && /usr/bin/systemctl enable oddjobd.service
 * Successfully enrolled machine in realm
0 [root@rhel7 ~ ]# 
0 [root@rhel7 ~ ]# getent passwd amy.qe
amy.qe:*:903600006:903600006:Amy Amy:/home/amy:/bin/sh
0 [root@rhel7 ~ ]# ssh amy.qe@localhost
amy.qe@localhost's password: 
Permission denied, please try again.
amy.qe@localhost's password: 

130 [root@rhel7 ~ ]# 
130 [root@rhel7 ~ ]# ls -l /tmp/krb5cc*
-rw-------. 1 1365200005 1365200005 530 Sep 17 09:03 /tmp/krb5cc_1365200005_zJcbJ6
0 [root@rhel7 ~ ]# 

// Notice the UID/GID; they are the original ones and not 903600006:903600006

0 [root@rhel7 ~ ]# mv /tmp/krb5cc_1365200005_zJcbJ6 /tmp/krb5cc_1365200005_zJcbJ6_BACKUP
0 [root@rhel7 ~ ]# ls -l /tmp/krb5cc*
-rw-------. 1 1365200005 1365200005 530 Sep 17 09:03 /tmp/krb5cc_1365200005_zJcbJ6_BACKUP
0 [root@rhel7 ~ ]# 
0 [root@rhel7 ~ ]# ssh amy.qe@localhost
amy.qe@localhost's password: 
Last failed login: Tue Sep 17 09:07:19 CEST 2013 from localhost on ssh:notty
There was 1 failed login attempt since the last successful login.
-sh-4.2$ exit
logout
Connection to localhost closed.
0 [root@rhel7 ~ ]# ls -l /tmp/krb5cc*
-rw-------. 1        1365200005        1365200005 530 Sep 17 09:03 /tmp/krb5cc_1365200005_zJcbJ6_BACKUP
-rw-------. 1 amy.qe amy.qe 530 Sep 17 09:08 /tmp/krb5cc_903600006_9EiQ8f
0 [root@rhel7 ~ ]# 

// Let's connect again to the first IPA server

0 [root@rhel7 ~ ]# echo 'nameserver 10.34.37.24' >/etc/resolv.conf 
0 [root@rhel7 ~ ]# realm -v join ipa.baseos.qe
 * Resolving: _ldap._tcp.ipa.baseos.qe
 * Performing LDAP DSE lookup on: 10.34.37.24
 * Successfully discovered: ipa.baseos.qe
Password for admin: 
 * Required files: /usr/sbin/ipa-client-install, /usr/sbin/oddjobd, /usr/libexec/oddjob/mkhomedir, /usr/sbin/sssd

... SNIP ...

 * Successfully enrolled machine in realm
0 [root@rhel7 ~ ]# 
0 [root@rhel7 ~ ]# getent passwd amy.qe
amy.qe:*:1365200005:1365200005:Amy Amy:/home/amy:/bin/sh
0 [root@rhel7 ~ ]# ssh amy.qe@localhost
amy.qe@localhost's password: 
Permission denied, please try again.
amy.qe@localhost's password: 

130 [root@rhel7 ~ ]# 
0 [root@rhel7 ~ ]# sss_cache -E
0 [root@rhel7 ~ ]# ssh amy.qe@localhost
amy.qe@localhost's password: 
Permission denied, please try again.
amy.qe@localhost's password: 
Permission denied, please try again.
amy.qe@localhost's password: 

130 [root@rhel7 ~ ]# 
130 [root@rhel7 ~ ]# systemctl restart sssd
0 [root@rhel7 ~ ]# ssh amy.qe@localhost
amy.qe@localhost's password: 
Permission denied, please try again.
amy.qe@localhost's password: 

130 [root@rhel7 ~ ]# rm -f /var/lib/sss/db/*
0 [root@rhel7 ~ ]# systemctl restart sssd
0 [root@rhel7 ~ ]# ssh amy.qe@localhost
amy.qe@localhost's password: 
Last failed login: Tue Sep 17 09:14:32 CEST 2013 from localhost on ssh:notty
There were 5 failed login attempts since the last successful login.
Last login: Tue Sep 17 09:03:13 2013 from localhost
-sh-4.2$ exit
logout
Connection to localhost closed.
0 [root@rhel7 ~ ]# ls -l /tmp/krb5cc*
-rw-------. 1 amy.qe amy.qe 530 Sep 17 09:14 /tmp/krb5cc_1365200005_M1a3OI
-rw-------. 1 amy.qe amy.qe 530 Sep 17 09:03 /tmp/krb5cc_1365200005_zJcbJ6_BACKUP
-rw-------. 1         903600006         903600006 530 Sep 17 09:08 /tmp/krb5cc_903600006_9EiQ8f
0 [root@rhel7 ~ ]#

Comment 3 Jakub Hrozek 2013-09-17 10:17:55 UTC

This is arguably an edge-case but on leave, I think it would be OK to also rm the database. AFAIK the domain is removed from sssd.conf as well and SSSD is restarted. That way, even root who could have installed ldb-tools won't have any access to cached domain data in the orphaned ldb files.

Comment 4 Stef Walter 2013-09-20 12:55:50 UTC

I think we should bump this to the next release. Do you agree Patrik?

Comment 5 Patrik Kis 2013-09-25 08:15:26 UTC

(In reply to Stef Walter from comment #4)
> I think we should bump this to the next release. Do you agree Patrik?

The described use case is quite a corner case and so far noting really was found that might cause problems. So yes, I agree.

Comment 6 Stef Walter 2015-06-09 11:30:37 UTC

So why doesn't sss_cache remove the ccache database? There are complaints against realmd about this:

https://bugs.freedesktop.org/show_bug.cgi?id=90810

Comment 7 Jakub Hrozek 2015-06-09 13:47:28 UTC

(In reply to Stef Walter from comment #6)
> So why doesn't sss_cache remove the ccache database? There are complaints
> against realmd about this:
> 
> https://bugs.freedesktop.org/show_bug.cgi?id=90810

There's an upstream bug for that - https://fedorahosted.org/sssd/ticket/1691

Comment 8 Stef Walter 2015-06-09 14:41:50 UTC

Jakub, is that really the same thing? It seems like people want the entire database file removed.

Comment 9 Jakub Hrozek 2015-06-09 21:21:32 UTC

(In reply to Stef Walter from comment #8)
> Jakub, is that really the same thing? It seems like people want the entire
> database file removed.

You're right, it's not, thanks for catching that. I could swear we had an upstream ticket to rm the whole cache, but I can't find it now...so I filed a new one - https://fedorahosted.org/sssd/ticket/2671

Comment 11 Stef Walter 2015-07-14 17:23:27 UTC

This needs to be solved by sssd properly. realmd shouldn't be screwing around with sssd internal database cache manually. We run sss_cache and that should be enough. If no domains exist when running sss_cache, then sss_cache should just remove the database.

Comment 13 Jakub Hrozek 2015-07-14 18:26:35 UTC

Upstream ticket:
https://fedorahosted.org/sssd/ticket/2671

Comment 16 Jakub Hrozek 2016-06-27 16:37:29 UTC

Implemented upstream as part of:
    e157b9f6cb370e1b94bcac2044d26ad66d640fba
    9e9ad4cb181c6c0ec70caacfb31319753f889e98
    bf83a0faacf16196ab9bd37dcf6190b4209ccaf7
    586fa3571753ab4a607d40fc31503fc0e8effd70
    2f18b8d67c86a1a277b59894f24ea6e09b41b7ea
    d6f1b16baf8106d709e3fac585a12789dcb6bd29
    725c291ccfa46b08d2713133c227ac8d7203eb2f
    2f75ad013f8410397e4efbf0adadc2e69621f12a
    edaadf8de0c86a2cfff2d29215775d42919476f3
    47ce713ef8c7b32f2ce19cc3ace8e88f123fafac
    7bf750f6b3b47dcc8a192cc7bcbdecfb94e6cefb
    d2d8f342cd5e90bb9fd947c448492225f959aa86
    aea1d5c0ca9bb1470759b024c8b97b6c1f577193
    e98ccef2609811186711b79d8ef5d0a4450ab6e0
    81cde110402e088508053aea79670b38d450cb83
    b03ccb2764a4ccdadb77599cb624b6a17b633438
    3bc651a611a3e5be508875f3ae58bfb5ece2525c
    a6cd927f298ff5c9a603db5acb6c1b0ebea178c0
    b963ed8079a4a284611d50d1b79695116c40295d
    cf3ba77997dfbd076a1f30fdbb33c7973766ac03
    36e262020c80479baa09b2c4c8dd045c7a0f32a1
    12d99da163b1efef7e982f04e03049e012857bae
    2a45f13e3139063d3a5842119e7377c8c98aea1d
    7f0b01bf0a8f5c5b3ef145e81511b6db2cb4f98f
    b420aae3becdbf501deb2637e2a06636bd6ce1fe 

Note that this was not implemented as part of sss_cache, but sssctl. sssctl talks to SSSD over D-Bus. Long-term, we would prefer to fold all the existing command-line tools to be driven through the D-Bus interface rather than touching the database directly.

Comment 20 Marc Muehlfeld 2016-08-01 14:47:08 UTC

I replaced the Doc Text according to #c19.

Jakub, can you please review the Doc Text to make sure it's correct like this?

Comment 21 Jakub Hrozek 2016-08-02 06:41:19 UTC

(In reply to Marc Muehlfeld from comment #20)
> I replaced the Doc Text according to #c19.
> 
> Jakub, can you please review the Doc Text to make sure it's correct like
> this?

sss_cache doesn't have the ability to restart sssd, only sssctl does.

Comment 22 Jakub Hrozek 2016-08-02 07:07:22 UTC

(In reply to Jakub Hrozek from comment #21)
> (In reply to Marc Muehlfeld from comment #20)
> > I replaced the Doc Text according to #c19.
> > 
> > Jakub, can you please review the Doc Text to make sure it's correct like
> > this?
> 
> sss_cache doesn't have the ability to restart sssd, only sssctl does.

Once again I got confused by the title of this bugzilla. I really need to change it..

The doc text is good, thank you Marc. ACK.

Comment 23 Dan Lavu 2016-09-14 13:24:30 UTC

Verified, the sssctl tool now does this. 


[root@qe-blade-05 db]# sssctl cache-remove
SSSD must not be running. Stop SSSD now? (yes/no) [yes] yes
Creating backup of local data...
SSSD backup of local data already exist, override? (yes/no) [no] yes
Removing cache files...
SSSD needs to be running. Start SSSD now? (yes/no) [yes]

Comment 25 errata-xmlrpc 2016-11-04 07:09:45 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHEA-2016-2476.html