Bug 658444 - sssd generates corrupt database when started by puppetd from kickstart %post
sssd generates corrupt database when started by puppetd from kickstart %post
Product: Fedora
Classification: Fedora
Component: sssd (Show other bugs)
Unspecified Linux
low Severity medium
: ---
: ---
Assigned To: Stephen Gallagher
Fedora Extras Quality Assurance
Depends On:
  Show dependency treegraph
Reported: 2010-11-30 06:54 EST by Daniel Piddock
Modified: 2011-01-10 16:30 EST (History)
4 users (show)

See Also:
Fixed In Version: sssd-1.5.0-1.fc14
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Last Closed: 2011-01-10 16:30:19 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)
sssd.log debug_level=9 (4.02 KB, text/plain)
2010-11-30 08:44 EST, Daniel Piddock
no flags Details
sssd.conf slightly sanitized (1.70 KB, text/plain)
2010-11-30 08:46 EST, Daniel Piddock
no flags Details
sssd.log command line restart (44.66 KB, text/plain)
2010-11-30 11:11 EST, Daniel Piddock
no flags Details
kickstart (2.71 KB, text/plain)
2010-11-30 12:00 EST, Daniel Piddock
no flags Details

  None (edit)
Description Daniel Piddock 2010-11-30 06:54:24 EST
Description of problem:
We use puppet to configure workstations. The %post section of our kickstart calls puppetd --test, which pulls in sssd and starts the service. However the service then falls over.

Version-Release number of selected component (if applicable):
1.4.1-1.fc14 i686 and x86_64

How reproducible:
Every kickstart install. I have not been able to successfully reproduce the problem outside kickstart.

Steps to Reproduce:
1. Puppet config:
package { ['sssd', 'sssd-client']:
   ensure => latest,
file { '/etc/sssd/sssd.conf':
   mode => 600,
   user => root,
   group => root,
   source => 'puppet:///files/sssd.conf',
   require => Package['sssd'],
service { 'sssd':
   hasstatus => true,
   subscribe => File['/etc/sssd/sssd.conf'],
   enable => true,
   ensure => running,
   hasrestart => true,
2. Fedora 14 kickstart %post section that calls puppetd --test --waitforcert 10
3. sssd has failed to start properly
Actual results:
sssd crashed. Running "sssd -d3 -i" gives:
($date) [sssd] [check_file] (1): lstat for [/var/run/nscd/socket] failed: [2][No such file or directory].
($date) [sssd] [confdb_get_domain_internal] (1): No enumeration for [default]!
($date) [sssd] [server_setup] (3): CONFDB: /var/lib/sss/db/config.ldb
($date) [sssd] [sysdb_domain_init_internal] (0): Failed to initialize DB (68, [Entry @ATTRIBUTES already exists]) for domain default!

Expected results:
sssd to start all happy

Additional info:
Deleting the contents of /var/lib/sss/db/ and starting sssd manually works fine.
Comment 1 Stephen Gallagher 2010-11-30 07:04:00 EST
Please include your (sanitized) sssd.conf in this bug.

Also, for best ability to track this down, please have your puppetized sssd.conf include


in both the [sssd] and [domain/default] sections. Then run your kickstart. SSSD should output some verbose information into /var/log/sssd/sssd.log and /var/log/sssd/sssd_default.log

Please attach that output as well.
Comment 2 Daniel Piddock 2010-11-30 08:44:20 EST
Created attachment 463730 [details]
sssd.log debug_level=9
Comment 3 Daniel Piddock 2010-11-30 08:46:30 EST
Created attachment 463733 [details]
sssd.conf slightly sanitized
Comment 4 Daniel Piddock 2010-11-30 08:47:10 EST
No /var/log/sssd/sssd_default.log was created. Only sssd.log was in that directory.
Comment 5 Stephen Gallagher 2010-11-30 10:41:31 EST
I'm inclined to say that this must be something unique to puppet. I just created a kickstart that drops an sssd.conf in place and then starts the service.

A VM generated from this kickstart started up the SSSD with no problems.
Comment 6 Daniel Piddock 2010-11-30 11:11:49 EST
Created attachment 463766 [details]
sssd.log command line restart

I'm beginning to wonder if it's a race condition. Puppet starts the service then immediately restarts it - there's a statement saying the service should be running, so it needs to be started. The subscribed file has just been changed so queue a refresh request too.

I'm not sure what puppet is doing between the start and restart calls as attempting to replicate from the command line doesn't exactly work in the same way:
rm -f /var/lib/sss/db/* && /etc/init.d/sssd start; /etc/init.d/sssd restart

The service is started, then killed but fails to start up again even though three OKs were printed. Another start request works successfully. Comparing the logs the command line restart call happens slightly before the one generated by puppet.
Comment 7 Stephen Gallagher 2010-11-30 11:20:57 EST
Would you mind including your sanitized kickstart file? I'm mostly interested in whether you're performing any actions using the sss_* or ldb* tools, but I'd like to see everything you're doing in %post.

Also, if puppet really is TOO fast in doing start then restart, it could very well be the cause of the problem. We have a certain minimal amount of initialization we do on the first install, and if that's interrupted it could explain this behavior.
Comment 8 Stephen Gallagher 2010-11-30 11:55:06 EST
Okay, I think we found the problem. It was a race-condition as you suspected. There was a limited window during the first-time startup where it was possible for a restart request to interrupt the cache initialization and leave it in an unusable state.

I have submitted a patch to the upstream development list here:
Comment 9 Daniel Piddock 2010-11-30 12:00:18 EST
Created attachment 463778 [details]

Attaching a sanitized kickstart. %post isn't very exciting.
Comment 10 Fedora Update System 2010-12-23 13:45:30 EST
sssd-1.5.0-1.fc14 has been submitted as an update for Fedora 14.
Comment 11 Fedora Update System 2010-12-24 19:22:37 EST
sssd-1.5.0-1.fc14 has been pushed to the Fedora 14 testing repository.  If problems still persist, please make note of it in this bug report.
 If you want to test the update, you can install it with 
 su -c 'yum --enablerepo=updates-testing update sssd'.  You can provide feedback for this update here: https://admin.fedoraproject.org/updates/sssd-1.5.0-1.fc14
Comment 12 Fedora Update System 2011-01-10 16:29:59 EST
sssd-1.5.0-1.fc14 has been pushed to the Fedora 14 stable repository.  If problems still persist, please make note of it in this bug report.

Note You need to log in before you can comment on or make changes to this bug.