Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1456451

Summary: Unable to start sssd container using atomic run rhel7/sssd
Product: Red Hat Enterprise Linux 7 Reporter: Niranjan Mallapadi Raghavender <mniranja>
Component: atomicAssignee: Brent Baude <bbaude>
Status: CLOSED ERRATA QA Contact: atomic-bugs <atomic-bugs>
Severity: high Docs Contact:
Priority: unspecified    
Version: 7.4CC: ajia, amurdaca, bbaude, ddarrah, dwalsh, fkluknav, gscrivan, imcleod, jhrozek, jpazdziora, lslebodn, lsm5, miabbott, mniranja
Target Milestone: rcKeywords: Extras, Reopened, TestBlocker
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: atomic-1.18.1-4.git64843d3.el7_4 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-09-05 10:39:10 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Niranjan Mallapadi Raghavender 2017-05-29 12:24:29 UTC 
Description of problem:
Start sssd application container using atomic command fails. 




Version-Release number of selected component (if applicable):

● rhel-atomic-host:rhel-atomic-host/7/x86_64/standard
             Version: 7.4.0 (2017-05-26 12:38:52)
              Commit: 2c64fd22b089607d7dbafd37008775fc982d3be6a0be7e673bc7d518f2488735
oci-systemd-hook-0.1.7-2.1.git2788078.el7.x86_64
docker-1.12.6-30.1.git1398f24.el7.x86_64

How reproducible:


Steps to Reproduce:
1. Get the sssd-docker container from 
http://download-node-02.eng.bos.redhat.com/brewroot/packages/sssd-docker/7.4/2/images/
2. Tag the image as RHEL7/sssd

REPOSITORY             TAG                                               IMAGE ID            CREATED             SIZE
lslebodn/sssd-docker   extras-rhel-7.4-docker-candidate-20170515090813   de28fff8645c        13 days ago         373.2 MB
rhel7/sssd             latest                                            de28fff8645c        13 days ago         373.2 MB

3. create a realm-join-password file in /etc/sssd/ directory Containing Administrator password
echo 'Secret123' > /etc/sssd/realm-join-password

4. Create an sssd container 
atomic install --opt1='--dns=192.168.122.187 --dns-search=centaur.test --hostname=mytest.centaur.test -e SSSD_CONTAINER_TYPE=application --net=default' rhel7/sssd realm join -v CENTAUR.TEST

5. Run the container

atomic run rhel7/sssd


Actual results:
Does't run sssd contaier and fails, journalctl shows 

May 29 11:29:24 titan.centaur.test oci-systemd-hook[16810]: systemdhook <debug>: Skipping as container command is /bin/run.sh, not init or systemd
May 29 11:29:24 titan.centaur.test dockerd-current[1245]: container_linux.go:247: starting container process caused "process_linux.go:359: container init caused \"rootfs_linux.go:54: mounting \\\"/etc/krb5.keytab\\\" to rootfs \\\"/var/lib/docker/devicemapper/mnt/17aff1cf09a3b1d1ab61f35b4b98a00556d781364f6e210b95d5ece12414097a/rootfs\\\" at \\\"/var/lib/docker/devicemapper/mnt/17aff1cf09a3b1d1ab61f35b4b98a00556d781364f6e210b95d5ece12414097a/rootfs/etc/krb5.keytab\\\" caused \\\"not a directory\\\"\""
May 29 11:29:24 titan.centaur.test dockerd-current[1245]: time="2017-05-29T11:29:24.622544289Z" level=error msg="containerd: start container" error="oci runtime error: container_linux.go:247: starting container process caused \"process_linux.go:359: container init caused \\\"rootfs_linux.go:54: mounting \\\\\\\"/etc/krb5.keytab\\\\\\\" to rootfs \\\\\\\"/var/lib/docker/devicemapper/mnt/17aff1cf09a3b1d1ab61f35b4b98a00556d781364f6e210b95d5ece12414097a/rootfs\\\\\\\" at \\\\\\\"/var/lib/docker/devicemapper/mnt/17aff1cf09a3b1d1ab61f35b4b98a00556d781364f6e210b95d5ece12414097a/rootfs/etc/krb5.keytab\\\\\\\" caused \\\\\\\"not a directory\\\\\\\"\\\"\"\n" id=a865470181f3ca0ba25da3de9ab3a31e4c0fbddba83f033529accc93d5e50d68
May 29 11:29:24 titan.centaur.test dockerd-current[1245]: time="2017-05-29T11:29:24.626004208Z" level=error msg="Create container failed with error: invalid header field value \"oci runtime error: container_linux.go:247: starting container process caused \\\"process_linux.go:359: container init caused \\\\\\\"rootfs_linux.go:54: mounting \\\\\\\\\\\\\\\"/etc/krb5.keytab\\\\\\\\\\\\\\\" to rootfs \\\\\\\\\\\\\\\"/var/lib/docker/devicemapper/mnt/17aff1cf09a3b1d1ab61f35b4b98a00556d781364f6e210b95d5ece12414097a/rootfs\\\\\\\\\\\\\\\" at \\\\\\\\\\\\\\\"/var/lib/docker/devicemapper/mnt/17aff1cf09a3b1d1ab61f35b4b98a00556d781364f6e210b95d5ece12414097a/rootfs/etc/krb5.keytab\\\\\\\\\\\\\\\" caused \\\\\\\\\\\\\\\"not a directory\\\\\\\\\\\\\\\"\\\\\\\"\\\"\\n\""
May 29 11:29:24 titan.centaur.test kernel: XFS (dm-4): Unmounting Filesystem
May 29 11:29:24 titan.centaur.test dockerd-current[1245]: time="2017-05-29T11:29:24.682725383Z" level=error msg="Handler for POST /v1.24/containers/sssd/start returned error: invalid header field value \"oci runtime error: container_linux.go:247: starting container process caused \\\"process_linux.go:359: container init caused \\\\\\\"rootfs_linux.go:54: mounting \\\\\\\\\\\\\\\"/etc/krb5.keytab\\\\\\\\\\\\\\\" to rootfs \\\\\\\\\\\\\\\"/var/lib/docker/devicemapper/mnt/17aff1cf09a3b1d1ab61f35b4b98a00556d781364f6e210b95d5ece12414097a/rootfs\\\\\\\\\\\\\\\" at \\\\\\\\\\\\\\\"/var/lib/docker/devicemapper/mnt/17aff1cf09a3b1d1ab61f35b4b98a00556d781364f6e210b95d5ece12414097a/rootfs/etc/krb5.keytab\\\\\\\\\\\\\\\" caused \\\\\\\\\\\\\\\"not a directory\\\\\\\\\\\\\\\"\\\\\\\"\\\"\\n\""
May 29 11:29:24 titan.centaur.test dockerd-current[1245]: time="2017-05-29T11:29:24.682757498Z" level=error msg="Handler for POST /v1.24/containers/sssd/start returned error: invalid header field value \"oci runtime error: container_linux.go:247: starting container process caused \\\"process_linux.go:359: container init caused \\\\\\\"rootfs_linux.go:54: mounting \\\\\\\\\\\\\\\"/etc/krb5.keytab\\\\\\\\\\\\\\\" to rootfs \\\\\\\\\\\\\\\"/var/lib/docker/devicemapper/mnt/17aff1cf09a3b1d1ab61f35b4b98a00556d781364f6e210b95d5ece12414097a/rootfs\\\\\\\\\\\\\\\" at \\\\\\\\\\\\\\\"/var/lib/docker/devicemapper/mnt/17aff1cf09a3b1d1ab61f35b4b98a00556d781364f6e210b95d5ece12414097a/rootfs/etc/krb5.keytab\\\\\\\\\\\\\\\" caused \\\\\\\\\\\\\\\"not a directory\\\\\\\\\\\\\\\"\\\\\\\"\\\"\\n\""


Expected results:

Should be able to start sssd contai.er

Additional info:
Comment 3 Niranjan Mallapadi Raghavender 2017-05-29 12:27:05 UTC 
This is a test blocker for testing sssd-docker for RHEL74 Atomic host.
Comment 4 Micah Abbott 2017-06-01 18:00:10 UTC 
I don't have all the test environment setup, but was able to run through most of the command successfully using the latest 7.4 compose.  Niranjan, can you try to reproduce this with the latest compose?


# atomic host status
State: idle
Deployments:
● 7.4_latest:rhel-atomic-host/7/x86_64/standard
             Version: 7.4.0 (2017-05-31 18:08:19)
              Commit: 589bc18b3fa01888230ea004c37465089b6098e67d2382ef435e583acc9b202b
        GPGSignature: 1 signature
                      Signature made Wed 31 May 2017 06:12:05 PM UTC using RSA key ID 199E2F91FD431D51
                      Good signature from "Red Hat, Inc. <security>"

# rpm -q atomic docker
atomic-1.17.2-5.1.git2760e30.el7.x86_64
docker-1.12.6-30.1.git1398f24.el7.x86_64

# docker images
REPOSITORY                                                        TAG                 IMAGE ID            CREATED             SIZE
brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/rhel7/sssd   7.4-2               1be78f0107f5        2 weeks ago         373.2 MB
registry.access.redhat.com/rhel7/sssd                             latest              1be78f0107f5        2 weeks ago         373.2 MB

# echo 'Secret123' > /etc/sssd/realm-join-password

# atomic install --opt1='--dns=192.168.122.187 --dns-search=centaur.test --hostname=mytest.centaur.test -e SSSD_CONTAINER_TYPE=application --net=default' rhel7/sssd realm join -v CE$
TAUR.TEST
docker run --rm=true --privileged --net=host -v /:/host -e NAME=sssd -e IMAGE=rhel7/sssd -e HOST=/host --dns=192.168.122.187 --dns-search=centaur.test --hostname=mytest.centaur.test -e SSSD_CONTAINER_TYPE=appli$
ation --net=default rhel7/sssd /bin/install.sh realm join -v CENTAUR.TEST
 * Resolving: _ldap._tcp.centaur.test
 ! Discovery timed out after 15 seconds
realm: No such realm found

# atomic run rhel7/sssd
docker run -d --restart=always --name sssd -e NAME=sssd -e IMAGE=rhel7/sssd rhel7/sssd /bin/run.sh
e36a38d86db90c76263e1bfd977e035f78b9a8eeeb5bca28bbd4e7b6425f2425

# atomic containers list
   CONTAINER ID IMAGE                COMMAND              CREATED          STATE      BACKEND    RUNTIME   
   e36a38d86db9 rhel7/sssd           /bin/run.sh          2017-06-01 17:54 running    docker     docker 

]# journalctl -b -u docker --since '5 minutes ago' --no-pager
-- Logs begin at Thu 2017-06-01 17:14:35 UTC, end at Thu 2017-06-01 17:57:36 UTC. --
Jun 01 17:54:04 micah-rhelah-vm0601a.localdomain dockerd-current[14705]: time="2017-06-01T17:54:04.185781301Z" level=info msg="{Action=_ping, Username=cloud-user, LoginUID=1000, PID=14946}"
Jun 01 17:54:04 micah-rhelah-vm0601a.localdomain dockerd-current[14705]: time="2017-06-01T17:54:04.191170800Z" level=info msg="{Action=_ping, Username=cloud-user, LoginUID=1000, PID=14946}"
Jun 01 17:54:04 micah-rhelah-vm0601a.localdomain dockerd-current[14705]: time="2017-06-01T17:54:04.193159340Z" level=error msg="Handler for GET /v1.24/containers/sssd/json returned error: No such container: sssd"
Jun 01 17:54:04 micah-rhelah-vm0601a.localdomain dockerd-current[14705]: time="2017-06-01T17:54:04.193180457Z" level=error msg="Handler for GET /v1.24/containers/sssd/json returned error: No such container: sssd"
Jun 01 17:54:04 micah-rhelah-vm0601a.localdomain dockerd-current[14705]: time="2017-06-01T17:54:04.198139365Z" level=info msg="{Action=_ping, Username=cloud-user, LoginUID=1000, PID=14946}"
Jun 01 17:54:05 micah-rhelah-vm0601a.localdomain dockerd-current[14705]: time="2017-06-01T17:54:05.523749134Z" level=info msg="{Action=create, Username=cloud-user, LoginUID=1000, PID=14963}"
Jun 01 17:54:05 micah-rhelah-vm0601a.localdomain dockerd-current[14705]: time="2017-06-01T17:54:05.861684258Z" level=info msg="{Action=attach, Username=cloud-user, LoginUID=1000, PID=14963}"
Jun 01 17:54:05 micah-rhelah-vm0601a.localdomain dockerd-current[14705]: time="2017-06-01T17:54:05.862669875Z" level=info msg="{Action=start, Username=cloud-user, LoginUID=1000, PID=14963}"
Jun 01 17:54:06 micah-rhelah-vm0601a.localdomain oci-register-machine[15075]: 2017/06/01 17:54:06 Register machine: prestart 24a7462c46dd0f2ac60e347d1b8afe5ff8b1ea10a3b6db671e6ccf66dd1d8f40 15063 /var/lib/docker/devicemapper/mnt/4734c330553782e429d52006595c6ec84ba41316b637aaf8f9e2e9e3b2096830/rootfs
Jun 01 17:54:08 micah-rhelah-vm0601a.localdomain dockerd-current[14705]:  * Resolving: _ldap._tcp.centaur.test
Jun 01 17:54:24 micah-rhelah-vm0601a.localdomain dockerd-current[14705]:  ! Discovery timed out after 15 seconds
Jun 01 17:54:24 micah-rhelah-vm0601a.localdomain dockerd-current[14705]: realm: No such realm found
Jun 01 17:54:24 micah-rhelah-vm0601a.localdomain oci-register-machine[15133]: 2017/06/01 17:54:24 Register machine: poststop 24a7462c46dd0f2ac60e347d1b8afe5ff8b1ea10a3b6db671e6ccf66dd1d8f40 0 /var/lib/docker/devicemapper/mnt/4734c330553782e429d52006595c6ec84ba41316b637aaf8f9e2e9e3b2096830/rootfs
Jun 01 17:54:24 micah-rhelah-vm0601a.localdomain dockerd-current[14705]: time="2017-06-01T17:54:24.705459310Z" level=info msg="{Action=wait, Username=cloud-user, LoginUID=1000, PID=14963}"
Jun 01 17:54:25 micah-rhelah-vm0601a.localdomain dockerd-current[14705]: time="2017-06-01T17:54:25.103413556Z" level=info msg="{Action=remove, Username=cloud-user, LoginUID=1000, PID=14963}"
Jun 01 17:54:31 micah-rhelah-vm0601a.localdomain dockerd-current[14705]: time="2017-06-01T17:54:31.406574612Z" level=info msg="{Action=_ping, Username=cloud-user, LoginUID=1000, PID=15173}"
Jun 01 17:54:31 micah-rhelah-vm0601a.localdomain dockerd-current[14705]: time="2017-06-01T17:54:31.412018587Z" level=info msg="{Action=_ping, Username=cloud-user, LoginUID=1000, PID=15173}"
Jun 01 17:54:31 micah-rhelah-vm0601a.localdomain dockerd-current[14705]: time="2017-06-01T17:54:31.414027740Z" level=error msg="Handler for GET /v1.24/containers/sssd/json returned error: No such container: sssd"
Jun 01 17:54:31 micah-rhelah-vm0601a.localdomain dockerd-current[14705]: time="2017-06-01T17:54:31.414049735Z" level=error msg="Handler for GET /v1.24/containers/sssd/json returned error: No such container: sssd"
Jun 01 17:54:31 micah-rhelah-vm0601a.localdomain dockerd-current[14705]: time="2017-06-01T17:54:31.419243614Z" level=info msg="{Action=_ping, Username=cloud-user, LoginUID=1000, PID=15173}"
Jun 01 17:54:34 micah-rhelah-vm0601a.localdomain dockerd-current[14705]: time="2017-06-01T17:54:34.747836417Z" level=info msg="{Action=create, Username=cloud-user, LoginUID=1000, PID=15191}"
Jun 01 17:54:34 micah-rhelah-vm0601a.localdomain dockerd-current[14705]: time="2017-06-01T17:54:34.978150342Z" level=info msg="{Action=start, Username=cloud-user, LoginUID=1000, PID=15191}"
Jun 01 17:54:35 micah-rhelah-vm0601a.localdomain oci-register-machine[15302]: 2017/06/01 17:54:35 Register machine: prestart e36a38d86db90c76263e1bfd977e035f78b9a8eeeb5bca28bbd4e7b6425f2425 15290 /var/lib/docker/devicemapper/mnt/7cdfa8fa1686d72c0466aaf3574f65300523a79c357ec50eaf8d0e8ea1a9965c/rootfs
Jun 01 17:54:35 micah-rhelah-vm0601a.localdomain dockerd-current[14705]: [start sssd.service]
Jun 01 17:54:35 micah-rhelah-vm0601a.localdomain dockerd-current[14705]: Running [/usr/sbin/sssd -i -f]
Jun 01 17:54:35 micah-rhelah-vm0601a.localdomain dockerd-current[14705]: Marked pid [9] for [sssd.service]
Jun 01 17:54:35 micah-rhelah-vm0601a.localdomain dockerd-current[14705]: Child process groupID is 9 args: /usr/sbin/sssd -i -f
Jun 01 17:56:33 micah-rhelah-vm0601a.localdomain dockerd-current[14705]: time="2017-06-01T17:56:33.475579933Z" level=info msg="{Action=_ping, Username=cloud-user, LoginUID=1000, PID=15373}"
Jun 01 17:56:33 micah-rhelah-vm0601a.localdomain dockerd-current[14705]: time="2017-06-01T17:56:33.481305262Z" level=info msg="{Action=_ping, Username=cloud-user, LoginUID=1000, PID=15373}"
Jun 01 17:56:33 micah-rhelah-vm0601a.localdomain dockerd-current[14705]: time="2017-06-01T17:56:33.497854962Z" level=info msg="{Action=_ping, Username=cloud-user, LoginUID=1000, PID=15373}"
Jun 01 17:56:33 micah-rhelah-vm0601a.localdomain dockerd-current[14705]: time="2017-06-01T17:56:33.503364630Z" level=info msg="{Action=_ping, Username=cloud-user, LoginUID=1000, PID=15373}"
Jun 01 17:57:36 micah-rhelah-vm0601a.localdomain dockerd-current[14705]: time="2017-06-01T17:57:36.295686172Z" level=info msg="{Action=_ping, Username=cloud-user, LoginUID=1000, PID=15384}"
Jun 01 17:57:36 micah-rhelah-vm0601a.localdomain dockerd-current[14705]: time="2017-06-01T17:57:36.301384740Z" level=info msg="{Action=_ping, Username=cloud-user, LoginUID=1000, PID=15384}"
Comment 5 Niranjan Mallapadi Raghavender 2017-06-01 18:37:59 UTC 
/etc/ostree/remotes.d/rhel-atomic-host.conf" 4L, 276C written
[root@titan ~]# rpm-ostree upgrade

2 metadata, 16 content objects fetched; 115147 KiB transferred in 252 seconds                                                                                                                                                                 
Copying /etc changes: 27 modified, 4 removed, 91 added
Transaction complete; bootconfig swap: yes deployment count change: 1
Upgraded:
  atomic 1:1.17.2-1.1.git2760e30.el7 -> 1:1.17.2-5.1.git2760e30.el7
  container-selinux 2:2.10-2.el7 -> 2:2.12-2.gite7096ce.el7
  kubernetes-client 1.5.2-0.5.gita552679.el7 -> 1.5.2-0.6.gitd33fd89.el7
  kubernetes-node 1.5.2-0.5.gita552679.el7 -> 1.5.2-0.6.gitd33fd89.el7
  ostree 2017.5-2.el7 -> 2017.6-2.el7
  ostree-fuse 2017.5-2.el7 -> 2017.6-2.el7
  ostree-grub2 2017.5-2.el7 -> 2017.6-2.el7
Downgraded:
  cockpit-bridge 138-7.el7 -> 138-1.el7
  cockpit-system 138-7.el7 -> 138-1.el7
Run "systemctl reboot" to start a reboot


[root@titan ~]# atomic host status
State: idle
Deployments:
  rhel-atomic-host:rhel-atomic-host/7/x86_64/standard
             Version: 7.4.0 (2017-05-31 18:08:19)
              Commit: 589bc18b3fa01888230ea004c37465089b6098e67d2382ef435e583acc9b202b

● rhel-atomic-host:rhel-atomic-host/7/x86_64/standard
             Version: 7.4.0 (2017-05-26 12:38:52)
              Commit: 2c64fd22b089607d7dbafd37008775fc982d3be6a0be7e673bc7d518f2488735

Reboot the system

[root@titan ~]# atomic host status
State: idle
Deployments:
● rhel-atomic-host:rhel-atomic-host/7/x86_64/standard
             Version: 7.4.0 (2017-05-31 18:08:19)
              Commit: 589bc18b3fa01888230ea004c37465089b6098e67d2382ef435e583acc9b202b

  rhel-atomic-host:rhel-atomic-host/7/x86_64/standard
             Version: 7.4.0 (2017-05-26 12:38:52)
              Commit: 2c64fd22b089607d7dbafd37008775fc982d3be6a0be7e673bc7d518f2488735
[root@titan ~]# 


[root@titan ~]# atomic install --opt1='--dns=192.168.122.187 --dns-search=centaur.test --hostname=mytest.centaur.test -e SSSD_CONTAINER_TYPE=application --net=default' rhel7/sssd /bin/install.sh realm join -v CENTAUR.TEST
docker run --rm=true --privileged --net=host -v /:/host -e NAME=sssd -e IMAGE=rhel7/sssd -e HOST=/host --dns=192.168.122.187 --dns-search=centaur.test --hostname=mytest.centaur.test -e SSSD_CONTAINER_TYPE=application --net=default rhel7/sssd /bin/install.sh /bin/install.sh realm join -v CENTAUR.TEST
 * Resolving: _ldap._tcp.centaur.test
 * Performing LDAP DSE lookup on: 192.168.122.187
 * Successfully discovered: CENTAUR.TEST
Password for Administrator:  * Required files: /usr/sbin/oddjobd, /usr/libexec/oddjob/mkhomedir, /usr/sbin/sssd, /usr/bin/net
 * LANG=C LOGNAME=root /usr/bin/net -s /var/cache/realmd/realmd-smb-conf.B4DC1Y -U Administrator ads join CENTAUR.TEST
Enter Administrator's password:
Using short domain name -- CENTAUR
Joined 'MYTEST' to dns domain 'CENTAUR.TEST'
 * LANG=C LOGNAME=root /usr/bin/net -s /var/cache/realmd/realmd-smb-conf.B4DC1Y -U Administrator ads keytab create
Enter Administrator's password:
 * /usr/bin/systemctl enable sssd.service
 * /usr/bin/systemctl restart sssd.service
 * /usr/bin/sh -c /usr/sbin/authconfig --update --enablesssd --enablesssdauth --enablemkhomedir --nostart && /usr/bin/systemctl enable oddjobd.service && /usr/bin/systemctl start oddjobd.service

 * Successfully enrolled machine in realm
Service sssd.service configured to run SSSD container.
Service sssd.service configured to run SSSD container.


[root@titan ~]# atomic run rhel7/sssd


[root@titan ~]# journalctl -xf
-- Logs begin at Mon 2017-05-29 10:08:04 UTC. --
Jun 01 18:37:23 titan.centaur.test systemd[1]: Stopping docker container a865470181f3ca0ba25da3de9ab3a31e4c0fbddba83f033529accc93d5e50d68.
-- Subject: Unit docker-a865470181f3ca0ba25da3de9ab3a31e4c0fbddba83f033529accc93d5e50d68.scope has begun shutting down
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit docker-a865470181f3ca0ba25da3de9ab3a31e4c0fbddba83f033529accc93d5e50d68.scope has begun shutting down.
Jun 01 18:37:23 titan.centaur.test oci-register-machine[15009]: 2017/06/01 18:37:23 Register machine: poststop a865470181f3ca0ba25da3de9ab3a31e4c0fbddba83f033529accc93d5e50d68 0 /var/lib/docker/devicemapper/mnt/17aff1cf09a3b1d1ab61f35b4b98a00556d781364f6e210b95d5ece12414097a/rootfs
Jun 01 18:37:23 titan.centaur.test oci-systemd-hook[15013]: systemdhook <debug>: parse 30928 bytes of config data from file
Jun 01 18:37:23 titan.centaur.test oci-systemd-hook[15013]: systemdhook <debug>: Skipping as container command is /bin/run.sh, not init or systemd
Jun 01 18:37:23 titan.centaur.test dockerd-current[985]: container_linux.go:247: starting container process caused "process_linux.go:359: container init caused \"rootfs_linux.go:54: mounting \\\"/etc/krb5.keytab\\\" to rootfs \\\"/var/lib/docker/devicemapper/mnt/17aff1cf09a3b1d1ab61f35b4b98a00556d781364f6e210b95d5ece12414097a/rootfs\\\" at \\\"/var/lib/docker/devicemapper/mnt/17aff1cf09a3b1d1ab61f35b4b98a00556d781364f6e210b95d5ece12414097a/rootfs/etc/krb5.keytab\\\" caused \\\"not a directory\\\"\""
Jun 01 18:37:23 titan.centaur.test dockerd-current[985]: time="2017-06-01T18:37:23.044430405Z" level=error msg="containerd: start container" error="oci runtime error: container_linux.go:247: starting container process caused \"process_linux.go:359: container init caused \\\"rootfs_linux.go:54: mounting \\\\\\\"/etc/krb5.keytab\\\\\\\" to rootfs \\\\\\\"/var/lib/docker/devicemapper/mnt/17aff1cf09a3b1d1ab61f35b4b98a00556d781364f6e210b95d5ece12414097a/rootfs\\\\\\\" at \\\\\\\"/var/lib/docker/devicemapper/mnt/17aff1cf09a3b1d1ab61f35b4b98a00556d781364f6e210b95d5ece12414097a/rootfs/etc/krb5.keytab\\\\\\\" caused \\\\\\\"not a directory\\\\\\\"\\\"\"\n" id=a865470181f3ca0ba25da3de9ab3a31e4c0fbddba83f033529accc93d5e50d68
Jun 01 18:37:23 titan.centaur.test dockerd-current[985]: time="2017-06-01T18:37:23.045915445Z" level=error msg="Create container failed with error: invalid header field value \"oci runtime error: container_linux.go:247: starting container process caused \\\"process_linux.go:359: container init caused \\\\\\\"rootfs_linux.go:54: mounting \\\\\\\\\\\\\\\"/etc/krb5.keytab\\\\\\\\\\\\\\\" to rootfs \\\\\\\\\\\\\\\"/var/lib/docker/devicemapper/mnt/17aff1cf09a3b1d1ab61f35b4b98a00556d781364f6e210b95d5ece12414097a/rootfs\\\\\\\\\\\\\\\" at \\\\\\\\\\\\\\\"/var/lib/docker/devicemapper/mnt/17aff1cf09a3b1d1ab61f35b4b98a00556d781364f6e210b95d5ece12414097a/rootfs/etc/krb5.keytab\\\\\\\\\\\\\\\" caused \\\\\\\\\\\\\\\"not a directory\\\\\\\\\\\\\\\"\\\\\\\"\\\"\\n\""
Jun 01 18:37:23 titan.centaur.test kernel: XFS (dm-4): Unmounting Filesystem
Jun 01 18:37:23 titan.centaur.test dockerd-current[985]: time="2017-06-01T18:37:23.095489760Z" level=error msg="Handler for POST /v1.24/containers/sssd/start returned error: invalid header field value \"oci runtime error: container_linux.go:247: starting container process caused \\\"process_linux.go:359: container init caused \\\\\\\"rootfs_linux.go:54: mounting \\\\\\\\\\\\\\\"/etc/krb5.keytab\\\\\\\\\\\\\\\" to rootfs \\\\\\\\\\\\\\\"/var/lib/docker/devicemapper/mnt/17aff1cf09a3b1d1ab61f35b4b98a00556d781364f6e210b95d5ece12414097a/rootfs\\\\\\\\\\\\\\\" at \\\\\\\\\\\\\\\"/var/lib/docker/devicemapper/mnt/17aff1cf09a3b1d1ab61f35b4b98a00556d781364f6e210b95d5ece12414097a/rootfs/etc/krb5.keytab\\\\\\\\\\\\\\\" caused \\\\\\\\\\\\\\\"not a directory\\\\\\\\\\\\\\\"\\\\\\\"\\\"\\n\""
Jun 01 18:37:23 titan.centaur.test dockerd-current[985]: time="2017-06-01T18:37:23.095522982Z" level=error msg="Handler for POST /v1.24/containers/sssd/start returned error: invalid header field value \"oci runtime error: container_linux.go:247: starting container process caused \\\"process_linux.go:359: container init caused \\\\\\\"rootfs_linux.go:54: mounting \\\\\\\\\\\\\\\"/etc/krb5.keytab\\\\\\\\\\\\\\\" to rootfs \\\\\\\\\\\\\\\"/var/lib/docker/devicemapper/mnt/17aff1cf09a3b1d1ab61f35b4b98a00556d781364f6e210b95d5ece12414097a/rootfs\\\\\\\\\\\\\\\" at \\\\\\\\\\\\\\\"/var/lib/docker/devicemapper/mnt/17aff1cf09a3b1d1ab61f35b4b98a00556d781364f6e210b95d5ece12414097a/rootfs/etc/krb5.keytab\\\\\\\\\\\\\\\" caused \\\\\\\\\\\\\\\"not a directory\\\\\\\\\\\\\\\"\\\\\\\"\\\"\\n\""
Comment 6 Niranjan Mallapadi Raghavender 2017-06-13 21:33:08 UTC 
Any update on this, This is a test blocker for us.
Comment 7 Daniel Walsh 2017-06-13 22:55:27 UTC 
Micah, do you see this as still a bug?

Is this a bug in the sssd container?
Comment 8 Micah Abbott 2017-06-14 13:34:58 UTC 
Dan, there hasn't been a successful 7.4 ostree compose since 5/31, so this problem still exists.

I have not been able to reproduce this in my own environment, but Niranjan appears to have an environment that reliably reproduces the problem.
Comment 9 Daniel Walsh 2017-06-14 13:38:02 UTC 
Antonio any ideas?
Comment 10 Antonio Murdaca 2017-06-14 13:52:51 UTC 
(In reply to Micah Abbott from comment #8)
> Dan, there hasn't been a successful 7.4 ostree compose since 5/31, so this
> problem still exists.
> 
> I have not been able to reproduce this in my own environment, but Niranjan
> appears to have an environment that reliably reproduces the problem.

Micah is there any way to see which docker version has been used at the last successful compose? that could narrow down the bisect I'm going to try (assuming it's a docker issue)
Comment 11 Micah Abbott 2017-06-14 13:57:40 UTC 
Last 7.4 compose has the following docker:  1.12.6-30.1.git1398f24.el7.x86_64
Comment 12 Antonio Murdaca 2017-06-14 14:01:16 UTC 
(In reply to Micah Abbott from comment #11)
> Last 7.4 compose has the following docker:  1.12.6-30.1.git1398f24.el7.x86_64

so it can't be a docker issue if the latest good compose used the same docker version as the one reported here in the first comment right?
Comment 14 Micah Abbott 2017-06-14 14:08:51 UTC 
(In reply to Antonio Murdaca from comment #12)

> so it can't be a docker issue if the latest good compose used the same
> docker version as the one reported here in the first comment right?

Well, the version is the same from the original report (compose from 5/26) and when it was reproduced, so I think it still could be in 'docker'
Comment 15 Antonio Murdaca 2017-06-14 14:15:37 UTC 
I guess I have another question then, what was the last compose w/o this issue? and which docker version did it have?
Comment 16 Micah Abbott 2017-06-14 14:17:47 UTC 
The last compose that had this issue was on 5/31 (which is also the most recent compose available).  The docker version in that compose is docker-1.12.6-30.1.git1398f24.el7.x86_64
Comment 17 Micah Abbott 2017-06-14 14:19:08 UTC 
(In reply to Micah Abbott from comment #16)
> The last compose that had this issue was on 5/31 (which is also the most
> recent compose available).  The docker version in that compose is
> docker-1.12.6-30.1.git1398f24.el7.x86_64

Doh!  I mis-read that question.


I don't think there has been a 7.4 compose that did not show this problem.

Niranjan, have you been able to test your sssd container successfully on any 7.4 compose?
Comment 18 Niranjan Mallapadi Raghavender 2017-06-15 05:22:18 UTC 
Nope, This was my first attempt.
Comment 19 Antonio Murdaca 2017-06-16 08:57:06 UTC 
another, probably silly, question, is the docker version in 7.3 working for this use case? can you try it out and let me know?
Comment 20 Niranjan Mallapadi Raghavender 2017-06-19 09:54:40 UTC 
I tried a more recent build of sssd-docker (7.4-3 sha256:17344f6cd2efb8919a66f5c238e921148536b6ccf8c65d2a7c6e9e77aa6fbd85).  On Atomic host version 7.4 (589bc18b3fa01888230ea004c37465089b6098e67d2382ef435e583acc9b202b). 

[root@dione sssd]# atomic host status
State: idle
Deployments:
● rhel-atomic-host:rhel-atomic-host/7/x86_64/standard
             Version: 7.4.0 (2017-05-31 18:08:19)
              Commit: 589bc18b3fa01888230ea004c37465089b6098e67d2382ef435e583acc9b202b

  rhel-atomic-host:rhel-atomic-host/7/x86_64/standard
             Version: 7.4.0 (2017-05-26 12:38:52)
              Commit: 2c64fd22b089607d7dbafd37008775fc982d3be6a0be7e673bc7d518f2488735


I did not see the same behaviour and i was able to start sssd container successfully. 

[root@dione sssd]# atomic install --opt1='--dns=192.168.122.187 --dns-search=centaur.test --hostname=abc.centaur.test -e SSSD_CONTAINER_TYPE=application --net=default' rhel7/sssd realm join -v CENTAUR.TEST
docker run --rm=true --privileged --net=host -v /:/host -e NAME=sssd -e IMAGE=rhel7/sssd -e HOST=/host --dns=192.168.122.187 --dns-search=centaur.test --hostname=abc.centaur.test -e SSSD_CONTAINER_TYPE=application --net=default rhel7/sssd /bin/install.sh realm join -v CENTAUR.TEST
sssd container is already configured on this system.
Run atomic uninstall $IMAGE first.

[root@dione sssd]# atomic uninstall rhel7/sssd
docker run --rm=true --privileged --net=host -v /:/host -e NAME=sssd -e IMAGE=rhel7/sssd -e HOST=/host rhel7/sssd /bin/uninstall.sh
[root@dione sssd]# atomic install --opt1='--dns=192.168.122.187 --dns-search=centaur.test --hostname=abc.centaur.test -e SSSD_CONTAINER_TYPE=application --net=default' rhel7/sssd realm join -v CENTAUR.TEST
docker run --rm=true --privileged --net=host -v /:/host -e NAME=sssd -e IMAGE=rhel7/sssd -e HOST=/host --dns=192.168.122.187 --dns-search=centaur.test --hostname=abc.centaur.test -e SSSD_CONTAINER_TYPE=application --net=default rhel7/sssd /bin/install.sh realm join -v CENTAUR.TEST
 * Resolving: _ldap._tcp.centaur.test
 * Performing LDAP DSE lookup on: 192.168.122.187
 * Successfully discovered: CENTAUR.TEST
Password for Administrator:  * Required files: /usr/sbin/oddjobd, /usr/libexec/oddjob/mkhomedir, /usr/sbin/sssd, /usr/bin/net
 * LANG=C LOGNAME=root /usr/bin/net -s /var/cache/realmd/realmd-smb-conf.9AVZ1Y -U Administrator ads join CENTAUR.TEST
Enter Administrator's password:DNS update failed: NT_STATUS_INVALID_PARAMETER

Using short domain name -- CENTAUR
Joined 'ABC' to dns domain 'CENTAUR.TEST'
No DNS domain configured for abc. Unable to perform DNS Update.
 * LANG=C LOGNAME=root /usr/bin/net -s /var/cache/realmd/realmd-smb-conf.9AVZ1Y -U Administrator ads keytab create
Enter Administrator's password:
 * /usr/bin/systemctl enable sssd.service
 * /usr/bin/systemctl restart sssd.service
 * /usr/bin/sh -c /usr/sbin/authconfig --update --enablesssd --enablesssdauth --enablemkhomedir --nostart && /usr/bin/systemctl enable oddjobd.service && /usr/bin/systemctl start oddjobd.service

 * Successfully enrolled machine in realm
Service sssd.service configured to run SSSD container.

[root@dione sssd]# atomic run rhel7/sssd
docker run -d --restart=always --name sssd -e NAME=sssd -e IMAGE=rhel7/sssd --hostname abc.centaur.test -v /var/lib/sssd_container/sssd/container/etc/dbus-1/system.d/:/etc/dbus-1/system.d/:Z -v /var/lib/sssd_container/sssd/container/etc/ipa/:/etc/ipa/:Z -v /var/lib/sssd_container/sssd/container/etc/openldap/:/etc/openldap/:Z -v /var/lib/sssd_container/sssd/container/etc/ssh/:/etc/ssh/:Z -v /var/lib/sssd_container/sssd/container/etc/sssd/:/etc/sssd/:Z -v /var/lib/sssd_container/sssd/container/var/cache/realmd/:/var/cache/realmd/:Z -v /var/lib/sssd_container/sssd/container/var/lib/authconfig/last/:/var/lib/authconfig/last/:Z -v /var/lib/sssd_container/sssd/container/var/lib/ipa-client/sysrestore/:/var/lib/ipa-client/sysrestore/:Z -v /var/lib/sssd_container/sssd/container/var/lib/samba/:/var/lib/samba/:Z -v /var/lib/sssd_container/sssd/container/var/lib/sss/db/:/var/lib/sss/db/:Z -v /var/lib/sssd_container/sssd/container/var/lib/sss/gpo_cache/:/var/lib/sss/gpo_cache/:Z -v /var/lib/sssd_container/sssd/container/var/lib/sss/secrets/:/var/lib/sss/secrets/:Z -v /var/lib/sssd_container/sssd/container/var/lib/sss/keytabs/:/var/lib/sss/keytabs/:Z -v /var/lib/sssd_container/sssd/container/var/log/sssd/:/var/log/sssd/:Z -v /var/lib/sssd_container/sssd/container/etc/krb5.keytab:/etc/krb5.keytab:Z -v /var/lib/sssd_container/sssd/container/etc/nsswitch.conf:/etc/nsswitch.conf:Z -v /var/lib/sssd_container/sssd/container/etc/sysconfig/authconfig:/etc/sysconfig/authconfig:Z -v /var/lib/sssd_container/sssd/container/etc/sysconfig/network:/etc/sysconfig/network:Z -v /var/lib/sssd_container/sssd/container/etc/yp.conf:/etc/yp.conf:Z -v /var/lib/sssd_container/sssd/container/etc/resolv.conf:/etc/resolv.conf:Z -v /var/lib/sssd_container/sssd/client/etc/krb5.conf.d/:/etc/krb5.conf.d/:z -v /var/lib/sssd_container/sssd/client/var/lib/sss/mc/:/var/lib/sss/mc/:z -v /var/lib/sssd_container/sssd/client/var/lib/sss/pipes/:/var/lib/sss/pipes/:z -v /var/lib/sssd_container/sssd/client/var/lib/sss/pubconf/:/var/lib/sss/pubconf/:z -v /var/lib/sssd_container/sssd/client/var/run/dbus/:/var/run/dbus/:z -v /var/lib/sssd_container/sssd/client/etc/krb5.conf:/etc/krb5.conf:z -v /var/lib/sssd_container/sssd/client/etc/pam.d/fingerprint-auth-ac:/etc/pam.d/fingerprint-auth:z -v /var/lib/sssd_container/sssd/client/etc/pam.d/password-auth-ac:/etc/pam.d/password-auth:z -v /var/lib/sssd_container/sssd/client/etc/pam.d/smartcard-auth-ac:/etc/pam.d/smartcard-auth:z -v /var/lib/sssd_container/sssd/client/etc/pam.d/system-auth-ac:/etc/pam.d/system-auth:z -e WITH_KCM=yes -e SSSD_CONTAINER_TYPE=application rhel7/sssd /bin/run.sh
1eb6bde4981b3e08791c54eded801d790004617602871710a2a060cf5c67ca7f
Comment 21 Niranjan Mallapadi Raghavender 2017-06-19 09:56:12 UTC 
I am unable to reproduce the issue again on another RHEL74 systems, I am not sure what combination or what situations caused the issue described in comment #1. If 
i see this issue again, i will open a new bug, This bug can be closed now.
Comment 22 Niranjan Mallapadi Raghavender 2017-06-19 12:25:02 UTC 
I tried 7.3 sssd-docker image on RHEL74 Atomic host 

-rw-r--r--. 1 root root 127708055 Jun 19 12:09 docker-image-sha256:639c91ea1e855d59badb0e3b6e17610176c21a8835f2a04bf6ae579d8c50ed08.x86_64.tar.gz
[root@dione old]# docker load -i docker-image-sha256:639c91ea1e855d59badb0e3b6e17610176c21a8835f2a04bf6ae579d8c50ed08.x86_64.tar.gz
f5bd5357a1de: Loading layer [==================================================>] 202.5 MB/202.5 MB
279bfd6c7049: Loading layer [==================================================>] 10.24 kB/10.24 kB
e1e57c80f769: Loading layer [==================================================>] 174.1 MB/174.1 MB
Loaded image: lslebodn/sssd-docker:extras-rhel-7.3-docker-candidate-62686-20170612142847
[root@dione old]# docker images
REPOSITORY             TAG                                                     IMAGE ID            CREATED             SIZE
lslebodn/sssd-docker   extras-rhel-7.3-docker-candidate-62686-20170612142847   d6dac5bdb6cf        6 days ago          357.6 MB
[root@dione old]# docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
[root@dione old]# docker images
REPOSITORY             TAG                                                     IMAGE ID            CREATED             SIZE
lslebodn/sssd-docker   extras-rhel-7.3-docker-candidate-62686-20170612142847   d6dac5bdb6cf        6 days ago          357.6 MB
[root@dione old]# docker tag d6dac5bdb6cf rhel7/sssd
[root@dione old]# docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
[root@dione old]# docker ^C
[root@dione old]# atomic install rhel7/sssd realm -v join CENTAUR.TEST
docker run --rm=true --privileged --net=host -v /:/host -e NAME=sssd -e IMAGE=rhel7/sssd -e HOST=/host rhel7/sssd /bin/install.sh realm -v join CENTAUR.TEST
Initializing configuration context from host ...
 * Resolving: _ldap._tcp.centaur.test
 * Performing LDAP DSE lookup on: 192.168.122.27
 * Successfully discovered: CENTAUR.TEST
Password for Administrator:  * Required files: /usr/sbin/oddjobd, /usr/libexec/oddjob/mkhomedir, /usr/sbin/sssd, /usr/bin/net
 * LANG=C LOGNAME=root /usr/bin/net -s /var/cache/realmd/realmd-smb-conf.N7P21Y -U Administrator ads join CENTAUR.TEST
Enter Administrator's password:DNS update failed: NT_STATUS_INVALID_PARAMETER

Using short domain name -- CENTAUR
Joined 'DIONE' to dns domain 'CENTAUR.TEST'
No DNS domain configured for dione. Unable to perform DNS Update.
 * LANG=C LOGNAME=root /usr/bin/net -s /var/cache/realmd/realmd-smb-conf.N7P21Y -U Administrator ads keytab create
Enter Administrator's password:
 * /usr/bin/systemctl enable sssd.service
 * /usr/bin/systemctl restart sssd.service
 * /usr/bin/sh -c /usr/sbin/authconfig --update --enablesssd --enablesssdauth --enablemkhomedir --nostart && /usr/bin/systemctl enable oddjobd.service && /usr/bin/systemctl start oddjobd.service

 * Successfully enrolled machine in realm
Copying new configuration to host ...
Service sssd.service configured to run SSSD container.
[root@dione old]# systemctl start sssd
[root@dione old]# docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
94ef7ca4744b        rhel7/sssd          "/bin/run.sh"       4 seconds ago       Up 2 seconds                            sssd

I could not see the same docker crash .
Comment 23 Micah Abbott 2017-06-19 13:38:27 UTC 
Thanks for the extra reproducer attempts, Niranjan.

Based on your comments, I'm going to close this as 'WORKSFORME'.
Comment 24 Niranjan Mallapadi Raghavender 2017-07-04 22:32:09 UTC 
I was able to reproduce the issue. 

[root@dione ~]# atomic host status
State: idle
Deployments:
● rhel-atomic-host:rhel-atomic-host/7/x86_64/standard
                Version: 7.4.0 (2017-06-30 18:37:40)
                 Commit: 8018f95c2f2f38a79e68f174dd5888b53769c0e4adcd89c87a802219091c9d0e

  rhel-atomic-host:rhel-atomic-host/7/x86_64/standard
                Version: 7.4.0 (2017-06-20 02:16:02)
                 Commit: c55bf46b4baaee58d637774e8515bc7e88b96e4acf099d8bca39c27757201442


Versions:
=========

[root@dione ~]# atomic install rhel7/sssd realm discover CENTAUR.TEST
docker run --rm=true --privileged --net=host -v /:/host -e NAME=sssd -e IMAGE=rhel7/sssd -e HOST=/host rhel7/sssd /bin/install.sh realm discover CENTAUR.TEST
Initializing configuration context from host ...
CENTAUR.TEST
  type: kerberos
  realm-name: CENTAUR.TEST
  domain-name: CENTAUR.TEST
  configured: no
  server-software: active-directory
  client-software: sssd
  required-package: oddjob
  required-package: oddjob-mkhomedir
  required-package: sssd
  required-package: adcli
  required-package: samba-common-tools
centaur.test
  type: kerberos
  realm-name: CENTAUR.TEST
  domain-name: centaur.test
  configured: no
Copying new configuration to host ...
[root@dione ~]# atomic install rhel7/sssd realm join -v --membership-software=samba CENTAUR.TEST
docker run --rm=true --privileged --net=host -v /:/host -e NAME=sssd -e IMAGE=rhel7/sssd -e HOST=/host rhel7/sssd /bin/install.sh realm join -v --membership-software=samba CENTAUR.TEST
Initializing configuration context from host ...
 * Resolving: _ldap._tcp.centaur.test
 * Performing LDAP DSE lookup on: 192.168.122.187
Password for Administrator:  * Performing LDAP DSE lookup on: 192.168.122.27
 * Successfully discovered: CENTAUR.TEST
 * Required files: /usr/sbin/oddjobd, /usr/libexec/oddjob/mkhomedir, /usr/sbin/sssd, /usr/bin/net
 * LANG=C LOGNAME=root /usr/bin/net -s /var/cache/realmd/realmd-smb-conf.RHJY2Y -U Administrator ads join CENTAUR.TEST
Enter Administrator's password:DNS update failed: NT_STATUS_UNSUCCESSFUL

Using short domain name -- CENTAUR
Joined 'DIONE' to dns domain 'CENTAUR.TEST'
DNS Update for dione.centaur.test failed: ERROR_DNS_UPDATE_FAILED
 * LANG=C LOGNAME=root /usr/bin/net -s /var/cache/realmd/realmd-smb-conf.RHJY2Y -U Administrator ads keytab create
Enter Administrator's password:
 * /usr/bin/systemctl enable sssd.service
 * /usr/bin/systemctl restart sssd.service
 * /usr/bin/sh -c /usr/sbin/authconfig --update --enablesssd --enablesssdauth --enablemkhomedir --nostart && /usr/bin/systemctl enable oddjobd.service && /usr/bin/systemctl start oddjobd.service
 * Successfully enrolled machine in realm

Copying new configuration to host ...
Service sssd.service configured to run SSSD container.

[root@dione ~]# systemctl start sssd

system_u:system_r:spc_t:s0      root      23196  23179  0 22:18 ?        00:00:00 tail -f /var/log/sssd/systemctl.log
system_u:system_r:spc_t:s0      root      23201  23179  0 22:18 ?        00:00:00 /usr/sbin/sssd -i -f
system_u:system_r:spc_t:s0      root      23202  23201  0 22:18 ?        00:00:00 /usr/libexec/sssd/sssd_be --domain CENTAUR.TEST --uid 0 --gid 0 --debug-to-files
system_u:system_r:spc_t:s0      root      23203  23201  0 22:18 ?        00:00:00 /usr/libexec/sssd/sssd_nss --uid 0 --gid 0 --debug-to-files
system_u:system_r:spc_t:s0      root      23204  23201  0 22:18 ?        00:00:00 /usr/libexec/sssd/sssd_pam --uid 0 --gid 0 --debug-to-files


[root@dione ~]# id Administrator
uid=1993600500(administrator) gid=1993600513(domain users) groups=1993600513(domain users),1993600520(group policy creator owners),1993600519(enterprise admins),1993600512(domain admins),1993600518(schema admins),1993601669(myunixgroup),1993601671(testgroup1),1993600572(denied rodc password replication group)

[root@dione ~]# systemctl stop sssd


[root@dione ~]# atomic uninstall rhel7/sssd realm leave -v CENTAUR.TEST
docker run --rm=true --privileged --net=host -v /:/host -e NAME=sssd -e IMAGE=rhel7/sssd -e HOST=/host rhel7/sssd /bin/uninstall.sh realm leave -v CENTAUR.TEST
Initializing configuration context from host ...
Warning: Failed to copy /etc/yp.conf to host. It cannot be a directory
 * Removing entries from keytab for realm
 * /usr/sbin/sss_cache --users --groups --netgroups --services --autofs-maps
 * Removing domain configuration from sssd.conf
 * /usr/sbin/authconfig --update --disablesssdauth --nostart
 * /usr/bin/systemctl disable sssd.service
 * Successfully unenrolled machine from realm
Copying new configuration to host ...
Removing /etc/krb5.keytab
Removing /etc/sssd/systemctl-lite-enabled/sssd.service
Removing /etc/yp.conf
Removing /var/lib/sss/pipes/private/sbus-dp_CENTAUR.TEST.69
Removing /var/lib/sss/pipes/private/pam
Removing /var/lib/sss/mc/passwd
Removing /var/lib/sss/mc/group
Removing /var/lib/sss/mc/initgroups


[root@dione ~]# atomic install --opt1='--dns=192.168.122.187 --dns-search=centaur.test --hostname=idm2.centaur.test -e SSSD_CONTAINER_TYPE=application --net=default' rhel7/sssd realm join -v CENTAUR.TEST
docker run --rm=true --privileged --net=host -v /:/host -e NAME=sssd -e IMAGE=rhel7/sssd -e HOST=/host --dns=192.168.122.187 --dns-search=centaur.test --hostname=idm2.centaur.test -e SSSD_CONTAINER_TYPE=application --net=default rhel7/sss
d /bin/install.sh realm join -v CENTAUR.TEST
 * Resolving: _ldap._tcp.centaur.test
 * Performing LDAP DSE lookup on: 192.168.122.187
 * Successfully discovered: CENTAUR.TEST
Password for Administrator:  * Required files: /usr/sbin/oddjobd, /usr/libexec/oddjob/mkhomedir, /usr/sbin/sssd, /usr/bin/net
 * LANG=C LOGNAME=root /usr/bin/net -s /var/cache/realmd/realmd-smb-conf.AHSK2Y -U Administrator ads join CENTAUR.TEST
Enter Administrator's password:DNS update failed: NT_STATUS_INVALID_PARAMETER

Using short domain name -- CENTAUR
Joined 'IDM2' to dns domain 'CENTAUR.TEST'
No DNS domain configured for idm2. Unable to perform DNS Update.
 * LANG=C LOGNAME=root /usr/bin/net -s /var/cache/realmd/realmd-smb-conf.AHSK2Y -U Administrator ads keytab create
Enter Administrator's password:
 * /usr/bin/systemctl enable sssd.service
 * /usr/bin/systemctl restart sssd.service
 * /usr/bin/sh -c /usr/sbin/authconfig --update --enablesssd --enablesssdauth --enablemkhomedir --nostart && /usr/bin/systemctl enable oddjobd.service && /usr/bin/systemctl start oddjobd.service

 * Successfully enrolled machine in realm
Service sssd.service configured to run SSSD container.
[root@dione ~]# atomic run rhel7/sssd

[root@dione ~]# echo $?
1
[root@dione ~]# journalctl -f
-- Logs begin at Tue 2017-07-04 21:50:50 UTC. --
Jul 04 22:21:24 dione.centaur.test oci-systemd-hook[24021]: systemdhook <debug>: GID: 0
Jul 04 22:21:24 dione.centaur.test oci-systemd-hook[24021]: systemdhook <info>: uidMappings not found in config
Jul 04 22:21:24 dione.centaur.test oci-systemd-hook[24021]: systemdhook <debug>: UID: 0
Jul 04 22:21:24 dione.centaur.test oci-systemd-hook[24021]: systemdhook <debug>: Skipping as container command is /bin/run.sh, not init or systemd
Jul 04 22:21:24 dione.centaur.test dockerd-current[1156]: container_linux.go:247: starting container process caused "process_linux.go:359: container init caused \"rootfs_linux.go:54: mounting \\\"/etc/krb5.keytab\\\" to rootfs \\\"/var/lib/docker/devicemapper/mnt/037f1b8d626442a6b98201f9c7ca254a05e0e986847e3408619c7b500bba4c16/rootfs\\\" at \\\"/var/lib/docker/devicemapper/mnt/037f1b8d626442a6b98201f9c7ca254a05e0e986847e3408619c7b500bba4c16/rootfs/etc/krb5.keytab\\\" caused \\\"not a directory\\\"\""
Jul 04 22:21:24 dione.centaur.test dockerd-current[1156]: time="2017-07-04T22:21:24.279450471Z" level=error msg="containerd: start container" error="oci runtime error: container_linux.go:247: starting container process caused \"process_linux.go:359: container init caused \\\"rootfs_linux.go:54: mounting \\\\\\\"/etc/krb5.keytab\\\\\\\" to rootfs \\\\\\\"/var/lib/docker/devicemapper/mnt/037f1b8d626442a6b98201f9c7ca254a05e0e986847e3408619c7b500bba4c16/rootfs\\\\\\\" at \\\\\\\"/var/lib/docker/devicemapper/mnt/037f1b8d626442a6b98201f9c7ca254a05e0e986847e3408619c7b500bba4c16/rootfs/etc/krb5.keytab\\\\\\\" caused \\\\\\\"not a directory\\\\\\\"\\\"\"\n" id=266c55fa99238a2262c9219ceb3a781932bb5bdba3e1b82bf0f115f20ebcd064
Jul 04 22:21:24 dione.centaur.test dockerd-current[1156]: time="2017-07-04T22:21:24.281200678Z" level=error msg="Create container failed with error: invalid header field value \"oci runtime error: container_linux.go:247: starting container process caused \\\"process_linux.go:359: container init caused \\\\\\\"rootfs_linux.go:54: mounting \\\\\\\\\\\\\\\"/etc/krb5.keytab\\\\\\\\\\\\\\\" to rootfs \\\\\\\\\\\\\\\"/var/lib/docker/devicemapper/mnt/037f1b8d626442a6b98201f9c7ca254a05e0e986847e3408619c7b500bba4c16/rootfs\\\\\\\\\\\\\\\" at \\\\\\\\\\\\\\\"/var/lib/docker/devicemapper/mnt/037f1b8d626442a6b98201f9c7ca254a05e0e986847e3408619c7b500bba4c16/rootfs/etc/krb5.keytab\\\\\\\\\\\\\\\" caused \\\\\\\\\\\\\\\"not a directory\\\\\\\\\\\\\\\"\\\\\\\"\\\"\\n\""
Jul 04 22:21:24 dione.centaur.test kernel: XFS (dm-4): Unmounting Filesystem
Jul 04 22:21:24 dione.centaur.test dockerd-current[1156]: time="2017-07-04T22:21:24.327542707Z" level=error msg="Handler for POST /v1.24/containers/sssd/start returned error: invalid header field value \"oci runtime error: container_linux.go:247: starting container process caused \\\"process_linux.go:359: container init caused \\\\\\\"rootfs_linux.go:54: mounting \\\\\\\\\\\\\\\"/etc/krb5.keytab\\\\\\\\\\\\\\\" to rootfs \\\\\\\\\\\\\\\"/var/lib/docker/devicemapper/mnt/037f1b8d626442a6b98201f9c7ca254a05e0e986847e3408619c7b500bba4c16/rootfs\\\\\\\\\\\\\\\" at \\\\\\\\\\\\\\\"/var/lib/docker/devicemapper/mnt/037f1b8d626442a6b98201f9c7ca254a05e0e986847e3408619c7b500bba4c16/rootfs/etc/krb5.keytab\\\\\\\\\\\\\\\" caused \\\\\\\\\\\\\\\"not a directory\\\\\\\\\\\\\\\"\\\\\\\"\\\"\\n\""
Jul 04 22:21:24 dione.centaur.test dockerd-current[1156]: time="2017-07-04T22:21:24.327577776Z" level=error msg="Handler for POST /v1.24/containers/sssd/start returned error: invalid header field value \"oci runtime error: container_linux.go:247: starting container process caused \\\"process_linux.go:359: container init caused \\\\\\\"rootfs_linux.go:54: mounting \\\\\\\\\\\\\\\"/etc/krb5.keytab\\\\\\\\\\\\\\\" to rootfs \\\\\\\\\\\\\\\"/var/lib/docker/devicemapper/mnt/037f1b8d626442a6b98201f9c7ca254a05e0e986847e3408619c7b500bba4c16/rootfs\\\\\\\\\\\\\\\" at \\\\\\\\\\\\\\\"/var/lib/docker/devicemapper/mnt/037f1b8d626442a6b98201f9c7ca254a05e0e986847e3408619c7b500bba4c16/rootfs/etc/krb5.keytab\\\\\\\\\\\\\\\" caused \\\\\\\\\\\\\\\"not a directory\\\\\\\\\\\\\\\"\\\\\\\"\\\"\\n\""
Comment 25 Niranjan Mallapadi Raghavender 2017-07-04 22:45:46 UTC 
This issue occurs when /etc/krb5.keytab directory is failed to remove by previous installs and when trying to start a new sssd application container, it fails .
Comment 26 Ian McLeod 2017-07-10 14:49:43 UTC 
(In reply to Niranjan Mallapadi Raghavender from comment #25)
> This issue occurs when /etc/krb5.keytab directory is failed to remove by
> previous installs and when trying to start a new sssd application container,
> it fails .

What is it that you believe is creating this krb5.keytab file initially?  Is it a prior sssd container install?  If so, is this something that needs to be fixed in the container INSTALL script, rather than the atomic command?
Comment 27 Ian McLeod 2017-07-10 19:22:08 UTC 
(In reply to Ian McLeod from comment #26)
> (In reply to Niranjan Mallapadi Raghavender from comment #25)
> > This issue occurs when /etc/krb5.keytab directory is failed to remove by
> > previous installs and when trying to start a new sssd application container,
> > it fails .
> 
> What is it that you believe is creating this krb5.keytab file initially?  Is
> it a prior sssd container install?  If so, is this something that needs to
> be fixed in the container INSTALL script, rather than the atomic command?

Partially answering my own question by looking at the sssd-docker component in dist-git.  It certainly looks as if this behavior is driven by scripts that are part of the container build, and that a fix can and probably should live there.  Reassigning component.  Please feel free to push back.
Comment 28 Lukas Slebodnik 2017-07-10 20:44:29 UTC 
I tried reproducer from  Comment 24 and I am not able to reproduce.
Please provide simpler reproducer. And in case of "-e SSSD_CONTAINER_TYPE=application", please provide content of directory: /var/lib/sssd_container/
Comment 29 Niranjan Mallapadi Raghavender 2017-07-11 06:48:24 UTC 
Below is the script to reproduce the issue.

#!/bin/bash
atomic install rhel7/sssd realm join -v CENTAUR.TEST 
systemctl start sssd
systemctl stop sssd
atomic uninstall rhel7/sssd realm leave -v CENTAUR.TEST
atomic install --opt1='--dns=192.168.122.187 \
--dns-search=centaur.test --hostname=abc.centaur.test \
-e SSSD_CONTAINER_TYPE=application --net=default' rhel7/sssd realm join -v CENTAUR.TEST
atomic run rhel7/sssd
ret=$?
if [ $ret -eq 1 ]
then
        journalctl -x -n 100 --no-pager
        ls -l /var/lib/sssd_container
fi
atomic stop sssd
atomic uninstall -f rhel7/sssd realm leave -v CENTAUR.TEST


[root@dione sssd_container]# ll
total 0
drwxr-xr-x. 4 root root 112 Jun 27 23:13 idm_sssd1
drwxr-xr-x. 4 root root 112 Jun 19 11:29 test2

[root@dione sssd_container]# ls -l /var/lib/sssd_container/idm_sssd1/
total 12
drwxr-xr-x. 4 root root   28 Jun 27 23:13 client
drwxr-xr-x. 4 root root   28 Jun 27 23:13 container
-rw-r--r--. 1 root root   12 Jun 27 23:13 container_type
-rw-r--r--. 1 root root  880 Jun 27 23:13 docker-client-run-opts
-rw-r--r--. 1 root root 2671 Jun 27 23:13 docker-run-opts
[root@dione sssd_container]# ls -l /var/lib/sssd_container/test2/
total 12
drwxr-xr-x. 4 root root   28 Jun 19 11:29 client
drwxr-xr-x. 4 root root   28 Jun 19 11:29 container
-rw-r--r--. 1 root root   12 Jun 19 11:29 container_type
-rw-r--r--. 1 root root  840 Jun 19 11:29 docker-client-run-opts
-rw-r--r--. 1 root root 2551 Jun 19 11:29 docker-run-opts

idm_sssd1, and test2 were my earlier runs where i had created container with different names, the above script uses default name sssd.
Comment 30 Lukas Slebodnik 2017-07-18 10:58:01 UTC 
(In reply to Niranjan Mallapadi Raghavender from comment #29)
> Below is the script to reproduce the issue.
> 
> #!/bin/bash
> atomic install rhel7/sssd realm join -v CENTAUR.TEST

I am kindly requested simple reproducer in comment 28 because I was not PTO
and I didn't have a time for reservation of machines in beaker.

BTW provided script is not a simple reproducer

-bash-4.2# atomic install rhel7/sssd realm join -v CENTAUR.TEST
docker run --rm=true --privileged --net=host -v /:/host -e NAME=sssd -e IMAGE=rhel7/sssd -e HOST=/host rhel7/sssd /bin/install.sh realm join -v CENTAUR.TEST
Initializing configuration context from host ...
 * Resolving: _ldap._tcp.centaur.test
 * Resolving: centaur.test
 * No results: centaur.test
realm: No such realm found

The problem is that I do not have an access to private network 192.168.122.187.
Comment 31 Lukas Slebodnik 2017-07-18 11:08:39 UTC 
There is much simpler reproducer:
* install sssd image
* start sssd container => so sssd container will be created and running
* stop sssd container => it will be stopped but not removed.
* unistall sssd image => all required data will be removed from host but container will *not* be removed
* run sssd container => it will fail because required data (keytab, some configuration files ... were removed)

It is not a bug in sssd image. It is a bug in atomic utility that it allows to uninstall image even though container exists. It should at least warn in such situation. Because "atomic uninstall --force" will remove all contianers based on that image. And atomic install might be called more times with different --name.



 -bash-4.2# atomic install rhel7/sssd realm join -v ADRELM12.COM                                                                                                                            
docker run --rm=true --privileged --net=host -v /:/host -e NAME=sssd -e IMAGE=rhel7/sssd -e HOST=/host rhel7/sssd /bin/install.sh realm join -v ADRELM12.COM
Initializing configuration context from host ...
 * Resolving: _ldap._tcp.adrelm12.com
 * Performing LDAP DSE lookup on: 10.16.71.181
 * Performing LDAP DSE lookup on: 2620:52:0:1040:6cb0:1b9b:99bf:e331
 * Successfully discovered: ADRELM12.COM
Password for Administrator:  * Required files: /usr/sbin/oddjobd, /usr/libexec/oddjob/mkhomedir, /usr/sbin/sssd, /usr/bin/net
 * LANG=C LOGNAME=root /usr/bin/net -s /var/cache/realmd/realmd-smb-conf.AQAM3Y -U Administrator ads join ADRELM12.COM
Enter Administrator's password:DNS update failed: NT_STATUS_INVALID_PARAMETER

Using short domain name -- ADRELM12COM
Joined 'ATOMIC' to dns domain 'ADRELM12.COM'
No DNS domain configured for atomic. Unable to perform DNS Update.
 * LANG=C LOGNAME=root /usr/bin/net -s /var/cache/realmd/realmd-smb-conf.AQAM3Y -U Administrator ads keytab create
Enter Administrator's password:
 * /usr/bin/systemctl enable sssd.service
 * /usr/bin/systemctl restart sssd.service
 * /usr/bin/sh -c /usr/sbin/authconfig --update --enablesssd --enablesssdauth --enablemkhomedir --nostart && /usr/bin/systemctl enable oddjobd.service && /usr/bin/systemctl start oddjobd.
service

 * Successfully enrolled machine in realm
Copying new configuration to host ...
Full path required for exclude: net:[4026531956].
Full path required for exclude: net:[4026532211].
Service sssd.service configured to run SSSD container.
-bash-4.2# 
-bash-4.2# docker ps -a
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS                        PORTS               NAMES
86000a3a9848        beaker-harness      "/usr/sbin/init"    2 hours ago         Exited (137) 34 minutes ago                       reverent_hopper
-bash-4.2# systemctl start sssd
-bash-4.2# systemctl stop sssd                                                                                                                                                             
-bash-4.2#
-bash-4.2# docker ps -a
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS                        PORTS               NAMES
6d1728c73fda        rhel7/sssd          "/bin/run.sh"       30 seconds ago      Exited (143) 23 seconds ago                       sssd
86000a3a9848        beaker-harness      "/usr/sbin/init"    2 hours ago         Exited (137) 35 minutes ago                       reverent_hopper
-bash-4.2# 
-bash-4.2# 
-bash-4.2# atomic uninstall rhel7/sssd                                                                                                                                                     
docker run --rm=true --privileged --net=host -v /:/host -e NAME=sssd -e IMAGE=rhel7/sssd -e HOST=/host rhel7/sssd /bin/uninstall.sh
Initializing configuration context from host ...
Warning: Failed to copy /etc/yp.conf to host. It cannot be a directory
Copying new configuration to host ...
Removing /etc/krb5.keytab
Removing /etc/sssd/systemctl-lite-enabled/sssd.service
Removing /etc/yp.conf
Removing /var/lib/authconfig/last/krb5.conf
Removing /var/lib/authconfig/last/sssd.conf
Removing /var/lib/sss/pipes/private/sbus-dp_ADRELM12.COM.67
Removing /var/lib/sss/pipes/private/pam
-bash-4.2# 
-bash-4.2# atomic run rhel7/sssd                                                                                                                                                           

-bash-4.2# echo $?
1

-bash-4.2# docker ps -a
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS                        PORTS               NAMES
6d1728c73fda        rhel7/sssd          "/bin/run.sh"       8 minutes ago       Exited (143) 8 minutes ago                        sssd
86000a3a9848        beaker-harness      "/usr/sbin/init"    2 hours ago         Exited (137) 43 minutes ago                       reverent_hopper

-bash-4.2# cat /var/lib/atomic/install.json
{}-bash-4.2#
Comment 33 Daniel Walsh 2017-07-18 12:58:47 UTC 
Giuseppe, can you take a look at this while Brent is on PTO?
Comment 34 Giuseppe Scrivano 2017-07-18 13:27:45 UTC 
@Dan are users allowed to install multiple containers from the same image?

With system containers this is allowed if the image is built in a certain way (as the etcd one) but I am not sure about atomic containers, do we ever allow it?
Comment 35 Lukas Slebodnik 2017-07-18 14:58:44 UTC 
sssd-docker image is created in such way that it stores data in different directories with different --name used at install time. So different client containers will use different configuration of sssd.

And I would assume you might do the same with any other based image (apache, same DB based images ...)
Comment 36 Giuseppe Scrivano 2017-07-18 15:29:39 UTC 
what is the "atomic uninstall X" semantic in the case of multiple containers based on image X. Should it remove the data for all the containers based on the specified image?

I am asking that to understand if it is enough to add a check to uninstall to prevent it when there are containers based on the image or we need something else to be changed.
Comment 37 Daniel Walsh 2017-07-18 15:42:57 UTC 
atomic uninstall X will remove the container with --name=X
atomic uninstall --name foobar X
Will uninstall the container named foobar
Comment 38 Daniel Walsh 2017-07-18 15:43:43 UTC 
We have always allowed users to install multiple containers from the same image, but they have to uninstall the containers installed from the image.
Comment 39 Giuseppe Scrivano 2017-07-18 15:57:59 UTC 
in that case it looks like a bug in the sssd image since uninstall deletes also files used by other containers (not just by the container X) and doesn't honor --name.

Alternatively, I've prepared a PR to disable uninstall when there are containers using the image: https://github.com/projectatomic/atomic/pull/1050

After the last comments, I am prone to close it since it doesn't look like the correct thing to do now.
Comment 40 Giuseppe Scrivano 2017-07-19 08:56:23 UTC 
Fixed upstream by: https://github.com/projectatomic/atomic/pull/1050
Comment 41 Lukas Slebodnik 2017-07-19 09:37:03 UTC 
(In reply to Daniel Walsh from comment #37)
> atomic uninstall X will remove the container with --name=X

It does not remove container with name "X". It just call uninstall label for container "X"

And IIRC it is since 7.3.4
Comment 42 Daniel Walsh 2017-07-19 11:49:17 UTC 
Then that is a bug.  Giuseppe Patch should fix this.
Comment 48 Niranjan Mallapadi Raghavender 2017-08-21 15:36:03 UTC 
Versions:

● atomic-host:rhel-atomic-host/7/x86_64/standard
                Version: 7.4.1 (2017-08-09 22:48:55)
                 Commit: fd2649205bbc185b07cb695893e425afa830a7b3caecd3ceefc0a1b9d7169aea


atomic-1.18.1-4.git64843d3.el7.x86_64


-bash-4.2# atomic info rhel7/sssd
Image Name: registry.access.redhat.com/rhel7/sssd:latest
RUN_OPTS_FILE: /var/lib/sssd_container/${NAME}/docker-run-opts
architecture: x86_64
authoritative-source-url: registry.access.redhat.com
build-date: 2017-07-27T11:37:38.452510
com.redhat.build-host: ip-10-29-120-57.ec2.internal
com.redhat.component: sssd-docker
description: The System Security Services Daemon (SSSD) provides a set of daemons to manage access to remote directories and authentication mechanisms. It provides Name Service Switch (NSS) and Pluggable Authentication Modules(PAM) interfaces toward the system and a pluggable back end system to connect to multiple different account sources.
distribution-scope: public
install: docker run --rm=true --privileged --net=host -v /:/host        -e NAME=${NAME} -e IMAGE=${IMAGE} -e HOST=/host         ${OPT1}         ${IMAGE} /bin/install.sh
io.k8s.description: The System Security Services Daemon (SSSD) provides a set of daemons to manage access to remote directories and authentication mechanisms. It provides Name Service Switch (NSS) and Pluggable Authentication Modules(PAM) interfaces toward the system and a pluggable back end system to connect to multiple different account sources.
io.k8s.display-name: System Security Services Daemon (SSSD)
io.k8s.openshift.tags: security sssd authentication authorisation LDAP kerberos krb5 Active Directory IdM
io.openshift.tags: base rhel7
name: rhel7/sssd
release: 7
run: docker run -d --restart=always --name ${NAME} -e NAME=${NAME} -e IMAGE=${IMAGE}    ${RUN_OPTS}     ${IMAGE} /bin/run.sh
stop: docker kill -s TERM ${NAME}
summary: System Security Services Daemon (SSSD) provides centralized user authentication for Atomic Host.
uninstall: docker run --rm=true --privileged --net=host -v /:/host -e NAME=${NAME} -e IMAGE=${IMAGE} -e HOST=/host ${IMAGE} /bin/uninstall.sh
url: https://access.redhat.com/containers/#/registry.access.redhat.com/rhel7/sssd/images/7.4-7
vcs-ref: c68d211e9ba0a60e48f7283669cfea3615157283
vcs-type: git
vendor: Red Hat, Inc.



Steps:

I used the below script to reproduce the issue earlier. The issue is atomic run rhel7/sssd failed earlier (line 9). 
 1 #!/bin/bash
      2 atomic install rhel7/sssd realm join -v JUNO.TEST
      3 systemctl start sssd
      4 systemctl stop sssd
      5 atomic uninstall rhel7/sssd realm leave -v JUNO.TEST
      6 atomic install --opt1='--dns=10.65.223.136 \
      7 --dns-search=juno.test --hostname=abc.juno.test \
      8 -e SSSD_CONTAINER_TYPE=application --net=default' rhel7/sssd realm join -v JUNO.TEST
      9 atomic run rhel7/sssd
     10 ret=$?
     11 if [ $ret -eq 1 ]
     12 then
     13         journalctl -x -n 100 --no-pager
     14         ls -l /var/lib/sssd_container
     15 fi
     16 atomic stop sssd
     17 atomic uninstall -f rhel7/sssd realm leave -v JUNO.TEST
     18 

In the below output, as you can see atomic run rhel7/sssd (line 9) passed wouth out any issues. 

+ atomic install rhel7/sssd realm join -v JUNO.TEST
docker run --rm=true --privileged --net=host -v /:/host -e NAME=sssd -e IMAGE=rhel7/sssd -e HOST=/host rhel7/sssd /bin/install.sh realm join -v JUNO.TEST
Initializing configuration context from host ...
 * Resolving: _ldap._tcp.juno.test
 * Performing LDAP DSE lookup on: 10.65.223.136
 * Successfully discovered: juno.test
Password for Administrator:  * Required files: /usr/sbin/oddjobd, /usr/libexec/oddjob/mkhomedir, /usr/sbin/sssd, /usr/bin/net
 * LANG=C LOGNAME=root /usr/bin/net -s /var/cache/realmd/realmd-smb-conf.VXBA5Y -U Administrator ads join juno.test
Enter Administrator's password:DNS update failed: NT_STATUS_UNSUCCESSFUL

Using short domain name -- JUNO
Joined 'MGMT3' to dns domain 'juno.test'
DNS Update for mgmt3.rhq.lab.eng.bos.redhat.com failed: ERROR_DNS_GSS_ERROR
 * LANG=C LOGNAME=root /usr/bin/net -s /var/cache/realmd/realmd-smb-conf.VXBA5Y -U Administrator ads keytab create
Enter Administrator's password:
 * /usr/bin/systemctl enable sssd.service
 * /usr/bin/systemctl restart sssd.service
 * /usr/bin/sh -c /usr/sbin/authconfig --update --enablesssd --enablesssdauth --enablemkhomedir --nostart && /usr/bin/systemctl enable oddjobd.service && /usr/bin/systemctl start oddjobd.service
 * Successfully enrolled machine in realm

Copying new configuration to host ...
Service sssd.service configured to run SSSD container.
+ systemctl start sssd
+ systemctl stop sssd
+ atomic uninstall rhel7/sssd realm leave -v JUNO.TEST
Containers `sssd` are using this image, delete them first or use --force
+ atomic install '--opt1=--dns=10.65.223.136 \
--dns-search=juno.test --hostname=abc.juno.test \
-e SSSD_CONTAINER_TYPE=application --net=default' rhel7/sssd realm join -v JUNO.TEST
docker run --rm=true --privileged --net=host -v /:/host -e NAME=sssd -e IMAGE=rhel7/sssd -e HOST=/host --dns=10.65.223.136 \ --dns-search=juno.test --hostname=abc.juno.test \ -e SSSD_CONTAINER_TYPE=application --net=default rhel7/sssd /bin/install.sh realm join -v JUNO.TEST
/usr/bin/docker-current: Error parsing reference: " --dns-search=juno.test" is not a valid repository/tag.
See '/usr/bin/docker-current run --help'.

+ atomic run rhel7/sssd
sssd
+ ret=0
+ '[' 0 -eq 1 ']'
+ atomic stop sssd
docker kill -s TERM sssd
sssd
+ atomic uninstall -f rhel7/sssd realm leave -v JUNO.TEST
docker run --rm=true --privileged --net=host -v /:/host -e NAME=sssd -e IMAGE=rhel7/sssd -e HOST=/host rhel7/sssd /bin/uninstall.sh realm leave -v JUNO.TEST
Initializing configuration context from host ...
Warning: Failed to copy /etc/yp.conf to host. It cannot be a directory
 * Removing entries from keytab for realm
 * /usr/sbin/sss_cache --users --groups --netgroups --services --autofs-maps
No cache object matched the specified search
 ! Flushing the sssd cache failed
 * Removing domain configuration from sssd.conf
 * /usr/sbin/authconfig --update --disablesssdauth --nostart
 * /usr/bin/systemctl disable sssd.service
 * Successfully unenrolled machine from realm
Copying new configuration to host ...
Removing /etc/krb5.keytab
Removing /etc/sssd/systemctl-lite-enabled/sssd.service
Removing /etc/yp.conf
Removing /var/lib/authconfig/last/krb5.conf
Removing /var/lib/authconfig/last/sssd.conf
Removing /var/lib/sss/pipes/private/sbus-dp_juno.test.66
Removing /var/lib/sss/pipes/private/pam
-
Comment 49 Niranjan Mallapadi Raghavender 2017-08-21 15:58:08 UTC 
Install a single container using below command .

1. Install a container named idm1 
atomic install --opt1='--dns=10.65.223.136 \
--dns-search=juno.test --hostname=abc.juno.test \
-e SSSD_CONTAINER_TYPE=application --net=default' --name idm1 rhel7/sssd realm join -v JUNO.TEST
docker run --rm=true --privileged --net=host -v /:/host -e NAME=idm1 -e IMAGE=rhel7/sssd -e HOST=/host --dns=10.65.223.136 --dns-search=juno.test --hostname=abc.juno.test -e SSSD_CONTAINER_TYPE=application --net=default rhel7/sssd /bin/install.sh realm join -v JUNO.TEST
 * Resolving: _ldap._tcp.juno.test
 * Performing LDAP DSE lookup on: 10.65.223.136
 * Successfully discovered: juno.test
Password for Administrator:  * Required files: /usr/sbin/oddjobd, /usr/libexec/oddjob/mkhomedir, /usr/sbin/sssd, /usr/bin/net
 * LANG=C LOGNAME=root /usr/bin/net -s /var/cache/realmd/realmd-smb-conf.0O934Y -U Administrator ads join juno.test
Enter Administrator's password:
Using short domain name -- JUNO
Joined 'ABC' to dns domain 'juno.test'
 * LANG=C LOGNAME=root /usr/bin/net -s /var/cache/realmd/realmd-smb-conf.0O934Y -U Administrator ads keytab create
Enter Administrator's password:
 * /usr/bin/systemctl enable sssd.service
 * /usr/bin/systemctl restart sssd.service
 * /usr/bin/sh -c /usr/sbin/authconfig --update --enablesssd --enablesssdauth --enablemkhomedir --nostart && /usr/bin/systemctl enable oddjobd.service && /usr/bin/systemctl start oddjobd.service
 * Successfully enrolled machine in realm

Service idm1.service configured to run SSSD container.




2. Start the container. 


docker run -d --restart=always --name idm1 -e NAME=idm1 -e IMAGE=rhel7/sssd --hostname abc.juno.test -v /var/lib/sssd_container/idm1/container/etc/dbus-1/system.d/:/etc/dbus-1/system.d/:Z -v /var/lib/sssd_container/idm1/container/etc/ipa/:/etc/ipa/:Z -v /var/lib/sssd_container/idm1/container/etc/openldap/:/etc/openldap/:Z -v /var/lib/sssd_container/idm1/container/etc/ssh/:/etc/ssh/:Z -v /var/lib/sssd_container/idm1/container/etc/sssd/:/etc/sssd/:Z -v /var/lib/sssd_container/idm1/container/var/cache/realmd/:/var/cache/realmd/:Z -v /var/lib/sssd_container/idm1/container/var/lib/authconfig/last/:/var/lib/authconfig/last/:Z -v /var/lib/sssd_container/idm1/container/var/lib/ipa-client/sysrestore/:/var/lib/ipa-client/sysrestore/:Z -v /var/lib/sssd_container/idm1/container/var/lib/samba/:/var/lib/samba/:Z -v /var/lib/sssd_container/idm1/container/var/lib/sss/db/:/var/lib/sss/db/:Z -v /var/lib/sssd_container/idm1/container/var/lib/sss/gpo_cache/:/var/lib/sss/gpo_cache/:Z -v /var/lib/sssd_container/idm1/container/var/lib/sss/secrets/:/var/lib/sss/secrets/:Z -v /var/lib/sssd_container/idm1/container/var/lib/sss/keytabs/:/var/lib/sss/keytabs/:Z -v /var/lib/sssd_container/idm1/container/var/log/sssd/:/var/log/sssd/:Z -v /var/lib/sssd_container/idm1/container/etc/krb5.keytab:/etc/krb5.keytab:Z -v /var/lib/sssd_container/idm1/container/etc/nsswitch.conf:/etc/nsswitch.conf:Z -v /var/lib/sssd_container/idm1/container/etc/sysconfig/authconfig:/etc/sysconfig/authconfig:Z -v /var/lib/sssd_container/idm1/container/etc/sysconfig/network:/etc/sysconfig/network:Z -v /var/lib/sssd_container/idm1/container/etc/yp.conf:/etc/yp.conf:Z -v /var/lib/sssd_container/idm1/container/etc/resolv.conf:/etc/resolv.conf:Z -v /var/lib/sssd_container/idm1/client/etc/krb5.conf.d/:/etc/krb5.conf.d/:z -v /var/lib/sssd_container/idm1/client/var/lib/sss/mc/:/var/lib/sss/mc/:z -v /var/lib/sssd_container/idm1/client/var/lib/sss/pipes/:/var/lib/sss/pipes/:z -v /var/lib/sssd_container/idm1/client/var/lib/sss/pubconf/:/var/lib/sss/pubconf/:z -v /var/lib/sssd_container/idm1/client/var/run/dbus/:/var/run/dbus/:z -v /var/lib/sssd_container/idm1/client/etc/krb5.conf:/etc/krb5.conf:z -v /var/lib/sssd_container/idm1/client/etc/pam.d/fingerprint-auth-ac:/etc/pam.d/fingerprint-auth:z -v /var/lib/sssd_container/idm1/client/etc/pam.d/password-auth-ac:/etc/pam.d/password-auth:z -v /var/lib/sssd_container/idm1/client/etc/pam.d/smartcard-auth-ac:/etc/pam.d/smartcard-auth:z -v /var/lib/sssd_container/idm1/client/etc/pam.d/system-auth-ac:/etc/pam.d/system-auth:z -e WITH_KCM=yes -e SSSD_CONTAINER_TYPE=application rhel7/sssd /bin/run.sh
742bdcab60c6195a525e75e9b0ec4861a0ad7367792e1c5cc5f6e35555ed7225



3. Verify container is running:

$ docker ps
-bash-4.2# docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
742bdcab60c6        rhel7/sssd          "/bin/run.sh"       18 seconds ago      Up 15 seconds                           idm1

5. The following Directories will be created under /var/lib/sssd_container
-bash-4.2# ls -l /var/lib/sssd_container/idm1/
total 12
drwxr-xr-x. 4 root root   28 Aug 21 11:54 client
drwxr-xr-x. 4 root root   28 Aug 21 11:54 container
-rw-r--r--. 1 root root   12 Aug 21 11:54 container_type
-rw-r--r--. 1 root root  830 Aug 21 11:54 docker-client-run-opts
-rw-r--r--. 1 root root 2518 Aug 21 11:54 docker-run-opts
-bash-4.2# cd ..




6. Stop the container idm1
-bash-4.2# atomic stop idm1
docker kill -s TERM idm1
idm1

-bash-4.2# docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES

6. Uninstall idm1 container

-bash-4.2# atomic uninstall --name idm1 rhel7/sssd
docker run --rm=true --privileged --net=host -v /:/host -e NAME=idm1 -e IMAGE=rhel7/sssd -e HOST=/host rhel7/sssd /bin/uninstall.sh


7.  Verify there is no container named idm1

-bash-4.2# ls -l /var/lib/sssd_container/
total 0


8. -bash-4.2# docker start idm1
Error response from daemon: No such container: idm1
Error: failed to start containers: idm1
Comment 50 Alex Jia 2017-08-21 16:07:08 UTC 
Moving the bug to VERIFIED status per Comment 48.
Comment 52 errata-xmlrpc 2017-09-05 10:39:10 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:2609