Bug 995192

Tried the below:
edited /etc/rwtab and added lines:
dirs    /root/.ipa
dirs    /etc/dirsrv

rebooted. 
# ipactl status
Directory Service: STOPPED
Directory Service must be running in order to obtain status of other services

# ipactl start
Starting Directory Service
Failed to start Directory Service: 

/var/log/messages had:
[08/Aug/2013:10:45:33 -0400] dse - Unable to write "/etc/dirsrv/slapd-TESTRELM-COM/dse.ldif": Netscape Portable Runtime error -5948 (Cannot write to a read-only file system.)
...
Aug  8 11:53:05 mgmt2 sshd[1940]: error: Could not load host key: /etc/ssh/ssh_host_ecdsa_key

 
from /var/log/dirsrv/slapd-TESTRELM-COM/errors:
        389-Directory/1.3.1.4 B2013.200.2348
        mgmt2.testrelm.com:636 (/etc/dirsrv/slapd-TESTRELM-COM)

[08/Aug/2013:10:39:33 -0400] dse - The DSE database stored in "/etc/dirsrv/slapd-TESTRELM-COM/dse.ldif" is not writeable
[08/Aug/2013:10:39:33 -0400] dse - Unable to write "/etc/dirsrv/slapd-TESTRELM-COM/dse.ldif": Netscape Portable Runtime error -5948 (Cannot write to a read-only file system.)
[08/Aug/2013:10:39:33 -0400] - SSL alert: Security Initialization: NSS initialization failed (Netscape Portable Runtime error -8015 - unknown error): certdir: /etc/dirsrv/slapd-TESTRELM-COM
[08/Aug/2013:10:39:33 -0400] - ERROR: NSS Initialization Failed.
[08/Aug/2013:10:45:33 -0400] dse - The DSE database stored in "/etc/dirsrv/slapd-TESTRELM-COM/dse.ldif" is not writeable
[08/Aug/2013:10:45:33 -0400] dse - Unable to write "/etc/dirsrv/slapd-TESTRELM-COM/dse.ldif": Netscape Portable Runtime error -5948 (Cannot write to a read-only file system.)
[08/Aug/2013:10:45:33 -0400] - SSL alert: Security Initialization: NSS initialization failed (Netscape Portable Runtime error -8015 - unknown error): certdir: /etc/dirsrv/slapd-TESTRELM-COM
[08/Aug/2013:10:45:33 -0400] - ERROR: NSS Initialization Failed.
~

What is the use case for running an IPA server on a stateless system?

(In reply to Rob Crittenden from comment #3)
> What is the use case for running an IPA server on a stateless system?

+1. I would understand Stateless system for HTTP server where you just serve a content and can route logs to other machine. But with Identity Management server you need to write sometimes, if only to update user passwords, renew certificates, Kerberos keytabs, join systems, etc. etc.

Adding Vaclav to help us understand the motivation of Stateless system for IdM solution.

Namita,

just to understand why that doesn't work, could you provide output of 
mount
ls -l /etc | grep dirsrv
ls -l /etc/dirsrv

Thanks

To the logging - ipa should probably use some standard way of logging or at least maybe store it's logs in /var/log, which is writeable in Stateless.

Martin,

I agree, I can't really think of any example of use case for IPA server running in Stateless Linux. You probably always want IM server to have persistent data. 

Alex, 
are you aware of any reason we want to support ipa in Stateless configuration? Otherwise, shouldn't it be removed from test plan?

Providing info from IdM server as Vaclav requested:

# mount
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime,seclabel)
devtmpfs on /dev type devtmpfs (rw,nosuid,seclabel,size=503304k,nr_inodes=125826,mode=755)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
selinuxfs on /sys/fs/selinux type selinuxfs (rw,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,seclabel)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,seclabel,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,seclabel,mode=755)
tmpfs on /sys/fs/cgroup type tmpfs (rw,nosuid,nodev,noexec,seclabel,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpuacct,cpu)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/net_cls type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
/dev/mapper/vg_root-lv_root on / type ext4 (rw,relatime,seclabel,data=ordered)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=32,pgrp=1,timeout=300,minproto=5,maxproto=5,direct)
configfs on /sys/kernel/config type configfs (rw,relatime)
mqueue on /dev/mqueue type mqueue (rw,relatime,seclabel)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,seclabel)
tmpfs on /tmp type tmpfs (rw,seclabel)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime)
sunrpc on /proc/fs/nfsd type nfsd (rw,relatime)
/etc/auto.misc on /misc type autofs (rw,relatime,fd=6,pgrp=10330,timeout=300,minproto=5,maxproto=5,indirect)
-hosts on /net type autofs (rw,relatime,fd=12,pgrp=10330,timeout=300,minproto=5,maxproto=5,indirect)
/etc/auto.home on /home type autofs (rw,relatime,fd=18,pgrp=10330,timeout=300,minproto=5,maxproto=5,indirect)
binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,relatime)
(rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.16.78.1,mountvers=3,mountport=4002,mountproto=udp,local_lock=none,addr=10.16.78.1)

# ls -l /etc | grep dirsrv
drwxrwxr-x.  5 root  dirsrv   4096 Aug  6 18:52 dirsrv

# ls -l /etc/dirsrv
total 16
drwxr-xr-x. 2 root   root   4096 Jul 15 19:07 config
-rw-------. 1 dirsrv dirsrv  410 Aug  6 18:52 ds.keytab
drwxr-xr-x. 2 root   root   4096 Jul 15 19:07 schema
drwxrwx---. 3 dirsrv dirsrv 4096 Aug  9 06:08 slapd-IDM-LAB-BOS-REDHAT-COM

Adding needinfo on Alex to help with a motivation for this effort with IdM.

Thank you Martin, there is the same problem as with mysql - all directories bind mounted over ro root are owned by root:root, thus are not writeable for applications running under different user (i.e. dirsrv in this case).

Whatever the use case might be, I think it would be nice to have as much running applications in stateless as possible, so I will think about the way how to enhance readonly-root script, so that it allows applications to set up dirs with the right owner.

Namita, could I get access to the machine where ipa and stateless support are set up?

Václav, any progress in your tests mentioned in Comment 9?

I am now tempted to just close this bug as WONTFIX as stateless does not make really sense for IdM solution, as it was said in several comments.

I am not able to login into that machine via ssh, so no tests happened thus no progress. Feel free to close this bug as we all agreed this is probably not valid use case for the stateless linux. 

We are investigating possibilities of how to solve the mysql bug, which might help IPA as well. I will reopen this bug if there are any significant changes to readonly-root script which we consider worth testing from your side.

Ok, thanks for info. Closing the bug until then.

Created attachment 786588 [details]
Rewritten fedora-readonly script

Hi Namita, 
I rewrote part of the readonly script and would like to ask you if you could test ipa again with this new version.

Just replace /lib/systemd/fedora-readonly with attached file.

Obviously, You will need to add files to rwtab again.

I followed Vaclav's suggestion, and replaced /lib/systemd/rhel-readonly with the attached script. Didn't check script to see if it had anything fedora specific, but I am testing on rhel7.


And behaviour this time.....
# ipactl start
Existing service file detected!
Assuming stale, cleaning and proceeding
Starting Directory Service
Failed to data from service file: Failed to get list of services to probe status:
Directory Server is stopped
Shutting down


/var/log/messages does not exist
/var/log/dirsrv/slapd-TESTRELM-COM is empty

The basic problem, I ran into, is that Directory Server tries to find some files in /etc/dirsrv/slapd-IPA-TEST, which are gone when the directory is bind mounted. I've created rwtab file that contained all files in this directory, but then Directory Server segfaulted. From my point of view I am not able to fix this on side of Stateless Linux. To provide IPA on readonly system would probably need to fix all compoments of IPA and/or prepare quite large rwtab files for them.

I will gladly help anyone interested in testing and debugging this, but for now I would say IPA is not available in Stateless configuration.