Bug 995192 - [RFE] IPA should provide Stateless support
[RFE] IPA should provide Stateless support
Status: CLOSED WONTFIX
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: ipa (Show other bugs)
7.0
Unspecified Unspecified
unspecified Severity unspecified
: rc
: ---
Assigned To: Martin Kosek
Namita Soman
: FutureFeature
Depends On:
Blocks: 922113
  Show dependency treegraph
 
Reported: 2013-08-08 14:00 EDT by Namita Soman
Modified: 2013-09-03 08:46 EDT (History)
4 users (show)

See Also:
Fixed In Version:
Doc Type: Enhancement
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-08-13 12:16:41 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
Rewritten fedora-readonly script (5.38 KB, text/plain)
2013-08-14 12:03 EDT, Václav Pavlín
no flags Details

  None (edit)
Description Namita Soman 2013-08-08 14:00:53 EDT
Description of problem:
Currently ipa doesn't support running on stateless linux env.


Version-Release number of selected component (if applicable):
ipa-server-3.2.2-1.el7.x86_64
389-ds-base-1.3.1.4-1.el7.x86_64

How reproducible:
always

Steps to Reproduce:
1. To set up env in Stateless mode
# Edit  /etc/fstab file - and changed - from:
/dev/mapper/rhel_storm-root /                       ext4    defaults        1 1
to:
/dev/mapper/rhel_storm-root /                       ext4    ro        1 1
# Edit /etc/sysconfig/readonly-root file - and changed - from
READONLY=no
to
READONLY=yes

2. install ipa-server
3.Reboot machine
4. When machine is back up, check ipa status : ipactl status

Actual results:
ipa: ERROR: Cannot open log file u'/root/.ipa/log/ipactl.log': [Errno 30] Read-only file system: u'/root/.ipa/log/ipactl.log'


Expected results:
ipa server should be up and running

Additional info:
Comment 2 Namita Soman 2013-08-08 14:13:06 EDT
Tried the below:
edited /etc/rwtab and added lines:
dirs    /root/.ipa
dirs    /etc/dirsrv

rebooted. 
# ipactl status
Directory Service: STOPPED
Directory Service must be running in order to obtain status of other services

# ipactl start
Starting Directory Service
Failed to start Directory Service: 

/var/log/messages had:
[08/Aug/2013:10:45:33 -0400] dse - Unable to write "/etc/dirsrv/slapd-TESTRELM-COM/dse.ldif": Netscape Portable Runtime error -5948 (Cannot write to a read-only file system.)
...
Aug  8 11:53:05 mgmt2 sshd[1940]: error: Could not load host key: /etc/ssh/ssh_host_ecdsa_key

 
from /var/log/dirsrv/slapd-TESTRELM-COM/errors:
        389-Directory/1.3.1.4 B2013.200.2348
        mgmt2.testrelm.com:636 (/etc/dirsrv/slapd-TESTRELM-COM)

[08/Aug/2013:10:39:33 -0400] dse - The DSE database stored in "/etc/dirsrv/slapd-TESTRELM-COM/dse.ldif" is not writeable
[08/Aug/2013:10:39:33 -0400] dse - Unable to write "/etc/dirsrv/slapd-TESTRELM-COM/dse.ldif": Netscape Portable Runtime error -5948 (Cannot write to a read-only file system.)
[08/Aug/2013:10:39:33 -0400] - SSL alert: Security Initialization: NSS initialization failed (Netscape Portable Runtime error -8015 - unknown error): certdir: /etc/dirsrv/slapd-TESTRELM-COM
[08/Aug/2013:10:39:33 -0400] - ERROR: NSS Initialization Failed.
[08/Aug/2013:10:45:33 -0400] dse - The DSE database stored in "/etc/dirsrv/slapd-TESTRELM-COM/dse.ldif" is not writeable
[08/Aug/2013:10:45:33 -0400] dse - Unable to write "/etc/dirsrv/slapd-TESTRELM-COM/dse.ldif": Netscape Portable Runtime error -5948 (Cannot write to a read-only file system.)
[08/Aug/2013:10:45:33 -0400] - SSL alert: Security Initialization: NSS initialization failed (Netscape Portable Runtime error -8015 - unknown error): certdir: /etc/dirsrv/slapd-TESTRELM-COM
[08/Aug/2013:10:45:33 -0400] - ERROR: NSS Initialization Failed.
~
Comment 3 Rob Crittenden 2013-08-08 14:26:39 EDT
What is the use case for running an IPA server on a stateless system?
Comment 5 Martin Kosek 2013-08-09 02:18:58 EDT
(In reply to Rob Crittenden from comment #3)
> What is the use case for running an IPA server on a stateless system?

+1. I would understand Stateless system for HTTP server where you just serve a content and can route logs to other machine. But with Identity Management server you need to write sometimes, if only to update user passwords, renew certificates, Kerberos keytabs, join systems, etc. etc.

Adding Vaclav to help us understand the motivation of Stateless system for IdM solution.
Comment 6 Václav Pavlín 2013-08-09 05:40:23 EDT
Namita,

just to understand why that doesn't work, could you provide output of 
mount
ls -l /etc | grep dirsrv
ls -l /etc/dirsrv

Thanks

To the logging - ipa should probably use some standard way of logging or at least maybe store it's logs in /var/log, which is writeable in Stateless.

Martin,

I agree, I can't really think of any example of use case for IPA server running in Stateless Linux. You probably always want IM server to have persistent data. 

Alex, 
are you aware of any reason we want to support ipa in Stateless configuration? Otherwise, shouldn't it be removed from test plan?
Comment 7 Martin Kosek 2013-08-09 06:12:18 EDT
Providing info from IdM server as Vaclav requested:

# mount
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime,seclabel)
devtmpfs on /dev type devtmpfs (rw,nosuid,seclabel,size=503304k,nr_inodes=125826,mode=755)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
selinuxfs on /sys/fs/selinux type selinuxfs (rw,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,seclabel)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,seclabel,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,seclabel,mode=755)
tmpfs on /sys/fs/cgroup type tmpfs (rw,nosuid,nodev,noexec,seclabel,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpuacct,cpu)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/net_cls type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
/dev/mapper/vg_root-lv_root on / type ext4 (rw,relatime,seclabel,data=ordered)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=32,pgrp=1,timeout=300,minproto=5,maxproto=5,direct)
configfs on /sys/kernel/config type configfs (rw,relatime)
mqueue on /dev/mqueue type mqueue (rw,relatime,seclabel)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,seclabel)
tmpfs on /tmp type tmpfs (rw,seclabel)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime)
sunrpc on /proc/fs/nfsd type nfsd (rw,relatime)
/etc/auto.misc on /misc type autofs (rw,relatime,fd=6,pgrp=10330,timeout=300,minproto=5,maxproto=5,indirect)
-hosts on /net type autofs (rw,relatime,fd=12,pgrp=10330,timeout=300,minproto=5,maxproto=5,indirect)
/etc/auto.home on /home type autofs (rw,relatime,fd=18,pgrp=10330,timeout=300,minproto=5,maxproto=5,indirect)
binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,relatime)
(rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.16.78.1,mountvers=3,mountport=4002,mountproto=udp,local_lock=none,addr=10.16.78.1)

# ls -l /etc | grep dirsrv
drwxrwxr-x.  5 root  dirsrv   4096 Aug  6 18:52 dirsrv

# ls -l /etc/dirsrv
total 16
drwxr-xr-x. 2 root   root   4096 Jul 15 19:07 config
-rw-------. 1 dirsrv dirsrv  410 Aug  6 18:52 ds.keytab
drwxr-xr-x. 2 root   root   4096 Jul 15 19:07 schema
drwxrwx---. 3 dirsrv dirsrv 4096 Aug  9 06:08 slapd-IDM-LAB-BOS-REDHAT-COM

Adding needinfo on Alex to help with a motivation for this effort with IdM.
Comment 8 Václav Pavlín 2013-08-09 08:43:15 EDT
Thank you Martin, there is the same problem as with mysql - all directories bind mounted over ro root are owned by root:root, thus are not writeable for applications running under different user (i.e. dirsrv in this case).

Whatever the use case might be, I think it would be nice to have as much running applications in stateless as possible, so I will think about the way how to enhance readonly-root script, so that it allows applications to set up dirs with the right owner.
Comment 9 Václav Pavlín 2013-08-12 07:34:00 EDT
Namita, could I get access to the machine where ipa and stateless support are set up?
Comment 11 Martin Kosek 2013-08-13 09:57:46 EDT
Václav, any progress in your tests mentioned in Comment 9?

I am now tempted to just close this bug as WONTFIX as stateless does not make really sense for IdM solution, as it was said in several comments.
Comment 12 Václav Pavlín 2013-08-13 10:07:55 EDT
I am not able to login into that machine via ssh, so no tests happened thus no progress. Feel free to close this bug as we all agreed this is probably not valid use case for the stateless linux. 

We are investigating possibilities of how to solve the mysql bug, which might help IPA as well. I will reopen this bug if there are any significant changes to readonly-root script which we consider worth testing from your side.
Comment 13 Martin Kosek 2013-08-13 12:16:41 EDT
Ok, thanks for info. Closing the bug until then.
Comment 14 Václav Pavlín 2013-08-14 12:03:53 EDT
Created attachment 786588 [details]
Rewritten fedora-readonly script

Hi Namita, 
I rewrote part of the readonly script and would like to ask you if you could test ipa again with this new version.

Just replace /lib/systemd/fedora-readonly with attached file.

Obviously, You will need to add files to rwtab again.
Comment 15 Namita Soman 2013-08-20 13:59:41 EDT
I followed Vaclav's suggestion, and replaced /lib/systemd/rhel-readonly with the attached script. Didn't check script to see if it had anything fedora specific, but I am testing on rhel7.


And behaviour this time.....
# ipactl start
Existing service file detected!
Assuming stale, cleaning and proceeding
Starting Directory Service
Failed to data from service file: Failed to get list of services to probe status:
Directory Server is stopped
Shutting down


/var/log/messages does not exist
/var/log/dirsrv/slapd-TESTRELM-COM is empty
Comment 16 Václav Pavlín 2013-09-03 08:46:18 EDT
The basic problem, I ran into, is that Directory Server tries to find some files in /etc/dirsrv/slapd-IPA-TEST, which are gone when the directory is bind mounted. I've created rwtab file that contained all files in this directory, but then Directory Server segfaulted. From my point of view I am not able to fix this on side of Stateless Linux. To provide IPA on readonly system would probably need to fix all compoments of IPA and/or prepare quite large rwtab files for them.

I will gladly help anyone interested in testing and debugging this, but for now I would say IPA is not available in Stateless configuration.

Note You need to log in before you can comment on or make changes to this bug.