Bug 1542567

As mentioned in the support ticket it would be also possible to change the gssproxy service to be configurable via a file /etc/sysconfig/gssproxy like e.g. openldap or openssh. This would allow to customize the default socket which otherwise cannot be done but by editing /lib/systemd/system/gssproxy.service directly.

This is not a solution for rhel7, but in the long run maybe we should move such stuff to /run. It is mounted on tmpfs which removes the problem with read-only root.

To give more information:
gssproxy has a default socket path which is a compile time option. This currently is "/var/lib/gssproxy/default.sock" . The only way to change that is to supply "-s <path to socket>" on the commandline which would need changes in "/lib/systemd/system/gssproxy.service" .

If there was "/etc/sysconfig/gssproxy" and "/lib/systemd/system/gssproxy.service" would include a line "EnvironmentFile=/etc/sysconfig/gssproxy" then the default socket could be configured by the end user without touching "/lib/systemd/system/gssproxy.service" .

The proposed solution avoids the problem by creating a separate tmpfs for gssproxy but lacks the possibility to influence the default socket. It keeps it more simple for the end user who wants to run a read-only / system but lacks a proper possibility to change the socket path.

Perhaps the best solution would be to "just" change the compile time value?

This bug can be reassigned to gssproxy if you believe it is better there.
Then, the solution would be to create a file under /etc/rwtab.d with the
required line.
Solution would work for rhel6 and rhel7.

(In reply to Uwe Sauter from comment #5)
> To give more information:
> gssproxy has a default socket path which is a compile time option. This
> currently is "/var/lib/gssproxy/default.sock" . The only way to change that
> is to supply "-s <path to socket>" on the commandline which would need
> changes in "/lib/systemd/system/gssproxy.service" .
> 
> If there was "/etc/sysconfig/gssproxy" and
> "/lib/systemd/system/gssproxy.service" would include a line
> "EnvironmentFile=/etc/sysconfig/gssproxy" then the default socket could be
> configured by the end user without touching
> "/lib/systemd/system/gssproxy.service" .
> 
> The proposed solution avoids the problem by creating a separate tmpfs for
> gssproxy but lacks the possibility to influence the default socket. It keeps
> it more simple for the end user who wants to run a read-only / system but
> lacks a proper possibility to change the socket path.
> 
> Perhaps the best solution would be to "just" change the compile time value?

Thank you, Uwe. This is actually something we were discussing yesterday as the right way to solve this problem. At least we see it that way as well.

I'm reassigning this BZ to gssproxy, to see if they are any objections to your proposed solution. :)

Um.  What.

Per FHS ( http://refspecs.linuxfoundation.org/FHS_3.0/fhs-3.0.html#purpose31 ):

"""
/var is specified here in order to make it possible to mount /usr read-only. Everything that once went into /usr that is written to during system operation (as opposed to installation and software maintenance) must be in /var.
"""

Mounting /var read-only defeats the entire point of having /var at all, and I read it as a violation of FHS.

I would like to close this ticket as NOTABUG.

Then please tell me how to setup a PXE booted host with NFS root that is shared by many hosts?

The whole point of /etc/sysconfig/readonly-root is to be able to have a read-only root (which contains /var) and some systemd service sets up the appropriate directories, which need to be writable, as tmpfs.

This bug is about gssproxy having a fixed default socket path in /var/lib/gssproxy/ . But no mechanism to actually create that directory as tmpfs OR provide the possibility to override this default. (You could edit /lib/systemd/system/gssproxy.service but this will be overridden by updates.)

You can create a /etc/systemd/system/gssproxy.service that adds the options you like for your environment, no need to touch system files, and no need for overcomplicated sysconfig stuff given creating a custom service file that takes precendece on the system one has the same complexity.

A lot of deamons use /var/lib for persistent databases and other data that should not be lost on reboot, so I do not see read-only /var as a good reason to modify gssproxy's defaults.

Alternatively if you think you can lose that data then you can mount /var on a tmpfs I guess.

As far as my knowledge goes, the correct way to override some service in systemd is to use "systemctl edit". But in the case of gssproxy (and every other systemd service where Type != oneshot) such editing of "ExecStart" causes systemd to loose this service completely.

You can follow this and some other experiments in customer support case 02021860.

My use case is that one host serves as a source for an OS image for an HPC cluster where the compute hosts are stateless. This hosts' disk is rsync'd to a bunch of NFS servers. Each NFS server exports one NFS share that is used by ca. 200 compute hosts as read-only / . It is only systemd-tmpfiles-setup.service and gssproxy.service that fail. systemd-tmpfiles-setup failes due to a symlink that cannot be created, gssproxy fails due to the impossibility to configure the arguments of ExecStart. Hence the proposal to create a /etc/sysconfig/gssproxy .

  Can a /etc/rwtab.d/gssproxy be created?

  This would make this issue not a (possible) FAQ, and follow the pattern
of other packages (that are in /etc/rwtab) that need a writable /var on
a read-only root mount.

  The "default.socket" needs to be created, and there are 2 directories
under /var/lib/gssproxy.

  A /etc/sysconfig/gssproxy would also be helpful, as well as the
systemd service using it, to specify an alternate socket.

(In reply to Paulo Andrade from comment #13)
>   Can a /etc/rwtab.d/gssproxy be created?

It could, but this is just dealing with consequences, not the source of the issue itself. IMHO this is not the correct long-term way how to fix this.

>   This would make this issue not a (possible) FAQ, and follow the pattern
> of other packages (that are in /etc/rwtab) that need a writable /var on
> a read-only root mount.

We will remove the /etc/rwtab at some point in the future. We have to stop overloading the initscripts and fix the issues where they really come from.

>   A /etc/sysconfig/gssproxy would also be helpful, as well as the
> systemd service using it, to specify an alternate socket.
Agreed.

/var is a place for variable data; per the FHS, we can depend on it being writable.

If your configuration cannot guarantee this, there are workarounds in comment#11.

As already stated (comment#9) this is not my specific configuration but the configuration that RHEL 7 provides for everyone who uses a read-only NFS share as root filesystem. You still haven't answered the question raised in that comment.

Yet I still think that this is an issue with gssproxy. If you take another look on the comments, your colleagues want to move away from /etc/rwtab.

And for runtime data, like the socket that is the matter of this discussion, you could also use /run, which is a tmpfs, at least in the distributions that I'm aware of. This was already mentioned in comment#3 and would just require changing the compile time option.


If you feel that this is not an issue for gssproxy then please reassign the bug to someone you feel is responsible for it instead of closing as NOTABUG.

(In reply to Uwe Sauter from comment #20)
> If you feel that this is not an issue for gssproxy then please reassign the bug to someone you feel is responsible for it instead of closing as NOTABUG.

NOTABUG means I think it's not a bug, not that I think it's someone else's bug.

I think wanting to mount /var read-only without any decrease in functionality is not reasonable.  We are far from the only system daemon exercising the guarantee that /var is writeable.  To pick some examples, sssd and freeipa will not work properly without writable /var either.

> As already stated (comment#9) this is not my specific configuration but the configuration that RHEL 7 provides for everyone who uses a read-only NFS share as root filesystem. You still haven't answered the question raised in that comment.

If that is the case, then the configuration is wrong and needs to be fixed because it is in violation of FHS, as I have already stated.

Where is this configuration?  I'm happy to go push back on that, but I've never seen it (I don't work on NFS, after all).

> Yet I still think that this is an issue with gssproxy. If you take another look on the comments, your colleagues want to move away from /etc/rwtab.

I don't think creating an entry in /etc/rwtab is the right answer either, though of course I don't control that file.  I don't think mounting /var read-only is a reasonable thing to do.  If someone wants to do it anyway, they're welcome to follow the workarounds already presented - either make /var a tmpfs and take the caching hit, or edit the unit file.

> And for runtime data, like the socket that is the matter of this discussion, you could also use /run, which is a tmpfs, at least in the distributions that I'm aware of.

It's not just a socket; that's just what notices first.  We store credential caches as well.  These are all expected to persist beyond a reboot because they are useful beyond a reboot.  This is why we didn't use a tmpfs in the first place, but rather used the FHS specified directory heirarchy for this kind of data.

> This was already mentioned in comment#3 and would just require changing the compile time option.

As stated in comment#11, there's no need to change the compile option if you want it in a different place.

> If that is the case, then the configuration is wrong and needs to be fixed because it is in violation of FHS, as I have already stated.
> 
> Where is this configuration?  I'm happy to go push back on that, but I've never seen it (I don't work on NFS, after all).

This is not really NFS related but related to the mechanism that RHEL 7 provides for stateless operation. In my case, NFS is just used to share the read-only root to several clients.

https://access.redhat.com/solutions/2196881

And more RHEL 7 documentation regarding diskless clients:

https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html-single/storage_administration_guide/index#ch-disklesssystems

Especially this part is about NFS:

https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html-single/storage_administration_guide/index#diskless-nfs-config

The mechanism to provide a stateless client uses at least two systemd services to set up additional tmpfs mountpoints where writeable paths are required:

rhel-readonly.service
systemd-tmpfiles-setup.service

systemd-tmpfiles-setup.service uses /etc/rwtab (/etc/rwtab.d/*) as configuration.

> To pick some examples, sssd and freeipa will not work properly without writable /var either.

And here the output of mount of a stateless, diskless system:

# mount 
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
devtmpfs on /dev type devtmpfs (rw,nosuid,size=57673256k,nr_inodes=14418314,mode=755)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpuacct,cpu)
cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_prio,net_cls)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
configfs on /sys/kernel/config type configfs (rw,relatime)
10.129.200.50:/ on / type nfs (ro,nosuid,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,nolock,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.129.200.50,mountvers=3,mountport=892,mountproto=tcp,local_lock=all,addr=10.129.200.50)
rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=27539)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime)
mqueue on /dev/mqueue type mqueue (rw,relatime)
nfsd on /proc/fs/nfsd type nfsd (rw,relatime)
tmpfs on /tmp type tmpfs (rw)
none on /var/lib/stateless/writable type tmpfs (rw,relatime)
none on /var/cache/man type tmpfs (rw,relatime)
none on /var/lib/xkb type tmpfs (rw,relatime)
none on /var/log type tmpfs (rw,relatime)
none on /var/lib/dbus type tmpfs (rw,relatime)
none on /tmp type tmpfs (rw,relatime)
none on /var/cache/httpd/ssl type tmpfs (rw,relatime)
none on /var/cache/httpd/proxy type tmpfs (rw,relatime)
none on /var/lib/dav type tmpfs (rw,relatime)
none on /var/lib/dhclient type tmpfs (rw,relatime)
none on /var/lib/pulse type tmpfs (rw,relatime)
none on /var/lib/systemd/timers type tmpfs (rw,relatime)
none on /var/tmp type tmpfs (rw,relatime)
none on /etc/adjtime type tmpfs (rw,relatime)
none on /etc/ntp.conf type tmpfs (rw,relatime)
none on /etc/resolv.conf type tmpfs (rw,relatime)
none on /etc/lvm/cache type tmpfs (rw,relatime)
none on /etc/lvm/archive type tmpfs (rw,relatime)
none on /etc/lvm/backup type tmpfs (rw,relatime)
none on /var/account type tmpfs (rw,relatime)
none on /var/lib/NetworkManager type tmpfs (rw,relatime)
none on /var/lib/gdm type tmpfs (rw,relatime)
none on /var/lib/iscsi type tmpfs (rw,relatime)
none on /var/lib/ntp type tmpfs (rw,relatime)
none on /var/lib/systemd/random-seed type tmpfs (rw,relatime)
none on /var/spool type tmpfs (rw,relatime)
none on /var/lib/samba type tmpfs (rw,relatime)
none on /var/lib/nfs type tmpfs (rw,relatime)
none on /var/lib/logrotate type tmpfs (rw,relatime)
none on /var/lib/rsyslog type tmpfs (rw,relatime)
none on /var/lib/sss type tmpfs (rw,relatime)
none on /etc/sysconfig/network-scripts type tmpfs (rw,relatime)
none on /var/lib/dhclient type tmpfs (rw,relatime)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime)


As you can see, there are several tmpfs mountpoints that are unusual as they do not exist on a statefull system, e.g. /var/lib/sss.

This is because sssd-common contains /etc/rwtab.d/sssd:

# cat /etc/rwtab.d/sssd
dirs /var/lib/sss


Now you're saying that gssproxy keeps data in /var/lib/gssproxy between boots. Knowing this I can understand why you wouldn't want to use the same mechanism because then that data wouldn't survive reboots.
But then there should be a mechanism that gives the user the ability to configure that directory. In case of a stateless system it is absoultely ok that this data doesn't survive reboots.

As stated in comment#12 editing the systemd unit to add the appropriate option causes a total loss of the gssproxy service.

Uwe, do I understand it correctly that an entry in rwtab provided by the gssproxy package would meet the needs ?
Robbie are we ok adding such an entry ?

As far as I understand now is that this would have the following consequences:

- systemd-tmpfiles-setup.service would mount a tmpfs at the provided path (compare to sssd)
- for a stateless system, the gssproxy service would start successfully
- for a stateful system, this would prevent persistent caching of credentials

So no, I don't think that an entry in rwtab would be a solution to this, though this might be a workaround for the stateless case.

From my POV the best fix to the problem would be

/etc/sysconfig/gssproxy

that is used in the gssproxy.service unit file. See comments 2, 5, & 7.

If people are ok putting sssd databases on tmpfs I guess gssproxy temporary credentials are fine there too.

Robbie, up to you, but I vote for rwtab

Fine, rwtab it is.

Verified
gssproxy-0.7.0-26.el7.x86_64

1) Machine set up with read-only root
[root@bender ~]# cat /proc/mounts | grep root
rootfs / rootfs rw 0 0
/dev/mapper/rhel_bender-root / xfs ro,relatime,attr2,inode64,noquota 0 0


2)Verifying gssproxy starts
[root@bender ~]# systemctl status gssproxy -l
● gssproxy.service - GSSAPI Proxy Daemon
   Loaded: loaded (/usr/lib/systemd/system/gssproxy.service; disabled; vendor preset: disabled)
   Active: inactive (dead) since Tue 2019-05-14 07:40:03 EDT; 2s ago
  Process: 1374 ExecStart=/usr/sbin/gssproxy -D (code=exited, status=0/SUCCESS)
 Main PID: 1378 (code=exited, status=0/SUCCESS)

May 14 07:26:03 bender.testrelm0513.test systemd[1]: Starting GSSAPI Proxy Daemon...
May 14 07:26:03 bender.testrelm0513.test systemd[1]: Started GSSAPI Proxy Daemon.
May 14 07:40:03 bender.testrelm0513.test systemd[1]: Stopping GSSAPI Proxy Daemon...
May 14 07:40:03 bender.testrelm0513.test systemd[1]: Stopped GSSAPI Proxy Daemon.

[root@bender ~]# systemctl start gssproxy

[root@bender ~]# systemctl status gssproxy -l
● gssproxy.service - GSSAPI Proxy Daemon
   Loaded: loaded (/usr/lib/systemd/system/gssproxy.service; disabled; vendor preset: disabled)
   Active: active (running) since Tue 2019-05-14 07:40:17 EDT; 3s ago
  Process: 2298 ExecStart=/usr/sbin/gssproxy -D (code=exited, status=0/SUCCESS)
 Main PID: 2299 (gssproxy)
   CGroup: /system.slice/gssproxy.service
           └─2299 /usr/sbin/gssproxy -D

May 14 07:40:17 bender.testrelm0513.test systemd[1]: Starting GSSAPI Proxy Daemon...
May 14 07:40:17 bender.testrelm0513.test systemd[1]: Started GSSAPI Proxy Daemon.


Based on the above observation, marking the bug VERIFIED

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:2050