Bug 1334881

Edit: Actual and Expected results should be reversed.

Can I ask why do you do that? /var/tmp should not be cleaned but /tmp is. This could break  daemons which put stuff to /var and expect that the data stay there.

We had a similar complaint from systemd-tmpfiles on a system where some of the directories under /var have been moved to a tmpfs to reduce read/write cycles on an SSD. Why is this bothering systemd-tmpfiles?

Note: this problem persists in 7.3.

To clarify, this is happening when systemd-tmpfiles-clean runs. I see this in the journalctl output:

Jan 05 12:51:56 ti133 systemd[1]: Starting Cleanup of Temporary Directories...
Jan 05 12:51:56 ti133 systemd-tmpfiles[6031]: Failed to open directory /var/tmp: Too many levels of symbolic link

In my case, /var/tmp is a symbolic link to a directory on tmpfs. I have no idea why it returns ELOOP, since there is only a single symbolic link. Looking in the code, it seems that opendir_nomod is failing with ELOOP. 
The opendir_nomod function calls xopendirat_nomod(AT_FDCWD, path),
and xopendirat_nomod calls xopendirat(dirfd, path, O_NOFOLLOW|O_NOATIME),
and xopendirat finally calls openat(fd, name, O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC|flags, 0);

So the basic problem here is that tmpfiles.c:xopendirat_nomod is passing the O_NOFOLLOW flag down to openat. Why is that flag being passed? The obvious patch is to remove it, but perhaps that would break something else...

Regards,
Andy

This error occurs every 24 hours when systemd-tmpfiles-clean.timer pops. How do we get this fixed? I think my usage case is legitimate, although I'm not sure about the OP's decision to point /var/tmp to /tmp.

Jan 08 11:14:37 ti133 systemd-tmpfiles[2092]: Failed to open directory /var/tmp: Too many levels of symbolic links
Jan 09 11:15:37 ti133 systemd-tmpfiles[4719]: Failed to open directory /var/tmp: Too many levels of symbolic links
Jan 10 11:16:37 ti133 systemd-tmpfiles[15367]: Failed to open directory /var/tmp: Too many levels of symbolic links

sh-4.2$ ls -ld /var/tmp
lrwxrwxrwx. 1 root root 19 Jan  5 12:08 /var/tmp -> impermanent/var/tmp

sh-4.2$ df /var/tmp
Filesystem     1K-blocks  Used Available Use% Mounted on
tmpfs            1535268 49624   1485644   4% /var/impermanent

Do I need to open a separate bug report for this, since my situation is slightly different? In my case, /var/impermanent is a tmpfs that is saved on system shutdown and repopulated at bootup. So I achieve persistence with fewer read/write cycles on my SSD.

Thanks,
Andy

(In reply to Andrew J. Schorr from comment #12)
> Do I need to open a separate bug report for this, since my situation is
> slightly different? In my case, /var/impermanent is a tmpfs that is saved on
> system shutdown and repopulated at bootup. So I achieve persistence with
> fewer read/write cycles on my SSD.
> 
> Thanks,
> Andy

Hi Andy,
  I wanted to drop you a quick update on this and thank you for the data you've submitted to this bugzilla.  It's great to see the analysis you've done and know your use case and requirements for making a change like this - it helps a lot, thank you!  For what it's worth, I'm also working with a large strategic Red Hat customer who is seeing this same issue under a different use case, so know that you're not the only one hitting this.  I'm working on getting engineering's assistance with identifying the root cause and then a patch that would resolve this.  Until I hear back from engineering let's keep any additional updates in this bugzilla and not open a new one until we know can confirm that this is the same issue or if you're seeing a new unique issue.  Once there's more progress to report we'll update this publicly so you'll have an idea of what we find and what our next steps are.

Andrew Sanders
Red Hat Inc.
Senior Technical Account Manager

Good morning,

The reason for the O_NOFOLLOW flag is due to the following upstream commit:

	commit a247755d5221411ffe7780677b270963682dadd1
	Author: Lennart Poettering <lennart>
	Date:   Tue Dec 28 14:20:21 2010 +0100

	    tmpfiles: ensure we do no follow symlinks when cleaning up dirs
	    
	    Patch suggested by Miloslav Trmac.


This is likely to avoid conditions where tmpfiles cleans up unexpected directories due to a stray symlink.

Such as:

    d /var/tmp/dir 0755 root root 1d

Where:

    /var/tmp/dir -> /application/data/


In the above instance, the administrator creating the tmpfiles cleanup rule would likely be unaware that the cleanup location would be /application/data.

That being said, I believe the issue can be avoided for the tmpfs use-case simply by setting the tmpfs mount point to the /var/tmp directory itself.

    # grep var\/tmp /etc/fstab 
    nodev /var/tmp tmpfs defaults 0 0


Is there a characteristic of the environment that renders the above solution infeasible?

- Kyle Walker

Hi,

Thanks for the further info. I think such a solution is possible, although it will require reworking a number of system administration scripts related to this
configuration scenario.

Regards,
Andy

Based on the feedback given from impacted users, I am currently closing this as NOTABUG. The issue can be avoided in a number of ways, and the alternative of allowing tmpfiles to follow symlinks introduces a number of possible unexpected results.

- Kyle Walker

Hello Kyle, I believe this should be closed as WONTFIX, not NOTABUG. It's a bug and a conscious choice is being made to not fix it (no arguments) due to impact and available workarounds. The symlink action is technically viable and against no "rules" of Linux or UNIX in general, it's introduced by the systemd code design not being able to handle the symlink correctly.