Those directories are created each time an service that has PrivateTmp=yes set is started on a fresh boot since it creates a directory with systemd-private-<BootID>-<service-with-PrivateTmp-set>-<uniqid>

Lennart and or Zbyszek needs to clarify on whether creating directory's in two places is intended ( both /var/tmp and /tmp ) since this happens with systemd components as well ( which rules out this being bugs in other applications ) and why the previous boot directories are not being deleted upon reboot ( which would happen automatically if this was placed in /tmp not /var/tmp ) and or restart of the unit containing PrivateTmp=yes.

This behavior is confirmed on systemd-218-1.fc22.x86_64 as well.

If package maintainers would have actually implemented Dan Walsh's Privatetmp feature ( as opposed to him simply flagging it 100% done hurray for working feature process )  we would have massive problem on our hands due to this behavior (atleast angry lynchmob of sysadmins with torches )

Arguably /tmp and /var/tmp should be split and PrivateVarTmp=yes introduced since current behavior is silly ( creating both ) if the intent is to use either /tmp or /var/tmp and looking at [¹] 

"int setup_tmpdirs( .." needs to be fixed as well as "if (private_tmp) {.."


1. http://cgit.freedesktop.org/systemd/systemd/tree/src/core/namespace.c?id=d8c9d3a468e61ee2a2b2c3454e662398b0885411

(In reply to Jóhann B. Guðmundsson from comment #1)
> Those directories are created each time an service that has PrivateTmp=yes
> set is started on a fresh boot since it creates a directory with
> systemd-private-<BootID>-<service-with-PrivateTmp-set>-<uniqid>
Yes.

> Lennart and or Zbyszek needs to clarify on whether creating directory's in
> two places is intended ( both /var/tmp and /tmp ) since this happens with
Yes. /var/tmp is disk based, /tmp is not, so two private directories are needed.

> systemd components as well ( which rules out this being bugs in other
> applications ) and why the previous boot directories are not being deleted
> upon reboot ( which would happen automatically if this was placed in /tmp
> not /var/tmp ) and or restart of the unit containing PrivateTmp=yes.
Those directories should be cleaned up by systemd-tmpfiles.service.
Currently, tmp.conf looks like this:

# Clear tmp directories separately, to make them easier to override
d /tmp 1777 root root 10d
d /var/tmp 1777 root root 30d

# Exclude namespace mountpoints created with PrivateTmp=yes
...

So those directories should be cleaned up after 30 days. Richard, do you see any directories older than 30 days? If yes, can you paste the output of systemctl status systemd-tmpfiles-clean.{timer,service}
If no, there's no bug here. You can reduce the time by creating an override file with a lower age.

(In reply to Zbigniew Jędrzejewski-Szmek from comment #4)
> (In reply to Jóhann B. Guðmundsson from comment #1)
> > Those directories are created each time an service that has PrivateTmp=yes
> > set is started on a fresh boot since it creates a directory with
> > systemd-private-<BootID>-<service-with-PrivateTmp-set>-<uniqid>
> Yes.
> 
> > Lennart and or Zbyszek needs to clarify on whether creating directory's in
> > two places is intended ( both /var/tmp and /tmp ) since this happens with
> Yes. /var/tmp is disk based, /tmp is not, so two private directories are
> needed.

I need you to clarify why that is. 

Why we cant have two switches, one for tmp and one for /var/tmp ( disk based tmp or "persistant private tmp ) as opposed to create two directories in two different places which server two different purposes ( surely we are not using both directories that have been created at the same time )

Why those directories are not cleaned up on start up or shutdown  of the service which to me seems to be more efficient that wiping those directories after period of time has passed and is more inline with what administrators expect.
( I restart a service and I get a new privatetmp directory and the old one is deleted as opposed to pile up and be deleted after a period of time ) 

Arguably this is an "design" bug not "functional" bug as in area we could improve in since it does not meet users expectation.

PID 1 should spawn a thread to remove the private directories when the unit is stopped. You can see it being launched when PID 1 has log-level debug:
  Spawning thread to nuke /var/tmp/...
The thing with tmpfiles removing the directories is just a fallback in case something goes wrong with that.

Jóhann, how were you testing this? If those directories are not removed, e.g. during service restart, then this indicates some problem. Can you paste debug output from around such a restart?

(In reply to Jóhann B. Guðmundsson from comment #5)
> Why we cant have two switches, one for tmp and one for /var/tmp ( disk based
> tmp or "persistant private tmp ) as opposed to create two directories in two
> different places which server two different purposes ( surely we are not
> using both directories that have been created at the same time )
I think just for simplicity. If you have two different switches, then some temporary files are protected, and some are not. If the service uses a few different programs, some of them might use one directory, some the other, and it's just easier to have a binary yes/no switch.

> 
> So those directories should be cleaned up after 30 days. Richard, do you see
> any directories older than 30 days? 

I do not have F21 for more than 30 days - but:

# ll -d -t /var/tmp/systemd*|tail
drwx------. 3 root root 4096 Dec 24  2013 /var/tmp/systemd-private-qCmH9c
drwx------. 3 root root 4096 Dec 23  2013 /var/tmp/systemd-private-8tFDSy
drwx------. 3 root root 4096 Dec 23  2013 /var/tmp/systemd-private-vHRbYl
drwx------. 3 root root 4096 Dec 23  2013 /var/tmp/systemd-private-EVacMi
drwx------. 3 root root 4096 Dec 23  2013 /var/tmp/systemd-private-sReKml
drwx------. 3 root root 4096 Dec 23  2013 /var/tmp/systemd-private-oCrc5r
drwx------. 3 root root 4096 Dec 22  2013 /var/tmp/systemd-private-sOaVgP
drwx------. 3 root root 4096 Dec 21  2013 /var/tmp/systemd-private-QPRRQK
drwx------. 3 root root 4096 Dec 20  2013 /var/tmp/systemd-private-3VQYhr
drwx------. 3 root root 4096 Dec 16  2013 /var/tmp/systemd-private-PAGknF

> ... If yes, can you paste the output of
> systemctl status systemd-tmpfiles-clean.{timer,service}

# locate systemd-tmpfiles-clean
/usr/lib/systemd/system/systemd-tmpfiles-clean.service
/usr/lib/systemd/system/systemd-tmpfiles-clean.timer
/usr/lib/systemd/system/timers.target.wants/systemd-tmpfiles-clean.timer
/usr/share/man/man8/systemd-tmpfiles-clean.service.8.gz
/usr/share/man/man8/systemd-tmpfiles-clean.timer.8.gz

# locate systemd-tmpfiles-clean|xargs rpm -qf
systemd-216-14.fc21.i686
systemd-216-14.fc21.i686
systemd-216-14.fc21.i686
systemd-216-14.fc21.i686
systemd-216-14.fc21.i686

# for i in `locate systemd-tmpfiles-clean|grep /usr/lib`; do echo "***** $i"; cat $i; done
***** /usr/lib/systemd/system/systemd-tmpfiles-clean.service
#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it
#  under the terms of the GNU Lesser General Public License as published by
#  the Free Software Foundation; either version 2.1 of the License, or
#  (at your option) any later version.

[Unit]
Description=Cleanup of Temporary Directories
Documentation=man:tmpfiles.d(5) man:systemd-tmpfiles(8)
DefaultDependencies=no
Conflicts=shutdown.target
After=systemd-readahead-collect.service systemd-readahead-replay.service local-fs.target time-sync.target
Before=shutdown.target

[Service]
Type=oneshot
ExecStart=/usr/bin/systemd-tmpfiles --clean
IOSchedulingClass=idle
***** /usr/lib/systemd/system/systemd-tmpfiles-clean.timer
#  This file is part of systemd.
#
#  systemd is free software; you can red# for i in `locate systemd-tmpfiles-clean|grep /usr/lib|xargs`; do echo "***** $i"; cat $i; done
***** /usr/lib/systemd/system/systemd-tmpfiles-clean.service
#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it
#  under the terms of the GNU Lesser General Public License as published by
#  the Free Software Foundation; either version 2.1 of the License, or
#  (at your option) any later version.

[Unit]
Description=Cleanup of Temporary Directories
Documentation=man:tmpfiles.d(5) man:systemd-tmpfiles(8)
DefaultDependencies=no
Conflicts=shutdown.target
After=systemd-readahead-collect.service systemd-readahead-replay.service local-fs.target time-sync.target
Before=shutdown.target

[Service]
Type=oneshot
ExecStart=/usr/bin/systemd-tmpfiles --clean
IOSchedulingClass=idle
***** /usr/lib/systemd/system/systemd-tmpfiles-clean.timer
#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it
#  under the terms of the GNU Lesser General Public License as published by
#  the Free Software Foundation; either version 2.1 of the License, or
#  (at your option) any later version.

[Unit]
Description=Daily Cleanup of Temporary Directories
Documentation=man:tmpfiles.d(5) man:systemd-tmpfiles(8)

[Timer]# for i in `locate systemd-tmpfiles-clean|grep /usr/lib|xargs`; do echo "***** $i"; cat $i; done
***** /usr/lib/systemd/system/systemd-tmpfiles-clean.service
#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it
#  under the terms of the GNU Lesser General Public License as published by
#  the Free Software Foundation; either version 2.1 of the License, or
#  (at your option) any later version.

[Unit]
Description=Cleanup of Temporary Directories
Documentation=man:tmpfiles.d(5) man:systemd-tmpfiles(8)
DefaultDependencies=no
Conflicts=shutdown.target
After=systemd-readahead-collect.service systemd-readahead-replay.service local-fs.target time-sync.target
Before=shutdown.target

[Service]
Type=oneshot
ExecStart=/usr/bin/systemd-tmpfiles --clean
IOSchedulingClass=idle
***** /usr/lib/systemd/system/systemd-tmpfiles-clean.timer
#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it
#  under the terms of the GNU Lesser General Public License as published by
#  the Free Software Foundation; either version 2.1 of the License, or
#  (at your option) any later version.

[Unit]
Description=Daily Cleanup of Temporary Directories
Documentation=man:tmpfiles.d(5) man:systemd-tmpfiles(8)

[Timer]
OnBootSec=15min
OnUnitActiveSec=1d
***** /usr/lib/systemd/system/timers.target.wants/systemd-tmpfiles-clean.timer
#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it
#  under the terms of the GNU Lesser General Public License as published by
#  the Free Software Foundation; either version 2.1 of the License, or
#  (at your option) any later version.

[Unit]
Description=Daily Cleanup of Temporary Directories
Documentation=man:tmpfiles.d(5) man:systemd-tmpfiles(8)

[Timer]
OnBootSec=15min
OnUnitActiveSec=1d

OnBootSec=15min
OnUnitActiveSec=1d
***** /usr/lib/systemd/system/timers.target.wants/systemd-tmpfiles-clean.timer
#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it
#  under the terms of the GNU Lesser General Public License as published by
#  the Free Software Foundation; either version 2.1 of the License, or
#  (at your option) any later version.

[Unit]
Description=Daily Cleanup of Temporary Directories
Documentation=man:tmpfiles.d(5) man:systemd-tmpfiles(8)

[Timer]
OnBootSec=15min
OnUnitActiveSec=1d
istribute it and/or modify it
#  under the terms of the GNU Lesser General Public License as published by
#  the Free Software Foundation; either version 2.1 of the License, or
#  (at your option) any later version.

[Unit]
Description=Daily Cleanup of Temporary Directories
Documentation=man:tmpfiles.d(5) man:systemd-tmpfiles(8)

[Timer]
OnBootSec=15min
OnUnitActiveSec=1d
***** /usr/lib/systemd/system/timers.target.wants/systemd-tmpfiles-clean.timer
#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it
#  under the terms of the GNU Lesser General Public License as published by
#  the Free Software Foundation; either version 2.1 of the License, or
#  (at your option) any later version.

[Unit]
Description=Daily Cleanup of Temporary Directories
Documentation=man:tmpfiles.d(5) man:systemd-tmpfiles(8)

[Timer]
OnBootSec=15min
OnUnitActiveSec=1d

Sorry for overwriting the last comment - hit the wrong button :(((

restoring the comment like this - dont know how to do it better


--- Comment #6 from Zbigniew Jędrzejewski-Szmek <zbyszek.pl> ---
PID 1 should spawn a thread to remove the private directories when the unit is
stopped. You can see it being launched when PID 1 has log-level debug:
  Spawning thread to nuke /var/tmp/...
The thing with tmpfiles removing the directories is just a fallback in case
something goes wrong with that.

Jóhann, how were you testing this? If those directories are not removed, e.g.
during service restart, then this indicates some problem. Can you paste debug
output from around such a restart?

(In reply to Jóhann B. Guðmundsson from comment #5)
> Why we cant have two switches, one for tmp and one for /var/tmp ( disk based
> tmp or "persistant private tmp ) as opposed to create two directories in two
> different places which server two different purposes ( surely we are not
> using both directories that have been created at the same time )
I think just for simplicity. If you have two different switches, then some
temporary files are protected, and some are not. If the service uses a few
different programs, some of them might use one directory, some the other, and
it's just easier to have a binary yes/no switch.

(In reply to Richard Z. from comment #7)
> > 
> > So those directories should be cleaned up after 30 days. Richard, do you see
> > any directories older than 30 days? 
> 
> I do not have F21 for more than 30 days - but:
> 
> # ll -d -t /var/tmp/systemd*|tail
> drwx------. 3 root root 4096 Dec 24  2013 /var/tmp/systemd-private-qCmH9c
> drwx------. 3 root root 4096 Dec 23  2013 /var/tmp/systemd-private-8tFDSy
> drwx------. 3 root root 4096 Dec 23  2013 /var/tmp/systemd-private-vHRbYl
> drwx------. 3 root root 4096 Dec 23  2013 /var/tmp/systemd-private-EVacMi
> drwx------. 3 root root 4096 Dec 23  2013 /var/tmp/systemd-private-sReKml
> drwx------. 3 root root 4096 Dec 23  2013 /var/tmp/systemd-private-oCrc5r
> drwx------. 3 root root 4096 Dec 22  2013 /var/tmp/systemd-private-sOaVgP
> drwx------. 3 root root 4096 Dec 21  2013 /var/tmp/systemd-private-QPRRQK
> drwx------. 3 root root 4096 Dec 20  2013 /var/tmp/systemd-private-3VQYhr
> drwx------. 3 root root 4096 Dec 16  2013 /var/tmp/systemd-private-PAGknF
The important part is truncated here: the end of the directory name tells us what unit the directory is for. Are those all for rtkit-daemon?

> > ... If yes, can you paste the output of
> > systemctl status systemd-tmpfiles-clean.{timer,service}
Try that command, literally, not locate.

(In reply to Zbigniew Jędrzejewski-Szmek from comment #6)
> PID 1 should spawn a thread to remove the private directories when the unit
> is stopped. You can see it being launched when PID 1 has log-level debug:
>   Spawning thread to nuke /var/tmp/...
> The thing with tmpfiles removing the directories is just a fallback in case
> something goes wrong with that.

Then obviously something is going wrong with that since my /var/tmp directory is filled with directories for rtkit-daemon,colord,systemd-hostnamed and systemd-timesyncd
( maybe not being cleaned when the system shutdowns or suspends/resume? ) 

> Jóhann, how were you testing this? If those directories are not removed,
> e.g. during service restart, then this indicates some problem. Can you paste
> debug output from around such a restart?

This works fine on stop/start/restart run manually ( both /tmp /var/tmp systemd-private directories do get cleaned up.

> 
> (In reply to Jóhann B. Guðmundsson from comment #5)
> > Why we cant have two switches, one for tmp and one for /var/tmp ( disk based
> > tmp or "persistant private tmp ) as opposed to create two directories in two
> > different places which server two different purposes ( surely we are not
> > using both directories that have been created at the same time )
> I think just for simplicity. If you have two different switches, then some
> temporary files are protected, and some are not. If the service uses a few
> different programs, some of them might use one directory, some the other,
> and it's just easier to have a binary yes/no switch.

Can you better explain the practicality of this implementation since to me it makes absolutely no sense. 

1. What determines which directory of the two the service/daemon uses?  

2. Does the service/daemon use two directories at once?

3. We create directories with uniq id's so 
   a) service/daemon is incapable of re-using the same directory after reboot
   b) service/daemon is incapable of re-using the same directory after stop/start/restart

4. At what point are we actually creating what would be considered "persistent" tmp data thus requires the retention of it? 

Based on 3. 

we should be able to... 
         a) Have only privatetmp directories created in /tmp and use that as the failsafe to cleanup those directories which would happen upon reboot   
         b) run the equallent of "rm -rf <path>/systemd-private-*-<daemon>.service-*" upon stop/start/restart and drop the related timer units

Even from Factory Reset, Stateless Systems, Reproducible Systems, Verifiable Systems perspective the current implementation of privatetmp and it's cleanup makes no sense
( and highlights at the same time we have not been testing this with that in mind since I'm pretty sure things would have crashed when we try to write to /var/tmp on those systems )

(In reply to Jóhann B. Guðmundsson from comment #11)
> (In reply to Zbigniew Jędrzejewski-Szmek from comment #6)
> Then obviously something is going wrong with that since my /var/tmp
> directory is filled with directories for
> rtkit-daemon,colord,systemd-hostnamed and systemd-timesyncd
> ( maybe not being cleaned when the system shutdowns or suspends/resume? ) 
They should be cleaned up during shutdown, and suspend/resume should not matter at all.
So, are they from the same boot? Are they empty?
 
> > Jóhann, how were you testing this? If those directories are not removed,
> > e.g. during service restart, then this indicates some problem. Can you paste
> > debug output from around such a restart?
> 
> This works fine on stop/start/restart run manually ( both /tmp /var/tmp
> systemd-private directories do get cleaned up.
OK, so this suggest that the problem occurs during shutdown. Maybe some race
with unmounting which prevents removal, dunno.

> 1. What determines which directory of the two the service/daemon uses?  
Both /tmp and /var/tmp are overmounted. The service can use either one as
it wants.

> 3. We create directories with uniq id's so 
>    a) service/daemon is incapable of re-using the same directory after reboot
>    b) service/daemon is incapable of re-using the same directory after
> stop/start/restart
Neither. A unique directory name is used because /tmp and /var/tmp
are shared, and an unpredictable suffix is necessary to prevent DOS attack.
The daemon is constrained by namespaces, and does not know or care where
the directory is actually located.

> 4. At what point are we actually creating what would be considered
> "persistent" tmp data thus requires the retention of it? 
I'm not sure if I understand this question. Normal rules for /tmp and /var/tmp
apply. PID 1 tries to remove those directories when stopping the service because
it know that they are unused. tmpfiles removes them by time according to normal
expiration rules, because it does not know who created and is using them. (If
we said that systemd "owns" the names {/var,}/tmp/systemd-private-*-*.* then
we could set the timeout to 0, since we exclude those from the current boot anyway,
but we don't say that, so the user in principle is free to create their own
directory called /var/tmp/systemd-private-x-y.z and can expect it to remain
on the file system for 30 days.)

> Based on 3. 
> 
> we should be able to... 
>          a) Have only privatetmp directories created in /tmp and use that as
> the failsafe to cleanup those directories which would happen upon reboot   
No, because /var/tmp must be disk based.

>          b) run the equallent of "rm -rf
> <path>/systemd-private-*-<daemon>.service-*" upon stop/start/restart and
> drop the related timer units
No, because we don't take ownership or those names.

(In reply to Jóhann B. Guðmundsson from comment #12)
> Even from Factory Reset, Stateless Systems, Reproducible Systems, Verifiable
> Systems perspective the current implementation of privatetmp and it's
> cleanup makes no sense
> ( and highlights at the same time we have not been testing this with that in
> mind since I'm pretty sure things would have crashed when we try to write to
> /var/tmp on those systems )
Why? /var/tmp should be writable even on stateless systems. Maybe just not persistent.

(In reply to Zbigniew Jędrzejewski-Szmek from comment #13)
> So, are they from the same boot? Are they empty?

Different boots, always the same units, all directories contain a /tmp folder within them which are all empty.

>          b) run the equallent of "rm -rf
> <path>/systemd-private-*-<daemon>.service-*" upon stop/start/restart and
> drop the related timer units

> No, because we don't take ownership or those names 

I dont follow 

we already delete the units current directory upon start/stop/restart so why cant PID1 delete all directories created by the same unit? 

Using systemd-hostnamed as an example that would be

"rm -rf /var/tmp/systemd-private-*-systemd-hostnamed.service-*" 

This would ensure all previously created directories would get wiped when systemd-hostnamed gets stop/started/restated 

And why must /var/tmp be disk based?

(In reply to Jóhann B. Guðmundsson from comment #15)
> (In reply to Zbigniew Jędrzejewski-Szmek from comment #13)
> > So, are they from the same boot? Are they empty?
> 
> Different boots, always the same units, all directories contain a /tmp
> folder within them which are all empty.
So this suggest some race with umounting. I'll take a look at the code.
I would be great if I have a way to reproduce this though.

> >          b) run the equallent of "rm -rf
> > <path>/systemd-private-*-<daemon>.service-*" upon stop/start/restart and
> > drop the related timer units
> 
> > No, because we don't take ownership or those names 
> 
> we already delete the units current directory upon start/stop/restart so why
> cant PID1 delete all directories created by the same unit? 
We delete a directory that we know that we created ourselves.

> Using systemd-hostnamed as an example that would be
> 
> "rm -rf /var/tmp/systemd-private-*-systemd-hostnamed.service-*" 
> 
> This would ensure all previously created directories would get wiped when
> systemd-hostnamed gets stop/started/restated 
Yes. But the administrator is for example allowed to do
  cp -av /var/tmp/systemd-private-XXXXXX-systemd-hostnamed.service-YYYY /var/tmp/systemd-private-XXXXXX-systemd-hostnamed.service-YYYY2

to keep it for later inspection. Removing everything with some glob
would break that.

> And why must /var/tmp be disk based?
Because that's how it's defined. For example large temporary files are downloaded
into /var/tmp.

(In reply to Zbigniew Jędrzejewski-Szmek from comment #16)
> (In reply to Jóhann B. Guðmundsson from comment #15)
> > (In reply to Zbigniew Jędrzejewski-Szmek from comment #13)
> > > So, are they from the same boot? Are they empty?
> > 
> > Different boots, always the same units, all directories contain a /tmp
> > folder within them which are all empty.
> So this suggest some race with umounting. I'll take a look at the code.
> I would be great if I have a way to reproduce this though.
> 
> > >          b) run the equallent of "rm -rf
> > > <path>/systemd-private-*-<daemon>.service-*" upon stop/start/restart and
> > > drop the related timer units
> > 
> > > No, because we don't take ownership or those names 
> > 
> > we already delete the units current directory upon start/stop/restart so why
> > cant PID1 delete all directories created by the same unit? 
> We delete a directory that we know that we created ourselves.
> 
> > Using systemd-hostnamed as an example that would be
> > 
> > "rm -rf /var/tmp/systemd-private-*-systemd-hostnamed.service-*" 
> > 
> > This would ensure all previously created directories would get wiped when
> > systemd-hostnamed gets stop/started/restated 
> Yes. But the administrator is for example allowed to do
>   cp -av /var/tmp/systemd-private-XXXXXX-systemd-hostnamed.service-YYYY
> /var/tmp/systemd-private-XXXXXX-systemd-hostnamed.service-YYYY2
> 
> to keep it for later inspection. Removing everything with some glob
> would break that.

That sounds like an awfully corner case

If you want to keep the directorias for later inspection you dont delete or clean them up et all 

Any real administrator in a real environment already is backing up his filesystem and bottom line these directories aren't being retained so not doing that makes no sense and we simply document "Dont do that". 

Copy those files into a directory under a new name 

> 
> > And why must /var/tmp be disk based?
> Because that's how it's defined. For example large temporary files are
> downloaded
> into /var/tmp.

Yeah sorry I knew that but was thinking about the systemd-private-* directories. 

The logic why we must create systemd-private directory in two places still eludes me.

(In reply to Jóhann B. Guðmundsson from comment #17)
> (In reply to Zbigniew Jędrzejewski-Szmek from comment #16)
> > (In reply to Jóhann B. Guðmundsson from comment #15)
> > > (In reply to Zbigniew Jędrzejewski-Szmek from comment #13)
> > > > So, are they from the same boot? Are they empty?
> > > 
> > > Different boots, always the same units, all directories contain a /tmp
> > > folder within them which are all empty.
> > So this suggest some race with umounting. I'll take a look at the code.
> > I would be great if I have a way to reproduce this though.
> > 
> > > >          b) run the equallent of "rm -rf
> > > > <path>/systemd-private-*-<daemon>.service-*" upon stop/start/restart and
> > > > drop the related timer units
> > > 
> > > > No, because we don't take ownership or those names 
> > > 
> > > we already delete the units current directory upon start/stop/restart so why
> > > cant PID1 delete all directories created by the same unit? 
> > We delete a directory that we know that we created ourselves.
> > 
> > > Using systemd-hostnamed as an example that would be
> > > 
> > > "rm -rf /var/tmp/systemd-private-*-systemd-hostnamed.service-*" 
> > > 
> > > This would ensure all previously created directories would get wiped when
> > > systemd-hostnamed gets stop/started/restated 
> > Yes. But the administrator is for example allowed to do
> >   cp -av /var/tmp/systemd-private-XXXXXX-systemd-hostnamed.service-YYYY
> > /var/tmp/systemd-private-XXXXXX-systemd-hostnamed.service-YYYY2
> > 
> > to keep it for later inspection. Removing everything with some glob
> > would break that.
> 
> That sounds like an awfully corner case
> 
> If you want to keep the directorias for later inspection you dont delete or
> clean them up et all 

It's cleaner and faster for systemd to only delete the directory it knows it has created and is not needed anymore. And this should work. Using a glob in systemd to delete directories would be a workaround for some problem with the current code.
 
> Copy those files into a directory under a new name
Exactly. Once you have a made a copy, even with a very similar name, even in /var/tmp or /tmp, you can expect your copy to stay. This is what I was trying to say.

> > > And why must /var/tmp be disk based?
> > Because that's how it's defined. For example large temporary files are
> > downloaded
> > into /var/tmp.
> 
> Yeah sorry I knew that but was thinking about the systemd-private-*
> directories. 
> 
> The logic why we must create systemd-private directory in two places still
> eludes me.
Because the private directories follow the same rules. A service can expect 
/var/tmp to be disk based and dump a large blob in there. It can also
expect /tmp to be tmp-based and fast.

Just an fyi if an administrator want's to inspect units tmp directory for whatever reason in 99.9% times that he does that he stops the running service/daemon first then wants to inspect the direction not "copy" it let alone lone copy it under a directory that bares the exact same name just with 2 behind it. ( and when he does we have already deleted that directory )

Anyway let's just agree we disagree how this is and should be handled. 

I comment with feedback once I find out the reproducible rigger that results in leaving those directories behind

Ok so I did massive testing both from the cmd as well using Gnome's shutdown/reboot/Suspend and everything worked as to be expected in all of them. 

The only way I manage to reliable reproduce this is when I forcefully "powered off" ( which is to be expected that those directories will exist after that and I myself have experience the bootup getting stuck when the laptop has been docked on few occasion which most likely explains the existence of those directory's ).

Richard can you confirm the existence of those directories on your computer might be after such an thing?

you have either manually held the power off button ( due to for example things might have gotten stuck on boot/shutdown or the system froze ) or you have experience power outage which might explain this?

(In reply to Zbigniew Jędrzejewski-Szmek from comment #10)
> (In reply to Richard Z. from comment #7)

> > # ll -d -t /var/tmp/systemd*|tail
> > drwx------. 3 root root 4096 Dec 24  2013 /var/tmp/systemd-private-qCmH9c
> > drwx------. 3 root root 4096 Dec 23  2013 /var/tmp/systemd-private-8tFDSy
> > drwx------. 3 root root 4096 Dec 23  2013 /var/tmp/systemd-private-vHRbYl
> > drwx------. 3 root root 4096 Dec 23  2013 /var/tmp/systemd-private-EVacMi
> > drwx------. 3 root root 4096 Dec 23  2013 /var/tmp/systemd-private-sReKml
> > drwx------. 3 root root 4096 Dec 23  2013 /var/tmp/systemd-private-oCrc5r
> > drwx------. 3 root root 4096 Dec 22  2013 /var/tmp/systemd-private-sOaVgP
> > drwx------. 3 root root 4096 Dec 21  2013 /var/tmp/systemd-private-QPRRQK
> > drwx------. 3 root root 4096 Dec 20  2013 /var/tmp/systemd-private-3VQYhr
> > drwx------. 3 root root 4096 Dec 16  2013 /var/tmp/systemd-private-PAGknF
> The important part is truncated here: the end of the directory name tells us
> what unit the directory is for. Are those all for rtkit-daemon?

Nothing truncated here, those are the real names. They are leftovers from previous Fedora versions, you can tell that by the date.

> > > ... If yes, can you paste the output of
> > > systemctl status systemd-tmpfiles-clean.{timer,service}
> Try that command, literally, not locate.

sorry, did not realize this is a command.

]# systemctl status -l systemd-tmpfiles-clean.{timer,service}
● systemd-tmpfiles-clean.timer - Daily Cleanup of Temporary Directories
   Loaded: loaded (/usr/lib/systemd/system/systemd-tmpfiles-clean.timer; static)
   Active: active (waiting) since Thu 2015-01-22 17:44:47 CET; 2h 49min ago
     Docs: man:tmpfiles.d(5)
           man:systemd-tmpfiles(8)

Jan 22 17:44:47 rz.localhost.localdomain systemd[1]: Starting Daily Cleanup of Temporary Directories.
Jan 22 17:44:47 rz.localhost.localdomain systemd[1]: Started Daily Cleanup of Temporary Directories.

● systemd-tmpfiles-clean.service - Cleanup of Temporary Directories
   Loaded: loaded (/usr/lib/systemd/system/systemd-tmpfiles-clean.service; static)
   Active: inactive (dead) since Thu 2015-01-22 17:58:27 CET; 2h 35min ago
     Docs: man:tmpfiles.d(5)
           man:systemd-tmpfiles(8)
 Main PID: 3840 (code=exited, status=0/SUCCESS)

Jan 22 17:58:27 rz.localhost.localdomain systemd[1]: Started Cleanup of Temporary Directories.

(In reply to Jóhann B. Guðmundsson from comment #20)
> Ok so I did massive testing both from the cmd as well using Gnome's
> shutdown/reboot/Suspend and everything worked as to be expected in all of
> them. 
> 
> The only way I manage to reliable reproduce this is when I forcefully
> "powered off" ( which is to be expected that those directories will exist
> after that and I myself have experience the bootup getting stuck when the
> laptop has been docked on few occasion which most likely explains the
> existence of those directory's ).
> 
> Richard can you confirm the existence of those directories on your computer
> might be after such an thing?

quite possible, looked at the logs around one of the creation dates and apparently I sent the system to sleep and it did not wake up.

(In reply to Zbigniew Jędrzejewski-Szmek from comment #4)

> So those directories should be cleaned up after 30 days. Richard, do you see
> any directories older than 30 days? If yes, can you paste the output of
> systemctl status systemd-tmpfiles-clean.{timer,service}
> If no, there's no bug here. You can reduce the time by creating an override
> file with a lower age.

just wondering - is it safe to remove any of them considering only file age? I mean some process may well run longer than 30 days.

Richard, can you show the output of
  find /var/tmp/systemd-private-qCmH9c | xargs stat
(for any one of those old directories)?

(In reply to Zbigniew Jędrzejewski-Szmek from comment #24)
> Richard, can you show the output of
>   find /var/tmp/systemd-private-qCmH9c | xargs stat
> (for any one of those old directories)?

#  find /var/tmp/systemd-private-qCmH9c | xargs stat
  File: ‘/var/tmp/systemd-private-qCmH9c’
  Size: 4096            Blocks: 8          IO Block: 4096   directory
Device: fd05h/64773d    Inode: 1056095     Links: 3
Access: (0700/drwx------)  Uid: (    0/    root)   Gid: (    0/    root)
Context: system_u:object_r:tmp_t:s0
Access: 2015-01-22 18:53:18.222628572 +0100
Modify: 2013-12-24 12:18:25.505126972 +0100
Change: 2013-12-24 12:18:25.505126972 +0100
 Birth: -
  File: ‘/var/tmp/systemd-private-qCmH9c/tmp’
  Size: 4096            Blocks: 8          IO Block: 4096   directory
Device: fd05h/64773d    Inode: 1056323     Links: 2
Access: (1777/drwxrwxrwt)  Uid: (    0/    root)   Gid: (    0/    root)
Context: system_u:object_r:tmp_t:s0
Access: 2015-01-23 14:42:31.314849279 +0100
Modify: 2013-12-24 12:18:25.505126972 +0100
Change: 2013-12-24 12:18:25.505126972 +0100
 Birth: -

[I wrote this comment before the previous one, but left it in an open browser
window. I'm adding it now, since it answers, I think, all other questions
apart from the main issue of the bug.]

(In reply to Richard Z. from comment #21)
> (In reply to Zbigniew Jędrzejewski-Szmek from comment #10)
> > (In reply to Richard Z. from comment #7)
> 
> > > # ll -d -t /var/tmp/systemd*|tail
> > > drwx------. 3 root root 4096 Dec 24  2013 /var/tmp/systemd-private-qCmH9c
> > > drwx------. 3 root root 4096 Dec 23  2013 /var/tmp/systemd-private-8tFDSy
> > > drwx------. 3 root root 4096 Dec 23  2013 /var/tmp/systemd-private-vHRbYl
> > > drwx------. 3 root root 4096 Dec 23  2013 /var/tmp/systemd-private-EVacMi
> > > drwx------. 3 root root 4096 Dec 23  2013 /var/tmp/systemd-private-sReKml
> > > drwx------. 3 root root 4096 Dec 23  2013 /var/tmp/systemd-private-oCrc5r
> > > drwx------. 3 root root 4096 Dec 22  2013 /var/tmp/systemd-private-sOaVgP
> > > drwx------. 3 root root 4096 Dec 21  2013 /var/tmp/systemd-private-QPRRQK
> > > drwx------. 3 root root 4096 Dec 20  2013 /var/tmp/systemd-private-3VQYhr
> > > drwx------. 3 root root 4096 Dec 16  2013 /var/tmp/systemd-private-PAGknF
> > The important part is truncated here: the end of the directory name tells us
> > what unit the directory is for. Are those all for rtkit-daemon?
> 
> Nothing truncated here, those are the real names. They are leftovers from
> previous Fedora versions, you can tell that by the date.
You're right, I should have looked at the date. This means that those directories
are not removed for some reason. 

>    Active: inactive (dead) since Thu 2015-01-22 17:58:27 CET; 2h 35min ago

systemd-tmpfiles-clean.service ran 2 hours ago, and it should have removed them.
So now we're getting somewhere.

(In reply to Richard Z. from comment #22)
> (In reply to Jóhann B. Guðmundsson from comment #20)
> quite possible, looked at the logs around one of the creation dates and
> apparently I sent the system to sleep and it did not wake up.
How they were created does not matter that much anymore. They should be removed by age.

(In reply to Richard Z. from comment #23)
> just wondering - is it safe to remove any of them considering only file age?
> I mean some process may well run longer than 30 days.
That's why directories with current boot id in the name are excepted: they might
correspond to running daemons and they cannot be removed. (Look at the x and X
patterns in tmp.conf).

So the question now is why those directories are not removed.

(In reply to Zbigniew Jędrzejewski-Szmek from comment #26)
> So the question now is why those directories are not removed.

are there some debugging options to turn on? Or can I run it manually with strace?

No, I think I found the problem. Doing stat(2) on a file or directory updates the atime of the parent directory. So whenever systemd-tmpfiles does its sweep, it'll bump atime. systemd-tmpfiles also includes atime in the age check, so essentially, running systemd-tmpfiles regularly prevents it from cleaning up.
systemd-tmpfiles takes some steps to avoid updating timestamps when doing it's sweep, but they don't work.

ok, btw assuming that the feature gets more widely used - would it be possible to hide the systemd-* directories in a topdir so that /tmp and /var/tmp isn't clutterded as much?

(In reply to Richard Z. from comment #29)
> ok, btw assuming that the feature gets more widely used - would it be
> possible to hide the systemd-* directories in a topdir so that /tmp and
> /var/tmp isn't clutterded as much?
No, because /var/tmp and /tmp and there for the purpose of holding temporary files. Also note that those directories are there only because of a crash, normally they shouldn't accumulate.

Perhaps I didnt express myself well, I did mean something like /vart/tmp/systemd-top/systemd-private-* and the corresponding in /tmp.

Even if there might be a dozen entries it would be nice to have them out of view.

(In reply to Richard Z. from comment #31)
> Perhaps I didnt express myself well, I did mean something like
> /vart/tmp/systemd-top/systemd-private-* and the corresponding in /tmp.
It could be done, but it would be papering over a bug. Right now it is fairly simply to find a private temporary directory because of the consistent naming.
 
Anyway, I think I fixed the bug (http://cgit.freedesktop.org/systemd/systemd/commit/?id=df99a9ef5b). This was one of those tricky and fun ones.

This message is a reminder that Fedora 21 is nearing its end of life.
Approximately 4 (four) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 21. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as EOL if it remains open with a Fedora  'version'
of '21'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 21 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.