Bug 955379 - systemctl enable fails for symlinks in /etc/systemd/system
Summary: systemctl enable fails for symlinks in /etc/systemd/system
Keywords:
Status: CLOSED NEXTRELEASE
Alias: None
Product: Fedora
Classification: Fedora
Component: systemd
Version: rawhide
Hardware: x86_64
OS: Linux
unspecified
medium
Target Milestone: ---
Assignee: Zbigniew Jędrzejewski-Szmek
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
: 887140 896670 (view as bug list)
Depends On:
Blocks: 1158576 1159308
TreeView+ depends on / blocked
 
Reported: 2013-04-22 22:44 UTC by Andrew McNabb
Modified: 2017-05-25 13:00 UTC (History)
25 users (show)

Fixed In Version: systemd-231-1.fc25.x86_64
Doc Type: Bug Fix
Doc Text:
Clone Of:
: 1159308 (view as bug list)
Environment:
Last Closed: 2016-08-04 04:34:37 UTC


Attachments (Terms of Use)
unit file for service (330 bytes, text/plain)
2013-04-23 15:06 UTC, Andrew McNabb
no flags Details
strace output (17.02 KB, text/plain)
2013-04-23 15:41 UTC, Andrew McNabb
no flags Details
systemd unit files are symlinks (28.29 KB, text/plain)
2014-10-22 14:26 UTC, Domen Kožar
no flags Details

Description Andrew McNabb 2013-04-22 22:44:12 UTC
If I create a custom service file at, say, `/etc/systemd/system/xyz.service` and then run `systemctl enable xyz.service`, it fails with the error:

"Failed to issue method call: No such file or directory"

It seems that systemd is trying to find this service at `/usr/lib/systemd/system/xyz.service`, but from the documentation, it sounds like `/etc/systemd/system` is the right place for system administrators to place custom unit files. The systemd documentation states:

"Unit files stored in /etc/systemd/system override those from /lib/systemd/system that carry the same name. The former directory is administrator territory, the latter terroritory of your package manager." (from http://0pointer.de/blog/projects/three-levels-of-off.html)

(It's also probably a bug that there's a "No such file or directory" message that doesn't identify the path of the missing file or directory)

Comment 1 Harald Hoyer 2013-04-23 06:18:33 UTC
Did you run:

# systemctl daemon-reload
before
# systemctl enable xyz.service

??

Comment 2 Michal Schmidt 2013-04-23 08:27:02 UTC
daemon-reload should not be necessary in this case (since "systemctl enable" has to operate on new unit files that systemd does not yet know anything about, and "systemctl daemon-reload" will reload only the units that have been loaded before).

Anyway, I am unable to reproduce the bug. What is the exact version of systemd and could you paste the xyz.service unit file here (in case its contents matter).

Comment 3 Andrew McNabb 2013-04-23 15:06:34 UTC
Created attachment 739058 [details]
unit file for service

(In reply to comment #1)
> Did you run:
> 
> # systemctl daemon-reload
> before
> # systemctl enable xyz.service

Yes, I made sure to do daemon-reload.

(In reply to comment #2)
> daemon-reload should not be necessary in this case (since "systemctl enable"
> has to operate on new unit files that systemd does not yet know anything
> about, and "systemctl daemon-reload" will reload only the units that have
> been loaded before).
> 
> Anyway, I am unable to reproduce the bug. What is the exact version of
> systemd and could you paste the xyz.service unit file here (in case its
> contents matter).

systemd-197-1.fc18.2.x86_64

I am attaching the unit file. Hopefully, I didn't do anything particularly stupid. :)

Comment 4 Michal Schmidt 2013-04-23 15:21:55 UTC
There is no [Install] section, so systemctl enable won't work. But the error message should be completely different. This is how it looks on my F18 system with the same systemd version:

# cp topical-guide-setup.service /etc/systemd/system
# systemctl enable topical-guide-setup.service
The unit files have no [Install] section. They are not meant to be enabled
using systemctl.
Possible reasons for having this kind of units are:
1) A unit may be statically enabled by being symlinked from another unit's
   .wants/ or .requires/ directory.
2) A unit's purpose may be to act as a helper for some other unit which has
   a requirement dependency on it.
3) A unit may be started when needed via activation (socket, path, timer,
   D-Bus, udev, scripted systemctl call, ...).

Comment 5 Michal Schmidt 2013-04-23 15:29:07 UTC
Do you get the same error if you use the --root parameter like this?:

systemctl --root=/ enable topical-guide-setup.service

If you do, you can run it under strace, which will tell us more:

strace -o trace.txt systemctl --root=/ enable topical-guide-setup.service

Comment 6 Andrew McNabb 2013-04-23 15:39:07 UTC
(In reply to comment #4)
> There is no [Install] section, so systemctl enable won't work. But the error
> message should be completely different. This is how it looks on my F18
> system with the same systemd version:

Sorry--I uploaded an old version of the file by accident. The newer version has an Install section:

[Install]
WantedBy=multi-user.target

Sorry about that.

Comment 7 Andrew McNabb 2013-04-23 15:41:34 UTC
Created attachment 739065 [details]
strace output

(In reply to comment #5)
> Do you get the same error if you use the --root parameter like this?:
> 
> systemctl --root=/ enable topical-guide-setup.service

The error is almost the same: "Operation failed: No such file or directory"

> If you do, you can run it under strace, which will tell us more:
> 
> strace -o trace.txt systemctl --root=/ enable topical-guide-setup.service

This is now attached.

Comment 8 Michal Schmidt 2013-04-23 15:43:18 UTC
open("///etc/systemd/system/topical-guide-setup.service", O_RDONLY|O_NOCTTY|O_NOFOLLOW|O_CLOEXEC) = -1 ELOOP (Too many levels of symbolic links)

Did you accidentally make it a symbolic link on itself, or something similar?

Comment 9 Andrew McNabb 2013-04-23 16:02:06 UTC
(In reply to comment #8)
> open("///etc/systemd/system/topical-guide-setup.service",
> O_RDONLY|O_NOCTTY|O_NOFOLLOW|O_CLOEXEC) = -1 ELOOP (Too many levels of
> symbolic links)
> 
> Did you accidentally make it a symbolic link on itself, or something similar?

The file is a symbolic link, but it points directly at a normal file. I can do `cat ///etc/systemd/system/topical-guide-setup.service` without any errors.

Comment 10 Zbigniew Jędrzejewski-Szmek 2013-04-23 17:31:17 UTC
O_NOFOLLOW implies that it won't be followed, no?

Comment 11 Andrew McNabb 2013-04-23 22:50:32 UTC
(In reply to comment #10)
> O_NOFOLLOW implies that it won't be followed, no?

Yeah, that must be the bug.

Comment 12 Zbigniew Jędrzejewski-Szmek 2013-04-23 23:38:57 UTC
Looking at the code, open() should be followed be readlink(). And it's not in your strace. What systemd version do you have?(In reply to comment #11)
> (In reply to comment #10)
> > O_NOFOLLOW implies that it won't be followed, no?
> 
> Yeah, that must be the bug.
Or a design choice. src/shared/install.c:1374 says unit_file_search(..., allow_symlink=false). It makes sense as far as systemd doesn't want to mess with arbitrary locations in the filesystem. I guess we need to get our story
with what is allowed and what straight, and then document it somewhere.

A separate issue is that reporting here is quite insufficient. Event at debug level, nothing useful is printed. But that can be fixed after the semantics are settled.

Comment 13 Andrew McNabb 2013-04-24 16:10:04 UTC
(In reply to comment #12)
> Or a design choice. src/shared/install.c:1374 says unit_file_search(...,
> allow_symlink=false). It makes sense as far as systemd doesn't want to mess
> with arbitrary locations in the filesystem. I guess we need to get our story
> with what is allowed and what straight, and then document it somewhere.

If it's a design choice, then it's a paternalistic one. If the "directory is administrator territory" as stated in the documentation, then the administrator should be allowed to use symbolic links as they deem necessary.

Comment 14 Lennart Poettering 2013-05-06 16:16:16 UTC
"systemctl enable" is about enabling vendor supplied unit files. It will only create and remove symlinks in /etc/ and /run/, that's all it does. So right now it's a pretty safe tool: it will create/override/remove the modifiable configuration via symlinks and strictly leave vendor supplied static data untouched, since it is stored in real files. However, if we suddenly allow enabling of symlinks, then this clear separation goes away.

This gets particularly nasty for disabling things, because that removes all symlinks to the destination file, and how should it know when to stop precisely?

So, yeah, I am pretty sure we shouldn't allow "enabling of symlinks".

What we should support however is enabling of unit files that are outside of the usualy search paths, via specifiying full absolute paths. i.e. "systemctl enable /var/lib/foo/bar.service" should link it to /etc/systemd/system/bar.service and do everything listed in [Install]. Now, I originally implemented things to work like that, but this might got broken one time...

Andrew, so if you'd call "systemctl enable" directly on the original unit file, instead of via a symlink, then everything should be fine for you, right?

Comment 15 Lennart Poettering 2013-05-06 16:18:45 UTC
So, I checked that "systemctl enable /absolute/path/foobar.service" still works, and it does. So I figure this bug can be closed.

Comment 16 Lennart Poettering 2013-05-06 17:20:46 UTC
*** Bug 896670 has been marked as a duplicate of this bug. ***

Comment 17 Lennart Poettering 2013-05-06 17:49:47 UTC
*** Bug 887140 has been marked as a duplicate of this bug. ***

Comment 18 Andrew McNabb 2013-05-06 18:38:36 UTC
(In reply to comment #14)
> "systemctl enable" is about enabling vendor supplied unit files. It will
> only create and remove symlinks in /etc/ and /run/, that's all it does. So
> right now it's a pretty safe tool: it will create/override/remove the
> modifiable configuration via symlinks and strictly leave vendor supplied
> static data untouched, since it is stored in real files. However, if we
> suddenly allow enabling of symlinks, then this clear separation goes away.
> 
> This gets particularly nasty for disabling things, because that removes all
> symlinks to the destination file, and how should it know when to stop
> precisely?
> 
> So, yeah, I am pretty sure we shouldn't allow "enabling of symlinks".
> 
> What we should support however is enabling of unit files that are outside of
> the usualy search paths, via specifiying full absolute paths. i.e.
> "systemctl enable /var/lib/foo/bar.service" should link it to
> /etc/systemd/system/bar.service and do everything listed in [Install]. Now,
> I originally implemented things to work like that, but this might got broken
> one time...
> 
> Andrew, so if you'd call "systemctl enable" directly on the original unit
> file, instead of via a symlink, then everything should be fine for you,
> right?

I have no idea. This is seriously undocumented.

Please, please, please, please, at least make the error message sane.

Comment 19 Andrew McNabb 2013-05-06 18:48:16 UTC
Just to clarify: the reason I dislike this resolution of the bug is because it's completely contrary to all official documentation and examples. The systemctl man page gives the usage for systemctl enable as: "enable [NAME...]". This does not indicate in any way that it's valid to give a full path, and the official blog-style documentation instructs people to copy files into /etc/systemd. It's unreasonable to expect people to know that they can put files anywhere on the system when this functionality isn't documented anywhere.

With respect to the safety of the disable command, it sounds like you're saying that systemd throws its symlinks to enable services into the directory that the documentation describes as "administrator territory". The directory for automatic symlinks should be a different place than the directory where the administrator is supposed to put custom unit files. If systemd has any trouble telling them apart, then it seems like it should stop putting symlinks into "administrator territory", and the two types of files should be in two different directories.

Comment 20 Fedora End Of Life 2013-12-21 15:29:32 UTC
This message is a reminder that Fedora 18 is nearing its end of life.
Approximately 4 (four) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 18. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as WONTFIX if it remains open with a Fedora 
'version' of '18'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version prior to Fedora 18's end of life.

Thank you for reporting this issue and we are sorry that we may not be 
able to fix it before Fedora 18 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior to Fedora 18's end of life.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Comment 21 Domen Kožar 2014-10-22 14:26:04 UTC
NixOS linux distribution has to patch systemd to overcome the limitation. The reason is that all our unit files are symlinks by design.

See https://github.com/NixOS/nixpkgs/issues/1083

The only security issue I see here is if symlink would point to untrusted location, which vendors should respect (and if not, such restrictions won't spare their foot).

I'll attach a file showing how we manage systemd unit files in stateless versioned manner.

Comment 22 Domen Kožar 2014-10-22 14:26:39 UTC
Created attachment 949412 [details]
systemd unit files are symlinks

Comment 23 Jan "Yenya" Kasprzak 2014-12-16 10:03:26 UTC
I want to deploy a small piece of software on a Fedora machine, but want the software including the unit files to live in its own git repository. It is thus not feasible for me to install the unit files to /lib/systemd, and want to have symlinks there. I think this is valid use case, and there is no reason of denying symlinks in /lib/systemd.

Moreover, systemd itself is inconsistent with respect to symlinks - on Fedora 20, I can "systemctl start" the unit file which is symlink, but cannot "systemctl enable" it.

And the last problem is that the error message from "systemctl enable" does not reveal the problem - it says "No such file or directory".

Please consider removing O_NOFOLLOW from the "systemctl enable" call.

Comment 24 Jaroslav Reznik 2015-03-03 16:52:45 UTC
This bug appears to have been reported against 'rawhide' during the Fedora 22 development cycle.
Changing version to '22'.

More information and reason for this action is here:
https://fedoraproject.org/wiki/Fedora_Program_Management/HouseKeeping/Fedora22

Comment 25 Jan Staněk 2015-03-17 13:10:10 UTC
(In reply to Jan "Yenya" Kasprzak from comment #23)
> Moreover, systemd itself is inconsistent with respect to symlinks - on
> Fedora 20, I can "systemctl start" the unit file which is symlink, but
> cannot "systemctl enable" it.
> 
> And the last problem is that the error message from "systemctl enable" does
> not reveal the problem - it says "No such file or directory".
> 
> Please consider removing O_NOFOLLOW from the "systemctl enable" call.

I would like to back up this proposal. I use systemctl --user calls for managing my user's service files. As I have several such files and are working on multiple machines, I have them stored in separate git repo and `systemctl --user link` them to the right place. This enables me to start the service, but not enable it - which in my opinion is kind of strange behaviour.

Comment 26 Zbigniew Jędrzejewski-Szmek 2015-03-18 12:43:00 UTC
Yes, current implementation is inconsistent. But it's not as simple as allowing all symlinks, because we have to distinguish links which point inside and outside of the directories systemd searches for units, because when uninstalling units, we delete links. When you factor in --root parameter support this becomes non-trivial. This definitely will be fixed at some point, but it's a matter of somebody taking the time to figure out all the use cases.

Comment 27 hashken 2015-03-29 06:55:55 UTC
(In reply to Zbigniew Jędrzejewski-Szmek from comment #26)
> Yes, current implementation is inconsistent. But it's not as simple as
> allowing all symlinks, because we have to distinguish links which point
> inside and outside of the directories systemd searches for units, because
> when uninstalling units, we delete links. When you factor in --root
> parameter support this becomes non-trivial. This definitely will be fixed at
> some point, but it's a matter of somebody taking the time to figure out all
> the use cases.

Given that the fix might take sometime, for the time being it would be useful if this is documented somewhere. I had to go through this whole bug report to figure this out. 

Also, shell completion for absolute paths for 'systemctl enable' command will be useful.

Comment 28 Dr. Gianluigi "Zane" Zanettini 2015-05-03 20:12:13 UTC
Hi, I found this topic Googling for the exact issue after banging my head against the desk for 2 days.

I've no problem with whatever design solution you choose, but I think  systemctl enable should at least spit a better error than "Failed to issue method call: No such file or directory"

"cannot enable symlink"?

Thanks,

--
Zane

Comment 29 mlissner@michaeljaylissner.com 2015-05-07 19:40:23 UTC
Alas, "systemctl enable /absolute/path/foobar.service" no longer works and now returns:

Excess arguments

So...as somebody that wants to use systemd for software that's released via Git, there appears to be no way to do the symlink trick that I need to.

Comment 30 Andrew McNabb 2015-07-19 02:43:33 UTC
mlissner, the "systemctl enable /absolute/path/foobar.service" workaround is working for me as of today. I'm running systemd-219-18.fc22.x86_64. Just out of curiosity, what version are you using?

Comment 31 Piotr Dobrogost 2016-03-20 15:13:27 UTC
I've got "Failed to execute operation: Too many levels of symbolic links" error when trying to set up OpenVPN client systemd service following instructions at https://fedoraproject.org/wiki/Openvpn and issuing `systemctl enable openvpn@MyClient.service` after `ln -s /lib/systemd/system/openvpn@.service /etc/systemd/system/openvpn@MyClient.service`

I'm surprised as linking from "etc/systemd/system/*" to "lib/systemd/system/*" seems to be official way of creating new services. Also, neither argument of `ln` above is a symbolic link.

I've read all comments in this bug but still don't understand the reason for the error. Is there any workaround? Is anyone working on a fix? Shouldn't the version of Fedora be upgraded in this bug?

Fedora 24 (as of 19-03-2016)
systemd 229

Comment 32 Piotr Dobrogost 2016-03-20 15:25:57 UTC
It seems this should be marked as duplicate of bug #1014311 wherein there's interesting discussion of the problem and the progress in the upstream is being tracked.

Comment 33 Michal Sekletar 2016-03-23 14:22:46 UTC
(In reply to Piotr Dobrogost from comment #31)
> I've got "Failed to execute operation: Too many levels of symbolic links"
> error when trying to set up OpenVPN client systemd service following
> instructions at https://fedoraproject.org/wiki/Openvpn and issuing
> `systemctl enable openvpn@MyClient.service` after `ln -s
> /lib/systemd/system/openvpn@.service
> /etc/systemd/system/openvpn@MyClient.service`
> 

I think that doing symlink manually is not necessary and actually misleading, because then you may not realize that all the changes to the unit file will be overwritten on next update. I don't understand why they mention this on the wiki page. Anyway...Issuing just plain systemctl enable openvpn@MyClient.service should do the right thing.

Comment 34 Jan Pazdziora 2016-03-23 14:31:39 UTC
I thought I'd note a potentially related bug 1285996 where systemctl disable fails when /etc/systemd/system itself is a symlink.

Comment 35 Fedora End Of Life 2016-07-19 19:26:02 UTC
Fedora 22 changed to end-of-life (EOL) status on 2016-07-19. Fedora 22 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.

Comment 36 Jan Pazdziora 2016-07-26 13:15:06 UTC
Reopening.

Comment 37 Zbigniew Jędrzejewski-Szmek 2016-08-04 04:34:37 UTC
Fixed in https://github.com/systemd/systemd/pull/3790, plus a follow up in https://github.com/systemd/systemd/pull/3806. systemd-231 in F25/rawhide has the fix.

Comment 41 Anze Zagar 2017-05-25 13:00:26 UTC
Is there any way to get this fix also for EL7?


Note You need to log in before you can comment on or make changes to this bug.