1285996 – systemctl disable fails when /etc/systemd/system is a symlink

RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1285996 - systemctl disable fails when /etc/systemd/system is a symlink

Summary: systemctl disable fails when /etc/systemd/system is a symlink

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat Enterprise Linux 7
Classification:	Red Hat
Component:	systemd
Sub Component:
Version:	7.2
Hardware:	Unspecified
OS:	Unspecified
Priority:	high
Severity:	unspecified
Target Milestone:	rc
Target Release:	---
Assignee:	systemd-maint
QA Contact:	Frantisek Sumsal
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:	1283777
TreeView+	depends on / blocked

Reported:	2015-11-27 07:51 UTC by Jan Pazdziora
Modified:	2016-11-04 00:47 UTC (History)
CC List:	9 users (show)
Fixed In Version:	systemd-219-29
Doc Type:	Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed:	2016-11-04 00:47:52 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Red Hat Product Errata	RHBA-2016:2216	0	normal	SHIPPED_LIVE	systemd bug fix and enhancement update	2016-11-03 13:24:51 UTC

Description Jan Pazdziora 2015-11-27 07:51:40 UTC

Description of problem:

When /etc/systemd/system is a symlink, systemctl enable works but disable fails.

Version-Release number of selected component (if applicable):

systemd-219-19.el7.x86_64

How reproducible:

Deterministic.

Steps to Reproduce:
1. mkdir -p /data/etc/systemd
2. mv /etc/systemd/system /data/etc/systemd && ln -s /data/etc/systemd/system /etc/systemd/system
3. yum install -y httpd
4. systemctl enable httpd
5. systemctl disable httpd

Actual results:

# systemctl enable httpd
Created symlink from /etc/systemd/system/multi-user.target.wants/httpd.service to /usr/lib/systemd/system/httpd.service.
# systemctl disable httpd
Failed to execute operation: Too many levels of symbolic links

Expected results:

# systemctl enable httpd
Created symlink from /etc/systemd/system/multi-user.target.wants/httpd.service to /usr/lib/systemd/system/httpd.service.
# systemctl disable httpd
Removed symlink /etc/systemd/system/multi-user.target.wants/httpd.service.

Additional info:

This is a blocker for systemd in containers when the state of the system needs to be captured in a data volume.

Comment 1 Lukáš Nykrýn 2016-01-11 13:43:43 UTC

> This is a blocker for systemd in containers when the state of the system
> needs to be captured in a data volume.

This is unsupported setup, you can't have that directory on different partition.

In your case, please use bind-mounts.

Comment 2 Jan Pazdziora 2016-01-11 16:04:05 UTC

(In reply to Lukáš Nykrýn from comment #1)
> > This is a blocker for systemd in containers when the state of the system
> > needs to be captured in a data volume.
> 
> This is unsupported setup, you can't have that directory on different
> partition.

Could you please point me to the documentation that forbids it being on different partition?
 
> In your case, please use bind-mounts.

That will however still make it a different partition, won't it?

The problem with using bind-mounts is that the user would need to keep the bind mount location for /etc/systemd/system and the one with the other configuration bits and data in sync, opening possibility for mismatches and unnecessary failures.

Also, the message

   Too many levels of symbolic links

seems incorrect since one level cannot reasonably be expected "too many".

Hence, reopening.

Comment 5 Lukáš Nykrýn 2016-08-30 12:16:24 UTC

> The problem with using bind-mounts is that the user would need to keep the
> bind mount location for /etc/systemd/system and the one with the other
> configuration bits and data in sync, opening possibility for mismatches and
> unnecessary failures.

I don't follow. If we are talking about containers, and you have the data elsewhere you can tell your container tool to bindmount it during start for all locations, which would be more cleaner solution.

But I don't understand your use-case at all. Why it needs to be elsewhere, do you have some read-only setup? Or are you sharing the configuration between multiple containers?

Anyway I will look once more in the issue with disable and try to find some fix, but I really think that you are doing something wrong. As a workaround you can just symlink the whole /etc/systemd

Comment 6 Jan Pazdziora 2016-08-30 12:57:13 UTC

(In reply to Lukáš Nykrýn from comment #5)
> 
> I don't follow. If we are talking about containers, and you have the data
> elsewhere you can tell your container tool to bindmount it during start for
> all locations, which would be more cleaner solution.

Not if we have 55 of such locations: https://github.com/adelton/docker-freeipa/blob/master/volume-data-list

That's why we create one bindmounted directory and have all data and config locations pointing with symlinks to that directory.

> But I don't understand your use-case at all.

It's described in more detail for example at https://www.adelton.com/docs/docker/complex-application-in-container

> Why it needs to be elsewhere,
> do you have some read-only setup?

Read-only setup is on the horizon.

> Or are you sharing the configuration
> between multiple containers?

No.

> Anyway I will look once more in the issue with disable and try to find some
> fix, but I really think that you are doing something wrong. As a workaround
> you can just symlink the whole /etc/systemd

That seems too wide for our purposes.

Thank you for looking into it.

I think that just the fact that enable passes and disable does not means that there is some issue in the systemd code.

Comment 7 Lukáš Nykrýn 2016-08-30 13:16:00 UTC

https://github.com/systemd/systemd/pull/4063

Comment 8 Lukáš Nykrýn 2016-08-30 13:43:51 UTC

(In reply to Jan Pazdziora from comment #6)
> (In reply to Lukáš Nykrýn from comment #5)
> > 
> > I don't follow. If we are talking about containers, and you have the data
> > elsewhere you can tell your container tool to bindmount it during start for
> > all locations, which would be more cleaner solution.
> 
> Not if we have 55 of such locations:
> https://github.com/adelton/docker-freeipa/blob/master/volume-data-list
> 
> That's why we create one bindmounted directory and have all data and config
> locations pointing with symlinks to that directory.

Looks similar to our fedora(rhel)-readonly. (although it is doing bind mounts) 
https://git.fedorahosted.org/cgit/initscripts.git/tree/rwtab
https://git.fedorahosted.org/cgit/initscripts.git/tree/systemd/fedora-readonly

Maybe we should think about splitting that from initscripts and make that as a general tool for both use-case.

> It's described in more detail for example at
> https://www.adelton.com/docs/docker/complex-application-in-container

Hm, I see. That would be a material to discussion on mailing list or in a pub :-). Lets not spam bugzilla with that.
 
> I think that just the fact that enable passes and disable does not means
> that there is some issue in the systemd code.
I agree with that part

Comment 11 Branislav Blaškovič 2016-09-01 11:12:40 UTC

qa acking

Comment 12 Lukáš Nykrýn 2016-09-05 13:49:15 UTC

fix merged to upstream staging branch ->
https://github.com/lnykryn/systemd-rhel/commit/55b81218e75888640d9127f176a28a6d9ba03d5d
https://github.com/lnykryn/systemd-rhel/commit/30c5299e91db30f07b64c6c47ac7417bd4e6988c
-> post

Comment 15 Frantisek Sumsal 2016-09-07 13:30:44 UTC

Verified with systemd-219-29.el7.

Old package:
:: [  BEGIN   ] :: Running 'mv /etc/systemd/system '/tmp/tmp.w6SDW4BUVJ/''
:: [   PASS   ] :: Command 'mv /etc/systemd/system '/tmp/tmp.w6SDW4BUVJ/'' (Expected 0, got 0)
:: [  BEGIN   ] :: Running 'ln -s '/tmp/tmp.w6SDW4BUVJ/system' /etc/systemd/system'
:: [   PASS   ] :: Command 'ln -s '/tmp/tmp.w6SDW4BUVJ/system' /etc/systemd/system' (Expected 0, got 0)
:: [  BEGIN   ] :: Running 'systemctl disable cups'
Failed to execute operation: Too many levels of symbolic links
:: [   FAIL   ] :: Command 'systemctl disable cups' (Expected 0, got 1)
:: [  BEGIN   ] :: Running 'systemctl enable cups'
:: [   PASS   ] :: Command 'systemctl enable cups' (Expected 0, got 0)

New package:
:: [  BEGIN   ] :: Running 'mv /etc/systemd/system '/tmp/tmp.jH5TSkY5Dg/''
:: [   PASS   ] :: Command 'mv /etc/systemd/system '/tmp/tmp.jH5TSkY5Dg/'' (Expected 0, got 0)
:: [  BEGIN   ] :: Running 'ln -s '/tmp/tmp.jH5TSkY5Dg/system' /etc/systemd/system'
:: [   PASS   ] :: Command 'ln -s '/tmp/tmp.jH5TSkY5Dg/system' /etc/systemd/system' (Expected 0, got 0)
:: [  BEGIN   ] :: Running 'systemctl disable cups'
Removed symlink /etc/systemd/system/multi-user.target.wants/cups.path.
Removed symlink /etc/systemd/system/multi-user.target.wants/cups.service.
Removed symlink /etc/systemd/system/printer.target.wants/cups.service.
Removed symlink /etc/systemd/system/sockets.target.wants/cups.socket.
:: [   PASS   ] :: Command 'systemctl disable cups' (Expected 0, got 0)
:: [  BEGIN   ] :: Running 'systemctl enable cups'
Created symlink from /etc/systemd/system/multi-user.target.wants/cups.service to /usr/lib/systemd/system/cups.service.
Created symlink from /etc/systemd/system/printer.target.wants/cups.service to /usr/lib/systemd/system/cups.service.
Created symlink from /etc/systemd/system/sockets.target.wants/cups.socket to /usr/lib/systemd/system/cups.socket.
Created symlink from /etc/systemd/system/multi-user.target.wants/cups.path to /usr/lib/systemd/system/cups.path.
:: [   PASS   ] :: Command 'systemctl enable cups' (Expected 0, got 0)

Comment 17 errata-xmlrpc 2016-11-04 00:47:52 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-2216.html

Note You need to log in before you can comment on or make changes to this bug.