Bug 1285996

Summary: systemctl disable fails when /etc/systemd/system is a symlink
Product: Red Hat Enterprise Linux 7 Reporter: Jan Pazdziora <jpazdziora>
Component: systemdAssignee: systemd-maint
Status: CLOSED ERRATA QA Contact: Frantisek Sumsal <fsumsal>
Severity: unspecified Docs Contact:
Priority: high    
Version: 7.2CC: bblaskov, fsumsal, jpazdziora, lnykryn, msekleta, ovasik, pvoborni, systemd-maint-list, systemd-maint
Target Milestone: rcKeywords: Reopened
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: systemd-219-29 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-11-04 00:47:52 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1283777    

Description Jan Pazdziora 2015-11-27 07:51:40 UTC
Description of problem:

When /etc/systemd/system is a symlink, systemctl enable works but disable fails.

Version-Release number of selected component (if applicable):

systemd-219-19.el7.x86_64

How reproducible:

Deterministic.

Steps to Reproduce:
1. mkdir -p /data/etc/systemd
2. mv /etc/systemd/system /data/etc/systemd && ln -s /data/etc/systemd/system /etc/systemd/system
3. yum install -y httpd
4. systemctl enable httpd
5. systemctl disable httpd

Actual results:

# systemctl enable httpd
Created symlink from /etc/systemd/system/multi-user.target.wants/httpd.service to /usr/lib/systemd/system/httpd.service.
# systemctl disable httpd
Failed to execute operation: Too many levels of symbolic links

Expected results:

# systemctl enable httpd
Created symlink from /etc/systemd/system/multi-user.target.wants/httpd.service to /usr/lib/systemd/system/httpd.service.
# systemctl disable httpd
Removed symlink /etc/systemd/system/multi-user.target.wants/httpd.service.

Additional info:

This is a blocker for systemd in containers when the state of the system needs to be captured in a data volume.

Comment 1 Lukáš Nykrýn 2016-01-11 13:43:43 UTC
> This is a blocker for systemd in containers when the state of the system
> needs to be captured in a data volume.

This is unsupported setup, you can't have that directory on different partition.

In your case, please use bind-mounts.

Comment 2 Jan Pazdziora 2016-01-11 16:04:05 UTC
(In reply to Lukáš Nykrýn from comment #1)
> > This is a blocker for systemd in containers when the state of the system
> > needs to be captured in a data volume.
> 
> This is unsupported setup, you can't have that directory on different
> partition.

Could you please point me to the documentation that forbids it being on different partition?
 
> In your case, please use bind-mounts.

That will however still make it a different partition, won't it?

The problem with using bind-mounts is that the user would need to keep the bind mount location for /etc/systemd/system and the one with the other configuration bits and data in sync, opening possibility for mismatches and unnecessary failures.

Also, the message

   Too many levels of symbolic links

seems incorrect since one level cannot reasonably be expected "too many".

Hence, reopening.

Comment 5 Lukáš Nykrýn 2016-08-30 12:16:24 UTC
> The problem with using bind-mounts is that the user would need to keep the
> bind mount location for /etc/systemd/system and the one with the other
> configuration bits and data in sync, opening possibility for mismatches and
> unnecessary failures.

I don't follow. If we are talking about containers, and you have the data elsewhere you can tell your container tool to bindmount it during start for all locations, which would be more cleaner solution.

But I don't understand your use-case at all. Why it needs to be elsewhere, do you have some read-only setup? Or are you sharing the configuration between multiple containers?

Anyway I will look once more in the issue with disable and try to find some fix, but I really think that you are doing something wrong. As a workaround you can just symlink the whole /etc/systemd

Comment 6 Jan Pazdziora 2016-08-30 12:57:13 UTC
(In reply to Lukáš Nykrýn from comment #5)
> 
> I don't follow. If we are talking about containers, and you have the data
> elsewhere you can tell your container tool to bindmount it during start for
> all locations, which would be more cleaner solution.

Not if we have 55 of such locations: https://github.com/adelton/docker-freeipa/blob/master/volume-data-list

That's why we create one bindmounted directory and have all data and config locations pointing with symlinks to that directory.

> But I don't understand your use-case at all.

It's described in more detail for example at https://www.adelton.com/docs/docker/complex-application-in-container

> Why it needs to be elsewhere,
> do you have some read-only setup?

Read-only setup is on the horizon.

> Or are you sharing the configuration
> between multiple containers?

No.

> Anyway I will look once more in the issue with disable and try to find some
> fix, but I really think that you are doing something wrong. As a workaround
> you can just symlink the whole /etc/systemd

That seems too wide for our purposes.

Thank you for looking into it.

I think that just the fact that enable passes and disable does not means that there is some issue in the systemd code.

Comment 7 Lukáš Nykrýn 2016-08-30 13:16:00 UTC
https://github.com/systemd/systemd/pull/4063

Comment 8 Lukáš Nykrýn 2016-08-30 13:43:51 UTC
(In reply to Jan Pazdziora from comment #6)
> (In reply to Lukáš Nykrýn from comment #5)
> > 
> > I don't follow. If we are talking about containers, and you have the data
> > elsewhere you can tell your container tool to bindmount it during start for
> > all locations, which would be more cleaner solution.
> 
> Not if we have 55 of such locations:
> https://github.com/adelton/docker-freeipa/blob/master/volume-data-list
> 
> That's why we create one bindmounted directory and have all data and config
> locations pointing with symlinks to that directory.

Looks similar to our fedora(rhel)-readonly. (although it is doing bind mounts) 
https://git.fedorahosted.org/cgit/initscripts.git/tree/rwtab
https://git.fedorahosted.org/cgit/initscripts.git/tree/systemd/fedora-readonly

Maybe we should think about splitting that from initscripts and make that as a general tool for both use-case.

> It's described in more detail for example at
> https://www.adelton.com/docs/docker/complex-application-in-container

Hm, I see. That would be a material to discussion on mailing list or in a pub :-). Lets not spam bugzilla with that.
 
> I think that just the fact that enable passes and disable does not means
> that there is some issue in the systemd code.
I agree with that part

Comment 11 Branislav Blaškovič 2016-09-01 11:12:40 UTC
qa acking

Comment 15 Frantisek Sumsal 2016-09-07 13:30:44 UTC
Verified with systemd-219-29.el7.

Old package:
:: [  BEGIN   ] :: Running 'mv /etc/systemd/system '/tmp/tmp.w6SDW4BUVJ/''
:: [   PASS   ] :: Command 'mv /etc/systemd/system '/tmp/tmp.w6SDW4BUVJ/'' (Expected 0, got 0)
:: [  BEGIN   ] :: Running 'ln -s '/tmp/tmp.w6SDW4BUVJ/system' /etc/systemd/system'
:: [   PASS   ] :: Command 'ln -s '/tmp/tmp.w6SDW4BUVJ/system' /etc/systemd/system' (Expected 0, got 0)
:: [  BEGIN   ] :: Running 'systemctl disable cups'
Failed to execute operation: Too many levels of symbolic links
:: [   FAIL   ] :: Command 'systemctl disable cups' (Expected 0, got 1)
:: [  BEGIN   ] :: Running 'systemctl enable cups'
:: [   PASS   ] :: Command 'systemctl enable cups' (Expected 0, got 0)

New package:
:: [  BEGIN   ] :: Running 'mv /etc/systemd/system '/tmp/tmp.jH5TSkY5Dg/''
:: [   PASS   ] :: Command 'mv /etc/systemd/system '/tmp/tmp.jH5TSkY5Dg/'' (Expected 0, got 0)
:: [  BEGIN   ] :: Running 'ln -s '/tmp/tmp.jH5TSkY5Dg/system' /etc/systemd/system'
:: [   PASS   ] :: Command 'ln -s '/tmp/tmp.jH5TSkY5Dg/system' /etc/systemd/system' (Expected 0, got 0)
:: [  BEGIN   ] :: Running 'systemctl disable cups'
Removed symlink /etc/systemd/system/multi-user.target.wants/cups.path.
Removed symlink /etc/systemd/system/multi-user.target.wants/cups.service.
Removed symlink /etc/systemd/system/printer.target.wants/cups.service.
Removed symlink /etc/systemd/system/sockets.target.wants/cups.socket.
:: [   PASS   ] :: Command 'systemctl disable cups' (Expected 0, got 0)
:: [  BEGIN   ] :: Running 'systemctl enable cups'
Created symlink from /etc/systemd/system/multi-user.target.wants/cups.service to /usr/lib/systemd/system/cups.service.
Created symlink from /etc/systemd/system/printer.target.wants/cups.service to /usr/lib/systemd/system/cups.service.
Created symlink from /etc/systemd/system/sockets.target.wants/cups.socket to /usr/lib/systemd/system/cups.socket.
Created symlink from /etc/systemd/system/multi-user.target.wants/cups.path to /usr/lib/systemd/system/cups.path.
:: [   PASS   ] :: Command 'systemctl enable cups' (Expected 0, got 0)

Comment 17 errata-xmlrpc 2016-11-04 00:47:52 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-2216.html