Bug 1406684 - docker daemon with --userns-remap=default fails to run systemd in container
Summary: docker daemon with --userns-remap=default fails to run systemd in container
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: oci-systemd-hook
Version: 7.3
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: ---
Assignee: Tom Sweeney
QA Contact: Martin Jenner
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-12-21 09:03 UTC by Jan Pazdziora
Modified: 2017-10-19 15:19 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-10-19 15:19:12 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2017:2963 normal SHIPPED_LIVE oci-systemd-hook bug fix update 2017-10-19 18:53:21 UTC

Description Jan Pazdziora 2016-12-21 09:03:14 UTC
Description of problem:

With systemd running in container without --userns-remap=default (make sure you do not hit bug 1406674), and with bash running under daemon with --userns-remap=default (use user_namespace.enable=1 to avoid hitting bug 1406026), I've now tried to run systemd under daemon with --userns-remap=default.

But it fails.

Version-Release number of selected component (if applicable):

docker-1.12.5-4.el7.x86_64
oci-systemd-hook-0.1.4-8.git45455fe.el7.x86_64
(also tried with oci-systemd-hook-0.1.4-6)

How reproducible:

Deterministic.

Steps to Reproduce:
1. Have RHEL 7 machine with user_namespace.enable=1 in /proc/cmdline and docker installed.
2. Verify that docker run --rm -ti rhel7 /usr/sbin/init runs.
3. Set OPTIONS in /etc/sysconfig/docker to OPTIONS='--log-driver=journald --userns-remap=default'
4. echo dockremap:808080:1000 >> /etc/subuid
5. echo dockremap:808080:1000 >> /etc/subgid
6. systemctl restart docker
7. docker run --rm -ti rhel7 /usr/sbin/init

Actual results:

# docker run --rm -ti rhel7 /usr/sbin/init
/usr/bin/docker-current: Error response from daemon: invalid header field value "oci runtime error: container_linux.go:247: starting container process caused \"process_linux.go:334: running prestart hook 2 caused \\\"error running hook: exit status 1, stdout: , stderr: \\\"\"\n".

Journal says

Dec 21 04:01:13 machine.example.com dockerd-current[22041]: time="2016-12-21T04:01:13.452729932-05:00" level=info msg="{Action=create, Username=root, LoginUID=0, PID=24610}"
Dec 21 04:01:13 machine.example.com kernel: XFS (dm-3): Mounting V5 Filesystem
Dec 21 04:01:13 machine.example.com kernel: XFS (dm-3): Ending clean mount
Dec 21 04:01:13 machine.example.com kernel: XFS (dm-3): Unmounting Filesystem
Dec 21 04:01:13 machine.example.com kernel: XFS (dm-3): Mounting V5 Filesystem
Dec 21 04:01:13 machine.example.com kernel: XFS (dm-3): Ending clean mount
Dec 21 04:01:13 machine.example.com kernel: XFS (dm-3): Unmounting Filesystem
Dec 21 04:01:13 machine.example.com dockerd-current[22041]: time="2016-12-21T04:01:13.680471653-05:00" level=info msg="{Action=attach, Username=root, LoginUID=0, PID=24610}"
Dec 21 04:01:13 machine.example.com dockerd-current[22041]: time="2016-12-21T04:01:13.682554880-05:00" level=info msg="{Action=start, Username=root, LoginUID=0, PID=24610}"
Dec 21 04:01:13 machine.example.com kernel: XFS (dm-3): Mounting V5 Filesystem
Dec 21 04:01:13 machine.example.com kernel: XFS (dm-3): Ending clean mount
Dec 21 04:01:13 machine.example.com kernel: device veth591e9fb entered promiscuous mode
Dec 21 04:01:13 machine.example.com kernel: IPv6: ADDRCONF(NETDEV_UP): veth591e9fb: link is not ready
Dec 21 04:01:13 machine.example.com kernel: docker0: port 1(veth591e9fb) entered forwarding state
Dec 21 04:01:13 machine.example.com kernel: docker0: port 1(veth591e9fb) entered forwarding state
Dec 21 04:01:13 machine.example.com kernel: docker0: port 1(veth591e9fb) entered disabled state
Dec 21 04:01:13 machine.example.com NetworkManager[648]: <info>  [1482310873.7117] manager: (veth9f1ac4b): new Veth device (/org/freedesktop/NetworkManager/Devices/60)
Dec 21 04:01:13 machine.example.com NetworkManager[648]: <info>  [1482310873.7132] manager: (veth591e9fb): new Veth device (/org/freedesktop/NetworkManager/Devices/61)
Dec 21 04:01:13 machine.example.com systemd[1]: Scope libcontainer-24670-systemd-test-default-dependencies.scope has no PIDs. Refusing.
Dec 21 04:01:13 machine.example.com systemd[1]: Scope libcontainer-24670-systemd-test-default-dependencies.scope has no PIDs. Refusing.
Dec 21 04:01:13 machine.example.com systemd[1]: Started docker container abefcc628923d20213f595cac63d922a36f4e15ed4188f7d9a764c1beabad62d.
Dec 21 04:01:13 machine.example.com systemd[1]: Starting docker container abefcc628923d20213f595cac63d922a36f4e15ed4188f7d9a764c1beabad62d.
Dec 21 04:01:13 machine.example.com NetworkManager[648]: <info>  [1482310873.8043] device (veth9f1ac4b): driver 'veth' does not support carrier detection.
Dec 21 04:01:13 machine.example.com NetworkManager[648]: <info>  [1482310873.8048] device (veth591e9fb): link connected
Dec 21 04:01:13 machine.example.com kernel: IPv6: ADDRCONF(NETDEV_CHANGE): veth591e9fb: link becomes ready
Dec 21 04:01:13 machine.example.com kernel: docker0: port 1(veth591e9fb) entered forwarding state
Dec 21 04:01:13 machine.example.com kernel: docker0: port 1(veth591e9fb) entered forwarding state
Dec 21 04:01:13 machine.example.com NetworkManager[648]: <info>  [1482310873.8084] device (docker0): link connected
Dec 21 04:01:13 machine.example.com oci-systemd-hook[24697]: systemdhook <debug>: 11:hugetlb:/system.slice/docker-abefcc628923d20213f595cac63d922a36f4e15ed4188f7d9a764c1beabad62d.scope
Dec 21 04:01:13 machine.example.com oci-systemd-hook[24697]: systemdhook <debug>: :hugetlb:/system.slice/docker-abefcc628923d20213f595cac63d922a36f4e15ed4188f7d9a764c1beabad62d.scope
Dec 21 04:01:13 machine.example.com oci-systemd-hook[24697]: systemdhook <debug>: 10:perf_event:/system.slice/docker-abefcc628923d20213f595cac63d922a36f4e15ed4188f7d9a764c1beabad62d.scope
Dec 21 04:01:13 machine.example.com oci-systemd-hook[24697]: systemdhook <debug>: :perf_event:/system.slice/docker-abefcc628923d20213f595cac63d922a36f4e15ed4188f7d9a764c1beabad62d.scope
Dec 21 04:01:13 machine.example.com oci-systemd-hook[24697]: systemdhook <debug>: 9:cpuacct,cpu:/system.slice/docker-abefcc628923d20213f595cac63d922a36f4e15ed4188f7d9a764c1beabad62d.scope
Dec 21 04:01:13 machine.example.com oci-systemd-hook[24697]: systemdhook <debug>: :cpuacct,cpu:/system.slice/docker-abefcc628923d20213f595cac63d922a36f4e15ed4188f7d9a764c1beabad62d.scope
Dec 21 04:01:13 machine.example.com oci-systemd-hook[24697]: systemdhook <debug>: 8:net_prio,net_cls:/system.slice/docker-abefcc628923d20213f595cac63d922a36f4e15ed4188f7d9a764c1beabad62d.scope
Dec 21 04:01:13 machine.example.com oci-systemd-hook[24697]: systemdhook <debug>: :net_prio,net_cls:/system.slice/docker-abefcc628923d20213f595cac63d922a36f4e15ed4188f7d9a764c1beabad62d.scope
Dec 21 04:01:13 machine.example.com oci-systemd-hook[24697]: systemdhook <debug>: 7:pids:/system.slice/docker-abefcc628923d20213f595cac63d922a36f4e15ed4188f7d9a764c1beabad62d.scope
Dec 21 04:01:13 machine.example.com oci-systemd-hook[24697]: systemdhook <debug>: :pids:/system.slice/docker-abefcc628923d20213f595cac63d922a36f4e15ed4188f7d9a764c1beabad62d.scope
Dec 21 04:01:13 machine.example.com oci-systemd-hook[24697]: systemdhook <debug>: 6:blkio:/system.slice/docker-abefcc628923d20213f595cac63d922a36f4e15ed4188f7d9a764c1beabad62d.scope
Dec 21 04:01:13 machine.example.com oci-systemd-hook[24697]: systemdhook <debug>: :blkio:/system.slice/docker-abefcc628923d20213f595cac63d922a36f4e15ed4188f7d9a764c1beabad62d.scope
Dec 21 04:01:13 machine.example.com oci-systemd-hook[24697]: systemdhook <debug>: 5:cpuset:/system.slice/docker-abefcc628923d20213f595cac63d922a36f4e15ed4188f7d9a764c1beabad62d.scope
Dec 21 04:01:13 machine.example.com oci-systemd-hook[24697]: systemdhook <debug>: :cpuset:/system.slice/docker-abefcc628923d20213f595cac63d922a36f4e15ed4188f7d9a764c1beabad62d.scope
Dec 21 04:01:13 machine.example.com oci-systemd-hook[24697]: systemdhook <debug>: 4:memory:/system.slice/docker-abefcc628923d20213f595cac63d922a36f4e15ed4188f7d9a764c1beabad62d.scope
Dec 21 04:01:13 machine.example.com oci-systemd-hook[24697]: systemdhook <debug>: :memory:/system.slice/docker-abefcc628923d20213f595cac63d922a36f4e15ed4188f7d9a764c1beabad62d.scope
Dec 21 04:01:13 machine.example.com oci-systemd-hook[24697]: systemdhook <debug>: Found
Dec 21 04:01:13 machine.example.com oci-systemd-hook[24697]: systemdhook <debug>: PATH: /system.slice/docker-abefcc628923d20213f595cac63d922a36f4e15ed4188f7d9a764c1beabad62d.scope
Dec 21 04:01:13 machine.example.com oci-systemd-hook[24697]: systemdhook <debug>: SUBSYSTEM_PATH: /sys/fs/cgroup/memory/system.slice/docker-abefcc628923d20213f595cac63d922a36f4e15ed4188f7d9a764c1beabad62d.scope
Dec 21 04:01:13 machine.example.com oci-systemd-hook[24697]: systemdhook <debug>: memory path: /sys/fs/cgroup/memory/system.slice/docker-abefcc628923d20213f595cac63d922a36f4e15ed4188f7d9a764c1beabad62d.scope/memory.limit_in_bytes
Dec 21 04:01:13 machine.example.com oci-systemd-hook[24697]: systemdhook <debug>: LIMIT: 9223372036854771712
Dec 21 04:01:13 machine.example.com oci-systemd-hook[24697]: systemdhook <debug>: Limit in bytes: 9223372036854771712
Dec 21 04:01:13 machine.example.com oci-systemd-hook[24697]: systemdhook <error>: Failed to mount /sys/fs/cgroup on /var/lib/docker/808080.808080/devicemapper/mnt/54f539f2766a6d5f62e549cae7db6bdd5c9ecfb4b435606bdf201dac0518b133/rootfs/sys/fs/cgroup: Invalid argument
Dec 21 04:01:13 machine.example.com systemd[1]: Stopped docker container abefcc628923d20213f595cac63d922a36f4e15ed4188f7d9a764c1beabad62d.
Dec 21 04:01:13 machine.example.com systemd[1]: Stopping docker container abefcc628923d20213f595cac63d922a36f4e15ed4188f7d9a764c1beabad62d.
Dec 21 04:01:13 machine.example.com systemd[1]: Scope libcontainer-24704-systemd-test-default-dependencies.scope has no PIDs. Refusing.
Dec 21 04:01:13 machine.example.com systemd[1]: Scope libcontainer-24704-systemd-test-default-dependencies.scope has no PIDs. Refusing.
Dec 21 04:01:13 machine.example.com dockerd-current[22041]: time="2016-12-21T04:01:13.86277234-05:00" level=error msg="containerd: start container" error="oci runtime error: container_linux.go:247: starting container process caused \"process_linux.go:334: running prestart hook 2 caused \\\"error running hook: exit status 1, stdout: , stderr: \\\"\"\n" id=abefcc628923d20213f595cac63d922a36f4e15ed4188f7d9a764c1beabad62d
Dec 21 04:01:13 machine.example.com dockerd-current[22041]: time="2016-12-21T04:01:13.863515263-05:00" level=error msg="Create container failed with error: invalid header field value \"oci runtime error: container_linux.go:247: starting container process caused \\\"process_linux.go:334: running prestart hook 2 caused \\\\\\\"error running hook: exit status 1, stdout: , stderr: \\\\\\\"\\\"\\n\""
Dec 21 04:01:13 machine.example.com systemd[1]: Scope libcontainer-24709-systemd-test-default-dependencies.scope has no PIDs. Refusing.
Dec 21 04:01:13 machine.example.com systemd[1]: Scope libcontainer-24709-systemd-test-default-dependencies.scope has no PIDs. Refusing.
Dec 21 04:01:13 machine.example.com kernel: docker0: port 1(veth591e9fb) entered disabled state
Dec 21 04:01:13 machine.example.com NetworkManager[648]: <info>  [1482310873.8824] manager: (veth9f1ac4b): new Veth device (/org/freedesktop/NetworkManager/Devices/62)
Dec 21 04:01:13 machine.example.com kernel: docker0: port 1(veth591e9fb) entered disabled state
Dec 21 04:01:13 machine.example.com kernel: device veth591e9fb left promiscuous mode
Dec 21 04:01:13 machine.example.com kernel: docker0: port 1(veth591e9fb) entered disabled state
Dec 21 04:01:13 machine.example.com NetworkManager[648]: <info>  [1482310873.8927] device (veth9f1ac4b): driver 'veth' does not support carrier detection.
Dec 21 04:01:13 machine.example.com NetworkManager[648]: <info>  [1482310873.8929] device (veth591e9fb): driver 'veth' does not support carrier detection.
Dec 21 04:01:13 machine.example.com kernel: XFS (dm-3): Unmounting Filesystem
Dec 21 04:01:13 machine.example.com dockerd-current[22041]: time="2016-12-21T04:01:13.950948064-05:00" level=error msg="Handler for POST /v1.24/containers/abefcc628923d20213f595cac63d922a36f4e15ed4188f7d9a764c1beabad62d/start returned error: invalid header field value \"oci runtime error: container_linux.go:247: starting container process caused \\\"process_linux.go:334: running prestart hook 2 caused \\\\\\\"error running hook: exit status 1, stdout: , stderr: \\\\\\\"\\\"\\n\""
Dec 21 04:01:13 machine.example.com dockerd-current[22041]: time="2016-12-21T04:01:13.950987079-05:00" level=error msg="Handler for POST /v1.24/containers/abefcc628923d20213f595cac63d922a36f4e15ed4188f7d9a764c1beabad62d/start returned error: invalid header field value \"oci runtime error: container_linux.go:247: starting container process caused \\\"process_linux.go:334: running prestart hook 2 caused \\\\\\\"error running hook: exit status 1, stdout: , stderr: \\\\\\\"\\\"\\n\""
Dec 21 04:01:13 machine.example.com dockerd-current[22041]: time="2016-12-21T04:01:13.952597174-05:00" level=info msg="{Action=remove, Username=root, LoginUID=0, PID=24610}"
Dec 21 04:01:14 machine.example.com systemd-udevd[24616]: inotify_add_watch(7, /dev/dm-3, 10) failed: No such file or directory
Dec 21 04:01:14 machine.example.com systemd-udevd[24616]: inotify_add_watch(7, /dev/dm-3, 10) failed: No such file or directory

Expected results:

systemd running fine in container as uid 808080.

Additional info:

Comment 1 Jan Pazdziora 2016-12-21 09:06:15 UTC
When I remove

/usr/libexec/oci/hooks.d/oci-systemd-hook

and run the container as

docker run -e container=docker -v /sys/fs/cgroup:/sys/fs/cgroup:ro --tmpfs /run --tmpfs /tmp --rm -ti rhel7 /usr/sbin/init

I get

systemd 219 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ -LZ4 -SECCOMP +BLKID +ELFUTILS +KMOD +IDN)
Detected virtualization docker.
Detected architecture x86-64.

Welcome to Red Hat Enterprise Linux Server 7.3 (Maipo)!

Set hostname to <9416b9a4f846>.
Initializing machine ID from random generator.
Failed to read AF_UNIX datagram queue length, ignoring: No such file or directory
Failed to create root cgroup hierarchy: Permission denied
Failed to allocate manager object: Permission denied
[!!!!!!] Failed to allocate manager object, freezing.

Not sure if this is the same case or not, and whether this bugzilla should be about oci-systemd-hook rather than docker, and perhaps different bugzilla should be failed for this explicit --tmpfs and -v execution failure.

Comment 2 Antonio Murdaca 2016-12-21 09:11:06 UTC
I highly doubt you're supposed to run systemd without the oci-systemd-hook, that's simply not supported. 
As for the component, this is an oci-systemd-hook issue, but can also probably be tied to runc.

Comment 3 Jan Pazdziora 2016-12-21 09:23:00 UTC
(In reply to Antonio Murdaca from comment #2)
> I highly doubt you're supposed to run systemd without the oci-systemd-hook,
> that's simply not supported. 

Well, even without the hook, without --userns-remap=default, running

docker run -e container=docker -v /sys/fs/cgroup:/sys/fs/cgroup:ro --tmpfs /run --tmpfs /tmp --rm -ti rhel7 /usr/sbin/init

works. That's why I tried it as well in the --userns-remap=default situation, to simply gather more data about the behaviour.

Comment 4 Antonio Murdaca 2016-12-21 09:42:06 UTC
(In reply to Jan Pazdziora from comment #3)
> (In reply to Antonio Murdaca from comment #2)
> > I highly doubt you're supposed to run systemd without the oci-systemd-hook,
> > that's simply not supported. 
> 
> Well, even without the hook, without --userns-remap=default, running
> 
> docker run -e container=docker -v /sys/fs/cgroup:/sys/fs/cgroup:ro --tmpfs
> /run --tmpfs /tmp --rm -ti rhel7 /usr/sbin/init
> 
> works. That's why I tried it as well in the --userns-remap=default
> situation, to simply gather more data about the behaviour.

Sure, it's interesting it's working without the hook though.

Comment 6 Daniel Walsh 2016-12-21 13:34:02 UTC
Systemd works without the hook. the Hook is doing what the docker run CLI is doing automatically and in a better way.  But you can run systemd without the hook. 

The problem with user namespace and systemd I believe is that /sys/fs/cgroup/systemd needs to be written to by systemd, but user namespace is preventing it.  Which means this file system is not UserNamespace aware.  We might have to work with the systemd guys on a fix for this after the break.

On Fedora we see other issues with changes to the kernel, which we are working to fix.

Great job Jan on testing these issues.

Comment 7 Jan Pazdziora 2016-12-21 13:46:06 UTC
Thanks for analysing and triaging the results, and driving it further. If there is anything else you can think of that I should try before the break, please let me know.

Comment 8 Daniel Walsh 2016-12-21 13:57:48 UTC
I would guess you could easily prove my point by running a container with bash and then attempting to write to /sys/fs/cgroup/systemd, and see who owns the files in that directory while running with userns

Comment 9 Jan Pazdziora 2016-12-21 14:32:11 UTC
# docker run --rm -ti rhel7 ls -la /sys/fs/cgroup/systemd
total 0
drwxr-xr-x.  2 65534 65534   0 Dec 21 14:30 .
drwxr-xr-x. 13 root  root  340 Dec 21 14:30 ..
-rw-r--r--.  1 65534 65534   0 Dec 21 14:30 cgroup.clone_children
--w--w--w-.  1 65534 65534   0 Dec 21 14:30 cgroup.event_control
-rw-r--r--.  1 65534 65534   0 Dec 21 14:30 cgroup.procs
-rw-r--r--.  1 65534 65534   0 Dec 21 14:30 notify_on_release
-rw-r--r--.  1 65534 65534   0 Dec 21 14:30 tasks

# docker run -v /sys/fs/cgroup:/sys/fs/cgroup --rm -ti rhel7 ls -la /sys/fs/cgroup/systemd
total 0
drwxr-xr-x.  4 65534 65534   0 Dec 21 14:17 .
drwxr-xr-x. 13 65534 65534 340 Dec 21 14:17 ..
-rw-r--r--.  1 65534 65534   0 Dec 21 14:17 cgroup.clone_children
--w--w--w-.  1 65534 65534   0 Dec 21 14:17 cgroup.event_control
-rw-r--r--.  1 65534 65534   0 Dec 21 14:17 cgroup.procs
-r--r--r--.  1 65534 65534   0 Dec 21 14:17 cgroup.sane_behavior
-rw-r--r--.  1 65534 65534   0 Dec 21 14:17 notify_on_release
-rw-r--r--.  1 65534 65534   0 Dec 21 14:17 release_agent
drwxr-xr-x. 68 65534 65534   0 Dec 21 14:30 system.slice
-rw-r--r--.  1 65534 65534   0 Dec 21 14:17 tasks
drwxr-xr-x.  3 65534 65534   0 Dec 21  2016 user.slice

Running bash in the container and attempting to write to that directory gives me

# echo 1 > /sys/fs/cgroup/systemd/cgroup.clone_children
bash: /sys/fs/cgroup/systemd/cgroup.clone_children: Permission denied
# echo 1 > /sys/fs/cgroup/systemd/test
bash: /sys/fs/cgroup/systemd/test: Permission denied

Comment 10 Daniel Walsh 2016-12-21 14:42:03 UTC
So to fix this these UID's should follow user namespace in order for systemd to be able to run in a User Namespace container.

Comment 11 Jan Pazdziora 2017-05-15 09:46:14 UTC
It looks like the RHEL 7.4 nightly kernel made some progress with the uid namespacing support and configuration: bug 1406026 and bug 1340238.

However, attempt to start /usr/sbin/init in the container still fails with

# docker run --rm -ti -e container=docker rhel7 /usr/sbin/init
/usr/bin/docker-current: Error response from daemon: invalid header field value "oci runtime error: container_linux.go:247: starting container process caused \"process_linux.go:334: running prestart hook 2 caused \\\"error running hook: exit status 1, stdout: , stderr: \\\"\"\n".

Comment 13 Daniel Walsh 2017-05-15 12:18:15 UTC
This will probably never work with user namespace


# docker run -v /sys/fs/cgroup:/sys/fs/cgroup --rm -ti rhel7 ls -la /sys/fs/cgroup/systemd

Since the /sys/fs/cgroup will be owned by REAL Root and root inside of the container will not be able to read/write content in these directories.

Comment 15 Jan Pazdziora 2017-05-15 12:37:15 UTC
What is the correct component to track this against? Kernel, to present  /sys/fs/cgroup via uid-namespaced manner (if I understand comment 10 correctly), or systemd to avoid using cgroups in containers altogether?

Or are we here hitting a blocker which will forever prevent us from running systemd in uid-namespaced containers?

Comment 18 Daniel Walsh 2017-05-15 13:12:21 UTC
I think oci-systemd-hook should prepare the cgroup file system in a way that systemd can use it.  It understands what systemd wants to write.  It needs to change ownership of this directory to be the "dockerroot", not real root and then systemd would be able to work normally.  (At least I think this will work.

Comment 20 Tom Sweeney 2017-06-07 15:33:01 UTC
Fix submitted here: https://github.com/projectatomic/oci-systemd-hook/pull/59  and as noted by Lokesh Mandvekar (lsm5@redhat.com) "built here: https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=13367868

I'm also including a "rhel74" in the release tag to more easily tell
it's a 7.4 build."

Comment 29 errata-xmlrpc 2017-10-19 15:19:12 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:2963


Note You need to log in before you can comment on or make changes to this bug.