RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1328684 - listen unix /run/docker/plugins/rhel-push-plugin.sock: bind: no such file or directory
Summary: listen unix /run/docker/plugins/rhel-push-plugin.sock: bind: no such file or ...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: docker-latest
Version: 7.4
Hardware: x86_64
OS: Linux
high
high
Target Milestone: rc
: ---
Assignee: Lokesh Mandvekar
QA Contact: atomic-bugs@redhat.com
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-04-20 04:00 UTC by Alex Jia
Modified: 2016-05-12 14:55 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-05-12 14:55:49 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHEA-2016:1057 0 normal SHIPPED_LIVE new packages: docker-latest 2016-05-12 18:51:24 UTC

Description Alex Jia 2016-04-20 04:00:48 UTC 
Description of problem:
The rhel-push-plugin doesn't work and got error like this 'listen unix /run/docker/plugins/rhel-push-plugin.sock: bind: no such file or directory'

Version-Release number of selected component (if applicable):
# rpm -q docker-latest docker-latest-rhel-push-plugin
docker-latest-1.10.3-10.el7.x86_64
docker-latest-rhel-push-plugin-1.10.3-10.el7.x86_64

How reproducible:
always

Steps to Reproduce:
1. if you're not using the systemd unit file then run rhel-push-plugin in background like /usr/lib/docker-latest/rhel-push-plugin &
2. in a new terminal to restart docker-latest daemon
3. return to step 1 terminal and check output 

Actual results:

# /usr/lib/docker-latest/rhel-push-plugin &
[1] 136028
[root@dell-per630-02 ~]# FATA[0000] listen unix /run/docker/plugins/rhel-push-plugin.sock: bind: no such file or directory 

# systemctl restart docker-latest
# systemctl is-active docker-latest
active

# ls /run/docker-latest/plugins/
ls: cannot access /run/docker-latest/plugins/: No such file or directory

# ls /run/docker/plugins/
ls: cannot access /run/docker/plugins/: No such file or directory


Expected results:


Additional info:

In fact, the rhel-push-plugin.sock haven't been created under the /run/docker-latest/plugins or /run/docker/plugins.

And docker-latest finds rhel-push-plugin.sock in /run/docker/plugins as default. 

In addition, users must use a absolute path to use rhel-push-plugin command w/o extra setting.
Comment 1 Alex Jia 2016-04-20 04:10:17 UTC 
(In reply to Alex Jia from comment #0)
> Steps to Reproduce:

> 1. if you're not using the systemd unit file then run rhel-push-plugin in
> background like /usr/lib/docker-latest/rhel-push-plugin &

I miss a step in here, you need to append --authorization-plugin=rhel-push-plugin to docker-latest daemon cmdline or add it into OPTIONS in /etc/sysconfig/docker-latest like this OPTIONS='--selinux-enabled --log-driver=journald --authorization-plugin=rhel-push-plugin'

> 2. in a new terminal to restart docker-latest daemon

To check if --authorization-plugin is added into docker-latest daemon cmdline

# ps auxf|grep docker-latest | grep -i '[R|r]hel-push-plugin'
root     136101  0.0  0.0 476172 30868 ?        Sl   11:35   0:00  \_ /usr/bin/docker-latest daemon --exec-opt native.cgroupdriver=systemd -g /var/lib/docker-latest --selinux-enabled --log-driver=journald --authorization-plugin=rhel-push-plugin
Comment 3 Antonio Murdaca 2016-04-20 09:50:00 UTC 
Alright, I suspect you never installed "docker" on the machine you're trying. Is it right?

There's an issue in the plugin itself which assumes that its socket should be placed inside "/run/docker/" which does not exists because what exists in docker-latest is "/run/docker-latest/". I can make this configurable in the plugin and rebuild it.

Curiosity, why didn't you start the plugin with systemd?
Comment 4 Antonio Murdaca 2016-04-20 09:51:21 UTC 
I think all the other plugins won't work also (novolume, lvm...) because they assume they'll listen on /run/docker/
Comment 5 Antonio Murdaca 2016-04-20 10:01:03 UTC 
So, right now the issue is that if the plugin isn't socket activated by systemd it assumes that it'll listen on "/run/docker-latest/plugins/rhel-push-plugin.sock"

I've added a flag --plugin-sock= to the plugin to change this behavior at startup time. But keep in mind that w/o systemd service and socket units you have to make sure the full patch to the socket exists.

In your case you have to make sure /run/docker-latest/plugins is there before starting the plugin. I've just tested this out and the plugin works.

But really, it should be started with systemd (not & in background)
Comment 6 Antonio Murdaca 2016-04-20 10:04:03 UTC 
Lokesh to make this work with systemd we have to edit the path inside the service and socket unit file:

[in service file]
ExecStart=/usr/libexec/docker-latest/rhel-push-plugin

[in socket file]
ListenStream=/run/docker-latest/plugins/rhel-push-plugin.sock

Also, I've noticed the plugin is installed into "/usr/lib/docker-latest/rhel-push-plugin" but it shouldn't because the systemd service file is looking for "/usr/libexec/docker[-latest].

Is this an error or am I missing something?
Comment 7 Antonio Murdaca 2016-04-20 10:05:11 UTC 
Lokesh, the novolume plugin should be changed as well when dealing with docker-latest
Comment 8 Alex Jia 2016-04-20 11:37:08 UTC 
(In reply to Antonio Murdaca from comment #3)
> Alright, I suspect you never installed "docker" on the machine you're
> trying. Is it right?
> 

Not really.

# rpm -q docker docker-latest
docker-1.9.1-28.el7.x86_64
docker-latest-1.10.3-10.el7.x86_64

# systemctl is-active docker-latest
active

# systemctl is-active docker
inactive


> There's an issue in the plugin itself which assumes that its socket should
> be placed inside "/run/docker/" which does not exists because what exists in
> docker-latest is "/run/docker-latest/". I can make this configurable in the
> plugin and rebuild it.
> 

Yes, I saw socket is placed inside /run/docker in upstream.

[Unit]
Description=Docker Block RHEL push plugin Socket for the API
Documentation=man:rhel-push-plugin(8)

[Socket]
ListenStream=/run/docker/plugins/rhel-push-plugin.sock

[Install]
WantedBy=sockets.target
 


> Curiosity, why didn't you start the plugin with systemd?

# systemctl list-units|grep push

NOTE: I can't find rhel-push-plugin related systemd unit file.

But in fact, it indeed exist under the /usr/lib/systemd/system/ and is disabled,
when start rhel-push-plugin service, the socket is created under the /run/docker-latest/plugins/, but the rhel-push-plugin is failed to EXEC spawning /usr/libexec/docker-latest/rhel-push-plugin: No such file or directory, the details as follows.


# systemctl enable rhel-push-plugin 
Created symlink from /etc/systemd/system/multi-user.target.wants/rhel-push-plugin.service to /usr/lib/systemd/system/rhel-push-plugin.service.

# systemctl is-enabled rhel-push-plugin 
enabled

# systemctl start rhel-push-plugin

# systemctl status rhel-push-plugin -l
● rhel-push-plugin.service - Docker Block RHEL push plugin authZ Plugin
   Loaded: loaded (/usr/lib/systemd/system/rhel-push-plugin.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Wed 2016-04-20 19:30:38 CST; 7s ago
     Docs: man:rhel-push-plugin(8)
  Process: 10291 ExecStart=/usr/libexec/docker-latest/rhel-push-plugin (code=exited, status=203/EXEC)
 Main PID: 10291 (code=exited, status=203/EXEC)

Apr 20 19:30:38 dell-per630-02.qe.lab.eng.nay.redhat.com systemd[1]: Started Docker Block RHEL push plugin authZ Plugin.
Apr 20 19:30:38 dell-per630-02.qe.lab.eng.nay.redhat.com systemd[1]: Starting Docker Block RHEL push plugin authZ Plugin...
Apr 20 19:30:38 dell-per630-02.qe.lab.eng.nay.redhat.com systemd[10291]: Failed at step EXEC spawning /usr/libexec/docker-latest/rhel-push-plugin: No such file or directory
Apr 20 19:30:38 dell-per630-02.qe.lab.eng.nay.redhat.com systemd[1]: rhel-push-plugin.service: main process exited, code=exited, status=203/EXEC
Apr 20 19:30:38 dell-per630-02.qe.lab.eng.nay.redhat.com systemd[1]: Unit rhel-push-plugin.service entered failed state.
Apr 20 19:30:38 dell-per630-02.qe.lab.eng.nay.redhat.com systemd[1]: rhel-push-plugin.service failed.
Comment 9 Antonio Murdaca 2016-04-20 12:19:17 UTC 
(In reply to Alex Jia from comment #8)
> (In reply to Antonio Murdaca from comment #3)
> > Alright, I suspect you never installed "docker" on the machine you're
> > trying. Is it right?
> > 
> 
> Not really.
> 
> # rpm -q docker docker-latest
> docker-1.9.1-28.el7.x86_64
> docker-latest-1.10.3-10.el7.x86_64

Alright, so the "/run/docker/*plugins*" directory should be created _before_ if not running with systemd (as I've exaplained above)

> 
> # systemctl is-active docker-latest
> active
> 
> # systemctl is-active docker
> inactive
> 
> 
> > There's an issue in the plugin itself which assumes that its socket should
> > be placed inside "/run/docker/" which does not exists because what exists in
> > docker-latest is "/run/docker-latest/". I can make this configurable in the
> > plugin and rebuild it.
> > 
> 
> Yes, I saw socket is placed inside /run/docker in upstream.

we'll fix this as well using sed I guess

> 
> [Unit]
> Description=Docker Block RHEL push plugin Socket for the API
> Documentation=man:rhel-push-plugin(8)
> 
> [Socket]
> ListenStream=/run/docker/plugins/rhel-push-plugin.sock
> 
> [Install]
> WantedBy=sockets.target
>  
> 
> 
> > Curiosity, why didn't you start the plugin with systemd?
> 
> # systemctl list-units|grep push
> 
> NOTE: I can't find rhel-push-plugin related systemd unit file.
> 
> But in fact, it indeed exist under the /usr/lib/systemd/system/ and is
> disabled,
> when start rhel-push-plugin service, the socket is created under the
> /run/docker-latest/plugins/, but the rhel-push-plugin is failed to EXEC
> spawning /usr/libexec/docker-latest/rhel-push-plugin: No such file or
> directory, the details as follows.

That's because the plugin it's installed in wrong directory, see my previous comment, we'll fix this and the unit file will work correctly.

> 
> 
> # systemctl enable rhel-push-plugin 
> Created symlink from
> /etc/systemd/system/multi-user.target.wants/rhel-push-plugin.service to
> /usr/lib/systemd/system/rhel-push-plugin.service.
> 
> # systemctl is-enabled rhel-push-plugin 
> enabled
> 
> # systemctl start rhel-push-plugin
> 
> # systemctl status rhel-push-plugin -l
> ● rhel-push-plugin.service - Docker Block RHEL push plugin authZ Plugin
>    Loaded: loaded (/usr/lib/systemd/system/rhel-push-plugin.service;
> enabled; vendor preset: disabled)
>    Active: failed (Result: exit-code) since Wed 2016-04-20 19:30:38 CST; 7s
> ago
>      Docs: man:rhel-push-plugin(8)
>   Process: 10291 ExecStart=/usr/libexec/docker-latest/rhel-push-plugin
> (code=exited, status=203/EXEC)
>  Main PID: 10291 (code=exited, status=203/EXEC)
> 
> Apr 20 19:30:38 dell-per630-02.qe.lab.eng.nay.redhat.com systemd[1]: Started
> Docker Block RHEL push plugin authZ Plugin.
> Apr 20 19:30:38 dell-per630-02.qe.lab.eng.nay.redhat.com systemd[1]:
> Starting Docker Block RHEL push plugin authZ Plugin...
> Apr 20 19:30:38 dell-per630-02.qe.lab.eng.nay.redhat.com systemd[10291]:
> Failed at step EXEC spawning /usr/libexec/docker-latest/rhel-push-plugin: No
> such file or directory
> Apr 20 19:30:38 dell-per630-02.qe.lab.eng.nay.redhat.com systemd[1]:
> rhel-push-plugin.service: main process exited, code=exited, status=203/EXEC
> Apr 20 19:30:38 dell-per630-02.qe.lab.eng.nay.redhat.com systemd[1]: Unit
> rhel-push-plugin.service entered failed state.
> Apr 20 19:30:38 dell-per630-02.qe.lab.eng.nay.redhat.com systemd[1]:
> rhel-push-plugin.service failed.
Comment 10 Antonio Murdaca 2016-04-22 10:09:29 UTC 
Update: we can't place the plugin socket nowhere out of /run/docker/plugins because the docker daemon always looks into this dir to find out about plugins. So we can't use /run/docker-latest/plugins unless we add a patch to the docker daemon to also look into this dir. Not sure what to do. Dan?
Comment 11 Daniel Walsh 2016-04-22 16:05:42 UTC 
We should not create a rhel-latest-push-plugin.  All content under /run should be the same for the docker daemon.

We are not creating a /run/docker-latest.sock.
Comment 12 Antonio Murdaca 2016-04-22 16:13:50 UTC 
Ack, does that apply to all /run/docker/ also? In this case, perfect, I still need to fix docker-latest spec.
Comment 13 Daniel Walsh 2016-04-22 17:09:03 UTC 
Yes we don't want to change anything in the docker package, and all support tools for docker should just work.

Docker storage setup and the systemd configuration for docker package is the only thing that should change.
Comment 14 Antonio Murdaca 2016-04-23 09:17:52 UTC 
alright, so the only bug in this bugzilla was that the systemd unit file was using in its ExecStart the path /usr/libexec/docker/rhel-push-plugin but the binary itself was installed under /usr/lib/docker/rhel-push-plugin, so it fails to start. I'll send a patch to Lokesh to fix this.
Comment 15 Antonio Murdaca 2016-04-23 09:23:01 UTC 
This plugin should be also enabled on installation and the flag added to the daemon to use it, is this possible? because this is replacing one of our patch and we should activate it by default
Comment 16 Antonio Murdaca 2016-04-23 18:21:55 UTC 
sent a patch to Lokesh
Comment 18 Antonio Murdaca 2016-04-27 13:38:58 UTC 
works on my RHEL 7.2 now
Comment 19 Alex Jia 2016-04-29 09:22:50 UTC 
Moving the bug to VERIFIED status per bellow testing.

# rpm -q docker-rhel-push-plugin docker-latest
docker-rhel-push-plugin-1.10.3-19.el7.x86_64
docker-latest-1.10.3-19.el7.x86_64

# systemctl start rhel-push-plugin
# systemctl status rhel-push-plugin
● rhel-push-plugin.service - Docker Block RHEL push plugin authZ Plugin
   Loaded: loaded (/usr/lib/systemd/system/rhel-push-plugin.service; disabled; vendor preset: disabled)
   Active: active (running) since Fri 2016-04-29 16:23:11 CST; 1min 34s ago
     Docs: man:rhel-push-plugin(8)
 Main PID: 9719 (rhel-push-plugi)
   Memory: 8.0K
   CGroup: /system.slice/rhel-push-plugin.service
           └─9719 /usr/libexec/docker/rhel-push-plugin

Apr 29 16:23:11 hp-dl360g9-04.qe.lab.eng.nay.redhat.com systemd[1]: Started Docker Block RHEL push plugin authZ Plugin.
Apr 29 16:23:11 hp-dl360g9-04.qe.lab.eng.nay.redhat.com systemd[1]: Starting Docker Block RHEL push plugin authZ Plugin...
Apr 29 16:24:41 hp-dl360g9-04.qe.lab.eng.nay.redhat.com systemd[1]: Started Docker Block RHEL push plugin authZ Plugin.
 
# ps auxf|grep docker-latest | grep -i '[R|r]hel-push-plugin'
root      9802  0.0  0.0 115236  1432 ?        Ss   16:23   0:00 /bin/sh -c /usr/bin/docker-latest daemon            --authorization-plugin=rhel-push-plugin            --exec-opt native.cgroupdriver=systemd            -g /var/lib/docker-latest            $OPTIONS            $DOCKER_STORAGE_OPTIONS            $DOCKER_NETWORK_OPTIONS            $INSECURE_REGISTRY            2>&1 | /usr/bin/forward-journald -tag docker-latest
root      9805  0.2  0.2 636124 32336 ?        Sl   16:23   0:00  \_ /usr/bin/docker-latest daemon --authorization-plugin=rhel-push-plugin --exec-opt native.cgroupdriver=systemd -g /var/lib/docker-latest --log-driver=journald --storage-driver devicemapper --storage-opt dm.fs=xfs --storage-opt dm.thinpooldev=/dev/mapper/rhel_hp--dl360g9--04-docker--latest--pool --storage-opt dm.use_deferred_removal=true --storage-opt dm.use_deferred_deletion=true
Comment 21 errata-xmlrpc 2016-05-12 14:55:49 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHEA-2016-1057.html
Note You need to log in before you can comment on or make changes to this bug.