Bug 1328684
Summary: | listen unix /run/docker/plugins/rhel-push-plugin.sock: bind: no such file or directory | ||
---|---|---|---|
Product: | Red Hat Enterprise Linux 7 | Reporter: | Alex Jia <ajia> |
Component: | docker-latest | Assignee: | Lokesh Mandvekar <lsm5> |
Status: | CLOSED ERRATA | QA Contact: | atomic-bugs <atomic-bugs> |
Severity: | high | Docs Contact: | |
Priority: | high | ||
Version: | 7.4 | CC: | amurdaca, dwalsh |
Target Milestone: | rc | Keywords: | Extras, ZStream |
Target Release: | --- | ||
Hardware: | x86_64 | ||
OS: | Linux | ||
Whiteboard: | |||
Fixed In Version: | Doc Type: | Bug Fix | |
Doc Text: | Story Points: | --- | |
Clone Of: | Environment: | ||
Last Closed: | 2016-05-12 14:55:49 UTC | Type: | Bug |
Regression: | --- | Mount Type: | --- |
Documentation: | --- | CRM: | |
Verified Versions: | Category: | --- | |
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
Cloudforms Team: | --- | Target Upstream Version: | |
Embargoed: |
Description
Alex Jia
2016-04-20 04:00:48 UTC
(In reply to Alex Jia from comment #0) > Steps to Reproduce: > 1. if you're not using the systemd unit file then run rhel-push-plugin in > background like /usr/lib/docker-latest/rhel-push-plugin & I miss a step in here, you need to append --authorization-plugin=rhel-push-plugin to docker-latest daemon cmdline or add it into OPTIONS in /etc/sysconfig/docker-latest like this OPTIONS='--selinux-enabled --log-driver=journald --authorization-plugin=rhel-push-plugin' > 2. in a new terminal to restart docker-latest daemon To check if --authorization-plugin is added into docker-latest daemon cmdline # ps auxf|grep docker-latest | grep -i '[R|r]hel-push-plugin' root 136101 0.0 0.0 476172 30868 ? Sl 11:35 0:00 \_ /usr/bin/docker-latest daemon --exec-opt native.cgroupdriver=systemd -g /var/lib/docker-latest --selinux-enabled --log-driver=journald --authorization-plugin=rhel-push-plugin Alright, I suspect you never installed "docker" on the machine you're trying. Is it right? There's an issue in the plugin itself which assumes that its socket should be placed inside "/run/docker/" which does not exists because what exists in docker-latest is "/run/docker-latest/". I can make this configurable in the plugin and rebuild it. Curiosity, why didn't you start the plugin with systemd? I think all the other plugins won't work also (novolume, lvm...) because they assume they'll listen on /run/docker/ So, right now the issue is that if the plugin isn't socket activated by systemd it assumes that it'll listen on "/run/docker-latest/plugins/rhel-push-plugin.sock" I've added a flag --plugin-sock= to the plugin to change this behavior at startup time. But keep in mind that w/o systemd service and socket units you have to make sure the full patch to the socket exists. In your case you have to make sure /run/docker-latest/plugins is there before starting the plugin. I've just tested this out and the plugin works. But really, it should be started with systemd (not & in background) Lokesh to make this work with systemd we have to edit the path inside the service and socket unit file: [in service file] ExecStart=/usr/libexec/docker-latest/rhel-push-plugin [in socket file] ListenStream=/run/docker-latest/plugins/rhel-push-plugin.sock Also, I've noticed the plugin is installed into "/usr/lib/docker-latest/rhel-push-plugin" but it shouldn't because the systemd service file is looking for "/usr/libexec/docker[-latest]. Is this an error or am I missing something? Lokesh, the novolume plugin should be changed as well when dealing with docker-latest (In reply to Antonio Murdaca from comment #3) > Alright, I suspect you never installed "docker" on the machine you're > trying. Is it right? > Not really. # rpm -q docker docker-latest docker-1.9.1-28.el7.x86_64 docker-latest-1.10.3-10.el7.x86_64 # systemctl is-active docker-latest active # systemctl is-active docker inactive > There's an issue in the plugin itself which assumes that its socket should > be placed inside "/run/docker/" which does not exists because what exists in > docker-latest is "/run/docker-latest/". I can make this configurable in the > plugin and rebuild it. > Yes, I saw socket is placed inside /run/docker in upstream. [Unit] Description=Docker Block RHEL push plugin Socket for the API Documentation=man:rhel-push-plugin(8) [Socket] ListenStream=/run/docker/plugins/rhel-push-plugin.sock [Install] WantedBy=sockets.target > Curiosity, why didn't you start the plugin with systemd? # systemctl list-units|grep push NOTE: I can't find rhel-push-plugin related systemd unit file. But in fact, it indeed exist under the /usr/lib/systemd/system/ and is disabled, when start rhel-push-plugin service, the socket is created under the /run/docker-latest/plugins/, but the rhel-push-plugin is failed to EXEC spawning /usr/libexec/docker-latest/rhel-push-plugin: No such file or directory, the details as follows. # systemctl enable rhel-push-plugin Created symlink from /etc/systemd/system/multi-user.target.wants/rhel-push-plugin.service to /usr/lib/systemd/system/rhel-push-plugin.service. # systemctl is-enabled rhel-push-plugin enabled # systemctl start rhel-push-plugin # systemctl status rhel-push-plugin -l ● rhel-push-plugin.service - Docker Block RHEL push plugin authZ Plugin Loaded: loaded (/usr/lib/systemd/system/rhel-push-plugin.service; enabled; vendor preset: disabled) Active: failed (Result: exit-code) since Wed 2016-04-20 19:30:38 CST; 7s ago Docs: man:rhel-push-plugin(8) Process: 10291 ExecStart=/usr/libexec/docker-latest/rhel-push-plugin (code=exited, status=203/EXEC) Main PID: 10291 (code=exited, status=203/EXEC) Apr 20 19:30:38 dell-per630-02.qe.lab.eng.nay.redhat.com systemd[1]: Started Docker Block RHEL push plugin authZ Plugin. Apr 20 19:30:38 dell-per630-02.qe.lab.eng.nay.redhat.com systemd[1]: Starting Docker Block RHEL push plugin authZ Plugin... Apr 20 19:30:38 dell-per630-02.qe.lab.eng.nay.redhat.com systemd[10291]: Failed at step EXEC spawning /usr/libexec/docker-latest/rhel-push-plugin: No such file or directory Apr 20 19:30:38 dell-per630-02.qe.lab.eng.nay.redhat.com systemd[1]: rhel-push-plugin.service: main process exited, code=exited, status=203/EXEC Apr 20 19:30:38 dell-per630-02.qe.lab.eng.nay.redhat.com systemd[1]: Unit rhel-push-plugin.service entered failed state. Apr 20 19:30:38 dell-per630-02.qe.lab.eng.nay.redhat.com systemd[1]: rhel-push-plugin.service failed. (In reply to Alex Jia from comment #8) > (In reply to Antonio Murdaca from comment #3) > > Alright, I suspect you never installed "docker" on the machine you're > > trying. Is it right? > > > > Not really. > > # rpm -q docker docker-latest > docker-1.9.1-28.el7.x86_64 > docker-latest-1.10.3-10.el7.x86_64 Alright, so the "/run/docker/*plugins*" directory should be created _before_ if not running with systemd (as I've exaplained above) > > # systemctl is-active docker-latest > active > > # systemctl is-active docker > inactive > > > > There's an issue in the plugin itself which assumes that its socket should > > be placed inside "/run/docker/" which does not exists because what exists in > > docker-latest is "/run/docker-latest/". I can make this configurable in the > > plugin and rebuild it. > > > > Yes, I saw socket is placed inside /run/docker in upstream. we'll fix this as well using sed I guess > > [Unit] > Description=Docker Block RHEL push plugin Socket for the API > Documentation=man:rhel-push-plugin(8) > > [Socket] > ListenStream=/run/docker/plugins/rhel-push-plugin.sock > > [Install] > WantedBy=sockets.target > > > > > Curiosity, why didn't you start the plugin with systemd? > > # systemctl list-units|grep push > > NOTE: I can't find rhel-push-plugin related systemd unit file. > > But in fact, it indeed exist under the /usr/lib/systemd/system/ and is > disabled, > when start rhel-push-plugin service, the socket is created under the > /run/docker-latest/plugins/, but the rhel-push-plugin is failed to EXEC > spawning /usr/libexec/docker-latest/rhel-push-plugin: No such file or > directory, the details as follows. That's because the plugin it's installed in wrong directory, see my previous comment, we'll fix this and the unit file will work correctly. > > > # systemctl enable rhel-push-plugin > Created symlink from > /etc/systemd/system/multi-user.target.wants/rhel-push-plugin.service to > /usr/lib/systemd/system/rhel-push-plugin.service. > > # systemctl is-enabled rhel-push-plugin > enabled > > # systemctl start rhel-push-plugin > > # systemctl status rhel-push-plugin -l > ● rhel-push-plugin.service - Docker Block RHEL push plugin authZ Plugin > Loaded: loaded (/usr/lib/systemd/system/rhel-push-plugin.service; > enabled; vendor preset: disabled) > Active: failed (Result: exit-code) since Wed 2016-04-20 19:30:38 CST; 7s > ago > Docs: man:rhel-push-plugin(8) > Process: 10291 ExecStart=/usr/libexec/docker-latest/rhel-push-plugin > (code=exited, status=203/EXEC) > Main PID: 10291 (code=exited, status=203/EXEC) > > Apr 20 19:30:38 dell-per630-02.qe.lab.eng.nay.redhat.com systemd[1]: Started > Docker Block RHEL push plugin authZ Plugin. > Apr 20 19:30:38 dell-per630-02.qe.lab.eng.nay.redhat.com systemd[1]: > Starting Docker Block RHEL push plugin authZ Plugin... > Apr 20 19:30:38 dell-per630-02.qe.lab.eng.nay.redhat.com systemd[10291]: > Failed at step EXEC spawning /usr/libexec/docker-latest/rhel-push-plugin: No > such file or directory > Apr 20 19:30:38 dell-per630-02.qe.lab.eng.nay.redhat.com systemd[1]: > rhel-push-plugin.service: main process exited, code=exited, status=203/EXEC > Apr 20 19:30:38 dell-per630-02.qe.lab.eng.nay.redhat.com systemd[1]: Unit > rhel-push-plugin.service entered failed state. > Apr 20 19:30:38 dell-per630-02.qe.lab.eng.nay.redhat.com systemd[1]: > rhel-push-plugin.service failed. Update: we can't place the plugin socket nowhere out of /run/docker/plugins because the docker daemon always looks into this dir to find out about plugins. So we can't use /run/docker-latest/plugins unless we add a patch to the docker daemon to also look into this dir. Not sure what to do. Dan? We should not create a rhel-latest-push-plugin. All content under /run should be the same for the docker daemon. We are not creating a /run/docker-latest.sock. Ack, does that apply to all /run/docker/ also? In this case, perfect, I still need to fix docker-latest spec. Yes we don't want to change anything in the docker package, and all support tools for docker should just work. Docker storage setup and the systemd configuration for docker package is the only thing that should change. alright, so the only bug in this bugzilla was that the systemd unit file was using in its ExecStart the path /usr/libexec/docker/rhel-push-plugin but the binary itself was installed under /usr/lib/docker/rhel-push-plugin, so it fails to start. I'll send a patch to Lokesh to fix this. This plugin should be also enabled on installation and the flag added to the daemon to use it, is this possible? because this is replacing one of our patch and we should activate it by default sent a patch to Lokesh works on my RHEL 7.2 now Moving the bug to VERIFIED status per bellow testing. # rpm -q docker-rhel-push-plugin docker-latest docker-rhel-push-plugin-1.10.3-19.el7.x86_64 docker-latest-1.10.3-19.el7.x86_64 # systemctl start rhel-push-plugin # systemctl status rhel-push-plugin ● rhel-push-plugin.service - Docker Block RHEL push plugin authZ Plugin Loaded: loaded (/usr/lib/systemd/system/rhel-push-plugin.service; disabled; vendor preset: disabled) Active: active (running) since Fri 2016-04-29 16:23:11 CST; 1min 34s ago Docs: man:rhel-push-plugin(8) Main PID: 9719 (rhel-push-plugi) Memory: 8.0K CGroup: /system.slice/rhel-push-plugin.service └─9719 /usr/libexec/docker/rhel-push-plugin Apr 29 16:23:11 hp-dl360g9-04.qe.lab.eng.nay.redhat.com systemd[1]: Started Docker Block RHEL push plugin authZ Plugin. Apr 29 16:23:11 hp-dl360g9-04.qe.lab.eng.nay.redhat.com systemd[1]: Starting Docker Block RHEL push plugin authZ Plugin... Apr 29 16:24:41 hp-dl360g9-04.qe.lab.eng.nay.redhat.com systemd[1]: Started Docker Block RHEL push plugin authZ Plugin. # ps auxf|grep docker-latest | grep -i '[R|r]hel-push-plugin' root 9802 0.0 0.0 115236 1432 ? Ss 16:23 0:00 /bin/sh -c /usr/bin/docker-latest daemon --authorization-plugin=rhel-push-plugin --exec-opt native.cgroupdriver=systemd -g /var/lib/docker-latest $OPTIONS $DOCKER_STORAGE_OPTIONS $DOCKER_NETWORK_OPTIONS $INSECURE_REGISTRY 2>&1 | /usr/bin/forward-journald -tag docker-latest root 9805 0.2 0.2 636124 32336 ? Sl 16:23 0:00 \_ /usr/bin/docker-latest daemon --authorization-plugin=rhel-push-plugin --exec-opt native.cgroupdriver=systemd -g /var/lib/docker-latest --log-driver=journald --storage-driver devicemapper --storage-opt dm.fs=xfs --storage-opt dm.thinpooldev=/dev/mapper/rhel_hp--dl360g9--04-docker--latest--pool --storage-opt dm.use_deferred_removal=true --storage-opt dm.use_deferred_deletion=true Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHEA-2016-1057.html |