Bug 1428608 - undercloud fails to deploy due to docker-registry service not found
Summary: undercloud fails to deploy due to docker-registry service not found
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: instack-undercloud
Version: 11.0 (Ocata)
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: beta
: 11.0 (Ocata)
Assignee: Flavio Percoco
QA Contact: Omri Hochman
URL:
Whiteboard:
Depends On:
Blocks: 1428826
TreeView+ depends on / blocked
 
Reported: 2017-03-02 23:24 UTC by Mike Burns
Modified: 2017-05-17 20:04 UTC (History)
18 users (show)

Fixed In Version: instack-undercloud-6.0.0-0.20170220135156.d695df8.1.el7ost
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1428826 (view as bug list)
Environment:
Last Closed: 2017-05-17 20:04:25 UTC


Attachments (Terms of Use)
undercloud.log (161.40 KB, text/plain)
2017-03-02 23:26 UTC, Mike Burns
no flags Details


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHEA-2017:1245 normal SHIPPED_LIVE Red Hat OpenStack Platform 11.0 Bug Fix and Enhancement Advisory 2017-05-17 23:01:50 UTC
OpenStack gerrit 436520 None None None 2017-03-03 12:31:35 UTC
OpenStack gerrit 440842 None None None 2017-03-03 01:44:12 UTC
Red Hat Knowledge Base (Solution) 2955121 None None None 2017-03-06 09:36:05 UTC

Description Mike Burns 2017-03-02 23:24:20 UTC
Description of problem:
During undercloud deployment, docker-registry is started in puppet.  A recent docker release removed docker-registry, so the service fails to start which causes undercloud deployment to fail

Version-Release number of selected component (if applicable):
Ocata RDO
OSP 11
OSP 10

How reproducible:
Always

Steps to Reproduce:
1.deploy undercloud with default options
2.
3.

Actual results:
undercloud fails

Expected results:
undercloud install succeeds

Additional info:

Related bug 1428606 has been filed to disable docker registry by default.


Workaround:

In the undercloud.conf file
set the following:

hieradata_override = foo.yaml

in foo.yaml
# turn off docker registry
enable_docker_registry: false

Comment 1 Mike Burns 2017-03-02 23:26:59 UTC
Created attachment 1259354 [details]
undercloud.log

Comment 2 Mike Burns 2017-03-02 23:43:10 UTC
RHEL bug that dropped docker-registry:  https://bugzilla.redhat.com/show_bug.cgi?id=1388923

Comment 3 Alan Pevec 2017-03-03 01:17:35 UTC
(In reply to Mike Burns from comment #2)
> RHEL bug that dropped docker-registry: 
> https://bugzilla.redhat.com/show_bug.cgi?id=1388923

That bug says docker-registry is unmaintained since long time and docker-distribution should be used.
So bug is in tripleo that it uses docker-registry at all.
On the RPM level, docker-distribution has correct Obsoletes/Provides:
 http://pkgs.devel.redhat.com/cgit/rpms/docker-distribution/tree/docker-distribution.spec?h=extras-rhel-7.3#n54
but they missed to add compat docker-registry.service symlink :(

Comment 4 Alan Pevec 2017-03-03 01:23:15 UTC
> So bug is in tripleo that it uses docker-registry at all.

ok, that's already fixed https://github.com/openstack/puppet-tripleo/commit/a21f1a193a7076a335d82f47cab04340da9c93ff

Why is it failing then in Ocata ?!

Comment 5 Mike Burns 2017-03-03 01:44:12 UTC
(In reply to Alan Pevec from comment #4)
> > So bug is in tripleo that it uses docker-registry at all.
> 
> ok, that's already fixed
> https://github.com/openstack/puppet-tripleo/commit/
> a21f1a193a7076a335d82f47cab04340da9c93ff
> 
> Why is it failing then in Ocata ?!

The other half of the change was only in master:

https://review.openstack.org/#/c/440842/

Comment 6 Jiri Stransky 2017-03-03 12:31:35 UTC
Linking another patch that has to be backported alongside, to prevent introducing this bug:

https://bugs.launchpad.net/tripleo/+bug/1666459

Comment 7 Marios Andreou 2017-03-03 14:25:49 UTC
Adding a note here as I hit it on two environments which needed slightly different workarounds, so may be helpful for anyone else working here today as I wasted a few runs getting them both to pass. 

----
Starting with OSP10 and upgrading to OSP11, the openstack undercloud upgrade fails like:

    2017-03-03 07:04:18,606 INFO: ^[[1;31mError: /Stage[main]/Main/Service[docker-registry]/ensure: change from stopped to running failed: Systemd start for docker-registry failed!

To fix this you need the missing patch as per comment #5 so you can do:

    # https://review.openstack.org/#/c/440842/ Move Docker registry setup into its own profile
    curl https://review.openstack.org/changes/440842/revisions/current/patch?download | \
        base64 -d | sudo patch  -d /usr/share/instack-undercloud/ -p2

and then the openstack undercloud upgrade should complete.


----
Starting at OSP10 from scratch (new deployment), the openstack undercloud install fails like:

    2017-03-03 07:25:28,926 INFO: 2017-03-03 07:25:28 - ^[[1;31mError: Could not start Service[docker-registry]: Execution of '/bin/systemctl start docker-registry' returned 5: Failed to start docker-registry.service: Unit not found.^[[0m

Here the patch from 440842 above doesn't apply cleanly so I manually applied it at /usr/share/instack-undercloud/puppet-stack-config/puppet-stack.config.pp

    [root@instack puppet-stack-config]# diff puppet-stack-config.pp puppet-stack-config.pp.ORIG 
    435c435,444
    <   include ::tripleo::profile::base::docker_registry
    ---
    >   package{'docker-registry': }
    >   augeas { 'docker-registry':
    >     context => '/files/etc/sysconfig/docker-registry',
    >     changes => 'set REGISTRY_PORT 8787',
    >     notify  => Service['docker-registry'],
    >   }
    >   service { 'docker-registry':
    >     ensure  => running,
    >     require => Package['docker-registry'],
    >   }

Here I also applied the two patches that add the docker registry into puppet-tripleo as they were missing (the profile::base::docker_registry):

    # https://review.openstack.org/#/c/441124 Add Docker Registry profile
    curl https://review.openstack.org/changes/441124/revisions/current/patch?download | \
        base64 -d | sudo patch  -d /usr/share/openstack-puppet/modules/tripleo/ -p1
&&
    # https://review.openstack.org/#/c/441130/ Stop accidentally removing docker-distribution
    curl https://review.openstack.org/changes/441130/revisions/current/patch?download | \
        base64 -d | sudo patch  -d /usr/share/openstack-puppet/modules/tripleo/ -p1

Both applied cleanly for me today and then the undercloud install completes.

Comment 8 Vispy Sarkari 2017-03-04 00:23:42 UTC
I have the same issue on OSP9 and here are my results of patch application failure (440842):

curl https://review.openstack.org/changes/441124/revisions/current/patch?download | base64 -d | sudo patch  -d /usr/share/openstack-puppet/modules/tripleo/ -p1
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  4864    0  4864    0     0   3806      0 --:--:--  0:00:01 --:--:--  3808
patching file manifests/profile/base/docker_registry.pp
patching file templates/docker_distribution/registry_config.yml.erb


curl https://review.openstack.org/changes/441130/revisions/current/patch?download |  base64 -d | sudo patch  -d /usr/share/openstack-puppet/modules/tripleo/ -p1
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1740    0  1740    0     0    341      0 --:--:--  0:00:05 --:--:--   435
patching file manifests/profile/base/docker_registry.pp


curl https://review.openstack.org/changes/440842/revisions/current/patch?download | base64 -d | sudo patch  -d /usr/share/instack-undercloud/ -p2
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  2468    0  2468    0     0   1315      0 --:--:--  0:00:01 --:--:--  1314
patching file puppet-stack-config/puppet-stack-config.pp
Hunk #1 FAILED at 515.
1 out of 1 hunk FAILED -- saving rejects to file puppet-stack-config/puppet-stack-config.pp.rej

Comment 9 Ruchika K 2017-03-06 15:34:36 UTC
Facing same problem with undercloud isntall on OSP9

Comment 10 Sai Sindhur Malleni 2017-03-06 15:39:50 UTC
Facing this on OSP 10 GA.

Comment 12 Alex Krzos 2017-03-08 16:09:26 UTC
I tried the patches on an already failed OSP10 GA undercloud install and that failed as well.  I then cleaned my undercloud machine (rebuilt the RHEL OS) and redeployed a newer build with the fix in it and I was able to get passed the undercloud install.  I would recommend as a current workaround to deploy the latest with the fix rather than trying to patch.

Comment 13 Sai Sindhur Malleni 2017-03-08 16:11:10 UTC
Yes, this puddle worked for me 2017-03-03.1

Comment 14 errata-xmlrpc 2017-05-17 20:04:25 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:1245


Note You need to log in before you can comment on or make changes to this bug.