Created attachment 1108678 [details] keystone.pp.log Packstack started failing completely the Mitaka keystone setup in current CI today with the following errors: Missing parameter(s): Set a service AUTH_URL, with --os-auth-url, OS_AUTH_URL or auth.auth_url Full manifest logs in attachment. [root@n31 latest(keystone_admin)]# rpm -qa |grep puppet openstack-puppet-modules-7.0.0-dev.1b721d549994b30794f01c4457a1babab1aca58b_41994f7f.el7.centos.noarch openstack-packstack-puppet-7.0.0-dev1704.gbb19d38.el7.centos.noarch puppet-3.6.2-3.el7.noarch [root@n31 latest(keystone_admin)]# rpm -qa |grep packstack openstack-packstack-puppet-7.0.0-dev1704.gbb19d38.el7.centos.noarch openstack-packstack-7.0.0-dev1704.gbb19d38.el7.centos.noarch [root@n31 latest(keystone_admin)]# yum repolist -v Loading "fastestmirror" plugin Loading "priorities" plugin Config time: 0.015 Yum version: 3.4.3 Loading mirror speeds from cached hostfile * base: mirror.centos.org * extras: mirror.centos.org * updates: mirror.centos.org Setting up Package Sacks pkgsack time: 0.898 Repo-id : base/7/x86_64 Repo-name : CentOS-7 - Base Repo-revision: 1449700451 Repo-updated : Wed Dec 9 22:35:45 2015 Repo-pkgs : 8,955 Repo-size : 6.5 G Repo-mirrors : http://mirrorlist.centos.org/?release=7&arch=x86_64&repo=os&infra=stock Repo-baseurl : http://mirror.centos.org/centos/7/os/x86_64/ (5 more) Repo-expire : 21,600 second(s) (last: Tue Dec 22 16:42:49 2015) Repo-excluded: 52 Repo-filename: /etc/yum.repos.d/CentOS-Base.repo Repo-id : cr/7/x86_64 Repo-name : CentOS-7 - cr Repo-revision: 1449798002 Repo-updated : Fri Dec 11 01:40:02 2015 Repo-pkgs : 0 Repo-size : 0 Repo-baseurl : http://mirror.centos.org/centos/7/cr/x86_64/ Repo-expire : 21,600 second(s) (last: Tue Dec 22 16:42:49 2015) Repo-filename: /etc/yum.repos.d/CentOS-CR.repo Repo-id : delorean Repo-name : delorean-openstack-nova-afd567292ef3204313c911f98008b5a2d86106f5 Repo-revision: 1450801948 Repo-updated : Tue Dec 22 16:32:44 2015 Repo-pkgs : 304 Repo-size : 72 M Repo-baseurl : http://trunk.rdoproject.org/centos7/af/d5/afd567292ef3204313c911f98008b5a2d86106f5_e5b119db/ Repo-expire : 21,600 second(s) (last: Tue Dec 22 16:42:49 2015) Repo-excluded: 125 Repo-filename: /etc/yum.repos.d/delorean.repo Repo-id : delorean-liberty-testing/x86_64 Repo-name : delorean-liberty-testing Repo-revision: 1450800722 Repo-updated : Tue Dec 22 16:12:07 2015 Repo-pkgs : 609 Repo-size : 377 M Repo-baseurl : http://buildlogs.centos.org/centos/7/cloud/x86_64/openstack-liberty/ Repo-expire : 21,600 second(s) (last: Tue Dec 22 16:42:49 2015) Repo-excluded: 527 Repo-filename: /etc/yum.repos.d/delorean-deps.repo Repo-id : extras/7/x86_64 Repo-name : CentOS-7 - Extras Repo-revision: 1450697582 Repo-updated : Mon Dec 21 11:33:18 2015 Repo-pkgs : 173 Repo-size : 540 M Repo-mirrors : http://mirrorlist.centos.org/?release=7&arch=x86_64&repo=extras&infra=stock Repo-baseurl : http://mirror.centos.org/centos/7/extras/x86_64/ (5 more) Repo-expire : 21,600 second(s) (last: Tue Dec 22 16:42:49 2015) Repo-excluded: 18 Repo-filename: /etc/yum.repos.d/CentOS-Base.repo Repo-id : updates/7/x86_64 Repo-name : CentOS-7 - Updates Repo-revision: 1450295334 Repo-updated : Wed Dec 16 19:49:31 2015 Repo-pkgs : 369 Repo-size : 1.2 G Repo-mirrors : http://mirrorlist.centos.org/?release=7&arch=x86_64&repo=updates&infra=stock Repo-baseurl : http://mirror.centos.org/centos/7/updates/x86_64/ (5 more) Repo-expire : 21,600 second(s) (last: Tue Dec 22 16:42:49 2015) Repo-filename: /etc/yum.repos.d/CentOS-Base.repo repolist: 10,410
I have been able to trace the issue back to commit https://github.com/openstack/os-client-config/commit/22d740b7007e1182c99370cb2629322384b17a14 to os-client-config. This patch is forcing auth_type to be "token" anytime a token is used in the openstack client command line, as is the case when the puppet-keystone module is executed. The openstack client should be using "token_endpoint" authentication, so it fails, causing the whole Packstack execution to fail. Upstream bug https://bugs.launchpad.net/os-client-config/+bug/1484373 is tracking this issue.
I have proposed https://review.openstack.org/260945 as a potential fix to os-client-config. I'm not sure if it is the best approach, let's wait for the upstream review.
Did something change ? The same job that was failing yesterday with this bug now passes without problems. Failed: https://ci.centos.org/job/trown-poc-packstack-promote-rdo-mitaka/7/console Passed: https://ci.centos.org/job/trown-poc-packstack-promote-rdo-mitaka/8/console
False alarm, the job that passed used a older repository somehow (from dec 21st)
I think this bug came back in a similar form https://ci.centos.org/job/weirdo-gate-ansible-puppet-openstack-scenario002/71/consoleFull === 23:05:15 Error: /Stage[main]/Heat::Keystone::Domain/Keystone_user[heat_admin::heat]: Could not evaluate: Execution of '/usr/bin/openstack token issue --format value' returned 1: Missing parameter(s): 23:05:15 Set a scope, such as a project or domain, set a project scope with --os-project-name, OS_PROJECT_NAME or auth.project_name, set a domain scope with --os-domain-name, OS_DOMAIN_NAME or auth.domain_name === Introduced by https://github.com/openstack/os-client-config/commit/a8532f6c8d221628b697ddb0d134e2a000ef61d6 ?
Please ignore comment #5, this is a puppet-heat/puppet-keystone/openstackclient issue with related bugs: - https://bugs.launchpad.net/puppet-keystone/+bug/1535384 - https://bugs.launchpad.net/python-openstackclient/+bug/1535449
Hmm, I think this can be closed now David?