Bug 1486140

The actual failure is:
2017-08-25 09:18:12,568 INFO: ^[[1;31mError: /Stage[main]/Neutron::Keystone::Auth/Keystone::Resource::Service_identity[neutron]/Keystone_user[neutron]: Could not evaluate: Execution of '/bin/openstack domain show --format shell Default' returned 1: /usr/lib/python2.7/site-packages/keyring/backends/Gnome.py:6: PyGIWarning: GnomeKeyring was imported without specifying a version first. Use gi.require_version('GnomeKeyring', '1.0') before import to ensure that the right version gets loaded.
2017-08-25 09:18:12,569 INFO:   from gi.repository import GnomeKeyring          
2017-08-25 09:18:12,569 INFO: Could not find resource Default^[[0m  


It appears that the existence of libgnome-keyring is breaking the openstackclient which causes the install to fail.

I suspect the keyring error messages are annoying and misleading, but not causing the actual failure. When looking at the last line after it:

/bin/openstack domain show --format shell Default' returned 1:
Could not find resource Default

It looks like the Keystone domain may not exist. Then on the following lines, if we ignore the keyring warning:

Error: Could not prefetch keystone_service provider 'openstack': Execution of '/bin/openstack service list --quiet --format csv --long' returned 1: 
Failed to parse: (Redacted IP):8000

I'm standing up an OSP8 environment to confirm this theory and attempt to reproduce the issue.

Would it be possible to confirm the output of "/bin/openstack service list --quiet --format csv --long" from the command line on the undercloud, and perhaps figure out where this particular IP:PORT is defined?

Sorry, probably better to try the command with --debug rather than --quiet.

Unfortunately when you get extraneous output in openstack commands it can break the parsing in puppet since we're parsing the output to determine the values.  This was a known issue in older versions and pops up from time to time.

Thank you for the extra information Alex. I can see the warnings on a new OSP8 environment, it doesn't seem to affect the return code ($? returns 0 as expected) but if puppet parses the entire output I can see how it might cause an issue.

The undercloud install still completes successfully locally despite the CLI warnings, so I am currently unable to reproduce the original issue.

Manually applying the fix in comment 2 on bug 1259747 removes the warning on my local env. The version of python-keyring installed is python-keyring-5.0-3. Testing with python-keyring-5.0-5 also resolves the warnings issue. It would be good to test with a more recent version of the package on the affected system (if it's possible) to check if removing the warning messages is indeed enough to resolve the puppet issue too.

As an additional point of information from grepping through the sosreport logs, the "Failed to parse" IP above seems to be referencing a proxy.

It looks like there may be another way to remove the warnings without modifying python-keyring.

First, could you try:

1. $ keystone user-list

Does it work, or does it hang? If it hangs this may be related to bug 1475969

2. Could you remove/uninstall gnome-keyring and libgnome-keyring, and try running the undercloud install again?

Uninstalling libgnome-keyring also worked to remove the warnings in my environment.

3. Could you provide the output of /bin/openstack service list --debug --format csv --long?

Thank you.

Sorry, I should have been more specific in comment 9 - would it be possible to run the commands again after sourcing the credentials from stackrc?

That is:

1. $ source stackrc
$ keystone user-list

2. $ sudo yum remove gnome-keyring libgnome-keyring

Please also provide the output of: $ rpm -qa | grep keyring

Then $ openstack undercloud install

Please provide the full output of the undercloud install in a separate file, not just the end. (The file should be made private when uploading it.)

3. $ source stackrc
$ service list --debug --format csv --long

Perhaps the output can be provided in a separate file as well as it is quite long. Sensitive information is likely to be included as well so the file can be made private.

Thank you.

The undercloud install errors are still there, even after removing the warnings related to the keyring:

Error: /Stage[main]/Neutron::Keystone::Auth/Keystone::Resource::Service_identity[neutron]/Keystone_user[neutron]: Could not evaluate: Execution of '/bin/openstack domain show --format shell Default' returned 1: Could not find resource Default
Error: /Stage[main]/Heat::Keystone::Auth/Keystone::Resource::Service_identity[heat]/Keystone_user[heat]: Could not evaluate: Execution of '/bin/openstack domain show --format shell Default' returned 1: Could not find resource Default
Error: Could not prefetch keystone_service provider 'openstack': Execution of '/bin/openstack service list --quiet --format csv --long' returned 1: Failed to parse: (redacted ip)
Error: Not managing Keystone_service[glance] due to earlier Keystone API failures.
Error: /Stage[main]/Glance::Keystone::Auth/Keystone::Resource::Service_identity[glance]/Keystone_service[glance::image]/ensure: change from absent to present failed: Not managing Keystone_service[glance] due to earlier Keystone API failures.

Based on this I am moving the bug back to DFG:DF to help debug the cause (full logs linked to in the previous comment). The IP that can't be parsed seems to be related to proxy settings.

So 'Could not find resource Default' is a message from openstackclient/common/utils.py L140.  Unfortunately I was unable to reproduce this issue on a clean 7.4.  Is it possible that the customer has installed a newer version of openstackclient on the undercloud (via pip)?  When I try and run '/bin/openstack domain show --format shell Default' that is not actually available on the version of openstackclient we shipped with 8. 


[stack@undercloud-0 ~]$ /bin/openstack domain show --format shell Default
/usr/lib/python2.7/site-packages/keyring/backends/Gnome.py:6: PyGIWarning: GnomeKeyring was imported without specifying a version first. Use gi.require_version('GnomeKeyring', '1.0') before import to ensure that the right version gets loaded.
  from gi.repository import GnomeKeyring
openstack: 'domain' is not an openstack command. See 'openstack --help'.
Did you mean one of these?
  command list
  container create
  container delete
  container list
  container save
  container show
[stack@undercloud-0 ~]$ echo $?
2
[stack@undercloud-0 ~]$ 


Additionally they don't have any OS_* vars set in their environment when they're running this command do they?  The only thing I could think is that the install is attempting to use keystone v3 which was not the default for OSP8 and should not be triggered during the install.

Hi Alex,

Is there a way to find out why puppet was trying to call openstack domain show (which shouldn't appear in OSP8) ? 

Error: /Stage[main]/Neutron::Keystone::Auth/Keystone::Resource::Service_identity[neutron]/Keystone_user[neutron]: Could not evaluate: Execution of '/bin/openstack domain show --format shell Default

I tried to have a look at the /usr/share/openstack-puppet/modules/keystone/manifests/resource/service_identity.pp but I'm sure there are many other places which need to check as well. Or do you need any puppet files from customer's environment ?

Best Regards,
Chen

Chen, the domain show is executed in the keystone_user provider. The question I have is how is that running with that error message as I spun up a fresh new environment and did not have that command if they had the same versions of everything as we ship they would have gotten the error message like in Comment 16.  Something is not right with the environment but I'm unable to determine what based on the rpm information.  

Additionally the fix for LP#1549867 was for mitaka which would be OSP9.  The bug was reported for 8, is it actually 9 or maybe we're getting a mix of 8/9 versions?  I haven't had time to version check all the components. I'm still trying to reproduce the problem but I'm currently unable to.

Hi Alex,

Thank you for your help.

I can not reproduce either even though I installed every rpm package where the customer has installed while I hasn't after comparing the installed rpms.

Is there any other method to proceed the investigation for example, to find out why the code goes to v3 credential path as we met big problem reproducing the issue ?

Best Regards,
Chen

Hi Alex,

Since that should be a fresh installation, then where could the non-ascii be created/used ?

Shall we dig into the keystone_user provider to figure out in what circumstances the code will go to v3 path ? Does keystone_user provider mean openstack-puppet/modules/keystone/lib/puppet/provider/keystone_user/openstack.rb ? 

Best Regards,
Chen

Hi Alex,

Do you have findings which could share with the customer ?

Best Regards,
Chen

I'm trying to reproduce this again. I did find from the sosreport that the hostname still appears to be localhost.localdomain which is probbaly not right. I tested that and it's still working fine.


I've tried a VM from the 7.4 qcow2 images via the website and installing from scratch using the DVD and enabling virt when installing.  Neither has successfully reproduced this case.   

You said they have a proxy, is this being set via environment params before they run the undercloud install?  For example, do they have http_proxy environment var set? If so that might be the cause

I attempted to test the http_proxy environment variable and it would appear that it is not the http_proxy environment setting because that results in a different error in the install process.  What system configurations are changed specifically to support the proxy?

Closing this issue due to lack of response. The issue seems to be related to incorrect proxy configuration.  Feel free to reopen if there is additional information.