Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1937458

Summary: debugging ironic with auth_type http_basic not possible due to: 'HTTPBasicAuth' object has no attribute 'get_auth_ref'
Product: OpenShift Container Platform Reporter: Andreas Karis <akaris>
Component: DocumentationAssignee: avbhatt
Status: CLOSED CURRENTRELEASE QA Contact: Xiaoli Tian <xtian>
Severity: medium Docs Contact: Tomas 'Sheldon' Radej <tradej>
Priority: medium    
Version: 4.7CC: avbhatt, derekh
Target Milestone: ---   
Target Release: 4.8.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-10-26 13:46:35 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Andreas Karis 2021-03-10 17:04:45 UTC
OCP 4.7-latest, installed today:

[root@localhost ~]# cat /var/opt/metal3/auth/clouds.yaml
clouds:
  metal3-bootstrap:
    auth_type: http_basic
    username: bootstrap-user
    password: TlR6ize2yugy8OGx
    baremetal_endpoint_override: http://192.168.123.25:6385/v1
    baremetal_introspection_endpoint_override: http://192.168.123.25:5050/v1

cp /var/opt/metal3/auth/clouds.yaml /tmp/clouds.yaml
export CLOUDS_YAML=/tmp/clouds.yaml

Then:
~~~
[root@localhost ~]# podman run -ti --rm --entrypoint /bin/bash -v $CLOUDS_YAML:/clouds.yaml -e OS_CLOUD=metal3-bootstrap quay.io/metal3-io/ironic-client
[root@2e679b2c682a /]# openstack baremetal node list
'HTTPBasicAuth' object has no attribute 'get_auth_ref'
~~~

With --debug output:
~~~
oot@2e679b2c682a /]# openstack --debug baremetal node list
START with options: --debug baremetal node list
options: Namespace(access_token='***', access_token_endpoint='', access_token_type='', application_credential_id='', application_credential_name='', application_credential_secret='***', auth_methods='', auth_type='', auth_url='', cacert=None, cert='', client_id='', client_secret='***', cloud='metal3-bootstrap', code='', debug=True, default_domain='default', default_domain_id='', default_domain_name='', deferred_help=False, discovery_endpoint='', domain_id='', domain_name='', endpoint='', identity_provider='', insecure=None, inspector_api_version='1', inspector_url=None, interface='public', key='', log_file=None, openid_scope='', os_baremetal_api_version='1.67', os_beta_command=False, os_compute_api_version='', os_identity_api_version='', os_image_api_version='', os_network_api_version='', os_object_api_version='', os_project_id=None, os_project_name=None, os_volume_api_version='', passcode='', password='***', project_domain_id='', project_domain_name='', project_id='', project_name='', protocol='', redirect_uri='', region_name='', remote_project_domain_id='', remote_project_domain_name='', remote_project_id='', remote_project_name='', service_provider='', system_scope='', timing=False, token='***', trust_id='', user_domain_id='', user_domain_name='', user_id='', username='', verbose_level=3, verify=None)
Auth plugin http_basic selected
auth_config_hook(): {'api_timeout': None, 'verify': True, 'cacert': None, 'cert': None, 'key': None, 'baremetal_status_code_retries': '5', 'baremetal_introspection_status_code_retries': '5', 'image_status_code_retries': '5', 'disable_vendor_agent': {}, 'interface': 'public', 'floating_ip_source': 'neutron', 'image_api_use_tasks': False, 'image_format': 'qcow2', 'message': '', 'network_api_version': '2', 'object_store_api_version': '1', 'secgroup_source': 'neutron', 'status': 'active', 'auth': {}, 'username': 'bootstrap-user', 'password': '***', 'baremetal_endpoint_override': 'http://192.168.123.25:6385/v1', 'baremetal_introspection_endpoint_override': 'http://192.168.123.25:5050/v1', 'verbose_level': 3, 'deferred_help': False, 'debug': True, 'cloud': 'metal3-bootstrap', 'default_domain': 'default', 'timing': False, 'inspector_api_version': '1', 'beta_command': False, 'baremetal_api_version': '1.67', 'region_name': '', 'auth_type': 'http_basic', 'networks': []}
defaults: {'api_timeout': None, 'verify': True, 'cacert': None, 'cert': None, 'key': None, 'auth_type': 'password', 'baremetal_status_code_retries': 5, 'baremetal_introspection_status_code_retries': 5, 'image_status_code_retries': 5, 'disable_vendor_agent': {}, 'interface': None, 'floating_ip_source': 'neutron', 'image_api_use_tasks': False, 'image_format': 'qcow2', 'message': '', 'network_api_version': '2', 'object_store_api_version': '1', 'secgroup_source': 'neutron', 'status': 'active'}
cloud cfg: {'api_timeout': None, 'verify': True, 'cacert': None, 'cert': None, 'key': None, 'baremetal_status_code_retries': '5', 'baremetal_introspection_status_code_retries': '5', 'image_status_code_retries': '5', 'disable_vendor_agent': {}, 'interface': 'public', 'floating_ip_source': 'neutron', 'image_api_use_tasks': False, 'image_format': 'qcow2', 'message': '', 'network_api_version': '2', 'object_store_api_version': '1', 'secgroup_source': 'neutron', 'status': 'active', 'auth': {}, 'username': 'bootstrap-user', 'password': '***', 'baremetal_endpoint_override': 'http://192.168.123.25:6385/v1', 'baremetal_introspection_endpoint_override': 'http://192.168.123.25:5050/v1', 'verbose_level': 3, 'deferred_help': False, 'debug': True, 'cloud': 'metal3-bootstrap', 'default_domain': 'default', 'timing': False, 'inspector_api_version': '1', 'beta_command': False, 'baremetal_api_version': '1.67', 'region_name': '', 'auth_type': 'http_basic', 'networks': []}
compute API version 2.1, cmd group openstack.compute.v2
identity API version 3, cmd group openstack.identity.v3
image API version 2, cmd group openstack.image.v2
network API version 2, cmd group openstack.network.v2
object_store API version 1, cmd group openstack.object_store.v1
volume API version 3, cmd group openstack.volume.v3
baremetal_introspection API version 1, cmd group openstack.baremetal_introspection.v1
baremetal API version 1.67, cmd group openstack.baremetal.v1
neutronclient API version 2, cmd group openstack.neutronclient.v2
command: baremetal node list -> ironicclient.osc.v1.baremetal_node.ListBaremetalNode (auth=True)
Auth plugin http_basic selected
auth_config_hook(): {'api_timeout': None, 'verify': True, 'cacert': None, 'cert': None, 'key': None, 'baremetal_status_code_retries': '5', 'baremetal_introspection_status_code_retries': '5', 'image_status_code_retries': '5', 'disable_vendor_agent': {}, 'interface': 'public', 'floating_ip_source': 'neutron', 'image_api_use_tasks': False, 'image_format': 'qcow2', 'message': '', 'network_api_version': '2', 'object_store_api_version': '1', 'secgroup_source': 'neutron', 'status': 'active', 'auth': {}, 'username': 'bootstrap-user', 'password': '***', 'baremetal_endpoint_override': 'http://192.168.123.25:6385/v1', 'baremetal_introspection_endpoint_override': 'http://192.168.123.25:5050/v1', 'additional_user_agent': [('osc-lib', '2.2.1')], 'verbose_level': 3, 'deferred_help': False, 'debug': True, 'cloud': 'metal3-bootstrap', 'default_domain': 'default', 'timing': False, 'inspector_api_version': '1', 'beta_command': False, 'baremetal_api_version': '1.67', 'region_name': '', 'auth_type': 'http_basic', 'networks': []}
Using auth plugin: http_basic
Using parameters {'username': 'bootstrap-user', 'password': '***'}
Get auth_ref
'HTTPBasicAuth' object has no attribute 'get_auth_ref'
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/cliff/app.py", line 393, in run_subcommand
    self.prepare_to_run_command(cmd)
  File "/usr/lib/python3.6/site-packages/osc_lib/shell.py", line 492, in prepare_to_run_command
    self.client_manager.auth_ref
  File "/usr/lib/python3.6/site-packages/osc_lib/clientmanager.py", line 199, in auth_ref
    self._auth_ref = self.auth.get_auth_ref(self.session)
AttributeError: 'HTTPBasicAuth' object has no attribute 'get_auth_ref'
clean_up ListBaremetalNode: 'HTTPBasicAuth' object has no attribute 'get_auth_ref'
END return value: 1
[root@2e679b2c682a /]# 
~~~

Comment 2 Andreas Karis 2021-03-10 17:38:20 UTC
This seems to be a dependency issue. I can use pip on my jumpserver and connect to ironic:

Copy clouds.yaml from the jumpserver in /var/opt/metal3/auth/clouds.yaml to the jumpserver which can reach the bootstrap node.

Then, use virtualenv and run:
~~~
virtualenv openstack
source openstack/bin/activate
pip install python-openstackclient
pip install python-ironicclient
export OS_CLOUD=metal3-bootstrap
~~~

Then:
~~~
(openstack) [root@openshift-jumpserver-0 ~]# cat clouds.yaml 
clouds:
  metal3-bootstrap:
    auth_type: http_basic
    username: bootstrap-user
    password: TlR6ize2yugy8OGx
    baremetal_endpoint_override: http://192.168.123.25:6385/v1
    baremetal_introspection_endpoint_override: http://192.168.123.25:5050/v1
(openstack) [root@openshift-jumpserver-0 ~]# export OS_CLOUD=metal3-bootstrap
(openstack) [root@openshift-jumpserver-0 ~]# openstack baremetal node list
+--------------------------------------+--------------------+---------------+-------------+--------------------+-------------+
| UUID                                 | Name               | Instance UUID | Power State | Provisioning State | Maintenance |
+--------------------------------------+--------------------+---------------+-------------+--------------------+-------------+
| b02583bc-a767-4ed2-806f-bf55b65ee1b2 | openshift-master-0 | None          | power on    | inspect failed     | False       |
| 6f618ff4-e0fa-4b7b-8699-0830354c54fb | openshift-master-2 | None          | power on    | inspect failed     | False       |
| b57aa939-a4b5-4f29-b322-f7ce2c12d65a | openshift-master-1 | None          | power on    | inspect failed     | False       |
+--------------------------------------+--------------------+---------------+-------------+--------------------+-------------+
~~~

Comment 3 Andreas Karis 2021-03-12 09:58:50 UTC
For completeness - I'm obviously using the instructions from: https://openshift-kni.github.io/baremetal-deploy/4.7/Troubleshooting.html

Comment 4 Derek Higgins 2021-03-16 16:49:06 UTC
(In reply to Andreas Karis from comment #0)
> OCP 4.7-latest, installed today:
> 
> [root@localhost ~]# cat /var/opt/metal3/auth/clouds.yaml
> clouds:
>   metal3-bootstrap:
>     auth_type: http_basic
>     username: bootstrap-user
>     password: TlR6ize2yugy8OGx
>     baremetal_endpoint_override: http://192.168.123.25:6385/v1
>     baremetal_introspection_endpoint_override: http://192.168.123.25:5050/v1
> 
> cp /var/opt/metal3/auth/clouds.yaml /tmp/clouds.yaml
> export CLOUDS_YAML=/tmp/clouds.yaml
> 
> Then:
> ~~~
> [root@localhost ~]# podman run -ti --rm --entrypoint /bin/bash -v
> $CLOUDS_YAML:/clouds.yaml -e OS_CLOUD=metal3-bootstrap
> quay.io/metal3-io/ironic-client
> [root@2e679b2c682a /]# openstack baremetal node list
> 'HTTPBasicAuth' object has no attribute 'get_auth_ref'


Looks like this is likely related using the ironic client vs the openstack cli,
can you try the baremetal command e.g.

[derekh@u07 dev-scripts]$ podman run -ti --rm -v $PWD/clouds.yaml:/clouds.yaml --entrypoint bash -e OS_CLOUD=metal3 quay.io/metal3-io/ironic-client 
[root@c807496e0f71 /]# openstack baremetal node list
'HTTPBasicAuth' object has no attribute 'get_auth_ref'

[root@c807496e0f71 /]# baremetal node list
+--------------------------------------+-----------------+--------------------------------------+-------------+--------------------+-------------+
| UUID                                 | Name            | Instance UUID                        | Power State | Provisioning State | Maintenance |
+--------------------------------------+-----------------+--------------------------------------+-------------+--------------------+-------------+
| 116cd4e2-0b78-40c1-bfba-57fd22e2d120 | ostest-worker-1 | 17cb754a-1711-4e6b-a235-2ee86fdfb43b | power on    | active             | False       |
| 61a15fc7-d505-499e-b832-3ac32d0a884e | ostest-master-1 | 1bc0b67d-d6db-4943-abba-aef6aeb582e3 | power on    | active             | False       |
| 48146f83-0f52-4625-b0e0-d078d4faec38 | ostest-worker-0 | 167bc747-dc23-428d-8444-3e72abceae35 | power on    | active             | False       |
| eab02f68-b63a-4c55-b181-d741fa922d2c | ostest-master-0 | 4b805e26-2352-4882-9177-3f5d84f05b25 | power on    | active             | False       |
| 83c237a1-370b-48c2-9868-3a97c601cbf6 | ostest-master-2 | 8c1a9eca-7246-4f8a-a5ee-f48a18695faf | power on    | active             | False       |
+--------------------------------------+-----------------+--------------------------------------+-------------+--------------------+-------------+
[root@c807496e0f71 /]# exit

Comment 5 Dmitry Tantsur 2021-03-16 17:02:56 UTC
> openstack --debug baremetal node list

We really need to stop using `openstack` CLI. Could you try with just

> baremetal --debug node list

Comment 6 Andreas Karis 2021-03-16 20:36:06 UTC
Yep: 

[core@localhost ~]$ sudo -i
[root@localhost ~]#  cat /var/opt/metal3/auth/clouds.yaml
clouds:
  metal3-bootstrap:
    auth_type: http_basic
    username: bootstrap-user
    password: I48hK2pRRuaIu07Q
    baremetal_endpoint_override: http://192.168.123.25:6385/v1
    baremetal_introspection_endpoint_override: http://192.168.123.25:5050/v1
[root@localhost ~]# cp /var/opt/metal3/auth/clouds.yaml /tmp/clouds.yaml
[root@localhost ~]# export CLOUDS_YAML=/tmp/clouds.yaml
[root@localhost ~]# 
[root@localhost ~]# podman run -ti --rm --entrypoint /bin/bash -v $CLOUDS_YAML:/clouds.yaml -e OS_CLOUD=metal3-bootstrap quay.io/metal3-io/ironic-client
Trying to pull quay.io/metal3-io/ironic-client...
Getting image source signatures
Copying blob 3c72a8ed6814 done  
Copying blob ef37e9b020f7 done  
Copying config b7c49cdff3 done  
Writing manifest to image destination
Storing signatures
[root@516e9a141721 /]# baremetal node list
+--------------------------------------+--------------------+---------------+-------------+--------------------+-------------+
| UUID                                 | Name               | Instance UUID | Power State | Provisioning State | Maintenance |
+--------------------------------------+--------------------+---------------+-------------+--------------------+-------------+
| 6d5378e7-549f-4941-bdc8-c3c095162a8c | openshift-master-1 | None          | power off   | inspecting         | False       |
| 92fb52d4-3c71-4056-911f-b0355a55de43 | openshift-master-0 | None          | power off   | inspecting         | False       |
| 72ef9fe2-5068-432a-9fef-c848123b2368 | openshift-master-2 | None          | power off   | inspecting         | False       |
+--------------------------------------+--------------------+---------------+-------------+--------------------+-------------+

So the problem are the instructions in the upstream guide: https://openshift-kni.github.io/baremetal-deploy/4.7/Troubleshooting.html

Thanks for the clarification, feel free to close this :-)

Comment 7 Dmitry Tantsur 2021-03-17 09:41:49 UTC
> https://openshift-kni.github.io/baremetal-deploy/4.7/Troubleshooting.html

I'm not sure I'm tagging the correct documentation team, but this document should not reference "openstack baremetal" any more, just "baremetal".

Comment 15 avbhatt 2022-10-17 11:38:22 UTC
Submitted https://github.com/openshift-kni/baremetal-deploy/pull/908 PR to the upstream repo, which is now merged.