Bug 1996816 - Cluster upgrade fails with: 'OAuthException invalid_grant: The provided authorization grant for the auth code has expired.
Summary: Cluster upgrade fails with: 'OAuthException invalid_grant: The provided autho...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: ovirt-ansible-collection
Classification: oVirt
Component: General
Version: unspecified
Hardware: x86_64
OS: Linux
high
high
Target Milestone: ovirt-4.4.8-1
: 1.6.1
Assignee: Martin Necas
QA Contact: Lukas Svaty
URL:
Whiteboard:
: 1996990 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-08-23 18:07 UTC by Lorenzo Sartoratti
Modified: 2021-09-24 09:27 UTC (History)
6 users (show)

Fixed In Version: ovirt-ansible-collection-1.6.1
Clone Of:
Environment:
Last Closed: 2021-09-24 09:27:42 UTC
oVirt Team: Infra
Embargoed:
sbonazzo: ovirt-4.4?
michal.skrivanek: blocker+
sbonazzo: planning_ack?
pm-rhel: devel_ack+
pm-rhel: testing_ack+


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github oVirt ovirt-ansible-collection pull 332 0 None None None 2021-08-25 06:42:44 UTC
Red Hat Issue Tracker RHV-43095 0 None None None 2021-08-23 18:10:07 UTC

Description Lorenzo Sartoratti 2021-08-23 18:07:56 UTC
Hi
we have updated in our cluster the hosted-engine and hosts to 4.4.8.
Having to raise cluster compatibility to 4.6,
the web operation fails, presumably due to a problem with authentication
I enclose the log extracts
Thanks in advance!

Lorenzo Sartoratti

--------------------------------
/var/log/ovirt-engine/server.log

2021-08-23 16:59:41,241+02 ERROR [org.jboss.threads.errors] (default
task-7) Thread Thread[default task-7,5,main] threw an uncaught
exception: java.lang.RuntimeException:
org.ovirt.engine.core.common.utils.ansible.AnsibleRunnerCallException:
Task Get API info failed to execute. Please check logs for more details:
/var/log/ovirt-engine/ansible/ansible-20210823165938-ovirt-cluster-upgrade_yml.log
         at
io.undertow.servlet.5.Final//io.undertow.servlet.spec.ServletContextImpl.invokeRunnable(ServletContextImpl.java:1060)
         at
io.undertow.servlet.5.Final//io.undertow.servlet.spec.AsyncContextImpl$4.run(AsyncContextImpl.java:326)
         at
org.jboss.threads.0.Final//org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
         at
org.jboss.threads.0.Final//org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1990)
         at
org.jboss.threads.0.Final//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1486)
         at
org.jboss.threads.0.Final//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1377)
         at
org.jboss.xnio.4.Final//org.xnio.XnioWorker$WorkerThreadFactory$1$1.run(XnioWorker.java:1280)
         at java.base/java.lang.Thread.run(Thread.java:829)
Caused by:
org.ovirt.engine.core.common.utils.ansible.AnsibleRunnerCallException:
Task Get API info failed to execute. Please check logs for more details:
/var/log/ovirt-engine/ansible/ansible-20210823165938-ovirt-cluster-upgrade_yml.log
         at
deployment.engine.ear.bll.jar//org.ovirt.engine.core.common.utils.ansible.AnsibleRunnerHttpClient.processEvents(AnsibleRunnerHttpClient.java:251)
         at
deployment.engine.ear.bll.jar//org.ovirt.engine.core.common.utils.ansible.AnsibleExecutor.runCommand(AnsibleExecutor.java:227)
         at
deployment.engine.ear.bll.jar//org.ovirt.engine.core.common.utils.ansible.AnsibleExecutor.runCommand(AnsibleExecutor.java:74)
         at
deployment.engine.ear.bll.jar//org.ovirt.engine.core.common.utils.ansible.AnsibleExecutor.runCommand(AnsibleExecutor.java:59)
         at
deployment.engine.ear.services.war//org.ovirt.engine.core.services.AnsibleServlet.lambda$doPost$0(AnsibleServlet.java:92)
         at
io.undertow.servlet.5.Final//io.undertow.servlet.spec.ServletContextImpl$2.call(ServletContextImpl.java:181)
         at
io.undertow.servlet.5.Final//io.undertow.servlet.spec.ServletContextImpl$2.call(ServletContextImpl.java:178)
         at
io.undertow.servlet.5.Final//io.undertow.servlet.spec.ServletContextImpl.invokeRunnable(ServletContextImpl.java:1058)


--------------------------------------
/var/log/ovirt-engine/ansible/ansible-20210823165938-ovirt-cluster-upgrade_yml.log


2021-08-23 16:59:41 CEST - TASK [ovirt.ovirt.cluster_upgrade : Get API
info] ******************************
2021-08-23 16:59:41 CEST -
2021-08-23 16:59:41 CEST - An exception occurred during task execution.
To see the full traceback, use -vvv. The error was: ovirtsdk4.AuthError:
Error during SSO authentication access_denied : Cannot authenticate u
ser 'None@N/A': No valid profile found in credentials..
fatal: [localhost]: FAILED! => {"changed": false, "msg": "Error during
SSO authentication access_denied : Cannot authenticate user 'None@N/A':
No valid profile found in credentials.."}
2021-08-23 16:59:41 CEST - {
   "status" : "OK",
   "msg" : "",
   "data" : {
     "uuid" : "74f26212-9717-4f86-9186-90737f5b51ec",
     "counter" : 11,
     "stdout" : "An exception occurred during task execution. To see the
full traceback, use -vvv. The error was: ovirtsdk4.AuthError: Error
during SSO authentication access_denied : Cannot authenticate user 'None@N
/A': No valid profile found in credentials..\r\nfatal: [localhost]:
FAILED! => {\"changed\": false, \"msg\": \"Error during SSO
authentication access_denied : Cannot authenticate user 'None@N/A': No
valid profile f
ound in credentials..\"}",
     "start_line" : 10,
     "end_line" : 12,
     "runner_ident" : "bca07b9a-0422-11ec-b999-00163e4e8aba",
     "event" : "runner_on_failed",
     "pid" : 8233,
     "created" : "2021-08-23T14:59:40.219831",
     "parent_uuid" : "00163e4e-8aba-6f50-17c2-000000000014",
     "event_data" : {
       "playbook" : "ovirt-cluster-upgrade.yml",
       "playbook_uuid" : "46b70a9d-6673-4a8a-b283-299a08e677c2",
       "play" : "oVirt cluster upgrade wizard target",
       "play_uuid" : "00163e4e-8aba-6f50-17c2-00000000000c",
       "play_pattern" : "localhost",
       "task" : "Get API info",
       "task_uuid" : "00163e4e-8aba-6f50-17c2-000000000014",
       "task_action" : "ovirt_api_info",
       "task_args" : "",
       "task_path" :
"/usr/share/ansible/collections/ansible_collections/ovirt/ovirt/roles/cluster_upgrade/tasks/main.yml:27",
       "role" : "cluster_upgrade",
       "host" : "localhost",
       "remote_addr" : "127.0.0.1",
       "res" : {
         "msg" : "Error during SSO authentication access_denied : Cannot
authenticate user 'None@N/A': No valid profile found in credentials..",
         "exception" : "Traceback (most recent call last):\n  File
\"/tmp/ansible_ovirt_api_info_payload_8fukbrop/ansible_ovirt_api_info_payload.zip/ansible_collections/ovirt/ovirt/plugins/modules/ovirt_api_info.py\",
line 73, in main\n  File
\"/usr/lib64/python3.6/site-packages/ovirtsdk4/services.py\", line
27517, in get\n    return self._internal_get(headers, query, wait)\n 
File \"/usr/lib64/python3.6/site-packages/ovirtsdk4/service.py\", line
211, in _internal_get\n    return future.wait() if wait else future\n 
File \"/usr/lib64/python3.6/site-packages/ovirtsdk4/service.py\", line
54, in wait\n    response = self._connection.wait(self._context)\n File
\"/usr/lib64/python3.6/site-packages/ovirtsdk4/__init__.py\", line 496,
in wait\n    return self.__wait(context, failed_auth)\n File
\"/usr/lib64/python3.6/site-packages/ovirtsdk4/__init__.py\", line 526,
in __wait\n    self._sso_token = self._get_access_token()\n  File
\"/usr/lib64/python3.6/site-packages/ovirtsdk4/__init__.py\", line 627,
in _get_access_token\n    sso_error[1]\novirtsdk4.AuthError: Error
during SSO authentication access_denied : Cannot authenticate user
'None@N/A': No valid profile found in credentials..\n",
         "invocation" : {
           "module_args" : {
             "fetch_nested" : false,
             "nested_attributes" : [ ],
             "follows" : [ ]
           }
         },
         "_ansible_no_log" : false,
         "changed" : false
       },
       "start" : "2021-08-23T14:59:39.604161",
       "end" : "2021-08-23T14:59:40.219582",
       "duration" : 0.615421,
       "ignore_errors" : null,
       "event_loop" : null,
       "uuid" : "74f26212-9717-4f86-9186-90737f5b51ec"
     }
   }
}

Comment 1 RHEL Program Management 2021-08-24 05:50:36 UTC
The documentation text flag should only be set after 'doc text' field is provided. Please provide the documentation text and set the flag to '?' again.

Comment 2 Dana 2021-08-24 07:10:12 UTC
Please attach the full log file - /var/log/ovirt-engine/ansible/ansible-20210823165938-ovirt-cluster-upgrade_yml.log

Comment 3 Lorenzo Sartoratti 2021-08-24 07:27:05 UTC
2021-08-23 16:59:41 CEST - 
2021-08-23 16:59:41 CEST - PLAY [oVirt cluster upgrade wizard target] *************************************
2021-08-23 16:59:41 CEST - TASK [ovirt.ovirt.cluster_upgrade : Initialize variables] **********************
2021-08-23 16:59:41 CEST - 
2021-08-23 16:59:41 CEST - ok: [localhost]
2021-08-23 16:59:41 CEST - {
  "status" : "OK",
  "msg" : "",
  "data" : {
    "uuid" : "4db4b354-ab3d-45fa-aaba-381c9244ae98",
    "counter" : 5,
    "stdout" : "ok: [localhost]",
    "start_line" : 4,
    "end_line" : 5,
    "runner_ident" : "bca07b9a-0422-11ec-b999-00163e4e8aba",
    "event" : "runner_on_ok",
    "pid" : 8233,
    "created" : "2021-08-23T14:59:39.054669",
    "parent_uuid" : "00163e4e-8aba-6f50-17c2-000000000011",
    "event_data" : {
      "playbook" : "ovirt-cluster-upgrade.yml",
      "playbook_uuid" : "46b70a9d-6673-4a8a-b283-299a08e677c2",
      "play" : "oVirt cluster upgrade wizard target",
      "play_uuid" : "00163e4e-8aba-6f50-17c2-00000000000c",
      "play_pattern" : "localhost",
      "task" : "Initialize variables",
      "task_uuid" : "00163e4e-8aba-6f50-17c2-000000000011",
      "task_action" : "set_fact",
      "task_args" : "",
      "task_path" : "/usr/share/ansible/collections/ansible_collections/ovirt/ovirt/roles/cluster_upgrade/tasks/main.yml:5",
      "role" : "cluster_upgrade",
      "host" : "localhost",
      "remote_addr" : "127.0.0.1",
      "res" : {
        "changed" : false,
        "ansible_facts" : {
          "stop_non_migratable_vms" : true,
          "provided_token" : "lUXEkJJgi7emCB1YGjLmgwUQTk85_lmIRwhFjuJDMcUTE6c59nGLaYsJhWwhV-UYKHlT0JxsokTH66bBjQZ32Q"
        },
        "_ansible_no_log" : false
      },
      "start" : "2021-08-23T14:59:38.981378",
      "end" : "2021-08-23T14:59:39.054467",
      "duration" : 0.073089,
      "event_loop" : null,
      "uuid" : "4db4b354-ab3d-45fa-aaba-381c9244ae98"
    }
  }
}

2021-08-23 16:59:41 CEST - TASK [ovirt.ovirt.cluster_upgrade : Login to oVirt] ****************************
2021-08-23 16:59:41 CEST - 
2021-08-23 16:59:41 CEST - ok: [localhost]
2021-08-23 16:59:41 CEST - {
  "status" : "OK",
  "msg" : "",
  "data" : {
    "uuid" : "b82cb8f0-2589-449d-979a-15612c3ba587",
    "counter" : 8,
    "stdout" : "ok: [localhost]",
    "start_line" : 7,
    "end_line" : 8,
    "runner_ident" : "bca07b9a-0422-11ec-b999-00163e4e8aba",
    "event" : "runner_on_ok",
    "pid" : 8233,
    "created" : "2021-08-23T14:59:39.598370",
    "parent_uuid" : "00163e4e-8aba-6f50-17c2-000000000013",
    "event_data" : {
      "playbook" : "ovirt-cluster-upgrade.yml",
      "playbook_uuid" : "46b70a9d-6673-4a8a-b283-299a08e677c2",
      "play" : "oVirt cluster upgrade wizard target",
      "play_uuid" : "00163e4e-8aba-6f50-17c2-00000000000c",
      "play_pattern" : "localhost",
      "task" : "Login to oVirt",
      "task_uuid" : "00163e4e-8aba-6f50-17c2-000000000013",
      "task_action" : "ovirt_auth",
      "task_args" : "",
      "task_path" : "/usr/share/ansible/collections/ansible_collections/ovirt/ovirt/roles/cluster_upgrade/tasks/main.yml:11",
      "role" : "cluster_upgrade",
      "host" : "localhost",
      "remote_addr" : "127.0.0.1",
      "res" : {
        "changed" : false,
        "ansible_facts" : {
          "ovirt_auth" : {
            "token" : "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER",
            "url" : "https://ovirt.localdomain:443/ovirt-engine/api",
            "ca_file" : null,
            "insecure" : true,
            "timeout" : 0,
            "compress" : true,
            "kerberos" : false,
            "headers" : { }
          }
        },
        "invocation" : {
          "module_args" : {
            "url" : "https://ovirt.localdomain:443/ovirt-engine/api",
            "username" : "",
            "hostname" : "",
            "password" : "",
            "ca_file" : "",
            "token" : "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER",
            "insecure" : true,
            "headers" : { },
            "timeout" : 0,
            "compress" : true,
            "kerberos" : false,
            "state" : "present",
            "ovirt_auth" : null
          }
        },
        "_ansible_no_log" : false
      },
      "start" : "2021-08-23T14:59:39.059610",
      "end" : "2021-08-23T14:59:39.598056",
      "duration" : 0.538446,
      "event_loop" : null,
      "uuid" : "b82cb8f0-2589-449d-979a-15612c3ba587"
    }
  }
}

2021-08-23 16:59:41 CEST - TASK [ovirt.ovirt.cluster_upgrade : Get API info] ******************************
2021-08-23 16:59:41 CEST - 
2021-08-23 16:59:41 CEST - An exception occurred during task execution. To see the full traceback, use -vvv. The error was: ovirtsdk4.AuthError: Error during SSO authentication access_denied : Cannot authenticate user 'None@N/A': No valid profile found in credentials..
fatal: [localhost]: FAILED! => {"changed": false, "msg": "Error during SSO authentication access_denied : Cannot authenticate user 'None@N/A': No valid profile found in credentials.."}
2021-08-23 16:59:41 CEST - {
  "status" : "OK",
  "msg" : "",
  "data" : {
    "uuid" : "74f26212-9717-4f86-9186-90737f5b51ec",
    "counter" : 11,
    "stdout" : "An exception occurred during task execution. To see the full traceback, use -vvv. The error was: ovirtsdk4.AuthError: Error during SSO authentication access_denied : Cannot authenticate user 'None@N/A': No valid profile found in credentials..\r\nfatal: [localhost]: FAILED! => {\"changed\": false, \"msg\": \"Error during SSO authentication access_denied : Cannot authenticate user 'None@N/A': No valid profile found in credentials..\"}",
    "start_line" : 10,
    "end_line" : 12,
    "runner_ident" : "bca07b9a-0422-11ec-b999-00163e4e8aba",
    "event" : "runner_on_failed",
    "pid" : 8233,
    "created" : "2021-08-23T14:59:40.219831",
    "parent_uuid" : "00163e4e-8aba-6f50-17c2-000000000014",
    "event_data" : {
      "playbook" : "ovirt-cluster-upgrade.yml",
      "playbook_uuid" : "46b70a9d-6673-4a8a-b283-299a08e677c2",
      "play" : "oVirt cluster upgrade wizard target",
      "play_uuid" : "00163e4e-8aba-6f50-17c2-00000000000c",
      "play_pattern" : "localhost",
      "task" : "Get API info",
      "task_uuid" : "00163e4e-8aba-6f50-17c2-000000000014",
      "task_action" : "ovirt_api_info",
      "task_args" : "",
      "task_path" : "/usr/share/ansible/collections/ansible_collections/ovirt/ovirt/roles/cluster_upgrade/tasks/main.yml:27",
      "role" : "cluster_upgrade",
      "host" : "localhost",
      "remote_addr" : "127.0.0.1",
      "res" : {
        "msg" : "Error during SSO authentication access_denied : Cannot authenticate user 'None@N/A': No valid profile found in credentials..",
        "exception" : "Traceback (most recent call last):\n  File \"/tmp/ansible_ovirt_api_info_payload_8fukbrop/ansible_ovirt_api_info_payload.zip/ansible_collections/ovirt/ovirt/plugins/modules/ovirt_api_info.py\", line 73, in main\n  File \"/usr/lib64/python3.6/site-packages/ovirtsdk4/services.py\", line 27517, in get\n    return self._internal_get(headers, query, wait)\n  File \"/usr/lib64/python3.6/site-packages/ovirtsdk4/service.py\", line 211, in _internal_get\n    return future.wait() if wait else future\n  File \"/usr/lib64/python3.6/site-packages/ovirtsdk4/service.py\", line 54, in wait\n    response = self._connection.wait(self._context)\n  File \"/usr/lib64/python3.6/site-packages/ovirtsdk4/__init__.py\", line 496, in wait\n    return self.__wait(context, failed_auth)\n  File \"/usr/lib64/python3.6/site-packages/ovirtsdk4/__init__.py\", line 526, in __wait\n    self._sso_token = self._get_access_token()\n  File \"/usr/lib64/python3.6/site-packages/ovirtsdk4/__init__.py\", line 627, in _get_access_token\n    sso_error[1]\novirtsdk4.AuthError: Error during SSO authentication access_denied : Cannot authenticate user 'None@N/A': No valid profile found in credentials..\n",
        "invocation" : {
          "module_args" : {
            "fetch_nested" : false,
            "nested_attributes" : [ ],
            "follows" : [ ]
          }
        },
        "_ansible_no_log" : false,
        "changed" : false
      },
      "start" : "2021-08-23T14:59:39.604161",
      "end" : "2021-08-23T14:59:40.219582",
      "duration" : 0.615421,
      "ignore_errors" : null,
      "event_loop" : null,
      "uuid" : "74f26212-9717-4f86-9186-90737f5b51ec"
    }
  }
}

Comment 4 Martin Necas 2021-08-24 08:29:34 UTC
Hi, which version of the collection do you use?
`rpm -q ovirt-ansible-collection`

Comment 5 Lorenzo Sartoratti 2021-08-24 08:46:48 UTC
ovirt-ansible-collection-1.6.0-1.el8.noarch

Comment 6 Michal Skrivanek 2021-08-25 06:30:35 UTC
*** Bug 1996990 has been marked as a duplicate of this bug. ***

Comment 7 Lorenzo Sartoratti 2021-08-27 09:50:38 UTC
Hi,
I've just installed the latest version: ovirt-ansible-collection-1.6.2-1.el8.noarch
Everything works!
Thank you all for the effective and fast support!

Lorenzo

Comment 8 Lorenzo Sartoratti 2021-09-01 13:42:07 UTC
Hi,
I'm sorry but, my fault, the error was related to the Cluster upgrade and not to the compatibility upgrade
In any case, the upgrade now works.
What still doesn't work is the compatibility upgrade from 4.5 to 4.6 because it can't change the compatibility for the HostedEngine vm:

from engine.log
2021-09-01 15:37:17,271+02 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-5) [28a98510] EVENT_ID: CLUSTER_CANNOT_UPDATE_VM_COMPATIBILITY_VERSION(12,005), Cannot update compatibility version of Vm/Template: [HostedEngine], Message: [No Message]
2021-09-01 15:37:17,457+02 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-5) [28a98510] EVENT_ID: USER_UPDATE_CLUSTER_FAILED(812), Failed to update Host cluster (User: admin@internal-authz)

from web ui:
Error while executing action: Cannot update cluster because the update triggered update of the VMs/Templates and it failed for the following: HostedEngine. To fix the issue, please go to each of them, edit, change the Custom Compatibility Version (or other fields changed previously in the cluster dialog) and press OK. If the save does not pass, fix the dialog validation. After successful cluster update, you can revert your Custom Compatibility Version change (or other changes). If the problem still persists, you may refer to the engine.log file for further details.

Thank you in advance!

Lorenzo

Comment 9 Lucie Leistnerova 2021-09-06 12:41:56 UTC
Verified in automation, cluster upgrade tests passed, no access_denied error.

In ovirt-engine-4.4.8.5-0.4.el8ev.noarch, ovirt-ansible-collection-1.6.2-1.el8ev.noarch


Note You need to log in before you can comment on or make changes to this bug.