Created attachment 1869658 [details] cannot-switch-rhv-node.png Description of problem: Can't switch rhv node to maintenance status because Image transfer is in progress although there is no image in transfer Version-Release number of selected component (if applicable): RHV4.4.10.6-0.1.el8ev How reproducible: 100% Steps to Reproduce: 1.Try to switch rhv node to maintenance status on rhv4.4 GUI, but popup error 'while executing action: Cannot switch Host 10.73.195.48 to Maintenance mode. Image transfer is in progress for the following (2) disks', pls refer to screenshot'cannot-switch-rhv-node.png' 2.Check storage disk option, found there is no disk in transfer, pls refer to screenshot'no-disk-in-transfer.png' Actual results: As above description Expected results: Can switch rhv node to maintenance status successfully on RHV4.4 GUI Additional info:
The message means there are transfers that are considered active in the image_transfers database table, you can check this with: select * from image_transfers where phase not in (9, 10); I can't tell from the logs why the transfers are stuck, if you have engine logs containing the transfers in the db please attach them. Some issues we know that can cause this were fixed in 4.5 for bug 2043984
(In reply to Benny Zlotnik from comment #3) > The message means there are transfers that are considered active in the > image_transfers database table, you can check this with: > > select * from image_transfers where phase not in (9, 10); engine=# select * from image_transfers where phase not in (9, 10); command_id | command_type | phase | last_updated | message | vds_id | disk_id | imaged_ticket_id | proxy_uri | bytes_sent | bytes_tot al | type | active | daemon_uri | client_inactivity_timeout | image_format | backend | backup_id | client_type | shall ow | timeout_policy --------------------------------------+--------------+-------+----------------------------+---------+--------------------------------------+-- ------------------------------------+------------------+-------------------------------------------------------------+-------------+---------- ---+------+--------+------------------------------------+---------------------------+--------------+---------+-----------+-------------+------ ---+---------------- 23e760b2-93cd-4d44-acec-7ba9608759f1 | 1024 | 11 | 2021-12-23 00:23:14.014+08 | | fa09062f-ec33-4d4a-a693-248e3af30101 | 2 970b1ec-547b-4ac2-8d08-a481de419956 | | https://dell-per740-22.lab.eng.pek2.redhat.com:54323/images | 21355560960 | 214748364 80 | 2 | t | https://10.73.194.236:54322/images | 3600 | 5 | 1 | | 2 | f | legacy 573324b9-0ace-4f8b-be8f-5d18985e5c70 | 1024 | 11 | 2022-02-22 12:17:00.932+08 | | f414a310-f0cc-4d5b-a3b7-29e5f808d1d4 | 5 ff0a36c-230b-446a-8ddf-ae764d8d5c13 | | https://dell-per740-22.lab.eng.pek2.redhat.com:54323/images | 20177170432 | 214748364 80 | 2 | t | https://10.73.195.48:54322/images | 3600 | 5 | 1 | | 2 | f | legacy 8ff8d5d1-65ff-4c41-8373-8d5c02f7c378 | 1024 | 11 | 2021-12-22 21:52:09.533+08 | | fa09062f-ec33-4d4a-a693-248e3af30101 | 2 46cb720-dbc8-40d6-9a7a-b7dc29afdb43 | | https://dell-per740-22.lab.eng.pek2.redhat.com:54323/images | 21269315584 | 214748364 80 | 2 | t | https://10.73.194.236:54322/images | 3600 | 5 | 1 | | 2 | f | legacy eeeab249-a401-44c5-8b57-235bd77cb2ff | 1024 | 11 | 2021-12-23 11:59:25.605+08 | | fa09062f-ec33-4d4a-a693-248e3af30101 | d e1a3313-4f24-4127-bc21-da237b4df545 | | https://dell-per740-22.lab.eng.pek2.redhat.com:54323/images | 17119051776 | 171798691 84 | 2 | t | https://10.73.194.236:54322/images | 3600 | 5 | 1 | | 2 | f | legacy 37404380-2d13-4324-bc55-71279197fbae | 1024 | 11 | 2021-12-23 11:20:51.638+08 | | fa09062f-ec33-4d4a-a693-248e3af30101 | a 1853db6-e7ff-4762-b39e-ecb148c277b3 | | https://dell-per740-22.lab.eng.pek2.redhat.com:54323/images | 21469069312 | 214748364 80 | 2 | t | https://10.73.194.236:54322/images | 3600 | 5 | 1 | | 2 | f | legacy 51455ec0-9539-41d8-bb5b-0bc8fbf40a8e | 1024 | 11 | 2021-12-23 02:28:34.638+08 | | fa09062f-ec33-4d4a-a693-248e3af30101 | 7 52afda5-25e3-4676-9f50-d4e713c95d53 | | https://dell-per740-22.lab.eng.pek2.redhat.com:54323/images | 15278112768 | 171798691 84 | 2 | t | https://10.73.194.236:54322/images | 3600 | 5 | 1 | | 2 | f | legacy a76f5ff8-9ee6-4aed-9fc6-78e481d9bc38 | 1024 | 11 | 2021-12-23 00:54:19.961+08 | | fa09062f-ec33-4d4a-a693-248e3af30101 | 7 0db1ffe-1028-4395-85c5-7126f25a114a | | https://dell-per740-22.lab.eng.pek2.redhat.com:54323/images | 21239955456 | 214748364 80 | 2 | t | https://10.73.194.236:54322/images | 3600 | 5 | 1 | | 2 | f | legacy 794325f6-a110-4c2e-80bd-4faa7b1572be | 1024 | 11 | 2021-12-22 21:33:04.879+08 | | fa09062f-ec33-4d4a-a693-248e3af30101 | 8 f1fee1a-4aec-45d9-9cf4-904d1c4e00e0 | | https://dell-per740-22.lab.eng.pek2.redhat.com:54323/images | 12871532544 | 128849018 88 | 2 | t | https://10.73.194.236:54322/images | 3600 | 5 | 1 | | 2 | f | legacy 43be7706-0f13-4975-a136-0ca3043932df | 1024 | 11 | 2022-02-16 17:13:17.144+08 | | f414a310-f0cc-4d5b-a3b7-29e5f808d1d4 | b 7c966e0-ada3-4065-9165-41f0a1b43cb9 | | https://dell-per740-22.lab.eng.pek2.redhat.com:54323/images | 15259041792 | 171798691 84 | 2 | t | https://10.73.195.48:54322/images | 3600 | 5 | 1 | | 2 | f | legacy 38ec8002-715e-4d30-82cc-58a45792096c | 1024 | 11 | 2021-12-23 11:38:07.946+08 | | fa09062f-ec33-4d4a-a693-248e3af30101 | 0 409d0b9-5387-4dd2-abfb-61ab02578b4d | | https://dell-per740-22.lab.eng.pek2.redhat.com:54323/images | 15338967040 | 171798691 84 | 2 | t | https://10.73.194.236:54322/images | 3600 | 5 | 1 | | 2 | f | legacy fcfca079-5715-4139-992a-2ed15b14d70d | 1024 | 11 | 2021-12-22 23:41:35.058+08 | | fa09062f-ec33-4d4a-a693-248e3af30101 | 0 b23486c-f60e-4f24-8e9c-2f19ea88fe8a | | https://dell-per740-22.lab.eng.pek2.redhat.com:54323/images | 21319647232 | 214748364 80 | 2 | t | https://10.73.194.236:54322/images | 3600 | 5 | 1 | | 2 | f | legacy 8db98a5f-4938-4e44-bc05-0cd9d5ef3088 | 1024 | 11 | 2021-12-22 23:05:24.679+08 | | fa09062f-ec33-4d4a-a693-248e3af30101 | c 833dea8-ca8a-43e9-827f-af34232c9815 | | https://dell-per740-22.lab.eng.pek2.redhat.com:54323/images | 21403533312 | 214748364 80 | 2 | t | https://10.73.194.236:54322/images | 3600 | 5 | 1 | | 2 | f | legacy fab65301-66c5-4337-aa91-6b4a9ea77dbd | 1024 | 11 | 2021-12-22 23:28:46.22+08 | | fa09062f-ec33-4d4a-a693-248e3af30101 | 9 89fb536-5515-467d-a2ea-94e84c08702b | | https://dell-per740-22.lab.eng.pek2.redhat.com:54323/images | 20985352192 | 214748364 80 | 2 | t | https://10.73.194.236:54322/images | 3600 | 5 | 1 | | 2 | f | legacy eae0ce86-f1c0-4990-8789-dbb3b24b1689 | 1024 | 11 | 2021-12-23 00:39:33.677+08 | | fa09062f-ec33-4d4a-a693-248e3af30101 | 5 6b6c5b5-663d-43bc-873c-0aa43d0aad13 | | https://dell-per740-22.lab.eng.pek2.redhat.com:54323/images | 21179400192 | 214748364 80 | 2 | f | https://10.73.194.236:54322/images | 3600 | 5 | 1 | | 2 | f | legacy (14 rows) > I can't tell from the logs why the transfers are stuck, if you have engine > logs containing the transfers in the db please attach them. > Some issues we know that can cause this were fixed in 4.5 for bug 2043984 I usually find log in /var/log/ovirt-engine/engine.log, could you please tell me where can find the other engine logs containing the transfers in the db?
(In reply to mxie from comment #4) > (In reply to Benny Zlotnik from comment #3) > > The message means there are transfers that are considered active in the > > image_transfers database table, you can check this with: > > > > select * from image_transfers where phase not in (9, 10); > > engine=# select * from image_transfers where phase not in (9, 10); > command_id | command_type | phase | > last_updated | message | vds_id | > disk_id | imaged_ticket_id | > proxy_uri | bytes_sent | bytes_tot > al | type | active | daemon_uri | > client_inactivity_timeout | image_format | backend | backup_id | client_type > | shall > ow | timeout_policy > --------------------------------------+--------------+-------+--------------- > -------------+---------+--------------------------------------+-- > ------------------------------------+------------------+--------------------- > ----------------------------------------+-------------+---------- > ---+------+--------+------------------------------------+-------------------- > -------+--------------+---------+-----------+-------------+------ > ---+---------------- > 23e760b2-93cd-4d44-acec-7ba9608759f1 | 1024 | 11 | 2021-12-23 > 00:23:14.014+08 | | fa09062f-ec33-4d4a-a693-248e3af30101 | 2 > 970b1ec-547b-4ac2-8d08-a481de419956 | | > https://dell-per740-22.lab.eng.pek2.redhat.com:54323/images | 21355560960 | > 214748364 > 80 | 2 | t | https://10.73.194.236:54322/images | > 3600 | 5 | 1 | | 2 | f > | legacy > 573324b9-0ace-4f8b-be8f-5d18985e5c70 | 1024 | 11 | 2022-02-22 > 12:17:00.932+08 | | f414a310-f0cc-4d5b-a3b7-29e5f808d1d4 | 5 > ff0a36c-230b-446a-8ddf-ae764d8d5c13 | | > https://dell-per740-22.lab.eng.pek2.redhat.com:54323/images | 20177170432 | > 214748364 > 80 | 2 | t | https://10.73.195.48:54322/images | > 3600 | 5 | 1 | | 2 | f > | legacy > 8ff8d5d1-65ff-4c41-8373-8d5c02f7c378 | 1024 | 11 | 2021-12-22 > 21:52:09.533+08 | | fa09062f-ec33-4d4a-a693-248e3af30101 | 2 > 46cb720-dbc8-40d6-9a7a-b7dc29afdb43 | | > https://dell-per740-22.lab.eng.pek2.redhat.com:54323/images | 21269315584 | > 214748364 > 80 | 2 | t | https://10.73.194.236:54322/images | > 3600 | 5 | 1 | | 2 | f > | legacy > eeeab249-a401-44c5-8b57-235bd77cb2ff | 1024 | 11 | 2021-12-23 > 11:59:25.605+08 | | fa09062f-ec33-4d4a-a693-248e3af30101 | d > e1a3313-4f24-4127-bc21-da237b4df545 | | > https://dell-per740-22.lab.eng.pek2.redhat.com:54323/images | 17119051776 | > 171798691 > 84 | 2 | t | https://10.73.194.236:54322/images | > 3600 | 5 | 1 | | 2 | f > | legacy > 37404380-2d13-4324-bc55-71279197fbae | 1024 | 11 | 2021-12-23 > 11:20:51.638+08 | | fa09062f-ec33-4d4a-a693-248e3af30101 | a > 1853db6-e7ff-4762-b39e-ecb148c277b3 | | > https://dell-per740-22.lab.eng.pek2.redhat.com:54323/images | 21469069312 | > 214748364 > 80 | 2 | t | https://10.73.194.236:54322/images | > 3600 | 5 | 1 | | 2 | f > | legacy > 51455ec0-9539-41d8-bb5b-0bc8fbf40a8e | 1024 | 11 | 2021-12-23 > 02:28:34.638+08 | | fa09062f-ec33-4d4a-a693-248e3af30101 | 7 > 52afda5-25e3-4676-9f50-d4e713c95d53 | | > https://dell-per740-22.lab.eng.pek2.redhat.com:54323/images | 15278112768 | > 171798691 > 84 | 2 | t | https://10.73.194.236:54322/images | > 3600 | 5 | 1 | | 2 | f > | legacy > a76f5ff8-9ee6-4aed-9fc6-78e481d9bc38 | 1024 | 11 | 2021-12-23 > 00:54:19.961+08 | | fa09062f-ec33-4d4a-a693-248e3af30101 | 7 > 0db1ffe-1028-4395-85c5-7126f25a114a | | > https://dell-per740-22.lab.eng.pek2.redhat.com:54323/images | 21239955456 | > 214748364 > 80 | 2 | t | https://10.73.194.236:54322/images | > 3600 | 5 | 1 | | 2 | f > | legacy > 794325f6-a110-4c2e-80bd-4faa7b1572be | 1024 | 11 | 2021-12-22 > 21:33:04.879+08 | | fa09062f-ec33-4d4a-a693-248e3af30101 | 8 > f1fee1a-4aec-45d9-9cf4-904d1c4e00e0 | | > https://dell-per740-22.lab.eng.pek2.redhat.com:54323/images | 12871532544 | > 128849018 > 88 | 2 | t | https://10.73.194.236:54322/images | > 3600 | 5 | 1 | | 2 | f > | legacy > 43be7706-0f13-4975-a136-0ca3043932df | 1024 | 11 | 2022-02-16 > 17:13:17.144+08 | | f414a310-f0cc-4d5b-a3b7-29e5f808d1d4 | b > 7c966e0-ada3-4065-9165-41f0a1b43cb9 | | > https://dell-per740-22.lab.eng.pek2.redhat.com:54323/images | 15259041792 | > 171798691 > 84 | 2 | t | https://10.73.195.48:54322/images | > 3600 | 5 | 1 | | 2 | f > | legacy > 38ec8002-715e-4d30-82cc-58a45792096c | 1024 | 11 | 2021-12-23 > 11:38:07.946+08 | | fa09062f-ec33-4d4a-a693-248e3af30101 | 0 > 409d0b9-5387-4dd2-abfb-61ab02578b4d | | > https://dell-per740-22.lab.eng.pek2.redhat.com:54323/images | 15338967040 | > 171798691 > 84 | 2 | t | https://10.73.194.236:54322/images | > 3600 | 5 | 1 | | 2 | f > | legacy > fcfca079-5715-4139-992a-2ed15b14d70d | 1024 | 11 | 2021-12-22 > 23:41:35.058+08 | | fa09062f-ec33-4d4a-a693-248e3af30101 | 0 > b23486c-f60e-4f24-8e9c-2f19ea88fe8a | | > https://dell-per740-22.lab.eng.pek2.redhat.com:54323/images | 21319647232 | > 214748364 > 80 | 2 | t | https://10.73.194.236:54322/images | > 3600 | 5 | 1 | | 2 | f > | legacy > 8db98a5f-4938-4e44-bc05-0cd9d5ef3088 | 1024 | 11 | 2021-12-22 > 23:05:24.679+08 | | fa09062f-ec33-4d4a-a693-248e3af30101 | c > 833dea8-ca8a-43e9-827f-af34232c9815 | | > https://dell-per740-22.lab.eng.pek2.redhat.com:54323/images | 21403533312 | > 214748364 > 80 | 2 | t | https://10.73.194.236:54322/images | > 3600 | 5 | 1 | | 2 | f > | legacy > fab65301-66c5-4337-aa91-6b4a9ea77dbd | 1024 | 11 | 2021-12-22 > 23:28:46.22+08 | | fa09062f-ec33-4d4a-a693-248e3af30101 | 9 > 89fb536-5515-467d-a2ea-94e84c08702b | | > https://dell-per740-22.lab.eng.pek2.redhat.com:54323/images | 20985352192 | > 214748364 > 80 | 2 | t | https://10.73.194.236:54322/images | > 3600 | 5 | 1 | | 2 | f > | legacy > eae0ce86-f1c0-4990-8789-dbb3b24b1689 | 1024 | 11 | 2021-12-23 > 00:39:33.677+08 | | fa09062f-ec33-4d4a-a693-248e3af30101 | 5 > 6b6c5b5-663d-43bc-873c-0aa43d0aad13 | | > https://dell-per740-22.lab.eng.pek2.redhat.com:54323/images | 21179400192 | > 214748364 > 80 | 2 | f | https://10.73.194.236:54322/images | > 3600 | 5 | 1 | | 2 | f > | legacy > (14 rows) > > > > I can't tell from the logs why the transfers are stuck, if you have engine > > logs containing the transfers in the db please attach them. > > Some issues we know that can cause this were fixed in 4.5 for bug 2043984 > > I usually find log in /var/log/ovirt-engine/engine.log, could you please > tell me where can find the other engine logs containing the transfers in the > db? engine.log is the right log, but it may have been rotated so it would be something like engine.log-20220314.gz I suggest just grepping for the the command id of the transfer in the /var/log/ovirt-engine/ directory and attaching the gz file here It looks like all transfers are in phase 11 which means they were cancelled by the user, however they were no longer polled for some reason and remained in this state instead of moving to phase 9 or 10
(In reply to Benny Zlotnik from comment #5) > engine.log is the right log, but it may have been rotated so it would be > something like engine.log-20220314.gz > I suggest just grepping for the the command id of the transfer in the > /var/log/ovirt-engine/ directory and attaching the gz file here There are too many engine.*.gz file in /var/log/ovirt-engine/, I'm not sure which one is useful, I will send our env info to by mail, thanks!
*** This bug has been marked as a duplicate of bug 2043984 ***