Description of problem: After deploying a clean RHV environment, we start building it using ovirt-ansible-collection/blob/master/roles/infra/tasks/main.yml. As part of this process, we start adding new storage domains. When adding iscsi storage domains, we discover using one IP address, and login into 3 available iscsi connections connection | iqn | port | portal ------------------------------------------------------------------+---------------------------------------+------+-------- x.x.x.9 | x.com.netapp:vserver-rhv-qe | 3260 | 1033 x.x.x.10 | x.com.netapp:vserver-rhv-qe | 3260 | 1034 <ipv6-addr> | x.com.netapp:vserver-rhv-qe | 3260 | 1048 When we finished adding all storage domains, we run 'Fetch storages' and 'Update storage parameters' - during this process, the iscsi connection list in the engine DB changes to 3 duplicate connections of x.x.x.10. From this point, if we try to deactivate and activate back a host (or restart vdsm) -> the host goes to 'Non-Operational' state, and we see "Could not login to target" errors in VDSM Log: 2022-04-28 10:56:30,332+0300 INFO (jsonrpc/7) [storage.iscsi] Adding iscsi node for target x.x.x.10:3260,1034 x.com.netapp:vserver-r hv-qe iface default (iscsi:209) 2022-04-28 10:56:30,483+0300 INFO (jsonrpc/7) [storage.iscsi] Adding iscsi node for target x.x.x.10:3260,1048 x.com.netapp:vserver-r hv-qe iface default (iscsi:209) 2022-04-28 10:56:30,623+0300 INFO (jsonrpc/7) [storage.iscsi] Adding iscsi node for target x.x.x.10:3260,1033 x.com.netapp:vserver-r hv-qe iface default (iscsi:209) 2022-04-28 10:56:30,753+0300 INFO (jsonrpc/7) [storage.storageServer] Log in to 3 targets using 3 workers (storageServer:614) 2022-04-28 10:56:30,754+0300 INFO (iscsi-login/0) [storage.iscsi] Logging in to iscsi target x.x.x.10:3260,1034 x.com.netapp:vserver -rhv-qe via iface default (iscsi:231) 2022-04-28 10:56:30,759+0300 INFO (iscsi-login/1) [storage.iscsi] Logging in to iscsi target x.x.x.10:3260,1048 x.com.netapp:vserver -rhv-qe via iface default (iscsi:231) 2022-04-28 10:56:30,768+0300 INFO (iscsi-login/2) [storage.iscsi] Logging in to iscsi target x.x.x.10:3260,1033 x.com.netapp:vserver -rhv-qe via iface default (iscsi:231) 2022-04-28 10:56:30,888+0300 INFO (iscsi-login/1) [storage.iscsi] Removing iscsi node for target x.x.x.10:3260,1048 x.com.netapp:vse rver-rhv-qe iface default (iscsi:244) 2022-04-28 10:56:30,897+0300 INFO (iscsi-login/2) [storage.iscsi] Removing iscsi node for target x.x.x.10:3260,1033 x.com.netapp:vse rver-rhv-qe iface default (iscsi:244) 2022-04-28 10:56:31,082+0300 INFO (iscsi-login/0) [storage.iscsi] Removing iscsi node for target x.x.x.10:3260,1034 x.com.netapp:vse rver-rhv-qe iface default (iscsi:244) 2022-04-28 10:56:31,331+0300 ERROR (iscsi-login/1) [storage.storageServer] Could not login to target Version-Release number of selected component (if applicable): Not sure when it exactly started, but was investigated in rhv-4.5.0-7 How reproducible: always in our lab. Steps to Reproduce: 1. Have a clean fresh installed 4.5 RHV environment. 2. Add Data center, cluster, hosts 3. Add iscsi storage domains (iscsi with multiple targets, 3 in our case) check DB to see these 3 connections: /usr/share/ovirt-engine/dbscripts/engine-psql.sh -c "select connection,iqn,port,portal from storage_server_connections;" 4. Run 'Fetch storages' and 'Update storage parameters' - during this process, the iscsi connection list in the engine DB changes to 3 duplicate connections check DB again to see it. Actual results: the iscsi connection list in the engine DB changes to 3 duplicate connections Expected results: the iscsi connection list in the engine DB should remain as it was (In our case, 3 different connections) Additional info:
This bug report has Keywords: Regression or TestBlocker. Since no regressions or test blockers are allowed between releases, it is also being identified as a blocker for this release. Please resolve ASAP.
let's: - document that ansible infra role supports only one storage connection - during update of existing domain - warn and skip updating all parameters if there is more than 1 storage connection defined for a storage domain
the previously removed target (https://github.com/oVirt/ovirt-ansible-collection/pull/252) should probably be put back as well
alternatively we can stop updating altogether, but that changes the behavior for all the storage types. another alternative would be not to update during "add" and keep it for "update", but that doesn't solve the problem of multiple storage connections when we do not have a way to describe them in parameters
(In reply to Michal Skrivanek from comment #10) > alternatively we can stop updating altogether, but that changes the behavior > for all the storage types. We doing that only for ISCSI > another alternative would be not to update during "add" and keep it for > "update", but that doesn't solve the problem of multiple storage connections > when we do not have a way to describe them in parameters This can be blocked on the engine side (see bz 2079903)
(In reply to Arik from comment #11) > (In reply to Michal Skrivanek from comment #10) > > alternatively we can stop updating altogether, but that changes the behavior > > for all the storage types. > > We doing that only for ISCSI We can start with doing that only for ISCSI
(In reply to Arik from comment #11) > (In reply to Michal Skrivanek from comment #10) > > alternatively we can stop updating altogether, but that changes the behavior > > for all the storage types. > > We can start doing that only for ISCSI It's just a different kind of exception in the code, i.e. declare the iscsi connection parameters unmodifiable. It still looks better to me to allow updates when there's just 1 instead of requiring to remove and add. > > another alternative would be not to update during "add" and keep it for > > "update", but that doesn't solve the problem of multiple storage connections > > when we do not have a way to describe them in parameters > > This can be blocked on the engine side (see bz 2079903) it's not about blocking, it's about not providing the expected functionality. I can imagine skipping updates for iscsi is fine (with a different warning message), but i wouldn't say it's fundamentally better. Right fix would be to provide a way to supply parameters for multiple connections, but that would be a new API
(In reply to Michal Skrivanek from comment #13) > (In reply to Arik from comment #11) > > (In reply to Michal Skrivanek from comment #10) > > > alternatively we can stop updating altogether, but that changes the behavior > > > for all the storage types. > > > > We can start doing that only for ISCSI > > It's just a different kind of exception in the code, i.e. declare the iscsi > connection parameters unmodifiable. > It still looks better to me to allow updates when there's just 1 instead of > requiring to remove and add. Right but we're not talking about update-flow here, we talk about add-flow I agree that update-flow should work differently > > > > another alternative would be not to update during "add" and keep it for > > > "update", but that doesn't solve the problem of multiple storage connections > > > when we do not have a way to describe them in parameters > > > > This can be blocked on the engine side (see bz 2079903) > > it's not about blocking, it's about not providing the expected functionality. > > I can imagine skipping updates for iscsi is fine (with a different warning > message), but i wouldn't say it's fundamentally better. Right fix would be > to provide a way to supply parameters for multiple connections, but that > would be a new API Again, I'm not talking about update-flow that should be implemented differently When it comes to add-flow, it doesn't make much sense to expect the user to provide the data for all connections - as it seems we expect a single point and then identify the other connections
We have decided that the role will support only storage with one connection. Adding multiple connections can cause breaking backward compatibility.
Verified on RHV-4.5.1-3 with ovirt-ansible-collection-2.1.0-1 Seeing this msg: "[WARNING]: The storage domain 'iscsi_0' has multiple connections. This is not supported by the role. The storage connection update of this domain will be skipped." It is indeed skipped and there are no dup connections in the engine's DB. Moving to 'Verified'.
This bugzilla is included in oVirt 4.5.1 release, published on June 22nd 2022. Since the problem described in this bug report should be resolved in oVirt 4.5.1 release, it has been closed with a resolution of CURRENT RELEASE. If the solution does not work for you, please open a new bug report.