DescriptionEduardo Olivares
2021-08-12 10:40:54 UTC
Description of problem:
The OVN TLS migration job failed running the start-ovn-migration.sh script. It failed during the following task:
TASK [migration : Sync neutron db with OVN db (container) - Run 1] *************
task path: /home/stack/ovn_migration/playbooks/roles/migration/tasks/sync-dbs.yml:7
Thursday 12 August 2021 06:21:18 +0000 (0:00:00.937) 0:41:59.450 *******
fatal: [controller-0]: FAILED! => {"changed": true, "cmd": ["podman", "exec", "ef44978bc506", "neutron-ovn-db-sync-util", "--config-file", "/etc/neutron/neutron.conf", "--config-file", "/etc/neutron/plugins/ml2/ml2_conf.ini", "--ovn-neutron_sync_mode", "migrate"], "delta": "0:00:23.218507", "end": "2021-08-12 06:21:41.902879", "msg": "non-zero return code", "rc": 1, "start": "2021-08-12 06:21:18.684372", "stderr": "Deprecated: Option \"vif_type\" from group \"ovn\" is deprecated for removal (The port VIF type is now determined based on the OVN chassis information when the port is bound to a host.). Its value may be silently ignored in the future.\nDeprecated: Option \"ovn_l3_mode\" from group \"ovn\" is deprecated for removal (This option is no longer used. Native L3 support in OVN is always used.). Its value may be silently ignored in the future.\nError: non zero exit code: 1: OCI runtime error", "stderr_lines": ["Deprecated: Option \"vif_type\" from group \"ovn\" is deprecated for removal (The port VIF type is now determined based on the OVN chassis information when the port is bound to a host.). Its value may be silently ignored in the future.", "Deprecated: Option \"ovn_l3_mode\" from group \"ovn\" is deprecated for removal (This option is no longer used. Native L3 support in OVN is always used.). Its value may be silently ignored in the future.", "Error: non zero exit code: 1: OCI runtime error"], "stdout": "", "stdout_lines": []}
This failure was reproduced on RHOS-16.1-RHEL-8-20210804.n.0:
http://rhos-ci-logs.lab.eng.tlv2.redhat.com/logs/rcj/DFG-network-networking-ovn-16.1_director-rhel-virthost-3cont_2comp-ipv4-vxlan-ml2ovs-to-ovn-migration_tls/12/undercloud-0/home/stack/ovn_migration/start-ovn-migration.sh.log.gz
The mentioned script did not fail on RHOS-16.1-RHEL-8-20210604.n.0:
http://rhos-ci-logs.lab.eng.tlv2.redhat.com/logs/rcj/DFG-network-networking-ovn-16.1_director-rhel-virthost-3cont_2comp-ipv4-vxlan-ml2ovs-to-ovn-migration_tls/9/undercloud-0/home/stack/ovn_migration/start-ovn-migration.sh.log.gz
The script failed on RHOS-16.1-RHEL-8-20210804.n.0 with other OVN migration jobs (for example, nondvr to dvr), but it failed at a later task and due to a known issue (BZ1987249):
http://rhos-ci-logs.lab.eng.tlv2.redhat.com/logs/rcj/DFG-network-networking-ovn-16.1_director-rhel-virthost-3cont_2comp-ipv4-vxlan-ml2ovs-to-ovn-migration_nodvr-to-dvr/39/undercloud-0/home/stack/ovn_migration/start-ovn-migration.sh.log.gz
task path: /home/stack/ovn_migration/playbooks/roles/migration/tasks/sync-dbs.yml:7
Wednesday 11 August 2021 03:27:59 +0000 (0:00:00.774) 0:39:13.604 ******
changed: [controller-0] => {"changed": true, "cmd": ["podman", "exec", "18124c7c0e58", "neutron-ovn-db-sync-util", "--config-file", "/etc/neutron/neutron.conf", "--config-file", "/etc/neutron/plugins/ml2/ml2_conf.ini", "--ovn-neutron_sync_mode", "migrate"], "delta": "0:00:25.984057", "end": "2021-08-11 03:28:25.577204", "rc": 0, "start": "2021-08-11 03:27:59.593147", "stderr": "Deprecated: Option \"vif_type\" from group \"ovn\" is deprecated for removal (The port VIF type is now determined based on the OVN chassis information when the port is bound to a host.). Its value may be silently ignored in the future.\nDeprecated: Option \"ovn_l3_mode\" from group \"ovn\" is deprecated for removal (This option is no longer used. Native L3 support in OVN is always used.). Its value may be silently ignored in the future.", "stderr_lines": ["Deprecated: Option \"vif_type\" from group \"ovn\" is deprecated for removal (The port VIF type is now determined based on the OVN chassis information when the port is bound to a host.). Its value may be silently ignored in the future.", "Deprecated: Option \"ovn_l3_mode\" from group \"ovn\" is deprecated for removal (This option is no longer used. Native L3 support in OVN is always used.). Its value may be silently ignored in the future."], "stdout": "", "stdout_lines": []}
Version-Release number of selected component (if applicable):
RHOS-16.1-RHEL-8-20210804.n.0
How reproducible:
So far, it happened only once
Steps to Reproduce:
1. run the TLS migration job
2.
3.
Verified that the issue does not occur on RHOS-16.1-RHEL-8-20211112.n.1 which uses python3-networking-ovn-7.3.1-1.20210714143310.el8ost.noarch
OVN migration succeeds when running downstream osp16.1 ovs2ovn CI jobs. All tempest tests pass after the migration.
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory (Red Hat OpenStack Platform 16.1.7 (Train) bug fix and enhancement advisory), and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.
https://access.redhat.com/errata/RHBA-2021:3762
Description of problem: The OVN TLS migration job failed running the start-ovn-migration.sh script. It failed during the following task: TASK [migration : Sync neutron db with OVN db (container) - Run 1] ************* task path: /home/stack/ovn_migration/playbooks/roles/migration/tasks/sync-dbs.yml:7 Thursday 12 August 2021 06:21:18 +0000 (0:00:00.937) 0:41:59.450 ******* fatal: [controller-0]: FAILED! => {"changed": true, "cmd": ["podman", "exec", "ef44978bc506", "neutron-ovn-db-sync-util", "--config-file", "/etc/neutron/neutron.conf", "--config-file", "/etc/neutron/plugins/ml2/ml2_conf.ini", "--ovn-neutron_sync_mode", "migrate"], "delta": "0:00:23.218507", "end": "2021-08-12 06:21:41.902879", "msg": "non-zero return code", "rc": 1, "start": "2021-08-12 06:21:18.684372", "stderr": "Deprecated: Option \"vif_type\" from group \"ovn\" is deprecated for removal (The port VIF type is now determined based on the OVN chassis information when the port is bound to a host.). Its value may be silently ignored in the future.\nDeprecated: Option \"ovn_l3_mode\" from group \"ovn\" is deprecated for removal (This option is no longer used. Native L3 support in OVN is always used.). Its value may be silently ignored in the future.\nError: non zero exit code: 1: OCI runtime error", "stderr_lines": ["Deprecated: Option \"vif_type\" from group \"ovn\" is deprecated for removal (The port VIF type is now determined based on the OVN chassis information when the port is bound to a host.). Its value may be silently ignored in the future.", "Deprecated: Option \"ovn_l3_mode\" from group \"ovn\" is deprecated for removal (This option is no longer used. Native L3 support in OVN is always used.). Its value may be silently ignored in the future.", "Error: non zero exit code: 1: OCI runtime error"], "stdout": "", "stdout_lines": []} This failure was reproduced on RHOS-16.1-RHEL-8-20210804.n.0: http://rhos-ci-logs.lab.eng.tlv2.redhat.com/logs/rcj/DFG-network-networking-ovn-16.1_director-rhel-virthost-3cont_2comp-ipv4-vxlan-ml2ovs-to-ovn-migration_tls/12/undercloud-0/home/stack/ovn_migration/start-ovn-migration.sh.log.gz The mentioned script did not fail on RHOS-16.1-RHEL-8-20210604.n.0: http://rhos-ci-logs.lab.eng.tlv2.redhat.com/logs/rcj/DFG-network-networking-ovn-16.1_director-rhel-virthost-3cont_2comp-ipv4-vxlan-ml2ovs-to-ovn-migration_tls/9/undercloud-0/home/stack/ovn_migration/start-ovn-migration.sh.log.gz The script failed on RHOS-16.1-RHEL-8-20210804.n.0 with other OVN migration jobs (for example, nondvr to dvr), but it failed at a later task and due to a known issue (BZ1987249): http://rhos-ci-logs.lab.eng.tlv2.redhat.com/logs/rcj/DFG-network-networking-ovn-16.1_director-rhel-virthost-3cont_2comp-ipv4-vxlan-ml2ovs-to-ovn-migration_nodvr-to-dvr/39/undercloud-0/home/stack/ovn_migration/start-ovn-migration.sh.log.gz task path: /home/stack/ovn_migration/playbooks/roles/migration/tasks/sync-dbs.yml:7 Wednesday 11 August 2021 03:27:59 +0000 (0:00:00.774) 0:39:13.604 ****** changed: [controller-0] => {"changed": true, "cmd": ["podman", "exec", "18124c7c0e58", "neutron-ovn-db-sync-util", "--config-file", "/etc/neutron/neutron.conf", "--config-file", "/etc/neutron/plugins/ml2/ml2_conf.ini", "--ovn-neutron_sync_mode", "migrate"], "delta": "0:00:25.984057", "end": "2021-08-11 03:28:25.577204", "rc": 0, "start": "2021-08-11 03:27:59.593147", "stderr": "Deprecated: Option \"vif_type\" from group \"ovn\" is deprecated for removal (The port VIF type is now determined based on the OVN chassis information when the port is bound to a host.). Its value may be silently ignored in the future.\nDeprecated: Option \"ovn_l3_mode\" from group \"ovn\" is deprecated for removal (This option is no longer used. Native L3 support in OVN is always used.). Its value may be silently ignored in the future.", "stderr_lines": ["Deprecated: Option \"vif_type\" from group \"ovn\" is deprecated for removal (The port VIF type is now determined based on the OVN chassis information when the port is bound to a host.). Its value may be silently ignored in the future.", "Deprecated: Option \"ovn_l3_mode\" from group \"ovn\" is deprecated for removal (This option is no longer used. Native L3 support in OVN is always used.). Its value may be silently ignored in the future."], "stdout": "", "stdout_lines": []} Version-Release number of selected component (if applicable): RHOS-16.1-RHEL-8-20210804.n.0 How reproducible: So far, it happened only once Steps to Reproduce: 1. run the TLS migration job 2. 3.