Back to bug 2179366

Who When What Removed Added
Red Hat One Jira (issues.redhat.com) 2023-03-17 14:06:57 UTC Link ID Red Hat Issue Tracker OSP-23217
Sanjay Upadhyay 2023-03-17 14:24:41 UTC CC supadhya
Robin Jarry 2023-03-17 15:18:59 UTC CC rjarry
David Marchand 2023-03-21 08:11:59 UTC CC dmarchan
Jakub Libosvar 2023-03-27 13:52:26 UTC CC jlibosva
Red Hat One Jira (issues.redhat.com) 2023-03-27 13:57:31 UTC Link ID Red Hat Issue Tracker NFV-2792
Haresh Khandelwal 2023-03-28 11:59:46 UTC CC hakhande
Robin Jarry 2023-03-29 12:49:37 UTC Summary [17.1][E810][OVN][DPDK] Very low performance using e810 nic in ovs-dpdk scenario [17.1][E810][OVN][DPDK] isolated cores interrupted by serial printk events
Lukas Svaty 2023-03-29 13:18:15 UTC Keywords Regression
CC lsvaty
Robin Jarry 2023-03-30 12:46:20 UTC Summary [17.1][E810][OVN][DPDK] isolated cores interrupted by serial printk events [17.1][DPDK] isolated cores interrupted by serial printk events
Marcelo Ricardo Leitner 2023-04-04 11:20:48 UTC CC mleitner
Vadim Khitrin 2023-04-04 11:54:32 UTC CC vkhitrin
Antoine Tenart 2023-04-05 11:06:16 UTC CC atenart
Harald Jensås 2023-04-05 14:25:00 UTC CC hjensas
Harald Jensås 2023-04-05 14:30:00 UTC CC sbaker
Flags needinfo?(sbaker)
Flavio Leitner 2023-04-05 15:10:47 UTC CC fleitner
Robin Jarry 2023-04-05 15:16:38 UTC Component openvswitch openstack-tripleo-common
QA Contact ekuris drosenfe
Assignee ovs-bugzilla nobody
CC mburns, slinaber
Robin Jarry 2023-04-05 15:30:06 UTC Assignee nobody rhos-hardprov
Robin Jarry 2023-04-05 15:39:43 UTC Priority unspecified urgent
Severity high urgent
Steve Baker 2023-04-05 21:45:48 UTC Component openstack-tripleo-common tripleo-ansible
Flags needinfo?(sbaker)
QA Contact drosenfe jhakimra
Assignee rhos-hardprov rhos-maint
Robin Jarry 2023-04-05 22:02:55 UTC Summary [17.1][DPDK] isolated cores interrupted by serial printk events [17.1][DPDK] packet drop caused by vga controller induced latency on memory bus
Franck Baudin 2023-04-07 14:09:43 UTC CC fbaudin
mhou 2023-04-07 15:23:39 UTC CC mhou
Steve Baker 2023-04-11 02:30:01 UTC Flags needinfo?(jkreger)
CC jkreger
Robin Jarry 2023-04-11 07:31:05 UTC Flags needinfo?(jfalempe)
CC jfalempe
Christophe Fontaine 2023-04-11 12:46:49 UTC Keywords Triaged
CC cfontain
Vincent S. Cojot 2023-04-11 22:32:42 UTC CC vcojot
Fabian Deutsch 2023-04-12 08:19:54 UTC CC fdeutsch
Bart Wensley 2023-04-12 11:53:11 UTC CC bwensley
Eder Zulian 2023-04-12 13:47:54 UTC CC ezulian
Julia Kreger 2023-04-12 14:18:13 UTC Flags needinfo?(jkreger)
Robin Jarry 2023-04-18 12:11:36 UTC Link ID Red Hat Bugzilla 1848102
Robin Jarry 2023-04-18 12:13:09 UTC Link ID Red Hat Bugzilla 2139901
Steve Baker 2023-04-20 22:56:17 UTC Link ID OpenStack gerrit 880010 OpenStack gerrit 880008 OpenStack gerrit 880009
Status NEW ON_DEV
Robin Jarry 2023-04-21 15:30:55 UTC CC mtosatti
Julia Kreger 2023-04-24 15:19:12 UTC Link ID OpenStack gerrit 881388
Robin Jarry 2023-04-24 15:36:34 UTC QA Contact jhakimra mnietoji
Assignee rhos-maint sbaker
Jaison Raju 2023-04-25 05:54:40 UTC CC jraju
Robin Jarry 2023-04-27 11:55:55 UTC CC eshulman
Eran Kuris 2023-05-09 08:24:59 UTC CC ekuris
Flags needinfo?(sbaker)
Eran Kuris 2023-05-09 08:25:18 UTC Flags needinfo?(sbaker)
Julia Kreger 2023-05-09 13:42:08 UTC Status ON_DEV MODIFIED
Fixed In Version openstack-tripleo-common-15.4.1-1.20230505015008.00bc21d.el8osttrunk
Julia Kreger 2023-05-09 20:03:28 UTC Assignee sbaker nobody
QA Contact mnietoji drosenfe
Component tripleo-ansible openstack-tripleo-common
Jason Joyce 2023-05-09 20:48:14 UTC Target Milestone --- beta
RHEL Program Management 2023-05-09 20:48:25 UTC Target Release --- 17.1
Steve Baker 2023-05-09 20:56:30 UTC Flags needinfo?(sbaker) needinfo?(sbaker)
Jocelyn Falempe 2023-05-15 09:43:58 UTC Flags needinfo?(jfalempe)
Robin Jarry 2023-05-15 13:18:23 UTC Flags needinfo?(jkreger)
Julia Kreger 2023-05-15 13:24:17 UTC Flags needinfo?(jkreger)
Assignee nobody sbaker
David Rosenfeld 2023-05-15 13:26:19 UTC QA Contact drosenfe jparoly
Ella Shulman 2023-05-21 07:21:54 UTC Doc Text unless the fix will be in the image for beta we require documentation for the WA

Cause: kernel args in overcloud image

Consequence: low performance

Workaround (if any): create a playbook that removes those kargs and run it as part of the node provisioning playbook:
# Additional reboot until optimization BZ #2179366 is closed
# https://bugzilla.redhat.com/show_bug.cgi?id=2179366
# [17.1][DPDK] packet drop caused by vga controller induced latency on memory bus
---
- name: Remove tty0
hosts: allovercloud
any_errors_fatal: true
gather_facts: false
vars:
reboot_wait_timeout: 900
pre_tasks:
- name: Wait for provisioned nodes to boot
wait_for_connection:
timeout: 600
delay: 10
tasks:
- name: Remove tty0 (duplicated)
become: true
replace:
dest: /etc/default/grub
regexp: '^GRUB_CMDLINE_LINUX_DEFAULT(.*)console=tty0 (.*)console=tty0 (.*)$'
replace: 'GRUB_CMDLINE_LINUX_DEFAULT\1\2\3'


Result: expected performance
Doc Type If docs needed, set a value Known Issue
Robin Jarry 2023-05-22 10:17:21 UTC Doc Text unless the fix will be in the image for beta we require documentation for the WA

Cause: kernel args in overcloud image

Consequence: low performance

Workaround (if any): create a playbook that removes those kargs and run it as part of the node provisioning playbook:
# Additional reboot until optimization BZ #2179366 is closed
# https://bugzilla.redhat.com/show_bug.cgi?id=2179366
# [17.1][DPDK] packet drop caused by vga controller induced latency on memory bus
---
- name: Remove tty0
hosts: allovercloud
any_errors_fatal: true
gather_facts: false
vars:
reboot_wait_timeout: 900
pre_tasks:
- name: Wait for provisioned nodes to boot
wait_for_connection:
timeout: 600
delay: 10
tasks:
- name: Remove tty0 (duplicated)
become: true
replace:
dest: /etc/default/grub
regexp: '^GRUB_CMDLINE_LINUX_DEFAULT(.*)console=tty0 (.*)console=tty0 (.*)$'
replace: 'GRUB_CMDLINE_LINUX_DEFAULT\1\2\3'


Result: expected performance
unless the fix will be in the image for beta we require documentation for the WA

Cause: kernel args in overcloud image

Consequence: low performance

Workaround (if any): create a playbook that removes those kargs and run it as part of the node provisioning playbook:
# Additional reboot until optimization BZ #2179366 is closed
# https://bugzilla.redhat.com/show_bug.cgi?id=2179366
# [17.1][DPDK] packet drop caused by vga controller induced latency on memory bus
# more info https://access.redhat.com/solutions/7007609
---
- name: Remove nomodeset
hosts: allovercloud
any_errors_fatal: true
gather_facts: false
vars:
reboot_wait_timeout: 900
pre_tasks:
- name: Wait for provisioned nodes to boot
wait_for_connection:
timeout: 600
delay: 10
tasks:
- name: Remove nomodeset
become: true
command: grubby --update-kernel=ALL --remove-args=nomodeset

Result: expected performance
Jelle Hoylaerts 2023-05-22 17:58:21 UTC CC jelle.hoylaerts.ext
Denise Hughes 2023-05-22 20:01:31 UTC CC dhughes
Julia Kreger 2023-05-23 19:52:14 UTC Fixed In Version openstack-tripleo-common-15.4.1-1.20230505015008.00bc21d.el8osttrunk openstack-tripleo-common-15.4.1-1.20230505011956.00bc21d.el9ost
Jesse Pretorius 2023-05-24 07:36:10 UTC CC jpretori
errata-xmlrpc 2023-05-25 15:19:00 UTC Status MODIFIED ON_QA
Vadim Khitrin 2023-05-29 08:01:23 UTC Flags needinfo?(sbaker)
Status ON_QA ASSIGNED
Luigi Tamagnone 2023-05-29 08:52:52 UTC CC ltamagno
Miguel Garcia 2023-05-30 09:27:35 UTC Fixed In Version openstack-tripleo-common-15.4.1-1.20230505011956.00bc21d.el9ost
Steve Baker 2023-05-30 22:55:26 UTC Status ASSIGNED ON_DEV
Flags needinfo?(sbaker)
Steve Baker 2023-05-30 22:56:25 UTC Link ID OpenStack gerrit 884780 OpenStack gerrit 884644
Jenny-Anne Lynch 2023-06-01 14:54:51 UTC Flags needinfo?(rjarry)
CC jelynch
Robin Jarry 2023-06-02 12:33:07 UTC Flags needinfo?(rjarry)
Julia Kreger 2023-06-02 14:02:33 UTC Status ON_DEV POST
Jon Schlueter 2023-06-02 14:09:56 UTC CC jschluet
Fixed In Version openstack-tripleo-common-15.4.1-1.20230518211051.cbb03c0.el9ost
Jon Schlueter 2023-06-02 14:12:42 UTC Status POST MODIFIED
Jenny-Anne Lynch 2023-06-02 14:21:31 UTC Doc Text unless the fix will be in the image for beta we require documentation for the WA

Cause: kernel args in overcloud image

Consequence: low performance

Workaround (if any): create a playbook that removes those kargs and run it as part of the node provisioning playbook:
# Additional reboot until optimization BZ #2179366 is closed
# https://bugzilla.redhat.com/show_bug.cgi?id=2179366
# [17.1][DPDK] packet drop caused by vga controller induced latency on memory bus
# more info https://access.redhat.com/solutions/7007609
---
- name: Remove nomodeset
hosts: allovercloud
any_errors_fatal: true
gather_facts: false
vars:
reboot_wait_timeout: 900
pre_tasks:
- name: Wait for provisioned nodes to boot
wait_for_connection:
timeout: 600
delay: 10
tasks:
- name: Remove nomodeset
become: true
command: grubby --update-kernel=ALL --remove-args=nomodeset

Result: expected performance
Currently, kernel arguments in an overcloud image cause low performance. Workaround: Create a playbook that removes the kernel arguments and run the playbook as part of the node provisioning playbook:

----
- name: Remove nomodeset
hosts: allovercloud
any_errors_fatal: true
gather_facts: false
vars:
reboot_wait_timeout: 900
pre_tasks:
- name: Wait for provisioned nodes to boot
wait_for_connection:
timeout: 600
delay: 10
tasks:
- name: Remove nomodeset
become: true
command: grubby --update-kernel=ALL --remove-args=nomodeset
----
Vadim Khitrin 2023-06-04 09:19:37 UTC Flags needinfo?(jschluet)
Jon Schlueter 2023-06-05 12:26:59 UTC Flags needinfo?(jschluet)
Steve Baker 2023-06-06 06:07:22 UTC Fixed In Version openstack-tripleo-common-15.4.1-1.20230518211051.cbb03c0.el9ost openstack-tripleo-common-15.4.1-1.20230603000754.dbfed0f.el9ost
errata-xmlrpc 2023-06-06 09:04:02 UTC Status MODIFIED ON_QA
Miguel Garcia 2023-06-06 09:43:52 UTC Status ON_QA MODIFIED
errata-xmlrpc 2023-06-06 10:49:52 UTC Status MODIFIED ON_QA
Vadim Khitrin 2023-06-08 05:11:08 UTC Status ON_QA VERIFIED
James Smith 2023-06-09 04:10:18 UTC CC jamsmith
Doc Text Currently, kernel arguments in an overcloud image cause low performance. Workaround: Create a playbook that removes the kernel arguments and run the playbook as part of the node provisioning playbook:

----
- name: Remove nomodeset
hosts: allovercloud
any_errors_fatal: true
gather_facts: false
vars:
reboot_wait_timeout: 900
pre_tasks:
- name: Wait for provisioned nodes to boot
wait_for_connection:
timeout: 600
delay: 10
tasks:
- name: Remove nomodeset
become: true
command: grubby --update-kernel=ALL --remove-args=nomodeset
----
Kernel arguments in an overcloud image degrade performance.
+
Workaround: Create a playbook that removes the kernel arguments and run the playbook as part of the node provisioning playbook:
+
----
- name: Remove nomodeset
hosts: allovercloud
any_errors_fatal: true
gather_facts: false
vars:
reboot_wait_timeout: 900
pre_tasks:
- name: Wait for provisioned nodes to boot
wait_for_connection:
timeout: 600
delay: 10
tasks:
- name: Remove nomodeset
become: true
command: grubby --update-kernel=ALL --remove-args=nomodeset
----
Jon Schlueter 2023-06-09 11:48:17 UTC Fixed In Version openstack-tripleo-common-15.4.1-1.20230603000754.dbfed0f.el9ost openstack-tripleo-common-15.4.1-1.20230518211051.cbb03c0.el9ost
Vadim Khitrin 2023-06-11 07:08:39 UTC Flags needinfo?(jamsmith)
James Smith 2023-06-13 16:46:09 UTC Flags needinfo?(jamsmith) needinfo?(vkhitrin)
Vadim Khitrin 2023-06-14 05:19:31 UTC Flags needinfo?(vkhitrin)
Doc Text Kernel arguments in an overcloud image degrade performance.
+
Workaround: Create a playbook that removes the kernel arguments and run the playbook as part of the node provisioning playbook:
+
----
- name: Remove nomodeset
hosts: allovercloud
any_errors_fatal: true
gather_facts: false
vars:
reboot_wait_timeout: 900
pre_tasks:
- name: Wait for provisioned nodes to boot
wait_for_connection:
timeout: 600
delay: 10
tasks:
- name: Remove nomodeset
become: true
command: grubby --update-kernel=ALL --remove-args=nomodeset
----
Doc Type Known Issue If docs needed, set a value
Lukas Svaty 2023-07-10 08:44:17 UTC Keywords Performance
Jenny-Anne Lynch 2023-07-19 12:15:37 UTC CC jelynch
errata-xmlrpc 2023-08-16 00:02:09 UTC Status VERIFIED RELEASE_PENDING
errata-xmlrpc 2023-08-16 01:14:22 UTC Resolution --- ERRATA
Status RELEASE_PENDING CLOSED
Last Closed 2023-08-16 01:14:22 UTC
errata-xmlrpc 2023-08-16 01:15:02 UTC Link ID Red Hat Product Errata RHEA-2023:4577

Back to bug 2179366