Bug 1488369

Summary: Manual reboot needed to apply tuned kernel arguments
Product: Red Hat OpenStack Reporter: Eyal Dannon <edannon>
Component: openstack-tripleo-heat-templatesAssignee: Saravanan KR <skramaja>
Status: CLOSED WONTFIX QA Contact: Yariv <yrachman>
Severity: high Docs Contact:
Priority: high    
Version: 11.0 (Ocata)CC: atelang, fbaudin, fherrman, itbrown, jskarvad, ksundara, mbabushk, mburns, oblaut, olysonek, rhel-osp-director-maint, skramaja, supadhya, vchundur, yrachman, zgreenbe
Target Milestone: z5Keywords: Regression, TestOnly, Tracking, Triaged, ZStream
Target Release: 11.0 (Ocata)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Known Issue
Doc Text:
Story Points: ---
Clone Of:
: 1488517 (view as bug list) Environment:
Last Closed: 2018-07-18 13:59:48 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1475571    
Bug Blocks: 1488517    

Description Eyal Dannon 2017-09-05 08:37:20 UTC 
Description of problem:

After deployment of DPDK and SR-IOV environment, manual reboot is needed to apply tuned kernel arguments,

The proper profile is active. To me the reboot has been done before activating tuned’s profile:

[root@overcloud-compute-0 ~]# tuned-adm active
Current active profile: cpu-partitioning
[root@overcloud-compute-0 ~]# cat /proc/cmdline 
BOOT_IMAGE=/boot/vmlinuz-3.10.0-693.el7.x86_64 root=UUID=8a1ee696-e3f1-416c-a126-e1e3350310e7 ro console=tty0 console=ttyS0,115200n8 crashkernel=auto rhgb quiet intel_iommu=on default_hugepagesz=1GB hugepagesz=1G hugepages=32 iommu=pt isolcpus=1-21,23-43,45-65,67-87

After Manual reboot:

[root@overcloud-compute-0 ~]# cat /proc/cmdline 
BOOT_IMAGE=/boot/vmlinuz-3.10.0-693.el7.x86_64 root=UUID=8a1ee696-e3f1-416c-a126-e1e3350310e7 ro console=tty0 console=ttyS0,115200n8 crashkernel=auto rhgb quiet intel_iommu=on default_hugepagesz=1GB hugepagesz=1G hugepages=32 iommu=pt isolcpus=1-21,23-43,45-65,67-87 nohz=on nohz_full=1-21,23-43,45-65,67-87 rcu_nocbs=1-21,23-43,45-65,67-87 tuned.non_isolcpus=00000004,00001000,00400001 intel_pstate=disable nosoftlockup

Using the following templates as reference:
https://code.engineering.redhat.com/gerrit/gitweb?p=nfv-qe.git;a=tree;f=ospd-11-vlan-sriov-single-port-composable-roles;hb=docs


Version-Release number of selected component (if applicable):
OSPD 11
tuned 2.8

How reproducible:
Always

Steps to Reproduce:
1. deploy the environemt
2. Check /proc/cmdline
3. reboot the system
4. Check /proc/cmdline

Actual results:
Kernel params are missing

Expected results:
Should exists

Additional info:
Comment 1 Saravanan KR 2017-09-05 10:24:38 UTC 
/var/log/tuned/tuned.log (with manual reboot - 2 reboots, first on cloud-init, second is manual)
http://chunk.io/krsacme/7e7eeed6c1574f3488bbe04547a24994

First reboot (cloud-init) is having below log
2017-09-05 05:14:48,657 INFO     tuned.daemon.daemon: terminating Tuned, rolling back all changes

When the reboot is triggered from first-boot script (which is run by cloud-init), the command "systemctl is-system-running" is returned as "starting", which is initiating the rollback of all changes.

Related change:
https://github.com/redhat-performance/tuned/commit/df9aa2f5c46e4db08a077081ca15b6da541b4514?diff=split#diff-ea63e0c5d4daa711fe01bc50e4db0145R151

Adding tuned team to comment on it.
Comment 2 Eyal Dannon 2017-09-05 13:53:18 UTC 
I've tested it on OSP10.
- latest OSPd10 provides tuned2.8 as OSPd11, gives the same result
- OSPd10 z3 provides:
[root@compute-0 ~]# rpm -qa | grep tuned
tuned-2.7.1-3.el7_3.2.noarch
tuned-profiles-cpu-partitioning-2.7.1-5.el7fdp.noarch

Gives us which result:
[root@compute-0 ~]# cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-3.10.0-514.21.1.el7.x86_64 root=UUID=fa9e939e-9e3c-4f1c-a07c-3f506756ad7b ro console=tty0 console=ttyS0,115200n8 crashkernel=auto rhgb quiet default_hugepagesz=1GB hugepagesz=1G hugepages=32 iommu=pt intel_iommu=on isolcpus=1,2,3,4,5,16,17,18,19,20,21 nohz=on nohz_full=1,2,3,4,5,16,17,18,19,20,21 rcu_nocbs=1,2,3,4,5,16,17,18,19,20,21 intel_pstate=disable nosoftlockup

I suppose it's related to the tuned package.
Comment 3 Yariv 2017-09-05 18:52:37 UTC 
Removing blocker flag added requires_doc_text ? as a known issue with w/a
Comment 4 Ondřej Lysoněk 2017-09-08 14:27:40 UTC 
This is indeed a regression in Tuned, caused by the commit listed in comment#1. The fix is available here:
https://github.com/redhat-performance/tuned/pull/66
Comment 5 atelang 2017-09-21 03:35:24 UTC 
Vijay, 
Please ack this for 11Z.

Thanks,
Anjali
Comment 6 Yariv 2017-10-16 07:40:25 UTC 
Automation is Ready
https://github.com/redhat-openstack/tempest-nfv-plugin/blob/devel/tempest_plugin/tests/scenario/test_nfv_epa.py#L205

Vijay
Please devel_ack
Anjali
Please pm_ack
Comment 9 Sanjay Upadhyay 2018-01-29 09:54:59 UTC 
tracking bug only, the tuned params have been verified as per comment - https://bugzilla.redhat.com/show_bug.cgi?id=1475571#c27

Moving this to QA for verifying from DFG:NFV perspective