Red Hat Bugzilla – Bug 1561162
[RHEL7.5] Extreme performance impact caused by raid resync
Last modified: 2018-10-30 04:54:10 EDT
Description of problem: This issue is a continuation of bz1455679. Through continued research, it was found that the proper fix here is to revert the patch that causes the issue, ac8fa4196d20 ("md: allow resync to go faster when there is competing IO.). Version-Release number of selected component (if applicable): kernel-3.10.0-693.17.1.el7 How reproducible: Issue large amount of I/O during raid resync. Easiest way to reproduce is to install to a md device that is resync'ing and watch partition creation & filesystem formatting time. Prior to patch revert 19:58:39,249 INFO anaconda: Creating disklabel on /dev/sdb ....snip 20:23:30,488 INFO anaconda: Created swap on /dev/md/3 ~ 25mins After patch revert: 16:45:17,990 INFO anaconda: Creating disklabel on /dev/sdb ....snip 16:45:36,935 INFO anaconda: Created swap on /dev/md/3 ~ 20secs Expected results: No detectable impact on I/O while raid resync is running. Additional info: Upstream discussion at https://marc.info/?l=linux-raid&m=152120839121813&w=2
Second attempt to get fix accepted upstream. https://marc.info/?l=linux-raid&m=152235148327824&w=2 If they don't accept, maybe we can get something temporary into the RH stream.
Hello what's performance you want to get while raid resync ? or what's the baseline time for the installing ?
Hello # cp test kernel to the dir /var/www/html/guazhang/repo/test_kernel # createrepo -u -o -d /var/www/html/guazhang/repo/ # setenforce 0 # lorax --product="RHEL" --version=7.6 --release=7.6 --source=http://download.lab.bos.redhat.com/rel-eng/RHEL-7.6-20180810.0/compose/Server/x86_64/os/ --source=http://download.lab.bos.redhat.com/rel-eng/RHEL-7.6-20180810.0/compose/Server-optional/x86_64/os/ --source http://pnate-control-01.lab.bos.redhat.com/guazhang/repo --variant=Server --nomacboot --buildarch=x86_64 --volid=RHEL-7.6_Server.x86_64 ./updated_kernel ks.cfg lang en_US.UTF-8 keyboard us url --url="http://pnate-control-01.lab.bos.redhat.com/guazhang/repo" firewall --disabled firstboot --disable rootpw redhat timezone Europe/Prague reboot clearpart --all timezone America/New_York ignoredisk --only-use=sda bootloader --append="loglevel=5 crashkernel=auto" --location=mbr zerombr clearpart --all --initlabel part raid.225 --ondisk=sda --size=512 part raid.231 --ondisk=sdb --size=512 part raid.261 --ondisk=sda --size=12288 part raid.255 --ondisk=sdb --size=12288 part raid.249 --ondisk=sda --size=12288 part raid.267 --ondisk=sdb --size=12288 part raid.237 --ondisk=sda --size=205824 part raid.243 --ondisk=sdb --size=205824 raid pv.285 --device=2 --fstype="lvmpv" --level=RAID1 raid.237 raid.243 raid / --device=1 --fstype="ext4" --level=RAID1 --mkfsoptions="-E nodiscard" raid.249 raid.255 raid /boot --device=0 --fstype="ext4" --level=RAID1 --mkfsoptions="-E nodiscard" raid.225 raid.231 raid swap --device=3 --fstype="swap" --level=RAID1 --mkfsoptions="-E nodiscard" raid.261 raid.267 volgroup vg00 --pesize=65536 pv.285 logvol /var/log --fstype="ext4" --grow --size=1 --mkfsoptions="-E nodiscard" --name=log --vgname=vg00 logvol /usr --fstype="ext4" --size=30000 --mkfsoptions="-E nodiscard" --name=usr --vgname=vg00 logvol /tmp --fstype="ext4" --size=30000 --mkfsoptions="-E nodiscard" --name=tmp --vgname=vg00 logvol /opt --fstype="ext4" --size=30000 --mkfsoptions="-E nodiscard" --name=opt --vgname=vg00 logvol /var --fstype="ext4" --size=30000 --mkfsoptions="-E nodiscard" --name=var --vgname=vg00 logvol /home --fstype="ext4" --size=30000 --mkfsoptions="-E nodiscard" --name=home --vgname=vg00 logvol /opt/mgtservices --fstype="ext4" --size=30000 --mkfsoptions="-E nodiscard" --name=mgtservices --vgname=vg00 get errors Traceback (most recent call last): File "/usr/lib64/python2.7/site-packages/pyanaconda/threads.py", line 227, in run threading.Thread.run(self, *args, **kwargs) Please make your choice from above: File "/usr/lib64/python2.7/threading.py", line 765, in run self.__target(*self.__args, **self.__kwargs) File "/usr/lib64/python2.7/site-packages/pyanaconda/ui/tui/spokes/software.py", line 238, in checkSoftwareSelection self.payload.checkSoftwareSelection() File "/usr/lib64/python2.7/site-packages/pyanaconda/packaging/yumpayload.py", line 1418, in checkSoftwareSelection self._applyYumSelections() File "/usr/lib64/python2.7/site-packages/pyanaconda/packaging/yumpayload.py", line 1360, in _applyYumSelections self._selectYumGroup("core") File "/usr/lib64/python2.7/site-packages/pyanaconda/packaging/yumpayload.py", line 1247, in _selectYumGroup raise NoSuchGroup(groupid, required=required) pyanaconda.packaging.NoSuchGroup: core beaker job https://beaker.engineering.redhat.com/jobs/2728229 I just have a test the ISO with beaker server, but it seem the ISO build failed, Could someone have a look the errors ? thanks Guazhang
Hello I want to know how to replace the default kernel with test kernel in DVD.iso , then boot with test kernel for testing. Could someone have detail commands or guide ? thanks Guazhang
Created attachment 1479489 [details] Proposed Patch
Created attachment 1479511 [details] Proposed Patch
Patch(es) committed on kernel repository and an interim kernel build is undergoing testing
Patch(es) available on kernel-3.10.0-945.el7
Nacking Heinz's patch for https://bugzilla.redhat.com/show_bug.cgi?id=1627563 I am resubmitting this patch, that broke LVM. -Nigel
Re-submitting the patch to rh kernel list now. And moving this bz back to POST. -Nigel
Wait. How did this bug go from POST -> ON_QA? And why hasn't the 'Fixed In Version' changed? I'm moving this back to POST.
*** Bug 1627563 has been marked as a duplicate of this bug. ***
*** Bug 1628499 has been marked as a duplicate of this bug. ***
Hello Could someone help to check the bug if can move to "ON_QA" status so that QE will verify it with fix kernel ? thanks Guazhang
Patch(es) available on kernel-3.10.0-951.el7
Hello the bug has test passed, move to verified
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2018:3083