May I ask you how are you sure that the particular issue is solved by the pointed out patch?

Also, a couple more questions on this case:
 a) do you have the console logs for the registered failure;
 b) how consistent is your reproducer; and 
 c) when it started to happen? at kernel-3.10.0-1059.el7 or at an earlier build?

Thanks in advance!
-- Rafael

(In reply to Rafael Aquini from comment #1)
> May I ask you how are you sure that the particular issue is solved by the
> pointed out patch?

The test#2(in move_pages12) is going to simulate the race condition, where move_pages() and soft offline are called on a single hugetlb page concurrently. But, it returns EBUSY and reports FAIL in soft-offline a moving hugepage as a result in upstream v5.2 kernel testing. 

I confirmed with Naoya Horiguchi and he pointed out that because of this new fix commit 6bc9b56433b7(mm: fix race on soft-offlining free huge pages) change on the return value of madvise(MADV_SOFT_OFFLINE), and we see -EBUSY when hugepage migration succeeded and error containment failed, that consider this EBUSY as error, but a good report for application.

And it says that patch also fixes another bz: a race condition between soft offline and hugetlb_fault which causes unexpected process SIGBUS killing and/or hugetlb allocation failure. So I tried it on rhel7 and get a failure like that:

err_log:
tst_test.c:1096: INFO: Timeout per run is 0h 05m 00s
move_pages12.c:236: INFO: Free RAM 119568 kB
move_pages12.c:254: INFO: Increasing 2048kB hugepages pool on node 0 to 83
move_pages12.c:264: INFO: Increasing 2048kB hugepages pool on node 1 to 94
move_pages12.c:180: INFO: Allocating and freeing 4 hugepages on node 0
move_pages12.c:180: INFO: Allocating and freeing 4 hugepages on node 1
move_pages12.c:170: PASS: Bug not reproduced
tst_test.c:1141: BROK: Test killed by SIGBUS!
move_pages12.c:114: FAIL: move_pages failed: ESRCH

dmesg
[ 9868.180669] MCE: Killing move_pages12:29616 due to hardware memory corruption fault at 2aaaaac00018
[ 9990.049875] Soft offlining page 50e00 at 2aaaaac00000
[ 9990.052218] Soft offlining page 50c00 at 2aaaaae00000
[ 9990.060395] Soft offlining page 51000 at 2aaaaac00000


This patch changes soft offline semantics where it sets PageHWPoison flag only after containment of the error page completes successfully.


-               if (PageHuge(page))
-                       dissolve_free_huge_page(page);
+               /*
+                * We set PG_hwpoison only when the migration source hugepage
+                * was successfully dissolved, because otherwise hwpoisoned
+                * hugepage remains on free hugepage list, then userspace will
+                * find it as SIGBUS by allocation failure. That's not expected
+                * in soft-offlining.
+                */
+               ret = dissolve_free_huge_page(page);
+               if (!ret) {
+                       if (set_hwpoison_free_buddy_page(page))
+                               num_poisoned_pages_inc();
+               }


> 
> Also, a couple more questions on this case:
>  a) do you have the console logs for the registered failure;

see above.

>  b) how consistent is your reproducer; and 
>  c) when it started to happen? at kernel-3.10.0-1059.el7 or at an earlier
> build?

Not sure, this reproducer is a new port to LTP, and I just run it from RHEL7.7(kernel-3.10.0-1059.el7) and mainline kernel v5.2, I guess the rhel8 also need to fix this problem too.

If anything I was wrong, feel free to correct me.


Li Wang

Btw, this is the original discussion on LTP ML:
  http://lists.linux.it/pipermail/ltp/2019-June/012299.html

(In reply to Li Wang from comment #2)
> (In reply to Rafael Aquini from comment #1)
> > May I ask you how are you sure that the particular issue is solved by the
> > pointed out patch?
> 
> The test#2(in move_pages12) is going to simulate the race condition, where
> move_pages() and soft offline are called on a single hugetlb page
> concurrently. But, it returns EBUSY and reports FAIL in soft-offline a
> moving hugepage as a result in upstream v5.2 kernel testing. 
> 
> I confirmed with Naoya Horiguchi and he pointed out that because of this new
> fix commit 6bc9b56433b7(mm: fix race on soft-offlining free huge pages)
> change on the return value of madvise(MADV_SOFT_OFFLINE), and we see -EBUSY
> when hugepage migration succeeded and error containment failed, that
> consider this EBUSY as error, but a good report for application.
> 
> And it says that patch also fixes another bz: a race condition between soft
> offline and hugetlb_fault which causes unexpected process SIGBUS killing
> and/or hugetlb allocation failure. So I tried it on rhel7 and get a failure
> like that:
> 
> err_log:
> tst_test.c:1096: INFO: Timeout per run is 0h 05m 00s
> move_pages12.c:236: INFO: Free RAM 119568 kB
> move_pages12.c:254: INFO: Increasing 2048kB hugepages pool on node 0 to 83
> move_pages12.c:264: INFO: Increasing 2048kB hugepages pool on node 1 to 94
> move_pages12.c:180: INFO: Allocating and freeing 4 hugepages on node 0
> move_pages12.c:180: INFO: Allocating and freeing 4 hugepages on node 1
> move_pages12.c:170: PASS: Bug not reproduced
> tst_test.c:1141: BROK: Test killed by SIGBUS!
> move_pages12.c:114: FAIL: move_pages failed: ESRCH
> 
> dmesg
> [ 9868.180669] MCE: Killing move_pages12:29616 due to hardware memory
> corruption fault at 2aaaaac00018
> [ 9990.049875] Soft offlining page 50e00 at 2aaaaac00000
> [ 9990.052218] Soft offlining page 50c00 at 2aaaaae00000
> [ 9990.060395] Soft offlining page 51000 at 2aaaaac00000
> 
> 
> This patch changes soft offline semantics where it sets PageHWPoison flag
> only after containment of the error page completes successfully.
> 
> 
> -               if (PageHuge(page))
> -                       dissolve_free_huge_page(page);
> +               /*
> +                * We set PG_hwpoison only when the migration source hugepage
> +                * was successfully dissolved, because otherwise hwpoisoned
> +                * hugepage remains on free hugepage list, then userspace
> will
> +                * find it as SIGBUS by allocation failure. That's not
> expected
> +                * in soft-offlining.
> +                */
> +               ret = dissolve_free_huge_page(page);
> +               if (!ret) {
> +                       if (set_hwpoison_free_buddy_page(page))
> +                               num_poisoned_pages_inc();
> +               }
> 
> 
> > 
> > Also, a couple more questions on this case:
> >  a) do you have the console logs for the registered failure;
> 
> see above.
> 
> >  b) how consistent is your reproducer; and 
> >  c) when it started to happen? at kernel-3.10.0-1059.el7 or at an earlier
> > build?
> 
> Not sure, this reproducer is a new port to LTP, and I just run it from
> RHEL7.7(kernel-3.10.0-1059.el7) and mainline kernel v5.2, I guess the rhel8
> also need to fix this problem too.
> 
> If anything I was wrong, feel free to correct me.

Nope, nothing wrong, I'm just double-checking the fact to be sure we're (a) really
hitting the condition described at the patch-fix; and (b) if we know that's a regression
or if it is something that have always been there.

Thanks for the information, Li.

-- Rafael

Also strange is that I'm unable to release most of hugepages. I run move_pages12 test couple times and hugepage count just keeps increasing:

# cat /sys/devices/system/node/node{0,1}/hugepages/hugepages-2048kB/nr_hugepages
38
24

# echo 0 > /proc/sys/vm/nr_hugepages
# echo 0 > /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages
# echo 0 > /sys/devices/system/node/node1/hugepages/hugepages-2048kB/nr_hugepages

# cat /sys/devices/system/node/node{0,1}/hugepages/hugepages-2048kB/nr_hugepages
36
24

# cat /proc/meminfo  | grep Huge
AnonHugePages:      6144 kB
HugePages_Total:      60
HugePages_Free:       16
HugePages_Rsvd:        0
HugePages_Surp:       24
Hugepagesize:       2048 kB

Doesn't look to be recent issue, I see same behavior with 7.7GA, 7.6GA kernels.

------- Comment From mbringm.com 2019-12-11 11:32 EDT-------
Aneesh:
Please take a look at this.

------- Comment From sadas034.com 2020-01-20 05:12 EDT-------
(In reply to comment #4)
> Also strange is that I'm unable to release most of hugepages. I run
> move_pages12 test couple times and hugepage count just keeps increasing:
> # cat
> /sys/devices/system/node/node{0,1}/hugepages/hugepages-2048kB/nr_hugepages
> 38
> 24
> # echo 0 > /proc/sys/vm/nr_hugepages
> # echo 0 >
> /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages
> # echo 0 >
> /sys/devices/system/node/node1/hugepages/hugepages-2048kB/nr_hugepages
> # cat
> /sys/devices/system/node/node{0,1}/hugepages/hugepages-2048kB/nr_hugepages
> 36
> 24
> # cat /proc/meminfo  | grep Huge
> AnonHugePages:      6144 kB
> HugePages_Total:      60
> HugePages_Free:       16
> HugePages_Rsvd:        0
> HugePages_Surp:       24
> Hugepagesize:       2048 kB
> Doesn't look to be recent issue, I see same behavior with 7.7GA, 7.6GA
> kernels.

With ppc64, this test triggers a kernel crash on 7.7 and older GA kernels as observed in BZ178206. The fix for that is now included but I don't see any issues with freeing huge pages even after running this test several times.

------- Comment From mbringm.com 2020-03-10 12:43 EDT-------
RedHat: Any updates on this one?