Bug 668340
Summary: | NUMA is not recognized for nec-em25.rhts.eng.bos.redhat.com | ||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Product: | Red Hat Enterprise Linux 6 | Reporter: | Zhang Kexin <kzhang> | ||||||||||
Component: | kernel | Assignee: | Cong Wang <amwang> | ||||||||||
Status: | CLOSED ERRATA | QA Contact: | Zhang Kexin <kzhang> | ||||||||||
Severity: | high | Docs Contact: | |||||||||||
Priority: | high | ||||||||||||
Version: | 6.1 | CC: | amwang, mzywusko, qcai, rkhan | ||||||||||
Target Milestone: | rc | ||||||||||||
Target Release: | --- | ||||||||||||
Hardware: | x86_64 | ||||||||||||
OS: | Linux | ||||||||||||
Whiteboard: | |||||||||||||
Fixed In Version: | kernel-2.6.32-112.el6 | Doc Type: | Bug Fix | ||||||||||
Doc Text: | Story Points: | --- | |||||||||||
Clone Of: | Environment: | ||||||||||||
Last Closed: | 2011-05-19 12:54:31 UTC | Type: | --- | ||||||||||
Regression: | --- | Mount Type: | --- | ||||||||||
Documentation: | --- | CRM: | |||||||||||
Verified Versions: | Category: | --- | |||||||||||
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||||||
Cloudforms Team: | --- | Target Upstream Version: | |||||||||||
Embargoed: | |||||||||||||
Bug Depends On: | |||||||||||||
Bug Blocks: | 668681 | ||||||||||||
Attachments: |
|
Created attachment 472517 [details]
kernel message for rhel5.6
sorry the whole numactl output for rhel6 is: [root@nec-em25 ~]# numactl --hardware available: 1 nodes (0) node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 node 0 size: 130927 MB node 0 free: 126026 MB node distances: node 0 0: 10 This sounds like a regression. On RHEL5 nodes are detected correctly with different distances which should be the distances physically, however, on RHEL6 all memory are put into one node which could hurt the performance since there are different distances to access different memory. REHL5 and RHEL6 detect same SRAT, but on RHEL6 we got: SRAT: PXMs only cover 98159MB of your 130927MB e820 RAM. Not used. It seems something wrong with __absent_pages_in_range(), 32768M is missed by its calculation! Boot with "loglevel=8 mminit_loglevel=4": Jan 10 04:36:10 nec-em25 kernel: mminit::memory_register Entering add_active_range(0, 0x10, 0x9b) 0 entries of 25600 used Jan 10 04:36:10 nec-em25 kernel: mminit::memory_register Entering add_active_range(0, 0x100, 0x7702b) 1 entries of 25600 used Jan 10 04:36:10 nec-em25 kernel: mminit::memory_register Entering add_active_range(0, 0x7724c, 0x7724e) 2 entries of 25600 used Jan 10 04:36:10 nec-em25 kernel: mminit::memory_register Entering add_active_range(0, 0x773e7, 0x7b206) 3 entries of 25600 used Jan 10 04:36:10 nec-em25 kernel: mminit::memory_register Entering add_active_range(0, 0x7b409, 0x7b600) 4 entries of 25600 used Jan 10 04:36:10 nec-em25 kernel: mminit::memory_register Entering add_active_range(0, 0x100000, 0x280000) 5 entries of 25600 used Jan 10 04:36:10 nec-em25 kernel: mminit::memory_register Entering add_active_range(0, 0x280000, 0x380000) 6 entries of 25600 used Jan 10 04:36:10 nec-em25 kernel: mminit::memory_register Entering add_active_range(0, 0x380000, 0x480000) 6 entries of 25600 used Jan 10 04:36:10 nec-em25 kernel: mminit::memory_register Entering add_active_range(2, 0x880000, 0x980000) 6 entries of 25600 used Jan 10 04:36:10 nec-em25 kernel: mminit::memory_register Entering add_active_range(2, 0x980000, 0xa80000) 7 entries of 25600 used Jan 10 04:36:10 nec-em25 kernel: mminit::memory_register Entering add_active_range(2, 0xa80000, 0xb80000) 7 entries of 25600 used Jan 10 04:36:10 nec-em25 kernel: mminit::memory_register Entering add_active_range(2, 0xb80000, 0xc80000) 7 entries of 25600 used Jan 10 04:36:10 nec-em25 kernel: mminit::memory_register Entering add_active_range(1, 0x480000, 0x580000) 7 entries of 25600 used Jan 10 04:36:10 nec-em25 kernel: mminit::memory_register Entering add_active_range(1, 0x580000, 0x680000) 8 entries of 25600 used Jan 10 04:36:10 nec-em25 kernel: mminit::memory_register Entering add_active_range(1, 0x680000, 0x780000) 8 entries of 25600 used Jan 10 04:36:10 nec-em25 kernel: mminit::memory_register Entering add_active_range(1, 0x780000, 0x880000) 8 entries of 25600 used Jan 10 04:36:10 nec-em25 kernel: mminit::memory_register Entering add_active_range(3, 0xc80000, 0xd80000) 8 entries of 25600 used Jan 10 04:36:10 nec-em25 kernel: mminit::memory_register Entering add_active_range(3, 0xd80000, 0xe80000) 9 entries of 25600 used Jan 10 04:36:10 nec-em25 kernel: mminit::memory_register Entering add_active_range(3, 0xe80000, 0xf80000) 9 entries of 25600 used Jan 10 04:36:10 nec-em25 kernel: mminit::memory_register Entering add_active_range(3, 0xf80000, 0x1080000) 9 entries of 25600 used Jan 10 04:36:10 nec-em25 kernel: mminit::memory_register Entering add_active_range(4, 0x1080000, 0x1180000) 9 entries of 25600 used Jan 10 04:36:10 nec-em25 kernel: mminit::memory_register Entering add_active_range(4, 0x1180000, 0x1280000) 10 entries of 25600 used Jan 10 04:36:10 nec-em25 kernel: mminit::memory_register Entering add_active_range(4, 0x1280000, 0x1380000) 10 entries of 25600 used Jan 10 04:36:10 nec-em25 kernel: mminit::memory_register Entering add_active_range(4, 0x1380000, 0x1480000) 10 entries of 25600 used Jan 10 04:36:10 nec-em25 kernel: mminit::memory_register Entering add_active_range(6, 0x1880000, 0x1980000) 10 entries of 25600 used Jan 10 04:36:10 nec-em25 kernel: mminit::memory_register Entering add_active_range(6, 0x1980000, 0x1a80000) 11 entries of 25600 used Jan 10 04:36:10 nec-em25 kernel: mminit::memory_register Entering add_active_range(6, 0x1a80000, 0x1b80000) 11 entries of 25600 used Jan 10 04:36:10 nec-em25 kernel: mminit::memory_register Entering add_active_range(6, 0x1b80000, 0x1c80000) 11 entries of 25600 used Jan 10 04:36:10 nec-em25 kernel: mminit::memory_register Entering add_active_range(5, 0x1480000, 0x1580000) 11 entries of 25600 used Jan 10 04:36:10 nec-em25 kernel: mminit::memory_register Entering add_active_range(5, 0x1580000, 0x1680000) 12 entries of 25600 used Jan 10 04:36:10 nec-em25 kernel: mminit::memory_register Entering add_active_range(5, 0x1680000, 0x1780000) 12 entries of 25600 used Jan 10 04:36:10 nec-em25 kernel: mminit::memory_register Entering add_active_range(5, 0x1780000, 0x1880000) 12 entries of 25600 used Jan 10 04:36:10 nec-em25 kernel: mminit::memory_register Entering add_active_range(7, 0x1c80000, 0x1d80000) 12 entries of 25600 used Jan 10 04:36:10 nec-em25 kernel: mminit::memory_register Entering add_active_range(7, 0x1d80000, 0x1e80000) 13 entries of 25600 used Jan 10 04:36:10 nec-em25 kernel: mminit::memory_register Entering add_active_range(7, 0x1e80000, 0x1f80000) 13 entries of 25600 used Jan 10 04:36:10 nec-em25 kernel: mminit::memory_register Entering add_active_range(7, 0x1f80000, 0x207c000) 13 entries of 25600 used Jan 10 04:36:10 nec-em25 kernel: mminit::memory_register Entering add_active_range(0, 0x10, 0x9b) 0 entries of 25600 used Jan 10 04:36:10 nec-em25 kernel: mminit::memory_register Entering add_active_range(0, 0x100, 0x7702b) 1 entries of 25600 used Jan 10 04:36:10 nec-em25 kernel: mminit::memory_register Entering add_active_range(0, 0x7724c, 0x7724e) 2 entries of 25600 used Jan 10 04:36:10 nec-em25 kernel: mminit::memory_register Entering add_active_range(0, 0x773e7, 0x7b206) 3 entries of 25600 used Jan 10 04:36:10 nec-em25 kernel: mminit::memory_register Entering add_active_range(0, 0x7b409, 0x7b600) 4 entries of 25600 used Jan 10 04:36:10 nec-em25 kernel: mminit::memory_register Entering add_active_range(0, 0x100000, 0x207c000) 5 entries of 25600 used Jan 10 04:36:10 nec-em25 kernel: mminit::pageflags_layout_widths Section 0 Node 9 Zone 2 Flags 26 Jan 10 04:36:10 nec-em25 kernel: mminit::pageflags_layout_shifts Section 19 Node 9 Zone 2 Jan 10 04:36:10 nec-em25 kernel: mminit::pageflags_layout_offsets Section 0 Node 55 Zone 53 Jan 10 04:36:10 nec-em25 kernel: mminit::pageflags_layout_zoneid Zone ID: 53 -> 64 Jan 10 04:36:10 nec-em25 kernel: mminit::pageflags_layout_usage location: 64 -> 53 unused 53 -> 26 flags 26 -> 0 Jan 10 04:36:10 nec-em25 kernel: mminit::memmap_init Initialising map node 0 zone 0 pfns 16 -> 4096 Jan 10 04:36:10 nec-em25 kernel: mminit::memmap_init Initialising map node 0 zone 1 pfns 4096 -> 1048576 Jan 10 04:36:10 nec-em25 kernel: mminit::memmap_init Initialising map node 0 zone 2 pfns 1048576 -> 34062336 Jan 10 04:36:10 nec-em25 kernel: BIOS-provided physical RAM map: Jan 10 04:36:10 nec-em25 kernel: BIOS-e820: 0000000000000000 - 000000000009b400 (usable) Jan 10 04:36:10 nec-em25 kernel: BIOS-e820: 000000000009b400 - 00000000000a0000 (reserved) Jan 10 04:36:10 nec-em25 kernel: BIOS-e820: 00000000000e0000 - 0000000000100000 (reserved) Jan 10 04:36:10 nec-em25 kernel: BIOS-e820: 0000000000100000 - 000000007702b000 (usable) Jan 10 04:36:10 nec-em25 kernel: BIOS-e820: 000000007702b000 - 00000000770ec000 (ACPI NVS) Jan 10 04:36:10 nec-em25 kernel: BIOS-e820: 00000000770ec000 - 000000007714e000 (ACPI data) Jan 10 04:36:10 nec-em25 kernel: BIOS-e820: 000000007714e000 - 0000000077167000 (reserved) Jan 10 04:36:10 nec-em25 kernel: BIOS-e820: 0000000077167000 - 0000000077168000 (ACPI NVS) Jan 10 04:36:10 nec-em25 kernel: BIOS-e820: 0000000077168000 - 0000000077179000 (reserved) Jan 10 04:36:10 nec-em25 kernel: BIOS-e820: 0000000077179000 - 000000007717c000 (ACPI NVS) Jan 10 04:36:10 nec-em25 kernel: BIOS-e820: 000000007717c000 - 0000000077230000 (reserved) Jan 10 04:36:10 nec-em25 kernel: BIOS-e820: 0000000077230000 - 0000000077239000 (ACPI NVS) Jan 10 04:36:10 nec-em25 kernel: BIOS-e820: 0000000077239000 - 000000007724c000 (reserved) Jan 10 04:36:10 nec-em25 kernel: BIOS-e820: 000000007724c000 - 000000007724e000 (usable) Jan 10 04:36:10 nec-em25 kernel: BIOS-e820: 000000007724e000 - 0000000077250000 (ACPI NVS) Jan 10 04:36:10 nec-em25 kernel: BIOS-e820: 0000000077250000 - 0000000077285000 (reserved) Jan 10 04:36:10 nec-em25 kernel: BIOS-e820: 0000000077285000 - 0000000077399000 (ACPI NVS) Jan 10 04:36:10 nec-em25 kernel: BIOS-e820: 0000000077399000 - 00000000773e7000 (reserved) Jan 10 04:36:10 nec-em25 kernel: BIOS-e820: 00000000773e7000 - 000000007b206000 (usable) Jan 10 04:36:10 nec-em25 kernel: BIOS-e820: 000000007b206000 - 000000007b409000 (ACPI NVS) Jan 10 04:36:10 nec-em25 kernel: BIOS-e820: 000000007b409000 - 000000007b600000 (usable) Jan 10 04:36:10 nec-em25 kernel: BIOS-e820: 000000007b600000 - 000000007b800000 (reserved) Jan 10 04:36:10 nec-em25 kernel: BIOS-e820: 0000000080000000 - 0000000091000000 (reserved) Jan 10 04:36:10 nec-em25 kernel: BIOS-e820: 00000000fed1c000 - 00000000fed20000 (reserved) Jan 10 04:36:10 nec-em25 kernel: BIOS-e820: 00000000ff000000 - 0000000100000000 (reserved) Jan 10 04:36:10 nec-em25 kernel: BIOS-e820: 0000000100000000 - 000000207c000000 (usable) __absent_pages_in_range() doesn't count e820 reserved ranges, while e820_hole_size() counts that, thus causes this bug. Created attachment 472564 [details]
Untested patch
Will test it now...
Created attachment 472596 [details]
Upstream patch
Upstream has fixed this, this should be the right patch...
With that patch applied, NUMA gets back: [root@nec-em25 ~]# numactl --hardware available: 6 nodes (1-3,5-7) node 1 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 node 1 size: 16303 MB node 1 free: 15616 MB node 2 cpus: 16 17 18 19 20 21 22 23 80 81 82 83 84 85 86 87 node 2 size: 16384 MB node 2 free: 15821 MB node 3 cpus: 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 node 3 size: 16384 MB node 3 free: 15619 MB node 5 cpus: 40 41 42 43 44 45 46 47 104 105 106 107 108 109 110 111 node 5 size: 16384 MB node 5 free: 15784 MB node 6 cpus: 48 49 50 51 52 53 54 55 112 113 114 115 116 117 118 119 node 6 size: 16384 MB node 6 free: 15668 MB node 7 cpus: 56 57 58 59 60 61 62 63 120 121 122 123 124 125 126 127 node 7 size: 16320 MB node 7 free: 15755 MB No distance information available. But there is still some problem, I reported it as Bug 668681. This request was evaluated by Red Hat Product Management for inclusion in a Red Hat Enterprise Linux maintenance release. Product Management has requested further review of this request by Red Hat Engineering, for potential inclusion in a Red Hat Enterprise Linux Update release for currently deployed products. This request is not yet committed for inclusion in an Update release. Patch(es) available on kernel-2.6.32-112.el6 An advisory has been issued which should help the problem described in this bug report. This report is therefore being closed with a resolution of ERRATA. For more information on therefore solution and/or where to find the updated files, please follow the link below. You may reopen this bug report if the solution does not work for you. http://rhn.redhat.com/errata/RHSA-2011-0542.html |
Created attachment 472516 [details] kernel message for rhel6 Description of problem: NUMA is not recognized for nec-em25.rhts.eng.bos.redhat.com when rhel6 and 6.1 x86_64 is installed on it. while rhel5 can recognize NUMA on it. Version-Release number of selected component (if applicable): rhel6(2.6.32-71) and rhel6.1(2.6.32-94) x86_64. How reproducible: always. Steps to Reproduce: 1. install rhel6 x86_64 on nec-em25.rhts.eng.bos.redhat.com. 2. ls /sys/devices/system/node/ 3. Actual results: only node1 is listed. Expected results: should have 8 nodes. Additional info: on rhel6: [root@nec-em25 ~]# numactl --hardware available: 1 nodes (0) node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 on rhel5.6 [root@nec-em25 ~]# numactl --hardware available: 8 nodes (0-7) node 0 size: 16045 MB node 0 free: 15346 MB node 1 size: 16160 MB node 1 free: 16114 MB node 2 size: 16160 MB node 2 free: 16127 MB node 3 size: 16160 MB node 3 free: 16134 MB node 4 size: 16160 MB node 4 free: 15973 MB node 5 size: 16160 MB node 5 free: 16134 MB node 6 size: 16160 MB node 6 free: 16133 MB node 7 size: 16096 MB node 7 free: 16058 MB node distances: node 0 1 2 3 4 5 6 7 0: 10 15 15 20 20 15 20 20 1: 15 10 20 15 20 20 20 20 2: 15 20 10 15 20 20 20 15 3: 20 15 15 10 15 20 20 20 4: 20 20 20 15 10 15 15 20 5: 15 20 20 20 15 10 20 15 6: 20 20 20 20 15 20 10 15 7: 20 20 15 20 20 15 15 10 /var/log/messages for rhel5.6 and rhel6 will be attached.