Bug 1262864 - pmap produces ludicrous output on shared memory using programs due to bogus parsing
pmap produces ludicrous output on shared memory using programs due to bogus p...
Status: CLOSED ERRATA
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: procps-ng (Show other bugs)
7.2
Unspecified Unspecified
urgent Severity high
: rc
: ---
Assigned To: Jan Rybar
Jan Houska
Lenka Špačková
: Upstream, ZStream
Depends On: 1287433 1287643
Blocks: 1203710 1284842
  Show dependency treegraph
 
Reported: 2015-09-14 09:45 EDT by Martin Poole
Modified: 2017-09-21 10:16 EDT (History)
8 users (show)

See Also:
Fixed In Version: procps-ng-3.3.10-4.el7
Doc Type: Release Note
Doc Text:
*pmap* no longer reports incorrect totals With the introduction of `VmFlags` in the kernel *smaps* interface, the *pmap* tool could no longer reliably process the content due to format differences of the `VmFlags` entry. As a consequence, *pmap* reported incorrect totals. The underlying source code has been patched, and *pmap* now works as expected.
Story Points: ---
Clone Of:
: 1262870 1284842 (view as bug list)
Environment:
Last Closed: 2016-11-04 02:36:36 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
somewhat redundant obvious patch to move continue statement to correct location. (475 bytes, patch)
2015-09-14 09:45 EDT, Martin Poole
no flags Details | Diff
Patch applied to the package (518 bytes, patch)
2016-06-15 08:42 EDT, Jan Rybar
no flags Details | Diff

  None (edit)
Description Martin Poole 2015-09-14 09:45:32 EDT
Created attachment 1073253 [details]
somewhat redundant obvious patch to move continue statement to correct location.

Description of problem:

pmap parsing of smap data has the continue statement in the wrong place for Keys and attempts to parse numeric values from non-numeric lines and then adds the badly initialised values into the process total.


Version-Release number of selected component (if applicable):

3.3.10-3

How reproducible:

Sometimes.

Steps to Reproduce:
1. pmap of a process using shared memory

Actual results:


8516:   /usr/sbin/AuditSp 0640 /var/run/audispd_events
Address           Kbytes     RSS   Dirty Mode   Mapping
000000000019d000      28      12       0 r-x--  librt-2.12.so
00000000001a4000       4       4       4 r----  librt-2.12.so
[snip]
0000000008049000       4       4       4 rw---  AuditSp
0000000009e59000     164      20      20 rw---    [ anon ]
00000000ef23f000  125496     932     932 rw-s-    [ shmid=0x3f8007 ]
00000000f6ccd000   10660      32      32 rw-s-    [ shmid=0x3f0006 ]
00000000f7736000      20      20      20 rw---    [ anon ]
00000000f7748000       8       8       8 rw---    [ anon ]
00000000ff7f6000     924      12      12 rw---    [ stack ]
----------------  ------  ------  ------
total kB 18014398501662120    2624    1224



Expected results:

sane value for total.


Additional info:

pmap is reading sections like

f6ccd000-f7736000 rw-s 00000000 00:04 4128774        /SYSV00000600 (deleted)
Size:              10660 kB
Rss:                  32 kB
Pss:                   0 kB
Shared_Clean:          0 kB
Shared_Dirty:         32 kB
Private_Clean:         0 kB
Private_Dirty:         0 kB
Referenced:           32 kB
Anonymous:             0 kB
AnonHugePages:         0 kB
Swap:                  0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
VmFlags: rd wr sh mr mw me ms ??


and reads the lines with (snipped and reformatted for clarity)

   /* hex values are lower case or numeric, keys are upper */
   if (mapbuf[0] >= 'A' && mapbuf[0] <= 'Z')
   {
      /* Its a key */
      if (sscanf (mapbuf, "%20[^:]: %llu", smap_key, &smap_value) == 2)
      {
          [snip - do stuff with key/values ]
          continue;
      }
   }
   sscanf(mapbuf, "%" KLF "x-%" KLF "x %31s %llx %x:%x %llu", &start,
           &end, perms, &file_offset, &dev_major, &dev_minor,
           &inode);


So when it meets the line


    VmFlags: rd wr sh mr mw me ms ??


it drops through to the sscanf for the address range data. Since this does not parse (and the return code is not checked) variations of the old values are used in subsequent calculations. Specifically the value for "diff" becomes a negative value in the  -4286537728  range.
Comment 2 Jaromír Cápík 2015-09-16 13:42:30 EDT
Hello Martin.

I remember we implemented the VmFlags support in the extended maps function, but for some reason it didn't make it in the default output.
Could You please test, whether you get correct value with the -X / -XX switch ?

Thanks,
Jaromir.
Comment 4 Martin Poole 2015-09-17 10:43:59 EDT
Not yet managed to create a reproducer on RHEL7, but the same logical bug is present as RHEL6

RHEL6 BZ now has reproducer.
Comment 6 Martin Poole 2015-09-18 08:27:52 EDT
Modified RHEL6 reproducer shows double accounting rather than ludicrous values

#include <sys/types.h>
#include <sys/stat.h>
#include <sys/shm.h>
#include <fcntl.h>
#include <sys/mman.h>
#include <stdio.h>


int
main( int argc, char *argv[] )
{
    int fd, shmid;
    void *vp;
    char cmdline[4096];
    char *pwdname = "/etc/passwd";

    /* memory map something */
    fd = open( pwdname, O_RDONLY );
    vp = mmap( NULL, 4096*4096, PROT_READ, MAP_SHARED , fd, 0 );
    fprintf( stderr, "mmap of %s at %p\n", pwdname, vp );


    /* call out to get our own maps */
    sprintf( cmdline, "/bin/cat /proc/%ld/smaps", getpid() );
    system( cmdline );
    sprintf( cmdline, "pmap -x %ld", getpid() );
    system( cmdline );
    sprintf( cmdline, "pmap -X %ld", getpid() );
    system( cmdline );
    return(0);
}
Comment 7 Martin Poole 2015-09-18 08:28:25 EDT
./badpmap
mmap of /etc/passwd at 0x7f5eaceeb000
00400000-00401000 r-xp 00000000 fd:05 113164774                          /data/mpoole/src/badpmap/badpmap
Size:                  4 kB
Rss:                   4 kB
Pss:                   4 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:         4 kB
Referenced:            4 kB
Anonymous:             0 kB
AnonHugePages:         0 kB
Swap:                  0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Locked:                0 kB
VmFlags: rd ex mr mw me dw
00600000-00601000 r--p 00000000 fd:05 113164774                          /data/mpoole/src/badpmap/badpmap
Size:                  4 kB
Rss:                   4 kB
Pss:                   4 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:         4 kB
Referenced:            4 kB
Anonymous:             4 kB
AnonHugePages:         0 kB
Swap:                  0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Locked:                0 kB
VmFlags: rd mr mw me dw ac
00601000-00602000 rw-p 00001000 fd:05 113164774                          /data/mpoole/src/badpmap/badpmap
Size:                  4 kB
Rss:                   4 kB
Pss:                   4 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:         4 kB
Referenced:            4 kB
Anonymous:             4 kB
AnonHugePages:         0 kB
Swap:                  0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Locked:                0 kB
VmFlags: rd wr mr mw me dw ac
7f5eaceeb000-7f5eadeeb000 r--s 00000000 fd:03 269079062                  /etc/passwd
Size:              16384 kB
Rss:                   0 kB
Pss:                   0 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:         0 kB
Referenced:            0 kB
Anonymous:             0 kB
AnonHugePages:         0 kB
Swap:                  0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Locked:                0 kB
VmFlags: rd mr me ms
7f5eadeeb000-7f5eae0a1000 r-xp 00000000 fd:03 537539217                  /usr/lib64/libc-2.17.so
Size:               1752 kB
Rss:                 284 kB
Pss:                   6 kB
Shared_Clean:        280 kB
Shared_Dirty:          0 kB
Private_Clean:         4 kB
Private_Dirty:         0 kB
Referenced:          284 kB
Anonymous:             0 kB
AnonHugePages:         0 kB
Swap:                  0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Locked:                0 kB
VmFlags: rd ex mr mw me
7f5eae0a1000-7f5eae2a1000 ---p 001b6000 fd:03 537539217                  /usr/lib64/libc-2.17.so
Size:               2048 kB
Rss:                   0 kB
Pss:                   0 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:         0 kB
Referenced:            0 kB
Anonymous:             0 kB
AnonHugePages:         0 kB
Swap:                  0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Locked:                0 kB
VmFlags: mr mw me
7f5eae2a1000-7f5eae2a5000 r--p 001b6000 fd:03 537539217                  /usr/lib64/libc-2.17.so
Size:                 16 kB
Rss:                  16 kB
Pss:                  16 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:        16 kB
Referenced:           16 kB
Anonymous:            16 kB
AnonHugePages:         0 kB
Swap:                  0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Locked:                0 kB
VmFlags: rd mr mw me ac
7f5eae2a5000-7f5eae2a7000 rw-p 001ba000 fd:03 537539217                  /usr/lib64/libc-2.17.so
Size:                  8 kB
Rss:                   8 kB
Pss:                   8 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:         8 kB
Referenced:            8 kB
Anonymous:             8 kB
AnonHugePages:         0 kB
Swap:                  0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Locked:                0 kB
VmFlags: rd wr mr mw me ac
7f5eae2a7000-7f5eae2ac000 rw-p 00000000 00:00 0
Size:                 20 kB
Rss:                  12 kB
Pss:                  12 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:        12 kB
Referenced:           12 kB
Anonymous:            12 kB
AnonHugePages:         0 kB
Swap:                  0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Locked:                0 kB
VmFlags: rd wr mr mw me ac
7f5eae2ac000-7f5eae2cd000 r-xp 00000000 fd:03 541551200                  /usr/lib64/ld-2.17.so
Size:                132 kB
Rss:                 112 kB
Pss:                   0 kB
Shared_Clean:        112 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:         0 kB
Referenced:          112 kB
Anonymous:             0 kB
AnonHugePages:         0 kB
Swap:                  0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Locked:                0 kB
VmFlags: rd ex mr mw me dw
7f5eae4a0000-7f5eae4a3000 rw-p 00000000 00:00 0
Size:                 12 kB
Rss:                  12 kB
Pss:                  12 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:        12 kB
Referenced:           12 kB
Anonymous:            12 kB
AnonHugePages:         0 kB
Swap:                  0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Locked:                0 kB
VmFlags: rd wr mr mw me ac
7f5eae4cc000-7f5eae4cd000 rw-p 00000000 00:00 0
Size:                  4 kB
Rss:                   4 kB
Pss:                   4 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:         4 kB
Referenced:            4 kB
Anonymous:             4 kB
AnonHugePages:         0 kB
Swap:                  0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Locked:                0 kB
VmFlags: rd wr mr mw me ac
7f5eae4cd000-7f5eae4ce000 r--p 00021000 fd:03 541551200                  /usr/lib64/ld-2.17.so
Size:                  4 kB
Rss:                   4 kB
Pss:                   4 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:         4 kB
Referenced:            4 kB
Anonymous:             4 kB
AnonHugePages:         0 kB
Swap:                  0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Locked:                0 kB
VmFlags: rd mr mw me dw ac
7f5eae4ce000-7f5eae4cf000 rw-p 00022000 fd:03 541551200                  /usr/lib64/ld-2.17.so
Size:                  4 kB
Rss:                   4 kB
Pss:                   4 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:         4 kB
Referenced:            4 kB
Anonymous:             4 kB
AnonHugePages:         0 kB
Swap:                  0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Locked:                0 kB
VmFlags: rd wr mr mw me dw ac
7f5eae4cf000-7f5eae4d0000 rw-p 00000000 00:00 0
Size:                  4 kB
Rss:                   4 kB
Pss:                   4 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:         4 kB
Referenced:            4 kB
Anonymous:             4 kB
AnonHugePages:         0 kB
Swap:                  0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Locked:                0 kB
VmFlags: rd wr mr mw me ac
7ffd9b8b5000-7ffd9b8d6000 rw-p 00000000 00:00 0                          [stack]
Size:                136 kB
Rss:                  20 kB
Pss:                  20 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:        20 kB
Referenced:           20 kB
Anonymous:            20 kB
AnonHugePages:         0 kB
Swap:                  0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Locked:                0 kB
VmFlags: rd wr mr mw me gd ac
7ffd9b915000-7ffd9b917000 r-xp 00000000 00:00 0                          [vdso]
Size:                  8 kB
Rss:                   4 kB
Pss:                   0 kB
Shared_Clean:          4 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:         0 kB
Referenced:            4 kB
Anonymous:             0 kB
AnonHugePages:         0 kB
Swap:                  0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Locked:                0 kB
VmFlags: rd ex mr mw me de
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
Size:                  4 kB
Rss:                   0 kB
Pss:                   0 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:         0 kB
Referenced:            0 kB
Anonymous:             0 kB
AnonHugePages:         0 kB
Swap:                  0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Locked:                0 kB
VmFlags: rd ex
25473:   ./badpmap
Address           Kbytes     RSS   Dirty Mode  Mapping
0000000000400000       4       4       4 r-x-- badpmap
0000000000600000       4       4       4 r---- badpmap
0000000000601000       4       4       4 rw--- badpmap
00007f5eaceeb000   16384       0       0 r--s- passwd
00007f5eadeeb000    1752     284       0 r-x-- libc-2.17.so
00007f5eae0a1000    2048       0       0 ----- libc-2.17.so
00007f5eae2a1000      16      16      16 r---- libc-2.17.so
00007f5eae2a5000       8       8       8 rw--- libc-2.17.so
00007f5eae2a7000      20      12      12 rw---   [ anon ]
00007f5eae2ac000     132     112       0 r-x-- ld-2.17.so
00007f5eae4a0000      12      12      12 rw---   [ anon ]
00007f5eae4cc000       4       4       4 rw---   [ anon ]
00007f5eae4cd000       4       4       4 r---- ld-2.17.so
00007f5eae4ce000       4       4       4 rw--- ld-2.17.so
00007f5eae4cf000       4       4       4 rw---   [ anon ]
00007ffd9b8b5000     132      20      20 rw---   [ stack ]
00007ffd9b915000       8       4       0 r-x--   [ anon ]
ffffffffff600000       4       0       0 r-x--   [ anon ]
---------------- ------- ------- -------
total kB           36928     496      96
25473:   ./badpmap
         Address Perm   Offset Device     Inode  Size Rss Pss Referenced Anonymous Swap Locked Mapping
        00400000 r-xp 00000000  fd:05 113164774     4   4   4          4         0    0      0 badpmap
        00600000 r--p 00000000  fd:05 113164774     4   4   4          4         4    0      0 badpmap
        00601000 rw-p 00001000  fd:05 113164774     4   4   4          4         4    0      0 badpmap
    7f5eaceeb000 r--s 00000000  fd:03 269079062 16384   0   0          0         0    0      0 passwd
    7f5eadeeb000 r-xp 00000000  fd:03 537539217  1752 284   6        284         0    0      0 libc-2.17.so
    7f5eae0a1000 ---p 001b6000  fd:03 537539217  2048   0   0          0         0    0      0 libc-2.17.so
    7f5eae2a1000 r--p 001b6000  fd:03 537539217    16  16  16         16        16    0      0 libc-2.17.so
    7f5eae2a5000 rw-p 001ba000  fd:03 537539217     8   8   8          8         8    0      0 libc-2.17.so
    7f5eae2a7000 rw-p 00000000  00:00         0    20  12  12         12        12    0      0
    7f5eae2ac000 r-xp 00000000  fd:03 541551200   132 112   0        112         0    0      0 ld-2.17.so
    7f5eae4a0000 rw-p 00000000  00:00         0    12  12  12         12        12    0      0
    7f5eae4cc000 rw-p 00000000  00:00         0     4   4   4          4         4    0      0
    7f5eae4cd000 r--p 00021000  fd:03 541551200     4   4   4          4         4    0      0 ld-2.17.so
    7f5eae4ce000 rw-p 00022000  fd:03 541551200     4   4   4          4         4    0      0 ld-2.17.so
    7f5eae4cf000 rw-p 00000000  00:00         0     4   4   4          4         4    0      0
    7ffd9b8b5000 rw-p 00000000  00:00         0   136  20  20         20        20    0      0 [stack]
    7ffd9b915000 r-xp 00000000  00:00         0     8   4   0          4         0    0      0 [vdso]
ffffffffff600000 r-xp 00000000  00:00         0     4   0   0          0         0    0      0 [vsyscall]
                                                ===== === === ========== ========= ==== ======
                                                20548 496 102        496        92    0      0 KB
Comment 8 Martin Poole 2015-09-18 08:32:30 EDT
Changing the size of the mmap to something large clearly illustrates the double accounting.  The mmap segment is added to the total twice in -x

Using 65336*4096

26493:   ./badpmap
Address           Kbytes     RSS   Dirty Mode  Mapping
0000000000400000       4       4       4 r-x-- badpmap
0000000000600000       4       4       4 r---- badpmap
0000000000601000       4       4       4 rw--- badpmap
00007f329f497000  262144       0       0 r--s- passwd
00007f32af497000    1752     284       0 r-x-- libc-2.17.so
00007f32af64d000    2048       0       0 ----- libc-2.17.so
00007f32af84d000      16      16      16 r---- libc-2.17.so
00007f32af851000       8       8       8 rw--- libc-2.17.so
00007f32af853000      20      12      12 rw---   [ anon ]
00007f32af858000     132     112       0 r-x-- ld-2.17.so
00007f32afa4c000      12      12      12 rw---   [ anon ]
00007f32afa78000       4       4       4 rw---   [ anon ]
00007f32afa79000       4       4       4 r---- ld-2.17.so
00007f32afa7a000       4       4       4 rw--- ld-2.17.so
00007f32afa7b000       4       4       4 rw---   [ anon ]
00007ffd38a8c000     132      20      20 rw---   [ stack ]
00007ffd38b19000       8       4       0 r-x--   [ anon ]
ffffffffff600000       4       0       0 r-x--   [ anon ]
---------------- ------- ------- -------
total kB          528448     496      96
26493:   ./badpmap
         Address Perm   Offset Device     Inode   Size Rss Pss Referenced Anonymous Swap Locked Mapping
        00400000 r-xp 00000000  fd:05 113164774      4   4   4          4         0    0      0 badpmap
        00600000 r--p 00000000  fd:05 113164774      4   4   4          4         4    0      0 badpmap
        00601000 rw-p 00001000  fd:05 113164774      4   4   4          4         4    0      0 badpmap
    7f329f497000 r--s 00000000  fd:03 269079062 262144   0   0          0         0    0      0 passwd
    7f32af497000 r-xp 00000000  fd:03 537539217   1752 284   6        284         0    0      0 libc-2.17.so
    7f32af64d000 ---p 001b6000  fd:03 537539217   2048   0   0          0         0    0      0 libc-2.17.so
    7f32af84d000 r--p 001b6000  fd:03 537539217     16  16  16         16        16    0      0 libc-2.17.so
    7f32af851000 rw-p 001ba000  fd:03 537539217      8   8   8          8         8    0      0 libc-2.17.so
    7f32af853000 rw-p 00000000  00:00         0     20  12  12         12        12    0      0
    7f32af858000 r-xp 00000000  fd:03 541551200    132 112   0        112         0    0      0 ld-2.17.so
    7f32afa4c000 rw-p 00000000  00:00         0     12  12  12         12        12    0      0
    7f32afa78000 rw-p 00000000  00:00         0      4   4   4          4         4    0      0
    7f32afa79000 r--p 00021000  fd:03 541551200      4   4   4          4         4    0      0 ld-2.17.so
    7f32afa7a000 rw-p 00022000  fd:03 541551200      4   4   4          4         4    0      0 ld-2.17.so
    7f32afa7b000 rw-p 00000000  00:00         0      4   4   4          4         4    0      0
    7ffd38a8c000 rw-p 00000000  00:00         0    136  20  20         20        20    0      0 [stack]
    7ffd38b19000 r-xp 00000000  00:00         0      8   4   0          4         0    0      0 [vdso]
ffffffffff600000 r-xp 00000000  00:00         0      4   0   0          0         0    0      0 [vsyscall]
                                                ====== === === ========== ========= ==== ======
                                                266308 496 102        496        92    0      0 KB
Comment 14 Jan Rybar 2016-06-15 08:42 EDT
Created attachment 1168367 [details]
Patch applied to the package
Comment 15 Jan Rybar 2016-06-17 10:19:04 EDT
Patch accepted by upstream in commit https://gitlab.com/procps-ng/procps/commit/f6abbb00f0b29a514955b864cd86dc1438728b88
Comment 20 errata-xmlrpc 2016-11-04 02:36:36 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-2447.html

Note You need to log in before you can comment on or make changes to this bug.