Bug 1475976 - multipathd show map <map_name> json sometimes has trailing garbage at the end
multipathd show map <map_name> json sometimes has trailing garbage at the end
Status: NEW
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: device-mapper-multipath (Show other bugs)
7.3
Unspecified Linux
unspecified Severity low
: rc
: ---
Assigned To: Ben Marzinski
Lin Li
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2017-07-27 12:56 EDT by Robert LeBlanc
Modified: 2017-07-27 23:36 EDT (History)
7 users (show)

See Also:
Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed:
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Robert LeBlanc 2017-07-27 12:56:08 EDT
Description of problem:
In some cases `multipathd show map <map_name> json` will have trailing garbage at the end producing bad json. It seems that this is a buffer overrun problem and sometimes goes away with a restart of multipathd, but not always.


Version-Release number of selected component (if applicable):


How reproducible:
Hard to reproduce, not sure what exactly causes it.

Steps to Reproduce:
1. Present a LUN to the target.
2. Read the map.
3. Present another LUN to the target. (Sometimes, this will corrupt the map output of the first LUN, but not always.)

Actual results:
00000000  7b 0a 20 20 20 22 6d 61  6a 6f 72 5f 76 65 72 73  |{.   "major_vers|
00000010  69 6f 6e 22 3a 20 30 2c  0a 20 20 20 22 6d 69 6e  |ion": 0,.   "min|
00000020  6f 72 5f 76 65 72 73 69  6f 6e 22 3a 20 31 2c 0a  |or_version": 1,.|
00000030  20 20 20 22 6d 61 70 22  3a 7b 0a 20 20 20 20 20  |   "map":{.     |
00000040  20 22 6e 61 6d 65 22 20  3a 20 22 33 36 30 30 31  | "name" : "36001|
00000050  34 30 35 61 30 34 35 61  30 62 31 30 34 38 38 31  |405a045a0b104881|
00000060  32 30 30 30 30 30 30 30  30 30 30 30 22 2c 0a 20  |200000000000",. |
00000070  20 20 20 20 20 22 75 75  69 64 22 20 3a 20 22 33  |     "uuid" : "3|
00000080  36 30 30 31 34 30 35 61  30 34 35 61 30 62 31 30  |6001405a045a0b10|
00000090  34 38 38 31 32 30 30 30  30 30 30 30 30 30 30 30  |4881200000000000|
000000a0  22 2c 0a 20 20 20 20 20  20 22 73 79 73 66 73 22  |",.      "sysfs"|
000000b0  20 3a 20 22 64 6d 2d 32  31 22 2c 0a 20 20 20 20  | : "dm-21",.    |
000000c0  20 20 22 66 61 69 6c 62  61 63 6b 22 20 3a 20 22  |  "failback" : "|
000000d0  69 6d 6d 65 64 69 61 74  65 22 2c 0a 20 20 20 20  |immediate",.    |
000000e0  20 20 22 71 75 65 75 65  69 6e 67 22 20 3a 20 22  |  "queueing" : "|
000000f0  6f 6e 22 2c 0a 20 20 20  20 20 20 22 70 61 74 68  |on",.      "path|
00000100  73 22 20 3a 20 31 2c 0a  20 20 20 20 20 20 22 77  |s" : 1,.      "w|
00000110  72 69 74 65 5f 70 72 6f  74 22 20 3a 20 22 72 77  |rite_prot" : "rw|
00000120  22 2c 0a 20 20 20 20 20  20 22 64 6d 5f 73 74 22  |",.      "dm_st"|
00000130  20 3a 20 22 61 63 74 69  76 65 22 2c 0a 20 20 20  | : "active",.   |
00000140  20 20 20 22 66 65 61 74  75 72 65 73 22 20 3a 20  |   "features" : |
00000150  22 31 20 71 75 65 75 65  5f 69 66 5f 6e 6f 5f 70  |"1 queue_if_no_p|
00000160  61 74 68 22 2c 0a 20 20  20 20 20 20 22 68 77 68  |ath",.      "hwh|
00000170  61 6e 64 6c 65 72 22 20  3a 20 22 30 22 2c 0a 20  |andler" : "0",. |
00000180  20 20 20 20 20 22 61 63  74 69 6f 6e 22 20 3a 20  |     "action" : |
00000190  22 22 2c 0a 20 20 20 20  20 20 22 70 61 74 68 5f  |"",.      "path_|
000001a0  66 61 75 6c 74 73 22 20  3a 20 30 2c 0a 20 20 20  |faults" : 0,.   |
000001b0  20 20 20 22 76 65 6e 64  22 20 3a 20 22 4c 49 4f  |   "vend" : "LIO|
000001c0  2d 4f 52 47 20 22 2c 0a  20 20 20 20 20 20 22 70  |-ORG ",.      "p|
000001d0  72 6f 64 22 20 3a 20 22  61 30 34 35 61 30 62 20  |rod" : "a045a0b |
000001e0  20 20 20 20 20 20 20 20  22 2c 0a 20 20 20 20 20  |        ",.     |
000001f0  20 22 72 65 76 22 20 3a  20 22 34 2e 30 20 22 2c  | "rev" : "4.0 ",|
00000200  0a 20 20 20 20 20 20 22  73 77 69 74 63 68 5f 67  |.      "switch_g|
00000210  72 70 22 20 3a 20 30 2c  0a 20 20 20 20 20 20 22  |rp" : 0,.      "|
00000220  6d 61 70 5f 6c 6f 61 64  73 22 20 3a 20 31 2c 0a  |map_loads" : 1,.|
00000230  20 20 20 20 20 20 22 74  6f 74 61 6c 5f 71 5f 74  |      "total_q_t|
00000240  69 6d 65 22 20 3a 20 30  2c 0a 20 20 20 20 20 20  |ime" : 0,.      |
00000250  22 71 5f 74 69 6d 65 6f  75 74 73 22 20 3a 20 30  |"q_timeouts" : 0|
00000260  2c 0a 20 20 20 20 20 20  22 70 61 74 68 5f 67 72  |,.      "path_gr|
00000270  6f 75 70 73 22 3a 20 5b  7b 0a 20 20 20 20 20 20  |oups": [{.      |
00000280  20 20 20 22 73 65 6c 65  63 74 6f 72 22 20 3a 20  |   "selector" : |
00000290  22 73 65 72 76 69 63 65  2d 74 69 6d 65 20 30 22  |"service-time 0"|
000002a0  2c 0a 20 20 20 20 20 20  20 20 20 22 70 72 69 22  |,.         "pri"|
000002b0  20 3a 20 31 2c 0a 20 20  20 20 20 20 20 20 20 22  | : 1,.         "|
000002c0  64 6d 5f 73 74 22 20 3a  20 22 61 63 74 69 76 65  |dm_st" : "active|
000002d0  22 2c 0a 20 20 20 20 20  20 20 20 20 22 67 72 6f  |",.         "gro|
000002e0  75 70 22 20 3a 20 31 2c  0a 20 20 20 20 20 20 20  |up" : 1,.       |
000002f0  20 20 22 70 61 74 68 73  22 3a 20 5b 7b 0a 20 20  |  "paths": [{.  |
00000300  20 20 20 20 20 20 20 20  20 20 22 64 65 76 22 20  |          "dev" |
00000310  3a 20 22 73 64 68 22 2c  0a 20 20 20 20 20 20 20  |: "sdh",.       |
00000320  20 20 20 20 20 22 64 65  76 5f 74 22 20 3a 20 22  |     "dev_t" : "|
00000330  38 3a 31 31 32 22 2c 0a  20 20 20 20 20 20 20 20  |8:112",.        |
00000340  20 20 20 20 22 64 6d 5f  73 74 22 20 3a 20 22 61  |    "dm_st" : "a|
00000350  63 74 69 76 65 22 2c 0a  20 20 20 20 20 20 20 20  |ctive",.        |
00000360  20 20 20 20 22 64 65 76  5f 73 74 22 20 3a 20 22  |    "dev_st" : "|
00000370  72 75 6e 6e 69 6e 67 22  2c 0a 20 20 20 20 20 20  |running",.      |
00000380  20 20 20 20 20 20 22 63  68 6b 5f 73 74 22 20 3a  |      "chk_st" :|
00000390  20 22 72 65 61 64 79 22  2c 0a 20 20 20 20 20 20  | "ready",.      |
000003a0  20 20 20 20 20 20 22 63  68 65 63 6b 65 72 22 20  |      "checker" |
000003b0  3a 20 22 64 69 72 65 63  74 69 6f 22 2c 0a 20 20  |: "directio",.  |
000003c0  20 20 20 20 20 20 20 20  20 20 22 70 72 69 22 20  |          "pri" |
000003d0  3a 20 31 2c 0a 20 20 20  20 20 20 20 20 20 20 20  |: 1,.           |
000003e0  20 22 68 6f 73 74 5f 77  77 6e 6e 22 20 3a 20 22  | "host_wwnn" : "|
000003f0  5b 75 6e 64 65 66 5d 22  2c 0a 20 20 20 20 20 20  |[undef]",.      |
00000400  20 20 20 20 20 20 22 74  61 72 67 65 74 5f 77 77  |      "target_ww|
00000410  6e 6e 22 20 3a 20 22 69  71 6e 2e 32 30 30 34 2d  |nn" : "iqn.2004-|
00000420  31 32 2e 63 6f 6d 2e 62  65 74 74 65 72 73 65 72  |12.com.betterser|
00000430  76 65 72 73 3a 74 63 70  2d 31 30 2e 34 2e 38 38  |vers:tcp-10.4.88|
00000440  2e 31 32 22 2c 0a 20 20  20 20 20 20 20 20 20 20  |.12",.          |
00000450  20 20 22 68 6f 73 74 5f  77 77 70 6e 22 20 3a 20  |  "host_wwpn" : |
00000460  22 5b 75 6e 64 65 66 5d  22 2c 0a 20 20 20 20 20  |"[undef]",.     |
00000470  20 20 20 20 20 20 20 22  74 61 72 67 65 74 5f 77  |       "target_w|
00000480  77 70 6e 22 20 3a 20 22  5b 75 6e 64 65 66 5d 22  |wpn" : "[undef]"|
00000490  2c 0a 20 20 20 20 20 20  20 20 20 20 20 20 22 68  |,.            "h|
000004a0  6f 73 74 5f 61 64 61 70  74 65 72 22 20 3a 20 22  |ost_adapter" : "|
000004b0  31 30 2e 34 2e 36 34 2e  31 33 22 0a 20 20 20 20  |10.4.64.13".    |
000004c0  20 20 20 20 20 7d 5d 0a  20 20 20 20 20 20 7d 5d  |     }].      }]|
000004d0  0a 20 20 20 7d 0a 7d 0a  81 09 02                 |.   }.}....|

Expected results:
00000000  7b 0a 20 20 20 22 6d 61  6a 6f 72 5f 76 65 72 73  |{.   "major_vers|
00000010  69 6f 6e 22 3a 20 30 2c  0a 20 20 20 22 6d 69 6e  |ion": 0,.   "min|
00000020  6f 72 5f 76 65 72 73 69  6f 6e 22 3a 20 31 2c 0a  |or_version": 1,.|
00000030  20 20 20 22 6d 61 70 22  3a 7b 0a 20 20 20 20 20  |   "map":{.     |
00000040  20 22 6e 61 6d 65 22 20  3a 20 22 33 36 30 30 31  | "name" : "36001|
00000050  34 30 35 61 30 34 35 61  30 62 31 30 34 38 38 31  |405a045a0b104881|
00000060  32 30 30 30 30 30 30 30  30 30 30 30 22 2c 0a 20  |200000000000",. |
00000070  20 20 20 20 20 22 75 75  69 64 22 20 3a 20 22 33  |     "uuid" : "3|
00000080  36 30 30 31 34 30 35 61  30 34 35 61 30 62 31 30  |6001405a045a0b10|
00000090  34 38 38 31 32 30 30 30  30 30 30 30 30 30 30 30  |4881200000000000|
000000a0  22 2c 0a 20 20 20 20 20  20 22 73 79 73 66 73 22  |",.      "sysfs"|
000000b0  20 3a 20 22 64 6d 2d 32  31 22 2c 0a 20 20 20 20  | : "dm-21",.    |
000000c0  20 20 22 66 61 69 6c 62  61 63 6b 22 20 3a 20 22  |  "failback" : "|
000000d0  69 6d 6d 65 64 69 61 74  65 22 2c 0a 20 20 20 20  |immediate",.    |
000000e0  20 20 22 71 75 65 75 65  69 6e 67 22 20 3a 20 22  |  "queueing" : "|
000000f0  6f 6e 22 2c 0a 20 20 20  20 20 20 22 70 61 74 68  |on",.      "path|
00000100  73 22 20 3a 20 31 2c 0a  20 20 20 20 20 20 22 77  |s" : 1,.      "w|
00000110  72 69 74 65 5f 70 72 6f  74 22 20 3a 20 22 72 77  |rite_prot" : "rw|
00000120  22 2c 0a 20 20 20 20 20  20 22 64 6d 5f 73 74 22  |",.      "dm_st"|
00000130  20 3a 20 22 61 63 74 69  76 65 22 2c 0a 20 20 20  | : "active",.   |
00000140  20 20 20 22 66 65 61 74  75 72 65 73 22 20 3a 20  |   "features" : |
00000150  22 31 20 71 75 65 75 65  5f 69 66 5f 6e 6f 5f 70  |"1 queue_if_no_p|
00000160  61 74 68 22 2c 0a 20 20  20 20 20 20 22 68 77 68  |ath",.      "hwh|
00000170  61 6e 64 6c 65 72 22 20  3a 20 22 30 22 2c 0a 20  |andler" : "0",. |
00000180  20 20 20 20 20 22 61 63  74 69 6f 6e 22 20 3a 20  |     "action" : |
00000190  22 22 2c 0a 20 20 20 20  20 20 22 70 61 74 68 5f  |"",.      "path_|
000001a0  66 61 75 6c 74 73 22 20  3a 20 30 2c 0a 20 20 20  |faults" : 0,.   |
000001b0  20 20 20 22 76 65 6e 64  22 20 3a 20 22 4c 49 4f  |   "vend" : "LIO|
000001c0  2d 4f 52 47 20 22 2c 0a  20 20 20 20 20 20 22 70  |-ORG ",.      "p|
000001d0  72 6f 64 22 20 3a 20 22  61 30 34 35 61 30 62 20  |rod" : "a045a0b |
000001e0  20 20 20 20 20 20 20 20  22 2c 0a 20 20 20 20 20  |        ",.     |
000001f0  20 22 72 65 76 22 20 3a  20 22 34 2e 30 20 22 2c  | "rev" : "4.0 ",|
00000200  0a 20 20 20 20 20 20 22  73 77 69 74 63 68 5f 67  |.      "switch_g|
00000210  72 70 22 20 3a 20 30 2c  0a 20 20 20 20 20 20 22  |rp" : 0,.      "|
00000220  6d 61 70 5f 6c 6f 61 64  73 22 20 3a 20 31 2c 0a  |map_loads" : 1,.|
00000230  20 20 20 20 20 20 22 74  6f 74 61 6c 5f 71 5f 74  |      "total_q_t|
00000240  69 6d 65 22 20 3a 20 30  2c 0a 20 20 20 20 20 20  |ime" : 0,.      |
00000250  22 71 5f 74 69 6d 65 6f  75 74 73 22 20 3a 20 30  |"q_timeouts" : 0|
00000260  2c 0a 20 20 20 20 20 20  22 70 61 74 68 5f 67 72  |,.      "path_gr|
00000270  6f 75 70 73 22 3a 20 5b  7b 0a 20 20 20 20 20 20  |oups": [{.      |
00000280  20 20 20 22 73 65 6c 65  63 74 6f 72 22 20 3a 20  |   "selector" : |
00000290  22 73 65 72 76 69 63 65  2d 74 69 6d 65 20 30 22  |"service-time 0"|
000002a0  2c 0a 20 20 20 20 20 20  20 20 20 22 70 72 69 22  |,.         "pri"|
000002b0  20 3a 20 31 2c 0a 20 20  20 20 20 20 20 20 20 22  | : 1,.         "|
000002c0  64 6d 5f 73 74 22 20 3a  20 22 61 63 74 69 76 65  |dm_st" : "active|
000002d0  22 2c 0a 20 20 20 20 20  20 20 20 20 22 67 72 6f  |",.         "gro|
000002e0  75 70 22 20 3a 20 31 2c  0a 20 20 20 20 20 20 20  |up" : 1,.       |
000002f0  20 20 22 70 61 74 68 73  22 3a 20 5b 7b 0a 20 20  |  "paths": [{.  |
00000300  20 20 20 20 20 20 20 20  20 20 22 64 65 76 22 20  |          "dev" |
00000310  3a 20 22 73 64 68 22 2c  0a 20 20 20 20 20 20 20  |: "sdh",.       |
00000320  20 20 20 20 20 22 64 65  76 5f 74 22 20 3a 20 22  |     "dev_t" : "|
00000330  38 3a 31 31 32 22 2c 0a  20 20 20 20 20 20 20 20  |8:112",.        |
00000340  20 20 20 20 22 64 6d 5f  73 74 22 20 3a 20 22 61  |    "dm_st" : "a|
00000350  63 74 69 76 65 22 2c 0a  20 20 20 20 20 20 20 20  |ctive",.        |
00000360  20 20 20 20 22 64 65 76  5f 73 74 22 20 3a 20 22  |    "dev_st" : "|
00000370  72 75 6e 6e 69 6e 67 22  2c 0a 20 20 20 20 20 20  |running",.      |
00000380  20 20 20 20 20 20 22 63  68 6b 5f 73 74 22 20 3a  |      "chk_st" :|
00000390  20 22 72 65 61 64 79 22  2c 0a 20 20 20 20 20 20  | "ready",.      |
000003a0  20 20 20 20 20 20 22 63  68 65 63 6b 65 72 22 20  |      "checker" |
000003b0  3a 20 22 64 69 72 65 63  74 69 6f 22 2c 0a 20 20  |: "directio",.  |
000003c0  20 20 20 20 20 20 20 20  20 20 22 70 72 69 22 20  |          "pri" |
000003d0  3a 20 31 2c 0a 20 20 20  20 20 20 20 20 20 20 20  |: 1,.           |
000003e0  20 22 68 6f 73 74 5f 77  77 6e 6e 22 20 3a 20 22  | "host_wwnn" : "|
000003f0  5b 75 6e 64 65 66 5d 22  2c 0a 20 20 20 20 20 20  |[undef]",.      |
00000400  20 20 20 20 20 20 22 74  61 72 67 65 74 5f 77 77  |      "target_ww|
00000410  6e 6e 22 20 3a 20 22 69  71 6e 2e 32 30 30 34 2d  |nn" : "iqn.2004-|
00000420  31 32 2e 63 6f 6d 2e 62  65 74 74 65 72 73 65 72  |12.com.betterser|
00000430  76 65 72 73 3a 74 63 70  2d 31 30 2e 34 2e 38 38  |vers:tcp-10.4.88|
00000440  2e 31 32 22 2c 0a 20 20  20 20 20 20 20 20 20 20  |.12",.          |
00000450  20 20 22 68 6f 73 74 5f  77 77 70 6e 22 20 3a 20  |  "host_wwpn" : |
00000460  22 5b 75 6e 64 65 66 5d  22 2c 0a 20 20 20 20 20  |"[undef]",.     |
00000470  20 20 20 20 20 20 20 22  74 61 72 67 65 74 5f 77  |       "target_w|
00000480  77 70 6e 22 20 3a 20 22  5b 75 6e 64 65 66 5d 22  |wpn" : "[undef]"|
00000490  2c 0a 20 20 20 20 20 20  20 20 20 20 20 20 22 68  |,.            "h|
000004a0  6f 73 74 5f 61 64 61 70  74 65 72 22 20 3a 20 22  |ost_adapter" : "|
000004b0  31 30 2e 34 2e 36 34 2e  31 33 22 0a 20 20 20 20  |10.4.64.13".    |
000004c0  20 20 20 20 20 7d 5d 0a  20 20 20 20 20 20 7d 5d  |     }].      }]|
000004d0  0a 20 20 20 7d 0a 7d 0a                           |.   }.}.|

Additional info:
Comment 2 Ben Marzinski 2017-07-27 15:22:22 EDT
I'm not able to reproduce this yet. Nor do I see where this could be happening in the json specific code.  Is it possible that you are getting garbage added to the end of the multiple multipathd commands, but only noticing it with the json code, since they might be non-printing characters?

If you could run some other multipath commands, such as 

# multipathd show map <name>
# multiapthd show config
# mulitpaths show maps topology

etc.

and pipe the output through hexdump, that would be great.  I'd like to see if there is garbage at the end of other commands, or just the json ones.

Another question. When this happens, is the output always the same size? Is the output always a different size when this doesn't happen?
Comment 3 Robert LeBlanc 2017-07-27 15:43:22 EDT
All of our maps are behaving at the moment, but once one of them starts acting up again, we will run the commands. Currently, the commands you asked for don't have any extra characters or data.

The garbage is always in addition to the good data. For now we are just truncating the string/output after the last '}' and that is working for us but thought we would raise the bug to clean up things. When we run into the garbage, if we run the command multiple times, it is the same number of garbage characters/bytes. I don't know if it is different across different maps or if it is a constant number regardless of the size of the map output. I'll keep an eye out for it when it happens again.

Note You need to log in before you can comment on or make changes to this bug.