Bug 506222 - memchr() with huge length returns unexpected NULL
memchr() with huge length returns unexpected NULL
Product: Fedora
Classification: Fedora
Component: glibc (Show other bugs)
x86_64 Linux
low Severity high
: ---
: ---
Assigned To: Jakub Jelinek
Fedora Extras Quality Assurance
Depends On:
  Show dependency treegraph
Reported: 2009-06-16 03:01 EDT by Petr Vandrovec
Modified: 2009-07-24 05:35 EDT (History)
4 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Last Closed: 2009-07-24 05:35:07 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)

  None (edit)
Description Petr Vandrovec 2009-06-16 03:01:33 EDT
Description of problem:

memchr() with long length returns NULL although searched character exists in the searched string.  Problem does not exist in glibc 2.9 or older.  Problem was found because VMware customers are reporting ASSERTs being hit in VMware's code due to unexpected memchr() return value.

Version-Release number of selected component (if applicable):

2.10.1 according to the http://communities.vmware.com/thread/214671.

How reproducible:

Problem is 100% reproducible using code below.

Steps to Reproduce:
/* make sure to build this as 64bit binary. Code should report 0x2525b80 0x2525bc9.  On current Fedora (with 2.10.1 glibc) it returns 0x2525b80 (nil). */
#include <sys/mman.h>
#include <stdio.h>
#include <string.h>

void main(void) {
   char* abc;

   abc = mmap(0x00002525000, 16384, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
   abc += 0xB80;
   strcpy(abc, "/mnt/data/dan/VMware/Windows XP Professional/Windows XP Professional.vmdk");
   printf("%p %p\n", abc, memchr(abc, 0, -(unsigned long)abc));

Actual results:

0x2525b80 (nil)

Expected results:

0x2525b80 0x2525bc9

Additional info:

memchr() behavior above causes vmware to crash, as it expects that memchr(string, 0, -(unsigned long)string) on NUL terminated string returns NULL only if string is not terminated at all.
Comment 1 Petr Vandrovec 2009-06-16 04:57:41 EDT
Introduced in 2.10.1 by  http://sourceware.org/cgi-bin/cvsweb.cgi/libc/sysdeps/x86_64/memchr.S.diff?cvsroot=glibc&r1=NONE&r2=1.1

It uses signed comparsion between number of characters processed (%rsi) and requested for processing (%rdx):

cmp %rsi,%rdx
jle xxx

It means that memchr() with length argument over 2^63 will immediately fail these tests, returning NULL.

Switching comparsions to unsigned (and depending on the fact that before %rsi can reach 2^63 process will be killed by #GP or #PF so there is no possibility of looping forever for length=-1..-15) seems to do the trick.
Comment 2 Jakub Jelinek 2009-06-19 08:57:49 EDT
Fixed in git so far.
Comment 3 Jakub Jelinek 2009-07-24 05:35:07 EDT
This is fixed in rawhide for quite sime time as well.

Note You need to log in before you can comment on or make changes to this bug.