Bug 506222 - memchr() with huge length returns unexpected NULL
Summary: memchr() with huge length returns unexpected NULL
Keywords:
Status: CLOSED RAWHIDE
Alias: None
Product: Fedora
Classification: Fedora
Component: glibc
Version: rawhide
Hardware: x86_64
OS: Linux
low
high
Target Milestone: ---
Assignee: Jakub Jelinek
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2009-06-16 07:01 UTC by Petr Vandrovec
Modified: 2009-07-24 09:35 UTC (History)
4 users (show)

Fixed In Version:
Clone Of:
Environment:
Last Closed: 2009-07-24 09:35:07 UTC
Type: ---
Embargoed:


Attachments (Terms of Use)

Description Petr Vandrovec 2009-06-16 07:01:33 UTC
Description of problem:

memchr() with long length returns NULL although searched character exists in the searched string.  Problem does not exist in glibc 2.9 or older.  Problem was found because VMware customers are reporting ASSERTs being hit in VMware's code due to unexpected memchr() return value.

Version-Release number of selected component (if applicable):

2.10.1 according to the http://communities.vmware.com/thread/214671.

How reproducible:

Problem is 100% reproducible using code below.

Steps to Reproduce:
  
/* make sure to build this as 64bit binary. Code should report 0x2525b80 0x2525bc9.  On current Fedora (with 2.10.1 glibc) it returns 0x2525b80 (nil). */
#include <sys/mman.h>
#include <stdio.h>
#include <string.h>

void main(void) {
   char* abc;

   abc = mmap(0x00002525000, 16384, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
   abc += 0xB80;
   strcpy(abc, "/mnt/data/dan/VMware/Windows XP Professional/Windows XP Professional.vmdk");
   printf("%p %p\n", abc, memchr(abc, 0, -(unsigned long)abc));
}

Actual results:

0x2525b80 (nil)

Expected results:

0x2525b80 0x2525bc9

Additional info:

memchr() behavior above causes vmware to crash, as it expects that memchr(string, 0, -(unsigned long)string) on NUL terminated string returns NULL only if string is not terminated at all.

Comment 1 Petr Vandrovec 2009-06-16 08:57:41 UTC
Introduced in 2.10.1 by  http://sourceware.org/cgi-bin/cvsweb.cgi/libc/sysdeps/x86_64/memchr.S.diff?cvsroot=glibc&r1=NONE&r2=1.1

It uses signed comparsion between number of characters processed (%rsi) and requested for processing (%rdx):

cmp %rsi,%rdx
jle xxx

It means that memchr() with length argument over 2^63 will immediately fail these tests, returning NULL.

Switching comparsions to unsigned (and depending on the fact that before %rsi can reach 2^63 process will be killed by #GP or #PF so there is no possibility of looping forever for length=-1..-15) seems to do the trick.

Comment 2 Jakub Jelinek 2009-06-19 12:57:49 UTC
Fixed in git so far.

Comment 3 Jakub Jelinek 2009-07-24 09:35:07 UTC
This is fixed in rawhide for quite sime time as well.


Note You need to log in before you can comment on or make changes to this bug.