Description of problem:
After opening a file, a stat call on the file sometimes returns an inode number with the high 32 bits all ones.  Subsequently it returns the normal inode number with high bits all zeroes.  A read between the open and stat calls seems to greatly increase the likelyhood of the inode mangling occurring.

Certain utilities depend on the inode number of a file staying constant.  For example, find uses inode numbers to determine whether it arrived back in the same directory after visiting a subdirectory.  If the inode number has changed, find gets confused and halts the search prematurely.

Another effect is that stat calls can fail on 32-bit systems.  The system call interface always uses a 64-bit struct stat, and for backwards (binary) compatibility glibc provides a 32-bit wrapper.  This wrapper will return EOVERFLOW if some of the values in the 64-bit struct, such as inode number of size, are too large for the 32-bit fields.  Presently I'm seeing this problem in libexiv2, which fails to extract metadata from images because it can't determine the size of the file.

How reproducible:
Always reproduces with the same access pattern on the same file.  Not all files are affected.

Steps to Reproduce:
1. execute a stat syscall on a file
2. open the file
3. read some data from the file (optional, increases likelyhood of bug occurrence)
4. execute the same stat syscall again
  
Actual results:
The second stat returns an st_ino which is 0xffffffff00000000|st_ino of the first call

Expected results:
Both stat calls return the same st_ino

Additional info:
Here's a sequence of system calls from a test program:
stat64("stattest", {st_dev=makedev(0, 17), st_ino=1056645660, st_mode=S_IFREG|0755, st_nlink=1, st_uid=1000, st_gid=1000, st_blksize=131072, st_blocks=16, st_size=5902, st_atime=2012/06/06-11:38:47, st_mtime=2012/06/06-11:38:46, st_ctime=2012/06/06-11:38:46}) = 0
open("stattest", O_RDONLY)              = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\2\0\3\0\1\0\0\0`\204\4\0104\0\0\0"..., 4096) = 4096
fstat64(3, {st_dev=makedev(0, 17), st_ino=18446744070471229980, st_mode=S_IFREG|0755, st_nlink=1, st_uid=1000, st_gid=1000, st_blksize=131072, st_blocks=16, st_size=5902, st_atime=2012/06/06-11:38:47, st_mtime=2012/06/06-11:38:46, st_ctime=2012/06/06-11:38:46}) = 0
stat64("stattest", {st_dev=makedev(0, 17), st_ino=1056645660, st_mode=S_IFREG|0755, st_nlink=1, st_uid=1000, st_gid=1000, st_blksize=131072, st_blocks=16, st_size=5902, st_atime=2012/06/06-11:38:47, st_mtime=2012/06/06-11:38:46, st_ctime=2012/06/06-11:38:46}) = 0
close(3)                                = 0

And another from exiv2:
stat64("IMG_3722.CR2", {st_dev=makedev(0, 17), st_ino=2273457493, st_mode=S_IFREG|0600, st_nlink=1, st_uid=1000, st_gid=1000, st_blksize=131072, st_blocks=55144, st_size=28232633, st_atime=2012/06/05-22:43:31, st_mtime=2012/04/21-14:43:46, st_ctime=2012/06/05-22:43:00}) = 0
open("IMG_3722.CR2", O_RDONLY)          = 3
fstat64(3, {st_dev=makedev(0, 17), st_ino=2273457493, st_mode=S_IFREG|0600, st_nlink=1, st_uid=1000, st_gid=1000, st_blksize=131072, st_blocks=55144, st_size=28232633, st_atime=2012/06/05-22:43:31, st_mtime=2012/04/21-14:43:46, st_ctime=2012/06/05-22:43:00}) = 0
read(3, "II*\0\20\0\0\0CR\2\0J\255\0\0\21\0\0\1\3\0\1\0\0\0\360\25\0\0\1\1"..., 131072) = 131072
_llseek(3, -131070, [2], SEEK_CUR)      = 0
_llseek(3, 0, [0], SEEK_SET)            = 0
read(3, "II*\0\20\0\0\0CR\2\0J\255\0\0\21\0\0\1\3\0\1\0\0\0\360\25\0\0\1\1"..., 131072) = 131072
_llseek(3, 131072, [131072], SEEK_SET)  = 0
_llseek(3, 131072, [131072], SEEK_SET)  = 0
_llseek(3, 131072, [131072], SEEK_SET)  = 0
_llseek(3, 131072, [131072], SEEK_SET)  = 0
close(3)                                = 0
open("IMG_3722.CR2", O_RDONLY)          = 3
fstat64(3, {st_dev=makedev(0, 17), st_ino=2273457493, st_mode=S_IFREG|0600, st_nlink=1, st_uid=1000, st_gid=1000, st_blksize=131072, st_blocks=55144, st_size=28232633, st_atime=2012/06/05-22:43:31, st_mtime=2012/04/21-14:43:46, st_ctime=2012/06/05-22:43:00}) = 0
read(3, "II*\0\20\0\0\0CR\2\0J\255\0\0\21\0\0\1\3\0\1\0\0\0\360\25\0\0\1\1"..., 131072) = 131072
_llseek(3, -131056, [16], SEEK_CUR)     = 0
_llseek(3, 0, [0], SEEK_SET)            = 0
close(3)                                = 0
open("IMG_3722.CR2", O_RDONLY)          = 3
fstat64(3, {st_dev=makedev(0, 17), st_ino=2273457493, st_mode=S_IFREG|0600, st_nlink=1, st_uid=1000, st_gid=1000, st_blksize=131072, st_blocks=55144, st_size=28232633, st_atime=2012/06/05-22:43:31, st_mtime=2012/04/21-14:43:46, st_ctime=2012/06/05-22:43:00}) = 0
read(3, "II*\0\20\0\0\0CR\2\0J\255\0\0\21\0\0\1\3\0\1\0\0\0\360\25\0\0\1\1"..., 131072) = 131072
_llseek(3, -131056, [16], SEEK_CUR)     = 0
_llseek(3, 0, [0], SEEK_SET)            = 0
stat64("IMG_3722.CR2", {st_dev=makedev(0, 17), st_ino=18446744071688041813, st_mode=S_IFREG|0600, st_nlink=1, st_uid=1000, st_gid=1000, st_blksize=131072, st_blocks=55144, st_size=28232633, st_atime=2012/06/05-22:43:31, st_mtime=2012/04/21-14:43:46, st_ctime=2012/06/05-22:43:00}) = 0
stat64("IMG_3722.CR2", {st_dev=makedev(0, 17), st_ino=2273457493, st_mode=S_IFREG|0600, st_nlink=1, st_uid=1000, st_gid=1000, st_blksize=131072, st_blocks=55144, st_size=28232633, st_atime=2012/06/05-22:43:31, st_mtime=2012/04/21-14:43:46, st_ctime=2012/06/05-22:43:00}) = 0
mmap2(NULL, 28232633, PROT_READ, MAP_SHARED, 3, 0) = 0xb5601000
close(3)                                = 0

GlusterFS volume info:
Volume Name: home
Type: Distribute
Status: Started
Number of Bricks: 1
Transport-type: tcp
Bricks:
Brick1: capybara:/home
Options Reconfigured:
performance.io-cache: on
nfs.enable-ino32: off