I've started a build on a an aarch64 box, and will continue to push this forward to analyze what went wrong.

Rawhide is effectively upstream glibc master, so I wonder why ARM or Linaro didn't see this failure, thus I suspect our tools are slightly out of sync with upstream.

> Rawhide is effectively upstream glibc master, so I wonder why ARM or Linaro
> didn't see this failure, thus I suspect our tools are slightly out of sync
> with upstream.

It isn't the first time they've submitted something upstream that has seen little to no testing to see if it actually works :-/

OK, so upstream build on an aarch64 box is just fine with no problems.

I need to look at the builder and do a mock chroot build to see what's possibly wrong.

(In reply to Peter Robinson from comment #2)
> > Rawhide is effectively upstream glibc master, so I wonder why ARM or Linaro
> > didn't see this failure, thus I suspect our tools are slightly out of sync
> > with upstream.
> 
> It isn't the first time they've submitted something upstream that has seen
> little to no testing to see if it actually works :-/

Well, I always do a `fedpkg build --scrach --srcpm foo.src.rpm` and then look at the logs before a final push. The problem is that in this process I don't get to peek at the secondary arch builders, and often forget.

> Well, I always do a `fedpkg build --scrach --srcpm foo.src.rpm` and then
> look at the logs before a final push. The problem is that in this process I
> don't get to peek at the secondary arch builders, and often forget.

"<arch>-koji build --scratch f22 foo.src.rpm" where <arch> is arm/ppc/s390 and that will give you the same for the secondary arches

(In reply to Peter Robinson from comment #5)
> > Well, I always do a `fedpkg build --scrach --srcpm foo.src.rpm` and then
> > look at the logs before a final push. The problem is that in this process I
> > don't get to peek at the secondary arch builders, and often forget.
> 
> "<arch>-koji build --scratch f22 foo.src.rpm" where <arch> is arm/ppc/s390
> and that will give you the same for the secondary arches

Dear RCM,

Please collate all secondary build results in a master review page for the build ;-)

Sincerely,
Carlos.

I can reproduce it at will on the builders.

Switching to mock builds.

Reading symbols from /builddir/build/BUILD/glibc-2.20-276-g0e7e69b/build-aarch64-redhat-linux/elf/ld-linux-aarch64.so.1...done.
(gdb) run
Starting program: /builddir/build/BUILD/glibc-2.20-276-g0e7e69b/build-aarch64-redhat-linux/elf/ld-linux-aarch64.so.1 --library-path /builddir/build/BUILD/glibc-2.20-276-g0e7e69b/build-aarch64-redhat-linux:/builddir/build/BUILD/glibc-2.20-276-g0e7e69b/build-aarch64-redhat-linux/math:/builddir/build/BUILD/glibc-2.20-276-g0e7e69b/build-aarch64-redhat-linux/elf:/builddir/build/BUILD/glibc-2.20-276-g0e7e69b/build-aarch64-redhat-linux/dlfcn:/builddir/build/BUILD/glibc-2.20-276-g0e7e69b/build-aarch64-redhat-linux/nss:/builddir/build/BUILD/glibc-2.20-276-g0e7e69b/build-aarch64-redhat-linux/nis:/builddir/build/BUILD/glibc-2.20-276-g0e7e69b/build-aarch64-redhat-linux/rt:/builddir/build/BUILD/glibc-2.20-276-g0e7e69b/build-aarch64-redhat-linux/resolv:/builddir/build/BUILD/glibc-2.20-276-g0e7e69b/build-aarch64-redhat-linux/crypt:/builddir/build/BUILD/glibc-2.20-276-g0e7e69b/build-aarch64-redhat-linux/nptl /builddir/build/BUILD/glibc-2.20-276-g0e7e69b/build-aarch64-redhat-linux/locale/localedef --alias-file=../intl/locale.alias --no-archive -i locales/en_US -c -f charmaps/UTF-8 --prefix=/builddir/build/BUILDROOT/glibc-2.20.90-10.fc22.aarch64 en_US.UTF-8

Program received signal SIGSEGV, Segmentation fault.
0x000003ffb7eab55c in _IO_vfprintf_internal (s=s@entry=0x3ffffffec00, format=format@entry=0x43d940 "%s%s/%.*s%s%s%c", ap=...) at vfprintf.c:1642
1642              process_string_arg (((struct printf_spec *) NULL));
(gdb) bt
#0  0x000003ffb7eab55c in _IO_vfprintf_internal (s=s@entry=0x3ffffffec00, format=format@entry=0x43d940 "%s%s/%.*s%s%s%c", ap=...) at vfprintf.c:1642
#1  0x000003ffb7ed1ec0 in _IO_vasprintf (result_ptr=0x3ffffffee88, format=0x43d940 "%s%s/%.*s%s%s%c", args=...) at vasprintf.c:62
#2  0x000003ffb7eb2070 in ___asprintf (string_ptr=<optimized out>, format=<optimized out>) at asprintf.c:35
#3  0x0000000000403bd0 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
(gdb) 

fun.

      /* We put an additional '\0' at the end of the string because at
	 the end of the function we need another byte for the trailing
	 '/'.  */
      ssize_t n;
      if (normal == NULL)
	n = asprintf (&result, "%s%s/%s%c",
		      output_prefix ?: "", LOCALEDIR, path, '\0');
      else
	n = asprintf (&result, "%s%s/%.*s%s%s%c",
		      output_prefix ?: "", LOCALEDIR,
		      (int) (startp - path), path, normal, endp, '\0');

looks like construct_output_path line 463.

(In reply to Kyle McMartin from comment #9)
>       /* We put an additional '\0' at the end of the string because at
> 	 the end of the function we need another byte for the trailing
> 	 '/'.  */
>       ssize_t n;
>       if (normal == NULL)
> 	n = asprintf (&result, "%s%s/%s%c",
> 		      output_prefix ?: "", LOCALEDIR, path, '\0');
>       else
> 	n = asprintf (&result, "%s%s/%.*s%s%s%c",
> 		      output_prefix ?: "", LOCALEDIR,
> 		      (int) (startp - path), path, normal, endp, '\0');
> 
> looks like construct_output_path line 463.

This really looks like a compiler issue. You can swap out vfprintf.os from another build, and relink, and see if the newly relinked version works properly.

inclined to agree, yes.

i've bisected this down to:

master@glibc:.% git log sysdeps/aarch64/strchrnul.S                                                                                                                           (kyle@dreadnought:~/src/glibc)
commit be9d4ccc7fe62751db1a5fdcb31958561dbbda9a
Author: Richard Earnshaw <rearnsha>
Date:   Wed Nov 5 13:51:56 2014 +0000

    [AArch64] Add optimized strchrnul.
    
    Here is an optimized implementation of __strchrnul.  The
    simplification that we don't have to track precisely why the loop
    terminates (match or end-of-string) means we have to do less work in
    both setup and the core inner loop.  That means this should never be
    slower than strchr.
    
    As with strchr, the use of LD1 means we do not need different versions
    for big-/little-endian.
master@glibc:.%    

which is more than a little strange...

(In reply to Kyle McMartin from comment #12)
> i've bisected this down to:
> 
> master@glibc:.% git log sysdeps/aarch64/strchrnul.S                         
> (kyle@dreadnought:~/src/glibc)
> commit be9d4ccc7fe62751db1a5fdcb31958561dbbda9a
> Author: Richard Earnshaw <rearnsha>
> Date:   Wed Nov 5 13:51:56 2014 +0000
> 
>     [AArch64] Add optimized strchrnul.
>     
>     Here is an optimized implementation of __strchrnul.  The
>     simplification that we don't have to track precisely why the loop
>     terminates (match or end-of-string) means we have to do less work in
>     both setup and the core inner loop.  That means this should never be
>     slower than strchr.
>     
>     As with strchr, the use of LD1 means we do not need different versions
>     for big-/little-endian.
> master@glibc:.%    
> 
> which is more than a little strange...

Could be an assembly bug in that it violates the EABI, or a compiler bug in calling this assembly function with incorrect EABI invariants e.g. stack alignment etc.

I'm inclined to look for the caller violating the EABI, the assembly expecting something and not getting it, and stomping on the stack, and a failure happening later in the chain of events.

indeed... currently hypothesis is that __find_specmb is failing interestingly. following that line of thinking now.

this ended up being a caller saved vector register getting clobbered by strchrnul. fixed in fd8c9e7.