Bug 1294897

When using a sha512 password on a system like this:

"""
Information
Server Name 	
Product Name                ProLiant DL380 Gen9
UUID                        88888888-8888-888A-8888-888888888888
Server Serial Number        AAA88888AA
Product ID                  719064-B21
System ROM                  P89 v1.40 (05/06/2015)
System ROM Date             05/06/2015
Backup System ROM           P89 v1.40 (05/06/2015)
Integrated Remote Console   .NET    Java
License Type                iLO 4 Advanced
iLO Firmware Version        2.20 May 20 2015
IP Address                  10.0.0.1
Link-Local IPv6 Address     AAAA::AAAA:AAAA:AAAA:AAAA
iLO Hostname                server.example.com
"""

the screen will blank after entering the password. It was reported
to me that it only happens when booting in UEFI mode.

  I made several test builds, and found where the problem
happens.

  *If* I do a quick hack patch like this:

---8<---
@@ -358,9 +370,10 @@ sha512_crypt_r (const char *key, const c
   char *p_bytes;
   char *s_bytes;
   /* Default number of rounds.  */
-  size_t rounds = ROUNDS_DEFAULT;
+  size_t rounds = 1;//ROUNDS_DEFAULT;
   bool rounds_custom = false;

it will not blank the screen, but will fail to match the
password.

  *If* I build grub with -O0 instead of -Os, it will not
blank the screen, but will fail to match the password as
well, and then, will boot half the time. When failing it
will show:

"""
Couldn t read file
Error 16: Inconsistent filesystem structure

Press any key to continue...
"""

  I believe it is a firmware bug, but am opening the bug
report in case this is somehow known.

  I made a test build as well, with this pseudo patch:

"""
---8<---
@@ -492,6 +528,7 @@ sha512_crypt_r (const char *key, const c
     }
   memcpy (cp, temp_result, cnt);
 
+  pause("26 ");
   /* Repeatedly run the collected hash value through SHA512 to burn
      CPU cycles.  */
   for (cnt = 0; cnt < rounds; ++cnt)
@@ -499,35 +536,42 @@ sha512_crypt_r (const char *key, const c
       /* New context.  */
       sha512_init_ctx (&ctx);
 
+  //pause("27 ");
       /* Add key or last result.  */
       if ((cnt & 1) != 0)
 	sha512_process_bytes (p_bytes, key_len, &ctx);
       else
 	sha512_process_bytes (alt_result, 64, &ctx);
 
+  //pause("28 ");
       /* Add salt for numbers not divisible by 3.  */
       if (cnt % 3 != 0)
 	sha512_process_bytes (s_bytes, salt_len, &ctx);
 
+  //pause("29 ");
       /* Add key for numbers not divisible by 7.  */
       if (cnt % 7 != 0)
 	sha512_process_bytes (p_bytes, key_len, &ctx);
 
+  //pause("30 ");
       /* Add key or last result.  */
       if ((cnt & 1) != 0)
 	sha512_process_bytes (alt_result, 64, &ctx);
       else
 	sha512_process_bytes (p_bytes, key_len, &ctx);
 
+  //pause("31 ");
       /* Create intermediate result.  */
       sha512_finish_ctx (&ctx, alt_result);
     }
 
+  pause("32 ");
   /* Now we can construct the result string.  It consists of three
      parts.  */
   cp = stpncpy (buffer, sha512_salt_prefix, MAX (0, buflen));
   buflen -= sizeof (sha512_salt_prefix) - 1;
 
+  pause("33 ");
   if (rounds_custom)
     {
       char sbuf[64];
@@ -539,6 +583,7 @@ sha512_crypt_r (const char *key, const c
       buflen -= n;
     }
 
+  pause("34 ");
---8<---
"""

with pause defined in sha512crypt.c as:
"""
static int
pause (char *arg)
{
  printf("%s\n", arg);

  /* If ESC is returned, then abort this entry.  */
  if (ASCII_CHAR (getkey ()) == 27)
    return 1;

  return 0;
}
"""

and after roughly 100+ loop iterations it did not
blank the screen, and I gave up on pressing Enter :),
when having pause("27 " ... "31 ") uncommented.

  The problem is that apparently it crashes, or jumps
to some random address during the loop, with the video
mode corrupted, so, no output to iLO remote console...