As far as I understand the problem is that a period of time after JBoss EAP reports "server started" in the log, the CLI interface is still not started.

(In reply to Aleksandar Kostadinov from comment #1)
> As far as I understand the problem is that a period of time after JBoss EAP
> reports "server started" in the log, the CLI interface is still not started.

Please provide some details behind this statement.

There is obviously a question if the printing of the "Server Started" message does indeed indicate with certainty that all subsystems are indeed fully functional/listening due to the multithreaded boot sequence and this needs more investigation, but I find it hard to believe we cannot introduce a small delay in some script somewhere to workaround the issue until then.

E.g. I'm looking at init.d/jboss-as-standalone.sh

 count=0
  launched=false

  until [ $count -gt $STARTUP_WAIT ]
  do
    grep 'JBAS015874:' $JBOSS_CONSOLE_LOG > /dev/null 
    if [ $? -eq 0 ] ; then
      launched=true
      ==> INTRODUCE sleep 1 HERE <==
      break
    fi 
    sleep 1
    let count=$count+1;
  done

A workaround sounds fine for now.

The "Server Started" message does not get written until all MSC services registered during the boot are started. It's possible there's a bug there, but unlikely, and I'd like to see a reproducer less complex than starting an EC2 instance.

Hynek, please provide some more information on how to reproduce this. In particular, I assume you're launching an instance using an EAP 6.0.1 GA AMI (since we haven't released new AMIs since then) and upgrading to EAP 6.2.0 after the instance has started. At what point do you see the error, i.e., after launching the instance the first time with EAP 6.0.1 or after restarting following the upgrade to EAP 6.2.0?

Is this issue observed with a standalone server, a domain host, or both?

(In reply to Brian Stansberry from comment #14)
> Is this issue observed with a standalone server, a domain host, or both?

Observed with standalone server, not observed with neither domain controller nor host controller.

I think I've found the underlying issue here. When "service jbossas start" is run, the jbossas RPM init script invokes standalone.sh and redirects the output to $JBOSS_CONSOLE_LOG. Note that this standalone.sh command is run in the *background*. Next, the jbossas RPM init script looks for the "Server Started" message in $JBOSS_CONSOLE_LOG. The problem with this is that if $JBOSS_CONSOLE_LOG already existed prior to running "service jbossas start", it's possible that at the point at which the init script looks for the "Server Started" message, $JBOSS_CONSOLE_LOG hasn't yet been overwritten with the new output. This means that the init script ends up finding a previous "Server Started" message in $JBOSS_CONSOLE_LOG and the "service jbossas start" command ends up erroneously returning before the server has actually finished starting up. 

I verified that this is indeed what's happening by reproducing the issue on EC2 using Hynek's updated AMI and inspecting the "Server Started" message that's actually found in this case. It was "JBoss EAP 6.0.1.GA (AS 7.1.3.Final-redhat-4) started..." instead of "JBoss EAP 6.2.0.Beta1 (AS 7.3.0.Final-redhat-8) started...", which means that after restarting the EC2 instance following the upgrade to EAP 6.2.0, at the point at which the init script looked for the message, $JBOSS_CONSOLE_LOG hadn't yet been overwritten with the new output. 

Since this underlying issue is a timing issue, it doesn't seem specific to EC2. To fix this, I think the jbossas RPM init script should truncate $JBOSS_CONSOLE_LOG to an empty file if it already exists before starting the server. Here's my proposed change:

--- a/jbossas.init
+++ b/jbossas.init
@@ -240,8 +240,8 @@ start() {
     fi
 
     if [ -n "$JBOSS_CONSOLE_LOG" -a ! -d "$JBOSS_CONSOLE_LOG" ]; then
-      # ensure the file exists
-      touch $JBOSS_CONSOLE_LOG
+      # ensure the file exists and is empty
+      truncate -s 0 $JBOSS_CONSOLE_LOG
       if [ "$JBOSS_USER" != "RUNASIS" ]; then
           chown $JBOSS_USER:$JBOSS_GROUP $JBOSS_CONSOLE_LOG
       fi

Permaine, if this change looks good to you, could we include it in the jbossas and jbossas-domain RPM init scripts for ER7?

With this change, I haven't been able to reproduce the issue. Hynek, if you'd like to try it out as well, you can add the following line to your $USER_SCRIPT in your EC2 user data for now:
sed -i 's/touch $JBOSS_CONSOLE_LOG/truncate -s 0 $JBOSS_CONSOLE_LOG/' /etc/init.d/jbossas

The change looks fine, I'll commit this for ER7.
Thanks for the patch, Farah!

Great catch Farah, well done!

I'm catching up late but didn't we change the way init script verifies server is started in bug 988864? Is the above change still relevant?

Truncating logs is not acceptable for customers who require preserving logs for auditing purposes.

Any options e.g. for backing up the file so that EAP would be started with a fresh file but the old log records would still be accessible?

(In reply to Hynek Mlnarik from comment #22)
> Truncating logs is not acceptable for customers who require preserving logs
> for auditing purposes.

Hynek, $JBOSS_CONSOLE_LOG already gets overwritten each time the "service {jbossas|jbossas-domain} start" command is run, i.e., customers do not have access to older versions of $JBOSS_CONSOLE_LOG. The only difference now is that we're making sure this file is empty just before overwriting it (to avoid the case where the init script checks for the "Server Started" message in this file before it has been overwritten).

We change the way to search for the server start message in the server.log for BZ#988864. 
As Farah mentioned, console.log always starts new and we shouldn't change the behaviour in the middle of a release, there maybe customers who expect this behaviour. I've added code to save existing console.log as console.log-timestamp so that it will solve this case and keep old logs around.

(In reply to Aleksandar Kostadinov from comment #21)
> I'm catching up late but didn't we change the way init script verifies
> server is started in bug 988864? Is the above change still relevant?

It looks like bug 988864 addresses an issue related to the full-ha profile where the "Server Started" message gets output to the server.log file but not to $JBOSS_CONSOLE_LOG. The fix there was to additionally check the server.log file for the "Server Started" message if it isn't found in $JBOSS_CONSOLE_LOG. Since the init script still checks $JBOSS_CONSOLE_LOG first, we'll still need to truncate this file to an empty file to make sure the script isn't searching for the message before this file has been overwritten.

Moving this to MODIFIED since Permaine has committed the changes to the jbossas and jbossas-domain RPM init scripts for ER7.

Hynek, please file another issue for that. I think the explained fixes don't represent a regression in this regard. And changing how CONSOLE log is handled needs a separate consideration. For example we can't keep CONSOLE log grow indefinitely without a rotation capability.
Moreover, whoever needs such log retention, will most likely use the full-ha production profile, which doesn't log to console at all.

That's my opinion at least.

Fixed in 6.2.0.ER7 (RPM installation).