Bug 990274

btw I'm not sure if this would qualify as a bug but I don't see domain server being able to start with default settings on a 32bit machine.

I have tried this upstream with the following setup. Server with small heap settings:
        <server name="server-one" group="main-server-group">
            <jvm name="default">
                <heap size="64m" max-size="256m"/>
                <permgen size="256m" max-size="256m"/>
                <jvm-options>
                    <option value="-server"/>
                </jvm-options>
            </jvm>
        </server>

It fails to start. Then in CLI change the minimum heap size:
[standalone@localhost:9990 /] /host=master/server-config=server-one:stop
{
    "outcome" => "success",
    "result" => "STOPPING"
}
[standalone@localhost:9990 /] /host=master/server-config=server-one/jvm=default:write-attribute(name=heap-size,value=64m)
{
    "outcome" => "success",
    "result" => undefined,
    "server-groups" => undefined
}
[standalone@localhost:9990 /] /host=master/server-config=server-one:start
{
    "outcome" => "success",
    "result" => "FAILED"
}
Then in CLI change the maximum heap size:
[standalone@localhost:9990 /] /host=master/server-config=server-one:stop
{
    "outcome" => "success",
    "result" => "STOPPING"
}
[standalone@localhost:9990 /] /host=master/server-config=server-one/jvm=default:write-attribute(name=max-heap-size,value=256m)
{
    "outcome" => "success",
    "result" => undefined,
    "server-groups" => undefined
}
[standalone@localhost:9990 /] /host=master/server-config=server-one:start
{
    "outcome" => "success",
    "result" => "STARTING"
}
The server gets started upstream, and I see the JVM options get recalculated in my debugger each time I call start(). I'll look at the current EAP 6.3 branch next.

                <heap size="64m" max-size="256m"/>
in my initial post should read
                <heap size="64k" max-size="256k"/>

I tried the same settings in EAP 6.3.0. From the CLI, the same sequence of commands work, however from the console I see the behaviour mentioned below.

The initial error on startup is:
18:34:57,228 INFO  [org.jboss.as.process.Server:server-one.status] (ProcessController-threads - 3) JBAS012017: Starting process 'Server:server-one'
[Server:server-one] Error occurred during initialization of VM
[Server:server-one] Too small initial heap
18:34:57,253 INFO  [org.jboss.as.process.Server:server-one.status] (reaper for Server:server-one) JBAS012010: Process 'Server:server-one' finished with an exit status of 1


I then change the 'Heap Size' for server-one in the Domain Server Configurations. 
Then I try to start the server under Runtime, and the logs report:
[Host Controller] 18:36:03,515 INFO  [org.jboss.as.host.controller] (HttpManagementService-threads - 2) JBAS010922: Starting server server-one
18:36:03,517 INFO  [org.jboss.as.process.Server:server-one.status] (ProcessController-threads - 5) JBAS012017: Starting process 'Server:server-one'
[Server:server-one] Error occurred during initialization of VM
[Server:server-one] Too small initial heap
18:36:03,536 INFO  [org.jboss.as.process.Server:server-one.status] (reaper for Server:server-one) JBAS012010: Process 'Server:server-one' finished with an exit status of 1
[Host Controller] 18:36:03,538 INFO  [org.jboss.as.host.controller] (ProcessControllerConnection-thread - 2) JBAS010926: Unregistering server server-one

I then change the 'Max Heap Size' in the Domain Server Configurations. Again I try to start the server under Runtime, and I get the same log as in my previous attempt.

At this stage, I see the correct values in the model. If I try to do the same to the server without stopping it first, e.g starting from scratch with max-heap-size=256k:
[domain@localhost:9999 /] /host=master/server-config=server-one/jvm=default:write-attribute(name=max-heap-size,value=256m)
{
    "outcome" => "success",
    "result" => undefined,
    "server-groups" => undefined
}
[domain@localhost:9999 /] /host=master/server-config=server-one:restart
{
    "outcome" => "failed",
    "failure-description" => "JBAS010946: Cannot restart server server-one as it is not currently started; it is FAILED",
    "rolled-back" => true
}
The restart problem appears to be in ServerRestartHandler, checking that the state is STARTED.

[domain@localhost:9999 /] /host=master/server-config=server-one:start
{
    "outcome" => "success",
    "result" => "FAILED"
}

The start problem gives the same log output as above, and appears to be something in the state machine of ManagedServer.

https://github.com/jbossas/jboss-eap/pull/1812

And what about restart operation?

# Start domain and run
/host=master/server-config=server-one/jvm=default:add
/host=master/server-config=server-one/jvm=default:write-attribute(name=heap-size,value=64k)
/host=master/server-config=server-one/jvm=default:write-attribute(name=max-heap-size,value=256k)
/host=master:reload
/host=master/server-config=server-one/jvm=default:write-attribute(name=heap-size,value=64m)
/host=master/server-config=server-one/jvm=default:write-attribute(name=max-heap-size,value=256m)

case 1) 
 /host=master/server-config=server-one:start 
 -> server-one is started
case 2)
 /host=master/server-config=server-one:stop
 /host=master/server-config=server-one:start
 -> server-one is started
case 3) 
 /host=master/server-config=server-one:restart
{
    "outcome" => "failed",
    "failure-description" => "JBAS010946: Cannot restart server server-one as it is not currently started; it is FAILED",
    "rolled-back" => true
}

Kabir Khan <kabir.khan> updated the status of jira WFCORE-165 to Resolved

I'll answer my own question :). According to the operation description server has to be in running state to be able to use restart operation thus behaviour I described in comment 7 is correct.

Verified on EAP 6.4.0.DR8.