I have access to the machine and rails console seems to be behaving well (detecting the region and accessing the database well) I'm digging into the errors
Unfortunately that appliance is working for me. I was able to run the status, I was able to run rails console, and able to startup the server I was also able to properly detect the region One error from the logs that I read here is: [----] E, [2019-04-17T15:09:20.503744 #1024:e08f58] ERROR -- : [MiqPassword::MiqPasswordError]: can not decrypt v2_key encrypted string Method:[block (2 levels) in <class:LogProxy>] If that log is present in the customer's site, then that would mean that the v2_key on disk is corrupted. Copying that file from a working appliance to the non-working appliance should get it running again. I would backup the current config/v2_key to config/v2_key.old - so you can diff the files and ensure that it was indeed the cause. full path for the file: /var/www/miq/vmdb/config/v2_key This would also cause a problem detecting the region (since it would not be able to connect to the database) But to be honest, if the filesystem is corrupted, then scrapping and rebuilding the broken worker appliance seems like a better approach. the v2_key file is not changed after initial configuration, so it seems at low odds at getting corrupted. I'll look back at the logs and look for other issues, including the missing user error, but I'm kinda stuck since this appliance doesn't look like a reproducer.
case was closed. please let me know if we need to look further into this