Created attachment 1642294 [details] error Description of problem: Version-Release number of selected component (if applicable): Version 5.11.1.1.20191122174937_707df01 How reproducible: 100% Steps to Reproduce: 1.Enable Cockpit Server Role 2.Add Provider(I have added Vsphere) 3.Go to VM details page 4.Click on Web Console From Access button Actual results: Unable to access Cockpit web console Expected results: "Service Unavailable" error on page after clicking on web console button Additional info: Last worked version 5.10.13.1
Thanks, we're getting close. I compared what processes are running on 5.10 and 5.11 and it seems that the newer doesn't run the cockpit proxy at all. 5.10: $ ps ax | grep cockpit 28561 ? SNl 0:14 /usr/libexec/cockpit-ws --port 9002 --address 127.0.0.1 --no-tls 31789 pts/0 S+ 0:00 grep --color=auto cockpit 5.11: $ ps ax | grep cockpit 20907 pts/0 S+ 0:00 grep --color=auto cockpit I'll try to take a look into how the cockpit proxy is forked from the worker. If I can't find a solution, we have to push this to the cockpit people who actually wrote this.
$ less log/evm.log | grep cockpit | tail -n 500 [----] I, [2019-12-06T06:37:45.916621 #8749:2afdb0c7e5c4] INFO -- : MIQ(MiqCockpitWsWorker::Runner#start_cockpit_ws) MIQ(MiqCockpitWsWorker::Runner) Starting cockpit-ws Process [----] I, [2019-12-06T06:37:45.916762 #8749:2afdb0c7e5c4] INFO -- : MIQ(MiqCockpitWsWorker::Runner#cockpit_ws_run) MIQ(MiqCockpitWsWorker::Runner) cockpit-ws process starting [----] I, [2019-12-06T06:37:45.942379 #8749:2afdb0c7e5c4] INFO -- : MIQ(MiqCockpitWsWorker::Runner#cockpit_ws_run) MIQ(MiqCockpitWsWorker::Runner) cockpit-ws process started - pid= [----] I, [2019-12-06T06:37:45.942625 #8749:2afdb0c7e5c4] INFO -- : MIQ(MiqCockpitWsWorker::Runner#start_cockpit_ws) MIQ(MiqCockpitWsWorker::Runner) Started cockpit-ws Process [----] I, [2019-12-06T06:37:48.944115 #8749:2afdb0c7e5c4] INFO -- : MIQ(MiqCockpitWsWorker::Runner#check_cockpit_ws) MIQ(MiqCockpitWsWorker::Runner) Cockpit-ws Process gone. Restarting... [----] I, [2019-12-06T06:37:48.944356 #8749:2afdb0c7e5c4] INFO -- : MIQ(MiqCockpitWsWorker::Runner#start_cockpit_ws) MIQ(MiqCockpitWsWorker::Runner) Starting cockpit-ws Process [----] I, [2019-12-06T06:37:48.944430 #8749:2afdb0c7e5c4] INFO -- : MIQ(MiqCockpitWsWorker::Runner#cockpit_ws_run) MIQ(MiqCockpitWsWorker::Runner) cockpit-ws process starting [----] I, [2019-12-06T06:37:48.963887 #8749:2afdb0c7e5c4] INFO -- : MIQ(MiqCockpitWsWorker::Runner#cockpit_ws_run) MIQ(MiqCockpitWsWorker::Runner) cockpit-ws process started - pid= [----] I, [2019-12-06T06:37:48.964069 #8749:2afdb0c7e5c4] INFO -- : MIQ(MiqCockpitWsWorker::Runner#start_cockpit_ws) MIQ(MiqCockpitWsWorker::Runner) Started cockpit-ws Process [----] I, [2019-12-06T06:37:51.967412 #8749:2afdb0c7e5c4] INFO -- : MIQ(MiqCockpitWsWorker::Runner#check_cockpit_ws) MIQ(MiqCockpitWsWorker::Runner) Cockpit-ws Process gone. Restarting... [----] I, [2019-12-06T06:37:51.967556 #8749:2afdb0c7e5c4] INFO -- : MIQ(MiqCockpitWsWorker::Runner#start_cockpit_ws) MIQ(MiqCockpitWsWorker::Runner) Starting cockpit-ws Process [----] I, [2019-12-06T06:37:51.967637 #8749:2afdb0c7e5c4] INFO -- : MIQ(MiqCockpitWsWorker::Runner#cockpit_ws_run) MIQ(MiqCockpitWsWorker::Runner) cockpit-ws process starting [----] I, [2019-12-06T06:37:51.987721 #8749:2afdb0c7e5c4] INFO -- : MIQ(MiqCockpitWsWorker::Runner#cockpit_ws_run) MIQ(MiqCockpitWsWorker::Runner) cockpit-ws process started - pid= [----] I, [2019-12-06T06:37:51.987946 #8749:2afdb0c7e5c4] INFO -- : MIQ(MiqCockpitWsWorker::Runner#start_cockpit_ws) MIQ(MiqCockpitWsWorker::Runner) Started cockpit-ws Process [----] I, [2019-12-06T06:37:54.989402 #8749:2afdb0c7e5c4] INFO -- : MIQ(MiqCockpitWsWorker::Runner#check_cockpit_ws) MIQ(MiqCockpitWsWorker::Runner) Cockpit-ws Process gone. Restarting... [----] I, [2019-12-06T06:37:54.989615 #8749:2afdb0c7e5c4] INFO -- : MIQ(MiqCockpitWsWorker::Runner#start_cockpit_ws) MIQ(MiqCockpitWsWorker::Runner) Starting cockpit-ws Process [----] I, [2019-12-06T06:37:54.989691 #8749:2afdb0c7e5c4] INFO -- : MIQ(MiqCockpitWsWorker::Runner#cockpit_ws_run) MIQ(MiqCockpitWsWorker::Runner) cockpit-ws process starting [----] I, [2019-12-06T06:37:55.008282 #8749:2afdb0c7e5c4] INFO -- : MIQ(MiqCockpitWsWorker::Runner#cockpit_ws_run) MIQ(MiqCockpitWsWorker::Runner) cockpit-ws process started - pid= [----] I, [2019-12-06T06:37:55.008464 #8749:2afdb0c7e5c4] INFO -- : MIQ(MiqCockpitWsWorker::Runner#start_cockpit_ws) MIQ(MiqCockpitWsWorker::Runner) Started cockpit-ws Process Seems like the EVM tries to start cockpit up over and over, but without success.
As far as I was able to debug, the cockpit proxy provides the following message in its STDERR: ``` couldn't load configuration file: /var/www/miq/vmdb/config/cockpit/cockpit.conf: Permission denied ``` I tried to recreate the same command and it was not failing: $ XDG_CONFIG_DIRS=/var/www/miq/vmdb/config /usr/libexec/cockpit-ws --port 9002 --address 127.0.0.1 --no-tls Anyway, this is either an issue in the cockpit proxy or in the appliance itself, so passing it further.
It's very possible that this change was caused by the selinux changes I needed to make to get cockpit running on the appliance itself https://github.com/ManageIQ/manageiq-appliance/pull/239 Can you try running these commands on an appliance before enabling the cockpit role and see if that fixes the problem? /usr/sbin/semanage fcontext -a -t usr_t /usr/libexec/cockpit-ws /usr/sbin/semanage fcontext -a -t usr_t /usr/libexec/cockpit-ssh [ -x /sbin/restorecon ] && /sbin/restorecon -v /usr/libexec/cockpit-ws [ -x /sbin/restorecon ] && /sbin/restorecon -v /usr/libexec/cockpit-ssh
https://github.com/ManageIQ/manageiq-appliance/pull/266
https://github.com/ManageIQ/manageiq-ui-classic/pull/6513
https://github.com/ManageIQ/manageiq-ui-classic/pull/6522
New commit detected on ManageIQ/manageiq-ui-classic/master: https://github.com/ManageIQ/manageiq-ui-classic/commit/9fc6c637234cb14d21dc636ce19e094733d7c973 commit 9fc6c637234cb14d21dc636ce19e094733d7c973 Author: Nick Carboni <ncarboni> AuthorDate: Tue Dec 10 13:52:12 2019 -0500 Commit: Nick Carboni <ncarboni> CommitDate: Tue Dec 10 13:52:12 2019 -0500 Reimplement token generation for cockpit webconsole proxy generate_ui_api_token was removed in f161abde29c273761a77113ef14652a9935f0d10 but this reference was left. This behavior is still required for cockpit so the contents of the removed method are now baked into #cockpit_redirect https://bugzilla.redhat.com/show_bug.cgi?id=1779988 app/controllers/dashboard_controller.rb | 5 +- 1 file changed, 4 insertions(+), 1 deletion(-)
https://github.com/ManageIQ/manageiq/pull/19631
New commits detected on ManageIQ/manageiq/master: https://github.com/ManageIQ/manageiq/commit/53207cb271f9bf9c460f68433992ceca2f18d0b4 commit 53207cb271f9bf9c460f68433992ceca2f18d0b4 Author: Nick Carboni <ncarboni> AuthorDate: Wed Dec 11 14:46:32 2019 -0500 Commit: Nick Carboni <ncarboni> CommitDate: Wed Dec 11 14:46:32 2019 -0500 Move cockpit-auth-miq to manageiq-appliance repo This doesn't need to live here, it isn't tested, doesn't use our models and moving it removes a few AVC denials when running cockpit-ws https://bugzilla.redhat.com/show_bug.cgi?id=1779988 lib/miq_cockpit.rb | 2 +- spec/lib/miq_cockpit_spec.rb | 6 +- tools/cockpit/cockpit-auth-miq | 274 - 3 files changed, 2 insertions(+), 280 deletions(-) https://github.com/ManageIQ/manageiq/commit/63b44ed6d0a69eec3d9d8a31cd8f455cbdce1917 commit 63b44ed6d0a69eec3d9d8a31cd8f455cbdce1917 Author: Nick Carboni <ncarboni> AuthorDate: Wed Dec 11 14:54:52 2019 -0500 Commit: Nick Carboni <ncarboni> CommitDate: Wed Dec 11 14:54:52 2019 -0500 Ensure the bundler and BUNDLE_GEMFILE env vars are not sent to cockpit-ws This was causing the ruby authentication process to look for our Gemfile and Gemfile.lock which caused a bunch of SELinux denials https://bugzilla.redhat.com/show_bug.cgi?id=1779988 app/models/miq_cockpit_ws_worker/runner.rb | 4 +- 1 file changed, 3 insertions(+), 1 deletion(-) https://github.com/ManageIQ/manageiq/commit/cbeb464ce5e3b465f1ef35af95bdeb5f4d9be6bb commit cbeb464ce5e3b465f1ef35af95bdeb5f4d9be6bb Author: Nick Carboni <ncarboni> AuthorDate: Wed Dec 11 15:02:00 2019 -0500 Commit: Nick Carboni <ncarboni> CommitDate: Wed Dec 11 15:02:00 2019 -0500 Run restorecon on the cockpit directory after writing the config Since we are creating the files, they will have the same context as our process (httpd_sys_content_t), we want these files to have the context of the parent directory (etc_t) so that they can be read by cockpit-ws without SELinux denials https://bugzilla.redhat.com/show_bug.cgi?id=1779988 lib/miq_cockpit.rb | 1 + spec/lib/miq_cockpit_spec.rb | 2 + 2 files changed, 3 insertions(+)
New commit detected on ManageIQ/manageiq-appliance/master: https://github.com/ManageIQ/manageiq-appliance/commit/afe24fb3afb543a1895b201c95b2e8affca00c45 commit afe24fb3afb543a1895b201c95b2e8affca00c45 Author: Nick Carboni <ncarboni> AuthorDate: Tue Dec 10 14:46:29 2019 -0500 Commit: Nick Carboni <ncarboni> CommitDate: Tue Dec 10 14:46:29 2019 -0500 Add cockpit selinux policy This was created by the following process: - set selinux to permissive - enable the cockpit role - Request a cockpit webconsole to a VM - audit2allow -a -m cockpit_ws_miq > cockpit_ws_miq.te https://bugzilla.redhat.com/show_bug.cgi?id=1779988 manageiq-setup.sh | 31 + 1 file changed, 31 insertions(+)