Created attachment 1156502 [details] sosreport Description of problem: If we change to other page during the HE setup after vm created, the cockpit shows can not be connected in some pages like logs, storage, networking, subscriptions, accounts, diagnostic report, terminal. Version-Release number of selected component (if applicable): rhev-hypervisor7-ng-20160506.0.el7 imgbased-0.6-0.1.el7ev.noarch ovirt-hosted-engine-setup-1.3.6.1-1.el7ev.noarch ovirt-host-deploy-1.4.1-1.el7ev.noarch ovirt-hosted-engine-ha-1.3.5.5-1.el7ev.noarch cockpit-ovirt-0.5.1-0.0.ovirt40.el7ev.x86_64 How reproducible: 100% Steps to Reproduce: 1. Install rhev-hypervisor7-ng-3.6-20160506.0.el7 2. Start Hosted Engine setup and run after vm was created 3. Change to Dashboard page 4. Check logs, storage, networking, subscriptions, accounts, diagnostic report, terminal page Actual results: 1. It reports "Unable to connect". Expected results: 1. It should show the correct content. Additional info:
Bug tickets must have version flags set prior to targeting them to a release. Please ask maintainer to set the correct version flags and only then set the target milestone.
Flags? target milestone? acks?
Test version: rhev-hypervisor7-ng-4.0-20160527.0 cockpit-ovirt-dashboard-0.10.1-0.0.1.el7ev.noarch Test steps: 1. Install rhev-hypervisor7-ng-4.0-20160527.0 2. Enter cockpit 3. Enter Hosted Engine page to start setup and run after vm was created 4. Change to Dashboard page 5. Check logs, storage, networking, subscriptions, accounts, diagnostic report, terminal page Test result: 1. Still shows Unable to connect. So this issue is not fixed in cockpit-ovirt-dashboard-0.10.1-0.0.1.el7ev.noarch. Change the status to assigned.
Target release should be placed once a package build is known to fix a issue. Since this bug is not modified, the target version has been reset. Please use target milestone to plan a fix for a oVirt release.
(In reply to wanghui from comment #3) > Test version: > rhev-hypervisor7-ng-4.0-20160527.0 > cockpit-ovirt-dashboard-0.10.1-0.0.1.el7ev.noarch > > Test steps: > 1. Install rhev-hypervisor7-ng-4.0-20160527.0 > 2. Enter cockpit > 3. Enter Hosted Engine page to start setup and run after vm was created > 4. Change to Dashboard page > 5. Check logs, storage, networking, subscriptions, accounts, diagnostic > report, terminal page > > Test result: > 1. Still shows Unable to connect. > > So this issue is not fixed in > cockpit-ovirt-dashboard-0.10.1-0.0.1.el7ev.noarch. Change the status to > assigned. I changed this to ON_QA because I'm not able to reproduce this bug at all. These steps are also somewhat unclear to me. Do you mean that you enter the hosted engine setup page, deploy the engine, then enter the page again and start setup (after deployment)? Then switch to the oVirt dashboard page, or the cockpit dashboard page? Can you please get a screenshot of what cockpit looks like when you're unable to connect?
(In reply to Ryan Barry from comment #5) > (In reply to wanghui from comment #3) > > Test version: > > rhev-hypervisor7-ng-4.0-20160527.0 > > cockpit-ovirt-dashboard-0.10.1-0.0.1.el7ev.noarch > > > > Test steps: > > 1. Install rhev-hypervisor7-ng-4.0-20160527.0 > > 2. Enter cockpit > > 3. Enter Hosted Engine page to start setup and run after vm was created > > 4. Change to Dashboard page > > 5. Check logs, storage, networking, subscriptions, accounts, diagnostic > > report, terminal page > > > > Test result: > > 1. Still shows Unable to connect. > > > > So this issue is not fixed in > > cockpit-ovirt-dashboard-0.10.1-0.0.1.el7ev.noarch. Change the status to > > assigned. > > I changed this to ON_QA because I'm not able to reproduce this bug at all. > > These steps are also somewhat unclear to me. > > Do you mean that you enter the hosted engine setup page, deploy the engine, > then enter the page again and start setup (after deployment)? Actually the deploy engine is not finished. Just need to wait for VM starts and then switch to other page. > > Then switch to the oVirt dashboard page, or the cockpit dashboard page? Should switch to cockpit dashboard page and then switch to oVirt dashboard page. > > Can you please get a screenshot of what cockpit looks like when you're > unable to connect?
Created attachment 1165859 [details] can not connect page in rhev-hypervisor7-ng-4.0-20160607.1
Hui, can you provide the sosreport from the host? I wonder if iptables might have closed 9090. What appliance were you using for setup?
(In reply to wanghui from comment #6) > Actually the deploy engine is not finished. Just need to wait for VM starts > and then switch to other page. > > Should switch to cockpit dashboard page and then switch to oVirt dashboard > page. These are the steps I took, but I didn't wait for the VM to start. I'll try that today. > Hui, can you provide the sosreport from the host? > I wonder if iptables might have closed 9090. It definitely looks this way, though I'd hope cockpit is still related/established. I'm guessing this is the time of the close, though: May 12 16:02:29 cshao.redhat.com cockpit-ws[12614]: WebSocket from 10.66.65.8 for root closed May 12 16:02:44 cshao.redhat.com cockpit-ws[12614]: root: timed out May 12 16:02:44 cshao.redhat.com cockpit-session[12620]: pam_unix(cockpit:session): session closed for user root Chain INPUT (policy ACCEPT 0 packets, 0 bytes) pkts bytes target prot opt in out source destination 4107 2080764 ACCEPT all -- lo * 0.0.0.0/0 0.0.0.0/0 0 0 ACCEPT icmp -- * * 0.0.0.0/0 0.0.0.0/0 icmptype 255 12495 57380948 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED 2 120 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:22 0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:5900 0 0 ACCEPT udp -- * * 0.0.0.0/0 0.0.0.0/0 state NEW udp dpt:5900 0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:5901 0 0 ACCEPT udp -- * * 0.0.0.0/0 0.0.0.0/0 state NEW udp dpt:5901 332 49192 REJECT all -- * * 0.0.0.0/0 0.0.0.0/0 reject-with icmp-host-proh(In reply to Fabian Deutsch from comment #8)
Port 9090 is closed. This sounds like you are using a 3.6 Engine. Hui, please let me know what version of Engine you are using.
Ping?
(In reply to Fabian Deutsch from comment #10) > Port 9090 is closed. > > This sounds like you are using a 3.6 Engine. > > Hui, please let me know what version of Engine you are using. Fabian, according to my bug, I report this issue in rhev-hypervisor7-ng-3.6-20160506.0.el7. So yes, I am using engine3.6. But I also try this issue with engine4.0 as screenshot shown. And I think the 9090 port is not closed because some pages can be accessed instead of all the pages.
It can be that some pages can be accessed, because they were accessed before. Can you try the following flow in a clean environment: 1. Use RHEV-M 4.0 2. Install NGN 4.0 3. Enter Cockpit of NGN 4. Leave cockpit again / Close browser window 5. Add NGN to Engine 6. Enter Cockpit After 6 you should be able to switch pages in cockpit
(In reply to Fabian Deutsch from comment #13) > It can be that some pages can be accessed, because they were accessed before. > > Can you try the following flow in a clean environment: > > 1. Use RHEV-M 4.0 > 2. Install NGN 4.0 > 3. Enter Cockpit of NGN > 4. Leave cockpit again / Close browser window > 5. Add NGN to Engine > 6. Enter Cockpit > > After 6 you should be able to switch pages in cockpit Yes, I can.
Moving this bug to ON_QA according to commetn 14. I do not see any further gap.
Hi Fabian, The test scenario in comment#13 is not the same as my original issue. Do you mean this issue can be verified with comment#13?
Checking original bug description, we can not verify this bug according to comment 13. And this bug is serious to cockpit UI + HE + NGN. The exact steps: 1. Installed RHEV-H-NG. 2. Access cockpit by browser. 3. Navigate to Hosted Engine UI. 4. Start Hosted-engine deployment. 5. During HE deploying, you need to monitor the UI, _AFTER_ vm is created, see VM-created.png 6. Navigate to other UI. e.g: dashboard - logs 7. Again navigate to other UI. e.g: oVirt - Hosted-Engine. 8. Re-access the cockpit by browser Results: step 6, step 7 and step 8, "Unable to connect" is displayed in UI, this issue cause the whole cockpit UI can not be accessed. Additional checking: In short, we did all, but it is always failed to access cockpit, we have to _reinstall_ NGN. More actions: # ps -aux ^^ you can see the attachment # systemctl status cockpit ● cockpit.service - Cockpit Web Service Loaded: loaded (/usr/lib/systemd/system/cockpit.service; static; vendor preset: disabled) Active: inactive (dead) since Thu 2016-06-23 16:21:08 CST; 21min ago Docs: man:cockpit-ws(8) Main PID: 19104 (code=exited, status=0/SUCCESS) Jun 23 14:07:18 dhcp-11-17.nay.redhat.com cockpit-session[19116]: pam_succeed_if(cockpit:auth): requirement "uid >= 1000" not met by user "root" Jun 23 14:07:27 dhcp-11-17.nay.redhat.com cockpit-session[19119]: pam_ssh_add: Failed adding some keys Jun 23 14:07:27 dhcp-11-17.nay.redhat.com cockpit-ws[19104]: logged in user: root Jun 23 14:07:29 dhcp-11-17.nay.redhat.com cockpit-ws[19104]: New connection from 10.72.7.15 for root Jun 23 15:38:31 dhcp-11-17.nay.redhat.com cockpit-ws[19104]: New connection from 10.72.7.15 for root Jun 23 15:41:33 dhcp-11-17.nay.redhat.com cockpit-ws[19104]: WebSocket from 10.72.7.15 for root closed Jun 23 15:42:19 dhcp-11-17.nay.redhat.com cockpit-ws[19104]: New connection from 10.72.7.15 for root Jun 23 15:44:12 dhcp-11-17.nay.redhat.com cockpit-ws[19104]: WebSocket from 10.72.7.15 for root closed Jun 23 16:19:38 localhost.localdomain cockpit-ws[19104]: Logging out user root from 10.72.7.15 Jun 23 16:19:38 localhost.localdomain cockpit-ws[19104]: WebSocket from 10.72.7.15 for root closed # systemctl restart cockpit # systemctl status cockpit ● cockpit.service - Cockpit Web Service Loaded: loaded (/usr/lib/systemd/system/cockpit.service; static; vendor preset: disabled) Active: active (running) since Thu 2016-06-23 16:43:43 CST; 1s ago Docs: man:cockpit-ws(8) Process: 23282 ExecStartPre=/usr/sbin/remotectl certificate --ensure --user=root --group=cockpit-ws --selinux-type=etc_t (code=exited, status=0/SUCCESS) Main PID: 23287 (cockpit-ws) CGroup: /system.slice/cockpit.service └─23287 /usr/libexec/cockpit-ws Jun 23 16:43:43 localhost.localdomain systemd[1]: Starting Cockpit Web Service... Jun 23 16:43:43 localhost.localdomain systemd[1]: Started Cockpit Web Service. Jun 23 16:43:43 localhost.localdomain cockpit-ws[23287]: Using certificate: /etc/cockpit/ws-certs.d/0-self-signed.cert ^^^ After doing above steps to restart cockpit service, it does not help, cockpit still can not be accessed. Then, I reboot the RHEV-H-NG, after the OS is booted, it does not help, cockpit still can not be access.
Created attachment 1171375 [details] ps -aux
Created attachment 1171377 [details] VM-created.png
Created attachment 1171379 [details] sosreport
Created attachment 1171380 [details] var_log
I will send my env. info to you by email, and I will keep the env. one working day for you debug if needed.
After the HE deployment port 9090 is not open anymore: [root@dhcp-11-17 ~]# iptables -L -n -x -v Chain INPUT (policy ACCEPT 0 packets, 0 bytes) pkts bytes target prot opt in out source destination 64 6492 ACCEPT all -- lo * 0.0.0.0/0 0.0.0.0/0 0 0 ACCEPT icmp -- * * 0.0.0.0/0 0.0.0.0/0 icmptype 255 3679 263133 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED 2 120 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:22 0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:5900 0 0 ACCEPT udp -- * * 0.0.0.0/0 0.0.0.0/0 state NEW udp dpt:5900 0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:5901 0 0 ACCEPT udp -- * * 0.0.0.0/0 0.0.0.0/0 state NEW udp dpt:5901 3224 400726 REJECT all -- * * 0.0.0.0/0 0.0.0.0/0 reject-with icmp-host-prohibited Chain FORWARD (policy ACCEPT 0 packets, 0 bytes) pkts bytes target prot opt in out source destination Chain OUTPUT (policy ACCEPT 1505 packets, 6754093 bytes) pkts bytes target prot opt in out source destination [root@dhcp-11-17 ~]# iptables -L -n -x -v | grep 9090
Simone, do you know what component is taking care of the iptables in the HE host case? We had bug 1314781 to take care that the cockpit port is opened when a host is getting added from Engine.
Raising priority because it will block remote access to the host after HE deployment.
The appliance image used for testing was rhevm-appliance-20160619.0-2.x86_64.rhevm.ova (4.0)
Moving to Integration - setting iptables on HE.
Not sure what's the exact question here. Tried a bit to understand what happens with hosted-engine in node-ng and so far failed... 1. hosted-engine --deploy asks the user whether to configure iptables. 2. If the answer is 'yes', the code in hosted-engine-setup configures iptables, opening a few ports and closing the rest. For an example about how to add rules there from the answer file, see bug 1288979 comment 8. 3. At a later point, when the host is added to the engine, the engine runs host-deploy on it, and if answer above was 'yes', also configures iptables on the host. I understand this part was handled by bug 1314781.
Ryan, please tell me if anything else is needed/missing.
As often iterated, we'd like to avoid setting anything in answerfiles if possible. It's something that can be done, but it isn't ideal, especially since installing through cockpit is a supported primary flow. hosted-engine-setup should "just work" with this use case, which is the purpose of this bug. To that end, what about adding port 9090 to @CUSTOM_RULES@ for hosted-engine-setup? Similar to the gluster bug, engine already supports this case, so hosted-engine-setup should as well.
Ryan, we need to move this bug to correct component, not Node component. And consider it is beta blocker as QE point of view.
Works for me on these components: sanlock-3.2.4-2.el7_2.x86_64 ovirt-hosted-engine-ha-2.0.1-1.el7ev.noarch ovirt-imageio-daemon-0.3.0-0.el7ev.noarch ovirt-host-deploy-1.5.1-1.el7ev.noarch ovirt-engine-sdk-python-3.6.7.0-1.el7ev.noarch qemu-kvm-rhev-2.3.0-31.el7_2.16.x86_64 mom-0.5.5-1.el7ev.noarch ovirt-setup-lib-1.0.2-1.el7ev.noarch ovirt-vmconsole-host-1.0.4-1.el7ev.noarch libvirt-client-1.2.17-13.el7_2.5.x86_64 vdsm-4.18.6-1.el7ev.x86_64 ovirt-hosted-engine-setup-2.0.1-1.el7ev.noarch ovirt-imageio-common-0.3.0-0.el7ev.noarch ovirt-vmconsole-1.0.4-1.el7ev.noarch Linux version 3.10.0-327.22.2.el7.x86_64 (mockbuild.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-4) (GCC) ) #1 SMP Thu Jun 9 10:09:10 EDT 2016 Linux 3.10.0-327.22.2.el7.x86_64 #1 SMP Thu Jun 9 10:09:10 EDT 2016 x86_64 x86_64 x86_64 GNU/Linux Red Hat Enterprise Linux release 7.2. Tested from Chrome Version 50.0.2661.86 (64-bit)