Bug 1223671

build sequence:
- otopi 1.4 (atm from git master) (rpms: otopi, otopi-java, otopi-devtools)
- ovirt-host-deploy from git master + patch 38091
  it may happen that ovirt-host-deploy rebuilt from rpms has lower priority than
  snapshots from maven, so ovirt-engine with patch 38095 won't build.
  workaround: overwrite jar and pom in local maven repo.
- engine from git master + patches 38092-38095 + 35887 + 35906
  patch 40832 is important from end result and must be merged but it is not 
  critical for testing
separately and independently:
- vdsm from git master + patc 40704
separately and independently:
- ovirt-vmconsole package + deps

pending issues, patch not yet posted (where applicable):
- wiki update (template, troubleshooting) - no need for patch
- selinux configuration for engine side, ovirt-vmconsole-proxy helper
- cert upgrade

(In reply to Francesco Romani from comment #2)
> pending issues, patch not yet posted (where applicable):
> - wiki update (template, troubleshooting) - no need for patch
> - selinux configuration for engine side, ovirt-vmconsole-proxy helper
> - cert upgrade

- implement key filtering engine side

ovirt-3.6.0-3 release

ovirt-3.6.0-3 release

Created attachment 1057013 [details]
logs from host and engine's VM

Returning to assigned as not worked for me.

I can see

failed 2015-07-28 09:20:27 ERROR otopi.context context._executeMethod:164 Failed to execute stage 'Setup validation': Failed to clear zombie commands. Please access support in attempt to resolve the problem

which is bug 1243125. Moving back as this is unrelated to the feature

Created attachment 1061633 [details]
logs from engine

Sorry for wrong usage of needinfo.

Nikolai, can you please try again using clean installation and see if it is helps?

(In reply to Francesco Romani from comment #15)
> Sorry for wrong usage of needinfo.
> 
> Nikolai, can you please try again using clean installation and see if it is
> helps?

the problem started at the problem report, rfe bugs are not for these discussions. if anyone of internal team has an issue, he can contact people for help to determine the proper method file it if any.

private comments are banned in any case.

Please open a bug on this feature and block the feature, but don't fail it. It needs to be tracked separately.

Added https://bugzilla.redhat.com/show_bug.cgi?id=1256214 to provide required information.

Due to bug 1256214 I can't proceed with vm-console-proxy to be enabled over the engine VM.

(In reply to Nikolai Sednev from comment #19)
> Due to bug 1256214 I can't proceed with vm-console-proxy to be enabled over
> the engine VM.

again, not the place for status/progress reports.

also, this bug happens only when performing upgrade, so the above statement is unclear and somewhat incorrect.

(In reply to Alon Bar-Lev from comment #20)
> (In reply to Nikolai Sednev from comment #19)
> > Due to bug 1256214 I can't proceed with vm-console-proxy to be enabled over
> > the engine VM.
> 
> again, not the place for status/progress reports.
> 
> also, this bug happens only when performing upgrade, so the above statement
> is unclear and somewhat incorrect.

Incorrect, it wasn't an upgrade.
I've tried to add the functionality of serial console to the engine and failed because we have a bug on second run of the engine setup.

I also tried to run the setup on clean environment first time and also failed, while serial-console-proxy package already installed on engine, but the engine-setup fails.

running setup 2nd time == upgrade.

"clean environment" != an environment in which there are left overs from previous setup.

please adjust your terms.

until this is resolved please progress using vanilla rhel while running setup once with all required features.

Looks more or less ok, top works, vi works, even able to play snakes[1] via serial.

[1] http://bruxy.regnet.cz/web/linux/EN/housenka-bash-game/

I've tried to connect to serial console of the el7.2VM running on top of the HE and failed.

Environment details:
HE-Host:
mom-0.5.1-1.el7ev.noarch
ovirt-vmconsole-1.0.0-1.el7ev.noarch
ovirt-hosted-engine-ha-1.3.3.6-1.el7ev.noarch
qemu-kvm-rhev-2.3.0-31.el7_2.5.x86_64
ovirt-vmconsole-host-1.0.0-1.el7ev.noarch
ovirt-host-deploy-1.4.1-1.el7ev.noarch
libvirt-client-1.2.17-13.el7_2.2.x86_64
sanlock-3.2.4-2.el7_2.x86_64
ovirt-setup-lib-1.0.1-1.el7ev.noarch
vdsm-4.17.15-0.el7ev.noarch
ovirt-hosted-engine-setup-1.3.2.1-1.el7ev.noarch

Engine:
ovirt-vmconsole-1.0.0-1.el6ev.noarch
rhevm-dwh-3.6.2-1.el6ev.noarch
rhevm-reports-3.6.2.1-1.el6ev.noarch
ovirt-engine-extension-aaa-jdbc-1.0.4-1.el6ev.noarch
ovirt-setup-lib-1.0.1-1.el6ev.noarch
ovirt-vmconsole-proxy-1.0.0-1.el6ev.noarch
ovirt-host-deploy-1.4.1-1.el6ev.noarch
ovirt-host-deploy-java-1.4.1-1.el6ev.noarch

I also checked the services on host and the engine, they were up and running:
Host:
systemctl is-enabled ovirt-vmconsole-host-sshd.service
enabled
systemctl status ovirt-vmconsole-host-sshd
● ovirt-vmconsole-host-sshd.service - oVirt VM Console SSH server daemon
   Loaded: loaded (/usr/lib/systemd/system/ovirt-vmconsole-host-sshd.service; enabled; vendor preset: disabled)
   Active: active (running) since Wed 2016-01-13 17:44:30 IST; 17h ago
 Main PID: 76722 (sshd)
   CGroup: /system.slice/ovirt-vmconsole-host-sshd.service
           └─76722 /usr/sbin/sshd -f /usr/share/ovirt-vmconsole/ovirt-vmconsole-host/ovirt-vmconsole-host-sshd/sshd_config -D

Jan 13 17:44:30 alma04.qa.lab.tlv.redhat.com systemd[1]: Started oVirt VM Console SSH server daemon.
Jan 13 17:44:30 alma04.qa.lab.tlv.redhat.com systemd[1]: Starting oVirt VM Console SSH server daemon...
Jan 13 17:44:31 alma04.qa.lab.tlv.redhat.com sshd[76722]: Server listening on 0.0.0.0 port 2223.
Jan 13 17:44:31 alma04.qa.lab.tlv.redhat.com sshd[76722]: Server listening on :: port 2223.

On engine:
service ovirt-vmconsole-proxy-sshd status
ovirt-vmconsole-proxy-sshd (pid  5493) is running...

On VM:
systemctl status  serial-getty
● serial-getty - Serial Getty on ttyS0
   Loaded: loaded (/usr/lib/systemd/system/serial-getty@.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2016-01-14 13:02:26 IST; 1h 56min left
     Docs: man:agetty(8)
           man:systemd-getty-generator(8)
           http://0pointer.de/blog/projects/serial-console.html
 Main PID: 682 (agetty)
   CGroup: /system.slice/system-serial\x2dgetty.slice/serial-getty
           └─682 /sbin/agetty --keep-baud 115200 38400 9600 ttyS0 vt220

Jan 14 13:02:26 RHEL7Server systemd[1]: Started Serial Getty on ttyS0.
Jan 14 13:02:26 RHEL7Server systemd[1]: Starting Serial Getty on ttyS0...
systemctl is-enabled  serial-getty
enabled

I've tried disabling the iptables on host and engine, but it did not helped.

Printing out the output from my laptop, while trying to connect over serial console to guest-vm:

ssh -v -t -i $HOME/.ssh/id_rsa -p 2222 ovirt-vmconsole@FQDNofMyEngine connect
OpenSSH_6.6.1, OpenSSL 1.0.1e-fips 11 Feb 2013
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 56: Applying options for *
debug1: Connecting to nsednev-he-1.qa.lab.tlv.redhat.com [10.35.97.61] port 2222.
debug1: Connection established.
debug1: permanently_set_uid: 0/0
debug1: identity file /root/.ssh/id_rsa type 1
debug1: identity file /root/.ssh/id_rsa-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.6.1
debug1: Remote protocol version 2.0, remote software version OpenSSH_5.3
debug1: match: OpenSSH_5.3 pat OpenSSH_5* compat 0x0c000000
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client aes128-ctr hmac-md5 none
debug1: kex: client->server aes128-ctr hmac-md5 none
debug1: kex: diffie-hellman-group-exchange-sha256 need=16 dh_need=16
debug1: kex: diffie-hellman-group-exchange-sha256 need=16 dh_need=16
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<3072<8192) sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
debug1: ssh_rsa_verify: signature correct
debug1: Server host key: RSA-CERT ad:86:e1:e2:bb:e5:4f:1e:63:01:52:64:73:f3:b3:a1
debug1: checking without port identifier
debug1: No matching CA found. Retry with plain key
debug1: No matching CA found. Retry with plain key
debug1: Host '[nsednev-he-1.qa.lab.tlv.redhat.com]:2222' is known and matches the RSA host key.
debug1: Found key in /root/.ssh/known_hosts:4
debug1: ssh_rsa_verify: signature correct
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: Roaming not allowed by server
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /root/.ssh/id_rsa
debug1: Remote: Forced command: exec "/usr/libexec/ovirt-vmconsole-proxy-shell"  accept --entityid="00000019-0019-0019-0019-0000000001cc" --entity="admin_internal-authz"
debug1: Remote: Agent forwarding disabled.
debug1: Remote: Port forwarding disabled.
debug1: Remote: User rc file execution disabled.
debug1: Remote: X11 forwarding disabled.
debug1: Server accepts key: pkalg ssh-rsa blen 279
debug1: key_parse_private2: missing begin marker
debug1: read PEM private key done: type RSA
debug1: Remote: Forced command: exec "/usr/libexec/ovirt-vmconsole-proxy-shell"  accept --entityid="00000019-0019-0019-0019-0000000001cc" --entity="admin_internal-authz"
debug1: Remote: Agent forwarding disabled.
debug1: Remote: Port forwarding disabled.
debug1: Remote: User rc file execution disabled.
debug1: Remote: X11 forwarding disabled.
debug1: Authentication succeeded (publickey).
Authenticated to nsednev-he-1.qa.lab.tlv.redhat.com ([10.35.97.61]:2222).
debug1: channel 0: new [client-session]
debug1: Requesting no-more-sessions
debug1: Entering interactive session.
debug1: Sending environment.
debug1: Sending env LANG = en_US.UTF-8
debug1: Sending env LANGUAGE = 
debug1: Sending command: connect
ERROR: No available running VMs
debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
debug1: client_input_channel_req: channel 0 rtype eow reply 0
debug1: channel 0: free: client-session, nchannels 1
Connection to nsednev-he-1.qa.lab.tlv.redhat.com closed.
Transferred: sent 4008, received 4144 bytes, in 0.8 seconds
Bytes per second: sent 5293.5, received 5473.1
debug1: Exit status 1

I've also tried disabling SElinux, it did not helped.

Created attachment 1114740 [details]
sosreport from the engine

Created attachment 1114741 [details]
sosreport from VM

Sosreport from the host: https://drive.google.com/a/redhat.com/file/d/0B85BEaDBcF88WFBkbG1lazhxNHc/view?usp=sharing

(In reply to Nikolai Sednev from comment #25)

what were the permissions on that VM?

(In reply to Michal Skrivanek from comment #30)
> (In reply to Nikolai Sednev from comment #25)
> 
> what were the permissions on that VM?

My bad, forgot to configure the Permissions for the VM, after adding admin as UserVmManager, serial connectivity was successfully established.
Working just fine now.

how can this still be on QA and planned for 3.6rc ? 3.6.0 is already out..

is this in 3.6.0? if it's not released yet, will it be in 3.6.2?

Thanks!

(In reply to Sven Kieske from comment #32)
> how can this still be on QA and planned for 3.6rc ? 3.6.0 is already out..
> 
> is this in 3.6.0? if it's not released yet, will it be in 3.6.2?

The basic feature is released since 3.6.0, is tested and works.
The thing is (minor) fixes, updates and RFEs both minor and major are attached to this BZ, so the state may get confusing.

Sure thing, bug #1306318 was opened accordingly.

Works for me on PPC:
Engine:
3.6.3-0.1.el6
rhevm-3.6.3-0.1.el6.noarch
ovirt-setup-lib-1.0.1-1.el6ev.noarch
ovirt-engine-extension-aaa-jdbc-1.0.5-1.el6ev.noarch
ovirt-engine-extension-aaa-ldap-1.1.1-1.el6ev.noarch
ovirt-host-deploy-1.4.1-1.el6ev.noarch
ovirt-vmconsole-1.0.0-1.el6ev.noarch
ovirt-engine-extension-aaa-misc-1.0.0-2.el6ev.noarch
ovirt-vmconsole-proxy-1.0.0-1.el6ev.noarch
ovirt-host-deploy-java-1.4.1-1.el6ev.noarch
ovirt-engine-extension-aaa-ldap-setup-1.1.1-1.el6ev.noarch
Linux version 2.6.32-573.8.1.el6.x86_64 (mockbuild.eng.bos.redhat.com) (gcc version 4.4.7 20120313 (Red Hat 4.4.7-16) (GCC) ) #1 SMP Fri Sep 25 19:24:22 EDT 2015

Host:
mom-0.5.2-1.el7ev.noarch
qemu-kvm-rhev-2.3.0-31.el7_2.7.ppc64le
libvirt-client-1.2.17-13.el7_2.3.ppc64le
sanlock-3.2.4-2.el7_2.ppc64le                                                   
vdsm-4.17.19-0.el7ev.noarch
ovirt-vmconsole-1.0.0-1.el7ev.noarch
ovirt-vmconsole-host-1.0.0-1.el7ev.noarch
Linux version 3.10.0-327.10.1.el7.ppc64le (mockbuild.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-4) (GCC) ) #1 SMP Sat Jan 23 04:55:20 EST 2016