Selinux policies should be ok for both 7.5 and 7.4.z so why can't you update to 2.8 or 2.9?

I verified the problem with 2.9.
It worked fine with my own build of 2.9, but I still see the problem with a version from brew.
It failed with the same issue in our system test:
    https://rhv-devops-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/rhv-system-tests/job/ds-ovirt-system-tests_manual/291/

which used the following 2.0 build:
    https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=655288

The error can be seen here:
https://rhv-devops-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/rhv-system-tests/job/ds-ovirt-system-tests_manual/291/artifact/exported-artifacts/test_logs/rhv-suite-4.2/post-098_ovirt_provider_ovn.py/lago-rhv-suite-4-2-engine/_var_log/ovirt-provider-ovn.log

The error seen is:
    BadRequestError: 'ascii' codec can't decode byte 0xe3 in position 87: ordinal not in range(128)

The openvswitch logs are here:
https://rhv-devops-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/rhv-system-tests/job/ds-ovirt-system-tests_manual/291/artifact/exported-artifacts/test_logs/rhv-suite-4.2/post-098_ovirt_provider_ovn.py/lago-rhv-suite-4-2-engine/_var_log/openvswitch/

The name used to fail the test: u'ネットワーク_1'

Could you please look at this?

Verified with 2.9.0-8.el7 and it works fine.
The automatic test failure was a result of a problem with package name provides in python-openvswitch (2.7 had python2-openvswitch and was not updated with 2.9 which has python-openvswitch).

Reopening due to Marcin reports that RHEV QA is still seeing the failures:

http://pastebin.test.redhat.com/569135

Reproduced outside of ovirt-provider-ovn.

Interestingly the error that points to the root cause is missing from ovirt-provider-ovn logs. May have something to do with logging handlers.

# ./repro.py
ERROR:ovsdbapp.backend.ovs_idl.transaction:Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 100, in run
    txn.results.put(txn.do_commit())
  File "/usr/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 92, in do_commit
    status = txn.commit_block()
  File "/usr/lib/python2.7/site-packages/ovs/db/idl.py", line 1402, in commit_block
    status = self.commit()
  File "/usr/lib/python2.7/site-packages/ovs/db/idl.py", line 1385, in commit
    if not self.idl._session.send(msg):
  File "/usr/lib/python2.7/site-packages/ovs/jsonrpc.py", line 539, in send
    return self.rpc.send(msg)
  File "/usr/lib/python2.7/site-packages/ovs/jsonrpc.py", line 244, in send
    self.run()
  File "/usr/lib/python2.7/site-packages/ovs/jsonrpc.py", line 203, in run
    retval = self.stream.send(self.output)
  File "/usr/lib/python2.7/site-packages/ovs/stream.py", line 808, in send
    return super(SSLStream, self).send(buf)
  File "/usr/lib/python2.7/site-packages/ovs/stream.py", line 391, in send
    buf = buf.encode('utf-8')
UnicodeDecodeError: 'ascii' codec can't decode byte 0xd7 in position 85: ordinal not in range(128)

Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner
    self.run()
  File "/usr/lib64/python2.7/threading.py", line 765, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 96, in run
    self.idl.run()
  File "/usr/lib/python2.7/site-packages/ovs/db/idl.py", line 179, in run
    assert not self.txn
AssertionError

ERROR:ovsdbapp.backend.ovs_idl.command:Error executing command
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 35, in execute
    txn.add(self)
  File "/usr/lib64/python2.7/contextlib.py", line 24, in __exit__
    self.gen.next()
  File "/usr/lib/python2.7/site-packages/ovsdbapp/api.py", line 94, in transaction
    self._nested_txn = None
  File "/usr/lib/python2.7/site-packages/ovsdbapp/api.py", line 54, in __exit__
    self.result = self.commit()
  File "/usr/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 62, in commit
    raise result.ex
UnicodeDecodeError: 'ascii' codec can't decode byte 0xd7 in position 85: ordinal not in range(128)
Traceback (most recent call last):
  File "./repro.py", line 40, in <module>
    main()
  File "./repro.py", line 36, in main
    add_switch(idl)
  File "./repro.py", line 30, in add_switch
    cmd.execute(check_error=True)
  File "/usr/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 35, in execute
    txn.add(self)
  File "/usr/lib64/python2.7/contextlib.py", line 24, in __exit__
    self.gen.next()
  File "/usr/lib/python2.7/site-packages/ovsdbapp/api.py", line 94, in transaction
    self._nested_txn = None
  File "/usr/lib/python2.7/site-packages/ovsdbapp/api.py", line 54, in __exit__
    self.result = self.commit()
  File "/usr/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 62, in commit
    raise result.ex
UnicodeDecodeError: 'ascii' codec can't decode byte 0xd7 in position 85: ordinal not in range(128)

Enviroment:
# cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.5 (Maipo)
# rpm -qf /usr/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/transaction.py
python-ovsdbapp-0.6.0-1.el7ev.noarch
# rpm -qf /usr/lib/python2.7/site-packages/ovs/stream.py
python-openvswitch-2.9.0-1.el7fdp.noarch

Created attachment 1417334 [details]
Reproducer for the bug.

Fix needed upstream.

Changes from:

commit 2254074e3067fd306c37ac80347ef75075b40a89
Author: Alin Balutoiu <abalutoiu>
Date:   Wed Aug 16 15:01:39 2017 +0000

    python: fix python3 encode/decode on Windows

are conflicting with an earlier patch:

commit d90ed7d65ba8cfbdada2286f5adb167ca26048c9
Author: Numan Siddique <nusiddiq>
Date:   Wed Oct 5 17:50:24 2016 +0530

    python: Add SSL support to the python ovs client library

and we end up attempting to encode a Unicode string twice.

Fix (with regression test) posted upstream:

https://mail.openvswitch.org/pipermail/ovs-dev/2018-April/346248.html

Fix is now backported to 2.9 branch upstream:

https://github.com/openvswitch/ovs/commit/cef8e1fa7a8f030ca221f8cac5893c686b9d0f00

Can you supply a scratch build ovs-2.9+this patch for us to verify?

(In reply to Dan Kenigsberg from comment #13)
> Can you supply a scratch build ovs-2.9+this patch for us to verify?

Yes, will do. This is what I should have done.

Jakub, did you intend to add a URL to the scratch build?

Yes, I'm on it. I apologize for the delay.

would you clear the needinfo? only when the needed info is provided? it helps me notice if action is needed on me.

(In reply to Dan Kenigsberg from comment #17)
> would you clear the needinfo? only when the needed info is provided? it
> helps me notice if action is needed on me.

Understood. Sorry for the confusion.

Here's a scratch build for RHEL-7.5:

https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=15904507

I'm not sure what RHEL version you would like to test with. BZ target is set for 7.7 but we don't have that release in Brew yet. Please let me know if you need another scratch build targeted for a different release.

Once I get a green light from RHV team, and manage to collect all needed flags (pm_ack, fast‑datapath‑rhel‑7 ack, qa_ack) I will backport it to FDP.

Correction, I believe the backport first has to go to Fast Datapath Next, not FDP.

(In reply to Jakub Sitnicki from comment #18) 
> Here's a scratch build for RHEL-7.5:
> 
> https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=15904507

Link to yum repo:

http://brew-task-repos.usersys.redhat.com/repos/scratch/jsitnick/openvswitch/2.9.0/19.el7_5.bz1547065.1/openvswitch-2.9.0-19.el7_5.bz1547065.1-scratch.repo

I just fired https://rhv-devops-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/ds-ovirt-system-tests_manual/310/ to test this (I must admit that I am not sure if I filled all the right data for this to succeed).

(I have not ran it properly, as openvswitch-2.9.0-19.el7_5.bz1547065.1 was not pulled in by the job.)

Since this bug has resurfaced after being solved before, I am asking to add qe coverage for it.

I can confirm that your scratch build supports creating of unicode-named logical switches. (ovirt, however, seem to have a bug elsewhere in enumerating them.)

(In reply to Dan Kenigsberg from comment #24)
> I can confirm that your scratch build supports creating of unicode-named
> logical switches. (ovirt, however, seem to have a bug elsewhere in
> enumerating them.)

Thank you. I will backport the fix to Fast Datapath Next branch, from there it will be automatically pulled into Fast Datapath Production branch for next release - FDP 18.06 (in planning for 19 June 2018). In case this doesn't work for RHV, please let us know.

Regarding QE coverage, I can chip in that regression tests have been added upstream but they have not been backported to 2.9 because they were not trivial additions (required some test suite refactoring).

https://mail.openvswitch.org/pipermail/ovs-dev/2018-April/346248.html

Ovirt system tests pass using this change:
https://rhv-devops-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/rhv-system-tests/job/ds-ovirt-system-tests_manual/315/

The tests needed a small adjustment to handle unicode properly too:
https://code.engineering.redhat.com/gerrit/#/c/137482/

I used the build provided by Jakub:
http://brew-task-repos.usersys.redhat.com/repos/scratch/jsitnick/openvswitch/2.9.0/19.el7_5.bz1547065.1/openvswitch-2.9.0-19.el7_5.bz1547065.1-scratch.repo

One remaining thing is to get the exceptions from Python IDL logged in ovirt logs so that it's easier to troubleshoot similar issues in the future.

Marcin, do you happen to have an env where we could experiment with the logger config?

I will set it up, and ping you once it's up.

The logger conf we are using:
https://github.com/oVirt/ovirt-provider-ovn/blob/master/provider/logger.conf

(In reply to Marcin Mirecki from comment #29)
> I will set it up, and ping you once it's up.
> 
> The logger conf we are using:
> https://github.com/oVirt/ovirt-provider-ovn/blob/master/provider/logger.conf

Thanks, Marcin.

this issue is verified on the latest verison:

[root@dell-per730-20 ~]# ovn-nbctl set-connection pssl:6641
[root@dell-per730-20 ~]# ovn-nbctl set-ssl /etc/openvswitch/northdb-privkey.pem \
>                                   /etc/openvswitch/northdb-cert.pem \
>                                   /var/lib/openvswitch/pki/controllerca/cacert.pem
[root@dell-per730-20 ~]# python 1.py
[root@dell-per730-20 ~]# ovn-nbctl show
switch a311658f-4eb4-4a76-bee6-bc998564d1c7 (רשת חיצונית)
[root@dell-per730-20 ~]# cat 1.py
#!/usr/bin/env python2

import logging
import ovs.stream

from ovsdbapp.schema.ovn_northbound.impl_idl import OvnNbApiIdlImpl
from ovsdbapp.backend.ovs_idl.connection import Connection
from ovsdbapp.backend.ovs_idl.connection import OvsdbIdl

# ovn-nbctl get-ssl
SSL_KEY_FILE = "/etc/openvswitch/northdb-privkey.pem"
SSL_CERT_FILE = "/etc/openvswitch/northdb-cert.pem"
SSL_CACERT_FILE = "/var/lib/openvswitch/pki/controllerca/cacert.pem"

def connect():
    ovs.stream.Stream.ssl_set_private_key_file(SSL_KEY_FILE)
    ovs.stream.Stream.ssl_set_certificate_file(SSL_CERT_FILE)
    ovs.stream.Stream.ssl_set_ca_cert_file(SSL_CACERT_FILE)
    
    ovsidl = OvsdbIdl.from_server("ssl:127.0.0.1:6641", "OVN_Northbound")
    ovsdb_connection = Connection(idl=ovsidl, timeout=100)    
    idl = OvnNbApiIdlImpl(ovsdb_connection)
    return idl


def add_switch(idl):
    name = u'\u05e8\u05e9\u05ea \u05d7\u05d9\u05e6\u05d5\u05e0\u05d9\u05ea'
    cmd = idl.ls_add(switch=name, may_exist=False)
    cmd.execute(check_error=True)

    
def main():
    logging.basicConfig()
    idl = connect()
    add_switch(idl)

    
if __name__ == "__main__":
    main()
[root@dell-per730-20 ~]# rpm -qa | grep openvswitch
openvswitch-selinux-extra-policy-1.0-3.el7fdp.noarch
openvswitch-ovn-host-2.9.0-36.el7fdp.x86_64
openvswitch-2.9.0-36.el7fdp.x86_64
openvswitch-ovn-central-2.9.0-36.el7fdp.x86_64
python-openvswitch-2.9.0-36.el7fdp.noarch
openvswitch-ovn-vtep-2.9.0-36.el7fdp.x86_64
openvswitch-ovn-common-2.9.0-36.el7fdp.x86_64
openvswitch-test-2.9.0-36.el7fdp.noarch
[root@dell-per730-20 ~]# 
[root@dell-per730-20 ~]# uname -a
Linux dell-per730-20.rhts.eng.pek2.redhat.com 3.10.0-862.el7.x86_64 #1 SMP Wed Mar 21 18:14:51 EDT 2018 x86_64 x86_64 x86_64 GNU/Linux
[root@dell-per730-20 ~]#

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:1962