Bug 1399234

1. To copy the DSCP mark:

Indeed, the DSCP mark implementation in the neutron QoS extension marks the packets coming out of the VM before encapsulation in VxLAN or GRE tunnels. The OVS implementation uses OVS flows in br-int to set the DSCP field of incoming packets (VM to network). The linuxbridge implementation uses iptables to mark the packets coming from the device connecting the VMs to the bridge.

When the packet is encapsulated, the DSCP field is not automatically copied from the inner header to the outer header. To do so, the solution in OVS is to set the tunnel interfaces (VxLAN or GRE) with the right option:

ovs-vsctl set interface vxlan-xxxxxxxx options:tos=inherit

The TOS field is comprised of the DSCP field and the ECN field. The ECN field is automatically copied as the RFC 6040 is implemented in the Linux kernel since version 3.13 at least. Setting the TOS field to "inherit" means that the value of the DSCP field of the inner header will be copied to the outer header.

The inherit option can be set from the neutron QoS extension code by updating the OVS tunnel ports (in br-tun) in all computes, or it can be done at the creation of the tunnel ports by the neutron OVS agent.

In linuxbridge, the VxLAN ports cannot be set to inherit, they can only be set to a specific value at the creation of the VxLAN tunnel port. This specific value can be set in the ml2_conf.ini configuration file, when using linuxbridge as a driver, in the section [VXLAN] by adding tos=value. The problem is that we cannot specify a DSCP mark per tenant as all tenants using this tunnel will have the same DSCP.

2. To have different DSCP marks in inner and outer header:

To extend further the problem of DSCP marking, we may want to have different values in the inner header and the outer header, rather than copy it, so that the underlay network using the outer header can make its own decisions based the outer DSCP value without being tied to the inner DSCP value.

In OVS, the elegant solution to set the outer DSCP mark would be to do this:

ovs-vsctl set interface vxlan-xxxxxxxx options:tos=flow

and manage all the tenant outer DSCP marks with flows in br-tun. But this is not implemented in OVS. So the big workaround would be to create a tunnel port with a specific TOS value for every DSCP mark we would need, and switch each tenant traffic to the corresponding port/DSCP mark. But OVS doesn't allow to have multiple ports with the same source and destination IP addresses and distinct TOS values.

A better solution I tested uses iptables with the MARK target. The idea is to set the packets coming out of a tenant VM with a specific MARK value with iptables (which sets the "mark" field in sk_buff structure associated to the packet). This value persists after encapsulation in VxLAN tunnels, so we can match this value with iptables to set the DSCP in the corresponding outer header.

This solution works using linuxbridge. We only need to see if their are no interference with other MARK values used in neutron. And also see how it works with namespaces as it seems that the MARK value is cleared when the packet is sent to another namespace (in skb_scrub_packet() in Linux kernel).

OVS also allows to use the sk_buff mark field with bridge flows, it can be modified with set_field:value->pkt_mark. This works between OVS bridges (we can match it with pkt_mark=value), but I noticed that the value does not persist after the packet exits the OVS tunnel ports (e.g., vxlan port in br-tun). So we cannot match it afterwards with iptables to mark the DSCP field on the outer header as with linuxbridge.

It seems that even if we mark the VM packets with a specific value using iptables (MARK target) before the packet enters the OVS bridges, the mark value cannot be used in the OVS flows, as if the mark value was cleared before entering OVS. We are still investigating why the mark value is cleared and cannot be used between iptables and OVS (maybe skb_scrub_packet() is called before entering or leaving OVS ports).

Nevertheless, the use of the mark field can be a solution to set the DSCP field of the outer header with a specific value per tenant.

After few verifications and tests with OVS, the mark field of sk_buff is indeed zeroed by skb_scrub_packet, but before entering OVS. When the mark is set with iptables at the egress of the VM (tap) in the PREROUTING chain, the value is cleared when the packet gets through the veth connected to br-int (qvb/qvo pair). But when the mark is set through flows with set_field:value->pkt_mark, the value can be used after VxLAN encapsulation to set the good DSCP mark in the outer header through iptables in the OUTPUT chain.

So we have a solution to set the DSCP of the outer header in both linuxbridge and OVS using the mark field.

I filed a RFE in the neutron launchpad concerning this issue: https://bugs.launchpad.net/neutron/+bug/1692951

Adding a note to mention that this was discussed during neutron_qos meeting[1]

[1] http://eavesdrop.openstack.org/meetings/neutron_qos/2017/neutron_qos.2017-06-20-15.00.log.html

We have reviewed this feature request and we won't be able to prioritize it for RHOSP 13 due to other priorities.

For the sake of completeness: step 1 (as described in comment #5) is being implemented ( https://review.openstack.org/#/c/501267 ), but an implementation of Step 2 and of an API to control these mappings, are not yet planned.

We decided to implement this feature with ODL, see BZ 1550668.

*** This bug has been marked as a duplicate of bug 1550668 ***