Created attachment 943621 [details] packstack_answers.txt Description of problem: Instances fail to start because of error running: sudo nova-rootwrap /etc/nova/rootwrap.conf ip link add qvb55064258-0a type veth peer name qvo55064258-0a Version-Release number of selected component (if applicable): [root@localhost ~(keystone_admin)]# rpm -q python-nova iproute rdo-release python-nova-2014.2-0.4.b3.fc22.noarch iproute-3.16.0-2.fc21.x86_64 rdo-release-juno-1.noarch How reproducible: Seems to be every time. Steps to Reproduce: 1. packstack --gen-answer-file ./answers.txt 2. packstack --answer-file ./answers.txt 3. glance image-create 4. nova boot... Actual results: Instance never spawns and ends in error state. In /var/log/nova/nova-compute.log: 2014-10-02 16:22:31.508 2180 TRACE nova.compute.manager [instance: d9ad23e8-bf6-4d21-9004-991f893fd500] Command: sudo nova-rootwrap /etc/nova/rootwrap.conf ip link add qvb55064258-0a type veth peer name qvo55064258-0a 2014-10-02 16:22:31.508 2180 TRACE nova.compute.manager [instance: d9ad23e8-bf6-4d21-9004-991f893fd500] Exit code: 255 2014-10-02 16:22:31.508 2180 TRACE nova.compute.manager [instance: d9ad23e8-bf6-4d21-9004-991f893fd500] Stdout: '' 2014-10-02 16:22:31.508 2180 TRACE nova.compute.manager [instance: d9ad23e8-bf6-4d21-9004-991f893fd500] Stderr: 'Error: argument "qvb55064258-0a" is wrong: Unknown device\n' Expected results: Instance boot without error. Additional info: Similar error for ubuntu at https://bugs.launchpad.net/ubuntu/+source/neutron/+bug/1364828. Their resolution involved fixing the ip command to honor the optional [ name ] argument so there was no code fix in openstack. We should probably also fix this in our iproute package.
Created attachment 943622 [details] nova-compute.log for one affected instance
To workaround on juno we can modify the ip command line arguments so that the command works. Run the following patch command and then restart services: patch -b -d/ -p0 << EOF --- /usr/lib/python2.7/site-packages/nova/network/linux_net.py.orig 2014-10-02 17:00:19.620007092 -0400 +++ /usr/lib/python2.7/site-packages/nova/network/linux_net.py 2014-10-02 16:40:31.514361599 -0400 @@ -1316,7 +1316,7 @@ for dev in [dev1_name, dev2_name]: delete_net_dev(dev) - utils.execute('ip', 'link', 'add', dev1_name, 'type', 'veth', 'peer', + utils.execute('ip', 'link', 'add', 'name', dev1_name, 'type', 'veth', 'peer', 'name', dev2_name, run_as_root=True) for dev in [dev1_name, dev2_name]: utils.execute('ip', 'link', 'set', dev, 'up', run_as_root=True) EOF openstack-service restart
Ubuntu cherry-picked the fix from Debian, who backported it from upstream: https://git.kernel.org/cgit/linux/kernel/git/shemminger/iproute2.git/commit/?id=f1b66ff83a0babbe99fef81b3a960d7a4ce8dbc6 is the fix. It is in 3.17.0, which was released 11 days ago, so if we put 3.17.0 into Rawhide/F21 it would resolve this, or the fix can be backported.
Rawhide now offers iproute 3.19.0 with all previous patches fixed. Not sure whether we should switch to 3.19.0 in released versions. Is there a still a good reason to either upgrade Fedora releases or backport the fix?
I don't know enough about the differences between 3.16 (current F22) and 3.19 to make that call. As for the need.. this is at least still listed on RDO's list of workarounds: https://www.rdoproject.org/Workarounds#nova_boot:_failure_creating_veth_devices
oops.. didn't mean to clear needinfo for awilliam.
Well, I don't know, I just did a bit of research. Don't even remember what my interest in this bug was.
(In reply to Dusty Mabe from comment #5) > I don't know enough about the differences between 3.16 (current F22) and > 3.19 to make that call. As for the need.. this is at least still listed on > RDO's list of workarounds: > > https://www.rdoproject.org/Workarounds#nova_boot: > _failure_creating_veth_devices Let's target F23 for now then and talk about released versions later.
(In reply to Pavel Šimerda (pavlix) from comment #8) > (In reply to Dusty Mabe from comment #5) > > I don't know enough about the differences between 3.16 (current F22) and > > 3.19 to make that call. As for the need.. this is at least still listed on > > RDO's list of workarounds: > > > > https://www.rdoproject.org/Workarounds#nova_boot: > > _failure_creating_veth_devices > > Let's target F23 for now then and talk about released versions later. Sorry, rawhide is fixed already. Please reopen with a specific fedora release if you have any concerns.