=== NOTE === This bug is not causing breakage but does need to be fixed. It is not a blocker for F25. Description of problem: Booting cloud image with cloud-init-0.7.8-3.fc25.noarch you see the following: Nov 08 19:10:56 digitalocean systemd[1]: multi-user.target: Found ordering cycle on multi-user.target/start Nov 08 19:10:56 digitalocean systemd[1]: multi-user.target: Found dependency on cloud-init.target/start Nov 08 19:10:56 digitalocean systemd[1]: multi-user.target: Found dependency on cloud-final.service/start Nov 08 19:10:56 digitalocean systemd[1]: multi-user.target: Found dependency on multi-user.target/start Nov 08 19:10:56 digitalocean systemd[1]: multi-user.target: Breaking ordering cycle by deleting job cloud-init.target/start Nov 08 19:10:56 digitalocean systemd[1]: cloud-init.target: Job cloud-init.target/start deleted to break ordering cycle starting with multi-user.target/start We need to backport this upstream fix: https://git.launchpad.net/cloud-init/commit/?id=3705bb5964a2ff3f9a67265e6d090a112b35e40c
Any update on this. I have noticed quite a few breaks on the atomic host. It's a bit random, but in slow vms, it happens a lot.
The following are logs from OpenStack/Magnum's tests in the openstack infrastructure. Two vms are created using Fedora Atomic 20170106. In one node cloud init configures the network. [1] On the other node it doesn't. [2] Deployed in the same environment with the same qcow2 image. In an environment built with the same process, we get the error mentioned in the bug description: [3] Jan 16 13:03:02 localhost.localdomain systemd[1]: cloud-final.service: Job cloud-final.service/start deleted to break ordering cycle starting with cloud-init.target/start [1] http://logs.openstack.org/57/417457/22/check/gate-functional-dsvm-magnum-k8s-ubuntu-xenial/f300eb2/logs/cluster-nodes/node-test_replication_controller_apis-172.24.5.8/cloud-init-output.txt.gz [2] http://logs.openstack.org/57/417457/22/check/gate-functional-dsvm-magnum-k8s-ubuntu-xenial/f300eb2/logs/cluster-nodes/master-test_replication_controller_apis-172.24.5.5/cloud-config.txt.gz [3] http://logs.openstack.org/57/417457/22/check/gate-functional-dsvm-magnum-swarm-ubuntu-xenial/f976e1b/logs/cluster-nodes/master-test_start_stop_container_from_api-172.24.5.8/cloud-final.txt.gz
Hi, This looks to be fixed in now release 0.7.9 version of cloud init.
Please can we update 0.7.9?
Building https://koji.fedoraproject.org/koji/taskinfo?taskID=17346019 I'll backport to f25 soon.
> Building https://koji.fedoraproject.org/koji/taskinfo?taskID=17346019 > > I'll backport to f25 soon. That build failed
https://koji.fedoraproject.org/koji/buildinfo?buildID=835362
also here is a scratch build for f25 if anyone wants to grab it and test things out: https://koji.fedoraproject.org/koji/taskinfo?taskID=17393273
I found an issue with the new cloud-init - basically during the run of cloud-init it tries to set the hostname using hostnamectl but dbus hasn't started yet and it fails. We need to figure this out. from the debug log: ``` 2017-01-25 08:59:38,338 - util.py[DEBUG]: Running module set_hostname (<module 'cloudinit.config.cc_set_hostname' from '/usr/lib/python3.5/site-packages/cloudinit/config/cc_set_hostname.py'>) failed Traceback (most recent call last): File "/usr/lib/python3.5/site-packages/cloudinit/stages.py", line 776, in _run_modules freq=freq) File "/usr/lib/python3.5/site-packages/cloudinit/cloud.py", line 54, in run return self._runners.run(name, functor, args, freq, clear_on_fail) File "/usr/lib/python3.5/site-packages/cloudinit/helpers.py", line 187, in run results = functor(*args) File "/usr/lib/python3.5/site-packages/cloudinit/config/cc_set_hostname.py", line 47, in handle cloud.distro.set_hostname(hostname, fqdn) File "/usr/lib/python3.5/site-packages/cloudinit/distros/__init__.py", line 84, in set_hostname self._write_hostname(writeable_hostname, self.hostname_conf_fn) File "/usr/lib/python3.5/site-packages/cloudinit/distros/rhel.py", line 130, in _write_hostname util.subp(['hostnamectl', 'set-hostname', str(hostname)]) File "/usr/lib/python3.5/site-packages/cloudinit/util.py", line 1850, in subp cmd=args) cloudinit.util.ProcessExecutionError: Unexpected error while running command. Command: ['hostnamectl', 'set-hostname', 'cloudhost.localdomain'] Exit code: 1 Reason: - Stdout: - Stderr: Failed to create bus connection: No such file or directory ``` and from the journal you can see that dbus starts after cloud-init: ``` -bash-4.3# journalctl -u cloud-init | grep -P 'Starting|hostname' Jan 25 08:59:36 localhost.localdomain systemd[1]: Starting Initial cloud-init job (metadata service crawler)... Jan 25 08:59:38 localhost.localdomain cloud-init[1308]: 2017-01-25 08:59:38,331 - util.py[WARNING]: Failed to set the hostname to cloudhost.localdomain (cloudhost) Jan 25 08:59:38 localhost.localdomain cloud-init[1308]: 2017-01-25 08:59:38,338 - util.py[WARNING]: Running module set_hostname (<module 'cloudinit.config.cc_set_hostname' from '/usr/lib/python3.5/site-packages/cloudinit/config/cc_set_hostname.py'>) failed -bash-4.3# -bash-4.3# -bash-4.3# journalctl -u dbus | head -n 2 -- Logs begin at Wed 2017-01-25 08:59:29 UTC, end at Wed 2017-01-25 09:02:04 UTC. -- Jan 25 08:59:38 localhost.localdomain systemd[1]: Started D-Bus System Message Bus. ``` We need to file a bug upstream and/or send a patch to get this fixed for us. I pulled the patch from the commit in the first comment of this BZ and applied it to cloud-init 0.7.8 in Fedora and built a scratch build: https://koji.fedoraproject.org/koji/taskinfo?taskID=17405006
cloud-init 0.7.9 includes: DefaultDependencies=no Which means that it does *not* have the implicit dependency on basic.target. So with 0.7.8, running systemd-analyze dot cloud-init.service yields: "cloud-init.service"->"basic.target" [color="green"]; But in 0.7.9 there is no such dependency. dbus.service runs before basic.target, and this is why everything works in 0.7.8 but hostname setting fails in 0.7.9.
With this build [1] from comment #9, I did a few runs in OpenStack's CI and I didn't see the error again. [1] https://koji.fedoraproject.org/koji/taskinfo?taskID=17405006
Related: https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1629797
And also: https://github.com/systemd/systemd/issues/4504
Filed the new issue as bug 1417025.
cloud-init-0.7.8-5.fc25 has been submitted as an update to Fedora 25. https://bodhi.fedoraproject.org/updates/FEDORA-2017-1f675fd070
cloud-init-0.7.8-5.fc25 has been pushed to the Fedora 25 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-1f675fd070
cloud-init-0.7.8-5.fc25 has been pushed to the Fedora 25 stable repository. If problems still persist, please make note of it in this bug report.