Testing a local live build which has systemd 6-2 and dbus-1.3.2-0.1.885483.fc14 , I can successfully boot the composed live image and do an install to hard disk. However, on booting the installed system, I wind up straight at gdm; firstboot doesn't show up, so no user creation step occurs and the system is effectively unusable. I can switch to a vt and manually invoke firstboot and run it, but on quit it leaves the system stuck at a black screen. There are some errors from systemd related to firstboot in the logs, I'll try and recreate the issue and provide those soon.
-- Fedora Bugzappers volunteer triage team https://fedoraproject.org/wiki/BugZappers
note this is not the same as https://bugzilla.redhat.com/show_bug.cgi?id=614538 , it's a new issue. blocks alpha under criterion "In most cases, the installed system must boot to a functional graphical environment (see Blocker_Bug_FAQ)". -- Fedora Bugzappers volunteer triage team https://fedoraproject.org/wiki/BugZappers
attaching a screenshot of the errors, from within my test vm. -- Fedora Bugzappers volunteer triage team https://fedoraproject.org/wiki/BugZappers
Created attachment 437277 [details] errors from /var/log/messages on boot
Investigating this a bit further, firstboot uses the result of the 'runlevel' command to decide whether to run in text or graphical mode, and the output of 'runlevel' on the installed system in question is 'unknown', which can't be helping. (I notice when I try and shut down the system in this situation that firstboot shows up, in text mode, behind the shutdown process - though it doesn't seem to run *before* I try and shut down). Given that with systemd the runlevel concept doesn't really apply any more, firstboot should probably grow code to first check the systemd target or something, before trying to check the runlevel. But even that would be 'wrong', given that we can have any number of arbitrary systemd targets, really. Maybe it should check whether prefdm is enabled in the currently active systemd target. CCing the firstboot maintainers. Lennart, it may be useful for you to suggest the best way forward here. (I'm going to hack up firstboot a bit to always try and start up in graphical mode, and see if that 'fixes' it). -- Fedora Bugzappers volunteer triage team https://fedoraproject.org/wiki/BugZappers
so yeah, seems like there's two problems here: somehow, with systemd, firstboot triggers on system *shutdown*, not startup, and with the problem with 'runlevel', it runs in text mode, not graphical mode (the code actually handles the case where the result of runlevel is 'unknown'; it treats it as runlevel 3). I'm not sure why the thing with it kicking in on shutdown happens. -- Fedora Bugzappers volunteer triage team https://fedoraproject.org/wiki/BugZappers
after the change to force it to run in graphical mode, as far as I can tell, on the next boot, it seems like the system wound up running firstboot's X on vt6 and the regular desktop on vt7, simultaneously. switching to vt6 gave me a black screen with an X cursor for a while, then X apparently crashed and it went to being a normal tty. does our current firstboot system rely on having a more-or-less serial init, so once the firstboot service kicks in, the regular desktop won't start up until firstboot is done, and this gets broken by systemd launching things in parallel? -- Fedora Bugzappers volunteer triage team https://fedoraproject.org/wiki/BugZappers
Created attachment 437426 [details] [PATCH] firstboot: add systemd service Steps to reproduce: chkconfig firstboot on rm /etc/sysconfig/firstboot reboot firstboot must be treated specially: - it interacts with the user, so it must not timeout - it must delay the start of prefdm until it's finished => it needs a native systemd service file. The attached patch adds it. It works for me.
scratch build with the patch applied: http://koji.fedoraproject.org/koji/taskinfo?taskID=2388036
I can reproduce it after F14-alpha-rc2-x86_64 virt-install. Firstboot didn't show up automatically. Had to turn to tty2 to start it manually.
Patch looks pretty good. one more addition: You might want to add "Before=prefdm.service" in the [Unit] section to make sure it is started before prefdm.service.
I haven't tested it yet, but AFAICT, it doesn't address the runlevel issue. If that's not addressed, we'll get the text 'firstboot' (which is a lot less useful than the graphical one), which is not desired. Michal, Lennart - did either of you check that out? -- Fedora Bugzappers volunteer triage team https://fedoraproject.org/wiki/BugZappers
The service file has: Environment=RUNLEVEL=5 firstboot uses the "runlevel" command to find out the active runlevel and the command accepts the value of this environment variable as an override. So we'll get firstboot in graphical mode.
So, the trick is to have two firstboot services: firstboot-text.service and firstboot.graphical.service, the first being pulled in (via a .wants/ link) by multi-user.target the other by graphical.target. The first should have RUNLEVEL=3 the other RUNLEVEL=5 in an env var. Then, firstboot-graphical.service should "conflict" firstboot-text.service, to make sure that only one of them is enabled. In the systemd version in git in such a case the unit that has the "conflicts" will then win over the one that is "conflicted by", and everything should work as intended. I will upload a new version of systemd shortly that has the mentioned "Who wins a conflict?" tweak added. The current version in f14 would arbitrarily remove one of the two conflicting services, giving the admin little control which one is removed in case of conflict.
that sounds like practically speaking it ought to work in all cases I can think of, yes. lennart, when you submit this new version of systemd as an update for f14, can you include the updated dbus from the system 5-2 submission too? the 6-1 and 6-2 submissions didn't have that dbus package, so we might have pulled the fixed systemd but not the fixed dbus...thanks! -- Fedora Bugzappers volunteer triage team https://fedoraproject.org/wiki/BugZappers
Adam, I already posted a new ticket in bodhi for this 1h ago or so.
really? when I search systemd in bodhi, it just takes me straight to https://admin.fedoraproject.org/updates/systemd-6-2.fc14 ... -- Fedora Bugzappers volunteer triage team https://fedoraproject.org/wiki/BugZappers
just tested a live image with the above firstboot scratch build, it still ends up with firstboot and gdm running together (gdm on vt1, firstboot on vt6). I will try doing my own build with Lennart's "Before=prefdm.service" suggestion. -- Fedora Bugzappers volunteer triage team https://fedoraproject.org/wiki/BugZappers
Heh. If I add Lennart's suggested fix, then firstboot gets run when booting to the *live* environment...:) there's a command in the %post of the live CD generation kickstart to disable firstboot: # turn off firstboot for livecd boots chkconfig --level 345 firstboot off 2>/dev/null I guess we need a systemd equivalent there, now we're using a systemd unit file. I think just adding 'systemctl disable firstboot.service' should do the job (and for Lennart's proposed modification, make it two commands to cover the two services, obviously). Note there's several others in a similar vein here, so if those get converted into systemd native format we'd have to add commands for those too. We should probably stick a comment in the kickstart file to that effect. I'll test another live compose with this proposed fix... -- Fedora Bugzappers volunteer triage team https://fedoraproject.org/wiki/BugZappers
mgracik - looks like we need your input here. Firstboot and systemd need to play nice together for F-14-Alpha. We have very little time remaining to fix, build and release on time. Michael Schmidt has posted a first draft patch for review (see attachment#437426 [details]). Any input you have is appreciated!
Note the first draft patch is not sufficient. It needs to be adjusted as per Lennart's comments #11 and #14. I can try to provide a patch that implements those. -- Fedora Bugzappers volunteer triage team https://fedoraproject.org/wiki/BugZappers
Okay, so I'm still having problems. Following on from comment #19, I tried my proposal there - this change to fedora-live-base.ks from spin-kickstarts: ------------------------ @@ -180,8 +180,13 @@ fi action "Adding live user" useradd \$USERADDARGS -c "Live System User" liveuser passwd -d liveuser > /dev/null +# For all the below, please be aware that if these services are made systemd +# native, a systemctl disable command should be added to / replace the +# chkconfig command + # turn off firstboot for livecd boots chkconfig --level 345 firstboot off 2>/dev/null +systemctl disable firstboot.service 2>/dev/null # don't start yum-updatesd for livecd boots chkconfig --level 345 yum-updatesd off 2>/dev/null ------------------------ However, it doesn't exactly work. Here's what happens. When I build a live image with this change, and with the proposed new systemd firstboot service, the boot dumps me to a text console. No firstboot, no gdm. Here's my reasoning as to *why* this happens. The %post section from live spin kickstarts actually gets run *during boot of the built live image* - not at any point during creation of the live image. So we have, to some degree, a race going on: 'systemctl disable firstboot.service' is being run early in boot, right around the time systemd is starting stuff up. If I log in to a VT in the affected live image and run 'systemctl status firstboot.service', the output includes: Active: activating (start) CGroup: name=systemd:/systemd-1/firstboot.service 1287 sd:exec this is far fewer processes than you see for the same command when the firstboot service is actually 'working', and running firstboot. I reckon I'm hitting a race condition here: the "systemctl disable firstboot.service" operation is getting run right in the middle of systemd trying to start the firstboot service. What 'systemctl disable foobar.service' actually does, according to the manpage, is remove all the symlinks which systemd uses to run the service. So I think systemd gets stuck, with the service forever - as far as it knows - in the middle of initialization, but with the symlinks pulled out from underneath it so it can't actually complete starting it up. Since we need to set firstboot.service to block the prefdm service running until it's complete, and firstboot.service is forever stuck, the prefdm service never gets started, so gdm never starts up. Presumably this worked with upstart because it's more serial; by the time upstart had got to the point where it would have started the firstboot service, the command to disable it had already been run. The race existed but was always won, easily, by the 'chkconfig' command used to disable the service, so everything worked. I'm not sure there's an easy way to fix this. I think it needs a re-think of how we accomplish the ultimate goal - have firstboot not run when you boot to the live environment, but run when you first boot into a system installed from the live environment. I'm not entirely sure what would be the best way to achieve this, though. Have the firstboot systemd service be able to detect whether it's running in a live environment or not? -- Fedora Bugzappers volunteer triage team https://fedoraproject.org/wiki/BugZappers
the thought occurs that if the %post operations get run as part of a service on boot, we can just adjust the firstboot service to not run until that's complete. I'm not sure they do get run that way, though. I'll investigate. -- Fedora Bugzappers volunteer triage team https://fedoraproject.org/wiki/BugZappers
okay, update: the %post stuff is actually run during live image creation, but what I missed is that the commands I saw aren't actually *run* at %post, but are stuck into a /etc/init.d/livesys file on the live image - i.e. it's creating a service, owned by no package, that gets run at boot. ew. still, that's good for us: we can make firstboot not run until this livesys service has run, and that should solve the race. will test that. -- Fedora Bugzappers volunteer triage team https://fedoraproject.org/wiki/BugZappers
damn. still doesn't work, but I can't figure out why. so, here's what I've done in total: to spin-kickstarts fedora-live-base.ks: ----------- @@ -180,8 +180,13 @@ fi action "Adding live user" useradd \$USERADDARGS -c "Live System User" liveuser passwd -d liveuser > /dev/null +# For all the below, please be aware that if these services are made systemd +# native, a systemctl disable command should be added to / replace the +# chkconfig command + # turn off firstboot for livecd boots chkconfig --level 345 firstboot off 2>/dev/null +systemctl disable firstboot.service 2>/dev/null # don't start yum-updatesd for livecd boots chkconfig --level 345 yum-updatesd off 2>/dev/null ------------ to firstboot.service from michal's scratch build: ------------ @@ -1,5 +1,7 @@ [Unit] Description=firstboot configuration program +After=livesys.service +Before=prefdm.service [Service] Environment=RUNLEVEL=5 ------------ somehow, building a live image with these changes, firstboot gets run at startup (I don't get the weird race condition). I've no idea why this is. It should be disabled, but it gets run - by systemd - just as if the systemctl disable command never happened. The systemctl disable command definitely wound up in /etc/init.d/livesys , and that script definitely ran (as the liveuser account is created, and all the other stuff it does happened). We could adjust the fedora-live-base.ks a bit to give some feedback so we can tell for sure when it kicks in and if it works. Lennart, did I miss a bit the command would need to work? Does it need a --system or a --global? sorry, I have to go out golfing now, but I'll poke this more when I get back. -- Fedora Bugzappers volunteer triage team https://fedoraproject.org/wiki/BugZappers
A new systemd version (v7) that ensures that if two conflicting jobs are scheduled the one that has the "conflicts" is kept while the one that is "conflicted" is removed is now in bodhi.
still can't make it work. :/ systemctl disable firstboot.service should have an implied systemctl daemon-reload , but if I add a systemctl daemon-reload command right after the systemctl disable firstboot.service command along with all the other above tweaks, I get back into the race condition with prefdm not starting and firstboot stuck at 'activating'. I really don't know why it don't work, but it don't. -- Fedora Bugzappers volunteer triage team https://fedoraproject.org/wiki/BugZappers
Hmm, do you have an image or so I can test this with? Or alternatively, can you get me the output of "dmesg" when bootet with "systemd.log_target=kmsg systemd.log_level=debug" on the kernel cmdline?
Adam, if you add "--no-reload" to the "systemctl disable" line, do things work for you then? If so, that might be a good temporary fix and I'll look into fixing this properly later on.
testing. I thought of an alternative workaround, too: we could simply have the firstboot service check if it's running live (parse cmdline for 'livesys') and not run the firstboot app if it is. -- Fedora Bugzappers volunteer triage team https://fedoraproject.org/wiki/BugZappers
I'm starting to be concerned that this bug (and a couple of others on the blockers list) are going to cause us to slip the schedule. It's very unclear to me whether we've found the root problem in this bug, of if we're just looking for workarounds. Is there anything I can do to help facilitate things here?
(In reply to comment #31) > I'm starting to be concerned that this bug (and a couple of others on the > blockers list) are going to cause us to slip the schedule. It's very unclear > to me whether we've found the root problem in this bug, of if we're just > looking for workarounds. Is there anything I can do to help facilitate things > here? Things are very dynamic with this issue at the moment. Adam, Martin and Lennart have met and developed a strategy on IRC. However, time is short, and in order to gain confidence that the Alpha has met release criteria, 1) this issue, 2) Additional remaining OPEN F14Alpha blockers, and 3) the test matrices must be completed in time for the go/no-go meeting scheduled for tomorrow. We can talk more about general readiness on the list (or irc).
jared: we know what the problem is, and I'm testing a fixed package now. The definition of whether the fix is a 'fix' or a 'workaround' is kind of just a semantic argument - the existing implementation is already something of a hack, and it's arguable whether it's less 'hacky' to successfully do the same thing with systemd or just do it a different, but in a way more robust, way. (Which is the current approach). Bottom line, we should have this behaving as it ought to for Alpha. Whether the way that behaviour is implemented is the best possible way is debatable, but then prior to systemd the same was true. -- Fedora Bugzappers volunteer triage team https://fedoraproject.org/wiki/BugZappers
okay, I think I've nailed this. Here's a firstboot package which, with systemd 7-1 and the appropriate changes to spin-kickstarts, produces the correct results. nirik has committed the spin-kickstarts change, and lennart has a pending update request for the necessary systemd 7-1. -- Fedora Bugzappers volunteer triage team https://fedoraproject.org/wiki/BugZappers
Created attachment 437987 [details] working firstboot build (.src.rpm)
Thanks for the update, Adam. I appreciate it!
firstboot-1.112-1.fc14 has been submitted as an update for Fedora 14. http://admin.fedoraproject.org/updates/firstboot-1.112-1.fc14
firstboot-1.112-2.fc14 has been submitted as an update for Fedora 14. http://admin.fedoraproject.org/updates/firstboot-1.112-2.fc14
firstboot-1.112-2.fc14 has been pushed to the Fedora 14 stable repository. If problems still persist, please make note of it in this bug report.
firstboot-1.112-3.fc14 has been submitted as an update for Fedora 14. http://admin.fedoraproject.org/updates/firstboot-1.112-3.fc14
firstboot-1.112-3.fc14 has been pushed to the Fedora 14 stable repository. If problems still persist, please make note of it in this bug report.
*** Bug 623559 has been marked as a duplicate of this bug. ***
firstboot-1.113-1.fc14 has been submitted as an update for Fedora 14. https://admin.fedoraproject.org/updates/firstboot-1.113-1.fc14
firstboot-1.113-4.fc14 has been submitted as an update for Fedora 14. https://admin.fedoraproject.org/updates/firstboot-1.113-4.fc14