Bug 622663

Well, systemd starts everything in parallel. Which has the effect that if every single start up script is connected to the tty all output will be mixed up which is not pretty. We thought about this quite a bit and came to the conclusion that the only really sensible solution is to hide the output in the normal case and live with the fact that it is all mangled up if the output is enable.

We bound this to the "quiet" kernel cmdline option. If you boot without "quiet" specified the sysv output will end up on the console as it always did. You may also use SysVConsole=yes in /etc/systemd/system.conf to make this permanent.

I we want a more verbose boot-up that shows output in a nice and clean way then we should probably hack something up with ncurses that properly visualizes what's going on without mixing up the output completely.

The problem is that you had this nice feature, that any boot problem would trigger a warning sign, and clicking on this warning in gdm of in your desktop would show you what was wrong

Now you still have the warning sign, but no info whatsoever on what failed exactly. And I don't think people will like the "no boot logs any-more, reboot after prodding grub manually if you want to know why you got a warning"

well, you can always just type "systemctl" in the command line and then "systemctl status" to figure out what's going wrong. It's not that bad.

But yepp, as mentioned I am thinking about adding some ncruses frontend which tries to mimic the old output but can deal with parallelization properly and works on all services, not just sysv stuff.

Well, given the warning sign, that's more an issue with plymouth's boot log viewer. I don't think a ncurses frontend will help with that.

CC'ing Ray.

*** Bug 627044 has been marked as a duplicate of this bug. ***

(In reply to comment #1)
> I we want a more verbose boot-up that shows output in a nice and clean way then
> we should probably hack something up with ncurses that properly visualizes
> what's going on without mixing up the output completely.


This sounds like a great idea to me, although ideally it'd also work on a line-oriented console. Even something that just listens and collects messages as they come in and serializes them, printing the results line by line.

Is there an RFE or todo item for this already?

(In reply to comment #3)
> well, you can always just type "systemctl" in the command line and then
> "systemctl status" to figure out what's going wrong. It's not that bad.
> 
> But yepp, as mentioned I am thinking about adding some ncruses frontend which
> tries to mimic the old output but can deal with parallelization properly and
> works on all services, not just sysv stuff.

Thinking about it a little more, if we forget about the way it used to work, and think about what would be useful for the end user, what should happen is:

1. any systemd unit launch that failed since boot up should trigger the warning icon (in gdm, the user panel, etc)
2. clicking on this icon should list all the failures since boot
3. (ideally) there should be more information that the old system used to provide on why a particular unit failed (unit logs and messages, abrt traces, selinux warnings, etc), maybe through a detailed status panel when clicking on an item in the list

And that does not specifically require parsing boot.log like the old system did, nor an ncurses frontend. In fact that would work better than the old system, since it would catch service failures that occur after plymouth exited, remove all the noise about successes, and provide more info

Of course it would be nice if it also ended up in boot.log, but dumping a raw log on the user is only good if you have no better mean to provide the info the user actually wanted (what failed; and why)

I've updated the summary of this to be more related to what we're tracking in this bug now. The Plymouth integration for whatever we come up with should probably be another report.

Hmm, does it really make sense to make this F14Target?

I personally believe that typing "systemctl" after boot is a lot more useful replacement for the boot screen that is almost invisible these days due early gdm and such.

"Hmm, does it really make sense to make this F14Target?"

I wouldn't lose any sleep over it. The Target bugs mean virtually nothing, they really aren't tracked for anything.



-- 
Fedora Bugzappers volunteer triage team
https://fedoraproject.org/wiki/BugZappers

I don't think it's an F14 blocker, but remember that early GDM just covers the runlevel 5 use case.

(In reply to comment #9)
> Hmm, does it really make sense to make this F14Target?
> 
> I personally believe that typing "systemctl" after boot is a lot more useful
> replacement for the boot screen that is almost invisible these days due early
> gdm and such.

It's a lost more useful than typing "systemctl" after boot because it only activates when there is a problem so you don't have to check manually the state just in case something was wrong (granted right now something *is* always wrong so there's not much difference)

Moving systemd bugs to f15, since the systemd feature got delayed.

It's really unacceptable that this fundamental usability issue has gotten ignored for six months and we are now about to ship a release wherein it's still broken.

Might I suggest as a stopgap the solution suggested in bug #684152, ie, divert all stdout/stderr output from SysV initscripts to syslog?  Then at least, when the thing says

Starting mysqld (via systemctl):  Job failed. See system logs and 'systemctl status' for details.
                                                           [FAILED]

it's not a bald-faced lie.  There are quite a few initscripts that expect that printing something to stdout or stderr is a useful thing to do, so I'm not happy at all to discover that systemd has decided to suppress potentially-critical information on the grounds that it might not print nicely.

This can be done by editing /etc/systemd/system.conf as follows:

DefaultStandardOutput=syslog
DefaultStandardError=syslog

or via systemd.default_standard_output=, systemd.default_standard_error= on the boot command line.

*** Bug 705102 has been marked as a duplicate of this bug. ***

*** Bug 708054 has been marked as a duplicate of this bug. ***

*** Bug 708662 has been marked as a duplicate of this bug. ***

Comment 1 flaunts everything that is WRONG with systemd.
If the developers "thought about this quite a bit" and still came to 
the conclusion they did, they just weren't thinking clearly.

Precisely because everything starts up in parallel, we are left with an
undecipherable pile of poo that leaves failures buried in an 
impenetrable morass of meaningless verbiage, instead of the clear list
of color-coded results of sysv init.

Parallelization might be attractive if it resulted in faster boot 
time, but it doesn't.  Measured times from luks passphrase to login 
prompt on my IBM T30 laptop are:
    F14:  55 sec
    F15:  62 sec
    
"If you boot without "quiet" specified the sysv output will end up on
the console as it always did."   You're kidding, right? 

I tried this and couldn't believe the vast spew of excess drivel.
It's totally useless.

Parallelization is a waste of effort.  Lose it!

This is a bug report, not a systemd discussion forum. Please refrain from discussing matters irrelevant to the bug report. We're talking about improving systemd's console output, not whether parallelization of service startup is a good thing.

> 
> Thinking about it a little more, if we forget about the way it used to work,
> and think about what would be useful for the end user, what should happen is:
> 
> 1. any systemd unit launch that failed since boot up should trigger the warning
> icon (in gdm, the user panel, etc)
> 2. clicking on this icon should list all the failures since boot

Arggh. . .
Bad idea. Let's not "forget about the way it used to work." That is basically telling the current user base to go to hell. We have Fedora on many servers. There is no GUI and we prefer not to need it. So the need to "click" on anything is not feasible. 

What was wrong with the text output we had before, and why is that so difficult to implement in systemd? It seems to me to be a simple matter of cleaning up the output and reordering it in the proper boot sequence.

We have used RH/Fedora since basically the inception of Red Hat. One of the reasons we prefer it is due to the clean console boot interface with easily noticeable error messages. The current console output mess is worse than Debian/Ubuntu. If we wanted that we would switch to Debian. SystemV is certainly outdated. The possible benefits of systemd are great. But currently, as seen from the countless systemd bug reports, it seems that systemd was incorporated into Fedora a bit prematurely. Fedora is supposed to be the base for RHEL. But if Red Hat puts systemd in its current state into RHEL they will see users leave in droves and their share price drop through the floor. So why is it in Fedora if there are something like 536 bugs (systemd bug search)?

I don't want to turn this bug report into a "systemd discussion forum," but it would be nice to know from the developers/maintainers when some of the more obvious systemd drawbacks, such as this bug and the lack of fsck feedback, will be resolved. This bug is almost a year old and nothing so far seems to have been done. We don't want to be stuck with F14 for several years, but currently, due to systemd, IMHO that is the only stable Fedora version.

I have also been frustrated by the inaccurate "See system logs and 'systemctl
status' for details" message, and I would really appreciate more helpful logs.

Note that that statement in current Fedora 16 is more accurate - output from services is going to syslog.

Please have another look at comment 21. ('easily noticeable error messages')

Hi, 
maybe we may start by updating systemd from .26 to .34, in Fedora 15 has as discussed in thread .
http://lists.fedoraproject.org/pipermail/devel/2011-September/156924.html

I (and likely the maintainer) would strongly prefer to keep this related to the topic in the bug, and not other issues.

Thank you to Bill and Lennart for continuing to work on this issue. I like systemd and feel it is much better than the old init process. We just need it to be a bit more legible at boot time. 

The biggest issue for us with our servers is the illegibility of the console output when combined with the kernel messages from the now closed Bug 683254.

Due to certain peculiarities of our hardware we MUST boot without RHGB and quiet option. When Plymouth was introduced the kernel messages after init were distracting, but tolerable. We understand and agree with the reasoning for marking Bug 683254 as closed. But we also agree with the statement from one of the that bug's commenters:

"1) Kernel boot messages often are the only resort of seeing and understanding
why the system doesn't boot properly or misbehaves. At the same time removing
quiet boot option makes the system overly verbose.

2) A lot of people may disagree why you are changing 13+ years defaults."

I know both of you are familiar with that bug.

When systemd was introduced it became almost impossible for us to easily isolate the systemd boot output from the kernel messages. That is where the biggest problem is for us. So I would kindly ask that when implementing systemd output changes that you also consider that issue and make it easier to isolate systemd messages from the kernel output. That could easily be done through ANSI color changes.

I agree with Russ here. I would also like to add that GNOME seems to be taking the philosophy (borrowed from the "If Operating Systems Ran the Airlines" joke):

Mac Airlines

 All the stewards, captains, baggage handlers, and ticket agents look and act exactly the same. Every time you ask questions about details, you are gently but firmly told that you don't need to know, don't want to know, and everything will be done for you without your ever having to know, so just shut up.

^ This is a bad philosophy. Specifically, the "so just shut up" part. Hiding the system only makes a more complex system. Making something pop up in GDM that shows boot status can be a problem if, say, the user doesn't use GDM (I use LXDM on all systems, personally), or if the system doesn't finish booting.

systemd is an upstart fork, is it not? Would the upstart code for clean boot messages provide a starting point, or is systemd just too different?

I noticed that when Debian Sid changed from BSD-style init to systemd, their boot progress readability did not change one single bit. The only visible difference was the change from "using BSD-style concurrent startup" to "using systemd". Other than that, a user watching the boot process would never know the difference.

Every time I have pointed to Debian Sid's implementation of systemd on IRC, except one, the response was one of three things: "Nope, bad idea," "Nope, user doesn't need to know" (see Mac Airlines above and my response), or "stop trolling." The last one because "So what if another distro got it working? This is Fedora, we don't care about other distros." This reasoning only hurts progress, which Fedora is supposed to be all about.

I admittedly don't know how Debian implements systemd. If it's all sysv-init scripts or what, I have no idea. I don't even know whether it's actually systemd that makes the output readable. All I know is that Debian Sid seems to have more readable startup progress than Fedora 15 (admittedly I haven't tried F16 yet). Someone please take a look if you have the time. I don't have the coding background to make sense of such things.

"systemd is an upstart fork, is it not?"

no.

(In reply to comment #23)
> Note that that statement in current Fedora 16 is more accurate - output from
> services is going to syslog.

I read in thread mention abive 
" And FYI to all those that gloriously want to upgrade and claim that
it's bug free or "they" ( all of what two people ) not encountered any
issues inetd-style socket activation is borked in .35 ( users need to
downgrade to .34 or add StandardOutput=socket if they cant wait until 36 goes
out )."

So "output from services is going to syslog" , is a know bug of systemd .35 ?

(In reply to comment #30)
> So "output from services is going to syslog" , is a know bug of systemd .35 ?

No. Output from services going to syslog by default is the intended behavior. Only the way it interacts with socket activation is a bug.

> When systemd was introduced it became almost impossible for us to easily
> isolate the systemd boot output from the kernel messages. That is where the
> biggest problem is for us. So I would kindly ask that when implementing systemd
> output changes that you also consider that issue and make it easier to isolate
> systemd messages from the kernel output. That could easily be done through ANSI
> color changes.

not sure if this helps you, but we have changed the kernel now so that dmesg log messages can carry full facility values. Then we have change all userspace tools which generate output to kmsg to set these facility values. The effect of that is that dmesg can filter the kmsg buffer for kernel messages. In newer util-linux you can hence type "dmesg -k" to see only kernel messages.

This package has changed ownership in the Fedora Package Database.  Reassigning to the new owner of this component.

You may be interested in this update:
https://admin.fedoraproject.org/updates/systemd-37-6.fc16

(In reply to comment #34)
> You may be interested in this update:
> https://admin.fedoraproject.org/updates/systemd-37-6.fc16


systemd-37-11.fc16.x86_64 has landed into stable.
Good work, I like it. The green OK and the red FAIL, but they aren't on center or the are on left of display.
I prefer is they are on center, or near of messages, can center it myself ? 
Thanks,

The code prints the status marks aligned to the right edge of the console. Some of the messages can be long and we don't want to limit the space for them artificially. The printing position is not configurable.
If you're getting the statuses on the left, something is wrong.

(In reply to comment #36)
> The code prints the status marks aligned to the right edge of the console. Some
> of the messages can be long and we don't want to limit the space for them
> artificially. The printing position is not configurable.
> If you're getting the statuses on the left, something is wrong.

sorry I swapped left from right. 
On right edge is to far , is line message is too big should warp it to 80 cols , like before I guess .

Ping what's the current status on this?

In F16, there's now a green OK and red FAILED on the right side of the display. as far as I'm concerned, the issue has been solved.

(In reply to comment #39)
> In F16, there's now a green OK and red FAILED on the right side of the display.
> as far as I'm concerned, the issue has been solved.

However, the applet that used to display those messages in GNOME is still gone

The errors really need to be pushed to the user to be useful

If you want a status icon that can browse system logs, please bring this up witht the GNOME folks.

To me it appears everything requested has been implemnted now. Closing.