Bug 576439 - Anaconda should be able to write logs to virtio-serial port in KVM
Anaconda should be able to write logs to virtio-serial port in KVM
Status: CLOSED CURRENTRELEASE
Product: Fedora
Classification: Fedora
Component: anaconda (Show other bugs)
14
All Linux
low Severity medium
: ---
: ---
Assigned To: Ales Kozumplik
Fedora Extras Quality Assurance
:
Depends On:
Blocks: 524980
  Show dependency treegraph
 
Reported: 2010-03-23 22:40 EDT by Liam Li
Modified: 2014-09-30 19:38 EDT (History)
9 users (show)

See Also:
Fixed In Version: anaconda-14.13
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2011-08-08 02:09:48 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
the sample output logs got via virtio (47.46 KB, application/x-gzip)
2010-11-14 21:30 EST, MingtaoNiu
no flags Details

  None (edit)
Description Liam Li 2010-03-23 22:40:44 EDT
Description of problem:
we want that anaconda can write logs to virtio-serial port in KVM,If Anaconda can successfully open something like
   /dev/virtio-ports/org.fedoraproject.anaconda.log
then the logs could be routed directly to the host.
On host,we want to get guest's anaconda log in real time during install.

the usage of virtio-serial port,please see:
https://fedoraproject.org/wiki/Features/VirtioSerial
Comment 1 Ales Kozumplik 2010-07-14 09:40:25 EDT
Liam,

the patch has been sent to the anaconda-devel-list for review if you'd like to take a look.

Ales
Comment 2 Ales Kozumplik 2010-07-20 03:32:59 EDT
Fixed by commits
49b7c9c3f21d83b4cdf86d86a5d4bcdf450b9c86
297340b0163295efd696d45a2514853a5b33acd6
7aa27d0ef140d8ac222d619609d7a01f62fb5128
e34ef1aaa720d59175972adb1277401e0b898cf1

Will be in anaconda-14.12.
Comment 3 Ales Kozumplik 2010-07-20 03:35:07 EDT
This is the promised guide about setting the virtio logging up:

https://fedoraproject.org/wiki/Anaconda/Logging#Remote_logging_via_virtio
Comment 4 Amit Shah 2010-07-20 03:46:19 EDT
Thanks, great to see this support.

I think it would be more useful to have the virtio-port name hard-coded in Anaconda, like 'org.fedoraproject.anaconda.install.log', so whenever virt-install installs an F14+ guest, it can create a new virtio-serial port with that name. Also, if Anaconda finds a port with that name available, it can just directly write logs to it.

The current method relies on users to manually make a port available when starting installation as well as tweak the kernel command line.

Automating is easy and also provides logs on the host which will surely help debugging early Anaconda problems, so the utility of this feature isn't just limited to sophisticated users.
Comment 5 Amit Shah 2010-07-20 03:46:49 EDT
Also could we have a F14 feature page on this (and ask the board to approve it since the feature is already in)?
Comment 6 Ales Kozumplik 2010-07-20 05:23:45 EDT
Hi Amit,

great you're involved on this.

(In reply to comment #4)
> Thanks, great to see this support.
> 
> I think it would be more useful to have the virtio-port name hard-coded in
> Anaconda, like 'org.fedoraproject.anaconda.install.log', so whenever
> virt-install installs an F14+ guest, it can create a new virtio-serial port
> with that name.

Of course, I can just add the auto detection mechanism once the virt-install change is in. Or do you guys think it would be better to have it in right now and get rid of the command line parameter? Liam? James?

By the way Amit, can one specify the virtio chardev parameters directly to virt-install? Or even use virt-manager?

Ales
Comment 7 Ales Kozumplik 2010-07-20 05:25:35 EDT
(In reply to comment #5)
> Also could we have a F14 feature page on this (and ask the board to approve it
> since the feature is already in)?    

I thought about it but I don't think it qualifies due to 
https://fedoraproject.org/wiki/Features/Policy/Is_this_a_feature

Certainly not requiring intervention and certainly not interesting to lay press. Perhaps slightly to the admins. I'll try to bring this up on the Freenode #anaconda channel this afternoon.
Comment 8 Amit Shah 2010-07-20 05:30:31 EDT
(In reply to comment #7)
> I thought about it but I don't think it qualifies due to 
> https://fedoraproject.org/wiki/Features/Policy/Is_this_a_feature
> 
> Certainly not requiring intervention and certainly not interesting to lay
> press. Perhaps slightly to the admins. I'll try to bring this up on the
> Freenode #anaconda channel this afternoon.    

It's a feature in the sense other distros don't have it yet :-)

It's a useful thing to do and tell others about, IMO.
Comment 9 Amit Shah 2010-07-20 05:34:07 EDT
(In reply to comment #6)
> > I think it would be more useful to have the virtio-port name hard-coded in
> > Anaconda, like 'org.fedoraproject.anaconda.install.log', so whenever
> > virt-install installs an F14+ guest, it can create a new virtio-serial port
> > with that name.
> 
> Of course, I can just add the auto detection mechanism once the virt-install
> change is in. Or do you guys think it would be better to have it in right now
> and get rid of the command line parameter? Liam? James?

I believe we should do it right away and then tell the libvirt folks the name Anaconda expects during installation. Also, since we already have this support in, even if libvirt can't add this change for F14, a libvirt from F15 can still install F14 guests and get the logs out to the host.

> By the way Amit, can one specify the virtio chardev parameters directly to
> virt-install? Or even use virt-manager?

I think libvirt just uses unix domain sockets for everything and so it's just a filename, but I believe parameters can be set, asking danpb for more info.
Comment 10 Daniel Berrange 2010-07-20 06:20:02 EDT
libvirt can configure character devices to point at anything supported by QEMU. 
A solution that requires running socat in the host manually though is not really a viable option for virt-manager. We need a solution that can be expressed purely in terms of the libvirt domain XML without having to run any extra processes on the side. Would it be possible to just point the QEMU proess directly at a syslog server listen on UDP, or does it require some special protocol to send syslog data over udp ?

Such an example would be:

  <devices>
    <channel type="udp">
      <source mode="connect" host="u.x.y.z" service="514"/>
      <target type='virtio' name='org.linux-kvm.port.foo'/>
    </channel>
  </devices>

In virt-manager we could configure what remote syslog service to send data to, and add this to any new enough Fedora guest.

Further examples are:

http://libvirt.org/formatdomain.html#elementsConsole
Comment 11 Liam Li 2010-07-20 06:29:36 EDT
Great, I just read the guide for virtio logging, very easy to use. Thanks Alex.

Auto detection mechanism to forward logs is good.But I have two questions:
1. Where to describe this Auto detection mechanism, create a new wiki to tell
people to use this anaconda arg? Just like the VNC, text, cmdline args, tell
anaconda via grub command line, anaconda will do as we expect. So the same as
virtio logging,use "virtiolog=syslog_forward_port" to tell anaconda that we
want it to write logs to virtio. I think this may be more consistent with other
options in http://fedoraproject.org/wiki/Anaconda/Options

2. If the previous used VM has virtio device,but the virtio name is not
anaconda expected, in this case, we have to recreate a VM(with complex args and
expected virtio name) if we want to get logs from virtio?
Comment 12 Amit Shah 2010-07-20 06:40:16 EDT
(In reply to comment #10)
> libvirt can configure character devices to point at anything supported by QEMU. 
> A solution that requires running socat in the host manually though is not
> really a viable option for virt-manager. We need a solution that can be
> expressed purely in terms of the libvirt domain XML without having to run any
> extra processes on the side. Would it be possible to just point the QEMU proess
> directly at a syslog server listen on UDP, or does it require some special
> protocol to send syslog data over udp ?

Sending directly should work. If a syslog server isn't configured, saving the logs to a local file (/var/log/libvirt/$VM/) should be fine.

> In virt-manager we could configure what remote syslog service to send data to,
> and add this to any new enough Fedora guest.

Cool, can this be done for F14?

> Further examples are:
> 
> http://libvirt.org/formatdomain.html#elementsConsole    

Thanks.
Comment 13 Daniel Berrange 2010-07-20 06:45:17 EDT
File a BZ against python-virtinst describing the necessary additions & Cole can evaluate whether its feasible.
Comment 14 Ales Kozumplik 2010-07-20 07:29:08 EDT
(In reply to comment #10)
>   <devices>
>     <channel type="udp">
>       <source mode="connect" host="u.x.y.z" service="514"/>
>       <target type='virtio' name='org.linux-kvm.port.foo'/>
>     </channel>
>   </devices>
> 
> 
> http://libvirt.org/formatdomain.html#elementsConsole    

Wow, I wouldnt do the socat trick if I knew there were options like this. Can I do TCP too? That would simplify things even more.
Comment 15 Daniel Berrange 2010-07-20 08:04:47 EDT
See that link I posted for details of TCP syntax.
Comment 16 Ales Kozumplik 2010-07-20 08:09:21 EDT
Daniel,

I added the following XML branch to my qemu libvirt machine:

    <channel type="tcp">
      <source mode="bind" host="127.0.0.1" service="6080"/>
      <target type='virtio' name='org.linux-kvm.port.foo'/>
    </channel>

Then restarted the virt-manager and then started the machine, but there is no port in /dev/virtio-ports and doing 'ss -a' reveals thre's nothing listening on port 6080. Is there a problem with the entry or am I trying to achieve something that is not possible?

Thanks.
Ales
Comment 17 Daniel Berrange 2010-07-20 08:19:10 EDT
Are you sure you added it to the right VM ? Also don't edit XML directly, use virsh edit $GUESTNAME, otherwise changes won't be loaded into libvirtd. virsh dumpxml should show the change if it was succesful
Comment 18 Ales Kozumplik 2010-07-20 08:24:12 EDT
Thanks, that was the problem. Now I'm getting:

Traceback (most recent call last):
  File "/usr/share/virt-manager/virtManager/engine.py", line 799, in run_domain
    vm.startup()
  File "/usr/share/virt-manager/virtManager/domain.py", line 1256, in startup
    self._backend.create()
  File "/usr/lib/python2.6/site-packages/libvirt.py", line 317, in create
    if ret == -1: raise libvirtError ('virDomainCreate() failed', dom=self)
libvirtError: internal error Process exited while reading console log output: char device redirected to /dev/pts/2
Error initializing device virtio-serial-pci

BTW are you on IRC somewhere?
Comment 19 Ales Kozumplik 2010-07-21 04:47:45 EDT
Amit, James, Liam,

I just sent a new patch to the anaconda devel list: it removes the virtiolog= parameter and autodetects the virtio port 

org.fedoraproject.anaconda.log.0

instead.

Wiki update will follow.
Comment 20 Ales Kozumplik 2010-07-26 04:40:33 EDT
Change pushed, including forwarding of anaconda traceback dumps and X server logs:

6977f0838e6d95ddfacdf0c284100ed0739cb10c
6127ada1fb285138794c6394af0ffa0470373df8
900916375dc97f6545494e18e5aecf4781d62ec8
Comment 21 Bug Zapper 2010-07-30 07:11:33 EDT
This bug appears to have been reported against 'rawhide' during the Fedora 14 development cycle.
Changing version to '14'.

More information and reason for this action is here:
http://fedoraproject.org/wiki/BugZappers/HouseKeeping
Comment 22 Liam Li 2010-08-04 00:03:59 EDT
I tested on anaconda 14.14, it works!
Comment 23 Liam Li 2010-08-04 00:06:30 EDT
but have to enter syslog=1 to enable it
Comment 24 Liam Li 2010-08-04 04:48:03 EDT
I am very excited that it works, and did get three logs in some directory. I tested on anaconda 14.14, but got two issues:
1. If reboot the guest, or restart install, have to kill rsyslog and restart rsyslog with: eval `scripts/analog -p 6080 ./ -o rsyslogd.conf -s`, does this by design?

2. I only can get first half of the anaconda.log, the logs after line "INFO anaconda: Starting graphical installation." can not get.
Comment 25 Ales Kozumplik 2010-08-06 12:59:59 EDT
> 1. If reboot the guest, or restart install, have to kill rsyslog and restart
> rsyslog with: eval `scripts/analog -p 6080 ./ -o rsyslogd.conf -s`, does this
> by design?

Is the logging not working at all when you skip this step?

> 
> 2. I only can get first half of the anaconda.log, the logs after line "INFO
> anaconda: Starting graphical installation." can not get.    

That's because nothing is receiving the remote syslog forwarding that you had to turn on with "syslog=1". I have a patch ready to fix the need for this.
Comment 26 MingtaoNiu 2010-11-14 21:27:57 EST
Hi all,
I succeeded in getting logs via virtio by following the steps below:
===============================================================
1. Get the anaconda repo

        $  git clone git://git.fedorahosted.org/git/anaconda.gitanaconda
        $ cd anaconda

    2. add the virtio-serial port to your virtual machine, direct it to the TCP port 6080 on the host:

	$ virsh edit <machine name>

        where you add following into <devices>:

      <channel type='tcp'>
        <source mode='connect' host='127.0.0.1' service='6080'/>
        <target type='virtio' name='org.fedoraproject.anaconda.log.0'/>
      </channel>


   3.start the listening rsyslogd process on the host, using the analog script

        $ eval `scripts/analog -p 6080 ./ -o rsyslogd.conf -s`

   4. start the virtual machine:

       $ virsh start <machine name>

   5. continue with the installation. Immediately after the Anaconda greeting is displayed the log messages will start appearing in the directory given to analog, in the 127.0.0.1 subdirectory.
===============================================================

All steps are already on wiki page "https://fedoraproject.org/wiki/Anaconda/Logging#Remote_logging_via_virtio" except step 1.
And the sample output is in the attachment.

BTW, James and I both have a question about step 3. Why do we use 'eval'?  Why is it needed?  Why not just run `scripts/analog -o rsyslogd.conf -s ./`?

Thanks,
Newgle
Comment 27 MingtaoNiu 2010-11-14 21:30:52 EST
Created attachment 460452 [details]
the sample output logs got via virtio

sample output logs got via virtio
Comment 28 Ales Kozumplik 2010-11-15 03:55:29 EST
(In reply to comment #26)
> Hi all,
> I succeeded in getting logs via virtio by following the steps below:
> ===============================================================
> 1. Get the anaconda repo
> 
>         $  git clone git://git.fedorahosted.org/git/anaconda.gitanaconda
>         $ cd anaconda

Just a note: if you do this only to get the 'analog' script in F14 you can just install the anaconda rpm with yum and you get the script in /usr/libexec (I know, I'm going to migrate it somewhere more suitable later).

Ales
Comment 29 James Laska 2010-11-17 16:06:38 EST
(In reply to comment #26)
> All steps are already on wiki page
> "https://fedoraproject.org/wiki/Anaconda/Logging#Remote_logging_via_virtio"
> except step 1.
> And the sample output is in the attachment.
> 
> BTW, James and I both have a question about step 3. Why do we use 'eval'?  Why
> is it needed?  Why not just run `scripts/analog -o rsyslogd.conf -s ./`?

Ales, I'm still confused by the eval line used to start rsyslogd.  Can you explain that in a bit more detail?  Why the eval stmt?
Comment 30 Ales Kozumplik 2010-11-18 02:35:34 EST
(In reply to comment #29)
> Ales, I'm still confused by the eval line used to start rsyslogd.  Can you
> explain that in a bit more detail?  Why the eval stmt?

This is because analog by default does not start the local rsyslog daemon that collects the log messages from the (remote) installation. Instead it just prints a command line with the exact set of parameters (rsyslogd, config file generated by analog, pid file) to do that. We discussed this with msivak and he suggested this could be a good practice, i.e. analog tells you how to start rsyslog but it's up to you to do that.

Would you prefer having a parameter in analog that would start rsyslogd directly? (Thinking about it now I have to admit it would make sense).
Comment 32 Hongqing Yang 2011-01-24 08:51:20 EST
Please create the installation media for the F-15 Pre Alpha Rawhide Acceptance Test. The plan schedule is from 20 Jan 2011 to 28 Jan 2011, testing time included.
Comment 33 Hongqing Yang 2011-01-24 08:52:14 EST
(In reply to comment #32)
> Please create the installation media for the F-15 Pre Alpha Rawhide Acceptance
> Test. The plan schedule is from 20 Jan 2011 to 28 Jan 2011, testing time
> included.

sorry, reply the wrong place
Comment 34 Hongqing Yang 2011-08-08 01:09:43 EDT
This has been solved since F15, can any one close this?
Comment 35 Ales Kozumplik 2011-08-08 02:09:48 EDT
Closing per comment 34.

Note You need to log in before you can comment on or make changes to this bug.