Bug 576439

Liam,

the patch has been sent to the anaconda-devel-list for review if you'd like to take a look.

Ales

Fixed by commits
49b7c9c3f21d83b4cdf86d86a5d4bcdf450b9c86
297340b0163295efd696d45a2514853a5b33acd6
7aa27d0ef140d8ac222d619609d7a01f62fb5128
e34ef1aaa720d59175972adb1277401e0b898cf1

Will be in anaconda-14.12.

This is the promised guide about setting the virtio logging up:

https://fedoraproject.org/wiki/Anaconda/Logging#Remote_logging_via_virtio

Thanks, great to see this support.

I think it would be more useful to have the virtio-port name hard-coded in Anaconda, like 'org.fedoraproject.anaconda.install.log', so whenever virt-install installs an F14+ guest, it can create a new virtio-serial port with that name. Also, if Anaconda finds a port with that name available, it can just directly write logs to it.

The current method relies on users to manually make a port available when starting installation as well as tweak the kernel command line.

Automating is easy and also provides logs on the host which will surely help debugging early Anaconda problems, so the utility of this feature isn't just limited to sophisticated users.

Also could we have a F14 feature page on this (and ask the board to approve it since the feature is already in)?

Hi Amit,

great you're involved on this.

(In reply to comment #4)
> Thanks, great to see this support.
> 
> I think it would be more useful to have the virtio-port name hard-coded in
> Anaconda, like 'org.fedoraproject.anaconda.install.log', so whenever
> virt-install installs an F14+ guest, it can create a new virtio-serial port
> with that name.

Of course, I can just add the auto detection mechanism once the virt-install change is in. Or do you guys think it would be better to have it in right now and get rid of the command line parameter? Liam? James?

By the way Amit, can one specify the virtio chardev parameters directly to virt-install? Or even use virt-manager?

Ales

(In reply to comment #5)
> Also could we have a F14 feature page on this (and ask the board to approve it
> since the feature is already in)?    

I thought about it but I don't think it qualifies due to 
https://fedoraproject.org/wiki/Features/Policy/Is_this_a_feature

Certainly not requiring intervention and certainly not interesting to lay press. Perhaps slightly to the admins. I'll try to bring this up on the Freenode #anaconda channel this afternoon.

(In reply to comment #7)
> I thought about it but I don't think it qualifies due to 
> https://fedoraproject.org/wiki/Features/Policy/Is_this_a_feature
> 
> Certainly not requiring intervention and certainly not interesting to lay
> press. Perhaps slightly to the admins. I'll try to bring this up on the
> Freenode #anaconda channel this afternoon.    

It's a feature in the sense other distros don't have it yet :-)

It's a useful thing to do and tell others about, IMO.

(In reply to comment #6)
> > I think it would be more useful to have the virtio-port name hard-coded in
> > Anaconda, like 'org.fedoraproject.anaconda.install.log', so whenever
> > virt-install installs an F14+ guest, it can create a new virtio-serial port
> > with that name.
> 
> Of course, I can just add the auto detection mechanism once the virt-install
> change is in. Or do you guys think it would be better to have it in right now
> and get rid of the command line parameter? Liam? James?

I believe we should do it right away and then tell the libvirt folks the name Anaconda expects during installation. Also, since we already have this support in, even if libvirt can't add this change for F14, a libvirt from F15 can still install F14 guests and get the logs out to the host.

> By the way Amit, can one specify the virtio chardev parameters directly to
> virt-install? Or even use virt-manager?

I think libvirt just uses unix domain sockets for everything and so it's just a filename, but I believe parameters can be set, asking danpb for more info.

libvirt can configure character devices to point at anything supported by QEMU. 
A solution that requires running socat in the host manually though is not really a viable option for virt-manager. We need a solution that can be expressed purely in terms of the libvirt domain XML without having to run any extra processes on the side. Would it be possible to just point the QEMU proess directly at a syslog server listen on UDP, or does it require some special protocol to send syslog data over udp ?

Such an example would be:

  <devices>
    <channel type="udp">
      <source mode="connect" host="u.x.y.z" service="514"/>
      <target type='virtio' name='org.linux-kvm.port.foo'/>
    </channel>
  </devices>

In virt-manager we could configure what remote syslog service to send data to, and add this to any new enough Fedora guest.

Further examples are:

http://libvirt.org/formatdomain.html#elementsConsole

Great, I just read the guide for virtio logging, very easy to use. Thanks Alex.

Auto detection mechanism to forward logs is good.But I have two questions:
1. Where to describe this Auto detection mechanism, create a new wiki to tell
people to use this anaconda arg? Just like the VNC, text, cmdline args, tell
anaconda via grub command line, anaconda will do as we expect. So the same as
virtio logging,use "virtiolog=syslog_forward_port" to tell anaconda that we
want it to write logs to virtio. I think this may be more consistent with other
options in http://fedoraproject.org/wiki/Anaconda/Options

2. If the previous used VM has virtio device,but the virtio name is not
anaconda expected, in this case, we have to recreate a VM(with complex args and
expected virtio name) if we want to get logs from virtio?

(In reply to comment #10)
> libvirt can configure character devices to point at anything supported by QEMU. 
> A solution that requires running socat in the host manually though is not
> really a viable option for virt-manager. We need a solution that can be
> expressed purely in terms of the libvirt domain XML without having to run any
> extra processes on the side. Would it be possible to just point the QEMU proess
> directly at a syslog server listen on UDP, or does it require some special
> protocol to send syslog data over udp ?

Sending directly should work. If a syslog server isn't configured, saving the logs to a local file (/var/log/libvirt/$VM/) should be fine.

> In virt-manager we could configure what remote syslog service to send data to,
> and add this to any new enough Fedora guest.

Cool, can this be done for F14?

> Further examples are:
> 
> http://libvirt.org/formatdomain.html#elementsConsole    

Thanks.

File a BZ against python-virtinst describing the necessary additions & Cole can evaluate whether its feasible.

(In reply to comment #10)
>   <devices>
>     <channel type="udp">
>       <source mode="connect" host="u.x.y.z" service="514"/>
>       <target type='virtio' name='org.linux-kvm.port.foo'/>
>     </channel>
>   </devices>
> 
> 
> http://libvirt.org/formatdomain.html#elementsConsole    

Wow, I wouldnt do the socat trick if I knew there were options like this. Can I do TCP too? That would simplify things even more.

See that link I posted for details of TCP syntax.

Daniel,

I added the following XML branch to my qemu libvirt machine:

    <channel type="tcp">
      <source mode="bind" host="127.0.0.1" service="6080"/>
      <target type='virtio' name='org.linux-kvm.port.foo'/>
    </channel>

Then restarted the virt-manager and then started the machine, but there is no port in /dev/virtio-ports and doing 'ss -a' reveals thre's nothing listening on port 6080. Is there a problem with the entry or am I trying to achieve something that is not possible?

Thanks.
Ales

Are you sure you added it to the right VM ? Also don't edit XML directly, use virsh edit $GUESTNAME, otherwise changes won't be loaded into libvirtd. virsh dumpxml should show the change if it was succesful

Thanks, that was the problem. Now I'm getting:

Traceback (most recent call last):
  File "/usr/share/virt-manager/virtManager/engine.py", line 799, in run_domain
    vm.startup()
  File "/usr/share/virt-manager/virtManager/domain.py", line 1256, in startup
    self._backend.create()
  File "/usr/lib/python2.6/site-packages/libvirt.py", line 317, in create
    if ret == -1: raise libvirtError ('virDomainCreate() failed', dom=self)
libvirtError: internal error Process exited while reading console log output: char device redirected to /dev/pts/2
Error initializing device virtio-serial-pci

BTW are you on IRC somewhere?

Amit, James, Liam,

I just sent a new patch to the anaconda devel list: it removes the virtiolog= parameter and autodetects the virtio port 

org.fedoraproject.anaconda.log.0

instead.

Wiki update will follow.

Change pushed, including forwarding of anaconda traceback dumps and X server logs:

6977f0838e6d95ddfacdf0c284100ed0739cb10c
6127ada1fb285138794c6394af0ffa0470373df8
900916375dc97f6545494e18e5aecf4781d62ec8

This bug appears to have been reported against 'rawhide' during the Fedora 14 development cycle.
Changing version to '14'.

More information and reason for this action is here:
http://fedoraproject.org/wiki/BugZappers/HouseKeeping

I tested on anaconda 14.14, it works!

but have to enter syslog=1 to enable it

I am very excited that it works, and did get three logs in some directory. I tested on anaconda 14.14, but got two issues:
1. If reboot the guest, or restart install, have to kill rsyslog and restart rsyslog with: eval `scripts/analog -p 6080 ./ -o rsyslogd.conf -s`, does this by design?

2. I only can get first half of the anaconda.log, the logs after line "INFO anaconda: Starting graphical installation." can not get.

> 1. If reboot the guest, or restart install, have to kill rsyslog and restart
> rsyslog with: eval `scripts/analog -p 6080 ./ -o rsyslogd.conf -s`, does this
> by design?

Is the logging not working at all when you skip this step?

> 
> 2. I only can get first half of the anaconda.log, the logs after line "INFO
> anaconda: Starting graphical installation." can not get.    

That's because nothing is receiving the remote syslog forwarding that you had to turn on with "syslog=1". I have a patch ready to fix the need for this.

Hi all,
I succeeded in getting logs via virtio by following the steps below:
===============================================================
1. Get the anaconda repo

        $  git clone git://git.fedorahosted.org/git/anaconda.gitanaconda
        $ cd anaconda

    2. add the virtio-serial port to your virtual machine, direct it to the TCP port 6080 on the host:

	$ virsh edit <machine name>

        where you add following into <devices>:

      <channel type='tcp'>
        <source mode='connect' host='127.0.0.1' service='6080'/>
        <target type='virtio' name='org.fedoraproject.anaconda.log.0'/>
      </channel>


   3.start the listening rsyslogd process on the host, using the analog script

        $ eval `scripts/analog -p 6080 ./ -o rsyslogd.conf -s`

   4. start the virtual machine:

       $ virsh start <machine name>

   5. continue with the installation. Immediately after the Anaconda greeting is displayed the log messages will start appearing in the directory given to analog, in the 127.0.0.1 subdirectory.
===============================================================

All steps are already on wiki page "https://fedoraproject.org/wiki/Anaconda/Logging#Remote_logging_via_virtio" except step 1.
And the sample output is in the attachment.

BTW, James and I both have a question about step 3. Why do we use 'eval'?  Why is it needed?  Why not just run `scripts/analog -o rsyslogd.conf -s ./`?

Thanks,
Newgle

Created attachment 460452 [details]
the sample output logs got via virtio

sample output logs got via virtio

(In reply to comment #26)
> Hi all,
> I succeeded in getting logs via virtio by following the steps below:
> ===============================================================
> 1. Get the anaconda repo
> 
>         $  git clone git://git.fedorahosted.org/git/anaconda.gitanaconda
>         $ cd anaconda

Just a note: if you do this only to get the 'analog' script in F14 you can just install the anaconda rpm with yum and you get the script in /usr/libexec (I know, I'm going to migrate it somewhere more suitable later).

Ales

(In reply to comment #26)
> All steps are already on wiki page
> "https://fedoraproject.org/wiki/Anaconda/Logging#Remote_logging_via_virtio"
> except step 1.
> And the sample output is in the attachment.
> 
> BTW, James and I both have a question about step 3. Why do we use 'eval'?  Why
> is it needed?  Why not just run `scripts/analog -o rsyslogd.conf -s ./`?

Ales, I'm still confused by the eval line used to start rsyslogd.  Can you explain that in a bit more detail?  Why the eval stmt?

(In reply to comment #29)
> Ales, I'm still confused by the eval line used to start rsyslogd.  Can you
> explain that in a bit more detail?  Why the eval stmt?

This is because analog by default does not start the local rsyslog daemon that collects the log messages from the (remote) installation. Instead it just prints a command line with the exact set of parameters (rsyslogd, config file generated by analog, pid file) to do that. We discussed this with msivak and he suggested this could be a good practice, i.e. analog tells you how to start rsyslog but it's up to you to do that.

Would you prefer having a parameter in analog that would start rsyslogd directly? (Thinking about it now I have to admit it would make sense).

Please create the installation media for the F-15 Pre Alpha Rawhide Acceptance Test. The plan schedule is from 20 Jan 2011 to 28 Jan 2011, testing time included.

(In reply to comment #32)
> Please create the installation media for the F-15 Pre Alpha Rawhide Acceptance
> Test. The plan schedule is from 20 Jan 2011 to 28 Jan 2011, testing time
> included.

sorry, reply the wrong place

This has been solved since F15, can any one close this?

Closing per comment 34.