Created attachment 994219 [details]
/usr/lib/udev/rules.d/99-hypervkvpd.rules

Created attachment 994220 [details]
/usr/lib/systemd/system/hypervkvpd.service

I've tried to workaround the problem by making the service wait for the hv_util driver to finish loading.

Initially I thought I could do this using Requires=sys-bus-vmbus-drivers-hv_util.device line in hypervkvpd.service but this caused boots to break on VMware systems where the hypervkvpd.service had been enabled (ConditionVirtualization doesn't stop the Requires line taking effect and since the Requires is never satisfied on VMware systems...).

The latest (attached) attempt forcefully loads the service from udev after hv_util is around. This is kinda bad because:

1. The service MUST no longer be installed to multi-user.target or it will load too soon. This kinda creates problems for people who are upgrading as they won't benefit from the fix...
2. There is no way to deactivate the service other than uninstalling the RPM/manually overriding udev rules.
3. The act of installing the RPM would "activate" the service on future boots.

Does anyone have something better? The only other thing I can think of is if service does the modprobe of hv_utils itself (but this makes a mess if the user compiled the module into the kernel).

The root cause of the issue is the fact that kernel/userspace daemon communication for kvp/vss (but not for fcopy) is done via netlink so all sorts of weird issues (packets being lost, racy startup, ...) are possible.

I'm going to send upstream patches to eliminate netlink communication for kvp/vss shortly.

Thanks Vitaly, if you want I can send you the kvp kernel/daemon code that I have for RHEL 5.x based on char device.

Thanks Vitaly, if you want I can send you the kvp kernel/daemon code that I have for RHEL 5.x based on char device.

Should be fixed in 0-0.12.20150702git.fc23 (requires kernel>=4.2-rc0).

Will this also be fixed in Fedora 22?

(In reply to Sitsofe Wheeler from comment #8)
> Will this also be fixed in Fedora 22?

Probably not until kernel-4.2 (which has this new transport) lands in updates.

Vitaly:
Understood. Will the existing hypervkvpd continue to work when a user boots the 4.2 kernel?

(In reply to Sitsofe Wheeler from comment #10)
> Vitaly:
> Understood. Will the existing hypervkvpd continue to work when a user boots
> the 4.2 kernel?

Yes (or at least it was supposed to). Newly introduced transport has an abstraction over both layers (old netlink and newly introduced chardev-based). Daemons upgrade under 4.2 kernel should also work.

Unfortunately this issue is still here with
Fedora 23
kernel-core-4.2.5-300.fc23.x86_64

Looking through the journal only shows the following:
Nov 09 10:28:48 localhost.localdomain systemd[1]: Starting Basic System.
Nov 09 10:28:48 localhost.localdomain systemd[1]: Started Hyper-V KVP daemon.
Nov 09 10:28:48 localhost.localdomain systemd[1]: Started Software RAID monitoring and management.
Nov 09 10:28:48 localhost.localdomain systemd[1]: Started VGAuth Service for open-vm-tools.
Nov 09 10:28:48 localhost.localdomain systemd[1]: Started Service for virtual machines hosted on VMware.
[...]
Nov 09 10:28:48 localhost.localdomain systemd[1]: Starting Update UTMP about System Runlevel Changes...
Nov 09 10:28:48 localhost.localdomain systemd[1]: Started Update UTMP about System Runlevel Changes.
Nov 09 10:28:48 localhost.localdomain systemd[1]: Startup finished in 819ms (kernel) + 900ms (initrd) + 917ms (userspace) = 2.637s.
Nov 09 10:28:48 localhost.localdomain kernel: hv_utils: Registering HyperV Utility Driver
Nov 09 10:28:48 localhost.localdomain kernel: hv_vmbus: registering driver hv_util

So I'm guessing by the time hv_utils showed up it was too late for the KVP daemon.

(How do I reopen this?)

(In reply to Sitsofe Wheeler from comment #12)
> Unfortunately this issue is still here with
> Fedora 23
> kernel-core-4.2.5-300.fc23.x86_64
> 
> Looking through the journal only shows the following:
> Nov 09 10:28:48 localhost.localdomain systemd[1]: Starting Basic System.
> Nov 09 10:28:48 localhost.localdomain systemd[1]: Started Hyper-V KVP daemon.
> Nov 09 10:28:48 localhost.localdomain systemd[1]: Started Software RAID
> monitoring and management.
> Nov 09 10:28:48 localhost.localdomain systemd[1]: Started VGAuth Service for
> open-vm-tools.
> Nov 09 10:28:48 localhost.localdomain systemd[1]: Started Service for
> virtual machines hosted on VMware.
> [...]
> Nov 09 10:28:48 localhost.localdomain systemd[1]: Starting Update UTMP about
> System Runlevel Changes...
> Nov 09 10:28:48 localhost.localdomain systemd[1]: Started Update UTMP about
> System Runlevel Changes.
> Nov 09 10:28:48 localhost.localdomain systemd[1]: Startup finished in 819ms
> (kernel) + 900ms (initrd) + 917ms (userspace) = 2.637s.
> Nov 09 10:28:48 localhost.localdomain kernel: hv_utils: Registering HyperV
> Utility Driver
> Nov 09 10:28:48 localhost.localdomain kernel: hv_vmbus: registering driver
> hv_util
> 
> So I'm guessing by the time hv_utils showed up it was too late for the KVP
> daemon.


Oh, it seems 

ConditionPathExists=/dev/vmbus/hv_fcopy

only prevents this service from starting too early, nobody will start it when this file appears. I think it should be possible to make systemd do magic like that, I'll take a look.

> 
> (How do I reopen this?)

Moved the bug back to ASSIGNED as there is still some work to do here.

Something like this in hypervkvpd.path:

[Unit]
Description=Hyper-V KVP daemon
ConditionVirtualization=microsoft

[Path]
PathExists=/dev/vmbus/hv_kvp

[Install]
WantedBy=multi-user.target

for hypervkvpd.service might have the desired effect (see http://www.freedesktop.org/software/systemd/man/systemd.path.html for details).

Can you please give this scratch build a try?
http://koji.fedoraproject.org/koji/taskinfo?taskID=11803666

I added .path files for systemd so units are supposed to be activated when devices come up. Device disappearance/reappearance (e.g. when guest services are disabled and then enabled back from host) should also be supported. Unfortunately I discovered an ugly kernel bug while testing this:
https://lkml.org/lkml/2015/11/12/197

You wouldn't happen to have a build that targets Fedora 23 with binary packages would you? I don't happen to have an appropriate build system around at the moment to rebuild the srpm...

OK I've just found the fc24 RPM and installed that. Here's what I get:

Nov 13 09:30:59 localhost kernel: hv_vmbus: registering driver hv_storvsc
Nov 13 09:31:00 localhost.localdomain systemd[1]: Started Hyper-V KVP device.
Nov 13 09:31:00 localhost.localdomain systemd[1]: Starting Hyper-V KVP device.
Nov 13 09:31:00 localhost.localdomain systemd[1]: Started Hyper-V KVP daemon.
Nov 13 09:31:00 localhost.localdomain systemd[1]: Starting Hyper-V KVP daemon...
Nov 13 09:31:00 localhost.localdomain KVP[493]: KVP starting; pid is:493
Nov 13 09:31:00 localhost.localdomain KVP[493]: open /dev/vmbus/hv_kvp failed; error: 2 No such file or directory
Nov 13 09:31:00 localhost.localdomain systemd[1]: hypervkvpd.service: Main process exited, code=exited, status=1/FAILURE
Nov 13 09:31:00 localhost.localdomain systemd[1]: hypervkvpd.service: Unit entered failed state.
Nov 13 09:31:00 localhost.localdomain systemd[1]: hypervkvpd.service: Failed with result 'exit-code'.
Nov 13 09:31:00 localhost.localdomain kernel: hv_utils: Registering HyperV Utility Driver
Nov 13 09:31:00 localhost.localdomain kernel: hv_vmbus: registering driver hv_util
Nov 13 09:31:00 localhost.localdomain systemd[1]: Started Hyper-V KVP daemon.
Nov 13 09:31:00 localhost.localdomain systemd[1]: Starting Hyper-V KVP daemon...
Nov 13 09:31:00 localhost.localdomain KVP[514]: KVP starting; pid is:514
Nov 13 09:31:00 localhost.localdomain KVP[514]: KVP LIC Version: 3.1
Nov 13 09:30:22 localhost.localdomain kernel: hv_vmbus: registering driver hv_netvsc
Nov 13 09:30:22 localhost.localdomain kernel: hv_netvsc: hv_netvsc channel opened successfully
Nov 13 09:30:22 localhost.localdomain multipathd[399]: sda: spurious uevent, path already in pathvec
Nov 13 09:30:22 localhost.localdomain kernel: hv_netvsc vmbus_14: Send section size: 6144, Section count:2560
Nov 13 09:30:22 localhost.localdomain kernel: hv_netvsc vmbus_14: Device MAC 00:15:5d:03:b6:32 link state up
Nov 13 09:30:22 localhost.localdomain kernel: hv_netvsc: hv_netvsc channel opened successfully
Nov 13 09:30:22 localhost.localdomain kernel: hv_netvsc vmbus_15: Send section size: 6144, Section count:2560
Nov 13 09:30:22 localhost.localdomain kernel: hv_netvsc vmbus_15: Device MAC 00:15:5d:03:b6:33 link state up
Nov 13 09:30:22 localhost.localdomain systemd[1]: Started Hyper-V KVP daemon.

It seems a bit messy that it starts up and then quits with failure only to be started again later...

I'm wondering if my first suggestion was wrong and it is better to use udev rules to help sort this out: https://lists.fedoraproject.org/pipermail/devel/2012-January/160917.html .

/etc/udev/rules.d/hypervkvpd.rules:
SUBSYSTEM=="misc", KERNEL=="vmbus/hv_kvp", TAG+="systemd"

/etc/systemd/system/hypervkvpd.service.d/override.conf:
[Unit]
Wants=sys-devices-virtual-misc-vmbus\x21hv_kvp.device
After=sys-devices-virtual-misc-vmbus\x21hv_kvp.device

looks awful but it seems to work...

Nov 13 09:56:47 localhost.localdomain kernel: hv_utils: Registering HyperV Utility Driver
Nov 13 09:56:47 localhost.localdomain kernel: hv_vmbus: registering driver hv_util
Nov 13 09:56:47 localhost.localdomain kernel: hv_vmbus: registering driver hv_netvsc
[...]
Nov 13 09:56:10 localhost.localdomain systemd[1]: Found device /sys/devices/virtual/misc/vmbus!hv_kvp.
Nov 13 09:56:10 localhost.localdomain systemd[1]: Started Hyper-V KVP daemon.
Nov 13 09:56:10 localhost.localdomain systemd[1]: Starting Hyper-V KVP daemon...
Nov 13 09:56:10 localhost.localdomain KVP[475]: KVP starting; pid is:475
Nov 13 09:56:10 localhost.localdomain KVP[475]: KVP LIC Version: 3.1

(I've no idea why the clock jumps around as it does when starting up - the host's clock is set correctly and there's no NTP syncing happening...)

The tricky part is the fact that these devices can disappear and reappear again at any time (e.g. try disabling "Data Exchange" in your VM properties and then enabling it back). I'd like to have daemons restarted in such cases, but udev rules don't do that. I played a little bit with other systemd features and it seems adding
ConditionPathExists=/dev/vmbus/hv_kvp
to the unit solves both problems -- we don't have fault startup when the device isn't there and .path file does the job when it comes up.

Here is a scratch build with the change:
http://koji.fedoraproject.org/koji/taskinfo?taskID=11821296

Please be aware that in case you'll be testing enabling/disabling services from your Hyper-V host your guest kernel will probably crash unless you rebuild it with patches from Comment#15.

Thanks!

The latest RPM produces the following:

journalctl -b | grep -i 'hv\|hyper\|KVP'
Nov 14 21:52:16 localhost kernel: hv_vmbus: registering driver hv_storvsc
Nov 14 21:52:17 localhost.localdomain systemd[1]: Started Hyper-V KVP device.
Nov 14 21:52:17 localhost.localdomain systemd[1]: Starting Hyper-V KVP device.
Nov 14 21:52:17 localhost.localdomain systemd[1]: Started Hyper-V KVP daemon.
Nov 14 21:52:17 localhost.localdomain kernel: hv_utils: Registering HyperV Utility Driver
Nov 14 21:52:17 localhost.localdomain kernel: hv_vmbus: registering driver hv_util
Nov 14 21:52:17 localhost.localdomain systemd[1]: Started Hyper-V KVP daemon.
Nov 14 21:52:17 localhost.localdomain systemd[1]: Starting Hyper-V KVP daemon...
Nov 14 21:52:17 localhost.localdomain KVP[478]: KVP starting; pid is:478
Nov 14 21:52:17 localhost.localdomain KVP[478]: KVP LIC Version: 3.1
Nov 14 21:52:17 localhost.localdomain kernel: hv_vmbus: registering driver hv_netvsc
Nov 14 21:52:17 localhost.localdomain kernel: hv_netvsc: hv_netvsc channel opened successfully
[...]
Nov 14 21:52:18 localhost.localdomain systemd[1]: Started Hyper-V KVP daemon.

It's not great seeing the (systemd hypervkvpd) service being started so many (4?) times but at least the KVP itself is only being started once (and I wish that daemon had chosen hypervkvpd as its syslog name) so arguably this is fixed.

One thing though - perhaps it's linked to your kernel patch but while disabling/enabling the "Data Exchange" integration service on the host didn't result in a kernel crash it DID result in the IP address never appearing in the host's networking tab ever again. The journal shows the service being started again but this has no impact - only manually restarting the service makes the IP address reporting work again.

OK try 3 (from http://unix.stackexchange.com/questions/63232/what-is-the-correct-way-to-write-a-udev-rule-to-stop-a-service-under-systemd ):

/etc/udev/rules.d/hypervkvpd.rules:
SUBSYSTEM=="misc", KERNEL=="vmbus/hv_kvp", TAG+="systemd", ENV{SYSTEMD_WANTS}+="hypervkvpd.service"

/etc/systemd/system/hypervkvpd.service.d/override.conf:
[Unit]
BindsTo=sys-devices-virtual-misc-vmbus\x21hv_kvp.device
After=sys-devices-virtual-misc-vmbus\x21hv_kvp.device

(.path file was masked away)

Brief testing shows this appeared to address your disappear and reappear requirement (service was correctly stopped when Data Exchange was disabled and started after Data Exchange was subsequently enabled) a few times before the kernel blew up with an oops.

Thanks,

here is the scratch build with udev rules:
http://koji.fedoraproject.org/koji/taskinfo?taskID=11893296

I'll test it for a while and submit an official build if there are no issues.

Thanks Vitaly - that package seems to work well for me (but I'm only testing the hyperkvpd portion of it).

(In reply to Sitsofe Wheeler from comment #24)
> Thanks Vitaly - that package seems to work well for me (but I'm only testing
> the hyperkvpd portion of it).

Thanks, I briefly tested all of them and everything seemed to work well. I'll be submitting builds for rawhide & f23 shortly.

Vitaly:
Something that has just occurred to me - are people going to get into a mess if they were to install this package on a system with an old (pre 4.2 kernel)? This isn't such an issue for Fedora 23 (as it never came with an older kernel) but might be for others...

(In reply to Sitsofe Wheeler from comment #26)
> Vitaly:
> Something that has just occurred to me - are people going to get into a mess
> if they were to install this package on a system with an old (pre 4.2
> kernel)? This isn't such an issue for Fedora 23 (as it never came with an
> older kernel) but might be for others...

We could have added an explicit 'Requires: kernel >= 4.2' to rpms but I think it would be an overkill:
- there never was an older kernel in F23;
- current hyperv-daemons packages in F23 and rawhide are not able to work with older kernels anyway;
- it is not a bullet proof solution as it is possible to have a newer kernel installed but an older one booted.

In general, people installing packages from newer Fedora on old Fedoras or building their own kernels are on their own: it might and might not work in each particular case...

Vitaly: Sounds reasonable!

hyperv-daemons-0-0.13.20150702git.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2015-0d1a796b7d

hyperv-daemons-0-0.13.20150702git.fc23 has been pushed to the Fedora 23 testing repository. If problems still persist, please make note of it in this bug report.
If you want to test the update, you can install it with
$ su -c 'dnf --enablerepo=updates-testing update hyperv-daemons'
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2015-0d1a796b7d

hyperv-daemons-0-0.13.20150702git.fc23 has been pushed to the Fedora 23 stable repository. If problems still persist, please make note of it in this bug report.