Bug 1062867
| Summary: | virt/install fails to submit logs and proceed with next task | ||
|---|---|---|---|
| Product: | [Retired] Beaker | Reporter: | Jan Stancek <jstancek> |
| Component: | beah | Assignee: | Dan Callaghan <dcallagh> |
| Status: | CLOSED DUPLICATE | QA Contact: | tools-bugs <tools-bugs> |
| Severity: | urgent | Docs Contact: | |
| Priority: | urgent | ||
| Version: | 0.15 | CC: | aigao, asaha, bpeck, dcallagh, jburke, llim, pbunyan, qwan, rmancy, skrishna |
| Target Milestone: | --- | ||
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | Bug Fix | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2014-02-17 03:02:26 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
|
Description
Jan Stancek
2014-02-08 10:04:56 UTC
(In reply to Jan Stancek from comment #0) > If I start guests manually with "virsh start", then sometimes they hit same > issue. Console log reports "Submitting logs", but no logs get submitted to > beaker and tasks eventually hit external watchdog. If I log to guest via ssh when this happens, all harness processes are running (with no child processes). If I run "systemctl restart beah-beaker-backend" logs get submitted to beaker immediately and guest continues with next task. A wild guess is the option: net.ipv6.conf.all.forwarding being turned off on the host. In addition to attempting to fix this directly, we will also add the ability to opt in to using older versions of the harness (see #1063090) (In reply to Jan Stancek from comment #4) > It looks like guests are not getting any response from LC. For example, set > filter to tcp.port==8000 and look at frame 130254. Both guests sent SYN to > LC, but there appears to be no response. I think this is actually from the logguestconsoles service created by /distribution/virt/install. I don't think it's related. From what I can see, when the task ends beah just sits there waiting for... nothing. There are no open connections to the LC waiting for anything. (In reply to Dan Callaghan from comment #7) Scratch that, the problem is definitely that IPv6 connectivity goes bad on the host. I can see beah talking to the LC over IPv6 at the start of the recipe, but once /distribution/virt/install runs IPv6 packets suddenly get dropped on the floor. It looks like the default IPv6 routes are missing from the routing table. Restarting beah-beaker-backend works, because it notices that IPv6 connections are timing out so it falls back to IPv4. Restarting the network service also works because it fixes up the IPv6 routing table to have default routes again. Okay, I don't think it's just the routing entries. I noticed that "service network restart" fixes IPv6 connectivity to the LC, and if I keep packets flowing to the LC (using ping6 left running for example) it will keep working for several hours. But if I leave it alone for about 60 seconds or more, IPv6 packets to the LC suddenly start dropping on the floor again. So I think there must be something going wrong with neighbour discovery/autoconfiguration, but I don't fully understand how that stuff works so I can't figure out what's going wrong. We only seem to hit this problem with /distribution/virt/install, and one of the things it does is disable NetworkManager and enable the network initscript instead. So the problem might be due to some difference between those two. Dropping this from Beaker's targets, as it appears to be failing due to a genuine issue in the kernel. We'll mark it as CLOSED/DUPLICATE once there's an appropriate BZ to reference. *** This bug has been marked as a duplicate of bug 1065257 *** |