Bug 2379916

I think the first step is to enable debugging[1] and collect the complete
log so we know what's actually going on.

[1] https://libguestfs.org/guestfs-faq.1.html#how-do-i-debug-when-using-any-libguestfs-program-or-tool-eg.-virt-customize-or-virt-df

Here are the logs I'm getting after enabling the debug:
GuestFish:Getting guestfish/libguestFS version:Error> 
libguestfs: trace: set_verbose true
libguestfs: trace: set_verbose = 0
libguestfs: trace: set_tmpdir "/libguestfs/tmp"
libguestfs: trace: set_tmpdir = 0
libguestfs: trace: set_cachedir "/libguestfs/tmp"
libguestfs: trace: set_cachedir = 0
libguestfs: trace: set_path "/libguestfs/appliance"
libguestfs: trace: set_path = 0
libguestfs: create: flags = 0, handle = 0xec55b40, program = guestfish
libguestfs: trace: shutdown
libguestfs: trace: shutdown = 0
libguestfs: trace: close
libguestfs: closing guestfs handle 0xec55b40 (state 0)


GuestFish:Exiting:Error> 
libguestfs: trace: set_verbose true
libguestfs: trace: set_verbose = 0
libguestfs: trace: set_tmpdir "/libguestfs/tmp"
libguestfs: trace: set_tmpdir = 0
libguestfs: trace: set_cachedir "/libguestfs/tmp"
libguestfs: trace: set_cachedir = 0
libguestfs: trace: set_path "/libguestfs/appliance"
libguestfs: trace: set_path = 0
libguestfs: create: flags = 0, handle = 0x3cceab40, program = guestfish
libguestfs: trace: shutdown
libguestfs: trace: shutdown = 0
libguestfs: trace: close
libguestfs: closing guestfs handle 0x3cceab40 (state 0)

That's definitely not all of the logs.  Set the environment variables as
described, and then copy *all* of the output generated.

We did set the environment variable and those are the only logs we are getting. After setting those environment variables, when the above debug logs print, out container crashes, so no more logs getting emmitted.
Here is the portion of DockerFile we use as part of our container to build and libguestfs and set the env variable.

FROM mcr.microsoft.com/mirror/docker/library/fedora:40

ARG PAT_TOKEN

# package dependencies
RUN dnf install dnf-plugins-core -y \
 && dnf builddep libguestfs -y \
 && dnf updateinfo \
 && dnf upgrade --refresh -y \
 && dnf install -y \
    autoconf \
    automake \
    libtool \
    gettext-devel\
    git \
    zstd \
    nginx \
    python3-pip \
    wget \
    supermin \
    koji \
    mono-devel \
    json-c-devel

# Download kernel 6.14 for Fedora 40. Reference: https://docs.fedoraproject.org/en-US/quick-docs/kernel/installing-koji-kernel/
RUN cd $(mktemp -d) \
  && koji download-build --arch=x86_64 --arch=noarch kernel-6.14.6-100.fc40 \
  && dnf install -y *

# patched libguestfs
WORKDIR /
RUN git clone -b chipate/Merge1.56 --single-branch --depth 1 https://github.com/Azure/libguestfs
WORKDIR /libguestfs
RUN git submodule update --init
RUN autoreconf -i
RUN ./configure
RUN make

# Enable Debug for LibGuestFS
ENV LIBGUESTFS_TRACE=1
ENV LIBGUESTFS_DEBUG=1

RUN make -C appliance clean-supermin-appliance

I don't know why, but only a tiny fraction of messages are being logged, so
you've got to solve that so you can get the complete output.

Maybe don't use a container and just run everything on the host as normal?

(In reply to Richard W.M. Jones from comment #5)
> I don't know why, but only a tiny fraction of messages are being logged, so
> you've got to solve that so you can get the complete output.
> 
> Maybe don't use a container and just run everything on the host as normal?

Hi Richard, I work with Chintan and I tried running the commands on a Fedora 40 VM without using a container, but the debug and tracelogs don't give much info. Attaching the disk and launching guestfish work as normal, but once I run the inspect-os command it just hangs endlessly without any response:

[azureuser@testFedora40 libguestfs]$ ./run guestfish --remote inspect-os
libguestfs: trace: set_verbose true
libguestfs: trace: set_verbose = 0
libguestfs: trace: set_tmpdir "/home/azureuser/Azure-DiskInspect-Service/src/pyServer/libguestfs/tmp"
libguestfs: trace: set_tmpdir = 0
libguestfs: trace: set_cachedir "/home/azureuser/Azure-DiskInspect-Service/src/pyServer/libguestfs/tmp"
libguestfs: trace: set_cachedir = 0
libguestfs: trace: set_path "/home/azureuser/Azure-DiskInspect-Service/src/pyServer/libguestfs/appliance"
libguestfs: trace: set_path = 0
libguestfs: create: flags = 0, handle = 0x115fe60, program = guestfish
libguestfs: trace: set_pgroup true
libguestfs: trace: set_pgroup = 0

If I download the vhd file and use that to attach the disk I don't see this issue and inspect-os works fine, but when I attach the disk using a SAS URL then inspect-os doesn't work. I tried the libguestfs_test_tool and it doesn't seem to report any issues. I will attach the test result to this.

Created attachment 2097918 [details]
libguestfs_test_tool results

Hi Richard, I did some more testing and found that I am sometimes able to get a response with inspect-os command, but the latency is significantly higher than before when we were running this in Fedora 39. It's not as consistent either as sometimes the command will finish in under a minute, while other times it will not complete for a few minutes. We didn't notice these latency issues when running on Fedora 39 though, do you know why the upgrade to Fedora 40 would cause these issues we're seeing? One thing to note is the command runs fast when I attach the disk directly with a vhd, this issue is isolated to disks which are attached via blob SAS URL.

(In reply to armanamjad from comment #6)
> [azureuser@testFedora40 libguestfs]$ ./run guestfish --remote inspect-os

The problem here is that you're enabling debugging for the guestfish --remote
command, not for libguestfs (running remotely).

(In reply to armanamjad from comment #7)
> Created attachment 2097918 [details]
> libguestfs_test_tool results

This is missing almost all of the debug messages for some reason.

I'm still cannot help unless you're able to produce full, unedited debug messages.
Would suggest removing all use of containers and use of --remote, and isolate it
down to a simple test case, then enable debugging and collect all of the
messages (both stdout & stderr).

(In reply to armanamjad from comment #8)
> Hi Richard, I did some more testing and found that I am sometimes able to
> get a response with inspect-os command, but the latency is significantly
> higher than before when we were running this in Fedora 39. It's not as
> consistent either as sometimes the command will finish in under a minute,
> while other times it will not complete for a few minutes. We didn't notice
> these latency issues when running on Fedora 39 though, do you know why the
> upgrade to Fedora 40 would cause these issues we're seeing? One thing to
> note is the command runs fast when I attach the disk directly with a vhd,
> this issue is isolated to disks which are attached via blob SAS URL.

No idea - I need the debugging to tell anything about failures or performance
regressions.

Hm, I tried removing --remote and this is the output I got:
><fs> inspect-os
libguestfs: trace: inspect_os
inspect_os: collect_coreos_inspection_info
inspect_os: check_for_duplicated_bsd_root
inspect_os: check_for_duplicated_btrfs_snapshots_of_root
inspect_os: collect_linux_inspection_info
libguestfs: trace: inspect_os = ["btrfsvol:/dev/sda3/root"]
btrfsvol:/dev/sda3/root

Are these the full logs? Please let me know if I should be using a different command to test this.

Created attachment 2098109 [details]
inspect-os test without --remote flag

https://libguestfs.org/guestfs-faq.1.html#how-do-i-debug-when-using-any-libguestfs-program-or-tool-eg.-virt-customize-or-virt-df

Set the environment variables before running guestfish.

Attach the complete, unedited debug output as a TEXT file to this bug.  It will be
quite lengthy.

Created attachment 2098135 [details]
fedora 40 disk inspect-os full debug logs

Thanks Richard, I was able to get the full logs now. Just attached the text file.

So the logs show inspecting a disk.  Actually the disk comes from an HTTPS server,
which is slightly unusual but shouldn't normally cause a problem (if the network
doesn't have too high latency).  We seem to successfully inspect the disk, find
a Fedora 40 guest on the disk, and print the btrfs root volume.

However I think you reported that mounting filesystem(s) from the disk failed.
You didn't try that explicitly here (although inspection does mount some filesystems
from the disk implicitly, apparently without problem).

As such I can't really see if there's an issue with this.  What happens when you
do something that fails?  (Again, full logs are required.)

You could also try using the guestfish -i option which will inspect and mount up
the filesystems from the original guest.

Created attachment 2103086 [details]
Testing libguestfs on fedora 42, connecting to a disk sas uri fails now.

Hi Richard, the error we were seeing on fedora 40 seemed to be more of a reliability issue where there the connection to the disk would get lost intermittently. It was difficult to reproduce the error in a local environment and get debug logs.

I have been trying out different setups and am currently facing an error after upgrading to fedora 42, this time connecting to the disk consistently fails. I attached the debug logs, it looks like there is an issue supporting HTTPS protocol. Do you know if HTTPS protocol is not supported in later versions of qemu img?

(In reply to armanamjad from comment #18)
> Hi Richard, the error we were seeing on fedora 40 seemed to be more of a
> reliability issue where there the connection to the disk would get lost
> intermittently. It was difficult to reproduce the error in a local
> environment and get debug logs.

This is because it's using HTTP to connect to the disk?  You'll need to
make sure the HTTP connection is reliable and low latency.

> I have been trying out different setups and am currently facing an error
> after upgrading to fedora 42, this time connecting to the disk consistently
> fails. I attached the debug logs, it looks like there is an issue supporting
> HTTPS protocol. Do you know if HTTPS protocol is not supported in later
> versions of qemu img?

qemu-img: /home/azureuser/libguestfs/tmp/libguestfsJgFtvV/overlay1.qcow2: Unknown protocol 'https'

That's pretty strange.  What version of qemu is this (full RPM NVR)?

I was able to get past this error actually, after installing qemu-block-curl disk connection via url and launch started working. But now the latest error I am seeing is the list-filesystems and inspect-os commands are failing. I am stumped on this one and not sure if it's related to my setup or incompatibility with libguestfs 1.52 and fedora 40. Could you take a look at my debug logs and let me know if anything sticks out to you?

Created attachment 2103560 [details]
Fedora 40 libguestfs 1.52 inspect os and list filesystems failing

The server is giving errors, lots of:
qemu-kvm: curl: The requested URL returned error: 503

Hey Richard,

The above errors we were seeing were related to setup /dependency issues, and I was able to resolve those. But the main issue still is that when we connect to the disk via SAS URL rather than a local VHD file, we experience intermittent disconnection errors where the inspect-os or mount command will time out and we need to reconnect to the disk. Are there any known issues similar to this?

Additionally, we are currently using a custom fork of the repo where we have a commit which would allow guestfish commands to be able to parse query parameters, as currently this feature is unavailable. Would it be possible to get this commit merged into the main libguestfs repo so we may pull from the official source? This would reduce potential setup issues and upkeep on our end as well. I can create a pull request with the change if this commit seems reasonable to you:
https://github.com/libguestfs/libguestfs/commit/cf0d959b8a0bbaf407e76c668a3884c93b612b4b

I guess the backing disk has to be stable, otherwise no tooling can work reliably.

I have specific comments about the commit, but more generally, I'm not sure what it's trying to do.  However we could look at it properly in a PR.

Created the PR: https://github.com/libguestfs/libguestfs/pull/215

I created a PR for the Common updates as well since it is in a submodule. Please take a look: https://github.com/libguestfs/libguestfs-common/pull/19

This product has been discontinued or is no longer tracked in Red Hat Bugzilla.

Reopening because Virtualization Tools has not been discontinued.