RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1825320 - Waagent on Gen2 RHEL VMs fails to mount the ephemeral disk (v 2.2.38)
Summary: Waagent on Gen2 RHEL VMs fails to mount the ephemeral disk (v 2.2.38)
Keywords:
Status: CLOSED DUPLICATE of bug 1714167
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: WALinuxAgent
Version: 7.7
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: rc
: ---
Assignee: Vitaly Kuznetsov
QA Contact: Yuxin Sun
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-04-17 17:08 UTC by Esteban Flores
Modified: 2023-09-14 05:55 UTC (History)
8 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-04-23 03:18:32 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)
sosreport from my test VM (8.73 MB, application/x-xz)
2020-04-17 17:08 UTC, Esteban Flores 		no flags Details
View All

Description Esteban Flores 2020-04-17 17:08:04 UTC 
Created attachment 1679712 [details]
sosreport from my test VM

Description of problem:

On Gen2 Images for RHEL where the ephemeral disk is GPT (where the agent is still the provisioning method), the agent fails to mount the resource disk with:

 

2020/04/13 16:19:51.819910 ERROR Daemon Failed to mount resource disk [ResourceDiskError] unable to detect disk topology

2020/04/13 16:19:51.820134 ERROR Daemon Event: name=WALinuxAgent, op=ActivateResourceDisk, message=[ResourceDiskError] unable to detect disk topology, duration=0

 

This was fixed in theory in version 2.2.45 of the waagent through PR https://github.com/Azure/WALinuxAgent/pull/1654

 

By default RHEL updates the agent to 2.2.46 (goal state) which should have PR1654, but still fails. Manually downloaded the source from the Github repo and updated to 2.2.47.2, verified that this version does mount the ephemeral disk. Ideally, the agent should auto update, but even with AutoUpdate.Enabled the agent remains on 2.2.46. Addionally, the images should be updated to use 2.2.47 as Gen2 become more popular.

The provisioning portion of the agent remains at 2.2.38 which falls under the issue mentioned above.

Ideally, version 2.2.46 needs to be rolled to RHEL 7 (and RHEL 8) to avoid this problem.
 

Version-Release number of selected component (if applicable):

Agent 2.2.38

How reproducible:

This can be easily reproed by simply deploying a Gen2 image (RedHat:RHEL:81gen2:8.1.2020020416 or RedHat:RHEL:77-gen2:7.7.2020020416) on a Gen2 supported size ( like a D2s_v3) and running df -h to confirm no ephemeral disk).

Steps to Reproduce:
1. az vm create --name rhel77gen2 --resource-group rhel77gen2 --image RedHat:RHEL:77-gen2:7.7.2020020416 --size Standard_D2s_v3 --location eastus2
2. df -h to confirm no /dev/sdb mount
3. less /var/log/waagent.log to confirm the agent fails to mount the ephemeral drive

Actual results:

Ephemeral drive not mounted

Expected results:

Ephemeral drive mounted through the waagent

Additional info:

Downloading the latest version of the WaLinuxAgent from source and updating does confirm it's an issue with the installed version, as the update allows the ephemeral disk to mount correctly.

Have not tested with Cloud-init
Comment 2 Yuxin Sun 2020-04-20 04:23:03 UTC 
Hi Esteban,

We have released WALinuxAgent-2.2.38-2.el7 last month which included this fix. I think the latest RHEL-7.8 images already have this package installed. Let's discuss with Alfred Sin if we need to rebuild RHEL-7.7 gen2 image.

For RHEL-8, we plan to release v2.2.46 later, so don't plan to backport this patch in v2.2.38. The v2.2.46 will be released in several months. Is it very urgent for any customer? Thanks!

For cloud-init, I have run basic test in Gen2 and didn't find this issue.
Comment 3 Esteban Flores 2020-04-20 16:21:17 UTC 
Hey Yuxin,

Thanks for the update! :) 

I've updated my 7.7 VM to 7.8 and updated to WALinuxAgent-2.2.38-2 which was able to mount the ephemeral disk. For now I've gotten a couple of people asking about this, but I'm seeing this happening more often as Gen2 becomes widely available.

Do you know if WALinuxAgent-2.2.38-2 would be backported to 7.7?

Thanks!
Comment 4 Vitaly Kuznetsov 2020-04-20 17:21:19 UTC 
(In reply to Esteban Flores from comment #3)
> Hey Yuxin,
> 
> Thanks for the update! :) 
> 
> I've updated my 7.7 VM to 7.8 and updated to WALinuxAgent-2.2.38-2 which was
> able to mount the ephemeral disk. For now I've gotten a couple of people
> asking about this, but I'm seeing this happening more often as Gen2 becomes
> widely available.
> 
> Do you know if WALinuxAgent-2.2.38-2 would be backported to 7.7?

In RHEL7 (unlike RHEL8) WALinuxAgent is distributed through 'extras' channel
which is not tied to a particular Y-stream release, it's the same repo for
all of them. The 'extras' repo needs to be enabled to receive the update
though and this may not be the default case for some EUS customers.
Comment 5 Yuxin Sun 2020-04-23 03:18:32 UTC 
Hi Esteban,

I'd like to close this BZ as a duplicate of BZ#1714167. Please update here if you have any concern. Thanks!

*** This bug has been marked as a duplicate of bug 1714167 ***
Comment 6 Esteban Flores 2020-04-23 16:39:32 UTC 
Hey Yuxin,

The only concern I have is how we'll distribute the updated version to EUS customers? Don't mind closing the BZ as long as we have a clear plan on bringing the 2.2.38-2 version to EUS customers.

Thx!
Comment 7 Yuxin Sun 2020-04-24 14:32:42 UTC 
Hi Esteban,

Sorry we don't provide WALA package in EUS repo. Current resolution is: "Contact Red Hat Support or Microsoft Support if you have EUS and need a copy of the waagent that is not corrupted or need a copy for a reinstall." (KCS: https://access.redhat.com/solutions/4134701)

Hi Alfred,

Do you know that if it is possible to build the new WALA package into those gen2 images(e.g. 74-gen2...) in Azure Marketplace? In such case I think we don't need WALA package in EUS repo. Currently WALinuxAgent-2.2.38-2.el7_7.noarch.rpm includes this fix.

Thanks!
Comment 8 Alfred Sin 2020-04-24 17:25:59 UTC 
Oops, added Mayank but accidentally cleared my needinfo without actually providing any info. Yuxin, just to clarify, the new WALA package should be available all the way back to 7.4? When we build the image we literally just run 'yum install -y WALinuxAgent' so it will install whatever version is available. I'll kick off a 7.4 to check.
Comment 9 Alfred Sin 2020-04-24 19:03:13 UTC 
This shows up in a RHEL 7.4 build that just ran:
"Package WALinuxAgent-2.2.38-2.el7_7.noarch already installed and latest version"

So I think we're good here... it seems like new images built will have 2.2.38-2 installed.
Comment 10 Yuxin Sun 2020-04-26 03:26:48 UTC 
(In reply to Alfred Sin from comment #9)
> This shows up in a RHEL 7.4 build that just ran:
> "Package WALinuxAgent-2.2.38-2.el7_7.noarch already installed and latest
> version"
> 
> So I think we're good here... it seems like new images built will have
> 2.2.38-2 installed.

Great! Thank you so much Alfred!

Hi Esteban,

Can this be a resolution of your question? Thanks!
Comment 11 Red Hat Bugzilla 2023-09-14 05:55:37 UTC 
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days
Note You need to log in before you can comment on or make changes to this bug.