2161969 – [leapp] IPU 7>8: EINVAL returned by mkdir() on overlayfs with XFS

This bug has been migrated to another issue tracking site. It has been closed here and may no longer be being monitored.

If you would like to get updates for this issue, or to participate in it, you may do so at Red Hat Issue Tracker .

RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 2161969 - [leapp] IPU 7>8: EINVAL returned by mkdir() on overlayfs with XFS

Summary: [leapp] IPU 7>8: EINVAL returned by mkdir() on overlayfs with XFS

Keywords:
Status:	CLOSED MIGRATED
Alias:	None
Product:	Red Hat Enterprise Linux 7
Classification:	Red Hat
Component:	leapp-repository
Sub Component:
Version:	7.9
Hardware:	All
OS:	Linux
Priority:	low
Severity:	low
Target Milestone:	rc
Target Release:	---
Assignee:	Petr Stodulka
QA Contact:	upgrades-and-conversions
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2023-01-18 13:34 UTC by Christophe Besson
Modified:	2023-09-12 12:39 UTC (History)
CC List:	3 users (show)
Fixed In Version:
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2023-09-12 12:39:22 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Priority	Status	Summary	Last Updated
Red Hat Issue Tracker	OAMG-8290	None	None	None	2023-01-18 13:39:17 UTC
Red Hat Issue Tracker	RHEL-3330	None	Migrated	None	2023-09-12 12:39:19 UTC
Red Hat Issue Tracker	RHELPLAN-145542	None	None	None	2023-01-18 13:39:14 UTC
Red Hat Knowledge Base (Solution)	7020579	None	None	None	2023-06-21 11:01:39 UTC

Description Christophe Besson 2023-01-18 13:34:20 UTC

Description of problem:
Leapp failed at the preupgrade step due to a directory that cannot be created on top of <overlayfs>/tmp.
The issue was not with this specific partition, if this one is not used it fails on the next one.

* They use the latest el7 kernel and the latest leapp packages
* They have no 3rd party tainting the kernel.
* All their XFS have an ftype (hence it supposedly supports overlays).
* The strace shows the first mkdir() on the mounted overlay fails with an EINVAL errno (excerpt of relevant calls):

24271 11:46:12.656828 mkdir("/var/lib/leapp/scratch", 0777) = 0
24271 11:46:12.656885 mkdir("/var/lib/leapp/scratch/mounts", 0777) = 0
24271 11:46:12.776710 mkdir("/var/lib/leapp/scratch/mounts/root_", 0777) = 0
24271 11:46:12.776772 mkdir("/var/lib/leapp/scratch/mounts/root_/upper", 0777) = 0
24271 11:46:12.776866 mkdir("/var/lib/leapp/scratch/mounts/root_/work", 0777) = 0
24271 11:46:12.776954 mkdir("/var/lib/leapp/scratch/mounts/root_/system_overlay", 0777) = 0
24282 11:46:12.801320 mkdir("/run/mount", 0755) = -1 EEXIST (File exists)
24282 11:46:12.801935 mount("overlay2", "/var/lib/leapp/scratch/mounts/root_/system_overlay", "overlay", MS_MGC_VAL, "lowerdir=/,upperdir=/var/lib/lea"...) = 0
24271 11:46:12.930815 mkdir("/var/lib/leapp/scratch/mounts/root_tmp", 0777) = 0
24271 11:46:12.930881 mkdir("/var/lib/leapp/scratch/mounts/root_tmp/upper", 0777) = 0
24271 11:46:12.930973 mkdir("/var/lib/leapp/scratch/mounts/root_tmp/work", 0777) = 0
24271 11:46:12.931066 mkdir("/var/lib/leapp/scratch/mounts/root_tmp/root_tmp", 0777) = 0
24288 11:46:12.956517 mkdir("/run/mount", 0755) = -1 EEXIST (File exists)
24288 11:46:12.957128 mount("overlay2", "/var/lib/leapp/scratch/mounts/root_tmp/root_tmp", "overlay", MS_MGC_VAL, "lowerdir=/tmp,upperdir=/var/lib/"...) = 0
24290 11:46:12.998509 newfstatat(AT_FDCWD, "/var/lib/leapp/scratch/mounts/root_/system_overlay/tmp", {st_mode=S_IFDIR|0755, st_size=6, ...}, AT_SYMLINK_NOFOLLOW) = 0
24290 11:46:12.998848 unlinkat(AT_FDCWD, "/var/lib/leapp/scratch/mounts/root_/system_overlay/tmp", AT_REMOVEDIR) = 0
24271 11:46:13.015461 mkdir("/var/lib/leapp/scratch/mounts/root_/system_overlay/tmp", 0777) = -1 EINVAL (Invalid argument)

My current assumption is a problem with overlay and xfs, in very rare conditions.
Attaching the leapp.db they shared.

Version-Release number of selected component (if applicable):
leapp-upgrade-el7toel8-0.17.0-1.el7_9

How reproducible:
Reproducible at will on customer site.
Not able to reproduce internally.

Actual results:

Upgrade is not done.

Additional info:

* A similar (but not identical) issue has been reported in the past (rhbz#1870668).
It has been closed due to insufficient data, and the KCS (#5339021) clearly does not apply for them.

* Workaround was to make Leapp believes it has to create ext4 images as upper layers (as it is done for XFS without ftype)

Edit /usr/share/leapp-repository/repositories/system_upgrade/common/libraries/overlaygen.py and insert one line (52) to define an array in _prepare_required_mounts():

 46 def _prepare_required_mounts(scratch_dir, mounts_dir, mount_points, xfs_info):
 47     result = {
 48         mount_point.fs_file: mounting.NullMount(
 49             _mount_dir(mounts_dir, mount_point.fs_file)) for mount_point in mount_points
 50     }
 51 
 52     xfs_info.mountpoints_without_ftype = [ "/", "/var", "/usr", "/boot" ]  # <<<      
 53     if not xfs_info.mountpoints_without_ftype:
 54         return result

Additionally, it has been suggested comment out the partitions which are not required during the in-place upgrade and use a larger LEAPP_OVL_SIZE (8192).


* A simple reproducer involving the same set of syscalls has been suggested but the customer didn't spend the time to do so.

yum install strace trace-cmd
strace -fTttyyvs 1024 -o /root/reproducer.strace -p $$ &
rm -rf /var/lib/leapp/scratch
mkdir -p /var/lib/leapp/scratch/mounts/root_/{system_overlay,upper,work}
mount -t overlay overlay2 -o lowerdir=/,upperdir=/var/lib/leapp/scratch/mounts/root_/upper,workdir=/var/lib/leapp/scratch/mounts/root_/work /var/lib/leapp/scratch/mounts/root_/system_overlay
ls -l /var/lib/leapp/scratch/mounts/root_/system_overlay
ls -l /var/lib/leapp/scratch/mounts/root_/system_overlay/tmp
rm -rf /var/lib/leapp/scratch/mounts/root_/system_overlay/tmp
trace-cmd record -o /root/mkdir.trc -b 10000 -F -p function_graph -O hex -- mkdir -m 777 /var/lib/leapp/scratch/mounts/root_/system_overlay/tmp
ls -l /var/lib/leapp/scratch/mounts/root_/system_overlay
pkill strace
umount -fl /var/lib/leapp/scratch/mounts/root_/system_overlay
trace-cmd report /root/mkdir.trc > /root/mkdir.trc.txt
tar zcf /tmp/debug.tgz /root/mkdir.trc /root/mkdir.trc.txt /root/reproducer.strace /var/lib/leapp/leapp.db

* From mkdir() man page:

       EINVAL The final component ("basename") of the new directory's pathname is invalid 
              (e.g., it contains characters not permitted by the underlying filesystem).

* lsblk

NAME          MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda             8:0    0   150G  0 disk 
|-sda1          8:1    0   953M  0 part /boot
`-sda2          8:2    0 139.7G  0 part 
  |-rhel-root 253:0    0  23.3G  0 lvm  /
  |-rhel-usr  253:1    0  18.6G  0 lvm  /usr
  |-rhel-tmp  253:2    0   9.3G  0 lvm  /tmp
  |-rhel-swap 253:3    0   9.3G  0 lvm  /swap
  |-rhel-home 253:4    0    28G  0 lvm  /home
  |-rhel-opt  253:5    0    28G  0 lvm  /opt
  `-rhel-var  253:6    0  23.3G  0 lvm  /var
sr0            11:0    1  1024M  0 rom  

* /etc/fstab

/dev/mapper/rhel-root   /                       xfs     defaults        0 0
UUID=7a37d1ea-22f6-43b4-bc36-e2dcad4d8212 /boot                   xfs     defaults        0 0
/dev/mapper/rhel-home   /home                   xfs     defaults        0 0
/dev/mapper/rhel-opt    /opt                    xfs     defaults        0 0
/dev/mapper/rhel-swap   /swap                   xfs     defaults        0 0
/dev/mapper/rhel-tmp    /tmp                    xfs     defaults        0 0
/dev/mapper/rhel-usr    /usr                    xfs     defaults        0 0
/dev/mapper/rhel-var    /var                    xfs     defaults        0 0

* blkid

/dev/sda1: UUID="7a37d1ea-22f6-43b4-bc36-e2dcad4d8212" TYPE="xfs" 
/dev/sda2: UUID="SwTtcI-Zs2a-acFH-MWED-Z1HO-We1q-Ln3JLg" TYPE="LVM2_member" 
/dev/mapper/rhel-root: UUID="961840e5-dbd0-4112-a243-043073a98092" TYPE="xfs" 
/dev/mapper/rhel-usr: UUID="a0f82a03-e72a-4109-abad-a0588c1ebf96" TYPE="xfs" 
/dev/mapper/rhel-tmp: UUID="8d569aed-4522-433f-9f59-2cf02651422b" TYPE="xfs" 
/dev/mapper/rhel-swap: UUID="25df4b5c-79b3-4453-a993-d1e2df3e3dde" TYPE="xfs" 
/dev/mapper/rhel-home: UUID="cb4b763e-d4dc-4277-bfb6-7d500b3dafd3" TYPE="xfs" 
/dev/mapper/rhel-opt: UUID="e547d839-3948-4638-91ca-ab0aba2d1582" TYPE="xfs" 
/dev/mapper/rhel-var: UUID="6765f39b-b8bc-4b83-814b-f312a522a191" TYPE="xfs" 

* grep -o ftype=. sos_commands/xfs/*

sos_commands/xfs/xfs_info:ftype=1
sos_commands/xfs/xfs_info_.boot:ftype=1
sos_commands/xfs/xfs_info_.home:ftype=1
sos_commands/xfs/xfs_info_.opt:ftype=1
sos_commands/xfs/xfs_info_.swap:ftype=1
sos_commands/xfs/xfs_info_.tmp:ftype=1
sos_commands/xfs/xfs_info_.usr:ftype=1
sos_commands/xfs/xfs_info_.var:ftype=1

* default sysctls

* cmdline

BOOT_IMAGE=/vmlinuz-3.10.0-1160.80.1.el7.x86_64 root=/dev/mapper/rhel-root ro crashkernel=auto rd.lvm.lv=rhel/root rd.lvm.lv=rhel/usr rhgb quiet LANG=en_IN.UTF-8

Comment 4 Christophe Besson 2023-01-18 14:42:53 UTC

I'm not sure we'll be able to reproduce such an issue.
As we rarely observed this issue, in my opinion one sufficient remediation could be to have the ability to use the same workaround as for XFS without ftype.
Something like LEAPP_ENABLE_EXT4_UPPER=<part1,part2,part3>

Comment 5 Petr Stodulka 2023-01-18 16:12:48 UTC

Thank you for the report and initial investigation. Regarding additional problems we have reported related to use of overlayfs, it seems that we will anyway start to create virtual fss/images for all partitions always, which would resolve this problem also. Just we need to figure out which technology to use to handle it right, without consumption of too much space (we have our horses in game already). We will update the ticket when we have additional info.

Comment 6 Christophe Besson 2023-01-27 15:09:50 UTC

Adding a new case in this BZ, the root cause being very likely *similar*.
No EINVAL errno this time, but an invisible /usr in the installroot leading to the below error:

STDERR:
Can not load RPM file: /installroot/usr/share/leapp-repository/repositories/system_upgrade/el7toel8/files/bundled-rpms/leapp-deps-el8-5.0.8-100.202208231727Z.29a77886.master.el8.noarch.rpm.
Can not load RPM file: /installroot/usr/share/leapp-repository/repositories/system_upgrade/el7toel8/files/bundled-rpms/leapp-repository-deps-el8-5.0.8-100.202208231727Z.29a77886.master.el8.noarch.rpm.
Can not load RPM file: /installroot/usr/share/leapp-repository/repositories/system_upgrade/el7toel8/files/bundled-rpms/kernel-workaround-0.1-1.el8.noarch.rpm.
Could not open: /installroot/usr/share/leapp-repository/repositories/system_upgrade/el7toel8/files/bundled-rpms/leapp-deps-el8-5.0.8-100.202208231727Z.29a77886.master.el8.noarch.rpm /installroot/usr/share/leapp-repository/repositories/system_upgrade/el7toel8/files/bundled-rpms/leapp-repository-deps-el8-5.0.8-100.202208231727Z.29a77886.master.el8.noarch.rpm /installroot/usr/share/leapp-repository/repositories/system_upgrade/el7toel8/files/bundled-rpms/kernel-workaround-0.1-1.el8.noarch.rpm

The system is very standard, upgraded, without noticeable 3rd party known to cause an issue.

The rootfs is presented as an overlayfs in /var/lib/leapp/scratch/mounts/root_/system_overlay (lower layer is a ro / and upper layer is rw from /var).
It is then bind-mounted into /var/lib/leapp/el8userspace/installroot:
2023-01-24 15:32:32.516 DEBUG    PID: 37375 leapp.workflow.TargetTransactionCheck.dnf_transaction_check: External command has finished: ['mount', '-o', 'bind', u'/var/lib/leapp/scratch/mounts/root_/system_overlay', u'/var/lib/leapp/el8userspace/installroot']

And then, any subsequent mount into /var/lib/leapp/scratch/mounts/root_/system_overlay _should_ be visible into /var/lib/leapp/el8userspace/installroot as it is a bind-mount.
2023-01-24 15:32:33.190 DEBUG    PID: 37375 leapp.workflow.TargetTransactionCheck.dnf_transaction_check: External command has finished: ['mount', '-t', 'overlay', 'overlay2', '-o', 'lowerdir=/usr,upperdir=/var/lib/leapp/scratch/mounts/root_usr/upper,workdir=/var/lib/leapp/scratch/mounts/root_usr/work', u'/var/lib/leapp/scratch/mounts/root_usr/root_usr']
 :
2023-01-24 15:32:33.205 DEBUG    PID: 37375 leapp.workflow.TargetTransactionCheck.dnf_transaction_check: External command has finished: ['rm', '-rf', u'/var/lib/leapp/scratch/mounts/root_/system_overlay/usr']
 :
2023-01-24 15:32:33.224 DEBUG    PID: 37375 leapp.workflow.TargetTransactionCheck.dnf_transaction_check: External command has finished: ['mount', '-o', 'bind', u'/var/lib/leapp/scratch/mounts/root_usr/root_usr', u'/var/lib/leapp/scratch/mounts/root_/system_overlay/usr']

From the strace of another execution we have the below relevant calls:
63742 16:29:41.450384 mkdir("/var/lib/leapp/scratch/mounts", 0777) = -1 EEXIST (File exists) <0.000011>
63742 16:29:41.604071 mkdir("/var/lib/leapp/scratch/mounts/root_/upper", 0777) = 0 <0.000064>
63742 16:29:41.604372 mkdir("/var/lib/leapp/scratch/mounts/root_/work", 0777) = 0 <0.000045>
63742 16:29:41.604662 mkdir("/var/lib/leapp/scratch/mounts/root_/system_overlay", 0777) = 0 <0.000045>
63754 16:29:41.640372 mkdir("/run/mount", 0755) = -1 EEXIST (File exists) <0.000010>
63754 16:29:41.641234 mount("overlay2", "/var/lib/leapp/scratch/mounts/root_/system_overlay", "overlay", MS_MGC_VAL, "lowerdir=/,upperdir=/var/lib/leapp/scratch/mounts/root_/upper,workdir=/v
ar/lib/leapp/scratch/mounts/root_/work") = 0 <0.000191>
63742 16:29:41.713580 mkdir("/var/lib/leapp/el8userspace/installroot", 0777) = 0 <0.000074>
63756 16:29:41.741549 mkdir("/run/mount", 0755) = -1 EEXIST (File exists) <0.000010>
63756 16:29:41.742459 mount("/var/lib/leapp/scratch/mounts/root_/system_overlay", "/var/lib/leapp/el8userspace/installroot", 0x55e64fdfd240, MS_MGC_VAL|MS_BIND, NULL) = 0 <0.000027>
 :
63742 16:29:43.659714 mkdir("/var/lib/leapp/scratch/mounts/root_usr/upper", 0777) = 0 <0.000096>
63742 16:29:43.660080 mkdir("/var/lib/leapp/scratch/mounts/root_usr/work", 0777) = 0 <0.000042>
63742 16:29:43.660339 mkdir("/var/lib/leapp/scratch/mounts/root_usr/root_usr", 0777) = 0 <0.000080>
63795 16:29:43.696451 mkdir("/run/mount", 0755) = -1 EEXIST (File exists) <0.000009>
63795 16:29:43.697315 mount("overlay2", "/var/lib/leapp/scratch/mounts/root_usr/root_usr", "overlay", MS_MGC_VAL, "lowerdir=/usr,upperdir=/var/lib/leapp/scratch/mounts/root_usr/upper,workdir=/var/lib/leapp/scratch/mounts/root_usr/work") = 0 <0.000191>
63742 16:29:43.770861 mkdir("/var/lib/leapp/scratch/mounts/root_/system_overlay/usr", 0777) = 0 <0.000105>
63797 16:29:43.802181 mkdir("/run/mount", 0755) = -1 EEXIST (File exists) <0.000009>
63797 16:29:43.803206 mount("/var/lib/leapp/scratch/mounts/root_usr/root_usr", "/var/lib/leapp/scratch/mounts/root_/system_overlay/usr", 0x564737735240, MS_MGC_VAL|MS_BIND, NULL) = 0 <0.000024>
63797 16:29:43.803267 access("/var/lib/leapp/scratch/mounts/root_/system_overlay/usr", W_OK) = 0 <0.000011>
 :
63742 16:29:43.812295 write(4</var/log/leapp/leapp-preupgrade.log>, "2023-01-24 16:29:43.812 DEBUG    PID: 63742 leapp.workflow.TargetTransactionCheck.dnf_transaction_check: External command has finished: ['mount', '-o', 'bind', u'/var/lib/leapp/scratch/mounts/root_usr/root_usr', u'/var/lib/leapp/scratch/mounts/root_/system"..., 271) = 271 <0.000018>
 :
 :
63896 16:30:02.306331 write(3</var/log/dnf.log>, "2023-01-24T16:30:02+0100 DEBUG Excludes in dnf.conf: leapp, leapp-upgrade-el7toel8, python2-leapp, snactor\n", 107) = 107 <0.000019>
63896 16:30:02.306946 stat("/installroot/usr/share/leapp-repository/repositories/system_upgrade/el7toel8/files/bundled-rpms/leapp-deps-el8-5.0.8-100.202208231727Z.29a77886.master.el8.noarch.
rpm", 0x7ffdacbfbc10) = -1 ENOENT (No such file or directory) <0.000023>
63896 16:30:02.307034 access("/installroot/usr/share/leapp-repository/repositories/system_upgrade/el7toel8/files/bundled-rpms/leapp-deps-el8-5.0.8-100.202208231727Z.29a77886.master.el8.noarc
h.rpm", R_OK) = -1 ENOENT (No such file or directory) <0.000013>
63896 16:30:02.307102 write(7</var/log/hawkey.log>, "2023-01-24T16:30:02+0100 WARN not a readable RPM file: /installroot/usr/share/leapp-repository/repositories/system_upgrade/el7toel8/files
/bundled-rpms/leapp-deps-el8-5.0.8-100.202208231727Z.29a77886.master.el8.noarch.rpm, skipping\n", 231) = 231 <0.000016>

The customer was not afraid to think outside the box and used a workaround a little bit rough, but claiming it worked for them:
mount -o bind,ro /var/lib/leapp/scratch/mounts/root_/system_overlay/usr /var/lib/leapp/el8userspace/installroot/usr

at the right instant.

I guess the workaround suggested in the description of this BZ is more difficult to implement but it could work and would be safer.

Comment 7 Christophe Besson 2023-06-16 16:52:35 UTC

Another EINVAL on mkdir()

9465  13:11:09.956191 mount("overlay2", "/var/lib/leapp/scratch/mounts/root_/system_overlay", "overlay", MS_MGC_VAL, "lowerdir=/,upperdir=/var/lib/leapp/scratch/mounts/root_/upper,workdir=/var/lib/leapp/scratch/mounts/root_/work") = 0 <0.000281>
9453  13:11:10.199485 mkdir("/var/lib/leapp/scratch/mounts/root_/system_overlay/var/cache/dnf", 0777) = -1 EINVAL (Invalid argument) <0.000096>

Comment 8 Christophe Besson 2023-06-21 11:01:23 UTC

Not supported as it requires to patch the code, but it's the easier workaround:

# sed -i 's/def is_xfs_without_ftype(mp):/def is_xfs_without_ftype(mp):\n    return True/' /usr/share/leapp-repository/repositories/system_upgrade/common/actors/xfsinfoscanner/libraries/xfsinfoscanner.py

It makes leapp believes it has to deal with XFS ftype=0 cases
Possibly adjust $LEAPP_OVL_SIZE as explained in https://access.redhat.com/solutions/5057391

(worked for the last case)

Comment 9 Petr Stodulka 2023-06-21 11:20:59 UTC

Thank you Chris for the investigation! The suggested workaround supports our expectation the issue will be fixed by the redesigned solution we are working on - as we are going to create a diskimage for each partition listed in FSTAB for temporary overlay filesystems we create during the upgrade process.

Comment 10 Petr Stodulka 2023-07-17 19:24:28 UTC

Should be fixed by upstream PR:
  * https://github.com/oamg/leapp-repository/pull/1097

If it is ever discovered again on any machine, we need to test it with the particular fix. I believe the current fix should work right also with XFS hopefully. If not, it will be possible to switch to the ext4 FS for upper directories using:
    LEAPP_OVL_IMG_FS_EXT4=1 leapp ...

It's possible the issue is caused also by an older XFS format or some exotic XFS params. Currently used disk images (for the upgrade purposes) will be formatted by XFS with up-to-date (and same) params. The possibility to switch between XFS and Ext4 will tell us whether there is hidden more deep issue between XFS and overlayfs.

Comment 12 RHEL Program Management 2023-09-12 12:38:16 UTC

Issue migration from Bugzilla to Jira is in process at this time. This will be the last message in Jira copied from the Bugzilla bug.

Comment 13 RHEL Program Management 2023-09-12 12:39:22 UTC

This BZ has been automatically migrated to the issues.redhat.com Red Hat Issue Tracker. All future work related to this report will be managed there.

Due to differences in account names between systems, some fields were not replicated.  Be sure to add yourself to Jira issue's "Watchers" field to continue receiving updates and add others to the "Need Info From" field to continue requesting information.

To find the migrated issue, look in the "Links" section for a direct link to the new issue location. The issue key will have an icon of 2 footprints next to it, and begin with "RHEL-" followed by an integer.  You can also find this issue by visiting https://issues.redhat.com/issues/?jql= and searching the "Bugzilla Bug" field for this BZ's number, e.g. a search like:

"Bugzilla Bug" = 1234567

In the event you have trouble locating or viewing this issue, you can file an issue by sending mail to rh-issues. You can also visit https://access.redhat.com/articles/7032570 for general account information.

Note You need to log in before you can comment on or make changes to this bug.