Bug 1430435

Created attachment 1261320 [details]
ironic deployment error with 4K native disks

Reference, 4K native sector support for RHEL:  https://access.redhat.com/solutions/56494

Hi,

Sorry for the delay on this. I have never seems or tried deploying a node with 4K native disks.

By the screenshot it seems that the problem occurs at the time we are installing the bootloader, specifically when the deploy ramdisk tries to mount the image partition (Ironic uses the GRUB contained in the image being deployed).

So I would suggest, just for testing purpose if it would be possible to disable local booting to see if it works so we can narrow down the problem. To disable it, we need to update the node's "boot_option" capabilities. If you run "ironic node-show <name>" and look at the properties/capabilities field you will see the "boot_option:local" there we need to change that to "boot_option:netboot", e.g:

$ ironic node-update <node uuid/name> add properties/capabilities="...,boot_option:netboot"

The flavor in nova needs to be modified to match that behavior as well.

And that would disable the boot loader installation. With "netboot" the nodes will always PXE boot (and chainload to the local disk after it's being deployed).

...

@Matt, the link you posted is also interesting [0]. And it mentions that " traditional BIOS cannot boot from 4k Native devices". This is something we could try as well, to boot the nodes in UEFI mode. 

Ironic does support UEFI deployments, similar to the "boot_option" capability described above, we need to set a "boot_mode:uefi" capability in the nodes and flavor. Here's the upstream documentation about it:

https://docs.openstack.org/project-install-guide/baremetal/draft/setup-drivers.html

[0] https://access.redhat.com/solutions/56494

Hope that helps,
Lucas

Hi Lucas,

Thanks for getting back with me.

I agree that the error is occurring during the procedure to install the boot loader but it seems the issue is that /dev/sda2 can not even be mounted due to the mismatch in the sector size of the image and physical disk.  I don't believe grub is even trying to install a boot loader yet as it needs to mount and chroot the root file system first.  It would seem the same error would occur during boot regardless of where the boot loader is; do you agree?

Thanks,

Matt

As a workaround, it should be relatively easy to script a conversion by extracting the raw fs image from the qcow2 image, creating a new xfs filesystem of similar size and geometry but with 4k sectors, loop mounting it, copying files to it, unmounting it, and recreating a qcow image from that new xfs filesystem.  I'm not sure how feasible that is in actual deployments...

Just got another case of the same problem

Derek was so kind to agree to take a look at this. We may also get hardware reasonably soon, so stay tuned.

Any helps with triaging is still appreciated.

I was told that the problem can be worked around using whole disk images as described in https://teknoarticles.blogspot.de/2016/12/start-using-whole-disk-images-with.html. Just make sure to change the sector size to 4k in the script you end up with.

I've reproduced this in virt by adding a block size to the disk exposed to the guest
<blockio logical_block_size='4096' physical_block_size='4096'/>

(I also tried using scsi_debug on the host and exposing this to the guest but qemu presented it with a 512 sector size)

ironic-python-agent[629]: 2017-11-21 00:21:14.925 629 DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): mount /dev/sda3 /tmp/tmpdmWBzG execute /usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py:367
kernel: SGI XFS with ACLs, security attributes, no debug enabled
kernel: XFS (sda3): device supports 4096 byte sectors (not 512)
ironic-python-agent[629]: 2017-11-21 00:21:14.996 629 DEBUG oslo_concurrency.processutils [-] CMD "mount /dev/sda3 /tmp/tmpdmWBzG" returned: 32 in 0.071s execute /usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py:404
ironic-python-agent[629]: 2017-11-21 00:21:14.997 629 DEBUG oslo_concurrency.processutils [-] u'mount /dev/sda3 /tmp/tmpdmWBzG' failed. Not Retrying. execute /usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py:452
ironic-python-agent[629]: 2017-11-21 00:21:14.997 629 ERROR ironic_python_agent.extensions.image [-] Installing GRUB2 boot loader to device /dev/sda failed with Unexpected error while running command.
ironic-python-agent[629]: Command: mount /dev/sda3 /tmp/tmpdmWBzG
ironic-python-agent[629]: Exit code: 32
ironic-python-agent[629]: Stdout: u''
ironic-python-agent[629]: Stderr: u'mount: mount /dev/sda3 on /tmp/tmpdmWBzG failed: Function not implemented\n'.: ProcessExecutionError: Unexpected error while running command.



I then built an overcloud-full image with "-s size=4096" passed into mkfs (mentioned in comment 16)

the resulting image appears to solve the problem, and can still be used with disks with a 512 sector size.

======== Image used with 4096 sector sizes =========
[root@t2 centos]# echo p | fdisk /dev/sda
Disk /dev/sda: 85.9 GB, 85899345920 bytes, 20971520 sectors
Units = sectors of 1 * 4096 = 4096 bytes
Sector size (logical/physical): 4096 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk label type: gpt
Disk identifier: C50C47EF-30DA-4BBF-9140-010A5B76E1F7
#         Start          End    Size  Type            Name
 1          256        51455    200M  EFI System      primary
 2        51456        51711      1M  Microsoft basic primary
 3        51712     20971514   79.8G  Microsoft basic primary
[root@t2 centos]# xfs_info /
meta-data=/dev/sda3              isize=512    agcount=69, agsize=305284 blks
         =                       sectsz=4096  attr=2, projid32bit=1
         =                       crc=1        finobt=0 spinodes=0
data     =                       bsize=4096   blocks=20919803, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=1
log      =internal               bsize=4096   blocks=2560, version=2
         =                       sectsz=4096  sunit=1 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0


======== The same image used with 512 sector sizes =========
[root@t3 centos]# echo p | fdisk /dev/sda
Disk /dev/sda: 85.9 GB, 85899345920 bytes, 167772160 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: gpt
Disk identifier: 480F61BE-53C5-414E-AAF6-7ADBFEE90C57
#         Start          End    Size  Type            Name
 1         2048       411647    200M  EFI System      primary
 2       411648       413695      1M  Microsoft basic primary
 3       413696    167772126   79.8G  Microsoft basic primary
[root@t3 centos]# xfs_info /
meta-data=/dev/sda3              isize=512    agcount=69, agsize=305284 blks
         =                       sectsz=4096  attr=2, projid32bit=1
         =                       crc=1        finobt=0 spinodes=0
data     =                       bsize=4096   blocks=20919803, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=1
log      =internal               bsize=4096   blocks=2560, version=2
         =                       sectsz=4096  sunit=1 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

Once we assess that there isn't any signifigant performance problems etc
introduced by using a 4k xfs filesystem on a traditional
512b disk, I'd suggest we just switch the default on the overcloud-full image.

Derek, could you please provide an example command you used to build an image? Such a workaround could help a lot.

cc: dchinner - Dave, can you spare any brain cycles on making sure this is headed in a safe direction?

"Once we assess that there isn't any signifigant performance problems etc
introduced by using a 4k xfs filesystem on a traditional
512b disk, I'd suggest we just switch the default on the overcloud-full image."

My worry is atomicity of 4k metadata writes on 512 disk, but I /think/ the torn write detection bfoster did makes that safe.

Thanks,
-Eric

(In reply to Dmitry Tantsur from comment #32)
> Derek, could you please provide an example command you used to build an
> image? Such a workaround could help a lot.

Note: I was testing this upstream so generating a centos image, but the RHEL equivalent would be similar but
1. use overcloud-images-rhel7.yaml in place of overcloud-images-centos7.yaml
2. set the various REG_* env variables relevant to your RHEL subscription (documented in the rhel-common disk-image-builder element), and no need for DIB_YUM_REPO_CONF

To create the image with a 4096 sector size, I had to patch two files in tripleo-common

1. Add -s size=4096 to the mkfs command (if building a rhel image then patch overcloud-images-rhel7.yaml instead)
diff -r /usr/share_/tripleo-common/image-yaml/overcloud-images-centos7.yaml /usr/share/tripleo-common/image-yaml/overcloud-images-centos7.yaml
9a10,11
>     options:
>       - "--mkfs-options '-s size=4096'"

2. support spaces in mkfs options correctly
diff -r /usr/lib/python2.7/site-packages_/tripleo_common/image/image_builder.py /usr/lib/python2.7/site-packages/tripleo_common/image/image_builder.py
19a20
> import shlex
97c98
<                 cmd.extend(option.split(' '))
---
>                 cmd.extend(shlex.split(option))

Then to build a centos overcloud-full image
$ export DIB_YUM_REPO_CONF="/etc/yum.repos.d/delorean-current.repo  /etc/yum.repos.d/delorean-queens-deps.repo  /etc/yum.repos.d/delorean.repo  /etc/yum.repos.d/quickstart-centos-qemu.repo" 
$ openstack overcloud image build \
    --config-file /usr/share/openstack-tripleo-common/image-yaml/overcloud-images.yaml \
    --config-file /usr/share/openstack-tripleo-common/image-yaml/overcloud-images-centos7.yaml

Once built, you can register it with 
$ openstack overcloud image upload --update-existing

(In reply to Eric Sandeen from comment #33)
> cc: dchinner - Dave, can you spare any brain cycles on making sure this is
> headed in a safe direction?
> 
> "Once we assess that there isn't any signifigant performance problems etc
> introduced by using a 4k xfs filesystem on a traditional
> 512b disk, I'd suggest we just switch the default on the overcloud-full
> image."

Performance will not be any different, because almost all of XFS's IO is filesystem block sized and aligned, which in both cases is already 4kB.

> My worry is atomicity of 4k metadata writes on 512 disk, but I /think/ the
> torn write detection bfoster did makes that safe.

Traditional 512b disk, or a 512e disk? Two very different behaviours between them....

With traditional (native) 512b sector disks, the torn log write issue for larger sectors then 512b should be solved (need to check what RHEL7.x version that was introduced). The log replay protects against torn metadata writes, and hence there should be no issues with undetected torn log and metadata writes. Torn data writes will still be an issue, but that indicates unsafe application data integrity practices so torn writes are the least of the user's worries here.

For 512e disks, they use 4k sectors on the media and so 512 byte writes are
actually RMW cycles hidden by the hardware. Using 4k sectors will stop this RMW cycle from happening for filesystem metadata and the log, but applications will still be able to do 512 byte aligned direct IO because the logical sector size is still 512. The page cache should insulate apps from this, so it's only direct IO where this gets exposed. IOWs, the torn write problem has a smaller "user data w/ direct IO" scope because of the internal RMW the drive does. The problem is still there, but very few users are likely going to be exposed to it on 512e drives.

Cheers,

Dave.

Hi, 

thanks to Derek procedure https://bugzilla.redhat.com/show_bug.cgi?id=1430435#c34 I've managed to get this working on OSP-11 with my 4K disks with UEFI enabled. 

Here's the procedure I used to create the overcloud image:

1. Add -s size=4096 to the mkfs command to /usr/share/openstack-tripleo-common/image-yaml/overcloud-images-rhel7.yaml 

>     options:
>       - "--mkfs-options '-s size=4096'"

2. support spaces in mkfs options correctly
diff -r /usr/lib/python2.7/site-packages_/tripleo_common/image/image_builder.py /usr/lib/python2.7/site-packages/tripleo_common/image/image_builder.py
19a20
> import shlex
97c98
<                 cmd.extend(option.split(' '))
---
>                 cmd.extend(shlex.split(option))

3. Generate image

export DIB_LOCAL_IMAGE=rhel-server-7.4-x86_64-kvm.qcow2
export REG_USER=user
export REG_PASSWORD=password
export REG_METHOD=portal
export REG_POOL_ID="poolid"
export REG_SERVER_URL="subscription.rhn.redhat.com"
export REG_SERVICE_LEVEL="Self-Support"
export REG_REPOS="rhel-7-server-rpms rhel-7-server-extras-rpms rhel-ha-for-rhel-7-server-rpms rhel-7-server-optional-rpms rhel-7-server-openstack-11-rpms"

openstack overcloud image build --config-file /usr/share/openstack-tripleo-common/image-yaml/overcloud-images.yaml --config-file /usr/share/openstack-tripleo-common/image-yaml/overcloud-images-rhel7.yaml

Thanks Derek :)

(In reply to Derek Higgins from comment #38)
> > Torn data writes will still be an issue, but that indicates unsafe application
> > data integrity practices so torn writes are the least of the user's worries
> > here.
> 
> Dave can you elaborate on this, what kind of practices would cause problems
> and what would the problems be?

The problem is applications not using fsync/fdatasync() where they need to to guarantee data is on stable storage. If they don't do this, then torn writes don't matter - critical data is going to be lost on crash/power loss regardless of the storage setup.

Really, bugzilla is not the place to discuss this. We've documented how to write data to stable storage safely in our developer guides and all over the web, such as this excellent article by Jeff Moyer:

https://lwn.net/Articles/457667/

-Dave.

(In reply to Dave Chinner from comment #39)
> (In reply to Derek Higgins from comment #38)
> > > Torn data writes will still be an issue, but that indicates unsafe application
> > > data integrity practices so torn writes are the least of the user's worries
> > > here.
> > 
> > Dave can you elaborate on this, what kind of practices would cause problems
> > and what would the problems be?
> 
> The problem is applications not using fsync/fdatasync() where they need to
> to guarantee data is on stable storage. If they don't do this, then torn
> writes don't matter - critical data is going to be lost on crash/power loss
> regardless of the storage setup.
> 
> Really, bugzilla is not the place to discuss this. We've documented how to
> write data to stable storage safely in our developer guides and all over the
> web, such as this excellent article by Jeff Moyer:
> 
> https://lwn.net/Articles/457667/
> 
> -Dave.


Hi,

in my case I will use 512 disks for Controller nodes and 4k for compute. My question is if this is safe or not? 

Thanks

Any update on when we might see this in a RHOSP release?

I've submitted this change into tripleo-common upstream, once it has landed we can consider potential backports etc.

Verified:

Environment:
instack-undercloud-8.4.1-4.el7ost.noarch
openstack-tripleo-common-containers-8.6.1-12.el7ost.noarch
openstack-tripleo-common-8.6.1-12.el7ost.noarch


Notes:
It requires UEFI.

Successfully booted a server with 4096 block size disk.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2018:2086