Bug 2210916

Summary: [RHEL9] Failed to commit changes to disk: Failed to format device: Input/output error
Product: Red Hat Enterprise Linux 9 Reporter: guazhang <guazhang>
Component: rhel-system-rolesAssignee: Rich Megginson <rmeggins>
Status: CLOSED ERRATA QA Contact: guazhang <guazhang>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 9.3CC: rmeggins, spetrosi
Target Milestone: rcKeywords: Triaged
Target Release: 9.3Flags: pm-rhel: mirror+
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: role:storage
Fixed In Version: rhel-system-roles-1.22.0-0.9.el9 Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of:
: 2211247 (view as bug list) Environment:
Last Closed: 2023-11-07 08:29:47 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 2211247    

Description guazhang@redhat.com 2023-05-30 03:43:03 UTC
Description of problem:
storage role testing found the error, please have a look

Version-Release number of selected component (if applicable):
ansible-core-2.15.0-1.el9.x86_64 
rhel-system-roles-1.22.0-0.6.el9.noarch
python3-blivet-1:3.6.0-5.el9.noarch

How reproducible:


Steps to Reproduce:
1. enable fips
2. ansible-playbook -vv -i host tests_luks.yml
3.

Actual results:


Expected results:


Additional info:

ok: [localhost] => {
    "storage_pools": [
        {
            "disks": [
                "sdb"
            ],
            "encryption": true,
            "encryption_cipher": "serpent-xts-plain64",
            "encryption_key_size": 512,
            "encryption_luks_version": "luks1",
            "encryption_password": "yabbadabbadoo",
            "name": "foo",
            "type": "lvm",
            "volumes": [
                {
                    "mount_point": "/opt/test1",
                    "name": "test1",
                    "size": "4g"
                }
            ]
        }
    ]
}


TASK [rhel-system-roles.storage : Failed message] ******************************
task path: /usr/share/ansible/roles/rhel-system-roles.storage/tasks/main-blivet.yml:95
fatal: [localhost]: FAILED! => {"changed": false, "msg": {"actions": [], "changed": false, "crypts": [], "failed": true, "invocation": {"module_args": {"disklabel_type": null, "diskvolume_mkfs_option_map": {}, "packages_only": false, "pool_defaults": {"disks": [], "encryption": false, "encryption_cipher": null, "encryption_key": null, "encryption_key_size": null, "encryption_luks_version": null, "encryption_password": null, "raid_chunk_size": null, "raid_device_count": null, "raid_level": null, "raid_metadata_version": null, "raid_spare_count": null, "state": "present", "type": "lvm", "volumes": []}, "pools": [{"disks": ["sdb"], "encryption": true, "encryption_cipher": "serpent-xts-plain64", "encryption_key": null, "encryption_key_size": 512, "encryption_luks_version": "luks1", "encryption_password": "yabbadabbadoo", "name": "foo", "raid_chunk_size": null, "raid_device_count": null, "raid_level": null, "raid_metadata_version": null, "raid_spare_count": null, "state": "present", "type": "lvm", "volumes": [{"_device": "/dev/mapper/foo-test1", "_mount_id": "/dev/mapper/foo-test1", "_raw_device": "/dev/mapper/foo-test1", "cache_devices": [], "cache_mode": null, "cache_size": 0, "cached": false, "compression": null, "deduplication": null, "disks": [], "encryption": false, "encryption_cipher": null, "encryption_key": null, "encryption_key_size": null, "encryption_luks_version": null, "encryption_password": null, "fs_create_options": "", "fs_label": "", "fs_overwrite_existing": true, "fs_type": "xfs", "mount_check": 0, "mount_device_identifier": "uuid", "mount_options": "defaults", "mount_passno": 0, "mount_point": "/opt/test1", "name": "test1", "raid_chunk_size": null, "raid_device_count": null, "raid_disks": [], "raid_level": null, "raid_metadata_version": null, "raid_spare_count": null, "size": "4g", "state": "present", "thin": false, "thin_pool_name": null, "thin_pool_size": null, "type": "lvm", "vdo_pool_size": null}]}], "safe_mode": false, "use_partitions": null, "volume_defaults": {"cache_devices": [], "cache_mode": null, "cache_size": 0, "cached": false, "compression": null, "deduplication": null, "disks": [], "encryption": false, "encryption_cipher": null, "encryption_key": null, "encryption_key_size": null, "encryption_luks_version": null, "encryption_password": null, "fs_create_options": "", "fs_label": "", "fs_overwrite_existing": true, "fs_type": "xfs", "mount_check": 0, "mount_device_identifier": "uuid", "mount_options": "defaults", "mount_passno": 0, "mount_point": "", "raid_chunk_size": null, "raid_device_count": null, "raid_level": null, "raid_metadata_version": null, "raid_spare_count": null, "size": 0, "state": "present", "thin": null, "thin_pool_name": null, "thin_pool_size": null, "type": "lvm", "vdo_pool_size": null}, "volumes": []}}, "leaves": [], "mounts": [], "msg": "Failed to commit changes to disk: Failed to format device: Input/output error", "packages": ["cryptsetup", "lvm2", "xfsprogs"], "pools": [], "volumes": []}}

TASK [rhel-system-roles.storage : Unmask the systemd cryptsetup services] ******
task path: /usr/share/ansible/roles/rhel-system-roles.storage/tasks/main-blivet.yml:99
skipping: [localhost] => {"changed": false, "skipped_reason": "No items in the list"}

PLAY RECAP *********************************************************************
localhost                  : ok=290  changed=9    unreachable=0    failed=1    skipped=205  rescued=7    ignored=0   
STDERR:
RETURN:2


https://beaker.engineering.redhat.com/recipes/13967235#task160735701

reproduce with rhel9.2
https://beaker.engineering.redhat.com/recipes/13967195#task160734998

Comment 5 guazhang@redhat.com 2023-05-31 03:50:43 UTC
still hit the bug with rhel-system-roles-1.22.0-0.8.el9.noarch 

https://beaker.engineering.redhat.com/recipes/14000454#task161033679



TASK [rhel-system-roles.storage : Failed message] ******************************
task path: /usr/share/ansible/roles/rhel-system-roles.storage/tasks/main-blivet.yml:95
fatal: [localhost]: FAILED! => {"changed": false, "msg": {"actions": [], "changed": false, "crypts": [], "failed": true, "invocation": {"module_args": {"disklabel_type": null, "diskvolume_mkfs_option_map": {}, "packages_only": false, "pool_defaults": {"disks": [], "encryption": false, "encryption_cipher": null, "encryption_key": null, "encryption_key_size": null, "encryption_luks_version": null, "encryption_password": null, "raid_chunk_size": null, "raid_device_count": null, "raid_level": null, "raid_metadata_version": null, "raid_spare_count": null, "state": "present", "type": "lvm", "volumes": []}, "pools": [{"disks": ["sda"], "encryption": false, "encryption_cipher": null, "encryption_key": null, "encryption_key_size": null, "encryption_luks_version": null, "encryption_password": null, "name": "foo", "raid_chunk_size": null, "raid_device_count": null, "raid_level": null, "raid_metadata_version": null, "raid_spare_count": null, "state": "present", "type": "lvm", "volumes": [{"_device": "/dev/mapper/luks-test1", "_mount_id": "/dev/mapper/luks-test1", "_raw_device": "/dev/mapper/foo-test1", "cache_devices": [], "cache_mode": null, "cache_size": 0, "cached": false, "compression": null, "deduplication": null, "disks": [], "encryption": true, "encryption_cipher": "serpent-xts-plain64", "encryption_key": null, "encryption_key_size": 512, "encryption_luks_version": "luks1", "encryption_password": "yabbadabbadoo", "fs_create_options": "", "fs_label": "", "fs_overwrite_existing": true, "fs_type": "xfs", "mount_check": 0, "mount_device_identifier": "uuid", "mount_group": null, "mount_mode": null, "mount_options": "defaults", "mount_passno": 0, "mount_point": "/opt/test1", "mount_user": null, "name": "test1", "raid_chunk_size": null, "raid_device_count": null, "raid_disks": [], "raid_level": null, "raid_metadata_version": null, "raid_spare_count": null, "size": "4g", "state": "present", "thin": false, "thin_pool_name": null, "thin_pool_size": null, "type": "lvm", "vdo_pool_size": null}]}], "safe_mode": false, "use_partitions": null, "volume_defaults": {"cache_devices": [], "cache_mode": null, "cache_size": 0, "cached": false, "compression": null, "deduplication": null, "disks": [], "encryption": false, "encryption_cipher": null, "encryption_key": null, "encryption_key_size": null, "encryption_luks_version": null, "encryption_password": null, "fs_create_options": "", "fs_label": "", "fs_overwrite_existing": true, "fs_type": "xfs", "mount_check": 0, "mount_device_identifier": "uuid", "mount_options": "defaults", "mount_passno": 0, "mount_point": "", "raid_chunk_size": null, "raid_device_count": null, "raid_level": null, "raid_metadata_version": null, "raid_spare_count": null, "size": 0, "state": "present", "thin": null, "thin_pool_name": null, "thin_pool_size": null, "type": "lvm", "vdo_pool_size": null}, "volumes": []}}, "leaves": [], "mounts": [], "msg": "Failed to commit changes to disk: Failed to format device: Input/output error", "packages": ["cryptsetup", "xfsprogs", "lvm2"], "pools": [], "volumes": []}}

Comment 6 Rich Megginson 2023-05-31 14:55:04 UTC
Cannot reproduce - https://beaker.engineering.redhat.com/recipes/14004781#task161062076

wow rhel-9.3 --ignore-panic --arch x86_64 --taskparam=VERSIONLOCK=true --brew-task 52982920 --taskparam=SYSTEM_ROLES_ONLY_TESTS=storage/tests_luks.yml --taskparam=ANSIBLE_VER=2 --fips --brew-method=multi --task 'git://pkgs.devel.redhat.com/tests/rhel-system-roles?master#Sanity/basic-smoke-test' --reservesys-if-warn --whiteboard 'System Roles testing build task 52982920 storage/tests_luks.yml legacy role ansible 2 FIPS'

One difference is that the system roles basic-smoke-test uses an inventory file that contains the following:

myself ansible_connection=local storage_udevadm_trigger=true

e.g. if this is in a file called /tmp/inventory, it does ansible-playbook -vv -i inventory tests/tests_luks.yml

Does your test pass if you use an inventory like this?

Comment 7 Rich Megginson 2023-05-31 15:10:46 UTC
I am now able to reproduce - https://beaker.engineering.redhat.com/recipes/14004781#task161062076 - investigating

Comment 8 Rich Megginson 2023-06-01 01:38:24 UTC
Looks like the problem is that the LUKS encryption cipher used by the test is not supported - changing to a supported cipher makes it work - https://github.com/linux-system-roles/storage/pull/361

Comment 10 guazhang@redhat.com 2023-06-13 08:51:43 UTC
can not reproduce the bug, but hit other bug bz2213691

Comment 12 errata-xmlrpc 2023-11-07 08:29:47 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (rhel-system-roles bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2023:6390