Bug 1691277

The commit below should fix the issue.  Until this fix is available, increase the max open files to 2048, or use a filter to accept only the PVs on the system.

Based on the info provided, every lvm command is reading 1569 devices to look for only 7 PVs (so there 1562 devices being unnecessarily read by every lvm command.)  In 8.1, lvm will improve this situation my keeping hints for itself to reduce unnecessary scanning, so most commands will read only the 7 PVs.  Until then, creating a filter to accept only the necessary PVs will make LVM commands much faster.


https://sourceware.org/git/?p=lvm2.git;a=commit;h=a991664dec02b62681a4a3eea7fbee2fc3a88700

commit a991664dec02b62681a4a3eea7fbee2fc3a88700
Author: David Teigland <teigland>
Date:   Fri Aug 24 14:46:51 2018 -0500

    bcache: reduce MAX_IO to 256
    
    This is the number of concurrent async io requests that
    the scan layer will submit to the bcache layer.  There
    will be an open fd for each of these, so it is best to
    keep this well below the default limit for max open files
    (1024), otherwise lvm may get EMFILE from open(2) when
    there are around 1024 devices to scan on the system.

diff --git a/lib/device/bcache.c b/lib/device/bcache.c
index b1f7d2a41bf7..531d83b10c03 100644
--- a/lib/device/bcache.c
+++ b/lib/device/bcache.c
@@ -196,7 +196,15 @@ static bool _async_issue(struct io_engine *ioe, enum dir d, int fd,
        return true;
 }
 
-#define MAX_IO 1024
+/*
+ * MAX_IO is returned to the layer above via bcache_max_prefetches() which
+ * tells the caller how many devices to submit io for concurrently.  There will
+ * be an open file descriptor for each of these, so keep it low enough to avoid
+ * reaching the default max open file limit (1024) when there are over 1024
+ * devices being scanned.
+ */
+
+#define MAX_IO 256
 #define MAX_EVENT 64
 
 static bool _async_wait(struct io_engine *ioe, io_complete_fn fn)

Even with the commit above, I'm still seeing some of the same errors in some tests with many devices.  So, I suspect there may be a second fix needed.

> Even with the commit above, I'm still seeing some of the same errors in some
> tests with many devices.  So, I suspect there may be a second fix needed.

The errors were due to testing with 1000 PVs and 1000 other non-lvm devices.  In this case, lvm does run into the open file limit as expected because it keeps each PV open (1000 fd's), and then needs to open 64 more non-lvm devices to scan for other PVs.  This goes past the default 1024 limit.  A safe current limit for the number of PVs that lvm can handle is around 900.  LVM should fix this by increasing the limit when it gets near.

For this particular report, the fix in comment 5 should be sufficient since there are only seven PVs.

This is already fixed in the recent 7.7 build

I just wanted to point out that the maximum supported limits for RHEL 7 documented here:

https://access.redhat.com/articles/rhel-limits#storage-11

As 10,000 which is a lot more than 900 (and there's the possibility of much larger amounts of paths being supported if you read the notes at the bottom) - if that's the supported limit shouldn't the LVM commands work within a supported limit in a default environment? Should instead of this the command look at current ulimits for open files and then do the opening of files and I/O in batches so it can work within the current ulimits of the process? It could complain if it's too low. It should really cope with the resource limits it is given on a default install shouldn't it?

Alternatively you would have to change the hard/soft resource limits in the command that needed to open a lot more files using setrlimit (which I'd argue would be unexpected). If it hits the issue just have it print a message explaining what the issue is and that the user must manually increase the resource limit for the command to run (see below though on large systems I expect that may end up breaking something when the issue gets hit).

My colleagues and I regularly see systems with more than 4000 sd devices presented to a system (I saw a customer system with 5000+ sd devices earlier today). It's not very unusual on large systems with TiBs of memory and hundreds of CPU cores. I don't think the implications of the change haven't been considered enough in terms of impact to systems with very large LUN count configurations.

> As 10,000 which is a lot more than 900 (and there's the possibility of much
> larger amounts of paths being supported if you read the notes at the bottom)
> - if that's the supported limit shouldn't the LVM commands work within a
> supported limit in a default environment? Should instead of this the command
> look at current ulimits for open files and then do the opening of files and
> I/O in batches so it can work within the current ulimits of the process? It
> could complain if it's too low. It should really cope with the resource
> limits it is given on a default install shouldn't it?

The problem described above is that the "small batches" were 1024, which were not small enough to fit within default open file limits.  Now the small batches are 256.  There is no inherent limit on the number of small batches that lvm can process.  Checking the current ulimits, and tuning sizes dynamically is a good suggestion, and might be a further improvement to look at.

If the batch sizes have been reduced in lvm2-7:2.02.180-10.el7_6.8 then it doesn't appear to be helping as I replicated this problem using that version.  The changelog shows:

* Mon May 13 2019 Marian Csontos <mcsontos> - 7:2.02.180-10.el7_6.8
- Set updating before scanning in lvmetad to avoid overwriting list of already
  seen devices.

* Tue Apr 09 2019 Marian Csontos <mcsontos> - 7:2.02.180-10.el7_6.7
- Add io_memory_size configuration option.
- Warn when metadata aproaches io_memory_size.
- Ensure migration_threshold for cache is at least 8 chunks.

* Thu Apr 04 2019 Marian Csontos <mcsontos> - 7:2.02.180-10.el7_6.6
- Reduce max concurrent aios to avoid EMFILE with many devices.              <<<<<-----
........


I'm unclear as to how the number of AIOs helps if all the devices are still open..??  The EMFILE is a device open problem not an IO issue as such.  Am I missing something..??





My replication steps:


### Create some devices:

for i in `seq 1 1200`
do
   j=$(printf "%04d" $i); echo $j;
   dmsetup create 17MB-zero_$j  --table '0 34816 zero'
done



### Now try the patched packages:

[root@alf-sdflex newer]# rpm -Uvh *
Preparing...                          ################################# [100%]
Updating / installing...
    1:device-mapper-libs-7:1.02.149-10.################################# [  8%]
    2:device-mapper-7:1.02.149-10.el7_6################################# [ 17%]
    3:device-mapper-event-libs-7:1.02.1################################# [ 25%]
    4:device-mapper-event-7:1.02.149-10################################# [ 33%]
    5:lvm2-libs-7:2.02.180-10.el7_6.8  ################################# [ 42%]
    6:lvm2-7:2.02.180-10.el7_6.8       ################################# [ 50%]
Cleaning up / removing...
    7:lvm2-7:2.02.180-8.el7            ################################# [ 58%]
    8:lvm2-libs-7:2.02.180-8.el7       ################################# [ 67%]
    9:device-mapper-event-7:1.02.149-8.################################# [ 75%]
   10:device-mapper-event-libs-7:1.02.1################################# [ 83%]
   11:device-mapper-libs-7:1.02.149-8.e################################# [ 92%]
   12:device-mapper-7:1.02.149-8.el7   ################################# [100%]
[root@alf-sdflex newer]#


### Errors once more:

[root@alf-sdflex newer]# pvscan
......
   Device open /dev/mapper/17MB-zero_0457 253:462 failed errno 24
   Device open /dev/mapper/17MB-zero_0457 253:462 failed errno 24
   Error opening device /dev/mapper/17MB-zero_0457 for reading at 0 length 512.
   Device open /dev/mapper/17MB-zero_0457 253:462 failed errno 24
   Device open /dev/mapper/17MB-zero_0457 253:462 failed errno 24
   Error opening device /dev/mapper/17MB-zero_0457 for reading at 0 length 4.
.......



### All happy again with the increased soft limit

[root@alf-sdflex newer]# ulimit -Sn 4096
[root@alf-sdflex newer]# pvscan
   PV /dev/md126p3   VG rhel00          lvm2 [423.57 GiB / 0    free]
   PV /dev/sdc2      VG system          lvm2 [1.09 TiB / 1.01 TiB free]
   Total: 2 [1.50 TiB] / in use: 2 [1.50 TiB] / in no VG: 0 [0   ]
[root@alf-sdflex newer]#

The problem caused by submitting reads to too many devices at once should be fixed by the MAX_IO 256 patch.  That problem would lead to going over the open file limit even when there were very few PVs on the system.

If there are less than around 900 LVM PVs on the system, lvm should not be hitting the open file limit, even if there are thousands of other non-PV devices on the system.  

LVM keeps PVs open, but closes non-PVs immediately.  So, the open file limit should only be an issue with somewhere between 900 and 1000 PVs on a system.  It's possible that there is some configuration difference that I'm not aware of that could be causing lvm to keep devs open, could you attach your lvm.conf?

In comments 14 and 16, it looks like the large number of devices are not LVM PVs, so the open file limit should not be causing a problem.

On my system I have set up 2001 non-PV loop devices and 902 PVs.  'pvs' reports all the PVs correctly without hitting any open file limits.

# losetup -l | wc -l
2902

# pvs | wc -l
904

# strace -c -e io_submit pvs > /dev/null
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
100.00    0.084574          25      3413           io_submit

# pvs --version
  LVM version:     2.02.180(2)-RHEL7 (2018-07-20)
  Library version: 1.02.149-RHEL7 (2018-07-20)

# ulimit -n
1024

I verified that lvm.conf is the default one.
When you setup duplication using mapper (dmsetup) as Darren suggested above, it's easily duplicable.

strace shows we open/close/open. The second open is the async one (before io_submit).
MAX_IO may be limited concurrent IOs but not concurrent fd's open/cached.

One device (near the end - open = 1022):

156951 open("/sys/devices/virtual/block/dm-205/uevent", O_RDONLY|O_CLOEXEC) = 1022
156951 fstat(1022, {st_dev=makedev(0, 17), st_ino=176353, st_mode=S_IFREG|0644, st_nlink=1, st_uid=0, st_gid=0, st_blksize=4096, st_blocks=0, st_size=4096, st_atime=2019/06/12-10:33:22.323403883, st_mtime=2019/06/12-10:33:22.323403883, st_ctime=2019/06/12-10:33:22.323403883}) = 0
156951 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fde15371000
156951 read(1022, "MAJOR=253\nMINOR=205\nDEVNAME=dm-205\nDEVTYPE=disk\n", 4096) = 48
156951 read(1022, "", 4096)             = 0
156951 close(1022)                      = 0
156951 munmap(0x7fde15371000, 4096)     = 0
156951 open("/run/udev/data/b253:205", O_RDONLY|O_CLOEXEC) = 1022
156951 fstat(1022, {st_dev=makedev(0, 19), st_ino=679089, st_mode=S_IFREG|0644, st_nlink=1, st_uid=0, st_gid=0, st_blksize=4096, st_blocks=8, st_size=271, st_atime=2019/06/14-05:13:54.921638881, st_mtime=2019/06/12-10:33:22.329403884, st_ctime=2019/06/12-10:33:22.329403884}) = 0
156951 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fde15371000
156951 read(1022, "S:disk/by-id/dm-name-17MB-zero_0200\nS:mapper/17MB-zero_0200\nW:3410\nI:2906817857\nE:DM_ACTIVATION=1\nE:DM_NAME=17MB-zero_0200\nE:DM_"..., 4096) = 271
156951 read(1022, "", 4096)             = 0
156951 close(1022)                      = 0
156951 munmap(0x7fde15371000, 4096)     = 0
156951 sendto(3, "<15>Jun 14 05:13:56 lvm[156951]: dm status   (253:205) [ noopencount noflush ]   [16384] (*1)", 93, MSG_NOSIGNAL, NULL, 0) = 93
156951 ioctl(8, DM_TABLE_STATUS, 0x564eb99fd390) = 0
156951 sendto(3, "<15>Jun 14 05:13:56 lvm[156951]: filter caching good /dev/mapper/17MB-zero_0200", 79, MSG_NOSIGNAL, NULL, 0) = 79
156951 sendto(3, "<15>Jun 14 05:13:56 lvm[156951]: dm status   (253:461) [ noopencount noflush ]   [16384] (*1)", 93, MSG_NOSIGNAL, NULL, 0) = 93
156951 ioctl(8, DM_TABLE_STATUS, 0x564eb99fd530) = 0
156951 stat("/dev/mapper/17MB-zero_0456", {st_dev=makedev(0, 5), st_ino=500414, st_mode=S_IFBLK|0660, st_nlink=1, st_uid=0, st_gid=6, st_blksize=4096, st_blocks=0, st_rdev=makedev(253, 461), st_atime=2019/06/12-10:33:24.740404029, st_mtime=2019/06/12-10:33:24.740404029, st_ctime=2019/06/12-10:33:24.740404029}) = 0
156951 open("/dev/mapper/17MB-zero_0456", O_RDONLY|O_DIRECT|O_NOATIME) = 1022
156951 fstat(1022, {st_dev=makedev(0, 5), st_ino=500414, st_mode=S_IFBLK|0660, st_nlink=1, st_uid=0, st_gid=6, st_blksize=4096, st_blocks=0, st_rdev=makedev(253, 461), st_atime=2019/06/12-10:33:24.740404029, st_mtime=2019/06/12-10:33:24.740404029, st_ctime=2019/06/12-10:33:24.740404029}) = 0
156951 sendto(3, "<15>Jun 14 05:13:56 lvm[156951]: Opened /dev/mapper/17MB-zero_0456 RO O_DIRECT", 78, MSG_NOSIGNAL, NULL, 0) = 78
156951 ioctl(1022, BLKGETSIZE64, [17825792]) = 0
156951 sendto(3, "<14>Jun 14 05:13:56 lvm[156951]: /dev/mapper/17MB-zero_0456: size is 34816 sectors", 82, MSG_NOSIGNAL, NULL, 0) = 82
156951 close(1022)                      = 0
156951 sendto(3, "<15>Jun 14 05:13:56 lvm[156951]: Closed /dev/mapper/17MB-zero_0456", 66, MSG_NOSIGNAL, NULL, 0) = 66
156951 open("/dev/mapper/17MB-zero_0456", O_RDONLY|O_DIRECT|O_NOATIME) = 1022
156951 io_submit(140591815401472, 1, [{pread, fildes=1022, buf=0x564eb97d1000, nbytes=131072, offset=0}]) = 1
156951 io_getevents(140591815401472, 1, 64, [{data=0, obj=0x564eb90eb1b8, res=131072, res2=0}], NULL) = 1
156951 sendto(3, "<14>Jun 14 05:13:56 lvm[156951]: /dev/mapper/17MB-zero_0456: using cached size 34816 sectors", 92, MSG_NOSIGNAL, NULL, 0) = 92
156951 readlink("/sys/dev/block/253:461", "../../devices/virtual/block/dm-461", 1024) = 34
156951 stat("/sys/devices/virtual/block/dm-461/uevent", {st_dev=makedev(0, 17), st_ino=198667, st_mode=S_IFREG|0644, st_nlink=1, st_uid=0, st_gid=0, st_blksize=4096, st_blocks=0, st_size=4096, st_atime=2019/06/12-10:33:24.736404029, st_mtime=2019/06/12-10:33:24.736404029, st_ctime=2019/06/12-10:33:24.736404029}) = 0
156951 readlink("/sys/devices/virtual/block/dm-461/subsystem", "../../../../class/block", 1024) = 23
156951 open("/sys/devices/virtual/block/dm-461/uevent", O_RDONLY|O_CLOEXEC) = 1023


// chosen one of them near the begining and checked for open/close pairing
[root@alf-sdflex ~]# grep -n -E "= 244$|close\(244\)" pvscand.str
52675:156951 open("/sys/devices/virtual/block/dm-45/uevent", O_RDONLY|O_CLOEXEC) = 244
52680:156951 close(244)                       = 0
52682:156951 open("/run/udev/data/b253:45", O_RDONLY|O_CLOEXEC) = 244
52687:156951 close(244)                       = 0
52695:156951 open("/dev/mapper/17MB-zero_0296", O_RDONLY|O_DIRECT|O_NOATIME) = 244
52700:156951 close(244)                       = 0
52702:156951 open("/dev/mapper/17MB-zero_0296", O_RDONLY|O_DIRECT|O_NOATIME) = 244
88854:156951 close(244)                       = 0

// the first close when exiting is  at line 88617, so the last close() is from exit
// counted closes at exit time:
# grep close last.close | wc -l
1024

It's a problem in the lvmetad code which I don't see since I never have lvmetad enabled.  Please try setting use_lvmetad=0 in lvm.conf and see if that helps.  I'll work on finding a solution.

I fixed this in Aug 2018 but I guess the fix has not made its way into rhel7 yet.

https://sourceware.org/git/?p=lvm2.git;a=commitdiff;h=5502f72e41dceb85d11ec562819ad6c5db6758ae

Confirmed - without lvmetad the pvscan passes fine.

Any updates?

Our customer has done the test to see how the changes of lvmetad affect this:
=======
I cherry-picked David Teiglands most relevant commits and patched the SRPM of the current LVM2 release in EL7.6  
( Version     : 2.02.180 , Release     : 10.el7_6.8). 
My last test included 2 patches:
Patch53: lvm2-2_02_180-pvscan-should-seed-devicelist-from-lvmetad-when-enabled.patch
Which is basically this one: https://sourceware.org/git/?p=lvm2.git;a=commitdiff;h=5502f72e41dceb85d11ec562819ad6c5db6758ae 
Patch54: lvm2-2_02_180-bcache-max-prefetches-256.patch
Which is this one: https://sourceware.org/git/?p=lvm2.git;a=commitdiff;h=a991664dec02b62681a4a3eea7fbee2fc3a88700
So there’s more to it than just these 2, according to my tests.

With lvmetad enabled, a pvscan -vv still fails:
<snip />
  Device open /dev/vg_el1232_root/lv_block_1004 253:1010 failed errno 24
  Error opening device /dev/vg_el1232_root/lv_block_1004 for reading at 0 length 512.
  Device open /dev/vg_el1232_root/lv_block_1004 253:1010 failed errno 24
  Device open /dev/vg_el1232_root/lv_block_1004 253:1010 failed errno 24
  Error opening device /dev/vg_el1232_root/lv_block_1004 for reading at 0 length 4.
  Device open /dev/vg_el1232_root/lv_block_1004 253:1010 failed errno 24
  Device open /dev/vg_el1232_root/lv_block_1004 253:1010 failed errno 24
  Error opening device /dev/vg_el1232_root/lv_block_1004 for reading at 4096 length 4.
^C

real    5m11.968s

With lvmetad disabled:
[root@el1232 RPMs]# time pvscan 2>/dev/null
  PV /dev/sda2   VG vg_el1232_root   lvm2 [278.84 GiB / 222.84 GiB free]
  PV /dev/sdf                        lvm2 [10.03 GiB]
  PV /dev/sdg                        lvm2 [10.03 GiB]
  Total: 3 [298.91 GiB] / in use: 1 [278.84 GiB] / in no VG: 2 [<20.07 GiB]

real    0m0.755s
user    0m0.328s
sys     0m0.407s


We’ll run without lvmetad for now and let engineering work on it. 
I would like to invite Red Hat to use my reproducer as David Teigland was convinced an earlier patch fixes the issue. 
Or I might have erred somewhere, of course.
=======

I cannot reproduce the problem using the stable branch of lvm upstream code.  This is going into 7.7.

lvm2-2.02.185-2.el7 is an early rhel 7.7 rpm that contains all the necessary fixes.  Someone in the support group should be able to get this to you to test to it fixes the issue.

I verified that I can dup on another rhel7.6 system using Darren's instructions and then installed lvm2-2.02.185-1.el7.x86_64+deps from RHEL7.7_snapshot3.
It no longer emits any errors from pvscan with that version.
Stan

Fix verified in the latest rpms.


3.10.0-1057.el7.x86_64

lvm2-2.02.180-10.el7    BUILT: Wed Oct 10 11:25:21 CDT 2018
lvm2-libs-2.02.180-10.el7    BUILT: Wed Oct 10 11:25:21 CDT 2018
lvm2-cluster-2.02.180-10.el7    BUILT: Wed Oct 10 11:25:21 CDT 2018
lvm2-lockd-2.02.180-10.el7    BUILT: Wed Oct 10 11:25:21 CDT 2018
lvm2-python-boom-0.9-18.el7    BUILT: Fri Jun 21 04:18:58 CDT 2019
cmirror-2.02.180-10.el7    BUILT: Wed Oct 10 11:25:21 CDT 2018
device-mapper-1.02.149-10.el7    BUILT: Wed Oct 10 11:25:21 CDT 2018
device-mapper-libs-1.02.149-10.el7    BUILT: Wed Oct 10 11:25:21 CDT 2018
device-mapper-event-1.02.149-10.el7    BUILT: Wed Oct 10 11:25:21 CDT 2018
device-mapper-event-libs-1.02.149-10.el7    BUILT: Wed Oct 10 11:25:21 CDT 2018
device-mapper-persistent-data-0.8.5-1.el7    BUILT: Mon Jun 10 03:58:20 CDT 2019



[root@hayes-01 tmp]# losetup -l | wc -l
1309
[root@hayes-01 tmp]# pvs | wc -l
1302
[root@hayes-01 tmp]# ulimit -n
1024

[root@hayes-01 tmp]# pvscan | more
  PV /dev/loop78                        lvm2 [16.00 MiB]
  PV /dev/loop88                        lvm2 [16.00 MiB]
  PV /dev/loop193                       lvm2 [16.00 MiB]
  [...]
  PV /dev/loop1197                      lvm2 [16.00 MiB]
  PV /dev/loop1266                      lvm2 [16.00 MiB]
  PV /dev/loop1289                      lvm2 [16.00 MiB]
  Total: 1301 [<20.33 GiB] / in use: 0 [0   ] / in no VG: 1301 [<20.33 GiB]
[root@hayes-01 tmp]# echo $?
0


[root@hayes-01 tmp]# systemctl start lvm2-lvmetad
[root@hayes-01 tmp]# systemctl status lvm2-lvmetad
â— lvm2-lvmetad.service - LVM2 metadata daemon
   Loaded: loaded (/usr/lib/systemd/system/lvm2-lvmetad.service; static; vendor preset: enabled)
   Active: active (running) since Mon 2019-06-24 14:01:41 CDT; 24h ago
     Docs: man:lvmetad(8)
 Main PID: 1040 (lvmetad)
   CGroup: /system.slice/lvm2-lvmetad.service
           └─1040 /usr/sbin/lvmetad -f

Jun 24 14:01:41 hayes-01.lab.msp.redhat.com systemd[1]: Started LVM2 metadata daemon.

[root@hayes-01 tmp]# pvs 
[...]
  /dev/loop995     lvm2 ---  16.00m 16.00m
  /dev/loop996     lvm2 ---  16.00m 16.00m
  /dev/loop997     lvm2 ---  16.00m 16.00m
  /dev/loop998     lvm2 ---  16.00m 16.00m
  /dev/loop999     lvm2 ---  16.00m 16.00m
[root@hayes-01 tmp]# echo $?
0



Downgrading to lvm2-2.02.180-10.el7 quickly shows the bug again:

[root@hayes-01 tmp]# pvs
[...]
  Device open /dev/loop1278 7:1278 failed errno 24
  Device open /dev/loop255 7:255 failed errno 24
  Device open /dev/loop255 7:255 failed errno 24
  Device open /dev/loop511 7:511 failed errno 24
[...]
  WARNING: PV h6xUI5-DcDP-kqKt-ko2Y-5vV9-L0Rp-JMOWvc prefers device /dev/loop6 because of previous preference.
  /run/lock/lvm/P_orphans:aux: open failed: Too many open files
  SELinux context reset: setfscreatecon failed: Too many open files
  Can't get lock for #orphans_lvm1
  Cannot process standalone physical volumes
  /run/lock/lvm/P_orphans:aux: open failed: Too many open files
  SELinux context reset: setfscreatecon failed: Too many open files
  Can't get lock for #orphans_pool
  Cannot process standalone physical volumes
  /run/lock/lvm/P_orphans:aux: open failed: Too many open files
  SELinux context reset: setfscreatecon failed: Too many open files
  Can't get lock for #orphans_lvm2
  Cannot process standalone physical volumes
  /run/lock/lvm/P_orphans:aux: open failed: Too many open files
  SELinux context reset: setfscreatecon failed: Too many open files
  Can't get lock for #orphans_lvm1
  Cannot process standalone physical volumes
  /run/lock/lvm/P_orphans:aux: open failed: Too many open files
  SELinux context reset: setfscreatecon failed: Too many open files
  Can't get lock for #orphans_pool
  Cannot process standalone physical volumes
  /run/lock/lvm/P_orphans:aux: open failed: Too many open files
  SELinux context reset: setfscreatecon failed: Too many open files
  Can't get lock for #orphans_lvm2
  Cannot process standalone physical volumes
[root@hayes-01 tmp]# echo $?
5

*** Bug 1727438 has been marked as a duplicate of this bug. ***

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:2253

*** Bug 1694632 has been marked as a duplicate of this bug. ***