Bug 1854035

Summary: Rebase PCP to 5.2.0+
Product: Red Hat Enterprise Linux 8 Reporter: Nathan Scott <nathans>
Component: pcpAssignee: Nathan Scott <nathans>
Status: CLOSED ERRATA QA Contact: Jan Kurik <jkurik>
Severity: high Docs Contact: Apurva Bhide <abhide>
Priority: high    
Version: 8.4CC: agerstmayr, bugproxy, byodlows, hannsj_uhl, jamills, jkurik, lilhuang, mgoodwin, mnewsome, nathans, patrickm, rmetrich, santhosh_m_r, shangsong1, ybabar
Target Milestone: rcKeywords: FutureFeature, Rebase, Triaged, ZStream
Target Release: 8.4Flags: pm-rhel: mirror+
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: pcp-5.2.5-2.el8 Doc Type: Enhancement
Doc Text:
.`pcp` rebased to version 5.2.5 The `pcp` package has been upgraded to version 5.2.5. Notable changes include: * SQL Server metrics support via a secure connection. * `eBPF/BCC` netproc module with per-process network metrics. * `pmdaperfevent(1)` support for the `hv_24x7 core-level` and `hv_gpci` event metrics. * New Linux process accounting metrics, Linux ZFS metrics, Linux XFS metric, Linux kernel socket metrics, Linux multipath TCP metrics, Linux memory and ZRAM metrics, and S.M.A.R.T. metric support for NVM Express disks. * New `pcp-htop(1)` utility to visualize the system and process metrics. * New pmrepconf(1) utility to generate the `pmrep/pcp2xxx` configurations. * New `pmiectl(1)` utility for controlling the `pmie` services. * New `pmlogctl(1)` utility for controlling the `pmlogger` services. * New `pmlogpaste(1)` utility for writing log string metrics. * New `pcp-atop(1)` utility to process accounting statistics and per-process network statistics reporting. * New `pmseries(1)` utility to query functions, language extensions, and REST API. * New `pmie(1)` rules for detecting OOM kills and socket connection saturation. * Bug fixes in the `pcp-atopsar(1)`, `pcp-free(1)`, `pcp-dstat(1)`, `pmlogger(1)`, and `pmchart(1)` utilities. * REST API and C API support for per-context derived metrics. * Improved OpenMetrics metric metadata (units, semantics). * Rearranged installed `/var` file system layouts extensively.
 Story Points: ---
Clone Of:
: 1913654 1913655 (view as bug list) Environment:
Last Closed: 2021-05-18 15:19:32 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1670029, 1716242, 1814526, 1857121, 1875659, 1881685, 1889613, 1901130, 1913654, 1913655, 1930390    
Attachments:
Description Flags
journalctl file none

Description Nathan Scott 2020-07-06 06:55:45 UTC 
Regular rebase of PCP for bugfixes and features.  Features in the 5.2.x series include scalability improvements for larger pmlogger(1) and pmie(1) farms, and new tools like pmlogpaste(1) for improved pbench integration.

A number of RHEL and upstream issues are also resolved in this release, and it is backwards compatible with previous PCP releases.
Comment 2 Nathan Scott 2020-09-07 07:08:41 UTC 
*** Bug 1875659 has been marked as a duplicate of this bug. ***
Comment 3 Nathan Scott 2020-09-07 23:22:22 UTC 
*** Bug 1876452 has been marked as a duplicate of this bug. ***
Comment 5 Nathan Scott 2020-10-08 05:04:27 UTC 
*** Bug 1886258 has been marked as a duplicate of this bug. ***
Comment 6 Nathan Scott 2020-10-08 05:38:04 UTC 
*** Bug 1886258 has been marked as a duplicate of this bug. ***
Comment 7 IBM Bug Proxy 2020-11-05 08:54:12 UTC 
------- Comment From pradeep.com 2020-09-23 13:22 EDT-------
(In reply to comment #14)
> > @Red Hat: How do we enable additional debug with NetworkManager. What logs do you need upon failure?
>
> Please read hints about logging here:
> https://cgit.freedesktop.org/NetworkManager/NetworkManager/tree/contrib/
> fedora/rpm/NetworkManager.conf#n28
>
> - configure level=TRACE in /etc/NetworkManager/NetworkManager.conf as
> indicated.
> - best disable ratelimiting of journal first (see link above).
> - then reboot and reproduce.
>
> Provide the full journal output of the boot that shows the issue (where
> NetworkManager is not running).
> That is, attach the output of `journalctl -b 0 > my-logfile`
>
> Thank you.

Thank you. Will request test team follow these steps and get back here

------- Comment From cdeadmin.com 2020-09-29 06:14 EDT-------
New issue with Defect SW507878 opened during the recreates. We are trying to recreate this issue on another lpar and will update the defect.

------- Comment From cdeadmin.com 2020-09-30 11:44 EDT-------
I have tried to recreate the issue with another lpar.
disabled ratelimiting of journal.
In the 2nd iteration, lpar went to grub rescue prompt, after activation

HMC: vhmccloudtst64
lpar is in zzg4p312
lpar: zpfp20p01

----------------------------------------------------------
Partitions On Managed System:  zzg4p312
OS/400 Partitions not listed
----------------------------------------------------------
1)    zingv1                               Running
2)    zingv2                               Running
3)    zpfp20p01                            Running
4)    zpfp6p11                             Not Activated
5)    zzfp379p8                            Running

Enter Number of Running Partition (q to quit): 3
3
Opening Virtual Terminal On Partition zpfp20p01 . . .

Open in progress

Open Completed.

grub rescue>
grub rescue>

I have opened a new defect, SW508173,for this issue.

Thanks
Anitha

------- Comment From bjking1.com 2020-10-12 14:45 EDT-------
Waiting on recreate with tracing enabled.

------- Comment From cdeadmin.com 2020-10-27 02:34 EDT-------
#=#=# 2020-10-27 01:31:05 (CDT) #=#=#
Action = [reopen]

I reran inactive lpm test and after 93 iterations, nmcli issue is reproduced.
I have enabled trace and also collected journalctl on every reboot.
Setup:
HMC: vhmccloudtst60
==================Iteration: 93===================

Partition: zingp4
Source: mac2fp1
Destination: zpfp6

Validating.....
^[[1;32m
Validation Passed...

Starting migration of zingp4 from mac2fp1 to zpfp6

Migration Passed...
Approximate time taken to migrate zingp4 is: 0 minutes and 27 seconds

Powering on partitions....

Waiting for 15 minutes...

Trying ping test for each LPAR
zingp4 Not pinging
Some LPARs did not ping.. Exiting Script zingp4

Leaving system in Failed state...
#=#=#=#=#=#=#=#=#=#=#=#=#=#=#=#=#=#=#

------- Comment From cdeadmin.com 2020-10-27 03:24 EDT-------
Setup:
HMC: vhmc15
Systems: mac2fp1,zpfp6
lpar: zingp4

LPM Script failed after 93 iterations, as the lpar network went down.
When I logged in through vtmenu, I saw that nmcli c is not working.

(0) root @ zingp4: /root
# systemctl is-enabled NetworkManager
enabled

(0) root @ zingp4: /root
# systemctl is-active NetworkManager
inactive

(0) root @ zingp4: /root
# nmcli c
Error: NetworkManager is not running.

Before powering off lpar, network manager was active
--------------------LPAR NetworkManager status---------------------
enabled
active

NAME                       UUID                                  TYPE      DEVICE
bond521ddf7f               f2f65468-f364-4929-b945-ed4a6a556366  bond      bond521ddf7f
bond39f40574               40e72e87-8da8-4b6d-a74b-a4c8da74ec4e  bond      bond39f40574
bond3b5b2bf7               b81586b0-0476-45b8-8015-841ab5bf5b88  bond      bond3b5b2bf7
bond48acb8df               b05ba086-d679-4840-b913-7efe2cf028af  bond      bond48acb8df
bond49b1795b               5d56fb58-fc8a-4263-a401-b1bc030c5a4b  bond      bond49b1795b
bond62de55eb               f503fea2-8e5c-4f91-8226-16f7ca0e1c7c  bond      bond62de55eb
bond62ef7412               e743e228-f890-49b7-818e-5fe8634d2e02  bond      bond62ef7412
bond6ad1eacf               837c0c5a-4151-4572-802f-8c7dccb51967  bond      bond6ad1eacf
bond7fb850a5               a1c98b6a-c86c-4c65-8341-2f39ef5770e8  bond      bond7fb850a5
virbr0                     d0e371c3-1453-4f6c-91a9-ed274fdd785c  bridge    virbr0
bond39f40574-net0          db265c3b-454b-4ead-b253-b3cd41e009bc  ethernet  net0
bond3b5b2bf7-env10         d2fed6b6-07c7-4199-90e0-5bd58817e399  ethernet  env10
bond48acb8df-env7          112dd8e6-a5bc-4da3-95fa-5a675c4f5d16  ethernet  env7
bond49b1795b-env5          491b9184-23f6-406c-9ec9-f0536b84b869  ethernet  env5
bond521ddf7f-env4          ee983048-9318-4d14-99ab-e0ce5d11a747  ethernet  env4
bond62de55eb-env11         0fd63e91-9028-4494-ae20-7e19e0b26c70  ethernet  env11
bond62ef7412-env8          bc5bbddf-505d-4047-90b5-4fbe917f0f40  ethernet  env8
bond6ad1eacf-env6          22bdd579-69eb-44ff-bf95-c175082fa109  ethernet  env6
bond7fb850a5-env9          b6789f4e-1a33-43c5-b566-0440a03d1101  ethernet  env9
bond39f40574-enP16387p1s0  48bf9558-5127-42d1-9a22-535e494fc655  ethernet  --
bond3b5b2bf7-enP16422p1s0  1376a3ae-33a3-4427-82a6-2c4e1aa9cce5  ethernet  --
bond48acb8df-enP16398p1s0  f0d269b5-86c0-4a73-89db-b58316ed10a2  ethernet  --
bond49b1795b-enP16416p1s0  f585aaeb-c99c-429d-9777-2579764e4348  ethernet  --
bond521ddf7f-enP16450p1s0  65298a4a-5756-484f-9b55-bdccc035f1fd  ethernet  --
bond62de55eb-enP16446p1s0  32010561-f32b-48d3-be18-da03a4cb8292  ethernet  --
bond62ef7412-enP16413p1s0  7a558ffd-daf9-40d5-b36f-88f2e6197bc9  ethernet  --
bond6ad1eacf-enP16423p1s0  d700bbdc-4b21-4a81-8ac1-9bf83014cfd2  ethernet  --
bond7fb850a5-enP16415p1s0  fabcb2bb-5e20-4d2c-b266-fd6728a018dd  ethernet  --
--------------------
Sosreport collected on 27thOct -- /var/tmp/sosreport-zingp4-SW506462-2020-10-27-usjtepk.tar.xz
Journalctl collected on 27thOct -- 'journalctl_Tue Oct 27 03:09:00 EDT 2020'

journalctl logs, in lpar /root:
-rw-r--r--  1 root root 1649185 Oct 26 05:26 'journalctl_Mon Oct 26 05:26:44 EDT 2020'
-rw-r--r--  1 root root 1755916 Oct 26 06:30 'journalctl_Mon Oct 26 06:30:01 EDT 2020'
-rw-r--r--  1 root root 1624185 Oct 26 07:32 'journalctl_Mon Oct 26 07:32:13 EDT 2020'
-rw-r--r--  1 root root 1742178 Oct 26 08:35 'journalctl_Mon Oct 26 08:35:21 EDT 2020'
-rw-r--r--  1 root root 1639253 Oct 26 09:37 'journalctl_Mon Oct 26 09:37:32 EDT 2020'
-rw-r--r--  1 root root   99378 Oct 26 10:40 'journalctl_Mon Oct 26 10:40:35 EDT 2020'

------- Comment From pradeep.com 2020-10-27 11:48 EDT-------
Redhat:

Here is what I found in the journalctl file that maybe relevant to why NetworkManager didn't start up:

Oct 26 10:40:24 zingp4.aus.stglabs.ibm.com systemd[1]: network-online.target: Found ordering cycle on network.target/start
Oct 26 10:40:24 zingp4.aus.stglabs.ibm.com systemd[1]: network-online.target: Found dependency on NetworkManager.service/start
Oct 26 10:40:24 zingp4.aus.stglabs.ibm.com systemd[1]: network-online.target: Found dependency on dbus.service/start
Oct 26 10:40:24 zingp4.aus.stglabs.ibm.com systemd[1]: network-online.target: Found dependency on basic.target/start
Oct 26 10:40:24 zingp4.aus.stglabs.ibm.com systemd[1]: network-online.target: Found dependency on paths.target/start
Oct 26 10:40:24 zingp4.aus.stglabs.ibm.com systemd[1]: network-online.target: Found dependency on pmlogger_check.path/start

I will attach the complete journalctl file.

After "systemctl restart NetworkManager" things appear to work as expected.

------- Comment From cdeadmin.com 2020-10-27 11:58 EDT-------
cde00 (cdeadmin.com) added native attachment /tmp/AIXOS13325782/journalctl_Tue_Oct27 on 2020-10-27 10:59:12

------- Comment From pradeep.com 2020-10-27 13:36 EDT-------
(In reply to comment #25)
> (In reply to IBM Bug Proxy from comment #10)
>
> > network-online.target: Found ordering cycle on network.target/start
> > network-online.target: Found dependency on NetworkManager.service/start
> > network-online.target: Found dependency on dbus.service/start
> > network-online.target: Found dependency on basic.target/start
> > network-online.target: Found dependency on paths.target/start
> > network-online.target: Found dependency on pmlogger_check.path/start
>
> What seems strange is that pmlogger_check.path is a dependency of
> paths.target but also depends on network.target. Do have custom
> modifications to systemd units?

Not that I am aware of. This is standard RHEL 8.3 (Snapshot 1 if I am not mistaken) install

------- Comment From cdeadmin.com 2020-10-28 03:24 EDT-------
(0) root @ zingp4: /root
# ver
ver - OS, HTX, Firmware and Machine details

OS: GNU/Linux
OS Version: Red Hat Enterprise Linux 8.3 (Ootpa)
Kernel Version: 4.18.0-235.el8.ppc64le
HTX Version: htxrhel8-572-LE.ppc64le
Host Name: zingp4.aus.stglabs.ibm.com
Machine Serial No: IBM,0213295CX
Machine Type/Model: IBM,9040-MR9
System FW Level: FW950.00 (VM950_038)

(1) root @ zingp4: /root
# uname -a
Linux zingp4.aus.stglabs.ibm.com 4.18.0-235.el8.ppc64le #1 SMP Thu Sep 3 08:15:47 EDT 2020 ppc64le ppc64le ppc64le GNU/Linux

(0) root @ zingp4: /root
# rpm -qa powerpc-utils
powerpc-utils-1.3.6-11.106.el8.ppc64le

------- Comment From Mingming.Cao 2020-10-28 16:18 EDT-------
(In reply to comment #25)
> (In reply to IBM Bug Proxy from comment #10)
>
> > network-online.target: Found ordering cycle on network.target/start
> > network-online.target: Found dependency on NetworkManager.service/start
> > network-online.target: Found dependency on dbus.service/start
> > network-online.target: Found dependency on basic.target/start
> > network-online.target: Found dependency on paths.target/start
> > network-online.target: Found dependency on pmlogger_check.path/start
>
> What seems strange is that pmlogger_check.path is a dependency of
> paths.target but also depends on network.target. Do have custom
> modifications to systemd units?
>
> Reassigning to the pcp team for investigation.

The Hybrid Network has a new systemd service registered (via powerpc-utils)

(0) root @ zingp4: /root
# ls -al /usr/lib/systemd/system/hcn-init.service
-rw-r--r-- 1 root root 206 Oct  9 15:14 /usr/lib/systemd/system/hcn-init.service

Also I noticed that the htx tests also registered systemd service

Oct 26 10:40:24 zingp4.aus.stglabs.ibm.com systemd[1]: Configuration file /usr/lib/systemd/system/htx.d.service is marked executable. Please remove executable permission bits. Proceeding anyway.
Oct 26 10:40:24 zingp4.aus.stglabs.ibm.com systemd[1]: network-online.target: Found ordering cycle on network.target/start
Oct 26 10:40:24 zingp4.aus.stglabs.ibm.com systemd[1]: network-online.target: Found dependency on NetworkManager.service/start
Oct 26 10:40:24 zingp4.aus.stglabs.ibm.com systemd[1]: network-online.target: Found dependency on dbus.service/start
Oct 26 10:40:24 zingp4.aus.stglabs.ibm.com systemd[1]: network-online.target: Found dependency on basic.target/start
Oct 26 10:40:24 zingp4.aus.stglabs.ibm.com systemd[1]: network-online.target: Found dependency on paths.target/start
Oct 26 10:40:24 zingp4.aus.stglabs.ibm.com systemd[1]: network-online.target: Found dependency on pmlogger_check.path/start

# ls -al /usr/lib/systemd/system/htx.d.service
-rwxr-xr-x. 1 root root 217 Oct 12 13:26 /usr/lib/systemd/system/htx.d.service

attach the two service files for review

(0) root @ zingp4: /root
# cat /usr/lib/systemd/system/htx.d.service
[Unit]
Description=htx Daemon

[Service]
Type=forking
ExecStart=/usr/lpp/htx/etc/scripts/htxd_run
ExecStop=/usr/lpp/htx/etc/scripts/htxd_shutdown
TasksMax=infinity
TimeoutSec=300

[Install]
WantedBy=multi-user.target

(0) root @ zingp4: /root
# cat /usr/lib/systemd/system/hcn-init.service
[Unit]
Description=hybrid virtual network scan and config
After=network-online.target
Wants=network-online.target

[Service]
Type=oneshot
ExecStart=/usr/sbin/hcnmgr -s

[Install]
WantedBy=multi-user.target

(0) root @ zingp4: /roo

------- Comment From cdeadmin.com 2020-10-29 02:24 EDT-------
Is the bug number incorrect ?

Upon search for 1875659
You must enter a valid bug number!

------- Comment From pradeep.com 2020-10-29 15:40 EDT-------
(In reply to comment #31)
>
> Is the bug number incorrect ?
>
> Upon search for 1875659
> You must enter a valid bug number!

It appears that several of us don't have access to Red Hat bugzilla (even after logging in). Can you please tells us what was the resolution of the Duped bug?

------- Comment From pradeep.com 2020-11-04 10:54 EDT-------
(In reply to comment #33)
> (In reply to comment #31)
> >
> > Is the bug number incorrect ?
> >
> > Upon search for 1875659
> > You must enter a valid bug number!
>
> It appears that several of us don't have access to Red Hat bugzilla (even
> after logging in). Can you please tells us what was the resolution of the
> Duped bug?

Hello Red Hat, Now have access to Bug# 1881685, which is duped to bug# 1875659. Don't have access to this (bug# 1875659). Still don't have a clue as to what the resolution is. Can we please get some help?
Comment 8 IBM Bug Proxy 2020-11-05 09:40:33 UTC 
Created attachment 1726821 [details]
journalctl file
Comment 9 Hanns-Joachim Uhl 2020-11-05 09:54:19 UTC 
Hello Red Hat,
fyi ... please ignore comment #7 and comment #8 in this Red Hat bugzilla ...
... it's a mirroring artifact ... sorry for the noise ...
Thanks for your understanding.
Comment 10 IBM Bug Proxy 2020-11-10 07:10:32 UTC 
------- Comment From cdeadmin.com 2020-11-10 02:04 EDT-------
Re-Run: I applied the workaround (Remove PCP packages) mentioned in this defect.
Updated powerpc-utils 108 version and reran the test. We did not hit nmcli / Network Manager issue.

Test completed more than 3 days and completed 84 iterations.

==================Iteration: 84===================

Partition: gdlzzg4stc55p05
Source: gdlzzg4stc55fp1
Destination: gdlfw53fp1

Validating.....

Validation Passed...

Starting migration of gdlzzg4stc55p05 from gdlzzg4stc55fp1 to gdlfw53fp1

Migration Passed...
Approximate time taken to migrate gdlzzg4stc55p05 is: 0 minutes and 34 seconds

Powering on partitions....

Waiting for 15 minutes...

Trying ping test for each LPAR
Starting HTX on each LPAR..

--------------------LPAR NetworkManager status---------------------
enabled
active

NAME                       UUID                                  TYPE      DEVICE
bond1faacd59               6d89166e-a4fc-4df2-b0d1-0e4d05549f16  bond      bond1faacd59
bond0ac56b2c               d4fd596c-21d6-4ba1-aff3-92fb9e8e0790  bond      bond0ac56b2c
bond1121ea56               ffb36062-3111-4d6d-b85f-db91f50d651a  bond      bond1121ea56
bond2962530a               b0bb10c7-e4f6-4a85-b7d8-6c7a2bc5c167  bond      bond2962530a
bond3086d6df               ec5eca13-0e95-4cb1-b0db-9c9cc2627d7d  bond      bond3086d6df
bond355aeefc               1f7d8ba6-8e25-48f9-9c4a-a790bea4e1d0  bond      bond355aeefc
bond3b385933               37144f3c-b3b3-46b9-a7f3-41c6b1f4a41e  bond      bond3b385933
bond3d06e19b               1fe3b92d-a104-41d7-9093-ad34c62063c7  bond      bond3d06e19b
bond65e3d3f5               445b2cfd-409c-4908-8c76-0b786db57e01  bond      bond65e3d3f5
bond7990a689               e3ac624e-3444-488d-a96c-5f8416217186  bond      bond7990a689
virbr0                     ee380998-8c11-49b5-84f9-8f01cea62b86  bridge    virbr0
bond0ac56b2c-enP32865p1s0  81cf1cad-2f93-40f7-a0eb-12e79123e11e  ethernet  enP32865p1s0
bond0ac56b2c-env8          b201e91e-ae42-474a-a88f-11fdd330c6dc  ethernet  env8
bond1121ea56-enP32868p1s0  ffdcf8c5-e7e0-4924-ba23-7594ee6ad37d  ethernet  enP32868p1s0
bond1121ea56-env12         251bf84c-dec1-4ac4-b002-152ecfb95680  ethernet  env12
bond1faacd59-enP32859p1s0  330a6cd2-1b5f-4acc-873c-3338e847b8cd  ethernet  enP32859p1s0
bond1faacd59-env3          a6e66bba-78f7-428a-8334-e316d572dea1  ethernet  env3
bond2962530a-enP32867p1s0  58d4a18b-5871-4c9c-b0ae-9f992fef2e57  ethernet  enP32867p1s0
bond2962530a-env11         8201fd73-9c89-4104-a439-c34cf72f7797  ethernet  env11
bond3086d6df-enP32862p1s0  24afb8e1-c6af-422f-8826-2c5a8f76adc5  ethernet  enP32862p1s0
bond3086d6df-env6          0bc77689-69bf-4440-a7ca-672d59f58cb1  ethernet  env6
bond355aeefc-enP32861p1s0  b4ad3dc4-2f36-4932-abe0-6e8736fb22ce  ethernet  enP32861p1s0
bond355aeefc-env5          2dbf0d62-c3ef-49c0-9926-864b29cf409f  ethernet  env5
bond3b385933-enP32863p1s0  6b83b7ec-720d-4677-a599-6a804bccb660  ethernet  enP32863p1s0
bond3b385933-env7          13b1fee9-69e2-44f4-8be8-3e3381312d9a  ethernet  env7
bond3d06e19b-enP32864p1s0  845b6033-4f48-409d-89a7-612b88727171  ethernet  enP32864p1s0
bond3d06e19b-net0          d4cae28a-9703-4906-aa72-195317165552  ethernet  net0
bond65e3d3f5-enP32866p1s0  35827b7d-306d-4e67-9a26-73f6971247e2  ethernet  enP32866p1s0
bond65e3d3f5-env9          452c9f83-61ae-4809-8e9e-9f88515145ed  ethernet  env9
bond7990a689-enP32860p1s0  e623b6bc-f4fc-4099-aaeb-0298e082d6f1  ethernet  enP32860p1s0
bond7990a689-env4          2184b976-0e9c-42da-a154-602aaa44c04c  ethernet  env4

Your host com_dev for comname gdlzzg4stc55p05.gdl.stglabs.ibm.com is set to bond1faacd59, netmask=255.255.0.0
Creating profile template... /usr/lpp/htx//bpt
onesys=y
Automation script will now try to detect your network topology.
In case of errors manually edit bpt file and run build_net bpt to setup hxecom networks
Running command ..
/usr/lpp/htx//bin//auto  bond355aeefc bond3d06e19b bond3b385933 bond65e3d3f5 bond2962530a bond0ac56b2c bond7990a689 bond3086d6df bond1121ea56
Your host com_dev for comname gdlzzg4stc55p05.gdl.stglabs.ibm.com is set to bond                                                                             1faacd59, netmask=255.255.0.0
Your network setup looks like ....
bond355aeefc -> bond3b385933
bond3d06e19b -> bond3086d6df
bond65e3d3f5 -> bond2962530a
bond0ac56b2c -> bond1121ea56

A real system somewhere on the network is configured with net_id=100. Skipping 100 net_id ....
A real system somewhere on the network is configured with net_id=101. Skipping 101 net_id ....
Running build_net bpt to configure test networks ....
Your host com_dev for comname gdlzzg4stc55p05.gdl.stglabs.ibm.com is set to bond1faacd59, netmask=255.255.0.0
Running Linux version of build_net.
...............................Found net0 in location N/A
...............................Found mlx5_0 in location U78D5.ND2.CSS3B52-P1-C3-C1-T1-S91
...............................Found mlx5_1 in location U78D5.ND2.CSS3B52-P1-C3-C1-T1-S92
...............................Found mlx5_2 in location U78D5.ND2.CSS3B52-P1-C3-C1-T1-S93
...............................Found mlx5_3 in location U78D5.ND2.CSS3B52-P1-C3-C1-T1-S94
...............................Found mlx5_4 in location U78D5.ND2.CSS3B52-P1-C3-C1-T1-S95
...............................Found mlx5_5 in location U78D5.ND2.CSS3B52-P1-C3-C1-T1-S96
...............................Found mlx5_6 in location U78D5.ND2.CSS3B52-P1-C3-C1-T1-S97
...............................Found mlx5_7 in location U78D5.ND2.CSS3B52-P1-C3-C1-T1-S98
...............................Found mlx5_8 in location U78D5.ND2.CSS3B52-P1-C3-C1-T1-S99
...............................Found mlx5_9 in location U78D5.ND2.CSS3B52-P1-C3-C1-T1-S100
...............................Found env3 in location U9080.M9S.7846EB8-V4-C3-T1
...............................Found env4 in location U9080.M9S.7846EB8-V4-C4-T1
...............................Found env5 in location U9080.M9S.7846EB8-V4-C5-T1
...............................Found env6 in location U9080.M9S.7846EB8-V4-C6-T1
...............................Found env7 in location U9080.M9S.7846EB8-V4-C7-T1
...............................Found env8 in location U9080.M9S.7846EB8-V4-C8-T1
...............................Found env9 in location U9080.M9S.7846EB8-V4-C9-T1
...............................Found env11 in location U9080.M9S.7846EB8-V4-C11-T1
...............................Found env12 in location U9080.M9S.7846EB8-V4-C12-T1
...............................Found enP32859p1s0 in location U78D5.ND2.CSS3B52-P1-C3-C1-T1-S91
...............................Found enP32860p1s0 in location U78D5.ND2.CSS3B52-P1-C3-C1-T1-S92
...............................Found enP32861p1s0 in location U78D5.ND2.CSS3B52-P1-C3-C1-T1-S93
...............................Found enP32862p1s0 in location U78D5.ND2.CSS3B52-P1-C3-C1-T1-S94
...............................Found enP32863p1s0 in location U78D5.ND2.CSS3B52-P1-C3-C1-T1-S95
...............................Found enP32864p1s0 in location U78D5.ND2.CSS3B52-P1-C3-C1-T1-S96
...............................Found enP32865p1s0 in location U78D5.ND2.CSS3B52-P1-C3-C1-T1-S97
...............................Found enP32866p1s0 in location U78D5.ND2.CSS3B52-P1-C3-C1-T1-S98
...............................Found enP32867p1s0 in location U78D5.ND2.CSS3B52-P1-C3-C1-T1-S99
...............................Found enP32868p1s0 in location U78D5.ND2.CSS3B52-P1-C3-C1-T1-S100
...............................Found bond355aeefc in location N/A
...............................Found bond3d06e19b in location N/A
...............................Found bond3b385933 in location N/A
...............................Found bond65e3d3f5 in location N/A
...............................Found bond1faacd59 in location N/A
bond1faacd59.. Detected site network. Skipping it .... .
...............................Found bond2962530a in location N/A
...............................Found bond0ac56b2c in location N/A
...............................Found bond7990a689 in location N/A
...............................Found bond3086d6df in location N/A
...............................Found bond1121ea56 in location N/A
Configuring Unknown, dev=bond355aeefc, rule=102net, hostname=102net98.250,ip address=102.1.98.250
Configuring Unknown, dev=bond3d06e19b, rule=103net, hostname=103net98.250,ip address=103.1.98.250
Configuring Unknown, dev=bond3b385933, rule=102nets, hostname=102nets98.250,ip address=102.2.98.250
Configuring Unknown, dev=bond65e3d3f5, rule=104net, hostname=104net98.250,ip address=104.1.98.250
Configuring Unknown, dev=bond2962530a, rule=104nets, hostname=104nets98.250,ip address=104.2.98.250
Configuring Unknown, dev=bond0ac56b2c, rule=105net, hostname=105net98.250,ip address=105.1.98.250
Configuring Unknown, dev=bond3086d6df, rule=103nets, hostname=103nets98.250,ip address=103.2.98.250
Configuring Unknown, dev=bond1121ea56, rule=105nets, hostname=105nets98.250,ip address=105.2.98.250

Class B n/w configured, thisid=98.250, mylastnib=250
Ping network 102.1.98.250 from dev bond3b385933---->OK
Ping network 102.2.98.250 from dev bond355aeefc---->OK
Ping network 103.1.98.250 from dev bond3086d6df---->OK
Ping network 103.2.98.250 from dev bond3d06e19b---->OK
Ping network 104.1.98.250 from dev bond2962530a---->OK
Ping network 104.2.98.250 from dev bond65e3d3f5---->OK
Ping network 105.1.98.250 from dev bond1121ea56---->OK
Ping network 105.2.98.250 from dev bond0ac56b2c---->OK
All networks ping Ok
######################## Result Starts Here ################################
mdts are created successfully.
######################### Result Ends Here #################################
Adding 8 devices to mdt
Open /usr/lpp/htx/mdt//mdt.bu to insert the new lines
Done OK
######################## Result Starts Here ################################
ECG (/usr/lpp/htx/mdt/mdt.bu) Activated.
######################### Result Ends Here #################################
Waiting for 30 minutes..

Checking HTX errors in each LPAR..

No ER entries

No DD entries

No HG entries

No PR entries
Everything looks Good with gdlzzg4stc55p05 stopping HTX and powering off LPAR..

Done..

Checking for System SRCs...

No Unrecoverable Errors Found after test started

No Unrecoverable Errors Found after test started

No issues Found.. Powering off all LPARS and Starting next Migration

--------------------LPAR NetworkManager status---------------------
enabled
active

NAME                       UUID                                  TYPE      DEVICE
bond1faacd59               6d89166e-a4fc-4df2-b0d1-0e4d05549f16  bond      bond1faacd59
bond0ac56b2c               d4fd596c-21d6-4ba1-aff3-92fb9e8e0790  bond      bond0ac56b2c
bond1121ea56               ffb36062-3111-4d6d-b85f-db91f50d651a  bond      bond1121ea56
bond2962530a               b0bb10c7-e4f6-4a85-b7d8-6c7a2bc5c167  bond      bond2962530a
bond3086d6df               ec5eca13-0e95-4cb1-b0db-9c9cc2627d7d  bond      bond3086d6df
bond355aeefc               1f7d8ba6-8e25-48f9-9c4a-a790bea4e1d0  bond      bond355aeefc
bond3b385933               37144f3c-b3b3-46b9-a7f3-41c6b1f4a41e  bond      bond3b385933
bond3d06e19b               1fe3b92d-a104-41d7-9093-ad34c62063c7  bond      bond3d06e19b
bond65e3d3f5               445b2cfd-409c-4908-8c76-0b786db57e01  bond      bond65e3d3f5
bond7990a689               e3ac624e-3444-488d-a96c-5f8416217186  bond      bond7990a689
virbr0                     ee380998-8c11-49b5-84f9-8f01cea62b86  bridge    virbr0
bond0ac56b2c-enP32865p1s0  81cf1cad-2f93-40f7-a0eb-12e79123e11e  ethernet  enP32865p1s0
bond0ac56b2c-env8          b201e91e-ae42-474a-a88f-11fdd330c6dc  ethernet  env8
bond1121ea56-enP32868p1s0  ffdcf8c5-e7e0-4924-ba23-7594ee6ad37d  ethernet  enP32868p1s0
bond1121ea56-env12         251bf84c-dec1-4ac4-b002-152ecfb95680  ethernet  env12
bond1faacd59-enP32859p1s0  330a6cd2-1b5f-4acc-873c-3338e847b8cd  ethernet  enP32859p1s0
bond1faacd59-env3          a6e66bba-78f7-428a-8334-e316d572dea1  ethernet  env3
bond2962530a-enP32867p1s0  58d4a18b-5871-4c9c-b0ae-9f992fef2e57  ethernet  enP32867p1s0
bond2962530a-env11         8201fd73-9c89-4104-a439-c34cf72f7797  ethernet  env11
bond3086d6df-enP32862p1s0  24afb8e1-c6af-422f-8826-2c5a8f76adc5  ethernet  enP32862p1s0
bond3086d6df-env6          0bc77689-69bf-4440-a7ca-672d59f58cb1  ethernet  env6
bond355aeefc-enP32861p1s0  b4ad3dc4-2f36-4932-abe0-6e8736fb22ce  ethernet  enP32861p1s0
bond355aeefc-env5          2dbf0d62-c3ef-49c0-9926-864b29cf409f  ethernet  env5
bond3b385933-enP32863p1s0  6b83b7ec-720d-4677-a599-6a804bccb660  ethernet  enP32863p1s0
bond3b385933-env7          13b1fee9-69e2-44f4-8be8-3e3381312d9a  ethernet  env7
bond3d06e19b-enP32864p1s0  845b6033-4f48-409d-89a7-612b88727171  ethernet  enP32864p1s0
bond3d06e19b-net0          d4cae28a-9703-4906-aa72-195317165552  ethernet  net0
bond65e3d3f5-enP32866p1s0  35827b7d-306d-4e67-9a26-73f6971247e2  ethernet  enP32866p1s0
bond65e3d3f5-env9          452c9f83-61ae-4809-8e9e-9f88515145ed  ethernet  env9
bond7990a689-enP32860p1s0  e623b6bc-f4fc-4099-aaeb-0298e082d6f1  ethernet  enP32860p1s0
bond7990a689-env4          2184b976-0e9c-42da-a154-602aaa44c04c  ethernet  env4
Comment 11 IBM Bug Proxy 2020-11-10 13:50:45 UTC 
------- Comment From chavez.com 2020-11-10 08:48 EDT-------
Since testing was successful with the workaround in place, what would be the next action?
Comment 12 IBM Bug Proxy 2020-11-13 00:20:24 UTC 
------- Comment From pradeep.com 2020-11-10 10:36 EDT-------
(In reply to comment #46)
> (In reply to comment #45)
> > Since testing was successful with the workaround in place, what would be the
> > next action?
> .
> ... maybe wait for RHEL8.4 with pcp 5.2.2+ included ..?

Should we add documentation that PCP needs to be removed on RHEL 8.3 to avod such problems?
Comment 13 Nathan Scott 2020-11-13 00:35:35 UTC 
(In reply to IBM Bug Proxy from comment #12)
> ------- Comment From pradeep.com 2020-11-10 10:36 EDT-------
> (In reply to comment #46)
> > (In reply to comment #45)
> > > Since testing was successful with the workaround in place, what would be the
> > > next action?
> > .
> > ... maybe wait for RHEL8.4 with pcp 5.2.2+ included ..?
> 
> Should we add documentation that PCP needs to be removed on RHEL 8.3 to avod
> such problems?

"Since testing was successful with the workaround in place" - so the existing
documentation seems just fine?  If this is a major issue not adequately fixed
via docs, a 8.3 zstream update should be requested (if so, best bet will be to
open a new BZ, with customer justification, linked to the two existing BZs and
assigned to component 'pcp' - we can take it from there, if that meets zstream
inclusion criteria).

cheers.
Comment 14 IBM Bug Proxy 2020-11-13 01:00:21 UTC 
------- Comment From pradeep.com 2020-11-12 19:53 EDT-------
> "Since testing was successful with the workaround in place" - so the existing
> documentation seems just fine?

Can you please point to the documentation so that we can make an assessment?
Comment 15 Nathan Scott 2020-11-29 22:56:14 UTC 
(In reply to IBM Bug Proxy from comment #14)
> [...]
> Can you please point to the documentation so that we can make an assessment?

Apologies, I was confused there - in this case bugzilla is the only documentation because AFAICT noone has written a kbase article for that issue as yet.
Comment 16 Karl Hastings 2020-12-07 22:11:09 UTC 
*** Bug 1904063 has been marked as a duplicate of this bug. ***
Comment 28 IBM Bug Proxy 2021-01-24 19:10:27 UTC 
------- Comment From cdeadmin.com 2021-01-24 14:03 EDT-------
I have NIM installed lpar and tried the test.

(0) root @ zzfp379p6: /root
# uname -a
Linux zzfp379p6 4.18.0-270.el8.ppc64le #1 SMP Wed Jan 6 07:26:39 EST 2021 ppc64le ppc64le ppc64le GNU/Linux

Added 10 MVFs to lpar and performed inactive lpm. Test ran for 3 days and completed 96 iterations.

Original issue was not found.

Thanks
Anitha
Comment 30 Jan Kurik 2021-02-09 05:10:56 UTC 
*** Bug 1926325 has been marked as a duplicate of this bug. ***
Comment 37 errata-xmlrpc 2021-05-18 15:19:32 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (pcp bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:1754