Bug 2034889 - `oc adm prune deployments` does not work
Summary: `oc adm prune deployments` does not work
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: oc
Version: 4.10
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 4.10.0
Assignee: Ross Peoples
QA Contact: zhou ying
URL:
Whiteboard: EmergencyRequest
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-12-22 11:52 UTC by zhou ying
Modified: 2022-03-11 18:15 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2022-03-11 18:15:11 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift oc pull 1005 0 None open Bug 2034889: Fix prune deployment panic 2021-12-23 14:32:13 UTC
Github openshift oc pull 1008 0 None open Bug 2034889: Fix another prune panic 2021-12-31 20:28:58 UTC

Description zhou ying 2021-12-22 11:52:53 UTC
Description of problem:
`oc adm prune deployments` does not work 

Version-Release number of selected component (if applicable):
oc version --client
Client Version: 4.10.0-202112201514.p0.g9214112.assembly.stream-9214112

How reproducible:
always

Steps to Reproduce:
1. Create dc and rollout more than 10 times;
[root@localhost ~]# oc get rc
NAME      DESIRED   CURRENT   READY   AGE
mydc-10   0         0         0       3h47m
mydc-11   0         0         0       3h26m
mydc-12   1         1         1       107s
mydc-2    0         0         0       6h44m
mydc-3    0         0         0       6h42m
mydc-4    0         0         0       6h39m
mydc-5    0         0         0       5h30m
mydc-6    0         0         0       5h21m
mydc-7    0         0         0       5h19m
mydc-8    0         0         0       5h7m
mydc-9    0         0         0       5h

2. Run command to prune the RC:
`oc adm prune deployments --confirm=true --keep-complete=2 --keep-failed=1 --keep-younger-than=1m`
3. Create deploy and rollout more times;
4. Run the command to prune the RS:
`oc adm prune deployments  --keep-complete=2 --keep-failed=1 --keep-younger-than=1m  --replica-sets=true`

Actual results:
2. Nothing was pruned:
[root@localhost ~]# oc adm prune deployments --confirm=true --keep-complete=2 --keep-failed=1 --keep-younger-than=1m  -v 6
I1222 19:50:20.754829   94321 loader.go:372] Config loaded from file:  /root/kubeconfig
I1222 19:50:21.752349   94321 round_trippers.go:553] GET https://api.yinzhou-ipv6b.qe.devcluster.openshift.com:6443/apis/apps.openshift.io/v1/deploymentconfigs 200 OK in 996 milliseconds
I1222 19:50:22.367199   94321 round_trippers.go:553] GET https://api.yinzhou-ipv6b.qe.devcluster.openshift.com:6443/api/v1/replicationcontrollers 200 OK in 305 milliseconds
I1222 19:50:22.985825   94321 prune.go:58] Creating deployment pruner with keepYoungerThan=1m0s, orphans=false, replicaSets=false, keepComplete=2, keepFailed=1

[root@localhost ~]# oc get rc
NAME      DESIRED   CURRENT   READY   AGE
mydc-10   0         0         0       4h48m
mydc-11   0         0         0       4h27m
mydc-12   1         1         1       62m
mydc-2    0         0         0       7h44m
mydc-3    0         0         0       7h43m
mydc-4    0         0         0       7h39m
mydc-5    0         0         0       6h30m
mydc-6    0         0         0       6h22m
mydc-7    0         0         0       6h20m
mydc-8    0         0         0       6h7m
mydc-9    0         0         0       6h1m


4. Hit painc error:
[root@localhost ~]# oc adm prune deployments  --keep-complete=2 --keep-failed=1 --keep-younger-than=1m  --replica-sets=true 
panic: runtime error: index out of range [17] with length 17

goroutine 1 [running]:
github.com/openshift/oc/pkg/cli/admin/prune/deployments.PruneDeploymentsOptions.Run({0x0, 0x0, 0x1, 0xdf8475800, 0x2, 0x1, {0x0, 0x0}, {0x40167e0, 0xc000bf5310}, ...})
	/builddir/build/BUILD/openshift-clients-4.10.0/__gopath/src/github.com/openshift/oc/pkg/cli/admin/prune/deployments/deployments.go:180 +0xaf7
github.com/openshift/oc/pkg/cli/admin/prune/deployments.NewCmdPruneDeployments.func1(0xc000ae0f00, {0xc000b5f440, 0x0, 0x4})
	/builddir/build/BUILD/openshift-clients-4.10.0/__gopath/src/github.com/openshift/oc/pkg/cli/admin/prune/deployments/deployments.go:78 +0x118
github.com/spf13/cobra.(*Command).execute(0xc000ae0f00, {0xc000b5f400, 0x4, 0x4})
	/builddir/build/BUILD/openshift-clients-4.10.0/__gopath/src/github.com/openshift/oc/vendor/github.com/spf13/cobra/command.go:860 +0x5f8
github.com/spf13/cobra.(*Command).ExecuteC(0xc00085ac80)
	/builddir/build/BUILD/openshift-clients-4.10.0/__gopath/src/github.com/openshift/oc/vendor/github.com/spf13/cobra/command.go:974 +0x3bc
github.com/spf13/cobra.(*Command).Execute(...)
	/builddir/build/BUILD/openshift-clients-4.10.0/__gopath/src/github.com/openshift/oc/vendor/github.com/spf13/cobra/command.go:902
k8s.io/component-base/cli.Run(0xc00085ac80)
	/builddir/build/BUILD/openshift-clients-4.10.0/__gopath/src/github.com/openshift/oc/vendor/k8s.io/component-base/cli/run.go:105 +0x389
main.main()
	/builddir/build/BUILD/openshift-clients-4.10.0/__gopath/src/github.com/openshift/oc/cmd/oc/oc.go:78 +0x385

Expected results:


Additional info:

Comment 1 Michal Fojtik 2021-12-22 12:22:31 UTC
** A NOTE ABOUT USING URGENT **

This BZ has been set to urgent severity and priority. When a BZ is marked urgent priority Engineers are asked to stop whatever they are doing, putting everything else on hold.
Please be prepared to have reasonable justification ready to discuss, and ensure your own and engineering management are aware and agree this BZ is urgent. Keep in mind, urgent bugs are very expensive and have maximal management visibility.

NOTE: This bug was automatically assigned to an engineering manager with the severity reset to *unspecified* until the emergency is vetted and confirmed. Please do not manually override the severity.

** INFORMATION REQUIRED **

Please answer these questions before escalation to engineering:

1. Has a link to must-gather output been provided in this BZ? We cannot work without. If must-gather fails to run, attach all relevant logs and provide the error message of must-gather.
2. Give the output of "oc get clusteroperators -o yaml".
3. In case of degraded/unavailable operators, have all their logs and the logs of the operands been analyzed [yes/no]
4. List the top 5 relevant errors from the logs of the operators and operands in (3).
5. Order the list of degraded/unavailable operators according to which is likely the cause of the failure of the other, root-cause at the top.
6. Explain why (5) is likely the right order and list the information used for that assessment.
7. Explain why Engineering is necessary to make progress.

Comment 3 zhou ying 2021-12-27 02:56:25 UTC
Checked with latest oc client, still hit error :

[root@localhost ~]#  oc adm prune deployments  --keep-complete=2 --keep-failed=1 --keep-younger-than=1m  --replica-sets=true 
Dry run enabled - no modifications will be made. Add --confirm to remove deployments
panic: interface conversion: interface {} is *v1.Deployment, not *v1.DeploymentConfig

goroutine 1 [running]:
github.com/openshift/oc/pkg/cli/admin/prune/deployments.(*dataSet).ListDeployments(0xc000b47f18)
	/go/src/github.com/openshift/oc/pkg/cli/admin/prune/deployments/data.go:169 +0x149
github.com/openshift/oc/pkg/cli/admin/prune/deployments.(*perDeploymentResolver).Resolve(0xc000d11de0)
	/go/src/github.com/openshift/oc/pkg/cli/admin/prune/deployments/resolvers.go:109 +0x42
github.com/openshift/oc/pkg/cli/admin/prune/deployments.(*mergeResolver).Resolve(0x1)
	/go/src/github.com/openshift/oc/pkg/cli/admin/prune/deployments/resolvers.go:28 +0xa2
github.com/openshift/oc/pkg/cli/admin/prune/deployments.(*pruner).Prune(0x401a640, {0x4016d20, 0xc0006d8e10})
	/go/src/github.com/openshift/oc/pkg/cli/admin/prune/deployments/prune.go:88 +0x2f
github.com/openshift/oc/pkg/cli/admin/prune/deployments.PruneDeploymentsOptions.Run({0x0, 0x0, 0x1, 0xdf8475800, 0x2, 0x1, {0x0, 0x0}, {0x4016780, 0xc0006f8a40}, ...})
	/go/src/github.com/openshift/oc/pkg/cli/admin/prune/deployments/deployments.go:205 +0xaaa
github.com/openshift/oc/pkg/cli/admin/prune/deployments.NewCmdPruneDeployments.func1(0xc000bc8280, {0xc000c86240, 0x0, 0x4})
	/go/src/github.com/openshift/oc/pkg/cli/admin/prune/deployments/deployments.go:78 +0x118
github.com/spf13/cobra.(*Command).execute(0xc000bc8280, {0xc000c86200, 0x4, 0x4})
	/go/src/github.com/openshift/oc/vendor/github.com/spf13/cobra/command.go:860 +0x5f8
github.com/spf13/cobra.(*Command).ExecuteC(0xc0008f2a00)
	/go/src/github.com/openshift/oc/vendor/github.com/spf13/cobra/command.go:974 +0x3bc
github.com/spf13/cobra.(*Command).Execute(...)
	/go/src/github.com/openshift/oc/vendor/github.com/spf13/cobra/command.go:902
k8s.io/component-base/cli.Run(0xc0008f2a00)
	/go/src/github.com/openshift/oc/vendor/k8s.io/component-base/cli/run.go:105 +0x389
main.main()
	/go/src/github.com/openshift/oc/cmd/oc/oc.go:78 +0x385
[root@localhost ~]# oc version 
Client Version: 4.10.0-0.nightly-2021-12-25-025639
Server Version: 4.10.0-0.nightly-2021-12-23-153012
Kubernetes Version: v1.22.1+6859754
[root@localhost ~]# oc version  --client -o yaml 
clientVersion:
  buildDate: "2021-12-24T22:49:12Z"
  compiler: gc
  gitCommit: 5fe7203294827ca91e2c94d1dacbf23477553ba3
  gitTreeState: clean
  gitVersion: 4.10.0-202112242226.p0.g5fe7203.assembly.stream-5fe7203
  goVersion: go1.17.2
  major: ""
  minor: ""
  platform: linux/amd64
releaseClientVersion: 4.10.0-0.nightly-2021-12-25-025639

[root@localhost oc]# git log
commit 5fe7203294827ca91e2c94d1dacbf23477553ba3 (HEAD -> master, origin/release-4.11, origin/release-4.10, origin/master, origin/HEAD)
Merge: b7970db36 bbfcffdef
Author: OpenShift Merge Robot <openshift-merge-robot.github.com>
Date:   Fri Dec 24 21:26:08 2021 +0100

    Merge pull request #1005 from deejross/bz2034889-fix-prune-panic
    
    Bug 2034889: Fix prune deployment panic

Comment 5 zhou ying 2022-01-04 05:59:07 UTC
The panic has fixed , but the prune function still has issue , will verify this one and open a new bug to follow the prune function. 

oc adm prune deployments  --keep-complete=2 --keep-failed=1 --keep-younger-than=1m  --replica-sets=true 
Dry run enabled - no modifications will be made. Add --confirm to remove deployments

[root@localhost ~]# oc version --client
Client Version: 4.10.0-202201031825.p0.g25851bd.assembly.stream-25851bd


Note You need to log in before you can comment on or make changes to this bug.