Bug 1869497
| Summary: | [LSO] Localvolumediscoveryresult objects are deleted after some time | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Chao Yang <chaoyang> |
| Component: | Storage | Assignee: | Santosh Pillai <sapillai> |
| Storage sub component: | Local Storage Operator | QA Contact: | Chao Yang <chaoyang> |
| Status: | CLOSED ERRATA | Docs Contact: | |
| Severity: | high | ||
| Priority: | high | CC: | aos-bugs, hekumar, jsafrane, rojoseph, sapillai |
| Version: | 4.6 | ||
| Target Milestone: | --- | ||
| Target Release: | 4.6.0 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2020-10-27 16:28:38 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
@Chao Good catch! Some queries to reproduce this bug: 1. Is there any approx time after which the results start getting deleted? 2. Was the daemonset also deleted around the same time? Also, @Chao. when you reproduce, next time, could you collect the following info at install time, after all the localvolumediscoveryresults have been created, after reproduction, and 2 minutes after reproduction: oc get po,ds,localvolumediscovery,localvolumediscoveryresult -n openshift-local-storage -o yaml 1.I found deleted at the second day. So hardly to notice the time 2.daemonset not deleted NAME READY STATUS RESTARTS AGE pod/diskmaker-discovery-8fd45 1/1 Running 0 15h pod/diskmaker-discovery-whlwr 1/1 Running 0 15h oc describe localvolumediscoveryresult discovery-result-ip-10-0-157-81.us-east-2.compute.internal
Name: discovery-result-ip-10-0-157-81.us-east-2.compute.internal
Namespace: openshift-local-storage
Labels: discovery-result-node=ip-10-0-157-81.us-east-2.compute.internal
Annotations: <none>
API Version: local.storage.openshift.io/v1alpha1
Kind: LocalVolumeDiscoveryResult
Metadata:
Creation Timestamp: 2020-08-24T09:59:21Z
Generation: 1
Managed Fields:
API Version: local.storage.openshift.io/v1alpha1
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:labels:
.:
f:discovery-result-node:
f:ownerReferences:
.:
k:{"uid":"ca793f25-a181-40bc-bfab-64f6114901a6"}:
.:
f:apiVersion:
f:kind:
f:name:
f:uid:
f:spec:
.:
f:nodeName:
f:status:
.:
f:discoveredDevices:
f:discoveredTimeStamp:
Manager: diskmaker
Operation: Update
Time: 2020-08-24T11:49:40Z
Owner References:
API Version: local.storage.openshift.io/v1alpha1
Kind: LocalVolumeDiscovery
Name: diskmaker-discovery-whlwr
UID: ca793f25-a181-40bc-bfab-64f6114901a6
Resource Version: 370027
Self Link: /apis/local.storage.openshift.io/v1alpha1/namespaces/openshift-local-storage/localvolumediscoveryresults/discovery-result-ip-10-0-157-81.us-east-2.compute.internal
UID: a14a5b96-fad2-4ddb-aabf-8d0c867a950a
Spec:
Node Name: ip-10-0-157-81.us-east-2.compute.internal
Status:
Discovered Devices:
Device ID: /dev/disk/by-id/nvme-Amazon_Elastic_Block_Store_vol09a4dae16fc37bddb-part3
Fstype:
Model:
Path: /dev/nvme0n1p3
Property: NonRotational
Serial:
Size: 1048576
Status:
State: NotAvailable
Type: part
Vendor:
Device ID: /dev/disk/by-id/nvme-Amazon_Elastic_Block_Store_vol09a4dae16fc37bddb-part1
Fstype: ext4
Model:
Path: /dev/nvme0n1p1
Property: NonRotational
Serial:
Size: 402653184
Status:
State: NotAvailable
Type: part
Vendor:
Device ID: /dev/disk/by-id/nvme-Amazon_Elastic_Block_Store_vol09a4dae16fc37bddb-part4
Fstype: crypto_LUKS
Model:
Path: /dev/nvme0n1p4
Property: NonRotational
Serial:
Size: 128311082496
Status:
State: NotAvailable
Type: part
Vendor:
Device ID: /dev/disk/by-id/nvme-Amazon_Elastic_Block_Store_vol09a4dae16fc37bddb-part2
Fstype: vfat
Model:
Path: /dev/nvme0n1p2
Property: NonRotational
Serial:
Size: 133169152
Status:
State: NotAvailable
Type: part
Vendor:
Device ID: /dev/disk/by-id/nvme-Amazon_Elastic_Block_Store_vol0a8412b50eff9fa12
Fstype:
Model: Amazon Elastic Block Store
Path: /dev/nvme3n1
Property: NonRotational
Serial: vol0a8412b50eff9fa12
Size: 536870912k
Status:
State: Available
Type: disk
Vendor:
Device ID: /dev/disk/by-id/nvme-Amazon_Elastic_Block_Store_vol0000b1b79eb46b065
Fstype:
Model: Amazon Elastic Block Store
Path: /dev/nvme2n1
Property: NonRotational
Serial: vol0000b1b79eb46b065
Size: 1073741824
Status:
State: Available
Type: disk
Vendor:
Device ID: /dev/disk/by-id/nvme-Amazon_Elastic_Block_Store_vol092cafed92e00170b
Fstype:
Model: Amazon Elastic Block Store
Path: /dev/nvme1n1
Property: NonRotational
Serial: vol092cafed92e00170b
Size: 1073741824
Status:
State: Available
Type: disk
Vendor:
Discovered Time Stamp: 2020-08-24T11:49:40Z
Events: <none>
apiVersion: v1
kind: Pod
metadata:
annotations:
k8s.v1.cni.cncf.io/network-status: |-
[{
"name": "",
"interface": "eth0",
"ips": [
"10.128.2.20"
],
"default": true,
"dns": {}
}]
k8s.v1.cni.cncf.io/networks-status: |-
[{
"name": "",
"interface": "eth0",
"ips": [
"10.128.2.20"
],
"default": true,
"dns": {}
}]
openshift.io/scc: privileged
creationTimestamp: "2020-08-24T09:59:12Z"
generateName: diskmaker-discovery-
labels:
app: diskmaker-discovery
controller-revision-hash: 74b6f69678
pod-template-generation: "3"
managedFields:
- apiVersion: v1
fieldsType: FieldsV1
fieldsV1:
f:metadata:
f:generateName: {}
f:labels:
.: {}
f:app: {}
f:controller-revision-hash: {}
f:pod-template-generation: {}
f:ownerReferences:
.: {}
k:{"uid":"7c45f78c-cfe2-454a-a6a7-55bff27cf1ab"}:
.: {}
f:apiVersion: {}
f:blockOwnerDeletion: {}
f:controller: {}
f:kind: {}
f:name: {}
f:uid: {}
f:spec:
f:affinity:
.: {}
f:nodeAffinity:
.: {}
f:requiredDuringSchedulingIgnoredDuringExecution:
.: {}
f:nodeSelectorTerms: {}
f:containers:
k:{"name":"diskmaker-discovery"}:
.: {}
f:args: {}
f:env:
.: {}
k:{"name":"MY_NODE_NAME"}:
.: {}
f:name: {}
f:valueFrom:
.: {}
f:fieldRef:
.: {}
f:apiVersion: {}
f:fieldPath: {}
k:{"name":"POD_NAME"}:
.: {}
f:name: {}
f:valueFrom:
.: {}
f:fieldRef:
.: {}
f:apiVersion: {}
f:fieldPath: {}
k:{"name":"UID"}:
.: {}
f:name: {}
f:valueFrom:
.: {}
f:fieldRef:
.: {}
f:apiVersion: {}
f:fieldPath: {}
k:{"name":"WATCH_NAMESPACE"}:
.: {}
f:name: {}
f:valueFrom:
.: {}
f:fieldRef:
.: {}
f:apiVersion: {}
f:fieldPath: {}
f:image: {}
f:imagePullPolicy: {}
f:name: {}
f:resources: {}
f:securityContext:
.: {}
f:privileged: {}
f:terminationMessagePath: {}
f:terminationMessagePolicy: {}
f:volumeMounts:
.: {}
k:{"mountPath":"/dev"}:
.: {}
f:mountPath: {}
f:mountPropagation: {}
f:name: {}
k:{"mountPath":"/mnt/local-storage"}:
.: {}
f:mountPath: {}
f:mountPropagation: {}
f:name: {}
k:{"mountPath":"/run/udev"}:
.: {}
f:mountPath: {}
f:mountPropagation: {}
f:name: {}
f:dnsPolicy: {}
f:enableServiceLinks: {}
f:restartPolicy: {}
f:schedulerName: {}
f:securityContext: {}
f:serviceAccount: {}
f:serviceAccountName: {}
f:terminationGracePeriodSeconds: {}
f:tolerations: {}
f:volumes:
.: {}
k:{"name":"device-dir"}:
.: {}
f:hostPath:
.: {}
f:path: {}
f:type: {}
f:name: {}
k:{"name":"local-disks"}:
.: {}
f:hostPath:
.: {}
f:path: {}
f:type: {}
f:name: {}
k:{"name":"run-udev"}:
.: {}
f:hostPath:
.: {}
f:path: {}
f:type: {}
f:name: {}
manager: kube-controller-manager
operation: Update
time: "2020-08-24T09:59:12Z"
- apiVersion: v1
fieldsType: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
f:k8s.v1.cni.cncf.io/network-status: {}
f:k8s.v1.cni.cncf.io/networks-status: {}
manager: multus
operation: Update
time: "2020-08-24T09:59:15Z"
- apiVersion: v1
fieldsType: FieldsV1
fieldsV1:
f:status:
f:conditions:
k:{"type":"ContainersReady"}:
.: {}
f:lastProbeTime: {}
f:lastTransitionTime: {}
f:status: {}
f:type: {}
k:{"type":"Initialized"}:
.: {}
f:lastProbeTime: {}
f:lastTransitionTime: {}
f:status: {}
f:type: {}
k:{"type":"Ready"}:
.: {}
f:lastProbeTime: {}
f:lastTransitionTime: {}
f:status: {}
f:type: {}
f:containerStatuses: {}
f:hostIP: {}
f:phase: {}
f:podIP: {}
f:podIPs:
.: {}
k:{"ip":"10.128.2.20"}:
.: {}
f:ip: {}
f:startTime: {}
manager: kubelet
operation: Update
time: "2020-08-24T09:59:19Z"
name: diskmaker-discovery-whlwr
namespace: openshift-local-storage
ownerReferences:
- apiVersion: apps/v1
blockOwnerDeletion: true
controller: true
kind: DaemonSet
name: diskmaker-discovery
uid: 7c45f78c-cfe2-454a-a6a7-55bff27cf1ab
resourceVersion: "300017"
selfLink: /api/v1/namespaces/openshift-local-storage/pods/diskmaker-discovery-whlwr
uid: ca793f25-a181-40bc-bfab-64f6114901a6
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchFields:
- key: metadata.name
operator: In
values:
- ip-10-0-157-81.us-east-2.compute.internal
containers:
- args:
- discover
env:
- name: MY_NODE_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: spec.nodeName
- name: WATCH_NAMESPACE
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
- name: POD_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
- name: UID
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.uid
image: registry.redhat.io/openshift4/ose-local-storage-diskmaker@sha256:d34ceab53092c190598556e1c4de5a63003bb246ee95cff1afbbe81b0aec089f
imagePullPolicy: IfNotPresent
name: diskmaker-discovery
resources: {}
securityContext:
privileged: true
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /mnt/local-storage
mountPropagation: HostToContainer
name: local-disks
- mountPath: /dev
mountPropagation: HostToContainer
name: device-dir
- mountPath: /run/udev
mountPropagation: HostToContainer
name: run-udev
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: local-storage-admin-token-9h5kp
readOnly: true
dnsPolicy: ClusterFirst
enableServiceLinks: true
imagePullSecrets:
- name: local-storage-admin-dockercfg-w2trz
nodeName: ip-10-0-157-81.us-east-2.compute.internal
preemptionPolicy: PreemptLowerPriority
priority: 0
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
serviceAccount: local-storage-admin
serviceAccountName: local-storage-admin
terminationGracePeriodSeconds: 30
tolerations:
- effect: NoExecute
key: node.kubernetes.io/not-ready
operator: Exists
- effect: NoExecute
key: node.kubernetes.io/unreachable
operator: Exists
- effect: NoSchedule
key: node.kubernetes.io/disk-pressure
operator: Exists
- effect: NoSchedule
key: node.kubernetes.io/memory-pressure
operator: Exists
- effect: NoSchedule
key: node.kubernetes.io/pid-pressure
operator: Exists
- effect: NoSchedule
key: node.kubernetes.io/unschedulable
operator: Exists
volumes:
- hostPath:
path: /mnt/local-storage
type: ""
name: local-disks
- hostPath:
path: /dev
type: Directory
name: device-dir
- hostPath:
path: /run/udev
type: ""
name: run-udev
- name: local-storage-admin-token-9h5kp
secret:
defaultMode: 420
secretName: local-storage-admin-token-9h5kp
status:
conditions:
- lastProbeTime: null
lastTransitionTime: "2020-08-24T09:59:12Z"
status: "True"
type: Initialized
- lastProbeTime: null
lastTransitionTime: "2020-08-24T09:59:19Z"
status: "True"
type: Ready
- lastProbeTime: null
lastTransitionTime: "2020-08-24T09:59:19Z"
status: "True"
type: ContainersReady
- lastProbeTime: null
lastTransitionTime: "2020-08-24T09:59:12Z"
status: "True"
type: PodScheduled
containerStatuses:
- containerID: cri-o://181630d57ab09d9d0a831ac716e912bc30053e350fe3ea4cf21af1d6e0fe732c
image: registry.redhat.io/openshift4/ose-local-storage-diskmaker@sha256:d34ceab53092c190598556e1c4de5a63003bb246ee95cff1afbbe81b0aec089f
imageID: registry.redhat.io/openshift4/ose-local-storage-diskmaker@sha256:a4f6676e69ed7dc124cb7765b57818587d588b62959a6891a5c4a0189593bbad
lastState: {}
name: diskmaker-discovery
ready: true
restartCount: 0
started: true
state:
running:
startedAt: "2020-08-24T09:59:19Z"
hostIP: 10.0.157.81
phase: Running
podIP: 10.128.2.20
podIPs:
- ip: 10.128.2.20
qosClass: BestEffort
startTime: "2020-08-24T09:59:12Z"
---------------------
oc get localvolumediscovery -o yaml
apiVersion: v1
items:
- apiVersion: local.storage.openshift.io/v1alpha1
kind: LocalVolumeDiscovery
metadata:
creationTimestamp: "2020-08-24T05:39:38Z"
generation: 2
managedFields:
- apiVersion: local.storage.openshift.io/v1alpha1
fieldsType: FieldsV1
fieldsV1:
f:spec:
.: {}
f:nodeSelector:
.: {}
f:nodeSelectorTerms: {}
manager: Mozilla
operation: Update
time: "2020-08-24T09:59:12Z"
- apiVersion: local.storage.openshift.io/v1alpha1
fieldsType: FieldsV1
fieldsV1:
f:status:
.: {}
f:conditions: {}
f:observedGeneration: {}
f:phase: {}
manager: local-storage-operator
operation: Update
time: "2020-08-24T09:59:33Z"
name: auto-discover-devices
namespace: openshift-local-storage
resourceVersion: "300188"
selfLink: /apis/local.storage.openshift.io/v1alpha1/namespaces/openshift-local-storage/localvolumediscoveries/auto-discover-devices
uid: ccbf1ffe-fc3a-409a-9fd5-e88e28ef29fb
spec:
nodeSelector:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- ip-10-0-173-172
- ip-10-0-157-81
status:
conditions:
- lastTransitionTime: "2020-08-24T09:59:33Z"
message: successfully running 2 out of 2 discovery daemons
status: "True"
type: Available
observedGeneration: 2
phase: Discovering
kind: List
metadata:
resourceVersion: ""
selfLink: ""
-------------------------------
oc get daemonset.apps/diskmaker-discovery -o yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
annotations:
deprecated.daemonset.template.generation: "3"
creationTimestamp: "2020-08-24T05:39:38Z"
generation: 3
labels:
app: diskmaker-discovery
managedFields:
- apiVersion: apps/v1
fieldsType: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.: {}
f:deprecated.daemonset.template.generation: {}
f:labels:
.: {}
f:app: {}
f:ownerReferences:
.: {}
k:{"uid":"ccbf1ffe-fc3a-409a-9fd5-e88e28ef29fb"}:
.: {}
f:apiVersion: {}
f:controller: {}
f:kind: {}
f:name: {}
f:uid: {}
f:spec:
f:revisionHistoryLimit: {}
f:selector:
f:matchLabels:
.: {}
f:app: {}
f:template:
f:metadata:
f:labels:
.: {}
f:app: {}
f:spec:
f:affinity:
.: {}
f:nodeAffinity:
.: {}
f:requiredDuringSchedulingIgnoredDuringExecution:
.: {}
f:nodeSelectorTerms: {}
f:containers:
k:{"name":"diskmaker-discovery"}:
.: {}
f:args: {}
f:env:
.: {}
k:{"name":"MY_NODE_NAME"}:
.: {}
f:name: {}
f:valueFrom:
.: {}
f:fieldRef:
.: {}
f:apiVersion: {}
f:fieldPath: {}
k:{"name":"POD_NAME"}:
.: {}
f:name: {}
f:valueFrom:
.: {}
f:fieldRef:
.: {}
f:apiVersion: {}
f:fieldPath: {}
k:{"name":"UID"}:
.: {}
f:name: {}
f:valueFrom:
.: {}
f:fieldRef:
.: {}
f:apiVersion: {}
f:fieldPath: {}
k:{"name":"WATCH_NAMESPACE"}:
.: {}
f:name: {}
f:valueFrom:
.: {}
f:fieldRef:
.: {}
f:apiVersion: {}
f:fieldPath: {}
f:image: {}
f:imagePullPolicy: {}
f:name: {}
f:resources: {}
f:securityContext:
.: {}
f:privileged: {}
f:terminationMessagePath: {}
f:terminationMessagePolicy: {}
f:volumeMounts:
.: {}
k:{"mountPath":"/dev"}:
.: {}
f:mountPath: {}
f:mountPropagation: {}
f:name: {}
k:{"mountPath":"/mnt/local-storage"}:
.: {}
f:mountPath: {}
f:mountPropagation: {}
f:name: {}
k:{"mountPath":"/run/udev"}:
.: {}
f:mountPath: {}
f:mountPropagation: {}
f:name: {}
f:dnsPolicy: {}
f:restartPolicy: {}
f:schedulerName: {}
f:securityContext: {}
f:serviceAccount: {}
f:serviceAccountName: {}
f:terminationGracePeriodSeconds: {}
f:volumes:
.: {}
k:{"name":"device-dir"}:
.: {}
f:hostPath:
.: {}
f:path: {}
f:type: {}
f:name: {}
k:{"name":"local-disks"}:
.: {}
f:hostPath:
.: {}
f:path: {}
f:type: {}
f:name: {}
k:{"name":"run-udev"}:
.: {}
f:hostPath:
.: {}
f:path: {}
f:type: {}
f:name: {}
f:updateStrategy:
f:rollingUpdate:
.: {}
f:maxUnavailable: {}
f:type: {}
manager: local-storage-operator
operation: Update
time: "2020-08-24T09:59:12Z"
- apiVersion: apps/v1
fieldsType: FieldsV1
fieldsV1:
f:status:
f:currentNumberScheduled: {}
f:desiredNumberScheduled: {}
f:numberAvailable: {}
f:numberReady: {}
f:observedGeneration: {}
f:updatedNumberScheduled: {}
manager: kube-controller-manager
operation: Update
time: "2020-08-24T09:59:33Z"
name: diskmaker-discovery
namespace: openshift-local-storage
ownerReferences:
- apiVersion: local.storage.openshift.io/v1alpha1
controller: true
kind: LocalVolumeDiscovery
name: auto-discover-devices
uid: ccbf1ffe-fc3a-409a-9fd5-e88e28ef29fb
resourceVersion: "300187"
selfLink: /apis/apps/v1/namespaces/openshift-local-storage/daemonsets/diskmaker-discovery
uid: 7c45f78c-cfe2-454a-a6a7-55bff27cf1ab
spec:
revisionHistoryLimit: 10
selector:
matchLabels:
app: diskmaker-discovery
template:
metadata:
creationTimestamp: null
labels:
app: diskmaker-discovery
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- ip-10-0-173-172
- ip-10-0-157-81
containers:
- args:
- discover
env:
- name: MY_NODE_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: spec.nodeName
- name: WATCH_NAMESPACE
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
- name: POD_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
- name: UID
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.uid
image: registry.redhat.io/openshift4/ose-local-storage-diskmaker@sha256:d34ceab53092c190598556e1c4de5a63003bb246ee95cff1afbbe81b0aec089f
imagePullPolicy: IfNotPresent
name: diskmaker-discovery
resources: {}
securityContext:
privileged: true
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /mnt/local-storage
mountPropagation: HostToContainer
name: local-disks
- mountPath: /dev
mountPropagation: HostToContainer
name: device-dir
- mountPath: /run/udev
mountPropagation: HostToContainer
name: run-udev
dnsPolicy: ClusterFirst
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
serviceAccount: local-storage-admin
serviceAccountName: local-storage-admin
terminationGracePeriodSeconds: 30
volumes:
- hostPath:
path: /mnt/local-storage
type: ""
name: local-disks
- hostPath:
path: /dev
type: Directory
name: device-dir
- hostPath:
path: /run/udev
type: ""
name: run-udev
updateStrategy:
rollingUpdate:
maxUnavailable: 1
type: RollingUpdate
status:
currentNumberScheduled: 2
desiredNumberScheduled: 2
numberAvailable: 2
numberMisscheduled: 0
numberReady: 2
observedGeneration: 3
updatedNumberScheduled: 2
Verify passed on 4.6.0-0.ci-2020-09-21-225645 oc get localvolumediscoveryresult NAME AGE discovery-result-ip-10-0-141-13.us-east-2.compute.internal 23h discovery-result-ip-10-0-188-236.us-east-2.compute.internal 23h discovery-result-ip-10-0-204-60.us-east-2.compute.internal 23h sharing results from my tests as well : Every 2.0s: oc get localvolumediscoveryresults.local.storage.openshift.io localhost.localdomain: Wed Sep 23 11:49:32 2020 NAME AGE discovery-result-5dc8b70350719b01ac8793cf5df371d9 20h discovery-result-cdffcec636a633f3280f621606b55044 20h discovery-result-eb2bd006c8ff4b3a7b07b1966d41e60a 21h Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (OpenShift Container Platform 4.6 GA Images), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:4196 |
Description of problem: Localvolumediscoveryresult objects are deleted after some time Version-Release number of selected component (if applicable): 4.6.0-0.nightly-2020-08-16-072105 local-storage-operator.4.6.0-202008111832.p0 How reproducible: Always Steps to Reproduce: 1.Deploy LSO 2.Create localvolumediscovery 3.Attach some disk and can found the result in the localvolumediscoveryresult 4.After some time, found the localvolumediscoveryresults objects are deleted. But localvolumediscovery is still working. I0818 03:54:58.241425 1 discovery.go:122] discovered devices: [{DeviceID:/dev/disk/by-id/nvme-Amazon_Elastic_Block_Store_vol0b88013a1ceef6acb-part3 Path:/dev/nvme0n1p3 Model: Type:part Vendor: Serial: Size:{i:{value:1048576 scale:0} d:{Dec:<nil>} s:1048576 Format:DecimalSI} Property:NonRotational FSType: Status:{State:NotAvailable}} {DeviceID:/dev/disk/by-id/nvme-Amazon_Elastic_Block_Store_vol0b88013a1ceef6acb-part1 Path:/dev/nvme0n1p1 Model: Type:part Vendor: Serial: Size:{i:{value:402653184 scale:0} d:{Dec:<nil>} s:402653184 Format:DecimalSI} Property:NonRotational FSType:ext4 Status:{State:NotAvailable}} {DeviceID:/dev/disk/by-id/nvme-Amazon_Elastic_Block_Store_vol0b88013a1ceef6acb-part4 Path:/dev/nvme0n1p4 Model: Type:part Vendor: Serial: Size:{i:{value:128311082496 scale:0} d:{Dec:<nil>} s:128311082496 Format:DecimalSI} Property:NonRotational FSType:crypto_LUKS Status:{State:NotAvailable}} {DeviceID:/dev/disk/by-id/nvme-Amazon_Elastic_Block_Store_vol0b88013a1ceef6acb-part2 Path:/dev/nvme0n1p2 Model: Type:part Vendor: Serial: Size:{i:{value:133169152 scale:0} d:{Dec:<nil>} s:133169152 Format:DecimalSI} Property:NonRotational FSType:vfat Status:{State:NotAvailable}} {DeviceID:/dev/disk/by-id/nvme-Amazon_Elastic_Block_Store_vol0dfa6f0c3fc3a1190 Path:/dev/nvme3n1 Model:Amazon Elastic Block Store Type:disk Vendor: Serial:vol0dfa6f0c3fc3a1190 Size:{i:{value:1073741824 scale:0} d:{Dec:<nil>} s:1073741824 Format:DecimalSI} Property:NonRotational FSType: Status:{State:Available}} {DeviceID:/dev/disk/by-id/nvme-Amazon_Elastic_Block_Store_vol06e49008b4694afd6 Path:/dev/nvme2n1 Model:Amazon Elastic Block Store Type:disk Vendor: Serial:vol06e49008b4694afd6 Size:{i:{value:1073741824 scale:0} d:{Dec:<nil>} s:1073741824 Format:DecimalSI} Property:NonRotational FSType: Status:{State:Available}} {DeviceID:/dev/disk/by-id/nvme-Amazon_Elastic_Block_Store_vol00e6bbe934326f6cf Path:/dev/nvme1n1 Model:Amazon Elastic Block Store Type:disk Vendor: Serial:vol00e6bbe934326f6cf Size:{i:{value:1073741824 scale:0} d:{Dec:<nil>} s:1073741824 Format:DecimalSI} Property:NonRotational FSType: Status:{State:Available}} {DeviceID:/dev/disk/by-id/nvme-Amazon_Elastic_Block_Store_vol046e4ed4094045b5e Path:/dev/nvme4n1 Model:Amazon Elastic Block Store Type:disk Vendor: Serial:vol046e4ed4094045b5e Size:{i:{value:1073741824 scale:0} d:{Dec:<nil>} s:1073741824 Format:DecimalSI} Property:NonRotational FSType: Status:{State:Available}}] I0818 03:54:58.241497 1 discovery.go:126] device list updated. Updating LocalVolumeDiscoveryResult status... W0818 03:54:58.250124 1 sync_result.go:89] result resource not found. Ignoring since object must be deleted. Actual results: Localvolumediscoveryresult objects are deleted after some time while at same time, localvolumediscovery is still working. oc get localvolumediscovery NAME AGE auto-discover-devices 19h Expected results: Localvolumediscoveryresult objects should not be deleted Master Log: Node Log (of failed PODs): PV Dump: PVC Dump: StorageClass Dump (if StorageClass used by PV/PVC): Additional info: I0817 19:36:50.904826 17 httplog.go:94] "HTTP" verb="DELETE" URI="/apis/local.storage.openshift.io/v1alpha1/namespaces/openshift-local-storage/localvolumediscoveryresults/discovery-result-ip-10-0-206-82.us-east-2.compute.internal" latency="12.355483ms" userAgent="kube-controller-manager/v1.19.0 (linux/amd64) kubernetes/99cb93a/system:serviceaccount:kube-system:generic-garbage-collector" srcIP="10.0.138.169:58369" resp=200 I0817 19:36:50.904988 17 httplog.go:94] "HTTP" verb="DELETE" URI="/apis/local.storage.openshift.io/v1alpha1/namespaces/openshift-local-storage/localvolumediscoveryresults/discovery-result-ip-10-0-137-29.us-east-2.compute.internal" latency="12.295317ms" userAgent="kube-controller-manager/v1.19.0 (linux/amd64) kubernetes/99cb93a/system:serviceaccount:kube-system:generic-garbage-collector" srcIP="10.0.138.169:58369" resp=200