Description of problem:

Following are two alerts i.e. etcdHighFsyncDurations and etcdHighCommitDurations for checking if underlying storage is perfomring well or not. 
~~~
      - alert: etcdHighFsyncDurations
        annotations:
          message: 'etcd cluster "{{ $labels.job }}": 99th percentile fync durations are {{ $value }}s on etcd instance {{ $labels.instance }}.'
        expr: |
          histogram_quantile(0.99, rate(etcd_disk_wal_fsync_duration_seconds_bucket{job=~".*etcd.*"}[5m]))
          > 0.5
        for: 10m
        labels:
          severity: warning

      - alert: etcdHighCommitDurations
        annotations:
          message: 'etcd cluster "{{ $labels.job }}": 99th percentile commit durations {{ $value }}s on etcd instance {{ $labels.instance }}.'
        expr: |
          histogram_quantile(0.99, rate(etcd_disk_backend_commit_duration_seconds_bucket{job=~".*etcd.*"}[5m]))
          > 0.25
        for: 10m
        labels:
          severity: warning

~~~

According to these alert rules, it seems the metrics values are compared and alert is getting fired at the time when wal_fsync_duration_seconds is going above 500ms and backend_commit_duration_seconds is going above 250ms. 

The values which are kept as boundary line i.e. 500ms and 250ms are too high as compared to etcd backend storage [erformance requirement. 
Backend performance requirement:
~~~
To rule out a slow disk from causing Etcd warnings, you can monitor metrics backend_commit_duration_seconds (p99 duration should be less than 25ms) and wal_fsync_duration_seconds (p99 duration should be less than 10ms) to confirm the storage is reasonably fast.
~~~

So, can we improvise the values from the alert rules? 

Customer's are relying on alerts if their OCP clusters are working properly or misbehaving. Also, the gap is too large as compared to the best practices. 10ms to 500ms is too much.
If the values stay less than 500ms wal_fsync_duration_seconds and 250ms for backend_commit_duration_seconds consistently and they are not as per etcd benchmarks then noone will get notified and the problem persists as it is. 


Version-Release number of selected component (if applicable):
I would say this would be impacting all OCP 4.x. I have checked alert rules on 4.6. 
Also, alert rules are same for 3.11 as well.

How reproducible:
Always

Actual results:
Alerting rules are not properly set according to etcd backend performance requirement.

Expected results:
Improvise on alerting rules so that the issues related to backend storage of etcd will be notified and get addressed.

Additional info:
I have discussed this situation on #forum-etcd slack channel with Sam Batschelet.