Description of problem:

The number of alerting rules shown in Console is higher than in prometheus. Some rules are duplicated.
The number of rules in Console is the same as number returned from thanos-querier.

Version-Release number of selected component (if applicable): 4.8


How reproducible: Always


Steps to Reproduce:
1. Launch cluster
2. Port-forward to in-cluster prometheus
3. curl -s http://localhost:9090/api/v1/rules | jq '[.data.groups[].rules[] | select(.type=="alerting")] | length'
4. Port-forward to thanos-querier
5. curl -s http://localhost:9090/api/v1/rules | jq '[.data.groups[].rules[] | select(.type=="alerting")] | length'
6. Compare numbers from #3 and #5

Actual results: Numbers don't match


Expected results: Numbers should match


Additional info: 
The primary suspect is incorrect alert deduplication in thanos querier.

Initial slack thread: https://coreos.slack.com/archives/G79AW9Q7R/p1616158687116100