Created attachment 1860879 [details]
turbostat output

Created attachment 1860880 [details]
tlp settings/stats

Ok, I got this working after a bit too much fiddling for my taste.
It's the combination of the Nvidia card (what a surprise ;) ), the original SSD the laptop came with (looks like a Sandisk disguised as Western Digital) and the bluetooth module that don't want to play along with power management.

After trying out many thins, this is what worked:
* ASPM to powersupersave *OR* ASPM to powersave + SSD off in BIOS (using TLP). This is now a two SSD setup with the primary SSD (hosting Fedora) being a Seagate Firecuda: after a week of usage this one seems to be OK with powersupersave, so that's good.
* added kernel param: pcie_aspm=force
* turn off RAID in BIOS (for some reason it's on by default)
* bluetooth off from BIOS (fortunately I can afford this luxury because I am not using it, but this might not be an option for others)
* Nvidia off with envycontrol as already mentioned in the ticket (this is the same story with bluetooth: I am always on AC when turning on dGPU so it's not a compromise for me, but it may be for others)

Now the package goes down to C8, the idle power is around 5-6W, which translates to 14h (this is actually fantastic, way better than the mildly tweaked Win11 the laptop came with).

Boot occasionally freezes after kernel updates, but so far went away with a forced power cycle. Ah yes: and i915.enable_psr=0 otherwise the screen flickers.

Note 1.: in the long term I would like to keep the problematic SSD under low power mode so refresh will still maintain the data on the disk (i.e. to keep the Win11 functioning just in case).

Note 2.: the downside is losing temperature info about the devices turned off (SSD and dGPU in this case), this may or may not throw some surprises in the future (I am thinking of a firmware/BIOS update screwing up turned off state).
This is a worry because if I do: 
echo 1 > /sys/bus/pci/devices/<PROBLEMATIC_SSD_DEVICE_ID>/remove
... wait 15 mins ...
echo 1 > /sys/bus/pci/rescan
tlp-stat -d
Then this disk comes back hot (~55 Celsius). Even under power management it stabilizes around 49 Celsius (idle), while everything else (including the other SSD that Fedora actually runs from) is below 40Celsius.

Reading this: https://docs.kernel.org/power/pci.html I got the naive impression that power management on pci is standardized. I was wondering: would in theory be possible to write a device-agnostic driver that does nothing else but tries to put the device into sleep?

(In reply to bthomasz from comment #3)
> Ok, I got this working after a bit too much fiddling for my taste.
> It's the combination of the Nvidia card (what a surprise ;) ), the original
> SSD the laptop came with (looks like a Sandisk disguised as Western Digital)
> and the bluetooth module that don't want to play along with power management.
> 
> After trying out many thins, this is what worked:
> * ASPM to powersupersave *OR* ASPM to powersave + SSD off in BIOS (using
> TLP). This is now a two SSD setup with the primary SSD (hosting Fedora)
> being a Seagate Firecuda: after a week of usage this one seems to be OK with
> powersupersave, so that's good.

Is either of the 2 SSDs a SATA SSD? If yes then it would be interesting to do:

cat /sys/class/scsi_host/host?/link_power_management_policy

And see what that says, you can also (as root) echo a different policy to that file and see if that helps combined with the regular ASPM powersave setting ?

> * bluetooth off from BIOS (fortunately I can afford this luxury because I am
> not using it, but this might not be an option for others)

Hmm, this should not be necessary. By default the Fedora kernels will enable autosuspend for USB attached BT HCIs. I assume that the bluetooth does show up in "lsusb" when enabled ?

(In reply to Hans de Goede from comment #4)
> (In reply to bthomasz from comment #3)
> > Ok, I got this working after a bit too much fiddling for my taste.
> > It's the combination of the Nvidia card (what a surprise ;) ), the original
> > SSD the laptop came with (looks like a Sandisk disguised as Western Digital)
> > and the bluetooth module that don't want to play along with power management.
> > 
> > After trying out many thins, this is what worked:
> > * ASPM to powersupersave *OR* ASPM to powersave + SSD off in BIOS (using
> > TLP). This is now a two SSD setup with the primary SSD (hosting Fedora)
> > being a Seagate Firecuda: after a week of usage this one seems to be OK with
> > powersupersave, so that's good.
> 
> Is either of the 2 SSDs a SATA SSD? If yes then it would be interesting to
> do:
> 
> cat /sys/class/scsi_host/host?/link_power_management_policy
> 
> And see what that says, you can also (as root) echo a different policy to
> that file and see if that helps combined with the regular ASPM powersave
> setting ?

Can't check that, both SSD-s are on PCIe/NVME so this doesn't apply to my case.

> 
> > * bluetooth off from BIOS (fortunately I can afford this luxury because I am
> > not using it, but this might not be an option for others)
> 
> Hmm, this should not be necessary. By default the Fedora kernels will enable
> autosuspend for USB attached BT HCIs. I assume that the bluetooth does show
> up in "lsusb" when enabled ?

Yes it shows up, did some research on the power consumption (single terminal open with powertop, watching it for a minute after at least 5 min idle):

Bluetooth off from BIOS: ~4.9-5.1W
Bluetooth off from XFCE BT manager: ~5.2-5.6W
Bluetooth on: ~6.8-7.5W (IRQ/s goes up from ~250 to ~370, ?discoverability?)

Thanks for the tip, it's not bad indeed. I remembered more but this was the first thing I disabled when I got the laptop weeks ago.

(TBH I really don't use bluetooth so I'm still trading that extra 0.3-0.5W for 20% more screen brightness.)

(In reply to bthomasz from comment #5)
> Can't check that, both SSD-s are on PCIe/NVME so this doesn't apply to my
> case.

Ok.

> > Hmm, this should not be necessary. By default the Fedora kernels will enable
> > autosuspend for USB attached BT HCIs. I assume that the bluetooth does show
> > up in "lsusb" when enabled ?
> 
> Yes it shows up, did some research on the power consumption (single terminal
> open with powertop, watching it for a minute after at least 5 min idle):
> 
> Bluetooth off from BIOS: ~4.9-5.1W
> Bluetooth off from XFCE BT manager: ~5.2-5.6W

That is worse then I would expect. Does this impact the PC-states in a negative manner, or do you still see mostly PC10 when the system is fully idle?

Can you do lsusb -t and then find the BT device? For me this looks like this:

/:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/10p, 480M
    |__ Port 9: Dev 11, If 0, Class=Wireless, Driver=btusb, 12M
    |__ Port 9: Dev 11, If 1, Class=Wireless, Driver=btusb, 12M


(with other devices omitted) then you can build a USB sysfs device path from that info which in this
case is just 1-9 since there are no hubs in between and then do:

cat /sys/bus/usb/devices/1-9/control
cat /sys/bus/usb/devices/1-9/power/runtime_status

This should output "auto" resp. "suspended" and if it does then the power-consumption impact really should not be this big.

> Bluetooth on: ~6.8-7.5W (IRQ/s goes up from ~250 to ~370, ?discoverability?)

Ugh, that is worse. Is that with the bluetooth control panel open? If yes what happens if you close the panel and then wait 10 seconds ?  Also again I wonder what PC-states you reach in this scenario ?

> (TBH I really don't use bluetooth so I'm still trading that extra 0.3-0.5W
> for 20% more screen brightness.)

I understand, but the impact when not in use really should be negligible, so something is not working as it should.

Just to avoid confusion, the wattages are the total system consumptions reported by the battery.

# lsusb -t
/:  Bus 04.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/8p, 20000M/x2
/:  Bus 03.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/16p, 480M
    |__ Port 5: Dev 2, If 0, Class=Video, Driver=uvcvideo, 480M
    |__ Port 5: Dev 2, If 1, Class=Video, Driver=uvcvideo, 480M
    |__ Port 9: Dev 3, If 0, Class=Vendor Specific Class, Driver=, 12M
    |__ Port 14: Dev 4, If 0, Class=Wireless, Driver=btusb, 12M
    |__ Port 14: Dev 4, If 1, Class=Wireless, Driver=btusb, 12M
/:  Bus 02.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/4p, 10000M
/:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/1p, 480M

# cat /sys/bus/usb/devices/3-14/power/control
auto
# cat /sys/bus/usb/devices/3-14/power/runtime_status
suspended

Yeah you're right, those in my previous reply were with the bluetooth control panel open and it shows on the consumption, sorry about that. 
With powertop --time=5 it actually shows the power consumption peaks at every 3rd-4th measurements.

Here are the udated values with control panel closed (2 min averages):
off from BIOS: 5.05W
off via control panel: 5.24W
on: 5.52W
systemctl disable bluetooth.service & reboot: 5.10W

Note 1.: once bluetooth.service is started, systemctl stop has no effect
Note 2.: if I turn off bluetooth through the control panel, it turns back on during reboot

Regarding the C-states (this is a Tiger Lake): I did not see noticable differences, typically this happens on idle:
package is 70% in C8 (pc8) and the rest is flipping between C2 & C3 
cores are 97-98% in C7 (cc7) 
threads are 99% in C3_ACPI

Note this is a generic comment added to several bugs at once:

There have been quite a few different bug reports now about suspend/resume issues with kernel versions >= 5.16.10, there are some reports that this is caused by the patch titled "ACPI: PM: s2idle: Cancel wakeup before dispatching EC GPE".

So to help debugging this further I've done 2 Fedora test kernel builds with that patch reverted:

5.16.13: https://koji.fedoraproject.org/koji/taskinfo?taskID=83901424
5.17-rc7: https://koji.fedoraproject.org/koji/taskinfo?taskID=83901915

Note these are still building atm, this should be finished in a couple of hours. See here for some generic instructions for installing a kernel directly from koji (the Fedora build-system): https://fedorapeople.org/~jwrdegoede/kernel-test-instructions.txt

If you are seeing suspend/resume issues with the recent Fedora kernels, please give both these builds a try and report back how they work for you.

This message is a reminder that Fedora Linux 35 is nearing its end of life.
Fedora will stop maintaining and issuing updates for Fedora Linux 35 on 2022-12-13.
It is Fedora's policy to close all bug reports from releases that are no longer
maintained. At that time this bug will be closed as EOL if it remains open with a
'version' of '35'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, change the 'version' 
to a later Fedora Linux version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora Linux 35 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora Linux, you are encouraged to change the 'version' to a later version
prior to this bug being closed.

Fedora Linux 35 entered end-of-life (EOL) status on 2022-12-13.

Fedora Linux 35 is no longer maintained, which means that it
will not receive any further security or bug fix updates. As a result we
are closing this bug.

If you can reproduce this bug against a currently maintained version of Fedora Linux
please feel free to reopen this bug against that version. Note that the version
field may be hidden. Click the "Show advanced fields" button if you do not see
the version field.

If you are unable to reopen this bug, please file a new report against an
active release.

Thank you for reporting this bug and we are sorry it could not be fixed.