Bug 1480844

This also happens on RHEL 7.3 and 7.4.

Thanks Christian, I just wanted to point out that I just found by co-incidence that suspending by closing the laptop screen (folding the laptop), and then resuming by opening it while the power cable is not connected, gives the chance for resuming sometimes without the fan running at full speed (in contrast to what happens always when resuming while connected to power=>fan running at full speed). 
I thought reporting this might help as a work-around, and I hope it could help in debugging.

Same with my T470s running Fedora26 4.12.5-300 kernel. Whether or not the fan starts spinning seems to depend on suspend (folding laptop) duration. Short suspends (up to one hour) worked fine with and without power connection. Longer suspend (4h) again led to the described problem.

Can confirm that it needs a cold restart to fix the problem.

Same experience as Jan in comment 3. I thought this demon had been banished but pulled my laptop out of its bag after lunch, docked it and the whirrrrrrr returned.

$ uname -a
Linux sepang 4.12.5-300.fc26.x86_64 #1 SMP Mon Aug 7 15:27:25 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

$ uptime
 16:32:15 up  7:51,  1 user,  load average: 0.45, 0.29, 0.28

$ cat /proc/acpi/ibm/fan 
status:		enabled
speed:		4412
level:		auto

(In reply to Oliver Haessler from comment #1)
> This also happens on RHEL 7.3 and 7.4.

I can also confirm that this is happening on RHEL 7.4 (3.10.0-693.1.1.el7.x86_64)

(In reply to Christian Kellner from comment #0)

Hi Chris, 
Today we are finally able to reproduce the fan issue.
1. We found a T470s and installed RHEL 7.4 (3.10.0-693.1.1.el7.x86_64), but we are not able to reproduce the issue yet. 
Doing suspend via two ways: lid close and "systemctl suspend", tried more than 10 times. 
We did not do any "yum update", as it need register subscription and we haven't.

2. We tried Fedora 26 with kernel 4.11, no system udpate, still not able to reproduce.

However, 
3. When we tried to update the kernel to 4.10.1, the issue starts to be reproduce very easily, especially when AC adapter connected.

We tried kernel 4.13, it works well by far. Seems for Redhat team, you may upgrade kernel to fix it.

Meanwhile, in order to dig out the root cause, I've already reported the issue to our ThinkPad BIOS team and wait for their comments.

(In reply to Peter Zhang from comment #6)

> Today we are finally able to reproduce the fan issue.
Great!

> 1. We found a T470s and installed RHEL 7.4 (3.10.0-693.1.1.el7.x86_64), but
> we are not able to reproduce the issue yet. 
Maybe we could also try again, to be sure.

> Doing suspend via two ways: lid close and "systemctl suspend", tried more
> than 10 times. 
How long did you let the notebook rest between suspend and resume? It seems the longer you wait the higher the chances of reproducing the bug.

> We did not do any "yum update", as it need register subscription and we
> haven't.
Maybe this is something we can "fix", Alberto?

> 2. We tried Fedora 26 with kernel 4.11, no system udpate, still not able to
> reproduce.
Interesting. With that kernel version it happened for me quite a lot.
 
> However, 
> 3. When we tried to update the kernel to 4.10.1, the issue starts to be
> reproduce very easily, especially when AC adapter connected.
> 
> We tried kernel 4.13, it works well by far. Seems for Redhat team, you may
> upgrade kernel to fix it.
I am also running 4.13, and it does indeed help. But as per the comment in the kernel bugzilla, it seems also not completely fixed (see https://bugzilla.kernel.org/show_bug.cgi?id=196129#c45)

> Meanwhile, in order to dig out the root cause, I've already reported the
> issue to our ThinkPad BIOS team and wait for their comments.
Glad to hear! People reported older firmware versions do not have this issue. Out of curiosity, what version do you have on your T470s test laptop?

(In reply to Christian Kellner from comment #7)

> How long did you let the notebook rest between suspend and resume? 
=> 1 min, we will check the status of sleeping longer.

> I am also running 4.13, and it does indeed help. But as per the comment in
> the kernel bugzilla, it seems also not completely fixed (see
> https://bugzilla.kernel.org/show_bug.cgi?id=196129#c45)
=> Yes, we saw it, the fail rate should be very low. However we will still continue to study it, that is why we forwarded it to BIOS team.
 
>  what version do you have on your T470s test laptop?
=> BIOS N1WET36W (1.15) and EC N1WHT34W (1.16)
Are you able to use the older firmware of T470s to fix the issue? 
What we saw is that someone reporting the older firmware of T460s works, instead of T470s'. Anyway, we will also go back to T460s to check it. 
Thanks for your reminding!

I'm running RHEL 7.4 (3.10.0-693.1.1.el7.x86_64) with BIOS Revision: 1.14 and am intermittently experiencing the issue after resuming from suspend to ram. I've found that after experiencing the issue, if I suspend again, let the fan stop, then resume it often fixes the issue.

I'm running RHEL 7.4 (3.10.0-693.1.1.el7.x86_64) with BIOS Revision: 1.14 and am intermittently experiencing the issue after resuming from suspend to ram. I've found that after experiencing the issue, if I suspend again, let the fan stop, then resume it often fixes the issue.

Hi Skip, Thanks for your report.

@Christian Kellner, 

Just want to share our status with you.
Last week our Japan BIOS team tried more than 7 units of system but are not able to reproduce it. So we sent the NG system from Shanghai to our Yokohama office. Today they received and are able to see this issue now, they will start to debug it from EC view soon.

@Peter

Thanks for the update, much appreciated!

Hi Christian, 
The problem may be caused by PECI hangs (PECI is CPU – Embedded Controller I/F including CPU temperature sensor access. Embedded Controller forced fan to run in max speed for safe. We have worked about PECI hang issue with Intel and applied some fixes in the past. But it seems not enough. Our BIOS team will continue to debug it with Intel. Will keep you posted about the progress.

(In reply to Peter Zhang from comment #13)
> Hi Christian, 
> The problem may be caused by PECI hangs (PECI is CPU – Embedded Controller
> I/F including CPU temperature sensor access. Embedded Controller forced fan
> to run in max speed for safe. We have worked about PECI hang issue with
> Intel and applied some fixes in the past. But it seems not enough. Our BIOS
> team will continue to debug it with Intel. Will keep you posted about the
> progress.

Thanks a lot Peter, keeps us posted.

@Albert and all, 
Sure. And there are some new findings - today our BIOS team found the issue is related to NVMe SSD of Samsung.

@Christian.K and all friends reported this bug, 
Would you mind helping to report if you are all using Samsung NVMe SSD? 
We hope to double confirm this point. Thanks.

@Peter , at least on the system that I have (T470s) with the fan issue, it is a Samsung Controller:

lshw -class storage

  *-storage
       description: Non-Volatile memory controller
       product: NVMe SSD Controller SM961/PM961
       vendor: Samsung Electronics Co Ltd
       physical id: 0
       bus info: pci@0000:3c:00.0
       version: 00
       width: 64 bits
       clock: 33MHz
       capabilities: storage pm msi pciexpress msix nvm_express bus_master cap_list
       configuration: driver=nvme latency=0
       resources: irq:16 memory:ec000000-ec003fff

@Peter, many thanks for keeping us updated. Yes, I also have a Samsung device. I get the very same lshw output as Oliver.

@Peter I also have the NVMe SSD Controller SM961/PM961 and am experiencing the issue.

@Peter the same as Oliver ...

Yep, same as Oliver here as well.

Thank you all, it is very helpful. 
We will continue to update BIOS team's debugging status.

I'm seeing this frequently on a fresh Fedora 26 install on a T470s with Toshiba so the Samsung controller might be a red herring.

  *-storage
       description: Non-Volatile memory controller
       product: Toshiba America Info Systems
       vendor: Toshiba America Info Systems
       physical id: 0
       bus info: pci@0000:3c:00.0
       version: 01
       width: 64 bits
       clock: 33MHz
       capabilities: storage pm msi pciexpress msix nvm_express bus_master cap_list
       configuration: driver=nvme latency=0
       resources: irq:16 memory:ec000000-ec003fff

Thanks Jeff, you are correct. 
These days there are some more findings at our side as following, I'd planned to update this on the day when we found root cause.
1. Samsung NVMe SSD => NG
2. Lenovo brand SSD => Good
3. Toshiba, Intel SSD => NG
At the beginning we were trying to track CPU C-state and looks that C6 can trigger the issue, now trying to track the difference of D-state of different SSD drivers and other directions...

Hi Christian and all,

Do you remember if you have an AC-attach action during suspend?
I mean this process: DC only -> sleep -> plug in AC -> resume -> found the Fan Noise issue.

DC=Battery, AC= AC adapter

This issue seems very complicated, we spent much time to discuss with Intel upstream kernel engineer Zheng Lv and ECFW engineer.

Besides the PECI direction, our another direction is to check why the AC attach event during S3 will cause this issue. However I never see that anyone of you mentioned it.
 
In other words, are you still able to see this issue when AC always attached? I mean without any AC plug in action, without any AC/DC status change during suspend. Thanks.

As for me, I only remember it happening when I work on DC, sleep, plug it into the docking station (i.e. move to AC) and resume. Sleeping and resuming so far always resolved the issue.

I do not recall it happening while staying on DC or staying on AC.

One extra data point here: I am on the road and this is occurring right now while never having reconnected to AC. Roughly, the order of operations was:

* laptop docked, lid closed, displays asleep (not suspended)
* pop off dock, put laptop in bag
* some time later it suspends
* wake; normal
* suspend again ... flight/travel etc
* wake; fan running high speed

Never reconnected to AC during this time.

4.12.14-300.fc26.x86_64 on 470s with Intel video.

Same issue here. Running T470s with Samsung NVMe SSD.

Had the fan come up after resume on both cases on many occasions: still on DC and connected to AC.

4.12.14-300.fc26.x86_64 

  *-storage
       description: Non-Volatile memory controller
       product: NVMe SSD Controller SM961/PM961
       vendor: Samsung Electronics Co Ltd
  
CPU: Version: Intel(R) Core(TM) i7-7600U CPU @ 2.80GHz

BIOS: BIOS Information
        Vendor: LENOVO
        Version: N1WET35W (1.14 )
        Release Date: 07/12/2017

System Information
        Manufacturer: LENOVO
        Product Name: 20HGS22D01
        Version: ThinkPad T470s

VGA compatible controller: Intel Corporation HD Graphics 620 (rev 02)

Hello Wout, Chris, Cenk, Thanks for your report. 

@All, When we lengthened PECI reset waiting time to max 2 secs, built a trial BIOS and found the issue disappeared. We are checking with our BIOS team if it can be as the final solution, please wait for our further update.

Sorry for the late response, we are just back to office from a long Chinese National Day holiday.

I'm having the same issue with X270 and latest BIOS on Fedora 26 with latest kernel. Does the potential fix include X270 and has the issue been observed on it as well?

Hey George,

I had seen the issue quite some time ago on a X270 we had for testing. I would guess it is the same issue.

Is there some info I can provide to help address this on X270?

Hello George,

Yes, could you please help to provide following info?
1. Did you do any actions when it's sleep? For example, plug in AC adapter or Dock or other devices.
2. Your BIOS & EC versions.
3. The SSD brand in your system.
4. dmesg info

Thanks,
Peter

Hey Peter,

1. I think it happens both when plugged in and unplugged and waking up from sleep without any additional actions in between.
2. BIOS R0IET43W (1.21)
   EC   R0IHT31W (1.13)
3. SSD  Samsung MZVLW256HEHP-000L7
4. At which do you want to see the dmesg? When the fans start spinning after resume?

Thank for looking into this.

Created attachment 1337939 [details]
dmesg from X270 after resume

Fans went full speed right after resume when plugged in.

Thank you George, it matches with what we know.

Dear Redhat friends,

Good news, today we received positive updates from our Yokohama BIOS team,and our Linux team verified the new candidate BIOS/EC on T470s and X1 Yoga Gen 2nd in our Shanghai office, so far both platforms work well and the issue is gone. Next we will ask BIOS team to start to work on X270 and other products.

Is there anyone want to try this candidate BIOS/EC on T470s first? If yes, can you please let me know? 

Please note that there are some risks to update this candidate BIOS/EC, so we don't want to post it here directly. It is better to have one person in Redhat to try it first in order to avoid to expose the risks to all. If it's safe enough, then we will be able to deliver it to more people.

However, our formal web release process will need more time, since we have to qualify it and confirm if this change has any side effects on Windows OS which is very time consuming.

Thanks,
Peter

(In reply to Peter Zhang from comment #36)
> Is there anyone want to try this candidate BIOS/EC on T470s first? If yes,
> can you please let me know? 

I'm interested in testing it.

Great Michal! I will send it to you by email soon. Thanks.

Peter, is there some intermediate solution? Like contacting support to replace SSD or is this Linux-specific and I should wait for the BIOS update?

Hi George, There are some options FYI:
a. To upgrade kernel to 4.13, the fail rate will reduce to an acceptable level.
b. So far we found that Lenovo brand SSD and SATA HDD work well
But we suggest to wait for the BIOS/EC update.

George, No Windows OS Users report this issue by far.

Peter, I'm on 4.13.5-200.fc26.x86_64 already. I haven't noticed any improvements.

George, Sorry to hear that. Is your issue a different one? Would you mind checking this bug link: https://bugzilla.kernel.org/show_bug.cgi?id=196129#c64

George, Anyway, I will push our BIOS team to prepare candidate BIOS for X270 ASAP next week.Then we will be able to know if it is the same issue.

Peter,
the firmware you sent me looks good. My T470s still works and the the fan issue is no longer reproduced.

dmidecode output before/after comparison:

--- dmi.before	2017-10-13 12:34:22.073635510 +0200
+++ dmi.after	2017-10-13 12:53:09.720265692 +0200
@@ -1,7 +1,7 @@
 # dmidecode 3.1
 Getting SMBIOS data from sysfs.
 SMBIOS 3.0.0 present.
-Table at 0x9A694000.
+Table at 0x9A693000.
 
 Handle 0x0000, DMI type 222, 14 bytes
 OEM-specific Type
@@ -174,12 +174,12 @@
 		TM (Thermal monitor supported)
 		PBE (Pending break enabled)
 	Version: Intel(R) Core(TM) i7-7600U CPU @ 2.80GHz
-	Voltage: 1.1 V
+	Voltage: 0.5 V
 	External Clock: 100 MHz
 	Max Speed: 2900 MHz
 	Current Speed: 2800 MHz
 	Status: Populated, Enabled
-	Upgrade: Other
+	Upgrade: Socket BGA1356
 	L1 Cache Handle: 0x0007
 	L2 Cache Handle: 0x0008
 	L3 Cache Handle: 0x0009
@@ -200,8 +200,8 @@
 Handle 0x000B, DMI type 0, 24 bytes
 BIOS Information
 	Vendor: LENOVO
-	Version: N1WET38W (1.17 )
-	Release Date: 09/01/2017
+	Version: N1WET40W (1.19_MaxFan )
+	Release Date: 10/12/2017
 	Address: 0xE0000
 	Runtime Size: 128 kB
 	ROM Size: 16 MB
@@ -224,8 +224,8 @@
 		BIOS boot specification is supported
 		Targeted content distribution is supported
 		UEFI is supported
-	BIOS Revision: 1.17
-	Firmware Revision: 1.16
+	BIOS Revision: 1.19
+	Firmware Revision: 0.18
 
 Handle 0x000C, DMI type 1, 27 bytes
 System Information
[...diff in OEM specific hex numbers trimmed...]

Great news Peter!
I am available for testing as well (kinda my job, tbh) ;)

@Michal, Thank you so much! It is very helpful.

@Christian.K, 
Will forward that email to you soon. Thanks.:)

Dear friends, 

For T470s I was told that there will be a new formal BIOS/EC FW release to fix this issue in about the first week of Nov. Just want to let you know about the schedule. 

X270 will be the next one. Its trial BIOS is ready, we are now testing it.

For other projects, the solution may be available in next or ne-next BIOS release.

If there are any projects you want to get a faster support, please don't hesitate to let us know.

Thanks,
Peter

(In reply to Peter Zhang from comment #48)
> Dear friends, 
> 
> For T470s I was told that there will be a new formal BIOS/EC FW release to
> fix this issue in about the first week of Nov. Just want to let you know
> about the schedule. 
> 
> X270 will be the next one. Its trial BIOS is ready, we are now testing it.
> 
> For other projects, the solution may be available in next or ne-next BIOS
> release.
> 
> If there are any projects you want to get a faster support, please don't
> hesitate to let us know.
> 

I'm glad the issue is fixed for T470s. On kernel bugzilla a similar problem was opened: https://bugzilla.kernel.org/show_bug.cgi?id=191181

On my t460s i have the same issue, will there a new firmware t460s as well?
For now i'm still at EC 1.09 because the issue startet with 1.10:

# dmidecode -t bios
BIOS Information
	Vendor: LENOVO
	Version: N1CET60W (1.28 )
	Release Date: 09/26/2017
..
	BIOS Revision: 1.28
	Firmware Revision: 1.9

# lshw -class storage
  *-storage                 
       description: Non-Volatile memory controller
       product: NVMe Controller
       vendor: Toshiba America Info Systems
       physical id: 0
       bus info: pci@0000:05:00.0
       version: 01
       width: 64 bits
       clock: 33MHz
       capabilities: storage pm msi pciexpress msix nvm_express bus_master cap_list
       configuration: driver=nvme latency=0
       resources: irq:16 memory:e1000000-e1003fff

Thanks,
-ap

(In reply to Peter Zhang from comment #48)
> Dear friends, 
> 
> For T470s I was told that there will be a new formal BIOS/EC FW release to
> fix this issue in about the first week of Nov. Just want to let you know
> about the schedule. 
> 
> X270 will be the next one. Its trial BIOS is ready, we are now testing it.
> 
> For other projects, the solution may be available in next or ne-next BIOS
> release.
> 
> If there are any projects you want to get a faster support, please don't
> hesitate to let us know.
> 
> Thanks,
> Peter

Good news for T470s!

I see the same symptoms on T470 (not T470s) with BIOS N1QET67W 1.42 and EC 1.27. The related kernel bug is tracked here: https://bugzilla.kernel.org/show_bug.cgi?id=196975
If I understand correctly, so far it seems recent kernels have C6-exits which sometimes trigger an EC failure when reading acpitz-virtual-0 temp1 so the fan spins up.

Is T470 perhaps also in line for a BIOS update to resolve this issue?

Thanks,
Tomislav

(In reply to Andreas Piesk from comment #49)
> On my t460s i have the same issue, will there a new firmware t460s as well?

Hello Andereas, 
Thanks for letting us know. We'd thought only Kabylake platforms having this issue. T460s is Skylake platform, we will check it if it is the same issue soon, and will keep you posted about the progress.

(In reply to Tomislav Ivek from comment #50)

> Is T470 perhaps also in line for a BIOS update to resolve this issue?

Hello Tomislav, 

Yes, the time frame of T470 new BIOS release is planned in middle of Nov.

Thanks,
Peter

(In reply to Peter Zhang from comment #52)
> (In reply to Tomislav Ivek from comment #50)
> 
> > Is T470 perhaps also in line for a BIOS update to resolve this issue?
> 
> Hello Tomislav, 
> 
> Yes, the time frame of T470 new BIOS release is planned in middle of Nov.
> 
> Thanks,
> Peter

Thank you Peter, I am looking forward to trying it out.

Best,
Tomislav

Hi Peter,

Is this the official release to fix this issue?

https://pcsupport.lenovo.com/nl/en/products/laptops-and-netbooks/thinkpad-t-series-laptops/thinkpad-t470s/20hg/20hgs22d04/downloads/ds120419

Thanks,
Cenk

Hi Cenk, 

Yes, you are correct. It is! 

Please don't hesitate to let us know if you have any questions.

Thanks,
Peter

(In reply to Cenk Kulacoglu from comment #54)
> https://pcsupport.lenovo.com/nl/en/products/laptops-and-netbooks/thinkpad-t-
> series-laptops/thinkpad-t470s/20hg/20hgs22d04/downloads/ds120419

After first testing, it appears to work (hope it is not premature optimism ;) ).

I wish the updated BIOS was easier installable via gnome-software and its firemware update feature. It would greatly help all Lenovo Linux users ...

I second Vít's suggestion. I would be great to have firmware updates distributed via https://fwupd.org/

Hello Vit, Michal,

Yes, agree. For LVFS support, our BIOS are working with Richard.
It might not be able to be finished in short time, but it is in progress.

(In reply to Peter Zhang from comment #58)
That is very good news! Thanks for working on that.

How are we supposed to apply this firmware update if we have no external CD? There are various steps online for extracting the image to a thumb drive, but none seem to work. Also tried booting the downloaded image from grub2.

(And yes, this is why fwupd would be nice, but supporting booting from thumb drives should be default now)

We used geteltorito to extract it then dd to write it to a USB key then booted from that key.  Details here (nod to Micah for finding this):

https://workaround.org/article/updating-the-bios-on-lenovo-laptops-from-linux-using-a-usb-flash-stick/

Beware that if it doesn't come up right away when you boot you might have to play with the Secure Boot options.

I can confirm that. I updated the firmware today, and used geteltorito + dd to create the bootable USB.

That site was the first thing I tried yesterday (long before the comment); I did get it to work today after changing some of the BIOS settings for boot preferences. There are comments on that page suggesting various, contradictory, settings -- for me it was "UEFI/Legacy Boot" set to "UEFI only" that let it work.

Since updating the BIOS I have gone through a few powered suspend -> battery resume/suspend -> powered resume cycles without running into this whereas before I was hitting the bug very consistently.  Looks to be fixed.  Thanks!

And just to consolidate the solution in one comment for simplicity:

1) Download the BIOS from the link in https://bugzilla.redhat.com/show_bug.cgi?id=1480844#c54

2) geteltorito -o bios.img n1wur15w.iso

3) dd if=bios.img of=<your USB key> bs=1M

4) Reboot to setup (Enter->F1) and enable Secure Boot and set UEFI/Legacy Boot to UEFI Only (both might work). 

5) Save settings and reboot from USB (Enter->F12 and select your USB key).  

5a) If you get only one line that says something like "Lenovo Systems" and nothing happens, go back and check your BIOS settings against the recommendations in https://download.lenovo.com/pccbbs/mobiles/n1wur15w.txt and try again.  You should see a page of text and some power warnings and other general OMG YOU ARE CHANGING THE BIOS ARE YOU SURE?!? stuff.

6) Follow instructions.

7) Profit

(In reply to Jeff Needle from comment #64)
> Since updating the BIOS I have gone through a few powered suspend -> battery
> resume/suspend -> powered resume cycles without running into this whereas
> before I was hitting the bug very consistently.  Looks to be fixed.  Thanks!
> 
> And just to consolidate the solution in one comment for simplicity:
> 
> 1) Download the BIOS from the link in
> https://bugzilla.redhat.com/show_bug.cgi?id=1480844#c54
> 
> 2) geteltorito -o bios.img n1wur15w.iso

One can do it without USB stick if OS was installed in UEFI mode,
here is how I always do/did it on my EFI capable Thinkpads:

geteltorito -o /tmp/bios.img /tmp/n1wur15w.iso
losetup /tmp/bios.img /dev/loop0
partprobe /dev/loop0
mount /dev/loop0p1 /mnt -o ro
cp -r /mnt/FLASH /boot/efi/
cp /mnt/EFI/BOOT/BootX64.efi /boot/efi/EFI/BOOT/bios.efi

# '-p 3' picks fat32 EFI partition (my setup),
# amend it to match your layout if necessary
efibootmgr -c -d /dev/nvme0n1 -p 3 -L "BIOS UPDATE" -l '\EFI\BOOT\bios.efi'


> 4) Reboot to setup (Enter->F1) and enable Secure Boot and set UEFI/Legacy
> Boot to UEFI Only (both might work). 
> 
> 5) Save settings and reboot from USB (Enter->F12 and select your USB key).
and at this step pick "BIOS UPDATE" option
  

> 5a) If you get only one line that says something like "Lenovo Systems" and
> nothing happens, go back and check your BIOS settings against the
> recommendations in https://download.lenovo.com/pccbbs/mobiles/n1wur15w.txt
> and try again.  You should see a page of text and some power warnings and
> other general OMG YOU ARE CHANGING THE BIOS ARE YOU SURE?!? stuff.
> 
> 6) Follow instructions.
> 
> 7) Profit

Having this same issue on a Thinkpad 13 (Skylake) using Fedora 27. Just started with recent kernel updates. Mine runs on high all the time. Suspend or not. 

I'm a novice so I'm not sure what logs if any would be helpful. I did update the bios to see if that would fix. No luck.

(In reply to mr.meierrr from comment #66)

Hello Mr Meier, 

The title issue is related with suspend. Your issue may be different.

Could you upload a screenshot of BIOS Setup and post the output of following command:
# lshw -class storage

Thanks

(In reply to Peter Zhang from comment #67)

Thanks for trying to help Peter!

lshw : https://paste.fedoraproject.org/paste/UiC4-24OExI6N2MpHyv7Tw



bios: https://i.imgur.com/afmJuAS.jpg

(In reply to mr.meierrr from comment #68)

So far what we observed about the title issue have following features.
a. Related to suspend, 
b. Related to NVMe storage, 
c. Warm reboot cannot make the issue disappear.

We are adding the solution to new BIOS release. Some products' new BIOS already available in our website, some others are still in the pipeline and will be available in a month.

But ThinkPad 13 (Skylake) is not in above product list.

However the issue you reported seems different, as your storage is common SATA, instead of NVMe. Let us clarify it separately. 

Thanks

In my case, ThinkPad T460s, fan was going crazy occasionally, once a month in average I'd say. Suspend/resume always helped. After upgrade to newest BIOS (1.30) things have gotten worse and now it goes crazy after each suspend.

I guess that you shouldn't always trust new BIOS.

I created new thread on Lenovo forum: https://forums.lenovo.com/t5/ThinkPad-T400-T500-and-newer-T/ThinkPad-T460s-BIOS-1-30-quot-fan-fix-quot-actually-worsen/m-p/3906892/highlight/false#M121042

(In reply to Łukasz Siudut from comment #70)
> In my case, ThinkPad T460s, fan was going crazy occasionally, once a month
> in average I'd say. Suspend/resume always helped. After upgrade to newest
> BIOS (1.30) things have gotten worse and now it goes crazy after each
> suspend.

Have you tried downgrading to EC 1.9?:

# dmidecode -t bios
BIOS Information
	Vendor: LENOVO
	Version: N1CET62W (1.30 )
	Release Date: 11/27/2017
	BIOS Revision: 1.30
	Firmware Revision: 1.9 <- EC

No problems with EC 1.9 and kernel 4.13 and 4.14 (Fedora Core 27).

(In reply to Andreas Piesk from comment #71)
> (In reply to Łukasz Siudut from comment #70)
> > In my case, ThinkPad T460s, fan was going crazy occasionally, once a month
> > in average I'd say. Suspend/resume always helped. After upgrade to newest
> > BIOS (1.30) things have gotten worse and now it goes crazy after each
> > suspend.
> 
> Have you tried downgrading to EC 1.9?:
> 
> # dmidecode -t bios
> BIOS Information
> 	Vendor: LENOVO
> 	Version: N1CET62W (1.30 )
> 	Release Date: 11/27/2017
> 	BIOS Revision: 1.30
> 	Firmware Revision: 1.9 <- EC
> 
> No problems with EC 1.9 and kernel 4.13 and 4.14 (Fedora Core 27).

I upgraded to EC 1.13 again (still running kernel 4.14) and haven't had the issue since then, so the combination kernel > 4.13 and EC > 1.9 works for me.

That's interesting.

Btw yes - EC 1.9 works fine for me. I'll check newer kernel later today or tomorrow.

I definitely also have this FAN problem after suspends with my T460s 20F9:

dmidecode -t bios
# dmidecode 3.0
Getting SMBIOS data from sysfs.
SMBIOS 2.8 present.

Handle 0x000C, DMI type 0, 24 bytes
BIOS Information
        Vendor: LENOVO
        Version: N1CET63W (1.31 )
        Release Date: 12/19/2017
        Address: 0xE0000
        Runtime Size: 128 kB
        ROM Size: 16384 kB
        Characteristics:
                PCI is supported
                PNP is supported
                BIOS is upgradeable
                BIOS shadowing is allowed
                Boot from CD is supported
                Selectable boot is supported
                EDD is supported
                3.5"/720 kB floppy services are supported (int 13h)
                Print screen service is supported (int 5h)
                8042 keyboard services are supported (int 9h)
                Serial services are supported (int 14h)
                Printer services are supported (int 17h)
                CGA/mono video services are supported (int 10h)
                ACPI is supported
                USB legacy is supported
                BIOS boot specification is supported
                Targeted content distribution is supported
                UEFI is supported
        BIOS Revision: 1.31
        Firmware Revision: 1.13

Handle 0x0025, DMI type 13, 22 bytes
BIOS Language Information
        Language Description Format: Abbreviated
        Installable Languages: 1
                en-US
        Currently Installed Language: en-US



I am currently using Linux 4.14.7 and my Storage is a Samsung NVMe

Btw. how can I downgrade the Firmware (EC) revision? I can only find full BIOS+Firmware updates on the Lenovo support site

I can reproduce it by removing the laptop from the docking station, close the lid, wait until red LED starts blinking and then opening the lid again+pressing the power button.

I can currently read following from my temperature sensors:

/sys/devices/virtual/hwmon/hwmon0/temp1_input: 48000
/sys/devices/virtual/hwmon/hwmon2/temp1_input: 35500
/sys/devices/virtual/hwmon/hwmon4/temp1_input: 30000

It is obvious that only the hwmon0 temperature is constant. So most likely the sensor doesn't return the correct temperature. The name of hwmon0 btw is acpitz

A reboot doesn't fix this problem. I really have to shut down the device and then do a coldstart. The readings of the temperature sensors after doing that:

/sys/devices/virtual/hwmon/hwmon0/temp1_input: 40000
/sys/devices/virtual/hwmon/hwmon1/temp1_input: 41500
/sys/devices/virtual/hwmon/hwmon4/temp1_input: 28000

All temperatures are changing over time - until I close the lid again and the device goes into suspend.

(In reply to Charlemagne Lasse from comment #75)
> Btw. how can I downgrade the Firmware (EC) revision? I can only find full
> BIOS+Firmware updates on the Lenovo support site

Jens Axboe posted a procedure in https://bugzilla.kernel.org/show_bug.cgi?id=191181

Peter Zhang, do you have any updates for T460s with Samsung NVMe SSD?

(In reply to Charlemagne Lasse from comment #79)
> Peter Zhang, do you have any updates for T460s with Samsung NVMe SSD?

Hello Charlemagne, 
 
Thanks for your asking. I just double checked with T460s BIOS team, they were still not able to reproduce this issue, but they prepared some trial BIOS/EC files to some customers, and their feedback are good by far. Do you have any interests to try it? If yes, you may get the firmware files from here:
http://www.mediafire.com/file/xby9gns6yhhfar6/n1cuj18a.zip

I still rely on BIOS team about formal release plan.

Thanks,
Peter

I was informed that I am not allowed to install bios versions which were shared via unofficial sources like mediafire.com. So I am unable to test this version and have to hope that your BIOS team releases the official version soon.

Also having the problem on a X260. Fedora 27. Noticed after updating to bios v1.35, still present on v1.36.

If anyone need some info I'll be happy to provide.

# dmidecode 3.1
Getting SMBIOS data from sysfs.
SMBIOS 2.8 present.

Handle 0x000B, DMI type 0, 24 bytes
BIOS Information
        Vendor: LENOVO
        Version: R02ET63W (1.36 )
        Release Date: 12/15/2017
        Address: 0xE0000
        Runtime Size: 128 kB
        ROM Size: 16 MB
        Characteristics:
                PCI is supported
                PNP is supported
                BIOS is upgradeable
                BIOS shadowing is allowed
                Boot from CD is supported
                Selectable boot is supported
                EDD is supported
                3.5"/720 kB floppy services are supported (int 13h)
                Print screen service is supported (int 5h)
                8042 keyboard services are supported (int 9h)
                Serial services are supported (int 14h)
                Printer services are supported (int 17h)
                CGA/mono video services are supported (int 10h)
                ACPI is supported
                USB legacy is supported
                BIOS boot specification is supported
                Targeted content distribution is supported
                UEFI is supported
        BIOS Revision: 1.36
        Firmware Revision: 1.14

Handle 0x0024, DMI type 13, 22 bytes
BIOS Language Information
        Language Description Format: Abbreviated
        Installable Languages: 1
                en-US
        Currently Installed Language: en-US

Hi,

for T460s BIOS 1.33 was released which fixes it for me (so far...; with Samsung NVMe SSD, some 4.15.4 kernel)
at least I can wakeup from sleep w/o the need of having the power cord attaced which was the only
working workaround for me.

SEE
https://download.lenovo.com/pccbbs/mobiles/n1cur20w.txt

CHANGES IN THIS RELEASE
  Version 1.33

[Important updates]
  Nothing.

[New functions or enhancements]
- Updated the Diagnostics module to version 03.12.002.

[Problem fixes]
- Fixed TPM firmware update issue with TPM 1.2.
- Fixed a mismatch issue between the Intel(R) ME configuration and FPF fuse settings.
- Fixed an issue where fan might rotated with max speed.

VERSION INFORMATION

  The following versions of UEFI BIOS and ECP (Embedded Controller Program) have
  been released to date.

       Package      (ID)     UEFI BIOS (BIOS ID)  ECP (ECP ID)      Rev.  Issue Date
       --------------------  -------------------  ---------------   ----  ----------
fixed?!   1.33      (N1CUR20W)  1.33 (N1CET65W)      1.14 (N1CHT32W)   01    2018/02/19
cont.     1.31      (N1CUR19W)  1.31 (N1CET63W)      1.13 (N1CHT31W)   02    2018/02/06
cont.     1.31      (N1CUR19W)  1.31 (N1CET63W)      1.13 (N1CHT31W)   01    2017/12/26
1st occ.  1.30      (N1CUR18W)  1.30 (N1CET62W)      1.13 (N1CHT31W)   01    2017/11/28
OK        1.28      (N1CUR17W)  1.28 (N1CET60W)      1.11 (N1CHT29W)   01    2017/10/04
 

YMMV

cheers


jg

Thanks Peter Zhang, the 1.33 BIOS release for T460s went successfully through the evaluation phase and is now deployed in the company

(In reply to Rafael José from comment #82)
> Also having the problem on a X260. Fedora 27. Noticed after updating to bios
> v1.35, still present on v1.36.

Hello Rafael, The BIOS/EC updating failed usually happens in the scenario that you did not attach AC adapter. Could you please try it again? This issue is more related to ECFW than BIOS. For X260, it is R02HT32W. Thanks.

(In reply to jg from comment #83)

Hello George, Very appreciate your comments! Thank you!

(In reply to Charlemagne Lasse from comment #84)

Hi Charlemagne, 
It is glad to know your status. The test results of 1.33 from other users are also good by far. Don't hesitate to let us know if any more questions. Thanks.

Created attachment 1402869 [details]
Debian dmesg,lsmod,dmidecode,storage,uname

This archive contains five files generated on a Debian testing system that exhibits the issue:

* dmesg.txt
* lsmod.txt
* dmidecode.txt (dmidecode -t bios)
* storage.txt (lshw -class storage)
* uname.txt (uname -a)

Created attachment 1402870 [details]
Fedora 27 dmesg,lmod,dmidecode,storage,uname

This archive contains five files generated on a Fedora 27 system that *does not* exhibit the issue:

* dmesg.txt
* lsmod.txt
* dmidecode.txt (dmidecode -t bios)
* storage.txt (lshw -class storage)
* uname.txt (uname -a)

I have a T460s with a Samsung NVMe SSD that will ramp up the fan speed every time I wake it when running Debian Stretch or Debian testing with BIOS 1.31.  As with the issue described above, a cold boot is required to return to normal fan operation.

The exact same system dual-boots Fedora 27, which does not ramp up the fan speed on wake, even with BIOS 1.31.

For a while, the Fedora system would sometimes (once a month or so) ramp up the fan on wake.  Closing and reopening the lid would correct it.  I don't remember if it was Fedora 26 or 27 at the time, and it may have stopped after a previous firmware upgrade.  It hasn't happened lately and didn't happen often enough to be a problem.

I've been using Fedora 27 regularly for several months, and Fedora 26 before that since the beta.  The Debian system was added in the last ten days.

Upgrading to BIOS 1.33 fixes the problem in Debian.  Fedora remains unaffected with 1.33.

For comparison, an archive of some system reports are attached in the previous two comments.  The dmesg output includes the boot sequence and one cycle of closing and opening the lid.

The UEFI CSM is enabled on this T460s.

Thanks for fixing the issue in the 1.33 BIOS.  I hope this data point can help narrow down the root cause.

This message is a reminder that Fedora 26 is nearing its end of life.
Approximately 4 (four) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 26. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as EOL if it remains open with a Fedora  'version'
of '26'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version'
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not
able to fix it before Fedora 26 is end of life. If you would still like
to see this bug fixed and are able to reproduce it against a later version
of Fedora, you are encouraged  change the 'version' to a later Fedora
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's
lifetime, sometimes those efforts are overtaken by events. Often a
more recent Fedora release includes newer upstream software that fixes
bugs or makes them obsolete.

Fedora 26 changed to end-of-life (EOL) status on 2018-05-29. Fedora 26
is no longer maintained, which means that it will not receive any
further security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.