Bug 527874
|
Description
Peng Huang
2009-10-08 02:46:07 UTC
Created attachment 364054 [details]
dmesg after pm-suspent
[phuang@phuang-notebook ~]$ lspci |grep VGA
01:00.0 VGA compatible controller: ATI Technologies Inc M52 [Mobility Radeon X1300]
After below commands, I got the dmesg output.
1> switch vt from X to text console
2> echo 1 > /sys/module/drm/parameters/debug
3> pm-suspend --quirk-none
Created attachment 364056 [details]
The photo of my screen is suspend/resume in X
Exactly the same here. can you retry with the 2.6.31.1-65 or higher kernel from koji? Just tired 2.6.31.1-65.fc12.i686.PAE. The problem still happens. But the dmesg output is different. The old error is *ERROR* radeon: couldn't schedule IB(11)., after update the kernel, IB(11) changed to IB(13). ADDRCONF(NETDEV_UP): eth0: link is not ready Registered led device: iwl-phy0::radio Registered led device: iwl-phy0::assoc Registered led device: iwl-phy0::RX Registered led device: iwl-phy0::TX ADDRCONF(NETDEV_UP): wlan0: link is not ready e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready [drm:radeon_ib_schedule] *ERROR* radeon: couldn't schedule IB(13). [drm:radeon_cs_ioctl] *ERROR* Faild to schedule IB ! [drm:radeon_ib_schedule] *ERROR* radeon: couldn't schedule IB(13). [drm:radeon_cs_ioctl] *ERROR* Faild to schedule IB ! [drm:radeon_ib_schedule] *ERROR* radeon: couldn't schedule IB(13). [drm:radeon_cs_ioctl] *ERROR* Faild to schedule IB ! [drm:radeon_ib_schedule] *ERROR* radeon: couldn't schedule IB(13). [drm:radeon_cs_ioctl] *ERROR* Faild to schedule IB ! [drm:radeon_ib_schedule] *ERROR* radeon: couldn't schedule IB(13). [drm:radeon_cs_ioctl] *ERROR* Faild to schedule IB ! [drm:radeon_ib_schedule] *ERROR* radeon: couldn't schedule IB(13). [drm:radeon_cs_ioctl] *ERROR* Faild to schedule IB ! [drm:radeon_ib_schedule] *ERROR* radeon: couldn't schedule IB(13). [drm:radeon_cs_ioctl] *ERROR* Faild to schedule IB ! [drm:radeon_ib_schedule] *ERROR* radeon: couldn't schedule IB(13). [drm:radeon_cs_ioctl] *ERROR* Faild to schedule IB ! [drm:radeon_ib_schedule] *ERROR* radeon: couldn't schedule IB(13). [drm:radeon_cs_ioctl] *ERROR* Faild to schedule IB ! [drm:radeon_ib_schedule] *ERROR* radeon: couldn't schedule IB(13). [drm:radeon_cs_ioctl] *ERROR* Faild to schedule IB ! [drm:radeon_ib_schedule] *ERROR* radeon: couldn't schedule IB(13). [drm:radeon_cs_ioctl] *ERROR* Faild to schedule IB ! [drm:radeon_ib_schedule] *ERROR* radeon: couldn't schedule IB(13). [drm:radeon_cs_ioctl] *ERROR* Faild to schedule IB ! [drm:radeon_ib_schedule] *ERROR* radeon: couldn't schedule IB(13). [drm:radeon_cs_ioctl] *ERROR* Faild to schedule IB ! [drm:radeon_ib_schedule] *ERROR* radeon: couldn't schedule IB(13). [drm:radeon_cs_ioctl] *ERROR* Faild to schedule IB ! [drm:radeon_ib_schedule] *ERROR* radeon: couldn't schedule IB(13). [drm:radeon_cs_ioctl] *ERROR* Faild to schedule IB ! Hello Peng I have discussed with Dave about your issue so it seems VRAM is not properly restored after suspend, we need you to run 3 tools to get information: http://people.freedesktop.org/~glisse/atomtools.tar.bz2 http://people.freedesktop.org/~glisse/vbetool-0.7.tar.bz2 You will need to install before building each tools (for vbetool make should be enough, for atomtools: cmake . then make) On you got the tools built we will need you to run each of them in init 3 without kms (add "radeon.modeset=0 3" to your kernel boot parameters). Then do the following as root: ./vbetool post 2> x1400-bios-post ./atomtools > x1400-asicinit-asm ./atomtoools2 > x1400-asicinit-reg And attach the 3 files x1400-bios-post, 1400-asicinit-asm, x1400-asicinit-reg to the bug report. Thanks for your help. Sorry you need libpciaccess-devel.i686 package Created attachment 365188 [details]
x1300-bios-post
Created attachment 365190 [details]
x1300-asicinit-asm
After executing atomtools, my screen can not display text correctly.
Created attachment 365191 [details]
x1300-asicinit-reg
After executing ./atomtools > x1400-asicinit-asm, my screen can not display correctly. And I tried to execute `vbetool post 2` again, it can recover my display, and everything will be OK. I need more dump, can you please download : http://people.freedesktop.org/~glisse/radeondump.tar.bz2 And do the following: After a cold boot in init 3 without KMS (add radeon.modeset=0 to kernel cmdline): sudo ./radeondump -d x1400.regs x1400-init3 Then suspend/resume launch vbetool post and then: sudo ./radeondump -d x1400.regs x1400-init3-resume Then reboot with kms enabled, suspend/resume and do through ssh: sudo ./radeondump -d x1400.regs x1400-kms-resume Hopefully it should give us enough information to address this bug :) Created attachment 365296 [details]
clean boot without kms, initlevel = 3
Created attachment 365297 [details]
without kms, after resume
Created attachment 365298 [details]
without kms, after resume and vbetool post
Created attachment 365299 [details]
with kms, initlevel = 3, before resume
Created attachment 365300 [details]
with kms, initlevel = 3, after resume
Hope those files could help you resolve this issue. Can you test if kernel at: http://people.freedesktop.org/~glisse/ Fix the issue, install with rpm -ivh --nodeps don't worry about few warnings. Created attachment 365622 [details]
output of dmesg
The problem still happens with the new kernel.
BTW, I have a git clone of upstream linux kernel. I can help you to test patches too. Created attachment 365625 [details]
dump for new kernel with kms
Created attachment 365626 [details]
dump for new kernel with kms, after resume
Peng please run : sudo lspci -vvv -nn -xxxx -s 01:00.0 Before and after resume with KMS (replace 01:00.0 by the correct busid of your GPU a simple lspci should give it to you). Attach output as 2 different files. Thanks. Should I test it on the kernel in comment #19 ? Or use the latest kernel for fedora 12? Created attachment 366381 [details]
outputs of lspci
Hi Jerome, This problem also happens in level 3 without Xserver. Why put this bug to xorg-x11-drv-ati? Peng we assign kms bugs to X drivers because the kernel gets too many bugs, hopefully we can separate kernel stuff out later. Can you run the same lcpi command after resume & vbetool post with KMS disable in init 3. I put the to ati because it's easier for us to find ati hw related bug their. Created attachment 366412 [details]
output of lspci for nokms
Here are my observation before i forget about them : Register dump i asked are all register the vbetool post ever read or write on this specific hw. Thus if there is any difference in the way vbetool or KMS restore the card it should reflect in various dumps. It's not the case. The dumps show that with KMS VGA is disable, PLL are different too (because video mode setup by KMS and vbetool are different), of course video mode related register are different. Interesting things that diff btw dump shows, is that MC is idle on KMS and the 3D pipe configuration isn't restored. MC being idle could be either the source of the bug or just reflect the fact that VRAM is not working. If MC is not properly restored or in bogus state it could report IDLE because it doesn't answer to any memory request from the GPU. Or if VRAM is not properly restored MC can simply fail at executing request from the GPU and thus report IDLE. Otherwise all others register have similar values. My first attempt to fix the issue tried to reset the MC at resume, i found a bug in my patch i am working on new one which will do the following (order matter) : -stop MC at suspend -reset MC at resume -restore MC -ASIC_Init I am relooking at Atombios dump as i was looking at the wrong disasm of the atom bios tools, to check if vbetool post takes a different path than ASIC_Init. I think I am seeing this problem also on an old ThinkPad T41 with RV250 on resume. I first see this: radeon 0000:01:00.0: PCI INT A -> Link[LNKA] -> GSI 11 (level, low) -> IRQ 11 Oct 29 10:02:08 t41 kernel: [drm] GPU reset succeed (RBBM_STATUS=0x00000140) Oct 29 10:02:08 t41 kernel: [drm] radeon: cp idle (0x02000000) Oct 29 10:02:08 t41 kernel: [drm] radeon: ring at 0x00000000D0000000 Oct 29 10:02:08 t41 kernel: [drm:r100_ring_test] *ERROR* radeon: ring test failed (sracth(0x15E4)=0xCAFEDEAD) Oct 29 10:02:08 t41 kernel: [drm:r100_cp_init] *ERROR* radeon: cp isn't working (-22). Oct 29 10:02:08 t41 kernel: radeon 0000:01:00.0: failled initializing CP (-22). Oct 29 10:02:08 t41 kernel: [drm] LVDS-13: set mode 1400x1050 1e After which I get a continuous stream of these errors: [drm:radeon_cs_ioctl] *ERROR* Faild to schedule IB ! [drm:radeon_ib_schedule] *ERROR* radeon: couldn't schedule IB(7). My display is garbled, but I can switch to a VT. Would it help if I collected the debug data also? kernel-2.6.31.5-97.fc12.i686 xorg-x11-drv-ati-6.13.0-0.10.20091006git457646d73.fc12.i686 xorg-x11-server-Xorg-1.7.1-1.fc12.i686 not sure if it helps, but after installing the -104 kernel from koji, IB(7) changed to IB(11) Robert i am confident your issue is different. Please open a new bug with following bug title: RADEON:RV250:KMS Suspend/Resume fails (ThinkPad T41) Attach full output of lspci -v and full dmesg after resume. Thanks. Created attachment 366649 [details]
Stop mc at suspend and reset it at resume
Please try the attached patch it apply on top of lastest drm-next branch of Dave repo.
I'm seeing the same issue on my T60 with the X1400 as in comment #5. Everything works until I suspend/resume, after which the whole screen is garbled and blinking, but otherwise responsive. This is with 2.6.32-rc5-git4. Unfortunately the drm-next branch didn't work (unclear why, produced a hard lockup), so I stuck with drm-linus which seemed to contain a few safe bugfixes. After applying your proposed workaround patch, things are still not working, and the error message you added is triggered: "[drm] (rv370_pcie_gart_set_page 78) VRAM seems to not work properly !", which seems to get emitted every time I am switching between a VT and the X server (none of which puts the graphics card into a sane state), with tons of the "couldn't schedule IB(15)" around them. Peng, Christophe can you try to build : http://people.freedesktop.org/~glisse/radeonvram.tar.bz2 You will need libpciaccess-dev (iirc name correctly). Than boot with KMS enabled in init 3 (add 3 to kernel boot cmd line). Suspend/resume and on resume when you get garbled screen run radeondump program (which is in radeonvram.tar.bz2) as root and report the output of the program, you likely need to do all this through ssh from another computer. Thanks. Before suspending: Found card 1002:7145 RV515 region: (base: 0x0000000000000000, bus: 0x00000000D8000000, size: 134217728, is_io: 0) region: (base: 0x0000000000000000, bus: 0x0000000000002000, size: 256, is_io: 1) region: (base: 0x0000000000000000, bus: 0x00000000EE100000, size: 65536, is_io: 0) region: (base: 0x0000000000000000, bus: 0x0000000000000000, size: 0, is_io: 0) region: (base: 0x0000000000000000, bus: 0x0000000000000000, size: 0, is_io: 0) region: (base: 0x0000000000000000, bus: 0x0000000000000000, size: 0, is_io: 0) BUS_CNTL: 0x00000001 CONFIG_CNTL: 0x00020100 CONFIG_MEMSIZE: 0x08000000 COMMAND|STATUS: 0x00100107 vram_test_hdp succeed After resuming: Found card 1002:7145 RV515 region: (base: 0x0000000000000000, bus: 0x00000000D8000000, size: 134217728, is_io: 0) region: (base: 0x0000000000000000, bus: 0x0000000000002000, size: 256, is_io: 1) region: (base: 0x0000000000000000, bus: 0x00000000EE100000, size: 65536, is_io: 0) region: (base: 0x0000000000000000, bus: 0x0000000000000000, size: 0, is_io: 0) region: (base: 0x0000000000000000, bus: 0x0000000000000000, size: 0, is_io: 0) region: (base: 0x0000000000000000, bus: 0x0000000000000000, size: 0, is_io: 0) BUS_CNTL: 0x00000001 CONFIG_CNTL: 0x00020000 CONFIG_MEMSIZE: 0x08000000 COMMAND|STATUS: 0x20100107 vram_test_hdp failed vram_test_gpu succeed Sorry, I cut off the last line containing a "vram_test_gup succeed" from the first output before suspending. This is BTW with the last patch you posted (where you attempt to reset the part that supposedly broke). *** Bug 522253 has been marked as a duplicate of this bug. *** Just an interesting observation from just now: I put my laptop into suspend out of habit, and now that resumed it about an hour later (as opposed to my tests where I always suspended it for like 5 seconds), surprisingly it came back correctly. It shows: BUS_CNTL: 0x00000001 CONFIG_CNTL: 0x00020000 CONFIG_MEMSIZE: 0x08000000 COMMAND|STATUS: 0x00100107 vram_test_hdp succeed vram_test_gpu succeed While CONFIG_CNTL is 0x20000 like for the non-working case after resuming, COMMAND|STATUS doesn't have 0x20 in the upper byte, but 0x00 instead like before resuming. So quick comment it seems vram is properly working but that we can't access it through HDP (pci aperture). I will do a patch to reset hdp after resume to see if it helps. (Btw this is kind of good news as it means VRAM is likely properly restored by atombios which was my feeling). Created attachment 367460 [details]
Reset HostDataPath at resume
Please test this patch which apply on top of drm-next branch of Dave repo:
git://git.kernel.org/pub/scm/linux/kernel/git/airlied/drm-2.6.git
I can generate rpm if you want but this will take me sometime.
I'm afraid to tell that this patch doesn't make any difference. If the power is plugged in, it always comes back in a broken state (and if it is not, chances are good that it does, as before) So, I guess it must be something else. If I had the slightest idea how modern graphics hardware works, I could have tried to help you figuring things out, but unfortunately I don't... Since this bugzilla report was filed, there have been several major updates in various components of the Xorg system, which may have resolved this issue. Users who have experienced this problem are encouraged to upgrade their system to the latest version of their packages (at least F12Beta, but even better if the very latest versions). Please, if you experience this problem on the up-to-date system, let us now in the comment for this bug, or whether the upgraded system works for you. If you won't be able to reply in one month, I will have to close this bug as INSUFFICIENT_DATA. Thank you. [This is a bulk message for all open Fedora Rawhide Xorg-related bugs. I'm adding myself to the CC list for each bug, so I'll see any comments you make after this and do my best to make sure every issue gets proper attention.] This problem still happens in F12 beta on a Clevo D870P with Mobility Radeon 9700. David you more than likely have another issue, this one is specific to X1400 on T60. Please open a new bug with dmesg, Xorg.log and a description of what you are seeing on resume. Also if it's an AGP GPU try booting with radeon.agpmode=-1 and report in the bug if it works when doing that. Thanks. Created attachment 367798 [details]
X1400 restore mc+hdp before asic_init and put vram at 0x10000000
Please try this patch, top of drm-next again, hope it works
The last two patches still have the problem. BTW, this week, I am on trip, so can not get more logs. Created attachment 368232 [details]
X1400 restore mc+hdp before asic_init and put vram at 0x10000000 + VGA HDP
Please test new patch. In this version i program the VGA HDP, maybe some VGA stuff happens at one point. Crossing fingers, but i don't think this one will help much.
ok, a step back :-( with the updates of today my laptop doesn't even boot anymore with KMS. I get a hard hang during modeset. using nomodeset allows the laptop to boot. kernel.x86_64 2.6.31.5-127.fc12 xorg-x11-drv-ati.x86_64 6.13.0-0.10.20091006git457646d73.fc12 xorg-x11-server-Xorg.x86_64 1.7.1-7.fc12 mesa-dri-drivers.x86_64 7.6-0.13.fc12 Yes, I saw this too with drm-next at some point, since then I decided to stick with drm-linus. Anyhow, no luck with the latest patch either. :( Can anyone of you confirm the strange effect that 90% of the times everything comes back as it should if you have the notebook unplugged when resuming? (In reply to comment #50) > Created an attachment (id=368232) [details] > X1400 restore mc+hdp before asic_init and put vram at 0x10000000 + VGA HDP > > Please test new patch. In this version i program the VGA HDP, maybe some VGA > stuff happens at one point. Crossing fingers, but i don't think this one will > help much. Hi Jerome, Which version does the patch base on? I can not apply the patch on drm-next of git://git.kernel.org/pub/scm/linux/kernel/git/airlied/drm-2.6.git successfully. Created attachment 368412 [details]
Shutdown lvds, force mc to be on, dump regs
Please try new patch, should apply cleanly on top of drm-next. This one take a different path i try to shutdown things at suspend and reactivate them at resume it also dump few registers which might be helpfull to further debug the issue. Please try it and attach full dmesg after a suspend/resume cycle. Thanks
Created attachment 368963 [details]
dmesg output
The problem and NMI still happens with last patch.
I see the same effect. Also, nothing useful in the debug output. I was wondering what could trigger the NMI. I read somewhere (I know that is not a very reliable citation, was somewhere in a forum) that it doesn't need to be the device itself, it might also be the bus. I looked at lspci output and also checked the PCIE bridge. It is some sort of memory access from the CPU through a PCIE "aperture" that isn't working, right? Can the bridge be at fault perhaps? Here are the diffs for the bridge (00:01.0) and the GPU (01:00.0) between a successful resume (with power unplugged) and a failed resume: Here the working full output of lspci -vvv 00:01.0 00:01.0 PCI bridge: Intel Corporation Mobile 945GM/PM/GMS, 943/940GML and 945GT Express PCI Express Root Port (rev 03) (prog-if 00 [Normal decode]) Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Bus: primary=00, secondary=01, subordinate=01, sec-latency=0 I/O behind bridge: 00002000-00002fff Memory behind bridge: ee100000-ee1fffff Prefetchable memory behind bridge: 00000000d8000000-00000000dfffffff Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR- BridgeCtl: Parity- SERR- NoISA+ VGA+ MAbort- >Reset- FastB2B- PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn- Capabilities: [88] Subsystem: Lenovo Device 2014 Capabilities: [80] Power Management version 2 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+) Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME- Capabilities: [90] MSI: Enable- Count=1/1 Maskable- 64bit- Address: 00000000 Data: 0000 Capabilities: [a0] Express (v1) Root Port (Slot+), MSI 00 DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us ExtTag- RBE- FLReset- DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- MaxPayload 128 bytes, MaxReadReq 128 bytes DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend- LnkCap: Port #2, Speed 2.5GT/s, Width x16, ASPM L0s L1, Latency L0 <1us, L1 <4us ClockPM- Surprise- LLActRep- BwNot- LnkCtl: ASPM L0s L1 Enabled; RCB 64 bytes Disabled- Retrain- CommClk- ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 2.5GT/s, Width x16, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surpise- Slot # 1, PowerLimit 75.000000; Interlock- NoCompl- SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg- Control: AttnInd Off, PwrInd On, Power- Interlock- SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt+ PresDet+ Interlock- Changed: MRL- PresDet- LinkState- RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna- CRSVisible- RootCap: CRSVisible- RootSta: PME ReqID 0000, PMEStatus- PMEPending- Capabilities: [100] Virtual Channel <?> Capabilities: [140] Root Complex Link <?> Kernel driver in use: pcieport and the diff to after a failed resume: @@ -1,6 +1,6 @@ 00:01.0 PCI bridge: Intel Corporation Mobile 945GM/PM/GMS, 943/940GML and 945GT Express PCI Express Root Port (rev 03) (prog-if 00 [Normal decode]) Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx- - Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- + Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR+ <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Bus: primary=00, secondary=01, subordinate=01, sec-latency=0 I/O behind bridge: 00002000-00002fff @@ -21,7 +21,7 @@ DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- MaxPayload 128 bytes, MaxReadReq 128 bytes - DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend- + DevSta: CorrErr+ UncorrErr+ FatalErr- UnsuppReq+ AuxPwr- TransPend- LnkCap: Port #2, Speed 2.5GT/s, Width x16, ASPM L0s L1, Latency L0 <1us, L1 <4us ClockPM- Surprise- LLActRep- BwNot- LnkCtl: ASPM L0s L1 Enabled; RCB 64 bytes Disabled- Retrain- CommClk- I am just wondering, the "Uncorr Err" and "UnsuppReq" went to +. Also, for the GPU MAbort went to + as well. Two patches here http://people.freedesktop.org/~airlied/scratch/0001-drm-radeon-kms-fix-handling-of-d1-d2-vga.patch http://people.freedesktop.org/~airlied/scratch/0002-drm-radeon-kms-read-back-register-before-writing-in-.patch Can you guys please try them, I'll try and make a Fedora kernel with them in it ASAP, they are also on the drm-radeon-testing of my drm-2.6 tree. I've tested them on Peng's laptop with my USB disk, hopefully when he tests them with his normal install they also work. I almost feel bad by telling you this, but I am now running 2.6.32-rc6 + drm-radeon-testing and made sure these two patches are in it - but it's still giving the same results as before. If the notebook is unplugged on resume, there's a 90% of it coming back correctly, otherwise not. In order to avoid any confusion: With "unplugged" I am referring to the power, not an external monitor (as in the description of patch no 1). OMG, I'm so stupid. I was actually booting the wrong kernel when doing my last test (I made sure the modules contained the patch but I never checked if I was actually loading it). I can confirm this patch is fixing the issue for me. Great. :-) Sorry for the confusion, I hope I didn't cause any additional work. Dave's patches also fixed the suspend/resume problem on a KMS-enabled T60/x1400 for me. The patch can fix this S/R problem. Thanks This bug appears to have been reported against 'rawhide' during the Fedora 12 development cycle. Changing version to '12'. More information and reason for this action is here: http://fedoraproject.org/wiki/BugZappers/HouseKeeping Thank you for letting us know. Will this update be available for F-12 (closed as rawhide)? |