Friday, 29 August 2008

The Checkerboard of Doom solved! :)

If you're like me, then you've probably been having a hell of a time (re)searching around, swearing, cursing, testing stuff, even trying to discuss things that have already been discussed regarding the screen corruption that occurrs when running a WINE application in fullscreen, while having moderately recent AMD/ATI proprietary catalyst/fglrx (yay rebranding) drivers. The screen corruption was symbolically called "The Checkerboard of Doom", since it makes the screen look like a checkerboard. And it stays that way until you restart your computer.

Well, ATI Catalyst (AMD, flgrx, whatever) version 8.8 has fixed the "permanent" checkerboard, which means you only have to close the fullscreen WINE application and the corruption is gone.

Some people did even better than that - they found various workarounds for this bug, some of which were better, while others weren't. Among the interesting ones I found were resizing your virtual display to a horizontal size which is divisible by 64 (e.g. if your monitor is 1400x1050 like mine, you just set the virtual to 1408x1050), while others used a combination of the open-source LibGL and the proprietary display driver.

This has lead me into thinking that it's the LibGL which has some broken stuff, and I've been trying out stuff regarding that for some time. Another clue was that, when you make a screenshot of the Checkerboard, the image is perfectly normal - not a trace of any corruption. Hmm. Would this mean that there is a problem in the communication between the display and the graphic driver? This may eventually also point to the LibGL. I don't have the brains for seeking out solutions there. At least not yet. Sorry.

zeddD1abl0, a good soul on the Arch Linux forum pointed out a fact that he hasn't seen the checkerboard in ages - and he mostly uses the defaults. This doesn't mean he's lazy or unimaginative, it just means he follows the KISS principle and guess what - it works for him.

Not having a better thing to do, and taking a look at what I've done to my Arch installation so far, I've decided to wipe out some stuff and install the whole GUI from scratch. Removed the config files and the stuff I've edited in /etc directory, and unmerged (uninstalled - damnit, Gentoo has obviously left some permanent marks in my brain) the whole xorg, together with the drivers, desktop environment (GNOME, e17, fluxbox and compiz-fusion-gtk), and everything that depended on it - which means, in essence, every single graphical application I've had installed. Good thing I wrote everything down in a textfile while I was doing it. I really didn't feel like reinstalling the whole system, bcos I've made some changes in the configuration which suit me, and don't feel like doing it all over again.

After reinstalling the xorg (and generating a new, plain xorg.conf), latest catalyst only (and simply running aticonfig --initial), GNOME and WINE, guess what happened.
I opened a Windows game in fullscreen, and it worked. I could actually see what's happening on my monitor. No squares. No checkerboard. It worked.

I was tempted to leave everything as is, but, craving for at least a little more performance as I always am, I tried adding lines - one by one - to my xorg.conf and see what will generate the checkerboard.

My first guess was that the "Option" "UseFastTLS" "2" had something to do with it, since setting it to 0 or 1 often broke compatibility with WINE. I thought this one was finally broken also, but I was wrong. Believe it or not, the critical stuff is Section "DRI".
Why? My first guess - it's the section that concerns the xorg DRI extension, not fglrx's. And now, having xorg driver removed and fglrx installed, and having in mind that it's DRI extension is far from similar to xorg's, the options you would give it could make it break. If my suspicion is correct, we're lucky to have only WINE broken (well, not really broken anymore, just not compatibile with the bullshit we're trying to feed it).

Anyway, to sum things up, here's how to solve the problem with the infamous Checkerboard of Doom:

1. download and install the latest AMD/ATI Catalyst (ex. fglrx) drivers. The way of doing this depends on your distro, but it seems they have to be at least version 8.7 or 8.8
2. run aticonfig --initial (you may add the "-f" switch in the end)
3. open your xorg.conf (in most cases it is located in /etc/X11/xorg.conf) with your favourite text editor. Mine is nano.
4. find the Section "Module" and comment out the lines which concern loading GLX and DRI. The proprietary fglrx (catalyst) driver has it's own modules, which have to be used rather than those.
5. find the Section "DRI" and comment (or even delete) everything out. That's right, from the Section "DRI" to EndSection, everything must disappear from your xorg.conf as if it was never there. The Section "DRI" was once pointed out by the Compiz Fusion team as necessary for running it under ATI graphics card with proprietary drivers (I don't know about the others, sorry), but this is not needed any longer. It's existence doesn't produce any problems with Compiz Fusion, but WINE hates it, as of Catalyst 8.5 (or 8.4, don't quite remember anymore).


That's it. You may restart your computer (I believe you should do so right now, since restarting X isn't completely stable yet) and check your favourite WINE application. It should work.

Cheerz =]

1 comment:

Steve said...

OH MY GOD!!! You have no idea how long I've been looking for a fix for this.

I haven't used any linux distro's in a while, and wanted to test out my new Radeon 4850 on linux to see both native linux performance and WINE performance.

I will try your method as soon as I can get the time to fiddle with my computer (probably over the weekend). I already thank you for posting your info concerning your problem with the checkerboard of doom, and thank you in advance if the fix works!!!