Update:

I thought the flickering problem was fixed, but there is a caveat. Under VLC and Xine the video flickers, but under SMPlayer, using X11 output the flicker is gone. Plus, the X11 output is the same quality as the XVideo or OpenGL output of other players.After upgrading to Ubuntu Hardy Xine crashed with two nasty errors: X Error of failed request… and option glx-shm does not exist. In addition to this, VLC output was blocky and I could not force VLC to work correctly with OpenGL simply changing the output method to OpenGL would crash VLC on startup. This is probably an issue users of modern ATi graphics cards will have, as the drivers aren’t really all that.

So here’s how to get Ubuntu Hardy video output sorted and have Compiz running too; as unlike Gutsy, Hardy does not fall foul of the ‘flickering video’ in Compiz. I know many people have issues with video under Compiz when using Fglrx, so this may help those people.

First, you should install Envy-Ng — you will need to configure Synaptic to use the Multiverse repository to install Envy-Ng– and let it sort out your graphics cards drivers. This program will keep your drivers uptodate and sort out your xorg.config for you. Use Envy-Ng to install the latest drivers for your card (at current Envy will install version 8-3.)You should then restart the X-Server (Ctrl-Alt-Backspace.)

Next up you will need to sort out your Xorg to run XVideo, without it being so blocky. This is due to Envy not setting the options for XVideo correctly. Open up xorg.conf with the following command in the terminal: sudo /etc/X11/xorg.conf

Now change the section “Device” to have TexturedVideoSync, TexturedVideo, and VideoOverlay on and have OpenGLOverlay off. The section should look as follows:

Section “Device”
Identifier    “aticonfig-Device[0]”
Driver        “fglrx”
Option        “TexturedVideoSync”    “on”
Option         “TexturedVideo” “on”
Option        “VideoOverlay”    “on”
Option        “OpenGLOverlay”    “off”
EndSection

You may want to check the following sections, with the following setting I get Compiz effects with no flickering video.

Section “ServerFlags”
Option        “AIGLX”    “true”
EndSection

Section “DRI”
Group        “video”
Mode    0666
EndSection

Section “Extensions”
Option        “RENDER”    “Enable”
Option        “DAMAGE”    “Enable”
Option        “XVideo”    “Enable”
Option        “Composite”    “Enable”
EndSection