How does hardware assisted video decoding work?

[dE_logics]

If you're doing hardware assisted video decoding, the GPU should assist decoding the video; but the GPU is limited to decoding a few set of videos formats, commonly mpeg4, h264 and mpeg2.

There're various libraries available to make GPU capabilities available as an easy API which video players (and libraries) can implement. e.gs include vaapi, vdapu, xvmc, and libxv

In most players (mplayer, VLC, xine, Gstreamer), you've to specify the 'output driver' which specifies the way in which the video will be decoded; some of these output drivers are implemented using the libraries that are hardware accelerated, e.g. the xv driver of VLC, mplayer and gstreamer, the vdapu driver of mplayer etc...

Is all this true? Cause I've not read any of this anywhere; this's just what I think, and I also thing it's wrong cause --

If I use a hardware accelerated (e.g. vaapi) driver for a video which the hardware can't decode, what will happen in mplayer atleast?

What's gl and gl2 output in mplayer? Hardware decode the video using openGL GPU capabilities? On my hardware, this doesn't appear to reduce any CPU usage compared to XV, instead it increases by 1 or 2%.

In VLC (which supports VAAPI and should use hardware to decode h264 on my i3 with HD2000), I cant figure it out if VLC is using the hardware or not. In codecs section, if I enable hardware h264 decoding, then the CPU usage increases instead, and there's no vaapi option in the video output drivers list.

Another question about VLC and mplayer is that, if it's the video output driver's responsibility to do hardware decoding, then what are those 'enable hardware acceleration' buttons doing?

[Dorsai!]

On Mplayer, most of the video output options just decide the way the video is drawn on the screen. xv should also be accelerated if you have a gl compatible driver active, but is the more compatible solution as gl works only if a capable driver is loaded. But the actual decoding is fully done by the CPU.

vdpau instead does something else. the full video stream, together with the place the final video should end up on the screen is handed to the graphics card which does all the decoding and displaying by itself.

vaapi is just a wrapper around different concepts, as vdpau and its AMD equivalent share different apis.

VLC uses a different approach by only letting the card decode the video and getting back the raw video stream for further processing. It does not work directly with vdpau but uses some sort of shader language based decoding if I am not mistaken (but I might be, I dont use VLC myself).

[Gusar]

vaapi is just a wrapper around different concepts

vaapi is not a wrapper, it's a full API just like VDPAU and XvBA are.

@dE_logics: What VLC does is it sends the bitstream to the graphic card to be decoded and then copies decoded frames back into system memory. MPlayer doesn't do such copying, the graphic card displays the video directly. That's why VLC is less efficient.

Also, decoding is *not* done on the GPU, but on a dedicated ASIC that's part of the graphic card. This dedicated ASIC is single-purpose - it decodes specific formats (mpeg2, h264, VC1), and that's it. The reason this was done is that GPUs suck at video decoding.

The gl and gl2 in mplayer outputs are just that - outputs. They are means to display the picture on the screen. Their primary job is colorspace conversion (the video is YV12 while displays are RGB, so you need to convert the video in order to display it) and scaling (so that you can watch the video in fullscreen).

MPlayer and other players have several such means of displaying the picture. The most common one is Xv. It's an X extension that in the past provided access to a graphic card's hardware overlay. This overlay was a dedicated part of the graphic chip responsible for colorspace conversion and scaling. Modern graphic cards don't have such overlays anymore, the 3D engine is used to perform colorspace conversion and scaling.

[dE_logics]

On Mplayer, most of the video output options just decide the way the video is drawn on the screen. xv should also be accelerated if you have a gl compatible driver active, but is the more compatible solution as gl works only if a capable driver is loaded. But the actual decoding is fully done by the CPU.

vdpau instead does something else. the full video stream, together with the place the final video should end up on the screen is handed to the graphics card which does all the decoding and displaying by itself.

vaapi is just a wrapper around different concepts, as vdpau and its AMD equivalent share different apis.

VLC uses a different approach by only letting the card decode the video and getting back the raw video stream for further processing. It does not work directly with vdpau but uses some sort of shader language based decoding if I am not mistaken (but I might be, I dont use VLC myself).

Ok, so, in mplayer, the driver actually changes the way the video gets decoded and displayed on the screen. So in vdapu, it may happen that the actual video after being decoded by the GPU gets displayed on the screen using Xv or gl/gl2; or maybe we can set options for the driver and customize it.

[dE_logics]

vaapi is just a wrapper around different concepts

vaapi is not a wrapper, it's a full API just like VDPAU and XvBA are.

@dE_logics: What VLC does is it sends the bitstream to the graphic card to be decoded and then copies decoded frames back into system memory. MPlayer doesn't do such copying, the graphic card displays the video directly. That's why VLC is less efficient.

Also, decoding is *not* done on the GPU, but on a dedicated ASIC that's part of the graphic card. This dedicated ASIC is single-purpose - it decodes specific formats (mpeg2, h264, VC1), and that's it. The reason this was done is that GPUs suck at video decoding.

The gl and gl2 in mplayer outputs are just that - outputs. They are means to display the picture on the screen. Their primary job is colorspace conversion (the video is YV12 while displays are RGB, so you need to convert the video in order to display it) and scaling (so that you can watch the video in fullscreen).

MPlayer and other players have several such means of displaying the picture. The most common one is Xv. It's an X extension that in the past provided access to a graphic card's hardware overlay. This overlay was a dedicated part of the graphic chip responsible for colorspace conversion and scaling. Modern graphic cards don't have such overlays anymore, the 3D engine is used to perform colorspace conversion and scaling.

But what happens if I change the output driver to vdapu? If the output driver handles only displaying the video on screen, then how can vdapu modify the decoding technique?

[pigeon768]

There are two different things to talk about here. There is the output driver, and the codec. You need both to display a video, and both can have dramatic effects on performance.

xv, gl, gl2 are output drivers. Once a video is decoded, the output driver will take care of transporting the raw video frames (at this point it's basically a sequence of bitmaps) from your media player's process memory to your display. xv uses a system designed for playing video, gl and gl2 will use the more general 3d graphics system.

vdpau and vaapi will both decode the compressed video and perform the function of the output driver. Since when the video is decoded to a raw frame, the raw frame is already in the video card's memory, it wouldn't make sense to use the same driver to display it to the screen. They've also gone and made vdpau and vaapi available as generalized output drivers, so you can use that functionality even if the compressed video can't be decoded by the video card.

In mplayer, with vdpau, you configure mplayer to decode the video with vdpau with the command line argument `-vc ffh264vdpau,ffodivxvdpau,ffmpeg12vdpau,ffwmv3vdpau,ffvc1vdpau`. It will attempt to use one of those codecs, all of which utilize vdpau, in that order, and if none of them work it will fail. If you use `-vc ffh264vdpau,ffodivxvdpau,ffmpeg12vdpau,ffwmv3vdpau,ffvc1vdpau,` (note the extra comma) if none of those codecs work it will fall back to the default libavcodec codecs.

You configure mplayer to use the vdpau output driver with `-vo vdpau`. This will work regardless of whether or not mplayer falls back to the libavcodec codec.

I don't use vlc, and mainline mplayer does not yet support vaapi, so I can't help you with your other questions.

[Gusar]

So in vdapu, it may happen that the actual video after being decoded by the GPU gets displayed on the screen using Xv or gl/gl2; or maybe we can set options for the driver and customize it.

That's possible in theory, but MPlayer doesn't support it. If you decode using VDPAU, you need to display using VDPAU. That's simply how MPlayer has been coded. Using other output methods would require copying decoded frames back into system memory, mplayer doesn't support that. But, like I said, VLC does.

But what happens if I change the output driver to vdapu? If the output driver handles only displaying the video on screen, then how can vdapu modify the decoding technique?

I don't get what you mean here. VDPAU, also VAAPI, have two parts: decoding and presentation (displaying the video). You can use VDPAU presentation regardless of how the video was decoded (you can decode the video in software, then use VDPAU to display it). But if you decode using VDPAU, then it depends how a particular player was coded.

[dE_logics]

Thanks everyone!

Things are 100% clear now and this post comes up on the first page of Google search result for "how hardware decoding works".

[pd1986]

Thanks everyone!

Things are 100% clear now and this post comes up on the first page of Google search result for "how hardware decoding works".

good news, good for spreading the knowledge and GENTOO, haha

In fact, I am interested in this too. as I have ATI card, but I often use mplayer. In mplayer of official version, vaapi isn't included, while it is in vlc. but I really don't find a large difference of cpu use between mplayer and vlc when I play a movie of mkv in 1080p. 15%-20% by vlc vs 20%-25% by mplayer.

One time I heard somewhere, if I choose xv, I could still get the hardware acceleration. can't remember where it is from.

[dE_logics]

Thanks everyone!

Things are 100% clear now and this post comes up on the first page of Google search result for "how hardware decoding works".

good news, good for spreading the knowledge and GENTOO, haha

In fact, I am interested in this too. as I have ATI card, but I often use mplayer. In mplayer of official version, vaapi isn't included, while it is in vlc. but I really don't find a large difference of cpu use between mplayer and vlc when I play a movie of mkv in 1080p. 15%-20% by vlc vs 20%-25% by mplayer.

One time I heard somewhere, if I choose xv, I could still get the hardware acceleration. can't remember where it is from.

Join the discussion --

http://forum.videolan.org/viewtopic.php?f=13&t=104718

[Gusar]

One time I heard somewhere, if I choose xv, I could still get the hardware acceleration.

Only in VLC, which copies frames into system memory. Which takes CPU time, so maybe that's why you don't see much difference between hardware and software decoding.

MPlayer doesn't do any copying, it displays the frames directly, which makes it more efficient than VLC. That's why you need to use a specific video output. For VDPAU you also need to specify the appropriate hardware codec (one of ffh264vdpau/ffmpeg12vdpau/ffvc1vdpau/ffodivxvdpau depending on the file you want to play).

To sum up regarding MPlayer:
Hardware decoding with VDPAU

Code: Select all

-vo vdpau -vc ffh264vdpau

Hardware decoding with VAAPI (requires a patched mplayer)

Code: Select all

-vo vaapi -va vaapi

[dE_logics]

I tried VLC on 1st generation mobile i3 and the CPU usage doped to half.

Anyway, XBMC is currently the only easily available alternative that works. Also, those mplayer-vaapi ebuilds don't work.

[BrandonKByrne]

hello everyone

In fact, I am interested in this too. as I have ATI card, but I often use mplayer. In mplayer of official version, vaapi isn't included, while it is in vlc. but I really don't find a large difference of cpu use between mplayer and vlc when I play a movie of mkv in 1080p. 15%-20% by vlc vs 20%-25% by mplayer. thanks for sharing such a nice question

[Gusar]

@BrandonKByrne:

Which driver are you using? The open source radeon driver only decodes mpeg1/2 in hardware using the Gallium3D VDPAU implementation. And even that is not done using the dedicated hardware decoder, but with shaders.

The closed source Catalyst/fglrx driver uses XvBA to access the hardware decoder. Only a separate branch of XBMC has direct support for XvBA. Though there's also xvba-video, a wrapper that allows using hardware decoding in VAAPI players.

[dE_logics]

I tried VLC on 1st generation Intel i3, and the CPU usage was halved; but not in my or anyone else's case.