mmal on rpi64

Message

axl · Post by **axl** » Wed Sep 25, 2019 9:34 pm

So I asked point blank Sakaki & anyone else (here - https://forums.gentoo.org/viewtopic-p-8 ... ml#8370868) if mmal in 64 is really that important.

I was expecting either sakaki or neddy to come and tell me I'm wrong.

Listen. aarch64 is great. works a little bit faster. i'm not sure it will fix all problems 6j and 7a had or maybe 8a will have, but it did show some improvements in terms of performance vs armv32. But that being said, in 64 bits the mmal components doesn't work.

I just want someone to say it out loud. It's not my incompetence, you can't make any multimedia thing work in 64 bits.

I wanted to repeat this question in other threads, but I thought it was impolite to stain a thread that is not mine.

So I made my own thread. I doubt sakaki will visit it, but anyone is invited to contribute.

My premise is that most of the video/3d things work only in 32 bits. 64 bits does work, but works best for HEAD-LESS setup.

In some weird sense, everybody is excited for the aarch64, but in reality, it has no practical application. I can't think of one.

Let me me something very clear. I'm ALL for "because i can". and aarch64 is exactly that. because i can.

In terms of raspberry pi gpu, you simply can't in 64 bits and someone should just say it out loud.

That's just my opinion... what's yours? I'd love to correct what I done wrong...

Post by **NeddySeagoon** » Wed Sep 25, 2019 9:57 pm

axl,

mmal is supposed to work on the Pi in 64 bit mode. I've not knowingly tested it myself.

The GPU on the Pi is 32 bit only, so any 64 bit code has to speak 32 bit to the GPU and understand 32 bit responses.
Its a hardware feature. The VC4 is 32 bit.

mmal is on my list of things to poke at but its very low down.
I unexpected got a 4k image out of my Pi4. My Pi is connected via HDMI on the display and the display is not supposed to support 4k on the HDMI ports.
The desktop was very slow in 4k and it shouldn't have been.

With mmal, the Pi should be able to play 4k videos.

Gavinmc42 · Post by **Gavinmc42** » Thu Sep 26, 2019 5:49 am

You have to read a bunch of 64bit Raspbian posts but buried in there is the answer.
It should be possible but it was not on the RPF list of important things at the time.

As Raspbian Buster is moving towards 64bit things like camera etc are starting to work.
With the PI4 being such a great device more smart people will figure things out.
And now it is probably on RPF's list of things to do.
Wish we had some manuals.
Time will tell.

I do miss the OpenVG

But I get Shaders as a consolation prize

axl · Post by **axl** » Thu Sep 26, 2019 11:32 am

@neddy 2 different things. vc4 is one thing, mmal is another. vc4 is the gpu itself used for rendering, and it's handled through kernel & mesa. Hmmm, I didn't know that worked in 32 bits as well, but hey, that was no problem getting it to work, now or in previous attempts. It's true I wasn't trying to get 4k to work, as I have no interest in that whatsoever, however omxplayer & kodi do interest me. And since mmal doesn't work in 64 bits, raspberry-pi flag is disabled in ffmpeg and since that is disabled ... no hardware decoding.

@Gavinmc42 in order for the camera to work, they changed the way it is used. Now instead of it being managed by raspberry pi userland, it is managed by v4l in kernel. that means instead of using raspistill and raspivid, you have to use /dev/video0.

My point is that before embarking into this long long ride, people should at least know what to expect at the end of it.

PS and based on the discussions I've read, the mmal port to 64 bits wont happen easily. Or soon. And it kinda is important.

Gavinmc42 · Post by **Gavinmc42** » Thu Sep 26, 2019 12:43 pm

If I have used mmal I was probably not aware of it.

I use Baremetal, it is based on Ultibo so I use the fpc based libraries and don't have much idea of how they work except it's OpenGLES/VG/EGL/Dispmanx.
Yep was aware of the cameras via V4L in the Pi4 but have not tested that.
dev/video0 is mainstream so will probably just work.
Emerge Cheese tonight, let you know tmr?
Got cameras here on the desk somewhere

SDL 2.0 seems to work, can it substitute?

The entire VC4 blob is not going to be ported to Pi4.
In fact RPF seem to be avoiding doing that.

What do people use mmal for?

axl · Post by **axl** » Thu Sep 26, 2019 1:22 pm

http://www.jvcref.com/files/PI/documentation/html/

MMAL (Multi-Media Abstraction Layer) is a framework which is used to provide a host-side, simple and relatively low-level interface to multimedia components running on VideoCore.

The short answer is that mmal is responsible for hardware video decoding/encoding. Keyword here is "hardware".

axl · Post by **axl** » Thu Sep 26, 2019 1:27 pm

A little correction. I was wrong about the name of the useflag. it's not raspberry-pi, it's actually mmal. Funny, I thought they changed it, but it seems I was remembering wrong or they changed it back.

Code: Select all

[ebuild   R    ] media-video/ffmpeg-4.2:0/56.58.58::gentoo  USE="X alsa amrenc bluray bzip2 cdio chromium codec2 cpudetection encode fdk flite fontconfig fribidi gcrypt gme gmp gnutls gpl gsm iconv iec61883 jpeg2k kvazaar libaom libass libcaca libdrm libilbc librtmp libsoxr libv4l lv2 lzma modplug mp3 network openal opengl openh264 openssl opus oss postproc pulseaudio rubberband samba sdl snappy speex srt ssh svg theora threads truetype twolame v4l vorbis wavpack webp x264 xcb xvid zeromq zimg zlib (-altivec) (-amr) (-appkit) (-bs2b) -chromaprint (-cuda) -dav1d -debug -doc -frei0r -hardcoded-tables (-ieee1394) -jack (-ladspa) -libaribb24 -libressl -libxml2 (-mipsdspr1) (-mipsdspr2) (-mipsfpu) (-mmal) (-opencl) -pic -static-libs -test -vaapi (-vdpau) -vidstab (-vpx) -x265 (-zvbi)" CPU_FLAGS_ARM="(neon) (thumb) (thumb2) (v6) (v8) (vfp) (vfpv3)" FFTOOLS="aviocat cws2fws ffescape ffeval ffhash fourcc2pixfmt graph2dot ismindex pktdumper qt-faststart sidxindex trasher" VIDEO_CARDS="(-nvidia)" 0 KiB

Anyway, as long that flag is disabled, you cannot hope to enjoy good video performance.

Post by **NeddySeagoon** » Thu Sep 26, 2019 7:22 pm

axl,

Enable the mmal flag and tell us what happens. We all want to know and someone has to be first.
This time, it could be you. :)

axl · Post by **axl** » Thu Sep 26, 2019 7:28 pm

?!

let me rephrase that. it is blocked, not disabled. mmal is blocked on aarch64 because it expects you to have raspberrypi-userland and that doesn't work as well. Well, some parts of that work, but mmal is not one them.

it's not just about enabling a flag.

Post by **NeddySeagoon** » Thu Sep 26, 2019 8:14 pm

axl,

I've just looked at the ffmpeg ebuild. You are correct
I recall someone posting here about mmal on 64 bit working.

axl · Post by **axl** » Thu Sep 26, 2019 8:29 pm

That would be awesum. Maybe I missed something. Reading through the issues on github for raspberrypi-userland, i doubt that completely though.

It's like a known issue, for years even (since the first 64bit pi dropped) that mmal is going to be a pain in the behind. Nobody is working on it. No efforts whatsoever have been done for this. It's just dead in the water. In fact if you read through the issues on raspberry pi github you can get a distinct idea that it's not ever supposed to work in 64bits. It's a given that it does, but not because it's supposed to.

Gavinmc42 mentioned the new interface for the pi camera, which in fact is the old interface for video. v4l. I don't know exactly if /dev/video0 is able to spit out h264 coded video. I doubt it can. which raspivid did. as far as I know, /dev/video0 provides uncompressed video. Which if it was a pc, would be usable. But on a PI... be it pi4... it's still not enough to process that data. /dev/video0 at those resolutions without mmal is just pointless. That was the whole thing of the rpi camera. it was realtime encoded in h264. and in case none of you know how, it was through mmal.

mmal is central to _hardware_ video encoding/decoding. if you are going the 64 bits way, you can say goodbye to hardware encoding/decoding. and without that... raspberry pi is just a very slow 64bits atom computer. it's whole hook was the hardware video encoding/decoding ability. I'm not saying you can't use it for other things, but mmal is brilliant. Shame it doesn't work in 64 bits.

Anyway, people trying 64 bits should know: kodi, omxplayer, pi camera and other alternatives of the sort are severely handicapped.

axl · Post by **axl** » Thu Sep 26, 2019 8:40 pm

When I say hardware video encoding/decoding I am talking about alternatives to cpu. We all know ffmpeg (which is the central piece of any video software) can process things through the cpu. even multi core. Amazing how many advances have been made.

But there's also alternatives to NOT use the cpu. Like vdpau for nvidia rendering. Or nvenc for nvidia encoding. I don't know amd, coz I don't have any, but I do know that Intel has it's own thing. Was it vaapi? Not sure, I don't have any intel either, I just picked it up while messing with mesa I think.

A while back, like 2 decades ago, I received a bundled creative cdrom and a creative dxr2 card. That card (you can still see the dxr2 flag in ffmeg) was able to video decode mpeg streams. You would put your vga out from video card into dxr2 input, and dxr2 output to vga output to monitor and basically it was an overlay. So old. I still have that card somewhere. Should be put in a museum. It was BAAAAD

at least with the software of the time.

But on the other hand did you see how videos look on the pi? The colors, the not tearing, the framerate... everything is crystal clear. I can connect my nvidia 960 through hdmi to the same tv where I have one of my pi's and there's just no comparison. I use mostly mplayer & smplayer and I tried xv, gl_nosw, sdl, vdpau... you can't make a pc with linux and nvidia show a video as beautiful as raspberry does it. You just can't. That's hardware video decoding at work. the mmal component does that.

Post by **NeddySeagoon** » Thu Sep 26, 2019 8:49 pm

axl,

axl wrote:A while back, like 2 decades ago, I received a bundled creative cdrom and a creative dxr2 card. That card (you can still see the dxr2 flag in ffmeg) was able to video decode mpeg streams. You would put your vga out from video card into dxr2 input, and dxr2 output to vga output to monitor and basically it was an overlay. So old. I still have that card somewhere. Should be put in a museum. It was BAAAAD :) at least with the software of the time.

I used to use one of those to play DVDs on a k6-2 system in the the late 90s.
That k6-2 system is long gone but I still have that dxr2 hardware overlay card and that early IDE DVD drive.

axl · Post by **axl** » Thu Sep 26, 2019 8:53 pm

NeddySeagoon wrote:axl,

axl wrote:A while back, like 2 decades ago, I received a bundled creative cdrom and a creative dxr2 card. That card (you can still see the dxr2 flag in ffmeg) was able to video decode mpeg streams. You would put your vga out from video card into dxr2 input, and dxr2 output to vga output to monitor and basically it was an overlay. So old. I still have that card somewhere. Should be put in a museum. It was BAAAAD at least with the software of the time.
I used to use one of those to play DVDs on a k6-2 system in the the late 90s.
That k6-2 system is long gone but I still have that dxr2 hardware overlay card and that early IDE DVD drive.

It was terrible. But then again it was during the windows 95/98 time. How good could have it been ?:))

I remember getting 2 bundled games with it. Star wars 4 I think and some cat platform game, which at the time I really enjoyed.

Good times

axl · Post by **axl** » Thu Sep 26, 2019 9:02 pm

and I'm sorry to myself for derailing my own thread, but talking about old hardware accelerated platforms, I had the original creative video blaster.

During that time, windows 3.1 was a novelty. Most things worked in dos.

And this thing would be able to take video in analogue signal and put it on screen. You couldn't record videos as far as I remember, but you could make pics. 320x240 pics (TGA format). I think. It was something ridiculously small.

But it was so cool. I remember we used to connect either the vcr to watch a movie on the computer, or the camera to monkey around. I might have been 12 at the time.

The computer holding it was a 486se. 8 megs of ram. 500 megs of hdd. had cdrom, had sound blaster. had tape. it was a beast at the time.

I can't believe how far we've come in my lifetime.

Post by **NeddySeagoon** » Thu Sep 26, 2019 9:09 pm

axl,

I was just on linux :)
It uses the em8300 driver, which still appears to be in the kernel.

axl · Post by **axl** » Thu Sep 26, 2019 9:14 pm

NeddySeagoon wrote:axl,

I was just on linux
It uses the em8300 driver, which still appears to be in the kernel.

I am pretty sure I can't run the original video blaster as it was an AT card. Not sure it's the right name. the one before pci. or pci-express.

As far as the dxr2, I think that's also at. Having flashbacks with agp ports. UHM... I doubt I'll ever try them in linux. I have the cards, but not the system to put them in.

Meanwhile... I'm pissed mmal is not working in 64 bits. going back to armv7a. GRRRRR

axl · Post by **axl** » Thu Sep 26, 2019 9:18 pm

I wonder... if you have a 64 bits kernel, and a 32 bits userland, could you use mmal? or if you boot 64bits, and chroot into a 32bits distro. could you mmal then? (if you mounted dev proc sys). if the answer is true...

maybe aarch64 could benefit from a multilib setup. one where you compile most things 64 bits, but not ffmpeg, it's dependencies, and anything that follows ffmpeg.

Should be fairly easy to test. just add 64 bits kernel to raspbian. it would boot and work, but is kodi still working? Maybe I'll try that tomorrow.

axl · Post by **axl** » Thu Sep 26, 2019 9:30 pm

Maybe I should try armv7a. Haven't done that in a while. On the pi4 should be a breeze.

Uhm, I'm especially curious if chromium (youtube) uses mmal to render. I think it does. Uses ffmpeg. There's no reason it shouldn't. Did you see how gpu heavy chrome/chromium is? they even have a chrome://gpu tab.

It's been at least one year since I wanted see what would that look on armv7a. I seen it in aarch64 and I'm not impressed.

Gavinmc42 · Post by **Gavinmc42** » Fri Sep 27, 2019 1:00 am

axl , you want mmal and VC4 hardware for h.264 encoding/decoding?
Then I defiantly use the h.264 encode, that's used for motion detection on my Pikrellcam cameras.
Guess that will limit Pikrellcam to VC4 Pi's, for the moment.

Years ago the Pi guys told me h.265 would be faster decoding on the Arm cores than the VC4 GPU.
I think that is what they have gone for in the PI4.
Arm decoding of h.264 and 265.
I was under the impression ffmpeg had that already.

RPF guys say Browsers are a bit laggy for >1920x1080 videos, so I went looking for a standalone YT viewer.
Freetube YouTube viewer works, that might be easier to dig into than Chromium/Firefox.
I think you are right about them using ffmpeg, but they do other stuff as well.

With the new mesa releases we get the OpenGLES 3.1 driver which has compute shaders.
OpenCL is supposed to work too.
Someone with some clues might be able to accelerate some stuff using those on the VC6.

Wish we had the manuals.

axl · Post by **axl** » Fri Sep 27, 2019 1:31 am

You know, this is not exactly clear. They did say the new rpi4 supports x265 as it is 4k. and as far as i know x265 was designed especially to push x264 into 4k. but people dont always use x265 for 4k streams.

What is not clear to me atm is if the ability to decode x265 is also backported to older pi's through the magic of mmal.

I guess the answer is in ffmpeg. which is central to any video rendering app. regardless of what app it is. even chrome.

Or things I could test if I could be bothered. Other than PI 1, I still have one of each. But they're in their place. It's not me that can't be bothered... I can't bother the girls. I call them by girls names. I'm weird. But I have a 4 and a 3b which are not "in their place". I'll experiment on those.

Honestly, this discussion helped. put things into perspective. it's hard to run tests without knowing what tests you want to run. which is not to say more input is not welcomed.

Gavinmc42 · Post by **Gavinmc42** » Fri Sep 27, 2019 12:41 pm

Looks like we don't have OpenCL either.

While the hardware might be capable of lots of stuff without the software it is not usable.
Tried to emerge CLinfo but it is masked.
Have not got to the masking part of the Gentoo manual yet.

OpenMAX is omx. = omxplayer, Is that fully open now?

omx101 for dummies?
https://kwasi-ich.de/blog/2017/11/26/omx/

https://www.yodeck.com/news/the-raspber ... te-review/

A few posts about OpenMax on Pi4, gooblygook to me.
The more I learn the less I know.
Starting to think I can never learn everything about Pi's.
Oh for the old days of <100 page manuals and one double sided laminated plastic sheet for coding.

Gavinmc42 · Post by **Gavinmc42** » Fri Sep 27, 2019 1:00 pm

I followed a white rabbit and found this.
https://www.raspberrypi.org/forums/view ... 8&t=246837

I am so glad I don't do complex video stuff or have large res screens.
Getting a straight answer is hard.

Maybe the Kodi guys have a clue?
Wish there were some manuals, so I could RTFMs.

Gavinmc42 · Post by **Gavinmc42** » Fri Sep 27, 2019 1:07 pm

Looks for the reasons mmal on Sakaki's wiki

As a (limited, 64-bit) build of media-libs/raspberrypi-userland is included, you can use vcgencmd etc. Note however that MMAL and OpenMAX IL features are not currently supported (this is an issue for all 64-bit OS on the RPi3/4 at present, please see here, here and here).

https://github.com/sakaki-/gentoo-on-rp ... -only-skip

I think I need to read Sakaki's wiki completely, lots of it are making sense now.

Sakaki · Post by **Sakaki** » Sat Sep 28, 2019 5:56 pm

As far as I know, the current 64-bit bcm2711_defconfig kernel supports MMAL, when called from a 32-bit userland (see e.g. here), but I don't believe the required userland libraries exist in a 64-bit form, yet.

The (optional) camera module, and (built-in) hardware video codecs can however be accessed via the V4L2{,M2M} framework from a 64-bit userland (this needs a patched ffmpeg to work, such as the version shipped on the current gentoo-on-rpi-64bit image). See e.g. my notes here.

When capturing from the camera, you can specify the use of e.g. H264:

Code: Select all

demouser@pi64 ~ $ ffmpeg -f video4linux2 -input_format h264 \
  -video_size 1280x720 -framerate 30 -i /dev/video0 \
  -vcodec copy -an -f matroska test.mkv

Play back the recorded stream using a hardware codec:

Code: Select all

demouser@pi64 ~ $ ffplay -vcodec h264_v4l2m2m -i test.mkv

kodi will support use of the v4l2 m2m endpoints from release 19.0, I think.