VL.MediaFoundation

Elias · April 9, 2020, 11:23am

Microsoft Windows has a lot of multimedia capabilities built-in which can be accessed through the Media Foundation SDK (About Media Foundation - Win32 apps | Microsoft Learn).

After scanning through a bunch of C++ and SharpDX (which provides the bindings for .NET) examples I’ve finally managed to put all pieces together for a nice and easy to use VideoIn node (as well as its reactive counter-part in case one wants to process data in the video thread).
It outputs a B8G8R8A8 image which can easily be made available to other nodes from Skia or OpenCV.
Hardware acceleration can be turned on if needed. I didn’t see much of a difference though with a 720p stream, maybe it’s different with higher resolutions. Would be eager to get some feedback on that topic.

Once VL.Xenko/VL.Stride gets available we can also think about making the video frames directly available as textures (by-passing the CPU completely) because the used DirectX11 device for hardware acceleration can be fed from outside.

The repository and install instructions can be found here: GitHub - vvvv/VL.Video.MediaFoundation: ARCHIVED! Microsoft Media Foundation for VL

bjoern · April 9, 2020, 12:13pm

Worked out of the box using a Kinect V2

Use Hardware Acceleration doesn’t seem to make any difference.

When using the reactive version it takes a while (~2,5 s) before the image is shown in the tooltip.

Maybe the tooltip could show the current FPS?

sebl · April 9, 2020, 1:02pm

would also make a nice videoplayer, no?

github.com

sharpdx/SharpDX-Samples/blob/master/Desktop/MediaFoundation/MediaEngineAppVideo/Program.cs

// Copyright (c) 2010-2013 SharpDX - Alexandre Mutel
// 
// Permission is hereby granted, free of charge, to any person obtaining a copy
// of this software and associated documentation files (the "Software"), to deal
// in the Software without restriction, including without limitation the rights
// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
// copies of the Software, and to permit persons to whom the Software is
// furnished to do so, subject to the following conditions:
// 
// The above copyright notice and this permission notice shall be included in
// all copies or substantial portions of the Software.
// 
// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
// THE SOFTWARE.

This file has been truncated. show original

Elias · April 9, 2020, 3:57pm

@bjoern
The ~2.5s you see when using the reactive version is because only when you hover over the pin the camera capture gets activated. The node is in that sense lazy. Other camera device nodes are patched in the same way. Reason being mainly about lifetime management.
I did however also notice the tooltip on the observable being a little sluggish and when looking in the task manager the CPU usage was much higher than going the manual route through DrawImage/Renderer - not sure what’s the reason for that, maybe something is off in the tooltip itself…

@sebl
I’ll definitely also have a look at normal video playback. Apart from having to install GStreamer it also never was very stable. So maybe Media Foundation is easier to handle in that regard. We’ll see.

Hadasi · April 10, 2020, 10:21am

Something that I tried to test a while back, but I couldn’t get to work with VL at the time was this library, essentially a cross-platform rewrite of media foundation based on FFMPG, but you were exploring Gstreamer so it seem unnecessary to discuss it.
GitHub - unosquare/ffmediaelement: FFME: The Advanced WPF MediaElement (based on FFmpeg).

Hadasi · April 10, 2020, 10:25am

On closer inspection

Target Framework must be 4.6.1 or above, or .Net Core 3.0 or above.

so I’m not sure if that excludes Gamma at the moment

Elias · April 10, 2020, 9:02pm

Just uploaded version 0.0.2 which adds VideoPlayer and VideoPlayer (Reactive) nodes.

sebescudie · April 11, 2020, 4:34pm

hei there, tried to export an app using that nuget. the patch works fine (nothing fancy, just a VideoIn drawn on a rectangle) but soon as I try to run the exported app, it throws this exception :

System.IO.FileNotFoundException: Could not load file or assembly 'SharpDX.MediaFoundation, Version=4.2.0.0, Culture=neutral, PublicKeyToken=b4dcf0f35e5521f1' or one of its dependencies. The system cannot find the file specified.

(full stack here)

I manually added SharpDX.MediaFoundation and SharpDX dlls in the lib folder next to the .exe (it would complain for both), the app starts but it shows a blank rectangle instead of my webcam feed (works fine in the editor).

any hints?

here’s a screenshot of the patch i’m exporting … nothing fancy :)

cheerz!

Elias · April 11, 2020, 4:44pm

Ah yes indeed, I think I can fix SharpDX not being referenced. Regarding the other error, you sure you stopped the patch? Access to camera is exclusive what I know. At least here it would show a white quad as long as the patch was still running.

sebescudie · April 11, 2020, 5:04pm

hey elias, thanks for the fix and yep, the patch was stopped. the small LED next to my webcam was off when I launched the .exe, and never turned on.

Elias · April 11, 2020, 5:08pm

Do you have multiple devices? Maybe it’s related to the dynamic enum I wonder. I just uploaded a new package (0.0.3) which should fix the reference issue. Maybe some other file was missing? Hard to say why you don’t get anything, it worked for me having one camera.

sebescudie · April 11, 2020, 5:20pm

I can confirm exported applications works fine with 0.0.3 ! Also for some reason my webcam is now showing up there.

Thanks again for the prompt fix !

lecloneur · May 31, 2021, 2:38am

Works fine here too, in editor and after being exported. Only with Preview Gamma version, adding reference get stuck and never finish but nuget packet is fine.

joreg · October 14, 2021, 9:22am

As of 2021.4 the VL.Video.MediaFoundation nuget is now shipping with vvvv. And there are even more options for video playback.