Wednesday, April 23, 2008

Webcam control with WPF or how to create high framerate player with DirectShow by using InteropBitmap in WPF application

Did you ever see, that MediaElement “eats” about 30% of CPU while playing movie in WPF? Did you thought, that you  can display live camera capture in WPF with 60 fps full screen (I have really high resolution 1920x1200) and 2% of  CPU? You did not? Let’s see how it can be done. Today we’ll create simple WebCam player control that can show you live video capturing with high frame rate. In order to do it, we’ll use DirectShow, WPF and make them work together.

  image

You, probably do not believe me. Let’s start. In order to build this application, we need to make DirectDraw working in C# managed code. We can use DirectShow.NET, but this time we’ll do it manually. Why? because I love to do things manually. So let’s understand what we need? Actually, not very much: one Sample Grabber (ISampleGrabber) and one Device input filter(IBaseFilter). Both we should connvert with Graph Builder (IGraphBuilder) and point to some grabber implementation (ISampleGrabberCB). Also, we do not want DirectShow to render video for use, thus we’ll send it’s default Video Window (IVideoWindow) to null with no AutoShow and then run the controller (IMediaControl). Do you tired enough to lost me? Let’s see the code. One Filter graph with one Device Filter and one Sample grabber.

graph = Activator.CreateInstance(Type.GetTypeFromCLSID(FilterGraph)) as IGraphBuilder;
sourceObject = FilterInfo.CreateFilter(deviceMoniker);

grabber = Activator.CreateInstance(Type.GetTypeFromCLSID(SampleGrabber)) as ISampleGrabber;
grabberObject = grabber as IBaseFilter;

graph.AddFilter(sourceObject, "source");
graph.AddFilter(grabberObject, "grabber");

Set media type for our grabber

using (AMMediaType mediaType = new AMMediaType())
                {
                    mediaType.MajorType = MediaTypes.Video;
                    mediaType.SubType = MediaSubTypes.RGB32;
                    grabber.SetMediaType(mediaType);

And then connect device filter to out pin and grabber to in pin. Then get capabilities of video received (thiss stuff come from your web camera manufacturer)

if (graph.Connect(sourceObject.GetPin(PinDirection.Output, 0), grabberObject.GetPin(PinDirection.Input, 0)) >= 0)
                    {
                        if (grabber.GetConnectedMediaType(mediaType) == 0)
                        {
                            VideoInfoHeader header = (VideoInfoHeader)Marshal.PtrToStructure(mediaType.FormatPtr, typeof(VideoInfoHeader));
                            capGrabber.Width = header.BmiHeader.Width;
                            capGrabber.Height = header.BmiHeader.Height;
                        }
                    }

Out pin to grabber without buffering and callback to grabber object (this one will get all images from our source).

graph.Render(grabberObject.GetPin(PinDirection.Output, 0));
grabber.SetBufferSamples(false);
grabber.SetOneShot(false);
grabber.SetCallback(capGrabber, 1);

Dump output window

IVideoWindow wnd = (IVideoWindow)graph;
wnd.put_AutoShow(false);
wnd = null;

And run the controller

control = (IMediaControl)graph;
control.Run();

We done. Now our video is captured and can be accessed from BufferCB method of ISampleGrabberCB. Next step is to do WPF related stuff

First of all, we’ll use InteropBitmap. This one will provide us with real performance bust. So, one our DirectShow graph is ready and we know result image capabilities, we can create memory section and map it in order to provide ISampleGrabberCB with place to put images. This will be always the same pointer, so all we have to do is to .Invalidate interop image.

if (capGrabber.Width != default(int) && capGrabber.Height != default(int))
                {

                    uint pcount = (uint)(capGrabber.Width * capGrabber.Height * PixelFormats.Bgr32.BitsPerPixel / 8);
                    section = CreateFileMapping(new IntPtr(-1), IntPtr.Zero, 0x04, 0, pcount, null);
                    map = MapViewOfFile(section, 0xF001F, 0, 0, pcount);
                    BitmapSource = System.Windows.Interop.Imaging.CreateBitmapSourceFromMemorySection(section, capGrabber.Width, capGrabber.Height, PixelFormats.Bgr32,
                        capGrabber.Width * PixelFormats.Bgr32.BitsPerPixel / 8, 0) as InteropBitmap;
                    capGrabber.Map = map;
                    if (OnNewBitmapReady != null)
                        OnNewBitmapReady(this, null);
                }

Now in capGrabber (ISampleGrabberCB) we’ll copy buffer, comes from our webcam to the mapped location for WPF usage

public int BufferCB(double sampleTime, IntPtr buffer, int bufferLen)
        {
            if (Map != IntPtr.Zero)
            {
                CopyMemory(Map, buffer, bufferLen);
                OnNewFrameArrived();
            }
            return 0;
        }

All we have to do is to call InteropBitmap.Invalidate() each frame to reread the image bytes from the mapped section.

if (BitmapSource != null)
                    {
                        BitmapSource.Invalidate();

How do display all this stuff? Simple – subclass from Image and set it’s Source property with the interop bitmap.

public class CapPlayer : Image,IDisposable
    {

void _device_OnNewBitmapReady(object sender, EventArgs e)
        {
            this.Source = _device.BitmapSource;
        }

Now, the usage from XAML is really simple

<l:CapPlayer x:Name="player"/>

We done :) As always, download full source code for this article

…and be good people and don’t tell anymore, that WPF performance in terms of imaging is sucks :)

P.S. small ‘r’ if you have more, then one WebCam connected. Inside CapDevice class there is member, public static FilterInfo[] DeviceMonikes, that provides you with all DirectShow devices installed. So, the only thing you should do in order to change the device is to set deviceMoniker = DeviceMonikes[0].MonikerString; with the moniker of your device. This sample works with first one.

7 comments:

Martin Tall said...

Great work! Thanks for posting!

Thomas said...

When I opened your example and ran it everything worked beautifully - however, the image displayed by my web cam is upside down. What should I change to correct this? And no, my cam isn't upside down :)

Thanks!

ogry said...
This comment has been removed by the author.
ogry said...

Waw! 60fps@1920x1200 is really impresive!

What WebCam are you using!


Saludos, OSCAR.

P.S. Great coding example!

Dipo said...

This is a great post, it really helped in a proof of concept for a project I'm working on.

The problem I'm facing now is capturing input from the video in port of a capture device (TV Tuner). I'm new to DirectShow and I've been studying the source code you provided but couldn't figure out it could be done. Any help will be greatly appreciated.

c# sockets1 said...

Hi,

This is a really good example but like |Thomas, the image is upside down - how could I change the code so its the correct way up?

Also how could I change the code so that I can record the video to a file?

Thanks - any pointers would be appreciated,

Mark

Martin Skov Nielsen said...

I had the same upside down problem - the solution was really simple actually:
<cap:CapPlayer x:Name="player" RenderTransformOrigin="0.5,0.5">
<cap:CapPlayer.RenderTransform>
<TransformGroup>
<ScaleTransform/>
<SkewTransform/>
<RotateTransform Angle="180"/>
<TranslateTransform/>
</TransformGroup>
</cap:CapPlayer.RenderTransform>
</cap:CapPlayer>