Create webcam LiveStreaming

Topics: Question
Mar 16, 2016 at 6:50 AM
Edited Mar 16, 2016 at 6:51 AM
Hi, I'm new to here.

I'm create a test program to stream my webcam frame to anthoer PC.
and I'm testing your MMA library, which look awesome!

I download the source from subversion (https://net7mma.svn.codeplex.com/svn) to get the latest code

and try this
        public void RunTest() {
            ILogger logger = LoggingSystem.GetLogger("Test");
            DsDevice[] devs = DsDevice.GetDevicesOfCat(FilterCategory.VideoInputDevice);
            DsCapture capture = new DsCapture();
            capture.SetCamera(
                moniker: devs[1].Mon,   //capture 2nd camera
                width: 640,
                height: 480,
                fps: 30);
            capture.StartCapture();
            using(RtspServer server = new RtspServer()) {
                RFC6184Media h264media = new RFC6184Media(640, 480, "test");
                server.TryAddMedia(h264media);
                server.Start();
                while(Console.KeyAvailable == false) {
                    Bitmap bmp = capture.SnapshotSourceImage();
                    h264media.Packetize(bmp);
                }
            }
            capture.Dispose();
        }
where DsCapture is a directshow capture lib to grab webcam image.

after run, I got exception on the line
server.Start();
which says
System.InvalidOperationException ---- ssrc and senderSsrc must be unique
Am I doing something wrong?

thanks.
Coordinator
Mar 16, 2016 at 12:52 PM
Definitely will take a look at this today or this week, there are a few other caveats in the encoder as there is no entropy encoding or quantization right now so your ouput won't really be compressed.

If your already using direct show you may want to check out this article especially if you have a GPU

http://www.codeproject.com/Articles/421869/H-CUDA-Encoder-DirectShow-Filter-in-Csharp
Marked as answer by juliusfriedman on 3/16/2016 at 4:52 AM
Coordinator
Mar 16, 2016 at 4:52 PM
I have made some updates in 111948 which should fix this as well as few other things.

Let me know if you still need help!
Marked as answer by juliusfriedman on 3/16/2016 at 8:52 AM
Mar 17, 2016 at 6:50 AM
Edited Mar 17, 2016 at 9:16 AM
Hi

Thank you for help.

I've update my code to 111950.
It's can run RtspServer successfully.

But I can't let VLC connect to the stream.

Here is the test code
        private void RtspTest() {
            DsDevice[] devs = DsDevice.GetDevicesOfCat(FilterCategory.VideoInputDevice);
            DsCapture capture = new DsCapture();
            capture.SetCamera(
                moniker: devs[1].Mon,   //capture 2nd camera
                width: 640,
                height: 480,
                fps: 30);
            capture.StartCapture();
            ManualResetEvent evt = new ManualResetEvent(false);
            Thread t = new Thread(new ThreadStart(()=> {
                using(RtspServer server = new RtspServer(IPAddress.Any, 554)) {
                    RFC6184Media media = new RFC6184Media(640, 480, "test");
                    //JPEGMedia media = new JPEGMedia("test", new Uri("", UriKind.Relative));
                    //RFC6416Media media = new RFC6416Media(640, 480, "test");
                    //RFC2435Media media = new RFC2435Media("test");
                    server.TryAddMedia(media);
                    server.Start();
                    while(true) {
                        if(evt.WaitOne(0)) { break; }
                        logger.AddLog("Before Grab Frame");
                        Bitmap bmp = capture.SnapshotSourceImage();
                        Bitmap bmp32 = new Bitmap(bmp.Width, bmp.Height, PixelFormat.Format32bppArgb);
                        using(Graphics g = Graphics.FromImage(bmp32)) {
                            g.DrawImage(bmp, 0, 0);
                        }
                        logger.AddLog("Grabbed frame");
                        media.Packetize(bmp32);
                    }
                }
            }));
            t.Start();
            Console.ReadLine();
            evt.Set();
            capture.Dispose();
        }
I use rtsp://localhost/live/test in VLC, the debug log is
core debug: processing request item: rtsp://localhost/live/test, node: 播放清單, skip: 0
core debug: resyncing on rtsp://localhost/live/test
core debug: rtsp://localhost/live/test is at 3
core debug: starting playback of the new playlist item
core debug: resyncing on rtsp://localhost/live/test
core debug: rtsp://localhost/live/test is at 3
core debug: creating new input thread
core debug: Creating an input for 'rtsp://localhost/live/test'
core debug: requesting art for rtsp://localhost/live/test
core debug: looking for meta fetcher module matching "any": 1 candidates
core debug: using timeshift granularity of 50 MiB, in path 'C:\Users\01021301\AppData\Local\Temp'
core debug: `rtsp://localhost/live/test' gives access `rtsp' demux `' path `localhost/live/test'
core debug: specified demux `any'
core debug: creating demux: access='rtsp' demux='any' location='localhost/live/test' file='live\test'
core debug: looking for access_demux module matching "rtsp": 12 candidates
live555 debug: version 2014.07.25
lua debug: Trying Lua scripts in C:\Users\01021301\AppData\Roaming\vlc\lua\meta\fetcher
lua debug: Trying Lua scripts in D:\Tools\vlc-2.2.1\lua\meta\fetcher
lua debug: Trying Lua playlist script D:\Tools\vlc-2.2.1\lua\meta\fetcher\tvrage.luac
qt4 debug: IM: Setting an input
lua debug: skipping script (unmatched scope) D:\Tools\vlc-2.2.1\lua\meta\fetcher\tvrage.luac
core debug: no meta fetcher modules matched
core debug: searching art for rtsp://localhost/live/test
core debug: looking for art finder module matching "any": 2 candidates
lua debug: Trying Lua scripts in C:\Users\01021301\AppData\Roaming\vlc\lua\meta\art
lua debug: Trying Lua scripts in D:\Tools\vlc-2.2.1\lua\meta\art
lua debug: Trying Lua playlist script D:\Tools\vlc-2.2.1\lua\meta\art\00_musicbrainz.luac
lua debug: skipping script (unmatched scope) D:\Tools\vlc-2.2.1\lua\meta\art\00_musicbrainz.luac
lua debug: Trying Lua playlist script D:\Tools\vlc-2.2.1\lua\meta\art\01_googleimage.luac
lua debug: skipping script (unmatched scope) D:\Tools\vlc-2.2.1\lua\meta\art\01_googleimage.luac
lua debug: Trying Lua playlist script D:\Tools\vlc-2.2.1\lua\meta\art\02_frenchtv.luac
lua debug: skipping script (unmatched scope) D:\Tools\vlc-2.2.1\lua\meta\art\02_frenchtv.luac
lua debug: Trying Lua playlist script D:\Tools\vlc-2.2.1\lua\meta\art\03_lastfm.luac
lua debug: skipping script (unmatched scope) D:\Tools\vlc-2.2.1\lua\meta\art\03_lastfm.luac
core debug: no art finder modules matched
core debug: looking for meta fetcher module matching "any": 1 candidates
lua debug: Trying Lua scripts in C:\Users\01021301\AppData\Roaming\vlc\lua\meta\fetcher
lua debug: Trying Lua scripts in D:\Tools\vlc-2.2.1\lua\meta\fetcher
lua debug: Trying Lua playlist script D:\Tools\vlc-2.2.1\lua\meta\fetcher\tvrage.luac
core debug: using meta fetcher module "lua"
core debug: removing module "lua"
core debug: searching art for rtsp://localhost/live/test
core debug: looking for art finder module matching "any": 2 candidates
lua debug: Trying Lua scripts in C:\Users\01021301\AppData\Roaming\vlc\lua\meta\art
lua debug: Trying Lua scripts in D:\Tools\vlc-2.2.1\lua\meta\art
lua debug: Trying Lua playlist script D:\Tools\vlc-2.2.1\lua\meta\art\00_musicbrainz.luac
lua debug: Trying Lua playlist script D:\Tools\vlc-2.2.1\lua\meta\art\01_googleimage.luac
lua debug: Trying Lua playlist script D:\Tools\vlc-2.2.1\lua\meta\art\02_frenchtv.luac
lua debug: Trying Lua playlist script D:\Tools\vlc-2.2.1\lua\meta\art\03_lastfm.luac
core debug: no art finder modules matched
core debug: art not found for rtsp://localhost/live/test
live555 debug: RTP subsession 'video/H264'
core debug: selecting program id=0
live555 debug: setup start: 0.000000 stop:0.000000
live555 debug: play start: 0.000000 stop:0.000000
core debug: using access_demux module "live555"
core debug: looking for decoder module matching "any": 43 candidates
avcodec debug: CPU flags: 0x010053db
avcodec debug: trying to use direct rendering
avcodec debug: allowing 4 thread(s) for decoding
avcodec debug: avcodec codec (H264 - MPEG-4 AVC (part 10)) started
avcodec debug: using frame thread mode with 4 threads
core debug: using decoder module "avcodec"
core debug: looking for packetizer module matching "any": 23 candidates
packetizer_h264 debug: found NAL_SPS (sps_id=0)
packetizer_h264 debug: found NAL_PPS (pps_id=0 sps_id=0)
core debug: using packetizer module "packetizer_h264"
core debug: looking for meta reader module matching "any": 2 candidates
lua debug: Trying Lua scripts in C:\Users\01021301\AppData\Roaming\vlc\lua\meta\reader
lua debug: Trying Lua scripts in D:\Tools\vlc-2.2.1\lua\meta\reader
lua debug: Trying Lua playlist script D:\Tools\vlc-2.2.1\lua\meta\reader\filename.luac
core debug: no meta reader modules matched
core debug: `rtsp://localhost/live/test' successfully opened
live555 warning: no data received in 10s. Switching to TCP
core debug: removing module "avcodec"
avcodec debug: ffmpeg codec (H264 - MPEG-4 AVC (part 10)) stopped
core debug: killing decoder fourcc `h264', 0 PES in FIFO
core debug: removing module "packetizer_h264"
core debug: Program doesn't contain anymore ES
live555 debug: RTP subsession 'video/H264'
core debug: looking for decoder module matching "any": 43 candidates
avcodec debug: CPU flags: 0x010053db
avcodec debug: trying to use direct rendering
avcodec debug: allowing 4 thread(s) for decoding
avcodec debug: avcodec codec (H264 - MPEG-4 AVC (part 10)) started
avcodec debug: using frame thread mode with 4 threads
core debug: using decoder module "avcodec"
core debug: looking for packetizer module matching "any": 23 candidates
packetizer_h264 debug: found NAL_SPS (sps_id=0)
packetizer_h264 debug: found NAL_PPS (pps_id=0 sps_id=0)
core debug: using packetizer module "packetizer_h264"
live555 debug: setup start: 0.000000 stop:0.000000
live555 debug: play start: 0.000000 stop:0.000000
live555 error: no data received in 10s, aborting
core debug: EOF reached
core debug: finished input
core debug: removing module "avcodec"
avcodec debug: ffmpeg codec (H264 - MPEG-4 AVC (part 10)) stopped
core debug: killing decoder fourcc `h264', 0 PES in FIFO
core debug: removing module "packetizer_h264"
core debug: removing module "live555"
core debug: Program doesn't contain anymore ES
core debug: dead input
core debug: changing item without a request (current 3/4)
core debug: nothing to play
qt4 debug: IM: Deleting the input
And I found the process time of
media.Packetize(bmp32); 
is about 5 second ....

//////
By the way, how can I change the taget framework to .Net Framework 4.0?
My program has to execute at Server 2003...which can't install .Net 4.5 ... T_T
Coordinator
Mar 17, 2016 at 3:55 PM
Edited Mar 17, 2016 at 3:56 PM
Thanks for your continued interest in the project.

Lets consider some improvements to your example for a moment.
                       //Let's try to conserve memory and cpu by not re-creating this memory every single iteration. 
                       Bitmap bmp32;
                       //Lets also preserve the Graphics state.
                       Graphics g;

while(true) {
                        if(evt.WaitOne(0)) { break; }

                        logger.AddLog("Before Grab Frame");

                        //Grab the snapshot and dispose when finished.
                        using(Bitmap bmp = capture.SnapshotSourceImage()){

                        //Allocate the memory and graphics one time
                        if(bmp32 == null) {
                               bmp32 = new Bitmap(bmp.Width, bmp.Height, PixelFormat.Format32bppArgb);
                               g = Graphics.FromImage(bmp32);
                        }
                        
                        //Draw it every time
                        g.DrawImage(bmp, 0, 0);

                        //Log
                        logger.AddLog("Grabbed frame");

                        //Packetize
                        media.Packetize(bmp32);                        

                    }

//Dispose when finished.
if(bmp32 != null){ 
bmp32.Dispose();
bmp32 = null;
}

if(g != null){
g.Dispose();
g = null;
}
That should reduce the time and memory the example takes quite a bit but it's possible the garbage collector may collect the Bitmap or the Graphics objects and you will want to take care to handle that also.

There is a similar example already without Directshow which can easily be adapted here:
System.Threading.Thread taker = new System.Threading.Thread(new System.Threading.ParameterizedThreadStart((o) =>
                {
                    using (var bmpScreenshot = new System.Drawing.Bitmap(Screen.PrimaryScreen.Bounds.Width,
                               Screen.PrimaryScreen.Bounds.Height,
                               System.Drawing.Imaging.PixelFormat.Format32bppArgb))
                    {

                        // Create a graphics object from the bitmap.
                        using (var gfxScreenshot = System.Drawing.Graphics.FromImage(bmpScreenshot))
                        {

                            //Could also use mirror.State.
                            while (server.IsRunning)
                            {
                                try
                                {
                                    // Take the screenshot from the upper left corner to the right bottom corner.
                                    gfxScreenshot.CopyFromScreen(Screen.PrimaryScreen.Bounds.X,
                                                                Screen.PrimaryScreen.Bounds.Y,
                                                                0,
                                                                0,
                                                                Screen.PrimaryScreen.Bounds.Size,
                                                                System.Drawing.CopyPixelOperation.SourceCopy);

                                    //Convert to JPEG and put in packets
                                    mirror.Packetize(bmpScreenshot);

                                    //REST
                                    System.Threading.Thread.Sleep(50);
                                }
                                catch (Exception ex)
                                {
                                    server.Logger.LogException(ex);
                                }
                            }

                            int exit = 1;

                        }
                    }
                }));
As for what VLC says, it may be because there are several streams being created with the same virtual name, "test" and your are not getting the information from the stream you think you are.

This is allowed for now just because it is not checked in the code but I should definitely change this convention in the future, so thanks for bringing that up.

One you have that sorted out you should be in better shape and you should be at least be able to consume the JPEG stream, the H.264 stream may have some other issues with pixel format which have to be worked out as well as SPS and PPS and certain other things.

Re - streaming / live streaming is a totally different animal and if the media is already in Rtp or Rtsp then you can just use the RtpSource or RtspSource which handles this without actually needing to decode the data to send it out.

Finally, as for .Net 4.0 There are like 5 things I can think of which prevent this for now:

1) StreamWriter does not have a easy way to leave the underlying stream open
2) BinaryReader does not have a easy way to leave the underlying stream open
3) MapToIpV4 for the IPV6 address family is not implemented
4) IsIPv4MappedToIPv6 is not implemented
5) InfiniteTimeSpan does not exist and there are several different conventions utilized for this concept throughout the framework depending on where exactly you are.

Then there are some of the attributes which I recently added which can easily avoided with some DEFINES.

It's not very hard, in fact I would consider this if you created an Issue and it received enough votes.... however what's to stop someone from asking for .Net 3.5 support or .Net 2.0 support or .Net 1.0 support?

What I will say is that if for some reason the .Net Micro Framework does move back into the spotlight for whatever reason, depending on if it gets generic support I will subsequently provide the same level of features for whatever the base line implementation would be, e.g. if it doesn't get generics then I guess I would have to support a completely 1.0 compliant subset which would force this to be handled a certain way, if it does get generics then depending on how the API relates to the same classes which are present I will address the attribute usage and compatibility issues in the API.

If you want to handle this yourself you may just want to consider using and older version and patching it to the latest code where possible which is much less intensive task, for example get the latest code which targets your framework and then compare the classes in the assemblies you use, adjust the code to either live with the existing API in the framework or add wrapper code to safely prevent the disposal of shared resources etc and provide those classes instead of the framework classes.

If you don't mind me asking what is the nature of your project and what do you hope to accomplish? With the world moving more towards HTML5 I can only imagine that your using it in some type of bridging system... I would like to know more about it so I can understand what your doing and how you use the code.

And as a final reminder, Since you also have Directshow you can probably use the overloads of Packetize which accept the Nal units directly from an H.264 encoder if you just strip off the start codes for the RFC6184Frame, you don't always have to get a snap shot especially if your camera can go into a pipeline where it's encoded to h.264 automatically.

Thank you again for your interest in the project and if you have any more questions let me know and sorry for being nosy, I am just pretty interested in this stuff!
Marked as answer by juliusfriedman on 3/17/2016 at 7:56 AM
Mar 17, 2016 at 5:20 PM
Edited Mar 17, 2016 at 5:20 PM
Hi,

Thank you for reply.
And sorry for my poor English, It's not my native language T_T
//////

In fact, I'm creating a camera streaming system in a LAN network (client's count may up to 100)
The system need capture number of webcam at same time, and steaming to client
The client can switch the channel (webcam) realtime.

I'm not familiar with Encoder/Decoder...so I just find the library that I can reuse in my project

I'll try the cached Bitmap object tomorrow.

Thanks again!
Coordinator
Mar 17, 2016 at 9:55 PM
Edited Mar 17, 2016 at 9:56 PM
This is just the scenario that this library helps with, depending on the amount of memory you have on the platform 100 clients or even 1000 clients should not be unreasonable, 10,000 clients can even be achieved and definitely more with some optimizations. (Specifically in how events are dispatched to client sessions from a source)

Encoding / Decoding is useful to save bandwidth. Sometimes just the differences in the individual frames are sent rather then sending the whole image over and over again. It also can further employ coding techniques which can reduce the amount of data required to represent the same data in different parts of the image, sort of like how zip can make files smaller.

Let me me know if you have any other questions and thanks for your interest and support!

If I can help more I will!

+-+
Marked as answer by juliusfriedman on 3/17/2016 at 1:55 PM
Mar 18, 2016 at 3:15 AM
Hi,

I found something wrong in RFC6184Media
        public override void Packetize(System.Drawing.Image image)
        {
            try
            {
                //Make the width and height correct
                using (var thumb = image.Width != Width || image.Height != Height ? image.GetThumbnailImage(Width, Height, null, IntPtr.Zero) : image)
when the size of input image same as the size of media,
the input image will get Dispose after the packetize job done...
so I can't reuse the bmp32 object to encode

/////////////

I have to build my project on .Net 4.0
And I don't have much time to tweak MMA project to .Net 4.0
I will try to found other solution.

I still interest in this lib, and thanks for your help!