Saving H264 from IP Camera to disk

Topics: Question
Dec 7, 2014 at 7:47 PM
Hello,
First of all let me thank you for the great job you have done with this project. Is really outstanding!
I've bought some Hikvision cameras for my new (still under construction) house (can't wait to move :) ). I decided to use a low power consumption computer in order to get and store the videos from my cameras and I would like to backup them also to Azure (real-time because I have 500 Mbit optical fiber internet). I was thinking to different approaches. First of all I tried with Hikvision client which seems to be very buggy and not reliable. Then I struggled one day to configure a NFS on windows and also to make my camera to use it to store files to it (the camera has a bug and I had to manually edit in hexa some content to make it work properly - I even bricked it and I had to debrick). Now it works but there are few problems. First of all it looks like the stored videos are losing some frames sometimes and therefore is not very reliable. However the live play seems to work smoothly.

In conclusion, I thought to capture RTSP video and store it to disk and also to Azure. My cameras are streaming in H264 (I guess). So my question is how can I simply store the video which I receive from my camera to disk and then to be able to play the video (of course). A bit of help would be much appreciated.
Dec 7, 2014 at 8:15 PM
Nevermind :).
Found the answer in thread http://net7mma.codeplex.com/discussions/569279

Here is my code:
    private void button1_Click(object sender, EventArgs e)
    {
        RtspClient rtspClient = new RtspClient("rtsp://192.168.1.211:554", RtspClient.ClientProtocolType.Tcp);
        rtspClient.Credential = new System.Net.NetworkCredential("admin", "12345");
        rtspClient.OnPlay += rtspClient_OnPlay;
        rtspClient.StartPlaying();
        rtspClient.Client.RtpFrameChanged += Client_RtpFrameChanged;
   }
void Client_RtpFrameChanged(object sender, Media.Rtp.RtpFrame frame)
    {

        if (!frame.Complete) return;


        var context = ((Media.Rtp.RtpClient)sender).GetContextByPayloadType(frame.PayloadTypeByte);
        Media.Rtsp.Server.MediaTypes.RFC6184Media.RFC6184Frame hframe = new Media.Rtsp.Server.MediaTypes.RFC6184Media.RFC6184Frame(frame);

        if (context == null || context.MediaDescription.MediaType != Media.Sdp.MediaType.video) return;
        if (bFirst == true)
        {
            Media.Sdp.SessionDescriptionLine fmtp = context.MediaDescription.FmtpLine;

            byte[] sps = null, pps = null;

            foreach (string p in fmtp.Parts)
            {
                string trim = p.Trim();
                if (trim.StartsWith("sprop-parameter-sets=", StringComparison.InvariantCultureIgnoreCase))
                {
                    string[] data = trim.Replace("sprop-parameter-sets=", string.Empty).Split(',');
                    sps = System.Convert.FromBase64String(data[0]);
                    pps = System.Convert.FromBase64String(data[1]);
                    break;
                }
            }

            bool hasSps, hasPps, sei, slice, idr;
            hframe.Depacketize(out hasSps, out hasPps, out sei, out slice, out idr);

            byte[] result = hframe.Buffer.ToArray();

            using (var stream = new System.IO.MemoryStream(result.Length))
            {
                if (!hasSps && sps != null)
                {
                    stream.Write(new byte[] { 0x00, 0x00, 0x00, 0x01 }, 0, 4);

                    stream.Write(sps, 0, sps.Length);
                }

                if (!hasPps && pps != null)
                {
                    stream.Write(new byte[] { 0x00, 0x00, 0x00, 0x01 }, 0, 4);

                    stream.Write(pps, 0, pps.Length);
                }

                hframe.Buffer.CopyTo(stream);

                stream.Position = 0;

                // Write All Bytes stream
                decode_stream(stream);
                bFirst = false;
            }
        }
        hframe.Depacketize();
        decode_stream(hframe.Buffer);

    }      

    public void decode_stream(Stream fin)
    {
        using (var fs = new FileStream(string.Format("c:\\!Temp\\CameraTests\\test2.h264"), FileMode.Append))
            fin.CopyTo(fs);

    }
Coordinator
Dec 8, 2014 at 12:10 AM
Edited Dec 8, 2014 at 12:10 AM
Thanks and yes, That's correct.

I am going to add few properties to make that easier to do and make packetization and such more useful (e.g. keeping a list of nal types and maybe their offsets)

Playback from .h264 should eventually be supported I just have to get around to making a NAL Reader which would also be used on the containers when h.264 was the codec in most cases.

I was just gonna wait until I implemented the Demuxer but I will see if I can get anything out sooner.

If you need anything else let me know!
Marked as answer by juliusfriedman on 12/7/2014 at 5:10 PM
Dec 8, 2014 at 6:25 AM
Well, VLC player knows how to play the files so I'm happy with the current solution. I just have to figure out some compression settings from my camera because currently I have ~ 1.5 MBytes/s so 5-6 cameras will be 7.5-9 MBytes/s which would be a bit too much for me.

Also I did some profiling because with one camera the application is using ~5-8% of my CPU (and I have an i7 4771 for my dev box and the server I bought is only an i5 4690S).
With 5-6 cameras I will get to ~50+% usage and I think it might be a bit too much because I plan to perform other tasks as well on that server and besides all the encoding is mainly done on camera and the current application should only get the bytes, identify frames and write them to disk.
Anyway based on my profiling the LINQ ToArray() method is using most of the CPU. I think there can be a different approach like to have a bigger/memory area buffer (like 1 MByte) and then just keep info about indexes within the memory area. I will change the code a bit and I will let you know the results (though it might take few days because I'm very busy this week and I don't know if I'll find any time except next weekend).
Once again thank you so much for this project!
Coordinator
Dec 8, 2014 at 8:53 AM
I would hope the load wouldn't be that high. I have tested with 1000 simulated clients and the load was never about 75%

Do you have a few cameras I can test against to check the load?

I agree the ToArray method is heavy for some reason when it could be a bit faster..
public static TSource [] ToArray<TSource> (this IEnumerable<TSource> source)
        {
            Check.Source (source);

            TSource[] array;
            var collection = source as ICollection<TSource>;
            if (collection != null) { //Seems like I could implement ICollection and possibly improve the performance a bit.
                if (collection.Count == 0)
                    return EmptyOf<TSource>.Instance;
                
                array = new TSource [collection.Count];
                collection.CopyTo (array, 0); 
                return array;
            }

            int pos = 0, arrLength = array.Length; //Could cache length here
            array = EmptyOf<TSource>.Instance;
            foreach (var element in source) {
                if (pos == arrLength) {
                    if (pos == 0)
                        array = new TSource [(arrLength=4)];
                    else
                        Array.Resize (ref array, (arrLength = pos * 2));
                }

                array[pos++] = element;
            }

            if (pos != arrLength)
                Array.Resize (ref array, pos);
            
            return array;
        }
In a way your saying that I should ditch IEnumerable or come up with something like VirtualEnumerable which can copy very quickly and determine count without enumeration.

I had a lot planned in this regard, especially with streams as you can see from these articles:

http://www.codeproject.com/Articles/575064/Complete-Managed-Media-Aggregat

http://www.codeproject.com/Articles/578116/Complete-Managed-Media-Aggregation-Part-III-Quantu

In the end I felt it was like GStreamer which didn't make me happy but now I may end up taking a few things from there.

The VirtualEnumerable is definitely something to look at as it would allow the code to be kept the same and still reduce the copy time / projection time.

The LinkedStream concept is also something to look at as it will allow streams to be 'joined' and modified in memory / then saved to disk without copying any stream and without re-sizing any streams if not allowed.

Let me know what you find and I may just wait until you take your approach so I can compare the two and then see what else I can come up with if needed.

Thank you for your feedback and definitely let me know if I can be of any further help!
Marked as answer by juliusfriedman on 12/8/2014 at 1:53 AM
Dec 8, 2014 at 10:26 AM
Hello,
I have to admit that I'm not very active in the open source community (shame on me) mostly because I don't have enough time. However I think you have done a fantastic job and in a return I will give you a VM and access to my cameras if you want to play around and perform some tests. I have Visual Studio Ultimate which has the profiler so it would be easy to check :D.
I only have one Hikvision camera at the moment DS-2CD2632F-I (Vari-Focal 3MP) (plus one with 4mm zoom should arrive any minute today), but after I "measure" the zoom which I need for every camera I will buy the version with fixed zoom because for this Vari-Focal camera you can only control the zoom manually from the camera itself and is not likely for me to climb to them from time to time to change the zoom amount.
By the end of the week I will order and install the other 4 cameras so I will have 5 cameras 3 MP each to test with.

As for the ToArray issue, don't bother for now because I will test myself as well and I will let you know the results. Eventually I will give you the changes I made and if you will find them suitable you can then push to official branch.
Coordinator
Dec 8, 2014 at 6:16 PM
It's only a priority to me if it's a priority to you! I was just worried the usage was so high.

Another option would be open up a camera to the outside world and password protect it and I could try remotely accessing it, I have worked with those cameras before I think (including the HD Versions) and I didn't seem to have any issues either with CPU or Memory.

Anyway I also think that there is definitely room for some improvements, I am just wondering if they can be generalized and more useful as Common classes rather than optimizing the MemorySegment and if they will gain much in the end.

Try the latest code when profiling again to make sure it wasn't due to a bug in the code :)

Thank you again for the kind words and let me know if you need anything else!

-Julius
Dec 8, 2014 at 8:15 PM
Well is not a priority at all. First of all I will move to my new house only in ~3 months. Until then I don't mind if my CPU is used 100% :). Secondly I think you should focus on other things which are more important for community like new features. For this particular issue I will contribute and I will give a solution after I perform many tests and profiling.

About cameras I can open them on internet with password protection but I'm from Romania and you are from USA. Although I have 100 MBits optical fiber internet, most likely you won't be able to get more than ~10 Mbits from me because you are too far. That's why I had the idea with VM on my machine which would be connected to cameras on a gigabit router.

Also as a small note my cameras are not Full HD (1920x1080) but greater :). They have 3 MP (some 2048 x 1536 resolution I think) and with full settings they stream 12280 kbits which is nearly 1.5 MBytes/s.
Coordinator
Dec 8, 2014 at 8:22 PM
Edited Dec 8, 2014 at 9:51 PM
100% Usage is very interesting but I think I have used those cameras before without any issues and I would just like to see if I can replicate this on my end.

110445 fixes a few bugs but nothing which should have caused excess usage, only fixed a little memory consumption and decreased CPU if anything.

It seems that one thing I remember is that the cameras had a MTU and MSS setting which you may want to check is not larger then 1456 [mtu] and 2048[mss] unless you increase the buffer size of the RtspClient.

Even if I am remote I should be able to see at least 10% usage from a single one of the cameras and I should be able to play around with debugging to see where the load comes from.

If I need to from there then I can use the VM but I will save that for last.

I asked someone here to try and replicate this also because they have about 800 cameras and they seem to indicate that everything is smooth.
https://net7mma.codeplex.com/discussions/569279

I am glad to hear you are moving into your new house, I know that process can be long and tiring so if I don't hear from you I will assume your just busy :)

I just updated the code again because I will wait for access to camera before I try to simulate something.

I also enabled optimization in all the code which helps a bit!

let me know if your still having issues or when I can test the camera!

Thanks!
Coordinator
Dec 10, 2014 at 4:02 AM
Edited Dec 10, 2014 at 4:03 AM
Just wanted to ask that when you profile to please use 110493, I have made some changes which may or may not improve your CPU usage.

BTW, what other features would you like to see?

Let me know what you find and thanks again for bringing this up!
Dec 14, 2014 at 4:29 PM
Hi again,
I got the latest version and the CPU usage problem is gone. Good job!
I have just ordered another 3 cameras so I will have a total of 5 cameras. Do you still need access to them? I haven't mounted them yet, but I hope I will mount them next weekend.
Currently I'm working on a functionality to store on disk and on Google drive real time. In that way I will always have a backup of my videos even if somebody would eventually break my house and steal my server. Now I have to think over a mechanism/algorithm to catch-up the cloud backup in case my internet goes down or it performs very slow for any reasons...

As for new features I think you could go slowly to a surveillance tool which can perform acquisitions from cameras, stream them together on a website and any other features :). I think doing that you can also sell it and make profit out of such tool :).
Marked as answer by juliusfriedman on 12/14/2014 at 9:44 AM
Coordinator
Dec 14, 2014 at 4:44 PM
Good to hear.

Believe it or not I can probably even optimize further.

I don't need access so long as your happy with the performance.

I like the idea, I would have to work on the decoding first for that.

My focus is around free software because I want to compete with paid software.

In the end storage is gonna require resources, I would rather focus on the infrastructure level.

Allowing for iptv with channel guides and dvr access is something I think could also be useful.

Anyway im glad your not having any more issues.

I will have facilities to support playback of h.264 as well as other formats soon, until then you can also use the rtpdump format for which I can read and write currently in the library.

Thanks again!

Julius
Coordinator
Dec 20, 2014 at 6:49 PM
Edited Dec 22, 2014 at 3:27 AM
FYI 110653 further reduces CPU and Memory usage.

Hope all else is well!
Marked as answer by juliusfriedman on 12/20/2014 at 11:49 AM