How to create a file from H.264 RFC6184Stream and play it by ffplay

Topics: Question
May 30, 2016 at 7:42 AM
Hello,
I try to stuck in grabbing frame from LIVE RTSP stream. Media format is 96 and i can get each frame in RtpFrameChanged event, i am using RFC6184Stream and Depacketize it and i save each frame in separate file. but ffplay can not replay the file.

[h264 @ 00000206c377dc40] concealing 3587 DC, 3587 AC, 3587 MV errors in P frame
[h264 @ 00000206c2c15480] concealing 3915 DC, 3915 AC, 3915 MV errors in P frame
[h264 @ 00000206c2c1a7c0] concealing 3844 DC, 3844 AC, 3844 MV errors in P frame
Invalid UE golomb code= 0 aq= 0KB vq= 79KB sq= 0B f=0/0
[h264 @ 00000206c3485820] mb_width/height overflow
[h264 @ 00000206c3485820] Truncating likely oversized SPS
Invalid UE golomb code
[h264 @ 00000206c3485820] mb_width/height overflow
Invalid UE golomb code
[h264 @ 00000206c3485820] mb_width/height overflow
[h264 @ 00000206c3485820] concealing 4658 DC, 4658 AC, 4658 MV errors in I frame
[h264 @ 00000206c2c1a260] concealing 3561 DC, 3561 AC, 3561 MV errors in P frame
nan M-V:    nan fd=   0 aq=    0KB vq=    0KB sq=    0B f=0/0
I found fmtp.HasFormatSpecificParameters is always return false.
This is my test code:

static void Main(string[] args)
    {
        var client = new RtspClient("rtsp://192.168.199.92:554/Streaming/Channels/1?transportmode=unicast&profile=Profile_1", RtspClient.ClientProtocolType.Tcp);
        client.AuthenticationScheme = System.Net.AuthenticationSchemes.Basic;
        client.AutomaticallyReconnect = true;
        client.Credential = new System.Net.NetworkCredential("admin", "12345");
        client.OnConnect += Client_OnConnect;
        client.Connect();

    }
private static void Client_OnConnect(RtspClient sender, object args)
    {
        if (!sender.IsConnected) return;
        if (!sender.IsPlaying)
        {
            sender.OnPlay += Sender_OnPlay;
            sender.StartPlaying();
        }
    }
private static void Sender_OnPlay(RtspClient sender, object args)
    {
        sender.Client.RtpFrameChanged += Client_RtpFrameChanged;

    }
private static void Client_RtpFrameChanged(object sender, Media.Rtp.RtpFrame frame = null, Media.Rtp.RtpClient.TransportContext tc = null, bool final = false)
    {
        string outputFileName = "e:\\Test.h264";
        using (var fs = new System.IO.FileStream(outputFileName, System.IO.FileMode.Append))
        {
            if (false == final || Media.Common.IDisposedExtensions.IsNullOrDisposed(frame) || false == frame.IsComplete) return;

            var context = tc ?? ((Media.Rtp.RtpClient) sender).GetContextByPayloadType(frame.PayloadType);

            if (context == null || context.MediaDescription.MediaType != Media.Sdp.MediaType.video) return;

            using (Media.Rtsp.Server.MediaTypes.RFC6184Media.RFC6184Frame profileFrame = new Media.Rtsp.Server.MediaTypes.RFC6184Media.RFC6184Frame(frame))
            {
                profileFrame.Depacketize();
                if (false == profileFrame.HasDepacketized) return;

                //If there is not a sps or pps in band and this is the first frame given to a decoder then it needs to contain a SPS and PPS
                //This is typically retrieved from the SessionDescription or CodecPrivateData but only the very first time.
                if (false == m_InitializedStream &&/* && false == profileFrame.ContainedUnitTypes.Any(nalType => nalType == Media.Codecs.Video.H264.NalUnitType.SequenceParameterSet || nalType == Media.Codecs.Video.H264.NalUnitType.PictureParameterSet)*/ false == profileFrame.ContainsSequenceParameterSet || false == profileFrame.ContainsPictureParameterSet)
                {
                    Media.Sdp.Lines.FormatTypeLine fmtp = new Media.Sdp.Lines.FormatTypeLine(context.MediaDescription.FmtpLine);

                    //Media.Sdp.Lines.FormatTypeLine fmtp = new Media.Sdp.Lines.FormatTypeLine(Media.Sdp.SessionDescriptionLine.Parse("a=fmtp:97 packetization-mode=1;profile-level-id=42C01E;sprop-parameter-sets=Z0LAHtkDxWhAAAADAEAAAAwDxYuS,aMuMsg=="));
                    //if (false == fmtp.HasFormatSpecificParameters) throw new System.Exception("HasFormatSpecificParameters is false");
                    byte[] sps = null, pps = null;

                    //If there was a fmtp line then iterate the parts contained.
                    foreach (string p in fmtp.Parts)
                    {
                        //Determine where in the string the desired token in.
                        string token = Media.Common.Extensions.String.StringExtensions.Substring(p, "sprop-parameter-sets=");

                        //If present extract it.
                        if (false == string.IsNullOrWhiteSpace(token))
                        {
                            //Get the strings which corresponds to the data without the datum split by ','
                            string[] data = token.Split(',');

                            //If there is any data then assign it

                            if (data.Length > 0) sps = System.Convert.FromBase64String(data[0]);

                            if (data.Length > 1) pps = System.Convert.FromBase64String(data[1]);

                            //Done
                            break;
                        }
                    }

                    //Prepend the SPS if it was found
                    if (sps != null)
                    {
                        //Emulation prevention, present for SPS or PPS
                        fs.WriteByte(0);

                        //Write the start code
                        fs.Write(Media.Codecs.Video.H264.NalUnitType.StartCodePrefix, 0, 3);

                        //Write the SPS
                        fs.Write(sps, 0, sps.Length);
                    }
                    else throw new System.Exception("SequenceParameterSet not found");

                    //Prepend the PPS if it was found.
                    if (pps != null)
                    {
                        //Emulation prevention, present for SPS or PPS
                        fs.WriteByte(0);

                        //Write the start code
                        fs.Write(Media.Codecs.Video.H264.NalUnitType.StartCodePrefix, 0, 3);

                        //Write the PPS
                        fs.Write(pps, 0, pps.Length);
                    }
                    else throw new System.Exception("PicutureParameterSet not found");

                    m_InitializedStream = true;
                }

                profileFrame.Buffer.CopyToStream(fs);


            }
        }
    }

wireshar dump:
Frame 209: 1514 bytes on wire (12112 bits), 1514 bytes captured (12112 bits) on interface 0
Interface id: 0 (\Device\NPF_{A4AA6F35-E538-4BEC-B542-5EBE8360D87F})
Encapsulation type: Ethernet (1)
Arrival Time: May 30, 2016 15:33:20.143997000 �й���׼ʱ��
[Time shift for this packet: 0.000000000 seconds]
Epoch Time: 1464593600.143997000 seconds
[Time delta from previous captured frame: 0.030575000 seconds]
[Time delta from previous displayed frame: 0.081562000 seconds]
[Time since reference or first frame: 16.871620000 seconds]
Frame Number: 209
Frame Length: 1514 bytes (12112 bits)
Capture Length: 1514 bytes (12112 bits)
[Frame is marked: False]
[Frame is ignored: False]
[Protocols in frame: eth:ethertype:ip:tcp:rtsp:rtp:rtp]
[Coloring Rule Name: TCP]
[Coloring Rule String: tcp]
Ethernet II, Src: HuaweiTe_86:71:44 (7c:a2:3e:86:71:44), Dst: AsustekC_0b:d4:60 (30:5a:3a:0b:d4:60)
Internet Protocol Version 4, Src: 192.168.199.92, Dst: 192.168.3.213
Transmission Control Protocol, Src Port: 554 (554), Dst Port: 51891 (51891), Seq: 1796, Ack: 962, Len: 1460
RTSP Interleaved Frame, Channel: 0x00, 32 bytes
Real-Time Transport Protocol
10.. .... = Version: RFC 1889 Version (2)
..0. .... = Padding: False
...0 .... = Extension: False
.... 0000 = Contributing source identifiers count: 0
0... .... = Marker: False
Payload type: DynamicRTP-Type-96 (96)
Sequence number: 32509
Timestamp: 359506518
Synchronization Source identifier: 0x3625f00d (908455949)
Payload: 674d40208b9500a00f34200000e100002bf21080
RTSP Interleaved Frame, Channel: 0x00, 16 bytes
Magic: 0x24
Channel: 0x00
Length: 16
Real-Time Transport Protocol
10.. .... = Version: RFC 1889 Version (2)
..0. .... = Padding: False
...0 .... = Extension: False
.... 0000 = Contributing source identifiers count: 0
0... .... = Marker: False
Payload type: DynamicRTP-Type-96 (96)
Sequence number: 32510
Timestamp: 359506518
Synchronization Source identifier: 0x3625f00d (908455949)
Payload: 68fe3880
Coordinator
May 31, 2016 at 9:12 PM
Thanks for pointing out the issues with 'HasFormatSpecificParameters' and the FormatTypeLine.

I will enhance SessionAttributeLine to contain a few properties : AttributeName, AttributeValue, HasAttributeValue, etc.

I will then utilize those properties from within the FormatTypeLine to ensure that HasFormatSpecificParameters looks at the correct part.

Expect an update later this week.

Thanks for your help!
Marked as answer by juliusfriedman on 5/31/2016 at 2:13 PM
Jun 2, 2016 at 6:20 AM
Hello,
Thank you for your reply.
I downloaded 112091 and built.
I used profileFrame.Buffer.CopyTo(fs) to making the file to play.
this is my code:

try
                {
                    if (profileFrame.IsKeyFrame())
                    {
                        Console.WriteLine("key");
                    }
                    else
                    {
                        Console.WriteLine("not key");
                    }
                    profileFrame.Buffer.CopyTo(fs);
                }
                catch (Exception e)
                {

                    Console.WriteLine(e.Message);
                }
I found that any time the keyframe come, profileFrame.Buffer.CopyTo throw an Exception:"System.ArgumentOutOfRangeException".so ffplay can not play the file.

[h264 @ 0000019951f62260] Format h264 detected only with low score of 1, misdetection possible!
[h264 @ 0000019951f633a0] non-existing PPS 0 referenced
Last message repeated 1 times
[h264 @ 0000019951f633a0] decode_slice_header error
[h264 @ 0000019951f633a0] no frame!
[h264 @ 0000019951f633a0] non-existing PPS 0 referenced
Last message repeated 1 times
Coordinator
Jun 2, 2016 at 10:52 AM
Could you post a stack trace of the exception? Or better a test which is able to replicate it?

SegmentStream has quite a few tests to ensure it reads and writes the same as a stream so I am having a hard time attempting to imagine where an exception would occur.

Let me know and thanks
Marked as answer by juliusfriedman on 6/2/2016 at 3:52 AM
Jun 3, 2016 at 5:58 AM
StackTrace:

在 System.IO.FileStream.Write(Byte[] array, Int32 offset, Int32 count)
在 Media.Common.SegmentStream.CopyTo(Stream s) 位置 F:\Projects\DotNet\Media-112091\Common\Classes\SegmentStream.cs:行号 295
在 RtspClientTester.Program.Client_RtpFrameChanged(Object sender, RtpFrame frame, TransportContext tc, Boolean final) 位置 G:\My Files\Documents\Visual Studio 2015\Projects\RtspClientTester\RtspClientTester\Program.cs:行号 136

I tested this code in a variety of brand camera.
Lower release like 111987 not this bug.
Coordinator
Jun 3, 2016 at 1:46 PM
Interesting.

I can't replicate that particular issue immediately but I'll try again today.

It is possible the frame is being disposed of before you write it out, can you see what the state of frame looks looks like when the exception is thrownot?

Thanks again for your testing!
Marked as answer by juliusfriedman on 6/3/2016 at 6:46 AM
Jun 6, 2016 at 2:34 AM
Hi,

The frame is not disposed before I write it out.
I tested by this code:
try
                {
                    profileFrame.Buffer.Position = 0;
                    Console.WriteLine(profileFrame.IsDisposed);
                    profileFrame.Buffer.CopyTo(fs);
                    profileFrame.Buffer.Dispose();
                }
                catch (Exception e)
                {
                    Console.WriteLine(profileFrame.IsDisposed);
                    Console.WriteLine(e.Message);
                }
profileFrame.IsDisposed is false when the exception is throwout.
Coordinator
Jun 6, 2016 at 1:41 PM
Can you provide a test case which replicates the exception?

The SegmentStream CopyTo method is tested within the UnitTests using totally random data...

I cannot replicate this issue...

Does the length of the buffer change in between where you check for it to be disposed and then right after?

That is the only way I can see an out of range exception could be thrown...

There is also the possibility of hitting a dead spot, e.g. a space with 0 bytes.. I will make an update for that soon and you can try the test again.
Marked as answer by juliusfriedman on 6/6/2016 at 6:41 AM
Jun 8, 2016 at 5:31 AM
Hello,
I downloaded the latest release and tested.
The SegmentStream CopyTo method work fine,but the ffplay still can not play the file.
[h264 @ 00000174e9f12280] Format h264 detected only with low score of 1, misdetection possible!
[h264 @ 00000174e9f133c0] non-existing PPS 0 referenced
Last message repeated 1 times
[h264 @ 00000174e9f133c0] decode_slice_header error
[h264 @ 00000174e9f133c0] no frame!
[h264 @ 00000174e9f133c0] non-existing PPS 0 referenced


The same code with the release 111987 work very well,ffplay can play the file any time.
Then I tested the code with 111988 and newer release,It is not work.
Coordinator
Jun 8, 2016 at 5:34 AM
The problem must be in the RFC6184Frame class...

Why don't you use the latest code but replace the RFC6184Frame class from the version which worked e.g 87

Let me know if that works well for you and if so I will take a closer look.

Thank you again for testing.

The only thing I imagine it could have something to do with is how I calculate the length of the packet in newer versions.

Let me know what you find and thanks again for your help!
Marked as answer by juliusfriedman on 6/7/2016 at 10:34 PM
Coordinator
Jun 8, 2016 at 5:48 AM
Edited Jun 8, 2016 at 5:49 AM
Okay actually I see it... You have Automatically Reconnect == true...

I am not sure why, if you disconnect and reconnect and then add duplicate event handler to play...

Can you provide a copy of the h.264 file and the Wireshark dump from when you created it with the library?

Using those two things I can create a diff of the results and see exactly what is going on...

I suspect some type of disposal is occurring on the packets before you process them although I am not sure where based on the limited example you provided.

Let me know if you can provide a way for me to be able to replicate this locally besides the capture it would also help, e.g. when I create h.264 files from the BigBuckBunny test video it works well for me.

You may want to use CsCodec to see if you can figure out exactly where the problem is as it would let you not only see how the decoder is handling it but how the frames are being kept in memory while this occurs.

Obviously ffmpeg will be faster but it should also work under cscodec for testing purposes.

I gotta get to bed but definitely keep me updated!

Thanks again for your help and testing!
Marked as answer by juliusfriedman on 6/7/2016 at 10:49 PM
Jun 8, 2016 at 8:27 AM
I try to use the latest code but replace the RFC6184Frame class from the version which worked e.g 87 ,but the file still not play.

I think the problem may be in RtpFrame.cs or RtspClient.cs
Coordinator
Jun 8, 2016 at 11:48 AM
If you can show me a bug I can replicate I will fix it.

If you can show me the Wireshark capture of the h264 stream and the same h264 file which you create in the application I can do a difference and see where the difference is and potentially why.

I have tested just recently but I will test more and I will double check with cscodec to make sure I can playback video and report the finding back.
Marked as answer by juliusfriedman on 6/8/2016 at 4:48 AM
Coordinator
Jun 8, 2016 at 9:20 PM
I have no issues playing with cscodec / quicktime / ffmpeg/ vlc or (s)mplayer.

Are you 110% sure the error is not on your end in your application somewhere?
Marked as answer by juliusfriedman on 6/8/2016 at 2:20 PM
Jul 7, 2016 at 6:53 AM
Hello.
Sorry for late.
I am taking a long holidays.
I found that ,the new version 112176 works very fine, but the memory is higher than older version.

thanks for your work.
Coordinator
Jul 8, 2016 at 1:27 AM
Np, thanks for the feedback.

I have already isolated the increased usage to the RtspMessage class.

I will have the fix up tommorow hopefully.

Thanks for testing!
Marked as answer by juliusfriedman on 7/7/2016 at 6:27 PM