Video playback delay (H264 rtp stream ) which is growing longer when a lot movement happens

Topics: Question
Nov 12, 2016 at 9:45 PM
Hello Julius,

Thanks again for your great library! hope you are well :-)

The Decoding is good, the Quality is okay. The Speed is most time good.
But I realize for example when I take something with much Details like a paper with a list on it, and fast move it in front of the camera, I get a delay. The Playback has still good Quality, but it Plays slower. If I do that for ten seconds and finish, it takes 10 seconds more if the Playback reaches that Point.

I tried to fasten Playback by Rendering with a persentation time, but did not help and crashed.

Do you have any idea or Suggestion? Should I drop packets, can I check a timestamp on a final packet and drop it, if the difference is too big?

Thanks a lot and have a nice Weekend!

Firlefanz
Nov 14, 2016 at 8:47 AM
Hi Julius,

I decreased the rtpclients buffer a lot. This one:

Media.Common.ISocketReferenceExtensions.SetReceiveBufferSize(context, BufferSize);

Now the delay is gone. But when I move a lot, I get artefacts again (only as Long as I Keep moving a lot).
Any idea if I can get rid of those, too without having a huge buffer?

I also tried to drop frames, but I am not sure how, dropping in my OnSourceFramechanged Listener does not seem to help.

Thanks :-)
Coordinator
Nov 21, 2016 at 1:40 AM
You really must pay attention to the GPU buffer; if you want the best performance you should be aware of how much ahead or behind of the playback you are such that seeking is possible; etc.

Each device will have different requirements and thus the CODEC / GPU implementation must be able to handle data loss and the like.

Hope that helps.
Marked as answer by juliusfriedman on 11/20/2016 at 6:40 PM
Nov 22, 2016 at 1:24 PM
Edited Nov 22, 2016 at 1:38 PM
Hi Julius,

If I turn off Audio in the SDP when initializing the RTP it gets much faster.
With Audio and also having lots of Animation in the Picture it gets much slower...

I have currently two contexts when initialising my rtp Client, one for Audio one for Video.
                foreach (RtpClient.TransportContext context in _rtpClient.GetTransportContexts())
                {

if ANDROID

            Media.Common.ISocketReferenceExtensions.SetReceiveBufferSize(context, _bufferSize);

endif

                     Media.Common.ISocketReferenceExtensions.SetReceiveTimeout(context, 4);
                     Media.Common.ISocketReferenceExtensions.SetSendTimeout(context, 2);
                }
I am receiving Video, sending and receiving Audio. On both iOS and Android devices it has a growing delay when animations Keep Happening. It is no Problem when I turn off Audio in the session description...
Nov 22, 2016 at 2:36 PM
Edited Nov 23, 2016 at 12:08 PM
The complete sdp with both Video and Audio Looks like this:

{v=0
o=- 1457065787 1457065787 IN IP4 192.168.0.24
s=Asterisk
c=IN IP4 192.168.0.24
t=0 0
m=audio 12984 RTP/AVP 0 8
a=rtpmap:0 PCMU/8000
a=rtpmap:8 PCMA/8000
a=ptime:20
a=maxptime:150
a=sendrecv
m=video 19370 RTP/AVP 34 99
a=rtpmap:34 H263/90000
a=rtpmap:99 H264/90000
a=sendrecv
}

If I remove audio totally before initialising the rtp this helps (no delay at all):
sdpTemp = sdpTemp.Remove(sdpTemp.IndexOf("m=audio"), sdpTemp.IndexOf("m=video") - sdpTemp.IndexOf("m=audio"));

If I set audio only to send or receive only, both do not help:
sdpTemp = sdpTemp.Replace("a=sendrecv\r\nm=video", "a=sendonly\r\nm=video");
sdpTemp = sdpTemp.Replace("a=sendrecv\r\nm=video", "a=recvonly\r\nm=video");

I removed the Audio time Patterns both at once, does also not help:
sdpTemp = sdpTemp.Replace("a=ptime:20","");
sdpTemp = sdpTemp.Replace("a=maxptime:150","");

I am using 112065 still. I initialise my rtpclient like this:
                Media.Sdp.SessionDescription sessionDescription = new Media.Sdp.SessionDescription(sdpTemp);
                sessionDescription.SessionName = sdp.Name;
                sessionDescription.SessionId = sdp.Owner.SessionID.ToString();

                try
                {
                    _rtpClient = Media.Rtp.RtpClient.FromSessionDescription(sessionDescription, rtpPort: 0, rtcpEnabled: false);
                }
                catch (Exception ex)
                {
                    ClientLogger.Instance.Log(string.Format("Error occured setting RTP SDP: " + ex.ToString()), ClientLogger.LogLevel.Error);
                }

                foreach (RtpClient.TransportContext context in _rtpClient.GetTransportContexts())
                {
        Media.Common.ISocketReferenceExtensions.SetReceiveBufferSize(context, _bufferSize);
                    Media.Common.ISocketReferenceExtensions.SetReceiveTimeout(context, 50);
                    Media.Common.ISocketReferenceExtensions.SetSendTimeout(context, 2);
                }

                _rtpClient.FrameChangedEventsEnabled = true;
                _rtpClient.RtpFrameChanged += OnSourceFrameChanged;

                foreach (var context in _rtpClient.GetTransportContexts())
                {
                    if (context.MediaDescription.MediaType == MediaType.audio)
                    {
                        IsReceivingAudio = true;
                        // Audioport for response Mediadescription
                        AudioPort = (context.RtpSocket.LocalEndPoint as System.Net.IPEndPoint).Port;
                    }
                    else if (context.MediaDescription.MediaType == MediaType.video)
                    {
                        IsReceivingVideo = true;
                        // Videoport for response Mediadescription
                        VideoPort = (context.RtpSocket.LocalEndPoint as System.Net.IPEndPoint).Port;
                        LocalEarlyMediaEndPoint = context.RtpSocket.LocalEndPoint as IPEndPoint;
                    }
                }

                //_rtpClient.ThreadEvents = true; // can be tested with newer version of rtpclient

                _rtpClient.Activate();


In my OnSourceFrameChanged I am filling a threadingqueue which very fast only stores the data and calls the Decoder in an other thread when there is data.
    internal void OnSourceFrameChanged(object sender, RtpFrame frame = null, RtpClient.TransportContext tc = null, bool final = false)
    {
            if (frame != null && frame.Count > 0 && final)
            {
                if (frame.PayloadType == H264PlayloadType)
                {
                    Media.Common.BaseDisposable.SetShouldDispose(frame, false);
                    if (_videoQueue.Count < 2)
                        _videoQueue.Add(frame);

if DEBUG

                    if (frame.IsMissingPackets)
                    {
                        ClientLogger.Instance.Log(string.Format("Attention! Data loss in RTPClient OnSourceFrameChanged with {0} packets at Time: {1}:{2} ", frame.Count, DateTime.Now.ToString(), DateTime.Now.Millisecond));
                    }

endif

                }
                else if (frame.PayloadType == MuLawPlayloadType)
                {
                    Media.Common.BaseDisposable.SetShouldDispose(frame, false);
                    _audioQueue.Add(frame);
                }
And if the videoqueue has data it is released in ist own thread like this:
    //[System.Runtime.CompilerServices.MethodImpl(System.Runtime.CompilerServices.MethodImplOptions.AggressiveInlining)]
    private byte[] DepacketizeH264(RtpFrame frame)
    {
        byte[] result = Media.Common.MemorySegment.EmptyBytes;

        using (RFC6184Media.RFC6184Frame profileFrame = new RFC6184Media.RFC6184Frame(frame))
        {
            profileFrame.Depacketize();
            if (profileFrame.HasDepacketized)
            {
                result = profileFrame.Buffer.ToArray();
            }
        }
        return result;
    }
Skipping Frames does not help much. You can see Pictures still in good Quality get much slower if there is much Animation. If I skip it is still slower, then you can see skipping and getting faster but then slower again.

What I am wondering why disabling Audio in sdp solves the Problem. Interesting Point: If I Keep Audio in sdp, but turn off both audioqueue for Decoding and sending Audio, it does not get faster, so I believe this is not an issue with too much Network traffic...

And since it is on both iOS and Android and better with Audio off I believe this is not an issue with Decoder.

Any idea?
Thanks a lot Julius!
Nov 23, 2016 at 9:48 AM
Edited Nov 23, 2016 at 9:56 AM
Hello Julius,

I added some Console Writelines to the RFC6184 ProcessPacket Method to Count stuff.

No Animation, empty room, one Minute:

Nalunit number - packet.Playload.Count
28: 1378
7: 50
8: 50
1: 680

Really heavy animations (only did it for 12 seconds, and These Count values *5)
28: 18115
7: 140
8: 140
1: 20

I bet this extremly increased 28 make it slow somehow, I don't know if it gets too slow in your or mine Code or both, will examine further :-)

Or would it make sense to declare some max Count in your codec for max Count of Frames in the depacketized sortedlist?

like
if (Depacketized.Count > 5) return;
in ProcessPacket at the beginning? Or just drop Special Packets this way, others not?
Nov 23, 2016 at 2:54 PM
I have found the Problem.

I was using a reference instead of copying the Frame. And at least in my Version of the rtpclient, it does do stuff with the last Frame after the Event for OnSourceFrameChanged was raised.

Sometimes this happened at the same time. Putting a short delay in top of my OnSourceFrameChanged fixed it.
This is also the cause when we tried together a few months ago it sometimes worked sometimes not.

Now I removed that delay again and make a copy from the Frame and use it. Much better now, delay is around 1 second, was 3-10 before with much animations...

thanks!
Nov 24, 2016 at 1:28 PM
There is still a delay wihich grows up to around 1,5 seconds when much Animation is before the camera (or if I rotate/move it, also a good test).
        Media.Common.ISocketReferenceExtensions.SetReceiveBufferSize(context, 4000000);
                    Media.Common.ISocketReferenceExtensions.SetReceiveTimeout(context, 4);
                    Media.Common.ISocketReferenceExtensions.SetSendTimeout(context, 2);
Are there more properties like this which I can test for Performance?

Thanks!
Nov 30, 2016 at 8:53 AM
Got it working on both Android and iOS by putting all Frames directly in a concurrent Queue and Decoding in own thread like described here:

https://net7mma.codeplex.com/discussions/659774
Dec 5, 2016 at 7:32 AM
Hello,

a remaining delay could be fixed like this, when initialising rtp Client for the Video context:

context.ReceiveInterval = TimeSpan.FromMilliseconds(30);