Same code and decoder: Ultrafast with no delay with video only, extremly slow with audio and video

Topics: Question
Nov 28, 2016 at 12:35 PM
Hello,

since I had so many Problems with the Performance, I now tried the latest Version 112204.

It is extremly slow, maybe 1 Frame per second with sometimes artefacts.

When I remove Audio from the sdp when initialising the rtp Client, it is ultrafast, fantastic Picture, no delay I would recognize.

If I have Audio in the sdp, but turn off Audio sending from the Client and Audio Decoding in my app, it is still slow.

So I guess there is something wrong with packet handling or playload type, is there something I can check? Some Special place in the rtp or rfc6184 I could look into if it works right?

Just some Information: I replaced the Depackteized -sortedlist with -dictionary, because I thought sortedlist may be slow, I didn't see a big Change in any direction...

Thanks a lot,

firlefanz
Coordinator
Nov 29, 2016 at 8:20 PM
It's really hard to say without doing some debugging first hand.

If your Audio process is any similar to the Video process you shouldn't have performance issues.

Are you sure it doesn't have something to do with inter stream synchronization during playback or are you playing each stream separately?

If your playing each stream separately then its possible that the decoder threads are getting caught up somewhere and blocking the application.

All my library does is facilitate the transfer of data from the socket according to the protocol; if you can ensure that the data is copied and used from another thread while handling the events that would be the best advice I can provide, see the ThreadEvents property and the opportunity it provides for you to handle the packets from a thread which doesn't block sending and receiving but be sure not to provide enough memory in the buffer to suit the application.
Marked as answer by juliusfriedman on 11/29/2016 at 1:20 PM
Nov 30, 2016 at 7:44 AM
Hello Julius,

thanks for your answer!

I can now tell that your latest Version is working on both Android and iOS devices. Even without having to Change something in the socket stuff for iOS.
Works out of the box now.

The big delay when having both Audio and Video the same time, with much Animation has gotten much better since I made a concurrentqueue for each Audio and Video frames, just take the data and have the Decoder working on an own thread each.

Now it is a lot better! Only when having real much Animation (like rotating the camera) I have a Little delay still.

Thanks, this is a great library!
Dec 2, 2016 at 6:38 AM
One error that happens with the latest Version (not with the older one I currently use again because of that):

When depacketizing Video Frames:

{System.NullReferenceException: Object reference not set to an instance of an object.
at Media.Common.MemorySegment.Equals (Media.Common.MemorySegment other) [0x00000] in c:\Projekte\net7mma-112204\Common\Classes\MemorySegment.cs:423
at Media.Common.MemorySegment.op_Inequality (Media.Common.MemorySegment a, Media.Common.MemorySegment b) [0x00000] in c:\Projekte\net7mma-112204\Common\Classes\MemorySegment.cs:435
at sks_Client.RFC6184Media+RFC6184Frame.Depacketize (Media.Rtp.RtpPacket packet) [0x00001] in C:\Projekte\sks-Kinkel\Repos\Apps Gateway 2.0\sks-Client\sks-Client\sks_Client\RFC6184Media.cs:394
at Media.Rtp.RtpFrame.Depacketize (System.Boolean allowIncomplete) [0x00022] in c:\Projekte\net7mma-112204\Rtp\RtpFrame.cs:1279
at Media.Rtp.RtpFrame.Depacketize () [0x00000] in c:\Projekte\net7mma-112204\Rtp\RtpFrame.cs:1249
at sks_Client.Manager.RTPClientManager.DepacketizeH264 (sks_Client.RFC6184Media+RFC6184Frame profileFrame) [0x00007] in C:\Projekte\sks-Kinkel\Repos\Apps Gateway 2.0\sks-Client\sks-Client\sks_Client\Manager\RTPClientManager.cs:185
at sks_Client.Manager.RTPClientManager.OnSourceFrameChanged (System.Object sender, Media.Rtp.RtpFrame frame, Media.Rtp.RtpClient+TransportContext tc, System.Boolean final) [0x0007f] in C:\Projekte\sks-Kinkel\Repos\Apps Gateway 2.0\sks-Client\sks-Client\sks_Client\Manager\RTPClientManager.cs:451 }


I will try to have a look.
Dec 2, 2016 at 6:47 AM
Edited Dec 2, 2016 at 6:54 AM
After Fixing that:
    public bool Equals(MemorySegment other)
    {
        return other!= null && other.GetHashCode().Equals(GetHashCode());
    }
now I get

{System.NullReferenceException: Object reference not set to an instance of an object.
at Media.Common.MemorySegment.op_Inequality (Media.Common.MemorySegment a, Media.Common.MemorySegment b) [0x00000] in C:\Projekte\net7mma-112204\Common\Classes\MemorySegment.cs:435
at Media.Common.MemorySegment.Equals (Media.Common.MemorySegment other) [0x00000] in C:\Projekte\net7mma-112204\Common\Classes\MemorySegment.cs:423
at Media.Common.MemorySegment.op_Inequality (Media.Common.MemorySegment a, Media.Common.MemorySegment b) [0x00000] in C:\Projekte\net7mma-112204\Common\Classes\MemorySegment.cs:435
at sks_Client.RFC6184Media+RFC6184Frame.Depacketize (Media.Rtp.RtpPacket packet) [0x00001] in C:\Projekte\sks-Kinkel\Repos\Apps Gateway 2.0\sks-Client\sks-Client\sks_Client\RFC6184Media.cs:394
at Media.Rtp.RtpFrame.Depacketize (System.Boolean allowIncomplete) [0x00022] in C:\Projekte\net7mma-112204\Rtp\RtpFrame.cs:1279
at Media.Rtp.RtpFrame.Depacketize () [0x00000] in C:\Projekte\net7mma-112204\Rtp\RtpFrame.cs:1249
at sks_Client.Manager.RTPClientManager.DepacketizeH264 (sks_Client.RFC6184Media+RFC6184Frame profileFrame) [0x00007] in C:\Projekte\sks-Kinkel\Repos\Apps Gateway 2.0\sks-Client\sks-Client\sks_Client\Manager\RTPClientManager.cs:185
at sks_Client.Manager.RTPClientManager.OnSourceFrameChanged (System.Object sender, Media.Rtp.RtpFrame frame, Media.Rtp.RtpClient+TransportContext tc, System.Boolean final) [0x0007f] in C:\Projekte\sks-Kinkel\Repos\Apps Gateway 2.0\sks-Client\sks-Client\sks_Client\Manager\RTPClientManager.cs:451 }


I am not sure and Change this to:
public static bool operator !=(MemorySegment a, MemorySegment b) { return a==null || b==null && a.Equals(b).Equals(false); }
Dec 2, 2016 at 7:12 AM
Edited Dec 2, 2016 at 7:18 AM
Now I get this one:

{System.NullReferenceException: Object reference not set to an instance of an object.
at Media.Common.MemorySegment.Equals (Media.Common.MemorySegment other) [0x00000] in C:\Projekte\net7mma-112204\Common\Classes\MemorySegment.cs:423
at Media.Common.MemorySegment.op_Inequality (Media.Common.MemorySegment a, Media.Common.MemorySegment b) [0x00000] in C:\Projekte\net7mma-112204\Common\Classes\MemorySegment.cs:435
at sks_Client.RFC6184Media+RFC6184Frame.Depacketize (Media.Rtp.RtpPacket packet) [0x00001] in C:\Projekte\sks-Kinkel\Repos\Apps Gateway 2.0\sks-Client\sks-Client\sks_Client\RFC6184Media.cs:394
at Media.Rtp.RtpFrame.Depacketize (System.Boolean allowIncomplete) [0x00022] in C:\Projekte\net7mma-112204\Rtp\RtpFrame.cs:1279
at Media.Rtp.RtpFrame.Depacketize () [0x00000] in C:\Projekte\net7mma-112204\Rtp\RtpFrame.cs:1249
at sks_Client.Manager.RTPClientManager.DepacketizeH264 (sks_Client.RFC6184Media+RFC6184Frame profileFrame) [0x00007] in C:\Projekte\sks-Kinkel\Repos\Apps Gateway 2.0\sks-Client\sks-Client\sks_Client\Manager\RTPClientManager.cs:185
at sks_Client.Manager.RTPClientManager.OnSourceFrameChanged (System.Object sender, Media.Rtp.RtpFrame frame, Media.Rtp.RtpClient+TransportContext tc, System.Boolean final) [0x0008a] in C:\Projekte\sks-Kinkel\Repos\Apps Gateway 2.0\sks-Client\sks-Client\sks_Client\Manager\RTPClientManager.cs:453 }


or sometimes


{System.NullReferenceException: Object reference not set to an instance of an object.
at Media.Common.MemorySegment.Equals (Media.Common.MemorySegment other) [0x00000] in C:\Projekte\net7mma-112204\Common\Classes\MemorySegment.cs:430
at Media.Common.MemorySegment.op_Inequality (Media.Common.MemorySegment a, Media.Common.MemorySegment b) [0x00000] in C:\Projekte\net7mma-112204\Common\Classes\MemorySegment.cs:442
at sks_Client.RFC6184Media+RFC6184Frame.Depacketize (Media.Rtp.RtpPacket packet) [0x00001] in C:\Projekte\sks-Kinkel\Repos\Apps Gateway 2.0\sks-Client\sks-Client\sks_Client\RFC6184Media.cs:394
at Media.Rtp.RtpFrame.Depacketize (System.Boolean allowIncomplete) [0x00022] in C:\Projekte\net7mma-112204\Rtp\RtpFrame.cs:1279
at Media.Rtp.RtpFrame.Depacketize () [0x00000] in C:\Projekte\net7mma-112204\Rtp\RtpFrame.cs:1249
at sks_Client.Manager.RTPClientManager.DepacketizeH264 (sks_Client.RFC6184Media+RFC6184Frame profileFrame) [0x00007] in C:\Projekte\sks-Kinkel\Repos\Apps Gateway 2.0\sks-Client\sks-Client\sks_Client\Manager\RTPClientManager.cs:185
at sks_Client.Manager.RTPClientManager.OnSourceFrameChanged (System.Object sender, Media.Rtp.RtpFrame frame, Media.Rtp.RtpClient+TransportContext tc, System.Boolean final) [0x0008a] in C:\Projekte\sks-Kinkel\Repos\Apps Gateway 2.0\sks-Client\sks-Client\sks_Client\Manager\RTPClientManager.cs:453 }


The same Video stream works good with the older Version (and never runs into an exception I could catch, I just checked that)! Could you please consider adding some null checks? I do not know where is the entry Point this should be checked before changing too much of your Code...


Thanks a lot and have a nice Weekend!
Dec 5, 2016 at 7:33 AM
It is working fine now with the older Version adding this when initialising rtp Client Video context:

context.ReceiveInterval = TimeSpan.FromMilliseconds(30);

Thanks :-)