RtpClient.FromSessionDescription crashes on Xamarain IOS device

Topics: Bug Archive
Apr 25, 2016 at 9:39 AM
Hello Julius,

since the Streaming Video now works on the Xamarin Android device (still too large delay), we also made our app working on an IOS (9.21). It is working until we try to create our RTPClient using RtpClient.FromSessionDescription.

It then crashes with exception:

2016-04-25 11:34:11.955 sks_ClientiOS[3346:669536] Problem occured during connection RTP: System.Net.Sockets.SocketException: Invalid arguments
at System.Net.Sockets.Socket.SetSocketOption (SocketOptionLevel optionLevel, SocketOptionName optionName, Int32 optionValue) [0x0001e] in /Users/builder/data/lanes/3051/5f11db87/source/maccore/_build/Library/Frameworks/Xamarin.iOS.framework/Versions/git/src/mono/mcs/class/System/System.Net.Sockets/Socket.cs:2872
at System.Net.Sockets.Socket.set_SendBufferSize (Int32 value) [0x0001d] in /Users/builder/data/lanes/3051/5f11db87/source/maccore/_build/Library/Frameworks/Xamarin.iOS.framework/Versions/git/src/mono/mcs/class/System/System.Net.Sockets/Socket.cs:687
at Media.Rtp.RtpClient+TransportContext.ConfigureRtpSocket (System.Net.Sockets.Socket socket) [0x0002f] in c:\Projekte
et7mma-111988\Rtp\RtpClient.cs:321
at Media.Rtp.RtpClient+TransportContext.Initialize (System.Net.IPEndPoint localRtp, System.Net.IPEndPoint remoteRtp, System.Net.IPEndPoint localRtcp,
System.Net.IPEndPoint remoteRtcp, Boolean punchHole, Int32 ttl) [0x00087] in c:\Projekte
et7mma-111988\Rtp\RtpClient.cs:1583

Could you please have a look? I'll try to find out more.
Apr 25, 2016 at 9:54 AM
Edited Apr 25, 2016 at 9:54 AM
We found out System.Net.Dns.GetHostAddresses does not seem to work on IOS.

It seems to be used 4 times in your Code. Here is descriped how to do the same for IOS, too.

https://theconfuzedsourcecode.wordpress.com/2015/05/16/how-to-easily-get-device-ip-address-in-xamarin-forms-using-dependencyservice/

We used it for getting our own IP which works in both cases now (Android + IOS).

Perhaps you can integrate that for your existing IsiOS / IOS ? :-)

I will try if the RTPClient works if I Change it for testing on IOS...
Apr 26, 2016 at 8:55 AM
It seems this one is causing the Problem, throwing an exception:

In your RTPClient.cs:
socket.SetSocketOption(SocketOptionLevel.IP, SocketOptionName.TypeOfService, 47);

2016-04-26 10:44:23.007 sks_ClientiOS[4168:840649] Problem occured during connection RTP: System.Net.Sockets.SocketException: Invalid arguments
at System.Net.Sockets.Socket.SetSocketOption (SocketOptionLevel optionLevel, SocketOptionName optionName, Int32 optionValue) [0x0001e] in /Users/builder/data/lanes/3051/5f11db87/source/maccore/_build/Library/Frameworks/Xamarin.iOS.framework/Versions/git/src/mono/mcs/class/System/System.Net.Sockets/Socket.cs:2872
at System.Net.Sockets.Socket.set_SendBufferSize (Int32 value) [0x0001d] in /Users/builder/data/lanes/3051/5f11db87/source/maccore/_build/Library/Frameworks/Xamarin.iOS.framework/Versions/git/src/mono/mcs/class/System/System.Net.Sockets/Socket.cs:687
at Media.Rtp.RtpClient+TransportContext.ConfigureRtpSocket (System.Net.Sockets.Socket socket) [0x0004d] in <filename unknown>:0
at Media.Rtp.RtpClient+TransportContext.Initialize (System.Net.IPEndPoint localRtp, System.Net.IPEndPoint remoteRtp, System.Net.IPEndPoint localR
tcp, System.Net.IPEndPoint remoteRtcp, Boolean punchHole, Int32 ttl) [0x000b1] in <filename unknown>:0
Apr 26, 2016 at 12:28 PM
Edited Apr 26, 2016 at 12:30 PM
Hello Julius,

the following Methods seem not to be implemented for IOS for the System.Sockets:

SendBufferSize
ReceiveBufferSize
TTL
SocketExtensions.GetFirstUnicastIPAddress (see above)

after we wrapped those with Try Catch we got until here in RTPClient.cs:
                //Assign the RemoteRtp EndPoint and Bind the socket to that EndPoint
                RtpSocket.Connect(RemoteRtp = remoteRtp);
which then throws the exception:

2016-04-26 14:29:28.135 sks_ClientiOS[4462:876619] Problem occured during connection RTP: System.Net.Sockets.SocketException: The requested address is not valid in this context
at System.Net.Sockets.Socket.Connect (System.Net.EndPoint remoteEP) [0x000bc] in /Users/builder/data/lanes/3051/5f11db87/source/maccore/_build/Library/Frameworks/Xamarin.iOS.framework/Versions/git/src/mono/mcs/class/System/System.Net.Sockets/Socket.cs:1235
at Media.Rtp.RtpClient+TransportContext.Initialize (System.Net.IPEndPoint localRtp, System.Net.IPEndPoint remoteRtp, System.Net.IPEndPoint localRtcp, System.Net.IPEndPoint remoteRtcp, Boolean punchHole, Int32 ttl) [0x000d7] in C:\Projekte\sks-Kinkel\Repos\Apps Gateway 2.0
et7mma-111988\Rtp\RtpClient.cs:1594
Apr 27, 2016 at 10:48 AM
Hello Julius,

I am still creating my RTPClient like this:

RTPClient = Media.Rtp.RtpClient.FromSessionDescription(sessionDescription, rtpPort: 0, rtcpEnabled: false);

The Problem for IOS seems to be the 0 for the port. If I try to use some other port, I do not run into that exception (but do not receive Frames).

There still Needs to be a try..catch around these socket methods:

SendBufferSize
ReceiveBufferSize
TTL

What port do I have to use for the rtpport? It also does not work if I do not set a port at all...

Thanks!
Apr 27, 2016 at 2:09 PM
Hello Julius,

we can receives Frames now (Decoder not working yet) on our IPhone!

Two changes made to RTPClient for get this working:

In RTPClient FromSessionDescription just after this line (because GetFirstUnicastIPAddress does not seem to work right, see top post)

IPAddress remoteIp = IPAddress.Parse(connectionLine.Host), localIp = Media.Common.Extensions.Socket.SocketExtensions.GetFirstUnicastIPAddress(remoteIp.AddressFamily);

Added this:
localIp = IPAddress.Parse("0.0.0.0");

Then in Transportcontext.Initialize just after the bind
                //Assign the LocalRtp EndPoint and Bind the socket to that EndPoint
                RtpSocket.Bind(LocalRtp = localRtp);
the Connect always crashes on IOS! But it is not needed on IOS, and Android is also working without.

We replaced the connect line with this line:

RemoteRtp = remoteRtp;

After that two changes IOS can receive RTPFramesChanged Events now with data. Will continue there later.

Soon we have a working Xamarin Solution :-)
Just the 98 is still not working, I am using the 88 still. And I cannot use the 98 CopyTo from the Frame, because I do not know the Byte[] Size!
Coordinator
Apr 27, 2016 at 7:11 PM
Interesting, I will work with you on these iOS problems after I resolve the allocation issues.

Today or tomorrow I will have the updates up and we can go from there.
Marked as answer by juliusfriedman on 4/27/2016 at 2:14 PM
Coordinator
Apr 27, 2016 at 9:48 PM
The Byte[] size is something you would define in the application.

E.g. a Decoding Buffer would probably need at least (width * height * bytes per pixel) bytes.

Make this byte[] once or create a MemoryStream that you keep around for the life of the application which uses that same byte[].

If you going to be doing a lot with memory streams you probably want to take a look at the Recycleable Stream as it can help to reduce GC pressure and the delay you experience.

https://github.com/Microsoft/Microsoft.IO.RecyclableMemoryStream

Anyway, in my latest updates the SegmentStream works well for reducing allocations by not relying on the MemoryStream @ all.

If you dispose the frame then the packets, stream and it's buffers go away with it.

If you need to keep the data around longer you will still will need either a memory stream or an array in the application to keep the data for decoding.

In your cases your dealing with Byte[] passed directly to a decoder, normally you could give the decoder the buffers available directly from Depacketized and Enqueue the array from the Depacketized member directly to the encoder.

Unfortunately due to bugs in various decoders or the software wrappers for their hardware counterparts this does not always work and crashes may occur (which are not the fault of this library), e.g. when I give a single byte in the Depacketized which is preceeded by a 3 or 4 byte start code the buffers queued to the decoder would be {start code}, {nalHeader}, {data}...repeating set of the same data.

When the decoder reaches {nalHeader} by itself the bitstream reading logic is incorrect (in their implementation) and the buffer is overrun which causes the application to crash, this is because the decoder expects more data; there is more data ready in the decoders buffers just not within that same buffer just dequeued, the buffer dequeued only contains the data for the nalHeader.

This should result in the next buffer being dequeued or at-least an error code stating that the buffer is malformed but instead crashes occur :)

That is unfortunate and not really my fault, it should be brought up with the developers of the Decoder so they can address the issue which is also probably a security concern.

I am not going to be able to get an update out today but I will have one out tomorrow and we can go from there.
Marked as answer by juliusfriedman on 4/27/2016 at 2:49 PM
Apr 28, 2016 at 6:06 AM
Hello Julius,

thanks. I am anxious to see the next update :-)
Coordinator
Apr 28, 2016 at 7:00 PM
No problem, what I am going to do is make the release with the updates I have and then seek feedback from you on how you think I should proceed if possible.

It will be easy to change the RtpFrame to use the SegmentStream just by un-commenting out the call which is already there, you can test like that to determine if it will help you more than the CopyTo methods.

e.g. Remove the Comment which looks like this in RtpFrame on the Buffer Property and Comment the existing getter.
//get { return HasBuffer ? m_Buffer : m_Buffer = new Common.SegmentStream(Depacketized.Values); }
From there it will be easy to judge exactly where you need to adjust for allocations in your application as no more memory will be allocated by using the 'Buffer' than would already be used for existing objects (plus the CLR overhead of the object references).

Then we can address the issues with GetFirstUnicastIPAddress... I think your looking for an overload which also takes a specific NetworkInterfaceType, I have added a method which can return all interfaces of a specific type for that purpose. You can combine that with 'GetFirstUnicastIPAddress' like so:
GetFirstUnicastIPAddress(GetNetworkInterface(System.Net.NetworkInformation.NetworkInterfaceType.Wireless80211), IPAddress.Any);
If that works then I will see about adding a NetworkInterfaceType overload to 'FromSessionDescription' as it would allow you to select the interface type you desire for the client type. Personally and Technically I feel that the SDP should be allowed to signal interface type such as 80211 to be specified separately as well but I will leave that for another time as IANA only has 4 types defined @ http://www.iana.org/assignments/sdp-parameters/sdp-parameters.xhtml (nettype) and the semantic is pretty much the same and another attribute could be used or an alternate connection line in a case where a server has a separate wifi end point.

Then we can address the issues with 'System.Net.Dns.GetHostAddresses', it will be a bigger fix but I can just add my own DNS implementation if needed as I find that it may be helpful to use alternate routes from the RoutingTable in certain unique cases although the main benefit would be a cross platform DNS Client and possibly server.

What I am curious about though is where you say I use it.... I only use it if the HostNameType is DNS in the RtspClient / HttpClient.

Where are you getting a DNS host specified instead of an IPAddress? If you can send me a Wireshark capture I will be able to understand better.

Thanks and I will pose the updates very soon!
Marked as answer by juliusfriedman on 4/28/2016 at 12:00 PM
Coordinator
Apr 28, 2016 at 7:17 PM
112013 has been added, I will be taking a break. If I don't hear from you I will continue with working in RtpFrame and SegmentStream / other issues as required.

Thanks again for your continued feedback and testing!
Marked as answer by juliusfriedman on 4/28/2016 at 12:17 PM
Apr 29, 2016 at 11:02 AM
Edited Apr 29, 2016 at 12:38 PM
Hello Julius,
//get { return HasBuffer ? m_Buffer : m_Buffer = new Common.SegmentStream(Depacketized.Values); }
This did not work. It cannot be casted to a Memorystream (the Segmentstream).

I just rebuild the 112013 and replaced my assemblies which were still from the 88.
I get one single Picture again which freezed and then I get the IllegalStateExceptions (with the same Code that worked with the 88).

I am still using
                using (RFC6184Media.RFC6184Frame profileFrame = new RFC6184Media.RFC6184Frame(frame))
                {
                    profileFrame.Depacketize(); //false true
                    if (profileFrame.HasDepacketized)
                    {
                        if (VideoFrameReceived != null)
                        {
                            VideoFrameReceived(this, profileFrame.Buffer.ToArray());
and just starting to examine this issue. thanks :-)

PS: I am again on the Android device with the delay Problems. I'll post this in the other thread, too. We are still working on IOS Decoder Problems.
Coordinator
Apr 29, 2016 at 2:23 PM
I can see what you mean about Depacketize giving you the information you need about not having to use HasDepacketized.

This is especially true if you wanted to update data within the RtpFrame by replacing a packet etc.

Depacketize should probably return true if new data has been depacketized.

This will also allow the skipping of the check for HasDepacketized in most cases.

I will add that my notes!
Marked as answer by juliusfriedman on 4/29/2016 at 7:23 AM
Apr 29, 2016 at 3:19 PM
Edited Apr 29, 2016 at 3:20 PM
I am in the Weekend now :-)

Thanks a lot for your help! I will try early monday, I can not try from here.

Have a nice Weekend Julius!
May 3, 2016 at 2:07 PM
Edited May 3, 2016 at 2:07 PM
Hi Julius,

just wanted to inform you it is even working on IOS now!

The Video is even slower and even more delayed on the IOS then on the Andriod, but it is working! Hooray :-)

I am still using 88.

I am using SourceFramechanged (same as Android) with the same Code, if final and correct payloadtype:
            using (RFC6184Media.RFC6184Frame profileFrame = new RFC6184Media.RFC6184Frame(frame))
            {
                profileFrame.Depacketize(); //false true
                if (profileFrame.HasDepacketized)
                    if (VideoFrameReceived != null)
                        VideoFrameReceived(this, profileFrame.Buffer.ToArray()); // give to Decoder
In RTPClient I made ll changes I mentioned on 27 April. Now after some work on the Decoder it is working on IOS.

It would be really great if we could get both working with a newer Version now :-)
Coordinator
May 3, 2016 at 2:32 PM
I am not ignoring you, I am having Internet issues which will hopefully be resolved today.

Once my Internet is back up an running I can post an update.

I have several updates which fix the Binary constructor to work in non full trust and some other performance features related to sending.

I have also updated RtpFrame to use the SegmentStream.

I have also added a few more experimental classes.

I will work with you on the 88 issue as well as anything else which remains once I post the updates.

I will update later today or tomorrow.
Marked as answer by juliusfriedman on 5/3/2016 at 7:32 AM
May 4, 2016 at 10:53 AM
Hi Julius,

no Problem, and good luck with your Internet :-)

Just to let you know: IOS is now working good. Even a bit better than the current Android Version. Still working with 88.
Coordinator
May 4, 2016 at 11:34 AM
Thanks, everything should be resolved. I will get an update out today at some point.

Then we can assess the immediate issues present.

Expect an update within like 4 hours, I just want to run some unit tests and test a few more things.
Marked as answer by juliusfriedman on 5/4/2016 at 4:34 AM
Coordinator
May 4, 2016 at 1:06 PM
Update is up!
Marked as answer by juliusfriedman on 5/4/2016 at 6:06 AM
May 9, 2016 at 8:15 AM
Hello Julius,

I just downloaded and test 112027. It is running with good Performance (compared to old 88) on Android, when animations are there sometimes a bit artefacts.

It is not running on IOS, it cannot connect to the RTPClient (using FromSessionDescription). again I set the local IP manually when connecting RTPClient like I did before etc (I used the RTPClient I have sent you on friday I think it is in the other post), I got SetSocketOption Errors and put all SetSocketOptions in the RTPSocket in a try..catch.
Now I can connect to RTPClient on IOS and receive Frames. I see the starting Frame and it freezes. I did no other changes, with 88 it works on IOS, too.

Just to let you know. Have a nice week :-)
Coordinator
May 9, 2016 at 1:33 PM
I just made an update which should prevent you from having to add Try / Catch, I may later log when those configurations failed also via the Logger.

I will be working on the RtpFrame further this week hopefully.
Marked as answer by juliusfriedman on 5/9/2016 at 6:33 AM
May 10, 2016 at 10:57 AM
Hello Julius,

I am right now caught in an other Topic. But I will gladly check the latest Version soon (next couple of days) :-)

Thanks a lot
Firlefanz
Marked as answer by juliusfriedman on 5/10/2016 at 8:15 AM
Coordinator
May 13, 2016 at 7:44 PM
The latest code is up.

It should get you even better performance.

Please take a look and let me know what issues still remain in a new thread as this one is kind of long.

Take care and talk to you soon!
Marked as answer by juliusfriedman on 5/13/2016 at 12:45 PM
Coordinator
Jun 8, 2016 at 5:48 PM
Just for the record I will also post in this thread:

FromSessionDescription uses GetFirstUnicastIPAddress.

Media.Common.Extensions.NetworkInterface.NetworkInterfaceExtensions.GetNetworkInterface

Is definitely what you want it to use there and not GetFirstUnicastIPAddress, this is because it's conceivable that you could have two wireless interfaces or even an Ethernet connection through the data port.

I use that because I want the first address I can find which matches the address family of RemoteIp,AddressFamily.

You should really be creating your socket and passing to the function to avoid this error totally.

System.Net.NetworkInformation.NetworkInterface localInterface;

            if (false.Equals(existingSocket.Equals(null)) && existingSocket.IsBound)
            {
                //This interface should be the interface you plan on using for the Rtp communication
                localIp = Media.Common.Extensions.Socket.SocketExtensions.GetFirstUnicastIPAddress(remoteIp.AddressFamily, out  localInterface);
            }
            else
            {
                //Use the localIp of the exsisting socket.
                localIp = ((System.Net.IPEndPoint)existingSocket.LocalEndPoint).Address;
            }
You could always pass in a socket and not have to worry about the issue.

I could also add an option to FromSessionDescription which allows a NetworkInterface but the semantic is the same and I think is better than the try catch in connect expect in certain scenarios e.g. where exceptions happen unexpectedly.

I could also just add a TryCreateFromSessionDescription which returns true or false and I think that would also be the same.

We can talk about it in the other thread which way you feel is better and why!

Thanks again for bringing this up!
Marked as answer by juliusfriedman on 6/8/2016 at 10:48 AM