UDP stream as source

Oct 21, 2014 at 10:35 PM
I currently emit .h264 video from FFMPEG with UDP or RTP and can play with something like:
ffplay udp://127.0.0.1:11001
ffplay rtp.sdp  
vlc -vvv udp://127.0.0.1:11001
Is there a way to use or derive from RtspSourceStream to make this work?

Cheers,
brett
Coordinator
Oct 22, 2014 at 1:45 PM
Edited Oct 14, 2015 at 4:56 PM
Post up the sdp just to be sure, I just need to finish the FromSessionDescription method and then make a RtpSourceStream constructor for it.

If you look in the example I create a sdp in the RtpClient test, this is the opportunity to implement the opposite.

Thanks
Marked as answer by juliusfriedman on 10/22/2014 at 5:46 AM
Oct 22, 2014 at 5:49 PM
SDP:
v=0
o=- 0 0 IN IP4 127.0.0.1
s=No Name
c=IN IP4 127.0.0.1
t=0 0
a=tool:libavformat 55.44.100
m=video 11001 RTP/AVP 96
a=rtpmap:96 H264/90000
a=fmtp:96 packetization-mode=1; sprop-parameter-sets=J2QAKKx2gHgCJ+XARAAAAwAEAAADAFA8IhG4,KM44gA==; profile-level-id=640028
Coordinator
Oct 22, 2014 at 8:30 PM
Edited Oct 22, 2014 at 8:31 PM
Looks appropriate, ill include support when I update the container alpha later in the week.

If you need something sooner just parse the sdp from the rtspsource class and instantiate the client.

The port is in the media description (m=) line.

Thanks for bringing this up!
Coordinator
Oct 24, 2014 at 2:21 AM
109858 Should fix your issue.

Let me know if you need anything else!
Marked as answer by juliusfriedman on 10/23/2014 at 6:22 PM
Oct 24, 2014 at 10:13 PM
Thanks! I'll try it out...
Oct 27, 2014 at 6:41 PM
So how would I use it?

Something like this?

rtspServer.AddStream( new RtspSourceStream("FromUDPSource", "udp://235.0.0.1:11002", RtspClient.ClientProtocolType.Udp) );
Oct 27, 2014 at 8:32 PM
Is this closer?
        //h264 is RFC6184 ??
        Media.Sdp.SessionDescription sd = new Media.Sdp.SessionDescription(1);
        Media.Sdp.MediaDescription md = new Media.Sdp.MediaDescription(Media.Sdp.MediaType.video, 11002, Media.Rtsp.Server.Streams.RtpSource.RtpMediaProtocol, Media.Rtsp.Server.Streams.RFC2435Stream.RFC2435Frame.RtpJpegPayloadType);
        Media.Sdp.Lines.SessionConnectionLine sdl = new Media.Sdp.Lines.SessionConnectionLine() { Address = "127.0.0.1", AddressType = "IP4", NetworkType = "IN" };
        sd.Add(md);
        sd.Add(sdl);
        RtpSource src = new RtpSource("Table", sd);
        server.AddStream(src);
It's not working though. Also, on the previous build I could stream rtsp://localhost/live/Omega with VLC. This build doesn't seem to work for that test case.
Coordinator
Oct 28, 2014 at 3:59 PM
Sorry, that was just fixed in 109895.

I should have made a test run before releasing.

Now it should work as expected and yes the 2nd is closer and probably exactly what you need unless you have the contents of the sdp you posted from vlc, that should also work.
Oct 28, 2014 at 6:35 PM
ok, the omega stream works now to VLC. Doesn't work to ffplay however.

And on my RTP stream, I get the exception "The requested address is not valid in its context" at RtpClient.cs.853. If I change my RTP stream to multicast(235.), there is no exception, but still no video showing in VLC.
Coordinator
Oct 28, 2014 at 7:37 PM
Double check @ 109901...

I'm confused because the UDP source has no meta information which is required to attach to the stream really, that's why I implemented the rtp://sdp uri support.

I would have to look into see if raw "udp or tcp" schemes would be able to be supported, also I just checked and both ffplay and vlc seem to be working (they use the same code) as well as Quick Time.

So essentially when you have the "sdp" from ffmpeg like you posted previously you just give that to the RtpSource constructor and it will aggregate the stream for you.

If you need anything else let me know.
Oct 28, 2014 at 9:59 PM
I'm not sure how FFPLAY knows how to decode the UDP streams. RTP will work for me tho if we can get it working.

I tried building the SessionDescription from the FFMPEG output and still get exceptions setting up the stream for 127.0.0.1. When going to 235.. it gets past that, but then when i make the request from VLC I see this in the debugger console:
Request OPTIONS rtsp://localhost/live/Test a8dc867c-58bb-4d3f-943e-c463e696952a 

Response 2 OK a8dc867c-58bb-4d3f-943e-c463e696952a 

Request DESCRIBE rtsp://localhost/live/Test a8dc867c-58bb-4d3f-943e-c463e696952a 

A first chance exception of type 'System.NullReferenceException' occurred in Media.dll
Object reference not set to an instance of an object. 
    at Media.Rtsp.ClientSession.CreateSessionDescription(SourceStream stream) in c:\Dev\Brett\vl\net7mma-100901\Rtsp\Server\ClientSession.cs:line 1015
   at Media.Rtsp.ClientSession.ProcessDescribe(RtspMessage describeRequest, SourceStream source) in c:\Dev\Brett\vl\net7mma-100901\Rtsp\Server\ClientSession.cs:line 378
   at Media.Rtsp.RtspServer.ProcessRtspDescribe(RtspMessage request, ClientSession session) in c:\Dev\Brett\vl\net7mma-100901\Rtsp\Server\RtspServer.cs:line 1544
   at Media.Rtsp.RtspServer.ProcessRtspRequest(RtspMessage request, ClientSession session) in c:\Dev\Brett\vl\net7mma-100901\Rtsp\Server\RtspServer.cs:line 1167 
 
Here's how I hook setup the stream:
            Media.Sdp.SessionDescription sdt = new Media.Sdp.SessionDescription(@"
SDP:
v=0
o=- 0 0 IN IP4 127.0.0.1
s=No Name
c=IN IP4 235.0.0.1
t=0 0
a=tool:libavformat 56.9.101
m=video 11002 RTP/AVP 96
a=rtpmap:96 H264/90000
a=fmtp:96 packetization-mode=1; sprop-parameter-sets=J2QAKKx2gHgCJ+XARAAAAwAEAAADAFA8IhG4,KM44gA==; profile-level-id=640028
");

            RtpSource src2 = new RtpSource("Test", sdt);
            server.AddStream(src2);
Coordinator
Oct 28, 2014 at 10:04 PM
Your right, I need to also assign in the given sdp.

I will fix this shortly.

Thanks for the heads up.
The udp stream may be in the format specifically for that library e.g. libav.

See if you can capture anything with wireshark and ill take a look.
Oct 28, 2014 at 11:49 PM
Ok, I have a wireshark capture. How should I get it to you?
Coordinator
Oct 29, 2014 at 2:20 AM
Try via email or by uploading it somewhere if it's to large. e.g. google drive.

109907 fixes the issue with SDP not being assigned which subsequently fixes the NullReferenceException.

In the next week a lot of changes and improvements are coming so if you notice anything else let me know!

Thanks again!
Marked as answer by juliusfriedman on 10/28/2014 at 6:20 PM
Oct 29, 2014 at 7:42 PM
ok, lets see how it goes next week. 109907 is still giving errors:
ImageStream39086ce5-e50c-48f5-a512-6fd94d5824c3 Encoding: C:\Dev\Brett\net7mma-109907\Tests\bin\x64\Debug\JpegTest\yh9r01459srgbsmall.jpg
ImageStream39086ce5-e50c-48f5-a512-6fd94d5824c3 Done Encoding: C:\Dev\Brett\net7mma-109907\Tests\bin\x64\Debug\JpegTest\yh9r01459srgbsmall.jpg
ImageStream39086ce5-e50c-48f5-a512-6fd94d5824c3 Encoding: C:\Dev\Brett\net7mma-109907\Tests\bin\x64\Debug\JpegTest\yuvj422.jpg
ImageStream39086ce5-e50c-48f5-a512-6fd94d5824c3 Done Encoding: C:\Dev\Brett\net7mma-109907\Tests\bin\x64\Debug\JpegTest\yuvj422.jpg
ImageStream39086ce5-e50c-48f5-a512-6fd94d5824c3 Started
The thread 'RtpClient-74.125.209.548743b11d-f951-4bc7-8b37-847c01a02488' (0xa18) has exited with code 0 (0x0).
The thread '<No Name>' (0x180c) has exited with code 0 (0x0).
Request OPTIONS rtsp://localhost/live/Test 8e0ef440-2bd2-499d-b60f-7613bb3f91b6 

Response 2 OK 8e0ef440-2bd2-499d-b60f-7613bb3f91b6 

Request DESCRIBE rtsp://localhost/live/Test 8e0ef440-2bd2-499d-b60f-7613bb3f91b6 

Response 3 OK 8e0ef440-2bd2-499d-b60f-7613bb3f91b6 

Request SETUP rtsp://127.0.0.1/live/c43c0227-8368-4e9c-a7f8-fa4d454be247/live/c43c0227-8368-4e9c-a7f8-fa4d454be247/video 8e0ef440-2bd2-499d-b60f-7613bb3f91b6 

Response 4 OK 8e0ef440-2bd2-499d-b60f-7613bb3f91b6 

Request PLAY rtsp://127.0.0.1/live/c43c0227-8368-4e9c-a7f8-fa4d454be247 8e0ef440-2bd2-499d-b60f-7613bb3f91b6 

Response 5 OK 8e0ef440-2bd2-499d-b60f-7613bb3f91b6 

Request GET_PARAMETER rtsp://127.0.0.1/live/c43c0227-8368-4e9c-a7f8-fa4d454be247 8e0ef440-2bd2-499d-b60f-7613bb3f91b6 

Response 6 OK 8e0ef440-2bd2-499d-b60f-7613bb3f91b6 

Request TEARDOWN rtsp://127.0.0.1/live/c43c0227-8368-4e9c-a7f8-fa4d454be247 8e0ef440-2bd2-499d-b60f-7613bb3f91b6 

Request OPTIONS rtsp://localhost/live/Test b13e9982-cceb-437c-946c-407bea758db4 

Response 2 OK b13e9982-cceb-437c-946c-407bea758db4 

Request DESCRIBE rtsp://localhost/live/Test b13e9982-cceb-437c-946c-407bea758db4 

A first chance exception of type 'System.InvalidOperationException' occurred in mscorlib.dll
Response 3 OK b13e9982-cceb-437c-946c-407bea758db4 

Response 7 OK 8e0ef440-2bd2-499d-b60f-7613bb3f91b6 

Request SETUP rtsp://127.0.0.1/live/c43c0227-8368-4e9c-a7f8-fa4d454be247/live/c43c0227-8368-4e9c-a7f8-fa4d454be247/video b13e9982-cceb-437c-946c-407bea758db4 

Response 4 OK b13e9982-cceb-437c-946c-407bea758db4 

Request PLAY rtsp://127.0.0.1/live/c43c0227-8368-4e9c-a7f8-fa4d454be247 b13e9982-cceb-437c-946c-407bea758db4 

A first chance exception of type 'System.Net.Sockets.SocketException' occurred in System.dll
The thread 'RtpClient-127.0.0.168226363-ad4a-4764-97c5-98a8edfe4480' (0x2afc) has exited with code 0 (0x0).
Response 5 OK b13e9982-cceb-437c-946c-407bea758db4 

An established connection was aborted by the software in your host machine 
    at System.Net.Sockets.Socket.DoBeginReceiveFrom(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags, EndPoint endPointSnapshot, SocketAddress socketAddress, OverlappedAsyncResult asyncResult)
   at System.Net.Sockets.Socket.BeginReceiveFrom(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags, EndPoint& remoteEP, AsyncCallback callback, Object state)
   at Media.Rtsp.RtspServer.ProcessSend(IAsyncResult ar) in c:\Dev\Brett\net7mma-109907\Rtsp\Server\RtspServer.cs:line 1053 
 

The thread '<No Name>' (0x1090) has exited with code 0 (0x0).
The thread '<No Name>' (0x29fc) has exited with code 0 (0x0).
Request TEARDOWN rtsp://127.0.0.1/live/c43c0227-8368-4e9c-a7f8-fa4d454be247 b13e9982-cceb-437c-946c-407bea758db4 

Response 6 OK b13e9982-cceb-437c-946c-407bea758db4 

A first chance exception of type 'System.ObjectDisposedException' occurred in System.dll
A first chance exception of type 'System.ObjectDisposedException' occurred in System.dll
A first chance exception of type 'System.ObjectDisposedException' occurred in System.dll
Coordinator
Oct 31, 2014 at 7:46 PM
Edited Nov 1, 2014 at 3:42 AM
Sorry for the delay, 109946 should fix this issue.

I have given it a few tests and it seems to be fine now, the only thing I have to work on is when TCP is being used and RFC4751 framing is used rather than RFC2326.

I also need to update the examples to allow for Standalone - TCP testing as right now they don't work for TCP because I need to re-structure the setup but that shouldn't be an issue for you or anyone else.

Let me know if you run into anything else.!

(I updated the code again, and verified streaming works)
Coordinator
Nov 21, 2014 at 5:41 AM
Udp seems like it is a Mpeg Transport Stream.

I have implemented the reader required, the MediaStream required to aggregate such a source is yet to be completed but it would be simply something which took in a TransportStreamReader from either a file or from the Network and then created a RFC2250 / etc frame for the packet.

e.g a 'TransportStreamSource'

I will work more on this when I rip apart the MediaTypes from the RtspServer.

Until then I hope your other issues are resolved.
Sep 15, 2015 at 11:56 PM
I'm also trying to make this work, but having some difficulties.

I have a couple H264 RTP streams. One I'm generating with GStreamer videotestsrc, the other is a from a file using FFMpeg. Using vlc, I can successfully connect to both using an sdp file. But creating an RtpSource doesn't work. It will send a PLAY response, then wait awhile and vlc will say it can't connect to the MRL. Also tried with mplayer and also no luck. Mplayer gives me end of stream messages after it gets the PLAY response.

If it helps, here's one of my RTP streams:
ffmpeg -re -i video.mp4 -vcodec h264 -an -f rtp rtp://127.0.0.1:5000
And the corresponding SDP:
v=0
o=- 0 0 IN IP4 127.0.0.1
c=IN IP4 192.168.16.33
s=H264 STREAM
m=video 5000 RTP/AVP 96
a=rtpmap:96 H264/90000
And some simple testing code (The RtspSource works fine, but RtpSource doesn't).
        private static RtpSource BuildRTPSource()
        {
            var sdt = new SessionDescription(@"
                v=0
                o=- 0 0 IN IP4 127.0.0.1
                c=IN IP4 192.168.16.33
                s=H264 STREAM
                m=video 5000 RTP/AVP 96
                a=rtpmap:96 H264/90000
                ");
            return new RtpSource("Test", sdt) {RtcpDisabled = true};
        }

        private static RtspSource BuildRTSPSource()
        {
            return new RtspSource("Bunny", "rtsp://wowzaec2demo.streamlock.net/vod/mp4:BigBuckBunny_115k.mov");
        }

        private static RtspServer BuildServer()
        {
            return new RtspServer {Logger = new RtspServerConsoleLogger()};
        }

        private static void Main(string[] args)
        {
            using (var server = BuildServer())
            {
                server.TryAddMedia(BuildRTPSource());
                server.TryAddMedia(BuildRTSPSource());

                server.Start();

                Console.ReadLine();

                server.DisableHttpTransport();
                server.DisableUnreliableTransport();
                server.Stop();
            }
        }
I don't know if there's something different in the newer releases of this library or if I'm just missing something. Hopefully it's just something simple I'm not seeing.
Sep 16, 2015 at 7:47 PM
Apparently the problem is in TransportContext.Initialize. It appears that you can't call RtpSocket.Connect, at least in my situation. I thought it was related to this, but the suggestions listed didn't help.
Coordinator
Sep 17, 2015 at 2:11 PM
...

Your example is a Rtsp server, it supports consumption of rtp sources without an rtsp layer, hence RtpSource or HttpSource.

If you're trying to eliminate rtsp your going about it wrong, if not I honestly can't understand what issues your having or what your trying to do.
Marked as answer by juliusfriedman on 9/17/2015 at 6:11 AM
Sep 17, 2015 at 4:12 PM
I'm simply trying to take an existing RTP stream and convert into an RTSP stream. Is that not what my code example does?
Coordinator
Sep 17, 2015 at 4:38 PM
Well in that case, It does appear to consume a RtpStream and make it available in the RtspServer, it also disabled Rtcp which may have something to do with it.

Try without disabling Rtcp and see if you have better luck, If the source is sending packets then you should be receiving them on the port(s) specified (5000 and 50001), there may be a problem with the source or receiving the data due to some type of networking issues.

One way to verify that would be to use ffmpeg to consume the same rtp stream and verify the results without using this software, afterwards you should be able to consume with this library just as easily.
Marked as answer by juliusfriedman on 9/17/2015 at 8:38 AM
Sep 17, 2015 at 5:01 PM
Thanks for the suggestions...

I've tried it without disabling Rtcp and there is no difference. I don't think any of my rtp sources use rtcp unless the tools I use are doing something behind the scenes that I don't know about.

My earlier post shows creating the rtp stream with ffmpeg, but in reality I am using gstreamer on a Raspberry Pi. I don't know if you've ever used gstreamer, but here's a simple gstreamer pipeline I use to send test video to my PC:
gst-launch-1.0 videotestsrc ! omxh264enc ! rtph264pay config-interval=1 ! udpsink host=192.168.16.33 port=5000 
This can be successfully consumed by gstreamer, mplayer, etc on the PC. However, I could not make it work with RtpSource. So I wrote my own implementations. The important part was inheriting TransportContext and doing this (note no call to Connect):
            RtpSocket.Bind(rtp);
            RemoteRtp = rtp;
            LocalRtp = RtpSocket.LocalEndPoint;
where rtp is my PC address, port 5000. I found this because there is something elsewhere in the code for this project, it polls whether the socket can read. This was always false, though write was true. I figured read should have been polling as true and that led me to this code. It did fix the problem for this particular stream and I was able to view the RTSP stream successfully.

But then I changed my rtp stream to something more useful, specifically camera input from my Pi using the following, which should essentially be the same kind of rtp stream:
raspivid -rot 270 -g 10 -o - -w 1024 -h 768 -ih -t 0 -n -fps 24 -b 1000000 | gst-launch-1.0 -v fdsrc ! h264parse ! rtph264pay config-interval=1 ! udpsink host=192.168.16.33 port=5000
Unfortunately, even with the change I did for the socket bind, this example still has the original problem. It's almost like it's not receiving any rtp traffic. And I've verified that other tools can consume the rtp stream, so I think the stream is fine. I've tried to determine what is different from this stream versus the test stream and haven't found anything that seemed to matter. That is where I'm currently at.
Coordinator
Sep 17, 2015 at 8:02 PM
Seems like a networking issue of some sort, I would put back the call to Connect and I would use 0 as the port and see if that changed anything. It maybe that the port is different or the source and destination ports are not matching.

You said you also have a Wireshark cap, post a link to that so I can try to understand things a littler better.
Marked as answer by juliusfriedman on 9/17/2015 at 12:02 PM
Sep 17, 2015 at 8:10 PM
I don't see how it would be a networking issue if I can successfully consume rtp streams with other software. However, I did try using connect with port 0 and it doesn't work.

I'm not the original poster. I don't have a wireshark cap.
Sep 17, 2015 at 8:53 PM
I think I'm going to give up on getting this project to work. I've downloaded the live555 project source and modified their proxy server to accept an RTP stream as input (it only accepts RTSP by default) and it's working fine.
Coordinator
Sep 18, 2015 at 12:13 PM
If you have a capture of the software working with other software versus it not working with this project that would probably help to both understand your setup and as well as diagnose the issue.

Since you got it working you should be fine anyway, keep in mind there are several problems with live's RTSP implementation not limited to but including:

Message and packet parsing is flawed and insecure, Rtsp especially however in just about all versions of QuickTime, GStreamer, FFmpeg, Mplayer and Helix they are also... When using TCP this problem is exacerbated by the fact the RTSP spec is poorly written and can be exploited even easier when framing under TCP, add HTTP to the mix and then you have an additional layer to worry about also.

There are connection management issues, this can effect Wifi users as well as users who implement a proxy to tunnel requests in or outside of a LAN. When you start to mix transport mechanisms in session this will also 'byte' you.

There are memory and performance related issues which prevent you from ever actually utilizing your hardware to it's full potential, if you have a fairly decent setup this will come up first when you start to have more then 10,000 clients per stream, or worse at about 1,000 different clients when reading from a file based source along different points in time.

In general, there are several mechanisms which appear to be standard such as the 'REGISTER' command however there is absolutely no support for such beyond the live software itself, one exception of this would be MPEG Transport Streams which can be supported over RTSP and RTP, it seems they appear to be working for other libraries outside of the live suite however there is not really a specification regarding such and as a result no matter how such a feature is implemented there will likely always be some level of fragmentation between implementations regarding such manifested implementations.

In short, This is why I created my own implementation and I have not had any such problems.

You shouldn't have to subclass anything, it's possible something got messed up with those changes but impossible to verify without seeing the changes.

In the latest version I can't seem to replicate such an issue and in such cases this usually indicated system or network configuration issues.

If you need anything further please let me know.
Marked as answer by juliusfriedman on 9/18/2015 at 4:14 AM
Sep 21, 2015 at 6:43 PM
Well, I'm back to looking at this.

I had Live555 working, but performance is absolutely horrible. I know the incoming streams will play fast by themselves, but once they go through Live555 they are unacceptably slow and I see a lot of CPU usage.

I have some stuff to send you if you like. It looks like the contact link on here won't let me send an attachment. How would you like me to send you what I have?

What I have is two test cases. Both sent from the same machine (Raspberry Pi 2) and both received on my Windows 7 PC (Xeon E3-1240, 16GB memory). In each case is a Wireshark recording, a recorded video that I hope will show the problem, mplayer output, and the output from this library.


Details on the successful case:
This is test video from gstreamer. This failed until I made the changes to RtpSocket.

Stream from Pi:
gst-launch-1.0 -v videotestsrc ! omxh264enc ! rtph264pay config-interval=1 pt=96 ! udpsink host=192.168.16.33 port=5000 

Save recorded video from Pi:
gst-launch-1.0 -v videotestsrc ! omxh264enc ! filesink location=success.h264
Details on the failing case:
This is pretty much just changing over to camera input instead of gstreamer test video. Both are h264 encoded video, so I was expecting it to just work, but it doesn't.

Stream from Pi:
raspivid -rot 270 -g 10 -o - -w 320 -h 240 -ih -t 0 -n -fps 24 -b 1000000 | gst-launch-1.0 -v fdsrc ! h264parse ! rtph264pay config-interval=1 pt=96 ! udpsink host=192.168.16.33 port=5000

Save recorded video from Pi:
raspivid -rot 270 -g 10 -o - -w 320 -h 240 -ih -t 0 -n -fps 24 -b 1000000 > failure.h264
I don't know if you've had any experience with gstreamer, but I can explain what it's doing if necessary.

A few other notes:
  • Both wireshark recordings start out with some SSH. That was just me starting the streams on the Pi through SSH.
  • I can play both RTP streams successfully on the PC using gstreamer or using an SDP with mplayer.
  • There are some framerate errors in the stream from the failing test. That's the only thing that sticks out to me. But since mplayer will play the direct RTP stream fine with the framerate error, I don't see how it'd be the problem.
Sep 21, 2015 at 9:32 PM
I'm working through the code for this library and I think I've gotten a little closer to the problem. In RtpFrame.Add(RtpPacket), there is a piece of code at the bottom:
//If the last packet has the marker bit then no more packets can be added unless they are from a lower sequence number
if (count > 0 && seq > HighestSequenceNumber && HasMarker) throw new InvalidOperationException("Complete frame cannot have additional packets added");
What I notice is that it gets to where it always throws this exception.

I might be wrong, but I suspect this library is stuck waiting for packets on the previous frame, but the server is already sending the next frame. Does that sound possible?
Coordinator
Sep 22, 2015 at 4:10 PM
You can enable 'PerPacket' which should work around this issue from a source and be a little more efficient as well but you would lose the 'FrameChanged' events, which also shouldn't matter if you are sending from this library as you would already have the frames. Why you are having this issue is unknown, especially from the receiving end but I know it is not possible to add packets to a completed frame in the real world, I would need some type of capture to understand what is happening and advise properly. To me, it seems you are experiencing reordering along the network route or there is some other issue which is causing this to appear but again without the capture I can't say. Post up what you have and I will take a look when I can.
Marked as answer by juliusfriedman on 9/22/2015 at 8:11 AM
Sep 22, 2015 at 4:17 PM
I think it's because raspivid generates h264 frames without timestamp information. I've been playing these streams with mplayer with no issues, but I think this library doesn't like that. It's doing some checking of timestamp in RtpClient.HandleIncomingRtpPacket and unfortunately the timestamp of all packets is the same.

I'll try the per packet and see what happens. If I still have issues, I'll try to get you what I have. Thanks.
Coordinator
Sep 22, 2015 at 4:24 PM
Yea, that would do it.

You should also add some code to re-timestamp the packets from that source which is quite easy as well, you could even make a small derived class to help with this so that if you encounter such problems with other sources you can easily handle this although 'PerPacket' will probably suit your needs for now.
Marked as answer by juliusfriedman on 9/22/2015 at 8:25 AM
Sep 22, 2015 at 5:33 PM
PerPacket seems to be causing a lot of NullReferenceExceptions in RtpPacket. Apparently there are some RtpPackets being created with a null payload.

In gstreamer there is apparently a v4l2src that will add timestamps. I'm experimenting with that now, but still not having great success.
Sep 22, 2015 at 6:37 PM
I think I may have gotten to a point where I can use this library. Using a timestamped source (v4l2src) was the answer, but not without some strangeness. I'll try to explain...

Test 1:
Stream recorded file. This test worked fine. On the Raspberry Pi, I used gstreamer to record H264 video to a file. After I had the file, I streamed RTP using gstreamer. Then I ran this library to consume that stream and provide it over RTSP. Finally used mplayer to play the RTSP. There was a few seconds of video artifacts at the beginning, but it always seemed to clear up.

Test 2:
Stream live (which is what I really want) using gstreamer. If I played the RTP stream directly with mplayer it worked fine. However, when I used this library, the resulting RTSP stream always had video artifacts for as long as I watched it. That was a step in the right direction because I couldn't even get it to play at all with the non-timestamped video.

Test 3:
I was surprised that I could record a H264 file, stream it later and it worked when a live stream wouldn't. They were both generated with v4l2src. So then I tried something crazy. I decided to decode the v4l2src output to raw, then re-encode it back into H264 and then stream with RTP. I'm not sure what this did, but finally this library could handle the RTP stream correctly.

I really think Test 2 should have been successful since the RTP stream plays fine by itself, but I suppose if I have to use the method I used in Test 3, I'll do that. I do have a wireshark capture from Test 2. I'll send you a link through the contact functionality on here.
Sep 23, 2015 at 12:03 AM
A little bit off topic, but there seems to be some buggy code related to the RtspServer. What I was noticing is that if I connected and then disconnected to the RTSP stream using MPlayer a handful of times it would get to where Mplayer would send a response (usually DESCRIBE or SETUP) and would not get a response from the RtspServer. I tracked this down to all the BeginSendTo calls in the code. I think the code was assuming that everything would finish asynchronously, but that's not the case. I had to add handlers immediately after BeginSendTo to process requests that ended synchronously. And so far it seems to be working flawlessly.
Coordinator
Sep 23, 2015 at 5:16 PM
With PerPacket enabled the packets are not re-ordered or checked for the 'Marker' bit to be set.

Sometimes as a result artifacts might be related to the decoder and gaps in the data related to time difference of the received packets, sometimes the buffering needs to be set higher for the artifacts to be removed. The reason why this is not a problem in the recorded stream is because there are less gaps in the data at decoding time.

I am not sure what BeginSend calls you refer to, give a line number or method so I can check into it.

I will check out the captures when I can and let you know what I find.

Also, make sure you are using the latest version of the code.
Marked as answer by juliusfriedman on 9/23/2015 at 9:16 AM
Sep 23, 2015 at 6:25 PM
As I recall, I couldn't get anywhere with PerPacket set to true. Artifacts are with it set to false.

I'm using the latest code as of last week.

There are a few BeginSend (or BeginSendTo) calls. For instance, there is ClientSession.SendRtspData. I think this was the most troublesome one for me, though I ended up changing all of them. Unfortunately after my changes, there was one time that I ended up with a StackOverflowException that I think may have been related to these changes. I can't seem to reproduce it though. Every other time it has worked flawlessly except I'm still having occasional hiccups with MPlayer saying I have reached end of stream. I suspect that doesn't have anything to do with the RTSP part of it. I'm guessing the RTP stream being sent by the RTSP server is interrupted. Just haven't found out why, but I'm digging deep into the code this morning to see if I can find it.

I don't know whether you'll be able to reproduce the BeginSend(To) problems. For me, all I needed was multiple connect/disconnects from MPlayer. I would start playing the rtsp stream with mplayer. Once the video started, I'd hit Ctrl-C. Then I'd start again. Seemed like it would work fine 3-5 times and then it would get stuck.
Coordinator
Sep 24, 2015 at 3:53 PM
Edited Sep 24, 2015 at 3:59 PM
I looked at the capture and it seems you are dropping some packets, you might want to try increasing the 'ReceiveBufferSize' of the socket being used for reception to help with the artifacts.

I'm interested in what changes you needed to make in 'SendRtspData', the function doesn't assume anything related to synchronicity, in fact the function actually waits synchronously.

For the sake of ensuring this is not the problem I have posted a slightly changed method below:
public void SendRtspData(byte[] data, int offset, int length, SocketFlags flags = SocketFlags.None, EndPoint other = null)
        {
            if (data == null || length == 0 && SharesSocket) return;

            try
            {

                long pollTime = (int)Media.Common.Extensions.TimeSpan.TimeSpanExtensions.MicrosecondsPerMillisecond;// (int)(m_RtspSocket.ReceiveTimeout * Media.Common.Extensions.TimeSpan.TimeSpanExtensions.MicrosecondsPerMillisecond);// Media.Common.Extensions.NetworkInterface.NetworkInterfaceExtensions.GetInterframeGapMicroSeconds(Media.Common.Extensions.Socket.SocketExtensions.GetNetworkInterface(m_RtspSocket));

                //while the socket cannot write in bit time
                while (false == IsDisposed && false == m_RtspSocket.Poll((int)pollTime, SelectMode.SelectWrite)) 
                {
                    ////Wait for the last send to complete
                    if (LastSend != null)
                    {
                        if (false == LastSend.IsCompleted)
                        {

                            WaitHandle wait = LastSend.AsyncWaitHandle;

                            Media.Common.Extensions.WaitHandle.WaitHandleExtensions.TryWaitOnHandleAndDispose(ref wait);
                        }
                    }
                }

                //If session is disposed then return
                if (IsDisposed) return;

                //Assign the buffer
                m_SendBuffer = data;

                //The state is this session.
                LastSend = m_RtspSocket.BeginSendTo(m_SendBuffer, offset, length, flags, other ?? RemoteEndPoint, m_Server.ProcessSendComplete, this);

                //Mark as not disconnected.
                IsDisconnected = false;
            }
            catch (Exception ex)
            {
                //Log the excetpion
                m_Server.Logger.LogException(ex);

                //if a socket exception occured then handle it.
                if (ex is SocketException) m_Server.HandleClientSocketException((SocketException)ex, this);

            }
        }
Use that method, the only real change in logic is that I change IsDisconnected only after the call succeeds which should fix the issues you were having but I somehow feel though that it was better set before making the call. I will check my notes as to why I changed it but I think that it had something to do with preventing the timeout of the session when a response couldn't be sent due to the socket being in use for other traffic.

Aside from that you shouldn't have to change anything in the code, there is nothing hard-coded and the logic is tested to work under any condition which can be configured on the underlying socket(s).

In relation to being able to reproduce a bug, MPlayer has issues with missing responses which come in short succession of each other because of how the polling in their code works combined with how they send requests which probe the server type, E.g. to determine if the server is a Real RTSP Server or otherwise the library sends two requests initially, if you cancel this process and start another one before the response is received the library can dead lock. I submitted a bug for this with MPlayer, and a separate bug for libav which had that problem as well as an injection problem in the Rtsp parser.

MPlayer acknowledged that their Rtsp client was buggy and to use libav or another, libav reacted by changing their reception code as well as their parsing code.

Both libraries still still send probe requests which can result in such a deadlock if used as described above...

There are some other updates for the code which I will be synchronizing eventually but nothing really which would be needed for complete operation at this time, it's more of a phase for allowing development in a more segregated manner.

-Julius
Marked as answer by juliusfriedman on 9/24/2015 at 7:54 AM
Sep 24, 2015 at 6:18 PM
The asynchronous problem isn't really with SendRtspData, but with the callback code it uses. The change I made was to the m_Server.ProcessSendComplete. At the beginning of that method, it checks to see if the BeginSendTo ended synchronously. If it did, it returns. Most of the time it ends asynchronously and proceeds to the rest of the code in ProcessSendComplete as expected. But I was seeing a few times where it did end synchronously and that seemed to coincide with when the RTSP commands would stop working. What I did was to split this into two methods.

The ProcessSendComplete method now looks like this:
        internal void ProcessSendComplete(IAsyncResult ar)
        {
            if (ar == null || ar.CompletedSynchronously || !ar.IsCompleted)
                return;

            ProcessSendCompleteResult(ar);
        }
And then everything else that was in ProcessSendComplete is now in ProcessSendCompleteResult. And then going back to the SendRtspData method, I have the following code after the BeginSendTo call...
        if (LastSend.CompletedSynchronously)
        {
            RtspServer.ProcessSendCompleteResult(LastSend);
        }
I've made similar changes everywhere BeginSendTo was being used. And looking at it now, I probably didn't need to split out ProcessSendComplete. I probably could have just removed the ar.CompletedSynchronously check. And I'll be honest, I don't usually write networking code, so I'm confused why the CompletedSychronously check would have been in there. But I do know my changes seemed to fix the problem.

Just curious if you were you able to look at the problem I had with the RtpSocket bind/connect problems I had earlier? I think it's probably due to how my RTP source is set up and not my network. Specifically I think it's related to the Pi sending RTP to a port on my PC and then having RtpSocket make a Connect to this, versus making a Connect to a remote host.

Regardless, the changes I've made have gotten me almost to the point of having it usable, so I'll probably just keep them in the code. I say almost usable because I'm still having a couple problems that may or may not be MPlayers fault...

Problem 1 is that MPlayer will stream fine. Nice and smooth playback. No lag. Exactly what I want. But I'm trying to show the video in a separate application using the -wid option. When I was utilizing straight RTP this was working perfectly fine. Now that I'm using the RTSP stream it has some serious lag. So I did a test by running one with -wid and one without at the same time. The one with -wid was laggy and had virtually no CPU usage. The one without -wid played fine and was taking up an entire core worth of CPU usage. I guess that proves that this library is working fine, but something about RTSP in MPlayer with -wid is messed up.

Problem 2 (maybe related to problem 1) is that MPlayer will start to play the stream and then about half the time will stop. Usually within the first couple seconds. I believe this has only ever happened when running with -wid. The MPlayer logs suggest end of stream was reached. I did a packet capture and I'm seeing that RTP traffic is being sent to MPlayer and then all of a sudden Mplayer responds with a TEARDOWN. I thought there must be something in those last few RTP packets that triggered it. If I was understanding the output correctly, it seemed like the earlier RTP packets would see an increase of 1 in sequence number and there'd be a few non-marker packets and then a marker. Then right before the TEARDOWN I saw a jump of 22 in sequence number. Also of interest is that the last few packets were all markers. I'm guessing some RTP packets are missing and MPlayer doesn't like that. I'm not yet sure if the packets are missing between the RTSP server and MPlayer or between my Pi and the RtpSource I'm using. I know MPlayer will play the RTP stream from the Pi with no problems though.

And a note on the networking... With the exception of the RTP stream that the Pi sends to my PC, everything else is running on the PC. If I understand the networking correctly, nothing actually goes out over the network adapter so I can probably ignore networking issues everywhere except for that connection to the Pi.

I will send you a link to the capture I was looking at for my second problem. It's probably useful to use the following display filter:
tcp.port eq 20270 or udp.port eq 30014 or udp.port eq 30015 or udp.port eq 49378 or udp.port eq 49379
Sep 24, 2015 at 7:01 PM
Also I'm going to try to set up a test application to send you to show the problems, but it might be awhile before I get it done. I think that's probably the best way I can help show the problems.
Coordinator
Sep 24, 2015 at 7:03 PM
The checks for 'CompletedSynchronously' were present because the handler code would have possibly already been called, subsequent requests or responses would then restart the server thread checking for data unless the transport has switched to TCP.

If it helps that's great, I will double check that those checks need to be in there but if that helps your issue then thats great. Please also be aware this might be a Framework related issue.

Looking at the capture the first thing I see is that the Uri used in the SETUP request is weird, 'SETUP rtsp://192.168.16.33/live/6f2349a4-ce7d-4a77-9e35-f4c433e5d8d4//live/6f2349a4-ce7d-4a77-9e35-f4c433e5d8d4/video RTSP/1.0'

That probably doesn't matter but I wanted to point that out, I am wondering if that is related to your changes or otherwise.

I see a bunch of dropped packets, I am also seeing duplicate packets, it's hard to say what the cause of this is besides networking configuration or resources on the receiver being maxed out.

I also see two streams were running and some of those packets may have been related to the other stream. I do see the Teardown but I think that is more or less related to packets not being received within a certain amount of time and not as a result of the sequence number.

Try increasing the send buffer size from mplayer and the receive buffer in the application.

In relation to the Connect stuff, there is a 'FromSessionDescription' function which accepts a socket already configured, you should be able to configure your sockets as required and use that socket through that function. You shouldn't have to modify the code as I have allowed for a variety of configuration options as well as the ability to configure the entire socket yourself.

If you need anything else let me know.
Marked as answer by juliusfriedman on 9/24/2015 at 11:03 AM
Sep 24, 2015 at 7:51 PM
I don't believe I've changed anything that would have affected the SETUP uri.

I think I did have two video streams running at the same time, but the display filter should have filtered those out so that only the RTSP, RTP, and RTCP traffic for the problematic stream would be shown.
Sep 24, 2015 at 8:05 PM
I noticed there was a way to pass in an existing socket. For some reason I thought that wouldn't work, but I will revisit that when I have time. Thanks for the suggestion.
Coordinator
Sep 25, 2015 at 1:24 PM
Edited Sep 25, 2015 at 1:26 PM
Indeed it will, although some constructors might need to be added.

For example try these if you need to.
 public RtspSource(string name, Uri source, bool perPacket, RtspClient client)
            : base(name, source, perPacket)
        {
            if (client == null) throw new ArgumentNullException("client");

            RtspClient.RtspSocket = client.RtspSocket;
        }


 public RtpSink(string name, Uri source, Rtp.RtpClient client)
            : this(name, source)
        {
            if (client == null) throw new ArgumentNullException("client");
            RtpClient = client;
        }

 public RtpSource(string name, Uri source, Rtp.RtpClient client)
            : base(name, source, client)
        {

        }
A source can then be created using the From methods which also allow you to configure the socket if required.

Please also see the 'ConfigureSocket' Action which can be further modified to allow the exact configuration you may require.
Marked as answer by juliusfriedman on 9/25/2015 at 5:24 AM
Sep 28, 2015 at 6:03 PM
Wanted to let you know I tried changing the ReceiveBufferSize and that seemed to fix it. I suspect my re-encoding of the video resulted in a more compressed stream so that's why it appeared to work.
Marked as answer by juliusfriedman on 9/28/2015 at 10:39 AM
Sep 29, 2015 at 9:45 PM
After I sent you the test application, I tried some more testing on why mplayer was sending teardown. I thought it was audio, but apparently not. Without the audio stream it would run longer, but would eventually stop. So I decided to revisit the PerPacket=true setting. Unfortunately I'm still having similar issues, although I think the playback performance is better.

I was going to mention that I think PerPacket has a bug that causes it not to work at all. As the packets come in, they get added to m_OutgoingRtpPackets via the RtpClient.EnqueuePacket method. And at some point, RtpClient.SendReceieve will send these out if they aren't disposed. Unfortunately most of them are disposed. If you track the EnqueuePacket call back to where the RtpPacket is first created, you'll see it's created via a using statement in RtpClient.ParseAndCompleteData. It appears most of the time this using statement exits and disposes the packet before SendReceieve can send it out. I've fixed this by enqueuing a clone of the packet, though I haven't yet verified if the clone will ever get disposed. That made it work, but like I said, it still won't keep playing.

It doesn't seem to matter if I'm doing per packet, or per frame it will play ok for up to a minute or two and then stop. Most of the time I can start another MPlayer process and it'll work for another minute or two. Obviously that's not so great for the end user. If I play the RTP stream by itself, I have no interruptions.

When I was using per frame, I had added some logging code to show when I had missing sequence numbers. I did this in ClientSession.OnSourceFrameChanged with the following code:
            var missing = Enumerable.Range(frame.Min(p => p.SequenceNumber), frame.Max(p => p.SequenceNumber) - frame.Min(p => p.SequenceNumber))
                .Where(p => frame.All(t => t.SequenceNumber != p))
                .ToList();

            if (missing.Any())
            {
                Console.WriteLine("Frame was missing packets: " + missing.Count + "/" + frame.Count);
            }
It will run for awhile no problem. Then I might see one or two of these messages printed out that suggest a few sequences were missing. And then it seems right before MPlayer stops playing I will see around 65000 sequences missing. I think that's a little strange because I didn't think a frame would have that many sequences. Most frames seem to have only a handful. I don't know if this is the cause of my problems or a symptom, but I suspect it's related.
Sep 29, 2015 at 10:52 PM
After some reading on the rtp protocol I see why I'm getting around 65000 missing. The sequence number wraps around. But still, it is in this wraparound situation that it usually kills my stream. I'm going to continue to look into this.
Coordinator
Sep 30, 2015 at 12:22 PM
I think the 2 minute issue is related to the session timeout on the server which defaults to 60 seconds.

MPlayer should send a keep alive or you need to disable session timeout at the server.

Sequence number wrapping seems to be handled appropriately, I will check into per packet and see if that causes issues but it's there as more of a work around for streams which don't set markers or timestamps appropriately.

That dispose call is appropriate because it occurs after the packet is handled already, also
In relation to the packet being disposed, it seems like a memory issue, I definitely wouldn't clone the packets because that's going to use twice the memory.

I'll check out the example application soon.
Marked as answer by juliusfriedman on 9/30/2015 at 4:22 AM
Sep 30, 2015 at 4:33 PM
I'll look into the timeout stuff, but with my recent changes I have gotten up to the point where it might play 10 or 15 minutes before stopping, so I'm not sure it's that timeout setting.

I did make a change in RtpFrame.IsMissingPackets. I don't think the original code was handling the sequence number wrapping correctly. I think it was relying on HighestSequenceNumber being the last in the sequence, but that's not the case when it wraps.

I'll look at a different option than cloning, but I'm positive these packets are being disposed before the server has a chance to send them out to a client. I've put logging statements in and 95%+ of the packets enqueued into m_OutgoingRtpPackets are disposed by the time RtpClient.SendReceive is ready to send them out.

If I understand the code correctly, there is an RtpClient for the source stream and it enqueues a packet into an RtpClient for each of the destination client(s). After enqueing the packet, the using statement finishes and the packet is disposed because it is considered "handled" at that point. Unfortunately it really isn't handled. The destination RtpClients run on their own threads and may or may not have been able to process the packets in their m_OutgoingRtpPackets buffer before the source RtpClient disposed of them. In other words, it's a race condition. Does that make sense?
Coordinator
Oct 1, 2015 at 12:48 PM
I will double check the sequence number wrapping, I should have an ExtendedHighestSequenceNumber property for that.

The packet being disposed is part of the logic, the packet is created an event is fired, all consumers of the event have access to the packet, the packet is disposed.

If the packet is disposed when it's time to be sent out there must be newer packets in the buffer waiting to be sent, this might be occurring because of the amount of marker packets in the audio stream or video stream or lack thereof.

Try first increasing the SenfBufferSize so more write calls will succeed.

Then check out in SendRecieve how Rtp packets are sent, the packet should still be alive there because it's not disposed until after the event is handled.
Marked as answer by juliusfriedman on 10/1/2015 at 4:48 AM
Oct 1, 2015 at 3:35 PM
Yes, the event is handled before the packet is disposed. However, the packet is put into a buffer that is only processed at some indeterminate time after. This could be long after the event is handled and the packet is disposed.
Coordinator
Oct 1, 2015 at 6:34 PM
Edited Oct 1, 2015 at 6:50 PM
There definitely was some issues when sequence numbers wrapped, I have updated the RtpFrame.cs class, I have posted a revision under Patches.

https://net7mma.codeplex.com/SourceControl/list/patches

https://net7mma.codeplex.com/workitem/17553

In relation to the disposition, I can say that the source's thread is running out of time to send the packet, this is possibly related to the SendBufferSize, try increasing that so that when the event occurs the packet can be written to the kernel buffer accordingly, without such the packet instance is only made to stay in memory until the event is completed, you can easily change this behavior by overriding the source and determine how much memory you should use to keep packets in memory however this will cause you more trouble than necessary and also causes a vulnerability in some cases where a re-transmit request is sent for a packet which was just removed from that buffer.

The latter is more advanced concept but is valid for security reasons, especially when you consider implementing some type of Negative Acknowledgement (which is found in some Rtcp profiles), the logic can also be justified with the following explination:

If the packet did not make it to the client, then there MUST be a newer packet which is already in the buffer, this packet should have precedence over the packet which is already disposed; in essence the packet is skipped because there are newer packets already ready to be sent, continually trying to send such a packet would cause discontinuity because the packet is already late and there more data which can probably be used to recover for the missing packet anyway. You definitely don't want someone to be able to request packets which are too far back in the stream or continuously as the boundary of where you are cycling packets out.

Another possibly easier way to visualize:

If you have 1000 clients and the packet has been sent to 500 clients before the packet was disposed then the next packet will be sent to the latter 500 clients, yes 500 clients miss a packet but you also save 500 * packetSize bytes of memory.

You can adapt for this on the source by creating a derivative of RtpSource or RtpClient, having packets which are received cached in the derivative and then spawning a new event for the packet which has been added to the cache, the cache will be emptied when all consumers of the packet have successfully aggregated it.

This will obviously cost more memory and should be handled accordingly, see the PacketBuffer in the ClientSession for a little insight.
Marked as answer by juliusfriedman on 10/1/2015 at 10:34 AM
Oct 1, 2015 at 7:51 PM
I guess I must not have described the problem well enough. This is specifically with PerPacket=true and has nothing to do with the source. The Source RtpClient is writing the packets to the Destination RtpClient(s). It's those RtpClients that don't have a chance to send the packet out to the video player. I will try to add a test case to my testing app to show it.
Oct 1, 2015 at 8:59 PM
Also I should note that after putting in some more logging statements, I saw I was having some issues with multiple threads accessing m_OutgoingRtpPackets. I have since added lock statements around every access to this list. I've also added lock statements to the RtspServerConsoleLogger methods as they obviously needed them (you could tell when you'd get weird mixes of console colors part way through writing a log message).

The lock statements seemed to help a bit. I went from an average of around 5 minutes before the video fails to around 15 minutes. I suspect maybe some more issues with incorrect locking, but I'm not sure yet.
Oct 1, 2015 at 9:42 PM
I'm having better luck with increasing the ReceiveBufferSize even larger than I had before. It's currently up to 65536.
Coordinator
Oct 1, 2015 at 9:55 PM
The lock will only decrease performance,in this case the contention is done at the method level with attributes.

I'll check into replication of the problems you are describing when I have more time and see if I come up with anything.

In your case you should see why gstreamer exits and diagnose further from there.
Marked as answer by juliusfriedman on 10/1/2015 at 1:55 PM
Oct 1, 2015 at 10:06 PM
gstreamer is not exiting. Mplayer is exiting and it appears because the rtsp server isn't sending it packets. I guess I don't explain well enough...

This works fine:
Gstreamer RTP stream -> Mplayer

This fails after a certain amount of time (sometimes right away, but usually within 15 minutes):
Gstreamer RTP stream -> Managed Media Aggregation Rtsp Server -> Mplayer

So I can only think there are bugs somewhere in the MMA code.
Coordinator
Oct 6, 2015 at 3:45 PM
Well I hope to have some free time later in the week, I will attempt to replicate the issue and let you know what I find.
Marked as answer by juliusfriedman on 10/6/2015 at 7:45 AM
Oct 6, 2015 at 5:02 PM
Thanks.

I've also been trying to test solutions, but haven't found the problem. What I know is that with live camera streams from a couple of Pis, eventually one of them will close MPlayer. And like I said, it's usually within 15 minutes but seems to vary a bit.

I added some logging to the RtpClient whenever it receives or sends rtp or rtcp. What I'm finding is that the RtpSource RtpClient eventually will stop part way through rtp packet receiving. And actually it's not even calling the ReceiveData method because the RtpSocket poll is showing that it's not readable. But it is writable. I had some logging statements prior to receiving rtcp, but those don't get called, so I'm assuming an exception is happening after the poll call and before it starts handling rtcp.

Trying to get my test application to reproduce the problem has been difficult. I've been running two of the gstreamer videotestsrc streams for the last 45 minutes and haven't had an issue yet. However, I ran this same test on my laptop at home and it did show a problem after about 6 minutes. So I suppose it may have something to do with my work PC being faster. But my laptop is a core i5, 8gb ram and I was running two low bandwidth streams. I would have expected that to be fine. And indeed, playing the two rtp streams directly didn't cause a problem. It was only the rtsp streams.

One thing I noticed on my laptop is that it'd play smoothly for 5 or 10 seconds, and then freeze for a second. I was also displaying a timeoverlay on the video and the test application shows the time. I noticed some drift. I'm guessing some buffer somewhere is getting more and more full until it runs out of space, but I haven't been able to find it. With what I'm seeing in both the test application and my actual application combined with the packet captures I've been looking at, I'm pretty sure the problem is with the RtpSource receiving data. I thought it would be the ReceiveBufferSize, but even large values for that property didn't seem to do much.

One thing that had me concerned is that when I start up the rtsp server, it uses very little cpu and mplayer streams fine. But eventually it will use up a decent amount of cpu. Like around 30%. I'm not sure what happens to make it start using cpu. I will probably try to track that down with my profiler.
Oct 6, 2015 at 5:18 PM
I can send another packet capture if necessary, but here's the interesting section (shortly before getting TEARDOWN):
No.     Time        Source                Destination           Protocol Length Info
     64 0.123012    192.168.16.33         192.168.16.33         UDP      1428   Source port: 30006  Destination port: 62198
     65 0.123012    10.64.40.11           192.168.16.33         UDP      45     Source port: 37983  Destination port: 20010
     66 0.123012    192.168.16.33         192.168.16.33         UDP      1382   Source port: 30006  Destination port: 62198
     67 0.123012    192.168.16.33         192.168.16.33         UDP      45     Source port: 63782  Destination port: 40010
     68 0.124012    192.168.16.33         192.168.16.33         UDP      45     Source port: 30006  Destination port: 62198
     69 0.125012    10.64.40.11           192.168.16.33         UDP      1428   Source port: 37983  Destination port: 20010
     70 0.125012    10.64.40.11           192.168.16.33         UDP      1428   Source port: 37983  Destination port: 20010
     71 0.125012    192.168.16.33         192.168.16.33         UDP      1428   Source port: 63782  Destination port: 40010
     72 0.125012    192.168.16.33         192.168.16.33         UDP      1428   Source port: 63782  Destination port: 40010
     73 0.125012    10.64.40.11           192.168.16.33         UDP      1428   Source port: 37983  Destination port: 20010
     74 0.125012    192.168.16.33         192.168.16.33         UDP      1428   Source port: 63782  Destination port: 40010
     75 0.125012    10.64.40.11           192.168.16.33         UDP      1428   Source port: 37983  Destination port: 20010
     76 0.126012    192.168.16.33         10.64.40.11           UDP      426    Source port: 63784  Destination port: 20010
     77 0.126012    192.168.16.33         192.168.16.33         UDP      1428   Source port: 63782  Destination port: 40010
     78 0.126012    10.64.40.11           192.168.16.33         UDP      1353   Source port: 37983  Destination port: 20010
     79 0.126012    192.168.16.33         192.168.16.33         UDP      1353   Source port: 63782  Destination port: 40010
     80 0.126012    10.64.40.11           192.168.16.33         UDP      55     Source port: 37983  Destination port: 20010
I should explain the ports...
  • Pi (10.64.40.11) sends rtp to PC (192.168.16.33) port 20010
  • Gstreamer on PC reads these rtp. It tees. One branch of the tee records the video to a file. The other branch forwards the packets on. One forward goes to another Pi. The other loops back to the PC on a different port (40010).
  • The RtpSource reads from port 40010 and sends out to whatever destinations are watching the stream. In my case, I have MPlayer running on my PC so the IP address is the same and it picked port 62198 to use.
So the flow is:

Pi Camera -> Gstreamer on Pi -> Gstreamer on PC -> Rtsp Server -> MPlayer

So in the packet capture, it shows packets coming into port 20010. Then those packets sent to port 40010. And then finally, those packets sent to 62198. But then it gets to a point where packets are being sent to port 40010 and I'm guessing the RtpSource is not reading them because you never see anything being sent to 62198. And after awhile, you see the TEARDOWN.
Oct 6, 2015 at 6:41 PM
Ignore my comments about the logging statements prior to rtcp processing. I think it just happened to be that the teardown happened at that point and I was displaying the log at that point. I think the RtpSource is continuing to process after that, though I think there is a significant period where the socket is writable but not readable. Seems to be about 10 seconds. I suspect 10 seconds of the RtpSource not being able to read data from the socket would cause some issues.
Coordinator
Oct 6, 2015 at 7:07 PM
What I can say immediately is that you should try making a RtpSource directly from the gstreamer and then using another gstreamer instance to consume from the RtpSource or just consume from the same gstreamer source as the RtpSource is already using.

You could also have the PI create the video.

What I'm interested in is if there is a bug which is preventing the server from functioning as designed.
Marked as answer by juliusfriedman on 10/6/2015 at 11:07 AM
Oct 6, 2015 at 7:20 PM
If I understand, you are asking me to try this:

RTP video stream created with GStreamer -> Consume and display RTP video stream with a second GStreamer process.

Prior to trying to provide the video through RTSP, I've done lots of testing with the above method and it has always worked fine. I've seen no evidence that anything is wrong with the RTP stream from GStreamer or with consuming an RTP stream using either GStreamer or MPlayer or with the network.

I will try again to update the test app to show the problem exists.
Oct 7, 2015 at 11:50 PM
On a hunch, I decided to completely rewrite the RtpClient receive code. Got rid of all the polling and changed the code to use BeginReceive. It appears to have gotten better, though not perfect yet. Seems to get around 30 minutes of playback before failing. And quite often the failing happens at the same time I get a ConnectionReset on the RTSP socket, so possibly my problem is on the sending side now.
Coordinator
Oct 8, 2015 at 5:02 PM
Edited Oct 8, 2015 at 5:08 PM
Well, I tried to tell you that originally :)

Your issue seems specific to your setup and also your changes are probably only relevant there also, by increasing the send buffer and the receive buffer you are really already doing what Begin will already do for you as they are already isolated to a new thread.

You may experience better performance by totally isolating events on the RtpClient to a new thread and only handling them there.

Check out this thread for help on the ConnectionReset if your using Windows.

http://stackoverflow.com/questions/10332630/connection-reset-on-receiving-packet-in-udp-server

There is also an AutomaticallyReconnect option on the RtspClient which can reconnect but when a ConnectionReset happens on TCP your usually not going to get any more data without a re-connect.
Marked as answer by juliusfriedman on 10/8/2015 at 9:02 AM
Oct 8, 2015 at 5:19 PM
Found another bug. I was having some problems with the RtspServer not handling requests maybe 20% of the time and it would eventually throw an exception that the thread was dead and priority could not be accessed. This happened when the maintainer was trying to set priority using the m_ServerThread priority.

So I looked at the code that creates the server thread...
            m_ServerThread = new Thread(new ThreadStart(RecieveLoop));
            
            //Configure the thread
            m_ServerThread.Name = ServerName + "@" + m_ServerPort;
            m_ServerThread.TrySetApartmentState(ApartmentState.MTA);
            m_ServerThread.Priority = ThreadPriority.BelowNormal;

            //Start it
            m_ServerThread.Start();

            //Indicate when start was finished.
            m_Started = DateTime.UtcNow;
The m_Started needs to be moved prior to m_ServerThread.Start();. The reason is because RecieveLoop only runs if IsRunning property is true. Unfortunately, it's not true until after m_Started is set. And what is happening is that sometimes m_ServerThread starts before m_Started gets set and so it exits.
Marked as answer by juliusfriedman on 10/8/2015 at 9:28 AM
Coordinator
Oct 8, 2015 at 5:27 PM
Edited Oct 8, 2015 at 5:29 PM
I added some updates for the RtpClient and RtspMessage class.

https://net7mma.codeplex.com/SourceControl/list/patches

You will also have to make a change in ClientSession to prevent the Context lookup, this should also help you with performance.

Your right about that bug with Start and that stats should be set to 0 before starting the other thread.

This happens more often if there are no streams to be started.
Marked as answer by juliusfriedman on 10/8/2015 at 9:31 AM
Oct 8, 2015 at 5:30 PM
juliusfriedman wrote:
Well, I tried to tell you that originally :)
I'm confused. What part were you trying to tell me originally? If it was the advice on increasing buffer sizes, that has not fixed the problem. It only seemed to fix the problem with screen artifacts.
Oct 8, 2015 at 5:54 PM
I'm testing the SIO_UDP_CONNRESET socket fix from the answer in the link you posted. So far so good, though I will want to test awhile longer to see if that fixes the problem. I've applied this to the RtpSocket, though the connection reset I was seeing had to do with the Rtsp socket.

I'm a little curious though... the stackoverflow answer said "This happens with UDP sockets." If this fixes my problem, doesn't that imply that the RtpSource is going to have this problem with any stream (not just my setup)?
Coordinator
Oct 8, 2015 at 6:18 PM
Edited Oct 8, 2015 at 6:26 PM
That your having a networking issue or cpu / memory issue or some other type of systems configuration issue.

The fact that your have increased the time to 30 minutes beyond 15 minutes just indicates you were able to allocate more resources to circumvent the problem for some amount of time.

The changes in Connect are not required for everyone, they are simply working for you because you have configured your system to require them through how the architecture receives the RtpPackets.

The only arguable point is that if you never want to send then you don't need a call to connect especially under UDP although I don't see why this would cause a problem because calling Connect only establishes a port for where packets will go if they are sent, as UDP doesn't have a notion of a Connection beyond that given by it's EndPoint.

You should be doing a root cause analysis to determine where the packets are being disposed, the best way to do this would be to output some statement at the dispose call, possibly also walk the stack to see which methods invoked dispose.

I have seen some issues where the lookup of the TransportContext is taking a bit longer either due to one being added or removed or otherwise, this lookup can be avoided which also decreased the time taken to handle events.

In such cases I would bet that it's possible for the RtpPacket to become disposed because of an additional lookup for the Context which isn't required since it can be passed to the event handler.

As you will probably see this will give you just about the same performance as isolating the events to a new thread unless the handler of the event is taking a long time to process. In such cases the handler should thread his handling code.

I should also remind you that using BeginSend doesn't solve the problem, it only causes the overhead to shift to other parts of the program...

Lastly, packets being copied is another poor example of a modification to fix the issue

Packets are cloned when FrameChangeEvents are enabled, this prevents the need for packets to be copied on the individual packet events and thus why I suggested PerPacket to reduce memory usage.

From this point you need to determine where the cycles are being used, with profiling or otherwise and then adjust the code if requires which I doubt it will, especially when you apply the fixes given in the Patched version.

Definitely let me know your results after applying the listed patches.
Marked as answer by juliusfriedman on 10/8/2015 at 10:18 AM
Coordinator
Oct 8, 2015 at 6:24 PM
Edited Oct 8, 2015 at 6:37 PM
Actually no, it's probably related to some type of configuration issue or networking issue.

It will then depend on the networking stack in the operating system, your most likely using Windows which is where that problem shows up and only in certain situations.

RtpSource can be given a configured RtpClient which prevents the need to modify the RtpClient code itself.

Lastly, You also shouldn't have to deal with Rtsp if your going Rtp -> Rtp, are you using Rtsp over UDP? If so the connection reset fix will also fix the issue, if not then you will need to determine why the Tcp socket is being reset, if your using Rtp -> Rtp where does a Rtsp or Tcp socket come into the equation?

Please also remember when your experience this ConnectionReset exception under UDP, it's usually the networking stack saying that data was unable to sent or received by your application, this usually indicates the socket is not listening at all or on the correct port, it could also indicate that the buffer for the operation by the application is full. It is also usually accompanied by a ICMP packet with the type set to Service Not Available.

In short, ignoring the exception usually doesn't fix the issue unless it's related to the environment.
Marked as answer by juliusfriedman on 10/8/2015 at 10:24 AM
Oct 8, 2015 at 7:00 PM
I'll try to respond to all your points...

I'm not yet convinced it's a cpu/networking issue. I did see sometimes where mma would be near 0 cpu, but other times it would go up to 30+%. I haven't noticed any high cpu usage after I made all the BeginReceive changes.

On the network side, I've looked at packet captures enough to know that there isn't a problem with the rtp stream coming from the pi. On the pc, gstreamer receives these packets and forwards to another port which the packet capture suggests is also working fine. Plus, my application had been using this way of viewing perfectly fine, but the requirements changed to where I needed multiple clients to be able to view, which is why I decided to try to use rtsp. At some point, these packets are either not received by the RtpSource RtpClient or are not sent out by the ClientSession RtpClients. And I'm pretty sure it's that they aren't received because when I have multiple MPlayer instances viewing the same stream, they all die at the same time. Because the gstreamer process, the rtsp server process, and all the MPlayer processes are running on the localhost I believe Windows completely bypasses the network, so that suggests to me there isn't a networking problem.

I suspect my changes to use bind only (and not connect) were due to the way Windows 7 is handling a connection to the localhost. I believe the mma code was only handling connections to a remote rtp stream, not an incoming rtp stream. Regardless, I have this working so I'm not worried about it. As a side note, I had originally tried using 127.0.0.1 in the connection line of my sdp, but that actually caused an exception.

There's been so many issues that maybe we aren't talking about the same thing in regards to the disposed packets. My current problem of MPlayer dying after 30 minutes is not due to disposed packets. I don't think the socket even receives the packets in the first place. The dispose problem I have fixed. Originally I had chosen to clone the packets, but since that's not a great solution I've done something different. I've removed the using block that was disposing them and added a finalizer to RtpPacket which disposes. So when the final ClientSession RtpClient thread has a chance to process the packet there should be no more references and it should get disposed at that point. I don't know if I've convinced you this was a bug, but there should have been a "Test PerPacket" test case in the last version of the test app I sent that should prove there is. But I should mention, the "Test PerPacket (Fixed)" case has the cloned packet fix, not the finalizer fix.

I will attempt to integrate your changes, but I've made so many modifications to get it to work that it may take some time to integrate.

I am not using UDP for Rtsp. When you say Rtp -> Rtp do you mean not using RTSP at all? No, I'm definitely using RTSP and over TCP.

I believe I may have gotten a test case working for this last problem with MPlayer dying. I have two test cases. Each test case shows two video streams. One stream might have been enough, but my actual application is using two so I thought it best to use two streams in the tests...


Test MPlayer (RTP):
  • Gstreamer videotestsrc to Port1 -> Gstreamer forward Port1 to Port2 -> MPlayer plays off Port2
  • I ran this test for probably 30+ minutes and it never stopped. I was seeing very small freezes once in awhile, but the important part is that it never stopped.
Test MPlayer Failure:
  • Gstreamer videotestsrc to Port1 -> Gstreamer forward Port1 to Port2 -> RtspServer with RtpSource binded to Port2 -> MPlayer plays off rtsp url.
  • On my PC (which is fairly decent hardware), it will run for a few minutes and then one stream stops. Awhile later the second stream stops.
  • This also had the small freezes
I will package up my newest version of the test app to send to you. I'll let you know when it's ready.
Oct 8, 2015 at 7:03 PM
I sent you the link to the latest test app. I would be interested to know whether all the tests fail for you like they do for me.
Coordinator
Oct 8, 2015 at 7:29 PM
I will double check the newest application you sent hopefully today.

Please also see this question : http://stackoverflow.com/questions/20019961/gstreamer-unable-to-read-input-video-when-source-stops-and-starts-again

You shouldn't have to modify the code, if you can't get something to work a specific way that should have been your question in the first place.

You can specify the external address of the adapter in the SDP which would be routed to 127.0.0.1 anyway, And keep in mind you may be bypassing the network but not the networking stack using localhost.

Using a finalizer isn't required either, just set ShouldDispose to false and then set it to true when your done processing the packet, using a finalizer is definitely going to reduce performance further.

If that's the route you really need to go then you can have ShouldDispose set to true by another mechanism which will indeed give you finer control of when the resources are being released but keep in mind the packet will still be located in the buffer... every-time you receive a new packet the buffer is modified... please also remember that another packet is not started receiving until the event for packet is handled. That should already prevent any race conditions related to disposition.

MPlayer has buggy code related to polling, no matter how much you remove checks for polling on the server or client code there is still the potential for MPlayer to freeze, I have submitted a bug for this but their solution was to use the ffmpeg RTSP classes which have their own set of issues...

I will check out the application here shortly and see what happens with the tests.
Marked as answer by juliusfriedman on 10/8/2015 at 11:29 AM
Coordinator
Oct 8, 2015 at 7:34 PM
Also can you link me to the versions of GStreamer and MPlayer you are using so the tests can be against the same version.

Thanks
Marked as answer by juliusfriedman on 10/8/2015 at 11:34 AM
Coordinator
Oct 8, 2015 at 7:56 PM
Using the latest versions of GStreamer I can't even run your tests...

I get an exception related to how the append calls are made in the TextBoxLogger. 'Cross-thread operation not valid'

I had to ' Control.CheckForIllegalCrossThreadCalls = false;'

TestRtp works as expected, can you give me some type of instructions on how to replicate the issue?

Thanks!
Marked as answer by juliusfriedman on 10/8/2015 at 11:56 AM
Oct 8, 2015 at 7:58 PM
You are right about some of the things I've done not requiring code changes. Such as passing an existing socket to bypass the issue with Connect or the ReceiveBufferSize to fix artificats. But the following things seem to have required code changes:

PerPacket=true disposing of packets before they have the chance to be sent out
Rtsp communication hangs due to not handling CompletedSynchronously situations
Rtsp thread m_Started bug mentioned earlier.

And quite possibly the issue I'm having now. with mplayer dying after a certain time. The problem doesn't happen if I'm not using mma. Only if I'm using mma. But I wish I could give more information on where the problem actually is. Perhaps it is just a setting, though I haven't figured it out.

You mention putting in the external address in the SDP. To clarify, that is what I've been doing all this time. It does not work without my changes to remove the RtpSocket Connect call. This should hopefully be proven by the Test RtpSocket Connect test case in my test app.

I'm not really sure that ShouldDispose would be the right answer. I see how it would stop the Dispose from happening when the using statement exits. But, it seems I would have to keep track of which ClientSessions have processed it, because the last one to process it would be the one that would set ShouldDispose to true (and also I assume would have to call Dispose on it). Seems like a lot of tracking code when it could just be put in the finalizer.

I'm not sure if we are on the same page about the dispose issue and the buffer. Hopefully the test app will show this, but I'll try to explain again. Here is the code:
            if (parseRtp && mRemaining >= RtpHeader.Length)
            {
                using (var subMemory = new Common.MemorySegment(memory.Array, offset + index, mRemaining))
                {
                    using (RtpPacket rtp = new RtpPacket(subMemory))
                    {
                        //Raise the event
                        HandleIncomingRtpPacket(this, rtp);

                        //Move the index past the length of the packet
                        index += rtp.Length;

                        //Calculate the amount of octets remaining in the segment.
                        mRemaining -= rtp.Length;
                    }
                }
            }
I think you are under the impression that the packet is handled by the time the HandleIncomingRtpPacket call is completed and that it is safe to dispose the packet at this point. I don't think this is true. The HandleIncomingRtpPacket does enque the packet into every ClientSession RtpClient buffer prior to disposing of the packet. However, that doesn't mean every ClientSession RtpClient has had a chance to read this buffer before the packet is disposed. In my testing, this is the vast majority of the time. Therefore, you have a race condition. Does that make sense?
Oct 8, 2015 at 8:00 PM
It sounds like you're running the code in the debugger if you are getting cross thread operation exceptions.

gst-launch-1.0 version 1.4.5
GStreamer 1.4.5
Unknown package origin

MPlayer Redxii-SVN-r37370-4.9.2 (i686) (C) 2000-2015 MPlayer Team
FFmpeg version: N-69658-g03cecf4
Build date: 2015-02-08 19:01:52 EST

Test Rtp is supposed to work. It shows that the RTP stream has no issues. It does not use your library at all.
Oct 8, 2015 at 8:03 PM
Test RtpSocket Connect should show no video. This is if you enter the external address of your local machine into the SDP.

Ignore the Test Camera H264. These were solved by your suggestion of ReceiveBufferSize

Test RTSP Hang should show the CompletedSynchronously issue. You may need to run this test for awhile since this sometimes fails and sometimes doesn't. It should loop the test automatically.

Test PerPacket should show no video because of massing packet loss due to disposed problem mentioned above.

Test MPlayer RTP should show that a straight RTP stream (no mma) works fine The Test MPlayer failure adds mma library into the test and should show that it fails after awhile.
Oct 8, 2015 at 8:10 PM
Also, I have not seen the gstreamer issue you linked to. According to the packet capture, rtp packets are successfully being sent from gstreamer to the port that RtpSource is listening to up to and beyond the unexpected TEARDOWN message.

I'd me more inclined to blame MPlayer except that it plays my direct Rtp stream fine and I'd be surprised if it handles the Rtp stream from mma any differently.
Coordinator
Oct 8, 2015 at 8:19 PM
Edited Oct 8, 2015 at 9:12 PM
The only thing I was able to replicate thus far was when using connect as the code was originally it would not work, this is because the ports were reversed, you have remoteRtp being the localRtp port...

You can verify this by adding a Connect to the same port.

rtpSocket.Connect(remoteRtp);

If you use the same ports with the old code the Bind and Connect calls proceed as they were originally.

Test Rtsp Hang as well as Test Mplayer Failure both seem to be working unless I am missing the point of the tests?

That only leaves Test Camera which I can't replicate without a Pi...

In the explanation about the race condition, if the client attempts to send a packet which is disposed it then it will be skipped, ensuring the packet goes to every client is not a function of the source's rtp client and must be dealt with at the server level, hence where tracking comes into play.

You could do a check against the packet created time and if longer then N seconds ago dispose it, that would alleviate complex tracking but probably cost more memory. Keep in mind you can also check the packet's Transferred property to determine when a packet is sent out at least one time.

I don't see this as a race condition because the same thing happens with PerPacket set to false but only if the frame changes before those packets are sent out..

Do I have a capture which shows the TEARDOWN you are referencing or can I replicate with one of the tests in your app? I need to try and understand in what test this is occurring to attempt replicate it and diagnose it.
Marked as answer by juliusfriedman on 10/8/2015 at 12:32 PM
Oct 8, 2015 at 8:33 PM
I'm not sure I understand. I assume you are talking about the Test RtpSocket Connect test cases? Are you asking me to add that rtpSocket.Connect(remoteRtp) line to my TestRtpSocketConnectTransportContext.cs file? Because I did and it made the (Fixed) test fail.

Would it be possible for you to describe what you see when you run those Test RtpSocket Connect test cases. What I am seeing is...

Test RtpSocket Connect: No video. Mplayer says it found video stream, but doesn't play. Times out in a few seconds and goes away.

Test RtpSocket Connect (fixed): Plays fine.
Oct 8, 2015 at 8:35 PM
You may have to wait a number of iterations on Test Rtsp Hang and Test MPlayer Failure. They fail in an indeterminate time. For me, the Rtsp Hang is usually within a minute or two. MPlayer failure is longer.
Oct 8, 2015 at 9:01 PM
Test camera doesn't use the pi. I original was going to include a packet capture to replay, but didn't. But I already said you should ignore those because receivebuffersize fixed them.

Sorry if I'm getting frustrated about the packet dispose. I'm still not sure we're on the same page. The "Test PerPacket" should show the problem, but I assume for whatever reason that it is working fine for you. I probably will need to run my test app on a separate machine, but it definitely fails every time for me on my PC. Maybe I can draw some kind of diagram here to show what is happening. And also, I did NOT see the same behavior with PerPacket=false.

What I'd expect:

GStreamer Sends RTP-> RtpSource gets RTP -> RtpSource enqueues RTP into ClientSession -> ClientSession sends RTP to MPlayer

Sometimes that's what happens. Most of the time I see this:

GStreamer Sends RTP-> RtpSource gets RTP -> RtpSource enqueues RTP into ClientSession -> RtpSource disposes RTP -> ClientSession ignores RTP because it's disposed


I could probably get you a capture, but if you can get the Test MPlayer Failure test to work that would show the problem and you could capture packets from that.
Oct 8, 2015 at 10:07 PM
I had someone else test on another machine. The Test MPlayer Failure has yet to show the problem, but we're going to let it run for awhile. All other tests worked as I expected.
Oct 8, 2015 at 10:15 PM
The other computer has now proven the Test MPlayer Failure issue. The first time we ran it, it couldn't reproduce the problem after 20 minutes or so. We ran the test a second time and it appeared to take about 8 minutes and then the video failed. It's kind of a difficult one to reproduce, sorry.
Coordinator
Oct 8, 2015 at 10:21 PM
Edited Oct 8, 2015 at 10:22 PM
I think I understand your issue when I combine it with the other issue you were having as far as Disposed.

The context switch is taking longer to occur and thus you are seeing some of these issues where I might be not. If I intentionally replicate the issue by intentionally holding up a thread then I see what you mean.

For an easy fix on the disposition you could simply Clone the packet a single time in the 'OnRtcpPacketReceieved' and use a copy of that packet to raise the event, only do this if packet.ShouldDispose is true.

I also think there might be a bug in clone when using the selfReference parameter, the header should also be cloned by reference in such cases.
Marked as answer by juliusfriedman on 10/8/2015 at 2:22 PM
Oct 8, 2015 at 10:30 PM
Yes, it's definitely a context switching issue. If the ClientSessions were able to execute prior to the packet being disposed, you wouldn't see the problem. But I don't know what would be different about the context switching. Maybe hardware differences.

Your clone suggestion was essentially what I originally tried and it fixed the problem. That's what you should see in the Test PerPacket (fixed) case. The difference is I didn't handle any kind of disposing of it. But like I said, I changed that fix to actually dispose in the finalizer. It doesn't appear to have had any negative impact on performance, so that's probably what I'll go with.

Really, the only thing I'm stuck on is the MPlayer failure test. I'd really like you to see the same results I'm seeing for that test, but it may prove a little difficult. You may need to run that test for 30 minutes. If it hasn't failed by then, I'd suggest restarting the test. I know it fails on both my PC and my co-workers PC. He is probably using different gstreamer and/or mplayer versions, but I didn't verify that.
Coordinator
Oct 8, 2015 at 11:59 PM
I do get an exit after a while, is that what I am looking for?

And what you are trying to figure out is what is causing that exit from Mplayer?

Well if that's the case the problem with that seems to be memory collection, finalizers are running which is blocking execution for long enough for the source to thinks it's and EOS..

The code seems different that the latest released code for me to change it, do you have an example using the latest code and not the code in Lib?
Marked as answer by juliusfriedman on 10/8/2015 at 3:59 PM
Oct 9, 2015 at 12:15 AM
If MPlayer exited by itself without any input from you, then yes that's what you are looking for and I'm trying to figure out why. It should keep playing until I close the app or stop the tests.

The code is probably a little old. The mma zip file I have shows that I downloaded it on 9/24, but I will build the latest and see what happens.
Oct 9, 2015 at 12:22 AM
I just downloaded the latest off of CodePlex and it has all the same issues.
Coordinator
Oct 9, 2015 at 1:07 PM
Edited Oct 9, 2015 at 1:33 PM
I will be running the test today against my latest code, I was able to get all the code in your solution updated and now I am just waiting for the MPlayerFailure to exit.

When I use your code in the /Lib folder the exit occurs immediately almost, within the first few minutes. I tracked this to GC finalizer threads, I didn't have the same source code so I couldn't really do more.

When I use my latest code and the same test I am over 15 minutes without any issue with exiting.

I will let it run for a while and see if it ever exits.

I am now over 40 minutes with no exiting...
Marked as answer by juliusfriedman on 10/9/2015 at 5:07 AM
Coordinator
Oct 9, 2015 at 3:13 PM
Edited Oct 9, 2015 at 3:16 PM
I was at about an hour without restarting, I have since restarted the test just to make sure and as of now it's running for over 40 minutes without any issue.

E.g. there seems to be something in the code in Lib versus my latest code which is causing that, I don't have the source and I don't feel like decompiling it.

If you would like I can zip up my latest library and send them back in a zip, I would be curious if on your machine the exit still occurs with those libs as it seems I can't replicate it once I update the code.
Marked as answer by juliusfriedman on 10/9/2015 at 7:16 AM
Oct 9, 2015 at 5:55 PM
Thanks for looking into this. I suppose I could zip up the library I'm using, but like I mentioned in my last post I had tried with the latest off of codeplex (Source Code -> Download). Is your latest code perhaps not on codeplex?
Coordinator
Oct 9, 2015 at 7:35 PM
Edited Oct 9, 2015 at 7:48 PM
Thats probably definitely the case (that my code is newer) and compiled for .Net 4.6, beyond that there aren't many changes.

If you would like I can zip up the compiled version I have and see what happens when you run it.

It's not a problem to look into issues I just need to be able to replicate them, if I can't reliably replicate the issue then I can't solve it.

I am going to stop the test now as it's been over 4 hours running.

I will start it again and see tonight if it's still running.

Thank you also for your interest in the library!
Marked as answer by juliusfriedman on 10/9/2015 at 11:35 AM
Oct 9, 2015 at 7:47 PM
I did notice when I would open your solution it prompted me to either upgrade my .net to 4.5.2 or to downgrade. I think I have 4.5.1. But I wouldn't think that could make much difference unless there was maybe some bug they fixed between 4.5.1 and 4.5.2.

If you could send me your compiled version that would be great. Then maybe I could see if it fixes the problem on my end. You can send it to uler3161 at hotmail.com. Thanks.
Coordinator
Oct 9, 2015 at 8:05 PM
Just sent everything, I would also try with 4.5.2 as that's what my code was designed to work against. It's possible it would still work with 4.5 but that may be exactly where your issue lies.

Lemme know if I can do anything else!
Marked as answer by juliusfriedman on 10/9/2015 at 12:05 PM
Oct 9, 2015 at 9:32 PM
I definitely see some differences in your newest version. And it's been running for 35 minutes with no issues so far. I'm going to probably run it overnight to see what happens and then try the new libs in my actual application. I'll let you know what happens. Thanks for your help.
Coordinator
Oct 9, 2015 at 11:58 PM
Cool, if you would like to compile a list of the changes I can try to narrow down the culprit in the original code. I wouldn't worry too much about it though since I do plan on updating the code eventually.

Let me know what you find and keep your eye out for updated versions of the library.
Marked as answer by juliusfriedman on 10/9/2015 at 3:58 PM
Oct 13, 2015 at 3:51 PM
When I was testing with my test app, instead of using the test app build that came with what you sent me, I just copied your library dlls over into my project, changed it to .net 4.6, made any necessary code changes and then tested. It worked flawlessly, so I'm quite happy.

Unfortunately making my actual application work is a bit tougher. For instance, my test app had it's own RtspServer implementation and that kept working. But in my actual app, I was using the RtspServer in your library and the parameterless constructor does not work. I can't remember the exception, but it's similar to the issue I had with the connect call on RtpSocket. I saw there was a constructor that took an IPEndPoint so that fixed it for me. However, I'm having other issues setting up the socket to pass into transportcontext. It'd help if I could get the up to date source code you are using. Are there any plans to get that up on codeplex anytime soon?
Oct 13, 2015 at 5:01 PM
I finally got it working. I was trying to use an existing socket with the TransportContext.Initialize(Socket) method. When it tries to create another socket for Rtcp, it checks the NoDelay on the Rtp socket which is throwing an exception. Something about an invalid argument. But I copied over my fixes from the test case app and that worked enough to get it going.

I'm noticing though that the live stream is sometimes pretty close to live and sometimes it's 5-10 seconds behind. That's something I'll need to get fixed as well because it has to have low delay. I'm not sure if that's a buffer size problem or not. I can't really think of anything else it could be. I would like to try to figure out what was changed in your current code versus what I downloaded off of CodePlex so I could test some things.
Oct 13, 2015 at 7:43 PM
So I've ignored the problem with the 5-10 second delay (which by the way, it will usually recover from in a little while). It's doing much better, but it will still fail after one to two hours of playing. I doubt I would need it to run this long. If I knew it would consistently work for an hour that might be fine. But since there's no guarantee it will run that long, I'm still worried.
Oct 13, 2015 at 8:34 PM
This happens quite a lot of the time when I stop viewing the streams in my application. Stopping the streams is essentially just closing mplayer and gstreamers processes and calling RtspServer.TryRemoveMedia

Object reference not set to an instance of an object.
at Media.Rtsp.RtspServer.ProcessAccept(IAsyncResult ar)
at System.Net.LazyAsyncResult.Complete(IntPtr userToken)
at System.Net.ContextAwareResult.CompleteCallback(Object state)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Net.ContextAwareResult.Complete(IntPtr userToken)
at System.Net.LazyAsyncResult.ProtectedInvokeCallback(Object result, IntPtr userToken)
at System.Net.Sockets.BaseOverlappedAsyncResult.CompletionPortCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* nativeOverlapped)
at System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* pOVERLAP)
Coordinator
Oct 14, 2015 at 1:58 PM
Edited Oct 14, 2015 at 3:31 PM
I do have plans to update the code but it's not my main priority as of now for several reasons, hopefully that will change sometime after the new year.

I am not sure why you need an alternate implementation of the RtspServer, the library includes one. Unless your adding something feature which it would probably be wise to leave the code as it is especially if you need support.

When you say there is a delay, how have you verified that the delay is coming from the RtspServer? Having 10 seconds of packets buffered seems unlikely unless there is some pausing going on..

Lastly, where can I replicate the issue? I ran the test application for well over 12 hours and experienced no issues, additionally I ran my own stress tests which have not encountered the issue so I would really like to know is how is this issues now manifesting itself? It seems obvious to me that even if I were to find a potential issue in the RtspServer it would serve your purpose as you have a different implementation of the RtspServer.

With that aside I would like to explain some of the difficulties I think your having.

1) There may be another service running on port 554, sometimes on windows the DLNA service runs on that port. 'wmpnetwork', If you need to check ports you can use netstat >netstat -an -p TCP to verify the port you need is clear, there are also extension methods to find open ports...

2) I was finally able to replicate the exception you had related to UDP Sockets, I had to run the application as Administrator, upon doing so I received 'Permission denied. An attempt was made to access a socket in a way forbidden by its access permissions.' e.g. WSA 10013 (https://msdn.microsoft.com/en-us/library/aa924071.aspx)

You may want to see this post to understand (http://stackoverflow.com/questions/11747545/getting-an-access-exception-with-multiple-pgm-listening-sockets-on-non-admin-acc), what essentially is happening in our cases is that the non admin process is unable to connect to the handle for the socket when the connection occurs from the local computer.

You can disable the firewall for this particular exception and it should go away, another way you can fix the exception is to run the program which is accessing the socket as an administrator, finally you could also pass a socket which doesn't connect.

3) I will check out the TryRemoveMedia exception shortly, do you have exact instructions on how to best reproduce?

E.g. I imagine it goes something like Add the stream, consume the stream, remove the stream. I just want to be sure.

4) The changes are mostly organizational, some things have been refactored or relocated to different namespaces and some new classes have been added which allow for more code reuse. There are some minor improvements in memory and cpu usage, some additional work has been done to allow MediaFileStream to work on media located on remote Uri's seamlessly. Most of all there is a lot of experimental code which I created simply either for completeness sake or the fact to say I did it.

In reality there is a lot more refactoring to do, there is a lot more separation which is required and also the components which are as of yet unimplemented e.g. Rtcp Feedback, Decoding, Encoding, the infrastructure for those components such as an (Media[.])Image class.

You can check out the Issue Tracker here:

http://net7mma.codeplex.com/workitem/list/basic

5) Lastly, I don't provide free support, I will fix bugs if they can be replicated and I will continue to take feedback to improve the library but usually what happens is that some entity which is making monetary gain off the project and has less then knowledgeable developers working on things takes the code, tries to modify it for their project (usually in an invalid or proprietary way) and then asks for support in their modifications. There are also those who pledged contribution in some way or another and have silently teetered off. I am not saying your in either of those two categories but you did indicate you have a co-worker, and that would indicate your working on something for your employer...

In short if your not providing any support for the project or contributing anything back to the source then there is no tangible benefit of me providing support for your problems.

I will let you know what I find in relation to the TryRemoveMedia shortly.
Marked as answer by juliusfriedman on 10/14/2015 at 5:58 AM
Coordinator
Oct 14, 2015 at 2:24 PM
Edited Oct 14, 2015 at 2:36 PM
I can't replicate the issue with TryRemoveMedia.

The way I tested was that I started the server, added some media, consumed the media, while consuming the media I called code to remove the media I was consuming.

If there is something I am missing let me know.

Otherwise let me know how I can help you as well as what you can do for the project!
Marked as answer by juliusfriedman on 10/14/2015 at 6:24 AM
Oct 14, 2015 at 4:02 PM
I'm not saying there's a problem with TryRemoveMedia. I was just trying to explain what my code was doing when I saw this issue.

I don't really want/need my own RtspServer implementation. I had that one from the test app because my implementation fixed the Rtsp hanging problem. I don't know if your more recent version has fixed that. Maybe it has. But I do know I couldn't get the RtspServer parameterless constructor to work. It would throw an exception immediately. I'm pretty sure it was the same kind of problem I was having with the bind/connect code...
The requested address is not valid in its context
   at System.Net.Sockets.Socket.DoBind(EndPoint endPointSnapshot, SocketAddress socketAddress)
   at System.Net.Sockets.Socket.Bind(EndPoint localEP)
   at Media.Rtsp.RtspServer.Start()
   at TestMMA.TestRtspHang.TestRtspHangTestCase.Execute() in c:\Users\danielu\Documents\Visual Studio 2012\Projects\TestMMA\TestMMA\TestRtspHang\TestRtspHangTestCase.cs:line 44
   at System.Threading.Tasks.Task.InnerInvoke()
   at System.Threading.Tasks.Task.Execute()
This was when trying to run the Rtsp hang test case in my test app.

As for the delay, this is still the rtp stream I've been using from my raspberry pi that has been verified to run smoothly when mplayer connects directly to the stream. The delay only happens when MMA is used. It does not appear to be a cpu/memory problem. And if it were a network problem, I would expect mplayer to have the same delay issues when playing the rtp stream directly.

No other service running on 554. This is proven by the Rtsp hang test case which connects to 554. If I recall, it's just your version of RtspServer from the Codeplex download with changes to fix the CompletedSynchrnously problem. It may be radically different from your current version because I don't know what you've changed, but my changes were small.

I don't believe any of my socket issues are related to being admin. My user account is an administrator.

I realize the issue with support and I don't really blame you because it does take time away from you. This is a project for my work, but I don't think it's something we would have any monetary gain on. I have no problem trying to fix the problems myself. I'm trying to explain there are a lot of bugs in this library. I tried to prove these things with my test app, though it appears the version of this library on Codeplex is so far behind that you have perhaps already fixed some of the problems. I am sorry you feel I'm not contributing anything. I feel that I've proven numerous bugs with my test app and I'd like to think that means something not just to you, but also to everyone who tries to use your project. But I suppose if you don't want me to find bugs, that's fine. I can start from scratch on my own server if that's what you'd rather me do.
Coordinator
Oct 14, 2015 at 8:36 PM
Edited Oct 14, 2015 at 8:39 PM
My answer wasn't properly worded so let me just clarify, these exceptions are not because you are not using an Administrator account, they are because you are using such an account. The source I linked to was an example that Socket behaves differently depending on the account they are used under.

The delay could very well be tracked down to my code and if that is the case I will fix it.

What I do know is that currently there are several lines in the ClientSession code which do cause some repeat packets to be sent but that is also fixed easily and doesn't occur with PerPacket.

Just encase it'a big deal I will also post that code here:
 /// <summary>
        /// Called for each RtpPacket received in the source RtpClient
        /// </summary>
        /// <param name="client">The RtpClient from which the packet arrived</param>
        /// <param name="packet">The packet which arrived</param>
        internal void OnSourceRtpPacketRecieved(object client, RtpPacket packet = null, RtpClient.TransportContext tc = null)
        {

            //If the packet is null or not allowed then return
            if (Common.BaseDisposable.IsNullOrDisposed(packet) || m_RtpClient == null) return;

            Thread.BeginCriticalRegion();

            //Get a source context
            RtpClient.TransportContext localContext = null, sourceContext = tc ?? GetSourceContext(packet);

            //Get the sourceContext incase the same payload type was used more then once otherwise fallback to the context for the Payloadtype
            if (sourceContext != null)
            {                
                localContext = m_RtpClient.GetContextForMediaDescription(sourceContext.MediaDescription);
            }
            else
            {
                sourceContext = localContext = m_RtpClient.GetContextByPayloadType(packet.PayloadType);
            }
            //This is the correct code
            if (false == localContext.UpdateSequenceNumber(packet.SequenceNumber)) return;

            //If there is no context then don't send.
            //OR
            //If the context already sent the packet don't send                       //(make sure the sequence number didn't wrap)
            //if (context == null || context.SequenceNumber >= packet.SequenceNumber && sourceContext.SequenceNumber != packet.SequenceNumber) goto Exit;

            if (PacketBuffer.ContainsKey(sourceContext.SynchronizationSourceIdentifier))
            {
                PacketBuffer.Add(sourceContext.SynchronizationSourceIdentifier, packet);
            }
            else if (m_RtpClient != null)
            {
                //Send packet on Client Thread
                m_RtpClient.EnquePacket(packet);
            }

        //Exit:
            Thread.EndCriticalRegion();

            return;
        }
There are definitely not a lot of bugs in the library at least in my opinion, then again any bug can be considered one to many especially if it causes some type of frustration.

If everyone who used the project provided feedback then the project wouldn't be "so far behind" or more appropriately not synchronized to. Usually when someone emails me or creates and issue I provide them with copies of the library or it's sources if it will help them as I also did for you.

I am not attempting to argue here and I do appreciate good rebuttal when someone doesn't agree with my points however I quite simply don't have the time right now especially for free. If some aspects of my personal life change sooner than expected or my employer needs a specific feature which can be released then I have no problem sharing such.

I do appreciate bugs being found and definitely if they can be re-produced and I am sorry that the latest code is not up yet and that you had so many problems with the code as it is or was.

Last but not least I would imagine that even if you did start to rewrite the code in some way for your purpose that it wouldn't benefit the project unless you contribute back and those changes help everyone, this implementation at the very least would provide you with a solid implementation of the client and server code in such cases, maybe you should consider joining the project or asking your employer if they are willing to pay you to donate some time.
Marked as answer by juliusfriedman on 10/14/2015 at 12:37 PM
Coordinator
Nov 20, 2015 at 7:46 PM
As an FYI a new version has been posted.

Thanks again for your interest in the library!
Marked as answer by juliusfriedman on 11/20/2015 at 11:46 AM