Telling a RTSP client to switch sources

Topics: Question
Jul 7, 2016 at 6:33 PM
Hi,

I would like to expand upon https://net7mma.codeplex.com/discussions/653017 and ask: how would I best inform a client to switch between two streams being published from my RTSP server? The linked-to article seems to discuss achieving switching at the stream level, but I'm interested in simply telling the client to switch to a different stream URL (still published from my server, however). How would I got about this?

The post seems to hint that it's possible to do this by adjusting the SDP; but does that mean sending out a new SDP? And if so, how is that done using this framework (warning: I'm a TOTAL newbie to this API).

I imagine it like this: client connects to my server (known URL), and then my server - after a period of time - notifies the client that it needs to switch to a second URL (still from my server). My server does something which notifies the client, but I don't know what that something is.

Thanks in advance.
Coordinator
Jul 7, 2016 at 8:04 PM
What do you mean 'Switch' between two streams... it's a little ambiguous.

You can use a Server Sent RTSP Message to send any kind of RTSP Message from Server to Client, if the client will honor the server sent message is another story.

If you can provide more information I can probably give a better response..

You can also use the Time Description within the Session Description to achieve this, but it depends on the client to honor and correctly utilize the Time Description within the Session Description.

Let me know how I can help further
Marked as answer by juliusfriedman on 7/7/2016 at 1:04 PM
Jul 7, 2016 at 8:12 PM
Julius,

Thanks for your response. Apologies for being ambiguous. I'm new to the world of RTSP and so my terminology may be a bit "lacking".

This is what I want to do. I want to have a client connect to my streaming server and stream, say, rtsp://<ip>/live/<mystream1>. Additionally my server is also providing rtsp://<ip>/live/<mystream2>. A period of time passes, I send a command to the player (from the server), and the player then stops playing <mystream1> and starts playing <mystream2>. To the viewer there is maybe nothing more than a "blip" as the streams switch in the player window, but otherwise it's seamless. Like the previously linked-to article I have multiple webcam sources and want to toggle between them. I may have N number of streams at some point (I'm not limited to two). I want this controlled from the server (so the server directing the client to stop playing its current stream, and open up a new stream).

I figured there's a way I can write a player to do this but my hope is there's some kind of pseudo-standard way of doing this stream swap or switch. I respect that not all (many?) players will honor this, but for those that do we'd like to have this feature.

You mentioned using the Time Description and Session Description to do what I described. Is there any way you could walk me through how this is done? For now I'm fine with a simple "hello world"-style example where I simply start up your API with two streams, and then direct, say, VLC (or my own custom player if that's necessary) to a URL, send a whatever sort of redirect command(s) I need to send from the server, and then have the player switch to my second stream.

I hope this is clearer. Thanks for your time.
Coordinator
Jul 7, 2016 at 8:57 PM
Edited Jul 7, 2016 at 8:58 PM
No problem on the ambiguity.

Yes, it's totally possible but it's up to the player to use the information within the Time Description to switch to the streams.

A more compatible or yet a method which will work even if the Time Description method does not would be to define multiple streams in the SDP and just send data related to each stream when you want, You will also see two video Tracks listed for the stream in the player and allow you to disable one or the other independently,

Rtcp should keep the other streams alive at the proper sequence numbers while the player decodes the data related to the stream being transmitted and will switch to the other stream when it gets data for that stream.

If you need more information let me know.
Marked as answer by juliusfriedman on 7/7/2016 at 1:57 PM
Jul 7, 2016 at 9:12 PM
OK. So, since you're smart in this area and I'm a newbie, let me see if I understand what you're saying.

I register N number of streams with the server, and when the SDP gets sent, it will have those streams declared. However, I ONLY send media data related to ONE of those streams to the client. When I want to switch the stream the client is playing, I stop sending data for the currently playing stream and start sending data for one of the not-currently-playing streams? The theory there is that the client will then just start displaying that data instead (and effectively "switch" the stream from the perspective of the player/viewer)?

I see RTSP commands like ANNOUNCE and REDIRECT. Are those useful/needed in any of this, or am I literally just doing as I described above?
Coordinator
Jul 7, 2016 at 9:29 PM
Edited Jul 7, 2016 at 9:30 PM
You don't have to keep putting yourself down... albeit one appreciates your modesty,

The player must support the ability to play more than one track at a time for this to work....

How the player displays that data is up to the player...

Usually the player will display the data within the same client viewing area but that's totally up to the player...

E.g. If there is already a movie playing I can imagine that the basic result would be 'mixed' video from both tracks which is not desirable.

Thus you don't want to send media from both tracks at the same time unless you want the player to display them at the same time :)

So...

You Source two streams {A, B}

You consume a SINGLE stream {C} which is composite of the {A,B} streams.

When data arrives from A you provide the data through C

When data arrives from B you provide the data through C

How you provide the data is key and will determine if timestamps have to be modified or otherwise which can be as complex as you require.

In layman terms, like I stated in the previous thread... if you handle this based on FrameChanges from each source and then swap the source your transmitting from when you accumulate the required amount of FrameChanges you can then queue the packets as required through C and they will be sent in the order you en-queue them to the client.

No, in this scenario you don't need ANNOUNCE or REDIRECT, everything is done in DESCRIBE, SETUP and PLAY.

Let me know what else I can help with!
Marked as answer by juliusfriedman on 7/7/2016 at 2:29 PM
Jul 7, 2016 at 9:40 PM
Great, thanks.

Now, the client has had BOTH streams described to it in the SDP, right? Your use of the stream {C} is a bit confusing as it seems like you're saying the client is only seeing ONE stream ({C}) and I'm simply changing the source of that stream on the server. Stream {C} in this case is, um...."virtual" (maybe that's a decent word to use here), it's not really described in the SDP. The SDP describes A or B, and I'm simply playing A or B; whichever is active, in your case, is {C}. Right?

Let me throw an additional curve ball. What if the streams are different aspect ratios? One stream by AxB and another being CxD? Those ratios are probably described in the SDP and therefore, if the player supports it, it ought to figure out how to work around this. No?

One final question before I stop chewing your ear off, are you AWARE of any players that do what we're saying? VLC? Exoplayer (https://github.com/google/ExoPlayer)?
Coordinator
Jul 7, 2016 at 10:45 PM
Yes.

C is a RtspSource which is not consumed from anywhere...

It is composite of A, B which are either RtpSource or RtspSource or otherwise.

C is where the Describe, Setup and Play occur.

E.g. Your server will set up A, and B, they are accessible as live/A and live/B

You will then create C

It will be accessible from 'live/C' but will have no source media.

You will attach A and B to C.

You will modify the logic in C to store the packets from each source and transmit them as required.

The SessionDescription of C will contain the m= lines from A and B that you want to source through C.

Yes, VLC, Exoplayer and pretty much every other player can be MADE to do this.... You have to change the SSRC depending on the viewer / transport layer / rtp implementation but YES it can be done.

The player will just display the frames as they appear from the source either A or B in the single player window which is attached to C.

I have verified this mostly with QuickTime and on accident due to how they sloppily reuse ports for RTP sessions and make no checks at the packet data being given to the decoder, this results is both streams attempt to be played in a single view.

When I was experimenting with JPEG the result was that view was mixed between the two JPEG sources I was using (Bandit and Mirror), the player would resize to the largest viewing size required, this is similar to VLC but QuickTime would not switch back to the smaller size where are VLC would.

In my experience because of most implementations utilize ANY available port from during reception of RTP but from a specific IPAddress, thus you may not have to do anything to the packet fields to get the player to display the results and then again you might depending on who is playing the data and how they implement RTP.

The results are the same for JPEG/ h264 or Audio IMHO

In short, no matter the player being used nor the algorithm they use where there is a will there is a way....

Hopefully that answers your questions!

Let me know if you need anything else or further clarification.
Marked as answer by juliusfriedman on 7/7/2016 at 3:45 PM
Jul 8, 2016 at 3:23 AM
Edited Jul 8, 2016 at 3:23 AM
Great. Thanks for your detailed and wonderful responses.

So, just to be clear. I'm a guy who needs pictures sometimes....
Video Source 1 ------> Stream A
                                                    \
                                                      \
                                                       ------ Stream C ------> Client
                                                      /
                                                    /
Video Source 2 ------> Stream B
The client is actually consuming Stream C, and Stream C is sourcing from Stream A/B. I'm correct, right?
Coordinator
Jul 8, 2016 at 3:49 AM
Indeed you are.

Good job!

To test this easily use two jpeg sources, make a third source which recieves from the other two.

Test in Player.

Add more tracks e.g. h264 or audio.

Re test.

See how the player responds.

Let me know if I can help you further.
Marked as answer by juliusfriedman on 7/7/2016 at 8:49 PM
Jul 8, 2016 at 4:40 PM
So here I am using your SDK for the first time. I've managed to spin up a server nicely and get it to source from an external RTSP stream just fine. However, I'm not sure how to code up making a third source which receives from the "other two".

This is what I have thus far:
        static void Main(string[] args)
        {
            const int ServerPort = 5554;

            IPAddress address;
            IPAddress.TryParse(GetLocalIPAddress(), out address);

            using (RtspServer server = new RtspServer(address, ServerPort))
            {
                Console.WriteLine("rtsp://" + address.ToString() + ":" + ServerPort.ToString() + "/live/Omega"); 

                server.Start();

                server.TryAddMedia(new RtspSource("Omega",
                    "rtsp://wowzaec2demo.streamlock.net/vod/mp4:BigBuckBunny_115k.mov",
                    Media.Rtsp.RtspClient.ClientProtocolType.Tcp));

                server.TryAddMedia(new RtspSource("Omega1",
                    "rtsp://mm2.pcslab.com/mm/7h1500.mp4",
                    Media.Rtsp.RtspClient.ClientProtocolType.Tcp));

                // create a third that sources from the above two

                while (server.IsRunning)
                {
                    Thread.Sleep(10); 
                }
            }
        }
So I need to add a new RtspSource and have it source from Omega and Omega1. I'm not sure how that's done. What block of code am I missing here (also, I know you said to source from JPEG, but I only know sample RTSP addresses).

Thanks for your continued support.
Coordinator
Jul 8, 2016 at 7:21 PM
No problem.

May I ask why you want to do this? e.g. In my experience camera touring is better handled in other ways...

May I also ask why you went directly to H.264 rather than attempting this in a controlled environment with the RFC2435Stream's as your sources until you were comfortable with the library... e.g. learn to crawl before you walk, and learn to swim before you sink....

None the less here is the way to make a composite source
                 //Make a new session description
                 Media.Sdp.SessionDescription compositeDescription = new Sdp.SessionDescription(0);

                //you need to access the session description from the media you wish to composite.

                //Add the first
                compositeDescription.Add(mirror.SessionDescription.MediaDescriptions.FirstOrDefault(), false);

                //Add the second
                compositeDescription.Add(otherSource.SessionDescription.MediaDescriptions.FirstOrDefault(), false);

                Media.Rtsp.Server.MediaTypes.RtpSource composite = new Rtsp.Server.MediaTypes.RtpSource("Composite", compositeDescription);

                mirror.RtpClient.RtpFrameChanged += (sender, frame, transportContext, final) =>
                {
                    composite.RtpClient.EnqueFrame(frame);
                };

                otherSource.RtpClient.RtpFrameChanged += (sender, frame, transportContext, final) =>
                {
                    composite.RtpClient.EnqueFrame(frame);
                };
Let me know if you need further help...
Marked as answer by juliusfriedman on 7/8/2016 at 12:21 PM
Jul 8, 2016 at 7:23 PM
Would you mind if I contacted you personally to describe, in greater detail, what we're trying to accomplish? Some of it is pseudo-internal company stuff and it might make people happier on my end if we took the details I'm trying to accomplish offline.

Thoughts?
Jul 8, 2016 at 8:13 PM
Edited Jul 8, 2016 at 8:18 PM
...also, I'm not sure this code is working 100%. I see what you're doing, but there's an exception being thrown when parsing out the host portion of the SessionDescriptionLine:
                SessionDescription compositeSDP = new SessionDescription(0);
                compositeSDP.Add(firstSource.SessionDescription.MediaDescriptions.FirstOrDefault());
                compositeSDP.Add(secondSource.SessionDescription.MediaDescriptions.FirstOrDefault());
                compositeSDP.ConnectionLine = new SessionDescriptionLine("c=IN IP4 " + myIp); 

                RtpSource compositeSource = new RtpSource("Composite", compositeSDP);
When RtpSource is instantiated there appears to be some code that attempts to extract Host from ConnectionLine. ConnectionLine is null.

I tried doing the following to no avail:
 compositeSDP.ConnectionLine = new SessionDescriptionLine("c=IN IP4 " + myIp); 
(there appears to be some kind of gremlin that gets into codeplex's editor when I type the string in the SessionDescriptionLine constructor. Not sure what's up with that.
Coordinator
Jul 8, 2016 at 8:50 PM
Edited Jul 8, 2016 at 8:52 PM
Absolutely not, either email or Skype or whatever method you prefer for the communication.

I would also appreciate feedback on the API and various other aspects of the library especially if it's going to be useful to your organization.

You need add the required information to the compositeSDP...

Host, Name, etc.

Do you need an example of that also?

Let me know if your still confused and what else I can do for you.
Marked as answer by juliusfriedman on 7/8/2016 at 1:50 PM
Jul 8, 2016 at 8:52 PM
juliusfriedman at gmail.com?
Coordinator
Jul 8, 2016 at 8:54 PM
Edited Jul 8, 2016 at 9:03 PM
Yes sir, and let me know if you don't understand about the lines you need to add...

Here is an example from RFC2435Stream :)
SessionDescription = new Sdp.SessionDescription(0, "v√ƒ", Name);
            SessionDescription.Add(new Sdp.Lines.SessionConnectionLine()
            {
                ConnectionNetworkType = Sdp.Lines.SessionConnectionLine.InConnectionToken,
                ConnectionAddressType = Sdp.SessionDescription.WildcardString,
                ConnectionAddress = System.Net.IPAddress.Any.ToString()
            });

            //Add a MediaDescription to our Sdp on any available port for RTP/AVP Transport using the RtpJpegPayloadType            
            SessionDescription.Add(new Sdp.MediaDescription(Sdp.MediaType.video, 0, Rtp.RtpClient.RtpAvpProfileIdentifier, RFC2435Media.RFC2435Frame.RtpJpegPayloadType));

            //Indicate control to each media description contained
            SessionDescription.Add(new Sdp.SessionDescriptionLine("a=control:*"));

            //Ensure the session members know they can only receive
            SessionDescription.Add(new Sdp.SessionDescriptionLine("a=sendonly")); // this directive is for their `tools`

            //that this a broadcast.
            SessionDescription.Add(new Sdp.SessionDescriptionLine("a=type:broadcast"));
For H.264 you will need to ensure that the FormatType lines are also conveyed... they may also require an rtpmap line or not depending on overlapping payloads or whatever other scenario is planned.

Try to keep the lines and packets the same as possible to the original at first....

You will have success when you register two tracks in the player, e.g. you can verify this in VLC under 'Tracks'.
//Add the control line, could be anything... this indicates the URI which will appear in the SETUP and PLAY commands
            SessionDescription.MediaDescriptions.First().Add(new Sdp.SessionDescriptionLine("a=control:trackID=video"));
This line will effect where the 'Track' commands in VLC go, e.g. PLAY, SETUP, PAUSE and TEARDOWN.

Meaning you can add many Tracks and let the player enable them as required... or disable them as required.

For your purposes this may be 'Video1' or 'Video2' or whatever you would like to name the 'Sub Tracks'.
Marked as answer by juliusfriedman on 7/8/2016 at 1:54 PM