Serve already encoded H264 Frames through RTSPServer

Topics: Question
Mar 19, 2015 at 1:47 PM
Hey,
first of all, nice work.
My problem is, I have an already encoded frame as byte[] and would like to serve it with the RTSPServer to connected clients.
The problem is, that the client doesn't receive anything when it calls the "StartPlaying()" method:

        /// <summary>
        /// Initiates the RTSP Server
        /// </summary>
        /// <param name="width"></param>
        /// <param name="height"></param>
        private static void InitRTSPServer(int width, int height)
        {
            Console.WriteLine("Initialising RTSPServer...");

            //Setup a Media.RtspServer (default port 554)
            _server = new RtspServer(listenPort: _PORT)
            {
                Logger = new Media.Rtsp.Server.RtspServerConsoleLogger()
            };

            Console.WriteLine("Creating stream with {0}x{1}...", width, height);
            _stream = new RFC6190Media(width, height, "stream", null, false);
            
            _server.TryAddMedia(_stream);

            // start server
            _server.Start();

            //Wait for the server to start.
            while(false == _server.IsRunning)
                Thread.Sleep(0);

            Console.WriteLine("Server started! Listening on: " + _server.LocalEndPoint);
        }
        
        private unsafe static void EncodeAndUpdateStream()
        {
            int result = 1;
            int width = 1920;
            int height = 1080;
            byte[] frameBuffer = new byte[BUFFERSIZE];
            int frameCounter = 0;

            // init decoder
            var encoder = new EncoderWrapper();
            if(encoder.InitEncoder() < 1)
            {
                Console.WriteLine("<!> Error initialising Encoder! Aborting...");
                return;
            }
            Console.WriteLine("Successful initialised Encoder! Starting grabbing and encoding frames...");

            while (result > 0 && _server.IsRunning)
            {
                fixed (byte* bufferPtr = frameBuffer)
                {
                    result =encoder.GrabAndEncodeFrame(bufferPtr, &width, &height);
                }
                if (result < 1)
                {
                    Console.WriteLine("<!> Error grabbing Frame!");
                    return;
                }
                
                try
                {
                    // packetize frame in Media-Object
                    //_stream.Packetize(frameBuffer, result);
                    AddFrameToStream(frameBuffer);
                    Thread.Sleep(50);
                }
                catch (Exception ex)
                {
                    _server.Logger.LogException(ex);
                }
            }
        }
        
        private static void AddFrameToStream(byte[] encodedFrame)
        {
            //create packet from a byte[]
            RtpPacket packet = new RtpPacket(encodedFrame, 0);
            RtpFrame frame = new RtpFrame(packet);
            //Create a new frame
            var newFrame = new RFC6184Media.RFC6184Frame(frame);
            _stream.AddFrame(newFrame);
        }
The server accepts the client, but then nothing else happens. Sometimes it throws also an "OutOfMemoryException". I think the way I'm adding the frame is wrong?!

Would it be possible to serve the encoded frame with already implemented classes or do i have to implement something by my own?

Thank you very much
Coordinator
Mar 19, 2015 at 3:02 PM
Edited Mar 19, 2015 at 3:03 PM
What is the stack trace of the OOM exception?

The only thing I see is that the frame is being manually added.

RFC6190Media has a Packetize method, you could also derive from that class and use the thread there to do the packetization.

` private static void AddFrameToStream(byte[] encodedFrame)
    {
        //create packet from a byte[]
        RtpPacket packet = new RtpPacket(encodedFrame, 0);
        RtpFrame frame = new RtpFrame(packet);
        //Create a new frame
        var newFrame = new RFC6184Media.RFC6184Frame(frame);
        _stream.AddFrame(newFrame);
    }
You do realize thatencodedFrame` is a pointer and the data inside is being overwritten each iteration correct?

The RtpPacket classes accept changed through the source array so please use a new array or copy the array if the packet owns this data.

If you need anything else let me know.
Marked as answer by juliusfriedman on 3/19/2015 at 7:02 AM
Mar 19, 2015 at 4:08 PM
Yes thank you, I realized that already.
There is no Exception anymore, but still nothing happens on client side.

I changed my method and copy the frameBuffer to a new byte array before.
private static void AddFrameToStream(byte[] encodedFrame)
        {
            RtpFrame frame = new RtpFrame(96);
            //Create a new frame
            var newFrame = new RFC6190Media.RFC6190Frame(frame);
            newFrame.Packetize(encodedFrame);
            _stream.AddFrame(newFrame);
        }
But still it doesn't work. Why would i need to derive from your class? Is it not providing the needed functionality?

This is the server output when i connect to it:
Accepted Client: 1df9a647-ed2d-4cec-bd00-3323284b50aa @ 19.03.2015 14:00:27
Request=> OPTIONS rtsp://192.168.30.52:80/live/stream RTSP/1.0
CSeq:  1

 Session=> 1df9a647-ed2d-4cec-bd00-3323284b50aa

Adding Client: 1df9a647-ed2d-4cec-bd00-3323284b50aa
Response=> RTSP/1.0 200 OK
CSeq: 1
Public: OPTIONS, DESCRIBE, SETUP, PLAY, PAUSE, TEARDOWN, GET_PARAMETER
Allow: ANNOUNCE, RECORD, SET_PARAMETER
Server: ASTI Media Server RTSP 1.0
Date: Thu, 19 Mar 2015 14:00:27 GMT

 Session=> 1df9a647-ed2d-4cec-bd00-3323284b50aa

Request=> DESCRIBE rtsp://192.168.30.52:80/live/stream RTSP/1.0
Accept:  application/sdp
CSeq:  2

 Session=> 1df9a647-ed2d-4cec-bd00-3323284b50aa

Response=> RTSP/1.0 200 OK
CSeq: 2
Content-Type: application/sdp
Cache-Control: no-cache
Content-Base: rtsp://192.168.30.52/live/347f4514-1f16-4d4f-b95f-c5bf66b41817/
Content-Length: 338
Content-Encoding: utf-8
Server: ASTI Media Server RTSP 1.0
Date: Thu, 19 Mar 2015 14:00:27 GMT

v=0
o=ASTI-Media-Server 15615480720716703707 -2831263352992847885 IN IP4 192.168.30.
52
s=ASTI-Streaming-Session-stream
a=sendonly
a=type:broadcast
m=video 0 RTP/AVP 96
a=rtpmap:96 H264-SVC/90000
a=fmtp:96 profile-level-id=4267;sprop-parameter-sets=Z0IACvhBog==,aM44gA==
a=control:/live/347f4514-1f16-4d4f-b95f-c5bf66b41817/video
 Session=> 1df9a647-ed2d-4cec-bd00-3323284b50aa

Request=> SETUP rtsp://192.168.30.52:80/live/stream//live/347f4514-1f16-4d4f-b95
f-c5bf66b41817/video RTSP/1.0
Transport:  RTP/AVP/UDP;unicast;client_port=10000-10001;mode="PLAY"
CSeq:  3

 Session=> 1df9a647-ed2d-4cec-bd00-3323284b50aa

Response=> RTSP/1.0 200 OK
CSeq: 3
Transport: RTP/AVP;source=192.168.30.134;unicast;client_port=10000-10001;server_
port=30000-30001;ssrc=91CCD49A
Session: 610701538;timeout=30
Server: ASTI Media Server RTSP 1.0
Date: Thu, 19 Mar 2015 14:00:27 GMT

 Session=> 1df9a647-ed2d-4cec-bd00-3323284b50aa

Request=> PLAY rtsp://192.168.30.52:80/live/stream RTSP/1.0
Range:  npt=now-
CSeq:  4
Session:  610701538

 Session=> 1df9a647-ed2d-4cec-bd00-3323284b50aa

Response=> RTSP/1.0 200 OK
CSeq: 4
Session:  610701538;timeout=30
Range: npt=now-
RTP-Info: url=rtsp://192.168.30.52/live/347f4514-1f16-4d4f-b95f-c5bf66b41817/vid
eo;ssrc=91CCD49A
Server: ASTI Media Server RTSP 1.0
Date: Thu, 19 Mar 2015 14:00:28 GMT

 Session=> 1df9a647-ed2d-4cec-bd00-3323284b50aa

Media.Rtp.RtpClient-da6b5a4f-dadb-4674-a1f4-f632762ed6aa@SendRecieve - Begin
RestartFaultedStreams
DisconnectAndRemoveInactiveSessions
Thanks for your help.
Coordinator
Mar 19, 2015 at 4:25 PM
Edited Mar 19, 2015 at 4:31 PM
The functionality is there but your not using it correctly.

The encoders do not give you a rtp packet but you create a rtp packet from the data.

You should be using the packetize method that you have commented out.

The sequence numbers need to be set on packets as well as the marker and that's why it doesn't work.

The Rfc6184 media class does this for you with the packeize method.

At the very least your data should only be in the Payload of a new packet and you should not create a rtp packet from data which is not rtp.

Also ensure svc Nal units are the same format So when you do call packetize the data is packetized correctly.

If you need anything else let me know.
Marked as answer by juliusfriedman on 3/19/2015 at 8:25 AM
Mar 19, 2015 at 4:42 PM
But I'm using Packetize in my new method now:
private static void AddFrameToStream(byte[] encodedFrame)
{
            RtpFrame frame = new RtpFrame(96);
            //Create a new frame
            var newFrame = new RFC6190Media.RFC6190Frame(frame);
            newFrame.Packetize(encodedFrame);
            _stream.AddFrame(newFrame);
}
Coordinator
Mar 19, 2015 at 4:45 PM
Edited Mar 19, 2015 at 4:52 PM
No, sorry that's still wrong.

You need to call packetize on the stream with the nal unit as a parameter.

Your just making an (invalid) rtp packet then trying to get the stream to send it out.

The encoders doesn't give you the rtp header also does it?

The stream class contains the sequence number and timestamps.

Your either going to have to put them in that packet or let the class do it for you.

Is that not clear?

Your mixing creating your own packet and asking the stream class to send packets you make. You don't need the Rfc6184 stream class for anything then.

Do you understand?
Marked as answer by juliusfriedman on 3/19/2015 at 8:45 AM
Mar 19, 2015 at 5:05 PM
Edited Mar 19, 2015 at 5:35 PM
Yes i got it. Sorry I'm pretty new to RTSP and to that streaming stuff.
Yes i could receive the header from the Encoder also.

I could read this both fields:
  • pSize: Contains size of the SPS\PPs header data written by the H.264 HW encoder.
  • pBuffer: Pointer to a client allocated buffer. H.264 encoder writes SPS\PPS data to this buffer.
Thank you for your help
Coordinator
Mar 19, 2015 at 6:26 PM
Edited Mar 20, 2015 at 3:28 PM
No problem, one is glad to be of service.

Please understand the sps and pps are different from the rtp header.


If you need anything else let me know!
Marked as answer by juliusfriedman on 3/19/2015 at 10:26 AM
Mar 20, 2015 at 11:02 AM
Edited Mar 20, 2015 at 11:17 AM
Hey! I am also trying out a similar implementation, I have raw RGBA frames in a byte buffer and I want to sent it through the RSTP server.

When I invoke stream.Packetize() it throws an error at

if (thumb.PixelFormat != System.Drawing.Imaging.PixelFormat.Format32bppArgb) throw new NotSupportedException("Only ARGB is currently supported.");

My image is ARGB and i still get this error. Any help will be appreciated.
Edit: I just realized that though my image is ARGB, when i get the thumbnail
thumb = image.GetThumbnailImage(Width, Height, null, IntPtr.Zero)
the thumbnail is Format32bppPArgb.

Also that would be great if @WaldemarLittau can post your final code here

Thanks
Suneesh Mohan
Coordinator
Mar 20, 2015 at 3:27 PM
So plug in the correct conversion routine and submit a patch.

There is a converting class in Media.Image

What code have you tried?
Marked as answer by juliusfriedman on 3/20/2015 at 7:27 AM
Mar 23, 2015 at 7:20 AM
Edited Mar 23, 2015 at 7:28 AM
This is what I have tried.
Note: I am a C++ developer, and just learning C#. Please don't mind those silly errors if any.
 public void InitServer()
        {
            InitVideoBuffer();
            IPAddress[] ip = Dns.GetHostAddresses("localhost");
            server = new RtspServer(ip[0], 554);
            stream = new RFC6190Media(640, 480, "stream", null, false);
            //RtspSource source = new RtspSource("RtspSourceTest", "rtsp://localhost:8554/display");
            server.TryAddMedia(stream);
            //server.TryAddMedia(source);
            server.Start();
            EncodeAndUpdateStream();
        }

        private   void EncodeAndUpdateStream()
        {
            while (server.IsRunning)
            {
                AddFrameToStream(videoFrameBuffer);
                Thread.Sleep(50);
            }
            
            
        }

        private void AddFrameToStream(Int32[] encodedFrame)
        {          
            bmp = new Bitmap("D:\\file.bmp");
            Bitmap b = bmp.Clone(new Rectangle(0, 0, bmp.Width, bmp.Height), System.Drawing.Imaging.PixelFormat.Format32bppArgb);      
            stream.Packetize(b);
        }
Mar 23, 2015 at 10:10 AM
For me it's also still doesn't work sorry.

I pass my encoded h264 frames to this adjusted packetize method:
public void Packetize(byte[] buffer, int size)
        {
            lock (m_Frames)
            {
                byte[] finalFrame = new byte[size];
                Array.Copy(buffer, finalFrame, size);

                //Create a new frame
                var newFrame = new RFC6184Frame(96); //should all payload type to come from the media description...
                
                //Packetize the data
                newFrame.Packetize(finalFrame);

                //Add the frame
                AddFrame(newFrame);

                finalFrame = null;
            }
        }
I have I-, P- and B-Frames. First my I-Frames had no header, now I'm adding a header to each I-Frame.

What is going wrong?
Mar 23, 2015 at 10:21 AM
Edited Mar 23, 2015 at 10:29 AM
SuneeshMohan2, try to use RFC2435Media:
private static RFC2435Media _stream;
private static RtspServer _server;

static void Main(string[] args)
        {
            // initialise the application
            GetAppConfig();

            // initialise server
            InitRTSPServer(1920, 1080);

            // start frame-grabber/server thread
            ThreadStart encodeThread = EncodeAndUpdateStream; // ReceiveFromNetwork; //
            new Thread(encodeThread).Start();
        }

private static void InitRTSPServer(int width, int height)
        {
            Console.WriteLine("Initialising RTSPServer...");

            //Setup a Media.RtspServer (default port 554)
            _server = new RtspServer(listenPort: _PORT)
            {
                Logger = new Media.Rtsp.Server.RtspServerConsoleLogger()
            };

            Console.WriteLine("Creating stream with {0}x{1}...", width, height);
            _stream = new RFC2435Media("stream", null, false, width, height, false);

            _server.TryAddMedia(_stream);

            // start server
            _server.Start();

            //Wait for the server to start.
            while(false == _server.IsRunning)
                Thread.Sleep(0);

            Console.WriteLine("Server started! Listening on: {0}", _server.LocalEndPoint);
        }

private static void EncodeAndUpdateStream()
        {
            using (var bmpScreenshot = new System.Drawing.Bitmap(Screen.PrimaryScreen.Bounds.Width, 
                Screen.PrimaryScreen.Bounds.Height, System.Drawing.Imaging.PixelFormat.Format32bppArgb))
            {
                // Create a graphics object from the bitmap.
                using (var gfxScreenshot = System.Drawing.Graphics.FromImage(bmpScreenshot))
                {

                    //Could also use _stream.State.
                    while (_server.IsRunning)
                    {
                        try
                        {
                            // Take the screenshot from the upper left corner to the right bottom corner.
                            gfxScreenshot.CopyFromScreen(Screen.PrimaryScreen.Bounds.X, Screen.PrimaryScreen.Bounds.Y, 0, 0, Screen.PrimaryScreen.Bounds.Size,
                                System.Drawing.CopyPixelOperation.SourceCopy);

                            //Convert to JPEG and put in packets
                            _stream.Packetize(bmpScreenshot);

                            //REST
                            Thread.Sleep(50);
                        }
                        catch (Exception ex)
                        {
                            _server.Logger.LogException(ex);
                        }
                    }
                }
            }
        }
Coordinator
Mar 23, 2015 at 4:31 PM
Edited Mar 24, 2015 at 12:51 AM
Hello Guys,

I am going to try and making this as simple as possible and if there are further questions we can address them, then it is my goal to take all of the relevant information and create a Documentation page for this subject and to remove this thread to alleviate confusion.

Lets start here:

Rtp

Rtp is an application level protocol, it was defined in internet request for comments 1889 however it was obsoleted by RFC 3550 at a later point due to various implementations not being compatible with each-other.

For information about Rtp please see one of the following source:

https://www.ietf.org/rfc/rfc3550.txt

http://en.wikipedia.org/wiki/Real-time_Transport_Protocol

http://www.codeproject.com/Articles/507218/Managed-Media-Aggregation-using-Rtsp-and-Rtp

H.264

H.264 is a video compression technology, to compress and decompress an image one typically has an encoder to perform the compression and then a decoder to decompress the data.

For more information on H.264 see these sources:

http://en.wikipedia.org/wiki/H.264/MPEG-4_AVC

H.264 has nothing to do with Rtp and subsequently it's data does not have the Rtp Header.

This is typically also true for various other audio and video compression and as a result the IETF decides that it will define an adaptation of the required format and specify these adaptations as profiles which are specific to the media being transported.

In this case H.264 would utilize the following profile:

https://tools.ietf.org/html/rfc6184

Please note that the format is updated slightly from the original request for comments which can be found here

https://tools.ietf.org/html/rfc3984

Please keep in mind H.264 is a Bit Stream and all contained data should be though of in such a sense.

To create a Rtp H.264 frame from existing data:

One would already have a compressed data segment, also know as a Network Access Layer Unit or NALU, the unit may contain anything from small amount of a picture to meta information within the stream or an entire frame.

In the data segment there is usually a Stop Bit which indicates the end of NALU and as such another NALU can start directly after (the stop bit), when there is no Stop Bit in use one can read for the next NALU by reading for the next start code and then determine where the previous header was in the stream to determine the size of the NALU.

Once a NALU was obtained (either from a file or a encoder card or otherwise) the NALU would be passed to the Packetize method of the RFC6184 stream class, at which point it will be copied into RtpPackets and the Stop Bit will be inserted for you.

E.g.

var myH264Stream = new RFC6184Media("Name Of Stream", knownHeight, knownWidth);

From there myH264Stream generates a Sdp which contains a PPS and SPS, this is currently not variable and need to be adjusted to the profile and width you desire for your stream.

There are two ways to achieve this, to add the overloads you need in the RFC6184Stream class so that the SDP is generated correctly or derive the class and generate the SDP the way you want it.

Currently the RFC6184Stream only generates the SDP correctly for the baseline profile and also it uses a SPS and PPS which are hard-coded in that class for now.

In the future I plan on having the API available to specify the profile and correctly generate the codec specific information however there is nothing you can use as is right now, what you can do however modify the logic I have which hard coded to put your SPS and PPS required in place.

In the end the goal is to get the Rtp header on the packets and set the Marker Bit in the last RtpPacket for a frame, the SequenceNumber and Timestamp should also be set during the process.

This is where the RFC6184Stream again helps you as it can take a NALU and prepares the Rtp Header with the correct values.

As a result given a data segment of 2 bytes {0xfe, 0xff} I would pass this to myH264Stream we created above, since this NALU is small it will fit directly in a single packet

The end result will be a RtpPacket with 12 extra bytes then you passed (of the RtpHeader) and then finally any profile header and last {0xfe, 0xff} copied to the packet.

This RtpPacket's created are kept in the RFC6184Stream as a RFC6184Frame where they are re-used again and again if 'Loop' is true otherwise it is discarded after it has been transmitted.

To create a Rtp H.264 frame from data on the fly:

This is more experimental and only works when the correct image format is chosen, unless work is done to convert other image formats by calling the appropriate methods.

One would have an 'Image', I suppose from GDI, this image would be passed to the RFC6184Stream which has a 'Packetize' method for 'Image'.

If the image is too large it is re-sized, the re-sized image is encoded as an instantaneous decoder refresh with an alternating slice header. The stop bit is added.

The result is now similar to the previous rendition.

The NALU containing the Slice Header and Stop Bit is then Packetized into RtpPacket instances and for each packet created from the NALU a single RtpFrame is created and all Rtp Packets created are inserted into that frame.

Playback via RFC6184Stream:

If 'Loop' is true each iteration of the source will de-queue a RtpFrame, this frame's Sequence Number and Timestamp will be updated for all contained packets for the current iteration of the loop and the packets will be sent out to viewers.

If 'Loop' is false then the RtpFrame will be sent out as it is and this discarded as it no longer used.

Hopefully that answers your questions.
Marked as answer by juliusfriedman on 3/23/2015 at 8:31 AM