RFC6184 Media Issue - Fragmented Nal Unit Type

Topics: Bug Archive
Feb 6, 2016 at 10:22 AM
Hi,

I'd just like to say that the work you are doing on this project is amazing and well appreciated.

I'm working on implementing the RTSP Client for a h264 stream coming from my IP Camera. Specifically, I am attempting to pull the image data out to grab images from the stream. I really only care about the IDR frames at this point.

Within the RFC6194Frame class, it is clear that the Nal types are being extracted from the RTP stream. It looks like both the RTP Nal and encapsulated h264 NAL are being extracted into a m_containednaltypes variable for quick analysis of the frame with various properties.

So through the m_containednaltypes I should be able to check the ContainsInstantaneousDecoderRefresh property to see if the frame is an IDR frame.

On debugging, I've found that I keep getting odd values in the Nal bytes. Most of my frames are coming back as fragemented. I've attached an image below.

It looks like the line:
byte nalHeader = (byte)((firstByte & 0xE0) | (FUHeader & Common.Binary.FiveBitMaxValue));

should actually be:
byte nalHeader = (FUHeader & Common.Binary.FiveBitMaxValue));

under where the packet is being processed as fragmented.

I've reviewed section 5.8 of the below article which I think corresponds with this:
https://tools.ietf.org/html/rfc3984#section-5.8

I'm very new to media processing - sorry if I got anything wrong - appreciate the help and work you are doing on this project!

Image

Regards,
Aron
Feb 7, 2016 at 12:02 AM
I'm going to amend my last post. Changing that line is not correct, please disregard.

It seems that the real problem is that the properties checking the nal type are doing a straight equals comparison instead of only checking the last 5 bits of the byte. It really doesn't matter what the first 3 bits of the byte are when added to m_containedNalTypes.

So in other words, the below code:
public bool ContainsPictureParameterSet
        {
            get
            {
                return m_ContainedNalTypes.Any(t => t == Media.Codecs.Video.H264.NalUnitType.PictureParameterSet);
            }
        }

        public bool ContainsInstantaneousDecoderRefresh
        {
            get
            {
                return m_ContainedNalTypes.Any(t => t == Media.Codecs.Video.H264.NalUnitType.InstantaneousDecoderRefresh);
            }
        }
Should change to:
        public bool ContainsPictureParameterSet
        {
            get
            {
                return m_ContainedNalTypes.Any(t => (t & Common.Binary.FiveBitMaxValue) == Media.Codecs.Video.H264.NalUnitType.PictureParameterSet);
            }
        }

        public bool ContainsInstantaneousDecoderRefresh
        {
            get
            {
                return m_ContainedNalTypes.Any(t => (t & Common.Binary.FiveBitMaxValue) == Media.Codecs.Video.H264.NalUnitType.InstantaneousDecoderRefresh);
            }
        }
And so on...
Coordinator
Feb 8, 2016 at 6:57 PM
Thanks for bringing this up.

I think rather than changing all the properties it's probably just best to mask the bits out before proceeding in the code.

//First 3 bits don't matter
nalHeader &= Common.Binary.FiveBitMaxValue;

//Store the nalType contained
m_ContainedNalTypes.Add(nalHeader);


I will make an update later today with the same code fix.
Marked as answer by juliusfriedman on 2/8/2016 at 11:57 AM
Feb 11, 2016 at 9:00 AM
Hi Julius,

This approach has introduced a bug. Both the 5 and 8 bit versions are needed in that block of code:
Image
Coordinator
Feb 11, 2016 at 7:04 PM
Yea, that's my bad, I probably should have only applied the mask when adding the NAL Header to ContainedNalTypes and left the byte alone in the stream being written.

There are also another few optimizations I wanted to make to re-use code in those blocks, I will see if I can get them posted today or tomorrow.

I should also have some much improved code for handling the NAL writing as well as some other goodies fairly soon so keep your eyes out an thanks again for bringing this up as well as your interest in the library.
Marked as answer by juliusfriedman on 2/11/2016 at 12:04 PM
Coordinator
Feb 11, 2016 at 10:46 PM
I just checked in #111886 which should fix this and a few other issues.

Let me know if your still having problems!
Marked as answer by juliusfriedman on 2/11/2016 at 3:46 PM