Decoder, Viewers, Filters, etc H265, 265, etc

Topics: Suggestion
Jun 1, 2016 at 8:05 PM
So I have been taking a look at this project along with the various discussions. First off, great job as I think this has some great potential. Aside from my own immediate need I think it will work well.

This is as you've stated a LOT of work for one person. As I have an immediate need within IOS to play a simple RTSP webcam stream, it poses significant issues in getting it done. This library solves the transport issue but not the remaining decoding, viewing, etc.

As there are many platforms (IOS, Android, PC App, Web, etc) no one solution is going to fit everything. I'd like to offer my assistance in moving this forward some. What I would propose is if you're open to help, and others are possibly willing to help contribute we define some simple interfaces to start with. Then we can backfill those with independent solution for decoding and so forth. This way not tying any one solution to the project. For example, IOS we can in fact use the hardware decoder (going to take some work, but it can be done), we can also implement the software only solution cscodec to the interface, maybe sharp-ffpmeg for pc use, and so forth. I think its a big mistake to tightly bind decoding, and even viewing to any one solution.

I'm still reading and reading at this point for AVSampleBufferDisplayLayer, and some other ways within VideoToolbox available on IOS. But it would be great to have a few folks working and sharing here as the longest part is just understanding what needs to be done. Writing the code is easy.

So if this is of interest I'd like to discuss how best we can go about working together to make this happen. As a lot of folks are looking for examples which go beyond the transport aspect and I think we can quickly get some real world solutions place.
Coordinator
Jun 1, 2016 at 8:43 PM
Thank you for your interest in the project!

I am sincerely and definitely interested and would not turn down the help.

There is already an Image processing framework being developed @ ImageProcessor, it uses System.Numerics and has a good starting point for quite a few transformations and other math.

There is also CoreLabs, which is essentially a LibGD Wrapper.

And finally there is also Microsoft - FFmpegInterop which provides yet another way.

What I am after is a PURE C# solution, the Media.Codec.dll outlined what seemed to be an even better starting point that ImageProcessor and the others for the following reasons:

Any ImageFormat can be represented and transformed accurately; including but not limited to the SubSampling, Alpha Channel Depth and Bit Precision required for non contiguous elements in varying orders even if they are mixed in some way; in short my Image class conversions when written once should be applicable to any derivative of the same ImageFormat, see Media.Codecs.Image.Transformations for an example of how RGB, BGR and GBR with or without Alpha channel is supported from a single class which required no other work but can be optimized for cases which are contiguous and additionally where SIMD or other intrinsics may benefit.

Audio and Image share the same interface and backing making it quite easy to setup a WaveFormTransformation / WaveletTransformation or other desired transform which works with both Audio and Video data accordingly; In short a FastForurierTransformation should be a type of DiscreteCosineTransform (with the -> Inverse method) already baked in.

You will eventually be able to Draw in the native ImageFormat thus eliminating conversions necessary for manipulating images in formats other than the System's default ImageFormat.

Finally...

It is my eventual goal that Kernel Drivers / System Calls have a managed abstraction such that it becomes possible to:

Interact with various hardware components including but not limited to:

Intrinsics, DMA, FPU, MMU, Chipset, PCI, GPU, Sound Card etc.

Support OpenGl in software and through any supported GraphicsAdapter (even as a CoProcessor of the CentralProcessingUnit)

Take C# code and cross compile to native code of the desired MachineFunction as well as the reverse where possible(take a MachineFunction and generate a managed analogous)

Eventually allowing you to either run C# and any other machine code on the GraphicsAdapter or elsewhere such as to interact directly with them for whatever purposes may be required.

Since those are all very high level and not yet complete I will offer the following:

Let me know where you would like to start and what you would like to provide (Diagrams is possible but along with code samples)

Start taking notes or making any changes required to complete the implementation of Image and AudioBuffer.

Consider defining WaveForm yourself along with Sine and see where you get with WaveFormTransformation.

I already have an interface for Codec and a few implementations basically working; consider working on the existing Codec implementations as well as testing them, there exists the eventual need to Register all codecs in some fashion and only minimal work has been done in this area.

Consider working on an existing Issue (for Drawing and Fonts perhaps)

Something else...

Let me know what you want to do and we can go from there!
Marked as answer by juliusfriedman on 6/1/2016 at 1:46 PM
Coordinator
Jun 1, 2016 at 8:45 PM
Consider this a 'Template' for the level of functionality I envision at the Machine level to optimize things to a point where not only can managed implementations then compete with native code but pretty much they can just offload the task directly :)
#region Copyright
/*
This file came from Managed Media Aggregation, You can always find the latest version @ https://net7mma.codeplex.com/
  
 Julius.Friedman@gmail.com / (SR. Software Engineer ASTI Transportation Inc. http://www.asti-trans.com)

Permission is hereby granted, free of charge, 
 * to any person obtaining a copy of this software and associated documentation files (the "Software"), 
 * to deal in the Software without restriction, 
 * including without limitation the rights to :
 * use, 
 * copy, 
 * modify, 
 * merge, 
 * publish, 
 * distribute, 
 * sublicense, 
 * and/or sell copies of the Software, 
 * and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
 * 
 * 
 * JuliusFriedman@gmail.com should be contacted for further details.

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 
 * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. 
 * 
 * IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, 
 * DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, 
 * TORT OR OTHERWISE, 
 * ARISING FROM, 
 * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
 * 
 * v//
 */
#endregion

namespace Media.Concepts.Hardware
{
    //IsVirtualMachine := (Machine.IsVirtualMachine)

    #region Hardware Support

    /// <summary>
    /// Provides a class which can be derived to support an intrinsic which is specific to a processor.
    /// </summary>
    public abstract class ProcessorIntrinsic : Intrinsic
    {
        [System.Runtime.CompilerServices.MethodImpl(System.Runtime.CompilerServices.MethodImplOptions.AggressiveInlining)]
        public ProcessorIntrinsic(bool shouldDispose) : base(shouldDispose) { }
    }

    /// <summary>
    /// Provides a class which can be derived to support an intrinsic which is specific to the central processor.
    /// </summary>
    public abstract class CentralProcessorIntrinsic : Intrinsic
    {
        [System.Runtime.CompilerServices.MethodImpl(System.Runtime.CompilerServices.MethodImplOptions.AggressiveInlining)]
        public CentralProcessorIntrinsic(bool shouldDispose) : base(shouldDispose) { }
    }

    /// <summary>
    /// Provides a class which can be derived to support an intrinsic which is specific to a graphics processor.
    /// </summary>
    public abstract class GraphicsProcessorIntrinsic : ProcessorIntrinsic
    {
        [System.Runtime.CompilerServices.MethodImpl(System.Runtime.CompilerServices.MethodImplOptions.AggressiveInlining)]
        public GraphicsProcessorIntrinsic(bool shouldDispose) : base(shouldDispose) { }
    }

    /// <summary>
    /// Provides a class which can be derived to support an intrinsic which is specific to a sound processor.
    /// </summary>
    public abstract class SoundProcessorIntrinsic : ProcessorIntrinsic
    {
        [System.Runtime.CompilerServices.MethodImpl(System.Runtime.CompilerServices.MethodImplOptions.AggressiveInlining)]
        public SoundProcessorIntrinsic(bool shouldDispose) : base(shouldDispose) { }
    }

    /// <summary>
    /// Provides a class which can be derived to support an intrinsic which is specific to a floating point processor.
    /// </summary>
    public abstract class FloatingPointProcessorIntrinsic : ProcessorIntrinsic
    {
        [System.Runtime.CompilerServices.MethodImpl(System.Runtime.CompilerServices.MethodImplOptions.AggressiveInlining)]
        public FloatingPointProcessorIntrinsic(bool shouldDispose) : base(shouldDispose) { }
    }

    /// <summary>
    /// Provides a class which can be derived to support an intrinsic which is specific to a math processor.
    /// </summary>
    public abstract class MathProcessorIntrinsic : ProcessorIntrinsic
    {
        [System.Runtime.CompilerServices.MethodImpl(System.Runtime.CompilerServices.MethodImplOptions.AggressiveInlining)]
        public MathProcessorIntrinsic(bool shouldDispose) : base(shouldDispose) { }
    }

    /// <summary>
    /// Provides a class which can be derived to support an intrinsic which is specific to a memory processor.
    /// </summary>
    public abstract class MemoryProcessorIntrinsic : Intrinsic
    {
        [System.Runtime.CompilerServices.MethodImpl(System.Runtime.CompilerServices.MethodImplOptions.AggressiveInlining)]
        public MemoryProcessorIntrinsic(bool shouldDispose) : base(shouldDispose) { }
    }

    /// <summary>
    /// Provides a class which can be derived to support an intrinsic which is specific to a chipset.
    /// </summary>
    public abstract class ChipsetIntrinsic : Intrinsic
    {
        [System.Runtime.CompilerServices.MethodImpl(System.Runtime.CompilerServices.MethodImplOptions.AggressiveInlining)]
        public ChipsetIntrinsic(bool shouldDispose) : base(shouldDispose) { }
    }

    public abstract class DirectModeAccess : Intrinsic
    {
        [System.Runtime.CompilerServices.MethodImpl(System.Runtime.CompilerServices.MethodImplOptions.AggressiveInlining)]
        public DirectModeAccess(bool shouldDispose) : base(shouldDispose) { }
    }

    public abstract class PCIIntrinsic : Intrinsic
    {
        [System.Runtime.CompilerServices.MethodImpl(System.Runtime.CompilerServices.MethodImplOptions.AggressiveInlining)]
        public PCIIntrinsic(bool shouldDispose) : base(shouldDispose) { }
    }

    public abstract class AGPIntrinsic : Intrinsic
    {
        [System.Runtime.CompilerServices.MethodImpl(System.Runtime.CompilerServices.MethodImplOptions.AggressiveInlining)]
        public AGPIntrinsic(bool shouldDispose) : base(shouldDispose) { }
    }

    public abstract class PCIExpressIntrinsic : Intrinsic
    {
        [System.Runtime.CompilerServices.MethodImpl(System.Runtime.CompilerServices.MethodImplOptions.AggressiveInlining)]
        public PCIExpressIntrinsic(bool shouldDispose) : base(shouldDispose) { }
    }

    #endregion
}
Marked as answer by juliusfriedman on 6/1/2016 at 1:46 PM