Consume stream from ffmpeg

Topics: Question
Dec 9, 2014 at 5:19 PM
Edited Dec 9, 2014 at 5:22 PM
I have a project where I am using ffmpeg to combine two live input streams (one is mjpeg video, one is PCM audio) and transcode them to a single h264/mp3 output. ffmpeg is the only program I could find that would accomplish this easily.

But I recently learned that ffmpeg itself does not act as a streaming server. i.e. ffmpeg can't host an rtsp stream so Managed Media Aggregation can't initiate a connection to ffmpeg to pull the stream. Unless I missed something... the ffmpeg designers want people to use ffserver to accomplish this, but ffserver is not available on Windows so I can't use it for my project.

However, ffmpeg is apparently capable of pushing out a stream in a variety of ways (udp, tcp, rtsp push?), as long as there is a streaming server listening for it.

Can Managed Media Aggregation listen for and accept one of these formats that ffmpeg will push? I don't really care which as long as it doesn't add a lot of latency and supports h264 video.
Coordinator
Dec 10, 2014 at 5:01 AM
Hello Sir,

Please see https://net7mma.codeplex.com/discussions/570380

I think FFMPEG can produce a 'SDP' which you can then feed directly into the RtpClient.FromSessionDescription function.

The Rtsp Push thing I have heard of before but I will not implement support for it because there is no reason for it, its a FFMPEG thing and should be treated as such.

The VLC and FFMPEG Clients already suffer from several security flaws related to the RTSP parsing and allowing a client to send in framed data is going to make that a much bigger problem especially since you can get directly into memory with the video decoder and then do whatever you want if you can overrun the decoder.

Furthermore RFC2326 / Rtsp itself doesn't explicitly say what to do when you received less data then expected and the remaining data is framed as new data (not the continuation to the old data)

It also doesn't say what to do when you have MORE bytes then received although you can probably ignore them, they could also be the next frame start or a RTSP interleaved request...

The whole standard is a bit funky in that regard and it's because the Big Endian Frame Control character is 0x36 which doesn't fit the exclusions set forth by the standard itself.

e.g. 0xC3 doesn't escape 0x24 and thus it becomes a bit of a cluster because RTSP and RTCP use the same encoding.

None the less if you just do exactly what the ALF states each time your program can be stable and secure, you just can't assume anything.

So in short, PUSH RTSP will never happen in this library and should be eliminated from others just as UDP Rtsp is being eliminated from the standard.

One other option you will have is 'udp' support in the near future, I will be working on the Mpeg Demxuing in the next few releasesm there is already a reader for Transport Streams, Packetized Elementary Streams and Program Streams, the next step would be to read the data from the network rather than a file.

If you need anything else let me know!
Marked as answer by juliusfriedman on 12/9/2014 at 9:01 PM
Dec 10, 2014 at 6:23 AM
Thanks for the response. I had read that thread before posting here, and most of that dealing with 'SDP' is over my head. I have no idea how to make ffmpeg produce this SDP file :) But that is okay, I wasn't even able to get ffmpeg to do any network streaming, even to VLC (for all I know it was VLC's fault though).

Luckily I can have ffmpeg write to stdout and pipe that into VLC which works. I am not confident that VLC will be able to handle my streaming needs, hopefully without adding too much latency.

Anyway, I am amazed by how quickly you have been developing Managed Media Aggregation! Amazing work!
Coordinator
Dec 10, 2014 at 6:36 AM
No problem,

You see when you create the stream with FFMPEG you usually have to tell it you want the SDP.

http://stackoverflow.com/questions/9914437/how-to-generate-sdp-file-from-ffmpeg

http://stackoverflow.com/questions/15712983/why-rtp-streaming-of-a-avi-video-file-fails-to-be-received/16469378#16469378

Once you do that you can give the contents of the SDP to my library and then you can consume it or host it.

Thanks for the kind words, I am doing my best and after the new year I plan on having the Demuxer release out the door which will be followed by Muxer.

Let me know if you need anything else!
Dec 10, 2014 at 3:29 PM
Interesting. I can get FFMPEG to produce the SDP but only when I take out my audio stream, otherwise I get this error message:

[rtp @ 00000000055300a0] Only one stream supported in the RTP muxer

I refuse to believe RTP does not support video with audio so I must still be doing something wrong. This is the command I feed it:
ffmpeg -f mjpeg -i http://localhost:18080/image.mjpg -ar 48000 -ac 2 -f s32be -i http://localhost:18080/audio.wav -vcodec libx264 -b:v 2048k -acodec libvo_aacenc -b:a 128k -f rtp rtp://127.0.0.1:1234
and this is the output:
E:\hdmi\ffmpeg-20141208-git-208c03a-win64-static\bin>ffmpeg -f mjpeg -i http://localhost:18080/image
.mjpg -ar 48000 -ac 2 -f s32be -i http://localhost:18080/audio.wav -vcodec libx264 -b:v 2048k -acode
c libvo_aacenc -b:a 128k -f rtp rtp://127.0.0.1:1234
ffmpeg version N-68289-g208c03a Copyright (c) 2000-2014 the FFmpeg developers
  built on Dec  7 2014 22:13:45 with gcc 4.9.2 (GCC)
  configuration: --enable-gpl --enable-version3 --disable-w32threads --enable-avisynth --enable-bzli
b --enable-fontconfig --enable-frei0r --enable-gnutls --enable-iconv --enable-libass --enable-libblu
ray --enable-libbs2b --enable-libcaca --enable-libfreetype --enable-libgme --enable-libgsm --enable-
libilbc --enable-libmodplug --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrw
b --enable-libopenjpeg --enable-libopus --enable-librtmp --enable-libschroedinger --enable-libsoxr -
-enable-libspeex --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvo-aacenc --
enable-libvo-amrwbenc --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enab
le-libx264 --enable-libx265 --enable-libxavs --enable-libxvid --enable-decklink --enable-zlib
  libavutil      54. 15.100 / 54. 15.100
  libavcodec     56. 14.100 / 56. 14.100
  libavformat    56. 15.102 / 56. 15.102
  libavdevice    56.  3.100 / 56.  3.100
  libavfilter     5.  2.103 /  5.  2.103
  libswscale      3.  1.101 /  3.  1.101
  libswresample   1.  1.100 /  1.  1.100
  libpostproc    53.  3.100 / 53.  3.100
Input #0, mjpeg, from 'http://localhost:18080/image.mjpg':
  Duration: N/A, bitrate: N/A
    Stream #0:0: Video: mjpeg, yuvj422p(pc, bt470bg/unknown/unknown), 1280x720 [SAR 1:1 DAR 16:9], 2
5 tbr, 1200k tbn, 25 tbc
Guessed Channel Layout for  Input Stream #1.0 : stereo
Input #1, s32be, from 'http://localhost:18080/audio.wav':
  Duration: N/A, bitrate: 3072 kb/s
    Stream #1:0: Audio: pcm_s32be, 48000 Hz, 2 channels, s32, 3072 kb/s
No pixel format specified, yuvj422p for H.264 encoding chosen.
Use -pix_fmt yuv420p for compatibility with outdated media players.
[libx264 @ 00000000057ce200] using SAR=1/1
[libx264 @ 00000000057ce200] using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX
[libx264 @ 00000000057ce200] profile High 4:2:2, level 3.1, 4:2:2 8-bit
[rtp @ 00000000003b2420] Only one stream supported in the RTP muxer
Output #0, rtp, to 'rtp://127.0.0.1:1234':
  Metadata:
    encoder         : Lavf56.15.102
    Stream #0:0: Video: h264 (libx264), yuvj422p(pc), 1280x720 [SAR 1:1 DAR 16:9], q=-1--1, 2048 kb/
s, 25 fps, 25 tbn, 25 tbc
    Metadata:
      encoder         : Lavc56.14.100 libx264
    Stream #0:1: Audio: aac (libvo_aacenc), 48000 Hz, stereo, s16 (32 bit), 128 kb/s
    Metadata:
      encoder         : Lavc56.14.100 libvo_aacenc
Stream mapping:
  Stream #0:0 -> #0:0 (mjpeg (native) -> h264 (libx264))
  Stream #1:0 -> #0:1 (pcm_s32be (native) -> aac (libvo_aacenc))
Could not write header for output file #0 (incorrect codec parameters ?): Error number -22 occurred
[libx264 @ 00000000057ce200] final ratefactor: 22.16
but when I remove the audio stream, like so:
ffmpeg -f mjpeg -i http://localhost:18080/image.mjpg -an -vcodec libx264 -b:v 2048k -f rtp rtp://127.0.0.1:1234
then I get the sdp in the output:
E:\hdmi\ffmpeg-20141208-git-208c03a-win64-static\bin>ffmpeg -f mjpeg -i http://localhost:18080/image
.mjpg -an -vcodec libx264 -b:v 2048k -f rtp rtp://127.0.0.1:1234
ffmpeg version N-68289-g208c03a Copyright (c) 2000-2014 the FFmpeg developers
  built on Dec  7 2014 22:13:45 with gcc 4.9.2 (GCC)
  configuration: --enable-gpl --enable-version3 --disable-w32threads --enable-avisynth --enable-bzli
b --enable-fontconfig --enable-frei0r --enable-gnutls --enable-iconv --enable-libass --enable-libblu
ray --enable-libbs2b --enable-libcaca --enable-libfreetype --enable-libgme --enable-libgsm --enable-
libilbc --enable-libmodplug --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrw
b --enable-libopenjpeg --enable-libopus --enable-librtmp --enable-libschroedinger --enable-libsoxr -
-enable-libspeex --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvo-aacenc --
enable-libvo-amrwbenc --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enab
le-libx264 --enable-libx265 --enable-libxavs --enable-libxvid --enable-decklink --enable-zlib
  libavutil      54. 15.100 / 54. 15.100
  libavcodec     56. 14.100 / 56. 14.100
  libavformat    56. 15.102 / 56. 15.102
  libavdevice    56.  3.100 / 56.  3.100
  libavfilter     5.  2.103 /  5.  2.103
  libswscale      3.  1.101 /  3.  1.101
  libswresample   1.  1.100 /  1.  1.100
  libpostproc    53.  3.100 / 53.  3.100
Input #0, mjpeg, from 'http://localhost:18080/image.mjpg':
  Duration: N/A, bitrate: N/A
    Stream #0:0: Video: mjpeg, yuvj422p(pc, bt470bg/unknown/unknown), 1280x720 [SAR 1:1 DAR 16:9], 2
5 tbr, 1200k tbn, 25 tbc
No pixel format specified, yuvj422p for H.264 encoding chosen.
Use -pix_fmt yuv420p for compatibility with outdated media players.
[libx264 @ 0000000002c02e60] using SAR=1/1
[libx264 @ 0000000002c02e60] using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX
[libx264 @ 0000000002c02e60] profile High 4:2:2, level 3.1, 4:2:2 8-bit
Output #0, rtp, to 'rtp://127.0.0.1:1234':
  Metadata:
    encoder         : Lavf56.15.102
    Stream #0:0: Video: h264 (libx264), yuvj422p(pc), 1280x720 [SAR 1:1 DAR 16:9], q=-1--1, 2048 kb/
s, 25 fps, 90k tbn, 25 tbc
    Metadata:
      encoder         : Lavc56.14.100 libx264
Stream mapping:
  Stream #0:0 -> #0:0 (mjpeg (native) -> h264 (libx264))
SDP:
v=0
o=- 0 0 IN IP4 127.0.0.1
s=No Name
c=IN IP4 127.0.0.1
t=0 0
a=tool:libavformat 56.15.102
m=video 1234 RTP/AVP 96
b=AS:2048
a=rtpmap:96 H264/90000
a=fmtp:96 packetization-mode=1

Press [q] to stop, [?] for help
frame=  701 fps= 30 q=-1.0 Lsize=      35kB time=00:00:27.96 bitrate=  10.2kbits/s
video:29kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 19.435801%
[libx264 @ 0000000002c02e60] frame I:3     Avg QP: 2.67  size:   489
[libx264 @ 0000000002c02e60] frame P:176   Avg QP: 0.16  size:    47
[libx264 @ 0000000002c02e60] frame B:522   Avg QP: 0.30  size:    39
[libx264 @ 0000000002c02e60] consecutive B-frames:  0.7%  0.0%  0.0% 99.3%
[libx264 @ 0000000002c02e60] mb I  I16..4: 100.0%  0.0%  0.0%
[libx264 @ 0000000002c02e60] mb P  I16..4:  0.0%  0.0%  0.0%  P16..4:  0.0%  0.0%  0.0%  0.0%  0.0%
   skip:100.0%
[libx264 @ 0000000002c02e60] mb B  I16..4:  0.0%  0.0%  0.0%  B16..8:  0.0%  0.0%  0.0%  direct: 0.0
%  skip:100.0%
[libx264 @ 0000000002c02e60] final ratefactor: -32.49
[libx264 @ 0000000002c02e60] 8x8 transform intra:0.0%
[libx264 @ 0000000002c02e60] coded y,uvDC,uvAC intra: 0.0% 0.0% 0.0% inter: 0.0% 0.0% 0.0%
[libx264 @ 0000000002c02e60] i16 v,h,dc,p: 98%  0%  2%  0%
[libx264 @ 0000000002c02e60] i8c dc,h,v,p: 100%  0%  0%  0%
[libx264 @ 0000000002c02e60] Weighted P-Frames: Y:0.0% UV:0.0%
[libx264 @ 0000000002c02e60] kb/s:8.53
Any ideas?
Coordinator
Dec 10, 2014 at 4:18 PM
Honestly no, but if you make a post on stack overflow its very likely to get answers one of which probably will be intelligent and have an explanation and answer.

I know that my library supports mjpeg completely and you can technically mix in the audio very easily with my library.

Just create a rfc2435 stream, configure an additional media description and supply the data ( which can come from another stream).

I have tested this to combine multiple audio amd video tracks and as long as the sdp is valid it works.

Lip synchronization works because each context has its own ntp timestamp which gets set per packet so the receiver never knows the difference.

If you need anything further let me know.
Marked as answer by juliusfriedman on 12/10/2014 at 8:18 AM