Working with raspivid for low latency and other issues with compression

So after I got ScaViSLAM working, I had an issue with getting my raspi stereo camera working at 640x480 as expected (I wanted to use 640x480 because the perf at 1920x1080 was too slow, but the perf would probably get better in a release build). The issue was that I was getting bursts of 17 frames in the stream. Instead of the h264 stream from the pi delivering one frame as often as possible, it was sending 17 frames in one packet. This meant that there was a latency of 33ms*17=500ms. This is not so usable. I eventually analyzed the H264 stream by detecting the NAL markers, which showed that the H264 decoder was decoding 17 frames for a I frame packet (buffer starting with 0x00000125). An I frame is a single frame that is only compressed against itself, there is no dependency to previous frames. A good reference for the H264 basics is
http://keyj.emphy.de/files/projects/SA_paper.pdf. I don't know if another decoder could handle this better or if this is normal, but then the solution was to increase the frequency of the I frames. using "raspivid -g 1" the I frames are then sent every frame. Then the problem is solved because then I receive as new a frame as fast possible and my video queue is always at most 1 deep (unless whatever algorithm that is running on the server is too slow to consume the stream).

I added code in
https://github.com/arcanon/raspbot/blob/master/h264decoder.h
to detect the NAL markers and also check how full the frame queue was.

 if ((data[i] == 0) &&

     (data[i+1] == 0) &&

     (data[i+2] == 0) &&


    (data[i+3] == 1))


{


      NALCount++;


      nalFound = true;


      ....


}

and


if ((m_reader->framesInQueue()- start) > 1)


{


   falseCount+= (m_reader->framesInQueue()- start);


   printf("%s, error frames more %d size %d fa %d, nc %d fc %d\n", this->m_vid_server.c_str(), (m_reader->framesInQueue()- start), prevSize, framesAdded, NALCount, falseCount);    }