4

I am grabbing video from a web cam using Microsoft Media Foundation. The image data is in YUY2 format. The pixels are packed in a 4:2:2 ratio. The color format is YCbCr. The color data gets repeated 2x. Ex the packed pixel data is:

y0 u y1 v

Gets unpacked to 2 pixels:

y0 u v and y1 u v

I want to use this data as an OpenGL texture. The question is: is there a way to pass this type of texture to glTexImage2D as-is? Or do I need to unpack it first? I would like you keep the YCbCr color and convert to RGBA in a shader.

(I know I can use OpenCV to get video - I've already done that. I am trying a more low level solution mostly as a learning experience).

Johnny Mopp
  • 165
  • 4
  • Normally you would use YUV semi-planar for this, which is a seperate single channel Y texture and a packed UV channel. You could also unpack inside a fragment shader by testing if the Y texel you are reading is on a odd or even column. So say you read Y0,U for the first pixel (Assuming a 2 channel texture), you test if Y0 is on a even-column, which means you need to read the texel to right to get the missing V. For odd-columns you would read the texel to the left to get the missing U. – PaulHK Jul 21 '16 at 04:36
  • Just did a little googling, you want NV12 pixel format in Microsoft Media Foundation if you want the semi-planar format. Should be the fastest format for doing colour space conversion on the GPU. – PaulHK Jul 21 '16 at 09:04
  • @PaulHK The camera only supports YUY2 and MJPG formats. I tried unpacking in the shader - the results were close to correct but there were vertical lines. I assume this is because I am incorrectly fetching the next pixel: `texture(tex, vec2(Texcoord.x + 1, Texcoord.y));` I'm guessing this is due to scaling. – Johnny Mopp Jul 22 '16 at 19:57
  • It might be simpler to deinterleave it into 2 buffers on the CPU ( Y & CrCb ), then the GPU can do CSC. – PaulHK Jul 25 '16 at 02:25

1 Answers1

3

The question is: is there a way to pass this type of texture to glTexImage2D as-is?

Yes; you can just pass it as-is, but (looking at the docs) it is not supported directly. You'll need a custom shader to interpret it.

Depending on your needs, expanding the pixels first (or even doing the full conversion CPU-side) may be worthwhile. At a guess, I'd say that expanding the pixels (as you copy from the camera's driver buffer into a staging buffer for upload to the GPU) is a pretty consistently good idea, unless you're extremely bandwidth-saturated. I'd add the color-space conversion for easier debugging, or if the image doesn't update frequently.

imallett
  • 1,870
  • 10
  • 23
  • 1
    Yes, I've decided it's best to unpack the data on the CPU (a task that lends itself well to multi-threading), then do color conversion in a shader. – Johnny Mopp Jul 27 '16 at 14:20