How to capture frames in a MacOS MTKView for recording to a video

Question

I have created a 3D model of a local outdoor performance space and I have an app that uses Metal on MacOS 10.14.2 to display the model. I want to create an animation by flying the camera around the scene, while recording each frame (actually every other frame for 30 fps). I know how to do the animated fly-around and I know how to create a video frame-by-frame with AVFoundation. The step for which I can find no information is how I can capture each frame.

I have a completion handler so I know when the gpu has finished each command buffer.

But what is the best way to get the image in CPU space?

I thought perhaps I could do this by attaching a second texture to colorAttachments[1] but this has resulted in some odd behavior where the original scene that used to fill my MTKView window now occupies just the upper left quadrant of the window.

What I was trying to do is write the same color to both colorAttachments[0] (my screen) and to colorAttachments[1]. In my shader I defined:

struct FragmentOut {
    float4 color0 [[ color(0) ]];
    float4 color1 [[ color(1) ]];
};

My fragment shader looks like:

fragment FragmentOut grove_fragment_function(
    VertexOut vIn [[ stage_in ]],
    constant Uniforms &uniforms [[buffer(1)]]) {
 ....
    float4 color = diffuseColor + ambientColor + specularColor;
    out.color0 = color;
    out.color1 = color;
    return out;
}

My hope was that I could then use something like:

offScreenTextureBuffer?.getBytes(buffer, bytesPerRow: 4 * w, from: MTLRegionMake2D(0, 0, w, h), mipmapLevel: 0)

to transfer the image data to a local buffer. This doesn't seem to work, plus I have the unexpected display behavior noted above.

I am configuring the offScreenTextureBuffer thus:

let pixelFormat = MTLPixelFormat.bgra8Unorm_srgb
var offScreenBufferDescriptor =  MTLTextureDescriptor.texture2DDescriptor(pixelFormat: pixelFormat,                                                                             width: 1000,                                                                             height: 600,                                                                            mipmapped: false)
offScreenBufferDescriptor.usage = [.renderTarget, .shaderRead]
offScreenTextureBuffer = device.makeTexture(descriptor: offScreenBufferDescriptor)

I also tried to use the presentation layer from my MTKView. I can get the texture from the presentation layer but when I try to use the getBytes() func on the returned texture I get an error because the buffer is configured as "framebufferonly". A presentation layer that can't be read doesn't seem particularly useful.

Any suggestions on how to capture each frame would be appreciated.

Answer 1

I've seen no answer to my question. But after lots of experimentation and further study of Apple documentation for clues I have figured out how to capture GPU generated frames.

First, I gave up on the attempt to have the shader write to the screen and an offscreen buffer in the same render pass. The odd (and undocumented as far as I can tell) behavior of MTKView in which the onscreen view occupies only a portion of the original area of the screen. I think this a bug in MTKView. It seems to act as if each attachment should appear somewhere on the screen. MTKView is not checking to see if the attachments are all on the screen.

Since MTKView calls the draw() function 60 times per second I decided to alternate. One draw() goes to the screen and the following draw() goes to my off screen buffer. This gives me a 30 Hz frame rate for the captured video frames, which is what I wanted anyway. I can watch the screen and see what will appear in the video.

For the video capture draw() calls I now define the MTLTextureDescriptor in this way:     

let offScreenPixelFormat = MTLPixelFormat.rgba32Float   // 32Float is convenient for AVFoundation video recording
let offScreenBufferDescriptor = MTLTextureDescriptor.texture2DDescriptor(pixelFormat: offScreenPixelFormat,
          width: offW, height: offH, mipmapped: false)
offScreenBufferDescriptor.usage = [.shaderWrite, .renderTarget]
offScreenBufferDescriptor.storageMode = .managed
offScreenBufferDescriptor.resourceOptions = [.storageModeManaged]
offScreenBufferDescriptor.textureType = MTLTextureType.type2D
offScreenTextureBuffer = device.makeTexture(descriptor:    offScreenBufferDescriptor)

It turns out this still did not work. I got nothing back from the GPU. That is, I got nothing back until I added this code to the end of the commandBuffer:

  

  let blitCommandEncoder = commandBuffer.makeBlitCommandEncoder()
  blitCommandEncoder?.synchronize(resource: offScreenTextureBuffer!)
  blitCommandEncoder?.endEncoding()

This causes the GPU memory version of the offScreenTextureBuffer to be copied back to the CPU memory version. I figured this out by looking at the Apple documentation. Like all Apple documentation, it doesn't tell you why you might want to synchronize. But you definitely do need to do this. I have seen several online code examples where this is NOT done and they claim to get data back from the GPU. I tried one example code without the synchronize command and nothing was returned. So I assume the requirement to synchronize is a new feature and the online code is just out of date. It would be nice if the Apple documentation made it clear this is now a requirement.

With these changes I can retrieve GPU frames. I have created a short video using GPU-generated frames.

If anyone is interested in seeing a complete code sample let me know.