新生代农民工的主页

简单了解 iOS CVPixelBuffer (上)

CVPixelBuffer
字数统计: 1.7k阅读时长: 8 min
2022/02/27

前言：

在iOS中，我们会常常看到 CVPixelBufferRef 这个类型，最常见到的场景是在Camera 采集的时候，返回的数据中有一个CMSampleBufferRef，而每个CMSampleBufferRef则包含一个 CVPixelBufferRef，在视频硬解码的返回数据里也是一个 CVPixelBufferRef（里面包含了所有的压缩的图片信息）。在了解了CVPixelBufferRef之后，我们将能够掌握并且运用CVPixelBufferRef的使用；

本篇我们主要熟悉下CVPixelBuffer的使用；

CVPixelBuffer 简介

CVPixelBuffer：核心视频像素缓冲区是在主存储器中保存像素的图像缓冲区。生成帧、压缩或解压缩视频或使用 Core Image 的应用程序都可以使用 CVPixelBuffer。

CVPixelBufferRef：是像素缓冲区类型，对CVPixelBuffer对象的引用。像素缓冲区类型基于图像缓冲区类型；像素缓冲器实现了图像缓冲器的存储器存储。

1
2
3
4
5
6
/*!
    @typedef	CVPixelBufferRef
    @abstract   Based on the image buffer type. The pixel buffer implements the memory storage for an image buffer.

*/
typedef CVImageBufferRef CVPixelBufferRef;

由于CVPixelBufferRef是C中的对象，所以没有ARC内存管理，必须由开发者自己去管理引用计数，控制对象生命周期；

首先，我们先来了解有关CVPixelBufferRef的基础方法，例如：CVPixelBufferRef的创建、持有、释放；

创建

创建的方法有四种，前三种常用到：

  • CVPixelBufferCreate()
  • CVPixelBufferCreateWithBytes()
  • CVPixelBufferCreateWithPlanarBytes()
  • CVPixelBufferCreateWithIOSurface()

普通的创建方法

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
/*!
    为给定大小和像素格式创建单个像素缓冲区
    @function   CVPixelBufferCreate
    @abstract   Call to create a single PixelBuffer for a given size and pixelFormatType.
    @discussion Creates a single PixelBuffer for a given size and pixelFormatType. It allocates the necessary memory based on the pixel dimensions, pixelFormatType and extended pixels described in the pixelBufferAttributes. Not all parameters of the pixelBufferAttributes will be used here.
    @param      width   Width of the PixelBuffer in pixels.
    @param      height  Height of the PixelBuffer in pixels.
    @param	pixelFormatType		Pixel format indentified by its respective OSType.
    @param	pixelBufferAttributes      A dictionary with additional attributes for a pixel buffer. This parameter is optional. See BufferAttributeKeys for more details.
    @param      pixelBufferOut          The new pixel buffer will be returned here
    @result	returns kCVReturnSuccess on success.
*/    
CV_EXPORT CVReturn CVPixelBufferCreate(
    CFAllocatorRef CV_NULLABLE allocator,
    size_t width,
    size_t height,
    OSType pixelFormatType,
    CFDictionaryRef CV_NULLABLE pixelBufferAttributes,
    CV_RETURNS_RETAINED_PARAMETER CVPixelBufferRef CV_NULLABLE * CV_NONNULL pixelBufferOut) __OSX_AVAILABLE_STARTING(__MAC_10_4,__IPHONE_4_0);

创建一个由内存位置指定数据的像素缓冲区

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
/*!
    为给定大小和像素格式创建一个像素缓冲区，其中包含由内存位置指定的数据
    @function CVPixelBufferCreateWithBytes
    @abstract Call to create a single PixelBuffer for a given size and pixelFormatType based on a passed in piece of memory.
    @discussion Creates a single PixelBuffer for a given size and pixelFormatType. Not all parameters of the pixelBufferAttributes will be used here. It requires a release callback function that will be called, when the PixelBuffer gets destroyed so that the owner of the pixels can free the memory.
    @param width   Width of the PixelBuffer in pixels
    @param height  Height of the PixelBuffer in pixels
    @param pixelFormatType Pixel format indentified by its respective OSType.
    @param baseAddress Address of the memory storing the pixels.
    @param bytesPerRow Row bytes of the pixel storage memory.
    @param releaseCallback         CVPixelBufferReleaseBytePointerCallback function that gets called when the PixelBuffer gets destroyed.
    @param releaseRefCon           User data identifying the PixelBuffer for the release callback.
    @param pixelBufferAttributes      A dictionary with additional attributes for a a pixel buffer. This parameter is optional. See PixelBufferAttributes for more details.
    @param pixelBufferOut          The new pixel buffer will be returned here
    @result returns kCVReturnSuccess on success.

*/

CV_EXPORT CVReturn CVPixelBufferCreateWithBytes(
    CFAllocatorRef CV_NULLABLE allocator,
    size_t width,
    size_t height,
    OSType pixelFormatType,
    void * CV_NONNULL baseAddress,
    size_t bytesPerRow,
    CVPixelBufferReleaseBytesCallback CV_NULLABLE releaseCallback,
    void * CV_NULLABLE releaseRefCon,
    CFDictionaryRef CV_NULLABLE pixelBufferAttributes,
    CV_RETURNS_RETAINED_PARAMETER CVPixelBufferRef CV_NULLABLE * CV_NONNULL pixelBufferOut) __OSX_AVAILABLE_STARTING(__MAC_10_4,__IPHONE_4_0);

创建一个平面格式的CVPixelBuffer

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
/*!
    @function   CVPixelBufferCreateWithPlanarBytes
    @abstract   Call to create a single PixelBuffer in planar format for a given size and pixelFormatType based on a passed in piece of memory.
    @discussion Creates a single PixelBuffer for a given size and pixelFormatType. Not all parameters of the pixelBufferAttributes will be used here. It requires a release callback function that will be called, when the PixelBuffer gets destroyed so that the owner of the pixels can free the memory.
    @param      width			Width of the PixelBuffer in pixels
    @param      height			Height of the PixelBuffer in pixels
    @param      pixelFormatType		Pixel format indentified by its respective OSType.
    @param	dataPtr			Pass a pointer to a plane descriptor block, or NULL.
    @param	dataSize		pass size if planes are contiguous, NULL if not.
    @param	numberOfPlanes		Number of planes.
    @param	planeBaseAddress	Array of base addresses for the planes.
    @param	planeWidth		Array of plane widths.
    @param	planeHeight		Array of plane heights.
    @param	planeBytesPerRow	Array of plane bytesPerRow values.
    @param	releaseCallback		CVPixelBufferReleaseBytePointerCallback function that gets called when the PixelBuffer gets destroyed.
    @param	releaseRefCon		User data identifying the PixelBuffer for the release callback.
    @param	pixelBufferAttributes      A dictionary with additional attributes for a a pixel buffer. This parameter is optional. See PixelBufferAttributes for more details.
    @param      pixelBufferOut          The new pixel buffer will be returned here
    @result	returns kCVReturnSuccess on success.
*/
CV_EXPORT CVReturn CVPixelBufferCreateWithPlanarBytes(
    CFAllocatorRef CV_NULLABLE allocator,
    size_t width,
    size_t height,
    OSType pixelFormatType,
    void * CV_NULLABLE dataPtr, // pass a pointer to a plane descriptor block, or NULL
    size_t dataSize, // pass size if planes are contiguous, NULL if not
    size_t numberOfPlanes,
    void * CV_NULLABLE planeBaseAddress[CV_NONNULL ],
    size_t planeWidth[CV_NONNULL ],
    size_t planeHeight[CV_NONNULL ],
    size_t planeBytesPerRow[CV_NONNULL ],
    CVPixelBufferReleasePlanarBytesCallback CV_NULLABLE releaseCallback,
    void * CV_NULLABLE releaseRefCon,
    CFDictionaryRef CV_NULLABLE pixelBufferAttributes,
    CV_RETURNS_RETAINED_PARAMETER CVPixelBufferRef CV_NULLABLE * CV_NONNULL pixelBufferOut) __OSX_AVAILABLE_STARTING(__MAC_10_4,__IPHONE_4_0);

修改

在使用 CPU 访问像素数据之前必须调用CVPixelBufferLockBaseAddress,然后再调用CVPixelBufferUnlockBaseAddress。
而使用 GPU 访问像素数据时，就没有这个必要了。

填充扩展的PixelBuffer

1
2
3
4
5
6
7
8
9
/*!
    填充扩展的PixelBuffer
    @function CVPixelBufferFillExtendedPixels
    @abstract Fills the extended pixels of the PixelBuffer.   This function replicates edge pixels to fill the entire extended region of the image.
    @param pixelBuffer Target PixelBuffer.

*/

CV_EXPORT CVReturn CVPixelBufferFillExtendedPixels( CVPixelBufferRef CV_NONNULL pixelBuffer ) __OSX_AVAILABLE_STARTING(__MAC_10_4,__IPHONE_4_0);

锁定PixelBuffer地址

1
2
3
4
5
6
7
8
9
10
11
12
13
/*!
    锁定PixelBuffer地址
    @function CVPixelBufferLockBaseAddress
    @abstract Description Locks the BaseAddress of the PixelBuffer to ensure that the memory is accessible.
    @discussion This API ensures that the CVPixelBuffer is accessible in system memory. This should only be called if the base address is going to be used and the pixel data will be accessed by the CPU.
    @param pixelBuffer Target PixelBuffer.
    @param lockFlags See CVPixelBufferLockFlags.
    @result kCVReturnSuccess if the lock succeeded, or error code on failure

*/

CV_EXPORT CVReturn CVPixelBufferLockBaseAddress( CVPixelBufferRef CV_NONNULL pixelBuffer, CVPixelBufferLockFlags lockFlags ) __OSX_AVAILABLE_STARTING(__MAC_10_4,__IPHONE_4_0);

解锁PixelBuffer地址

1
2
3
4
5
6
7
8
9
10
/*!

    @function CVPixelBufferUnlockBaseAddress
    @abstract Description Unlocks the BaseAddress of the PixelBuffer.
    @param pixelBuffer Target PixelBuffer.
    @param unlockFlags See CVPixelBufferLockFlags.
    @result kCVReturnSuccess if the unlock succeeded, or error code on failure
*/

CV_EXPORT CVReturn CVPixelBufferUnlockBaseAddress( CVPixelBufferRef CV_NONNULL pixelBuffer, CVPixelBufferLockFlags unlockFlags ) __OSX_AVAILABLE_STARTING(__MAC_10_4,__IPHONE_4_0);

内存管理（保留或者释放）

这里在开篇的时候也提到过，由于CVPixelBufferRef是C中的对象，所以不存在ARC内存管理，由开发者自己去管理引用计数，从而控制对象的生命周期；

持有

1
2
3
4
5
6
7
8
9
/*!
    引用计数+1
    @function   CVPixelBufferRetain
    @abstract   Retains a CVPixelBuffer object
    @discussion Equivalent to CFRetain, but NULL safe
    @param      buffer A CVPixelBuffer object that you want to retain.
    @result     A CVPixelBuffer object that is the same as the passed in buffer.
*/
CV_EXPORT CVPixelBufferRef CV_NULLABLE CVPixelBufferRetain( CVPixelBufferRef CV_NULLABLE texture ) __OSX_AVAILABLE_STARTING(__MAC_10_4,__IPHONE_4_0);

释放

1
2
3
4
5
6
7
8
/*!
    引用计数-1
    @function   CVPixelBufferRelease
    @abstract   Releases a CVPixelBuffer object
    @discussion Equivalent to CFRelease, but NULL safe
    @param      buffer A CVPixelBuffer object that you want to release.
*/
CV_EXPORT void CVPixelBufferRelease( CV_RELEASES_ARGUMENT CVPixelBufferRef CV_NULLABLE texture ) __OSX_AVAILABLE_STARTING(__MAC_10_4,__IPHONE_4_0);

检查

CVPixelBufferGetHeight 返回像素缓冲区的高度
CVPixelBufferGetWidth 返回像素缓冲区的宽度

这里方法较多，就不一一举例，可以查看官方文档

像素格式类型 kCVPixelFormatType

RGB :

kCVPixelFormatType_32BGRA = 'BGRA'
kCVPixelFormatType_32BGRA = 'BGRA',
kCVPixelFormatType_32ABGR = 'ABGR',  
kCVPixelFormatType_32RGBA = 'RGBA',

NV12 :

kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange = '420v',
kCVPixelFormatType_420YpCbCr8BiPlanarFullRange = '420f',

YUV420P :
kCVPixelFormatType_420YpCbCr8Planar = 'y420',

1
kCVPixelFormatType_{长度|序列}{颜色空间}{Planar|BiPlanar}{VideoRange|FullRange}

从这里类型可以看出，YUV格式和RGB格式都是可以创建成CVPixelBuffer；

那么拿到CVPixelBufferRef后，如果要显示的话， 我们可以通过CVPixelBufferRef转成UIImage，或者通过绘制纹理的形式展示。

想要更清楚的了解kCVPixelFormatType这些类型，那么就需要我们对YUV格式和RGB格式有更深入的了解，所以在下一篇中，我们将对两种格式进行详细的说明，然后再来看kCVPixelFormatType，那样将会加深我们对此的了解程度。

下一章：「简单了解 iOS CVPixelBuffer （中）」

原文作者：新生代农民工

原文链接：https://caohanchao.github.io/2022/02/27/%E7%AE%80%E5%8D%95%E4%BA%86%E8%A7%A3-iOS-CVPixelBuffer%EF%BC%88%E4%B8%8A%EF%BC%89/

发表日期：February 27th 2022, 11:13:48 pm

更新日期：February 28th 2022, 12:00:07 am

版权声明：本文采用知识共享署名-非商业性使用 4.0 国际许可协议进行许可

CATALOG
  1. 1. 前言：
  2. 2. CVPixelBuffer 简介
    1. 2.1. 创建
      1. 2.1.1. 普通的创建方法
      2. 2.1.2. 创建一个由内存位置指定数据的像素缓冲区
      3. 2.1.3. 创建一个平面格式的CVPixelBuffer
    2. 2.2. 修改
      1. 2.2.1. 填充扩展的PixelBuffer
      2. 2.2.2. 锁定PixelBuffer地址
      3. 2.2.3. 解锁PixelBuffer地址
    3. 2.3. 内存管理（保留或者释放）
      1. 2.3.1. 持有
      2. 2.3.2. 释放
    4. 2.4. 检查
  3. 3. 像素格式类型 kCVPixelFormatType
Total : 14
2022
Invalid date
2022
CocoaPods 音视频 架构 原理篇 UIStackView 布局 Objective-C CVPixelBuffer