|
|
Android MediaCodec图片合成视频
利用MediaCodec可以录制视频，可是可以将图片合成视频吗？之前使用ffmpeg来实现。但是，ffmpeg却是c++写的，而且非常占用内存，虽然它是非常棒的音视频处理库，但是杀鸡焉用牛刀，所以今天就讲一下：如何利用Android API中的MediaCodec来实现图片合成视频
YUV是为了解决彩色电视与黑白电视的兼容性。黑白视频只有Y值，也就是灰度。而彩色电视则有YUV3个分量，如果只读取Y值，就只能显示黑白画面了。YUV最大的优点在于只需占用极少的带宽。
Android MediaCodec 硬编码器封装 - https://blog.csdn.net/devil__lee/article/details/49508773
图文详解YUV420数据格式 - https://www.cnblogs.com/Sharley/p/5595768.html
如何正确使用ImageReader与YUV_420_888和MediaCodec将视频编码为h264格式？- https://stackoverrun.com/cn/q/12725625
AVFrame 与 yuv420那些事 - https://blog.csdn.net/lanxiaziyi/article/details/74139729?utm_source=blogxgwz6
YUV420P和YUV420有什么区别? - https://bbs.csdn.net/topics/80129347
Java实现的RGB转YUV420方法 - https://blog.csdn.net/u012149399/article/details/78799990
YUV简介
YUV是一种也是编码方法。比如我们常用的RGB，是指红(red)、绿(green)，蓝(blue)，而YUV则是指：
Y：亮度(Luminance、Luma)； U：色度(Chrominance )； V：浓度(Chroma)。 rgb与yuv互转：
rgb转yuv
Y = 0.299 R + 0.587 G + 0.114 B U = - 0.1687 R - 0.3313 G + 0.5 B + 128 V = 0.5 R - 0.4187 G - 0.0813 B + 128yuv转rgb
R = Y + 1.402 (V-128) G = Y - 0.34414 (U-128) - 0.71414 (V-128) B = Y + 1.772 (U-128)将图片编码为YUV格式的数据时，将对图片上的点进行采样存储。
以黑点表示采样该像素点的Y分量，以空心圆圈表示采用该像素点的UV分量
YUV420sp：YYYYYYYY UVUV 由VU顺序的不同YUV420p可分为I420和YV12，上诉例子是YV12；YUV420sp可分为 NV12与NV21，上诉例子是NV12；
1. 获取设备可渲染的颜色空间模式
由于不同手机生产商对颜色空间的渲染模式不尽相同，所以需要区别对待。不过大多是手机都是支持YUV420p、YUV420sp其中的一种。
public int[] getMediaCodecList() { //获取解码器列表 int numCodecs = MediaCodecList.getCodecCount(); MediaCodecInfo codecInfo = null; for (int i = 0; i < numCodecs && codecInfo == null; i++) { MediaCodecInfo info = MediaCodecList.getCodecInfoAt(i); if (!info.isEncoder()) { continue; String[] types = info.getSupportedTypes(); boolean found = false; //轮训所要的解码器 for (int j = 0; j < types.length && !found; j++) { if (types[j].equals("video/avc")) { found = true; if (!found) { continue; codecInfo = info; Log.d(TAG, "found" + codecInfo.getName() + "supporting" + " video/avc"); MediaCodecInfo.CodecCapabilities capabilities = codecInfo.getCapabilitiesForType("video/avc"); return capabilities.colorFormats;得到颜色空间模式后，可以判断选择其中一种来对图片进行编码：
public int getColorFormat(){ int colorFormat; int[] formats = this.getMediaCodecList(); for (int format : formats) { switch (format) { case MediaCodecInfo.CodecCapabilities.COLOR_FormatYUV420SemiPlanar: // yuv420sp colorFormat = format; break lab; case MediaCodecInfo.CodecCapabilities.COLOR_FormatYUV420Planar: // yuv420p colorFormat = format; break lab; case MediaCodecInfo.CodecCapabilities.COLOR_FormatYUV420PackedSemiPlanar: // yuv420psp colorFormat = format; break lab; case MediaCodecInfo.CodecCapabilities.COLOR_FormatYUV420PackedPlanar: // yuv420pp colorFormat = format; break lab; if (colorFormat <= 0) { colorFormat = MediaCodecInfo.CodecCapabilities.COLOR_FormatYUV420SemiPlanar; return colorFormat;然后就是设置
MediaFormat:MediaFormat mediaFormat = MediaFormat.createVideoFormat(MediaFormat.MIMETYPE_VIDEO_AVC, width, height); mediaFormat.setInteger(MediaFormat.KEY_COLOR_FORMAT, colorFormat); mediaFormat.setInteger(MediaFormat.KEY_BIT_RATE, width, * height); mediaFormat.setInteger(MediaFormat.KEY_FRAME_RATE, 16); mediaFormat.setInteger(MediaFormat.KEY_I_FRAME_INTERVAL, 10);2. rgb转YUV420p、YUV420sp、YUV420pp、YUV420psp
这里只贴出rgb转YUV420p、YUV420sp，rgb转YUV420pp和YUV420psp的代码并没有找到，只能自己写，虽然也写了，但是还没有验证过，就不贴出来了。
由于YUV420不是全采样，U和V的数据都是with*height*(1/4)，所以数据长度为：1(Y)+1/4(U)+1/4(V) = 3/2。
具体代码如下：encodeYUV420SP：
private void encodeYUV420SP(byte[] yuv420sp, int[] argb, int width, int height) { final int frameSize = width * height; int yIndex = 0; int uvIndex = frameSize; int a, R, G, B, Y, U, V; int index = 0; for (int j = 0; j < height; j++) { for (int i = 0; i < width; i++) { a = (argb[index] & 0xff000000) >> 24; // a is not used obviously R = (argb[index] & 0xff0000) >> 16; G = (argb[index] & 0xff00) >> 8; B = (argb[index] & 0xff) >> 0; // well known RGB to YUV algorithm Y = ((66 * R + 129 * G + 25 * B + 128) >> 8) + 16; V = ((-38 * R - 74 * G + 112 * B + 128) >> 8) + 128; // Previously U U = ((112 * R - 94 * G - 18 * B + 128) >> 8) + 128; // Previously V yuv420sp[yIndex++] = (byte) ((Y < 0) ? 0 : ((Y > 255) ? 255 : Y)); if (j % 2 == 0 && index % 2 == 0) { yuv420sp[uvIndex++] = (byte) ((V < 0) ? 0 : ((V > 255) ? 255 : V)); yuv420sp[uvIndex++] = (byte) ((U < 0) ? 0 : ((U > 255) ? 255 : U)); index++;encodeYUV420P:
private void encodeYUV420P(byte[] yuv420sp, int[] argb, int width, int height) { final int frameSize = width * height; int yIndex = 0; int uIndex = frameSize; int vIndex = frameSize + width * height / 4; int a, R, G, B, Y, U, V; int index = 0; for (int j = 0; j < height; j++) { for (int i = 0; i < width; i++) { a = (argb[index] & 0xff000000) >> 24; // a is not used obviously R = (argb[index] & 0xff0000) >> 16; G = (argb[index] & 0xff00) >> 8; B = (argb[index] & 0xff) >> 0; // well known RGB to YUV algorithm Y = ((66 * R + 129 * G + 25 * B + 128) >> 8) + 16; V = ((-38 * R - 74 * G + 112 * B + 128) >> 8) + 128; // Previously U U = ((112 * R - 94 * G - 18 * B + 128) >> 8) + 128; // Previously V yuv420sp[yIndex++] = (byte) ((Y < 0) ? 0 : ((Y > 255) ? 255 : Y)); if (j % 2 == 0 && index % 2 == 0) { yuv420sp[vIndex++] = (byte) ((U < 0) ? 0 : ((U > 255) ? 255 : U)); yuv420sp[uIndex++] = (byte) ((V < 0) ? 0 : ((V > 255) ? 255 : V)); index++;然后统一处理：
private byte[] getNV12(int inputWidth, int inputHeight, Bitmap scaled) { int[] argb = new int[inputWidth * inputHeight]; //Log.i(TAG, "scaled : " + scaled); scaled.getPixels(argb, 0, inputWidth, 0, 0, inputWidth, inputHeight); byte[] yuv = new byte[inputWidth * inputHeight * 3 / 2]; switch (colorFormat) { case MediaCodecInfo.CodecCapabilities.COLOR_FormatYUV420SemiPlanar: // yuv420sp encodeYUV420SP(yuv, argb, inputWidth, inputHeight); break; case MediaCodecInfo.CodecCapabilities.COLOR_FormatYUV420Planar: // yuv420p encodeYUV420P(yuv, argb, inputWidth, inputHeight); break; case MediaCodecInfo.CodecCapabilities.COLOR_FormatYUV420PackedSemiPlanar: // yuv420psp encodeYUV420PSP(yuv, argb, inputWidth, inputHeight); break; case MediaCodecInfo.CodecCapabilities.COLOR_FormatYUV420PackedPlanar: // yuv420pp encodeYUV420PP(yuv, argb, inputWidth, inputHeight); break; // scaled.recycle(); return yuv;3. 保存为mp4格式的视频
视频处理需要用到MediaMuxer：
mediaMuxer = new MediaMuxer(out.getAbsolutePath(), MediaMuxer.OutputFormat.MUXER_OUTPUT_MPEG_4);其中out为视频输出文件。
生成MediaCodec对象：
try { mediaCodec = MediaCodec.createEncoderByType(MediaFormat.MIMETYPE_VIDEO_AVC); //创建生成MP4初始化对象 mediaMuxer = new MediaMuxer(out.getAbsolutePath(), MediaMuxer.OutputFormat.MUXER_OUTPUT_MPEG_4); } catch (IOException e) { e.printStackTrace(); mediaCodec.configure(mediaFormat, null, null, MediaCodec.CONFIGURE_FLAG_ENCODE); mediaCodec.start();导出视频文件后的处理：
public void finish() { isRunning = false; if (mediaCodec != null) { mediaCodec.stop(); mediaCodec.release(); if (mediaMuxer != null) { try { if (mMuxerStarted) { mediaMuxer.stop(); mediaMuxer.release(); } catch (Exception e) { e.printStackTrace();核心处理逻辑：
public void encode(Bitmap bitmap) { final int TIMEOUT_USEC = 10000; isRunning = true; long generateIndex = 0; MediaCodec.BufferInfo info = new MediaCodec.BufferInfo(); ByteBuffer[] buffers = null; if (Build.VERSION.SDK_INT <= Build.VERSION_CODES.LOLLIPOP) { buffers = mediaCodec.getInputBuffers(); while (isRunning) { int inputBufferIndex = mediaCodec.dequeueInputBuffer(TIMEOUT_USEC); if (inputBufferIndex >= 0) { long ptsUsec = computePresentationTime(generateIndex); if (generateIndex >= mProvider.size()) { mediaCodec.queueInputBuffer(inputBufferIndex, 0, 0, ptsUsec, MediaCodec.BUFFER_FLAG_END_OF_STREAM); isRunning = false; drainEncoder(true, info); } else { if (bitmap == null) { bitmap = mProvider.next(); byte[] input = getNV12(getSize(bitmap.getWidth()), getSize(bitmap.getHeight()), bitmap); bitmap = null; //有效的空的缓存区 ByteBuffer inputBuffer = null; if (Build.VERSION.SDK_INT <= Build.VERSION_CODES.LOLLIPOP) { inputBuffer = buffers[inputBufferIndex]; } else { inputBuffer = mediaCodec.getInputBuffer(inputBufferIndex);//inputBuffers[inputBufferIndex]; inputBuffer.clear(); inputBuffer.put(input); //将数据放到编码队列 mediaCodec.queueInputBuffer(inputBufferIndex, 0, input.length, ptsUsec, 0); drainEncoder(false, info); generateIndex++; } else { Log.i(TAG, "input buffer not available"); try { Thread.sleep(50); } catch (InterruptedException e) { e.printStackTrace();其中对InputBuffer的获取做了兼容处理，mProvider是一个接口，用来获取位图。computePresentationTime的代码：
private long computePresentationTime(long frameIndex) { return 132 + frameIndex * 1000000 / mFrameRate;然后就是buffer输出：
private void drainEncoder(boolean endOfStream, MediaCodec.BufferInfo bufferInfo) { final int TIMEOUT_USEC = 10000; ByteBuffer[] buffers = null; if (Build.VERSION.SDK_INT <= Build.VERSION_CODES.LOLLIPOP) { buffers = mediaCodec.getOutputBuffers(); if (endOfStream) { try { mediaCodec.signalEndOfInputStream(); } catch (Exception e) { while (true) { int encoderStatus = mediaCodec.dequeueOutputBuffer(bufferInfo, TIMEOUT_USEC); if (encoderStatus == MediaCodec.INFO_TRY_AGAIN_LATER) { if (!endOfStream) { break; // out of while } else { Log.i(TAG, "no output available, spinning to await EOS"); } else if (encoderStatus == MediaCodec.INFO_OUTPUT_FORMAT_CHANGED) { if (mMuxerStarted) { throw new RuntimeException("format changed twice"); MediaFormat mediaFormat = mediaCodec.getOutputFormat(); mTrackIndex = mediaMuxer.addTrack(mediaFormat); mediaMuxer.start(); mMuxerStarted = true; } else if (encoderStatus < 0) { Log.i(TAG, "unexpected result from encoder.dequeueOutputBuffer: " + encoderStatus); } else { ByteBuffer outputBuffer = null; if (Build.VERSION.SDK_INT <= Build.VERSION_CODES.LOLLIPOP) { outputBuffer = buffers[encoderStatus]; } else { outputBuffer = mediaCodec.getOutputBuffer(encoderStatus); if (outputBuffer == null) { throw new RuntimeException("encoderOutputBuffer " + encoderStatus + " was null"); if ((bufferInfo.flags & MediaCodec.BUFFER_FLAG_CODEC_CONFIG) != 0) { Log.d(TAG, "ignoring BUFFER_FLAG_CODEC_CONFIG"); bufferInfo.size = 0; if (bufferInfo.size != 0) { if (!mMuxerStarted) { throw new RuntimeException("muxer hasn't started"); // adjust the ByteBuffer values to match BufferInfo outputBuffer.position(bufferInfo.offset); outputBuffer.limit(bufferInfo.offset + bufferInfo.size); Log.d(TAG, "BufferInfo: " + bufferInfo.offset + "," + bufferInfo.size + "," + bufferInfo.presentationTimeUs); try { mediaMuxer.writeSampleData(mTrackIndex, outputBuffer, bufferInfo); } catch (Exception e) { Log.i(TAG, "Too many frames"); mediaCodec.releaseOutputBuffer(encoderStatus, false); if ((bufferInfo.flags & MediaCodec.BUFFER_FLAG_END_OF_STREAM) != 0) { if (!endOfStream) { Log.i(TAG, "reached end of stream unexpectedly"); } else { Log.i(TAG, "end of stream reached"); break; // out of while如果看过其他类似博客的代码，就会发现代码其实差不多，但是却不是我想要的，所以只能自己改。其中一些博客还是写的非常棒，从中学到了很多。
这篇文章讲的是利用纯Android API实现的图片合成视频文件，其中我有查询到利用ffmpeg的，利用opencv/javacv的，但是这边文章介绍的方式没有引用第三方库，因此打包出来的apk文件肯定是很小的。
为了解决这个问题查了不少资料，花了不少时间，不过还好，终于搞定了。