背景

隨著智能門禁等物聯網產品的普及，越來越多的開發者對音視頻互動體驗提出了更高的要求。目前市面上大多一對一互動都是基于WebRTC，優點不再贅述，我們這里先說說可能需要面臨的問題：WebRTC的服務器部署非常復雜，可以私有部署，但是非常復雜。傳輸基于UDP，很難保證傳輸質量，由于UDP是不可靠的傳輸協議，在復雜的公網網絡環境下，各種突發流量、偶爾的傳輸錯誤、網絡抖動、超時等等都會引起丟包異常，都會在一定程度上影響音視頻通信的質量，難以應對復雜的互聯網環境，如跨區跨運營商、低帶寬、高丟包等場景，行話說的好：從demo到實用,中間還差1萬個WebRTC。

其他技術方案

內網環境下的RTSP輕量級服務；

基于RTMP的公網或內網技術方案。

本方案系基于現有RTMP或內置RTSP服務、RTMP/RTSP直播播放模塊，產品穩定度高，在保證超低延遲的基礎上，加入噪音抑制、回音消除、自動增益控制等特性，確保通話體驗（如需更好的消除效果，亦可考慮如麥克風陣列等技術方案），采用通用的RTMP服務器（如nginx、SRS）或自身的輕量級RTSP服務，更有利于私有部署，便于支持H.264的擴展SEI消息發送機制，方便擴展特定機型H.265編碼支持。

技術實現

廢話不多說，先上圖：

關鍵demo代碼說明：

拉流播放：

btnPlaybackStartStopPlayback.setOnClickListener(new Button.OnClickListener() { // @Override public void onClick(View v) { if(isPlaybackViewStarted){Log.i(PLAY_TAG, "Stop playback stream++");btnPlaybackStartStopPlayback.setText("開始播放 ");//btnPopInputText.setEnabled(true);btnPlaybackPopInputUrl.setEnabled(true);btnPlaybackHardwareDecoder.setEnabled(true);btnPlaybackSetPlayBuffer.setEnabled(true);btnPlaybackFastStartup.setEnabled(true);if ( playerHandle != 0 ){libPlayer.SmartPlayerStopPlay(playerHandle);libPlayer.SmartPlayerClose(playerHandle);playerHandle = 0;}isPlaybackViewStarted = false;Log.i(PLAY_TAG, "Stop playback stream--");}else{Log.i(PLAY_TAG, "Start playback stream++");playerHandle = libPlayer.SmartPlayerOpen(curContext);if(playerHandle == 0){Log.e(PLAY_TAG, "surfaceHandle with nil..");return;}libPlayer.SetSmartPlayerEventCallbackV2(playerHandle,new EventHandePlayerV2());libPlayer.SmartPlayerSetSurface(playerHandle, playerSurfaceView); //if set the second param with null, it means it will playback audio only..// libPlayer.SmartPlayerSetSurface(playerHandle, null); libPlayer.SmartPlayerSetRenderScaleMode(playerHandle, 1);// External Render test//libPlayer.SmartPlayerSetExternalRender(playerHandle, new RGBAExternalRender());//libPlayer.SmartPlayerSetExternalRender(playerHandle, new I420ExternalRender());libPlayer.SmartPlayerSetExternalAudioOutput(playerHandle, new PlayerExternalPcmOutput());libPlayer.SmartPlayerSetAudioOutputType(playerHandle, 1);libPlayer.SmartPlayerSetBuffer(playerHandle, playbackBuffer);libPlayer.SmartPlayerSetFastStartup(playerHandle, isPlaybackFastStartup?1:0);if ( isPlaybackMute ){libPlayer.SmartPlayerSetMute(playerHandle, isPlaybackMute?1:0);}if (isPlaybackHardwareDecoder) {int isSupportHevcHwDecoder = libPlayer.SetSmartPlayerVideoHevcHWDecoder(playerHandle,1);int isSupportH264HwDecoder = libPlayer.SetSmartPlayerVideoHWDecoder(playerHandle,1);Log.i(TAG, "isSupportH264HwDecoder: " + isSupportH264HwDecoder + ", isSupportHevcHwDecoder: " + isSupportHevcHwDecoder);}if( playbackUrl == null ){Log.e(PLAY_TAG, "playback URL with NULL..."); return;}libPlayer.SmartPlayerSetAudioVolume(playerHandle, curAudioVolume);libPlayer.SmartPlayerSetUrl(playerHandle, playbackUrl);int iPlaybackRet = libPlayer.SmartPlayerStartPlay(playerHandle);if( iPlaybackRet != 0 ){libPlayer.SmartPlayerClose(playerHandle);playerHandle = 0;Log.e(PLAY_TAG, "StartPlayback strem failed.."); return;}btnPlaybackStartStopPlayback.setText("停止播放 ");btnPlaybackPopInputUrl.setEnabled(false);btnPlaybackHardwareDecoder.setEnabled(false);btnPlaybackSetPlayBuffer.setEnabled(false);btnPlaybackFastStartup.setEnabled(false);isPlaybackViewStarted = true;Log.i(PLAY_TAG, "Start playback stream--");}}});

拉流端實時音量調節：

audioVolumeBar.setOnSeekBarChangeListener(new SeekBar.OnSeekBarChangeListener() {@Overridepublic void onProgressChanged(SeekBar seekBar, int i, boolean b) {}@Overridepublic void onStartTrackingTouch(SeekBar seekBar) {//Log.i(TAG, "開始拖動");}@Overridepublic void onStopTrackingTouch(SeekBar seekBar) {Log.i(TAG, "停止拖動, CurProgress: " + seekBar.getProgress());curAudioVolume = seekBar.getProgress();audioVolumeText.setText("當前音量: " + curAudioVolume);if(playerHandle != 0){libPlayer.SmartPlayerSetAudioVolume(playerHandle, curAudioVolume);}}});}

回調后的PCM數據，傳給推送端，用于音頻處理

class NTAudioRecordV2CallbackImpl implements NTAudioRecordV2Callback{@Overridepublic void onNTAudioRecordV2Frame(ByteBuffer data, int size, int sampleRate, int channel, int per_channel_sample_number){/*Log.i(TAG, "onNTAudioRecordV2Frame size=" + size + " sampleRate=" + sampleRate + " channel=" + channel+ " per_channel_sample_number=" + per_channel_sample_number);*/if ( (isPushingRtmp || isRTSPPublisherRunning) && publisherHandle != 0 ){libPublisher.SmartPublisherOnPCMData(publisherHandle, data, size, sampleRate, channel, per_channel_sample_number);}}} class PlayerExternalPcmOutput implements NTExternalAudioOutput{ private int sample_rate_ = 0;private int channel_ = 0;private int sample_size = 0;private int buffer_size = 0;private ByteBuffer pcm_buffer_ = null;@Overridepublic ByteBuffer getPcmByteBuffer(int size){//Log.i("getPcmByteBuffer", "size: " + size);if(size < 1){return null;}if(buffer_size != size){buffer_size = size;pcm_buffer_ = ByteBuffer.allocateDirect(buffer_size);}return pcm_buffer_;}public void onGetPcmFrame(int ret, int sampleRate, int channel, int sampleSize, int is_low_latency){/*Log.i("onGetPcmFrame", "ret: " + ret + ", sampleRate: " + sampleRate + ", channel: " + channel + ", sampleSize: " + sampleSize +",is_low_latency:" + is_low_latency + " buffer_size：" + buffer_size);*/if ( pcm_buffer_ == null)return;pcm_buffer_.rewind();if ( ret == 0 && (isPushingRtmp || isRTSPPublisherRunning)){libPublisher.SmartPublisherOnFarEndPCMData(publisherHandle, pcm_buffer_, sampleRate, channel, sampleSize, is_low_latency);if (is_audio_mix_){libPublisher.SmartPublisherOnMixPCMData(publisherHandle, 1, pcm_buffer_, 0, buffer_size, sampleRate, channel, sampleSize);}/*java.nio.ByteOrder old_order = pcm_buffer_.order();pcm_buffer_.order(java.nio.ByteOrder.nativeOrder());java.nio.ShortBuffer short_buffer = pcm_buffer_.asShortBuffer();pcm_buffer_.order(old_order);short[] short_array = new short[short_buffer.remaining()];short_buffer.get(short_array);libPublisher.SmartPublisherOnMixPCMShortArray(publisherHandle, 1, short_array, 0, short_array.length, sampleRate, channel, sampleSize);*/}// test/*byte[] test_buffer = new byte[16];pcm_buffer_.get(test_buffer);Log.i(TAG, "onGetPcmFrame data:" + bytesToHexString(test_buffer));*/}}

推送端：

RTMP推送：

class ButtonPushStartListener implements OnClickListener{public void onClick(View v){ if (isPushingRtmp){stopPush();if (!isRTSPPublisherRunning) {ConfigControlEnable(true);}btnPushStartStop.setText("推送RTMP");isPushingRtmp = false;return;}Log.i(PUSH_TAG, "onClick start push rtmp..");if (libPublisher == null)return;if (!isRTSPPublisherRunning) {InitPusherAndSetConfig();}if ( inputPushURL != null && inputPushURL.length() > 1 ){publishURL = inputPushURL;Log.i(PUSH_TAG, "start, input publish url:" + publishURL);}else{publishURL = basePushURL + String.valueOf((int)( System.currentTimeMillis() % 1000000));Log.i(PUSH_TAG, "start, generate random url:" + publishURL);}printPushText = "URL:" + publishURL;Log.i(PUSH_TAG, printPushText);textPushCurURL = (TextView)findViewById(R.id.txt_push_cur_url);textPushCurURL.setText(printPushText);Log.i(PUSH_TAG, "videoWidth: "+ pushVideoWidth + " videoHeight: " + pushVideoHeight + " pushType:" + pushType);if ( libPublisher.SmartPublisherSetURL(publisherHandle, publishURL) != 0 ){Log.e(PUSH_TAG, "Failed to set rtmp pusher URL..");}int startRet = libPublisher.SmartPublisherStartPublisher(publisherHandle);if (startRet != 0) {isPushingRtmp = false;Log.e(TAG, "Failed to start push stream..");return;}if ( !isRTSPPublisherRunning ) {if (pushType == 0 || pushType == 1) {CheckInitAudioRecorder(); //enable pure video publisher..}ConfigControlEnable(false);}btnPushStartStop.setText("停止推送 ");isPushingRtmp = true;}};

輕量級RTSP服務模式：

//啟動/停止RTSP服務class ButtonRtspServiceListener implements OnClickListener {public void onClick(View v) {if (isRTSPServiceRunning) {stopRtspService();btnRtspService.setText("啟動RTSP服務");btnRtspPublisher.setEnabled(false);isRTSPServiceRunning = false;return;}Log.i(TAG, "onClick start rtsp service..");rtsp_handle_ = libPublisher.OpenRtspServer(0);if (rtsp_handle_ == 0) {Log.e(TAG, "創建rtsp server實例失敗! 請檢查SDK有效性");} else {int port = 8554;if (libPublisher.SetRtspServerPort(rtsp_handle_, port) != 0) {libPublisher.CloseRtspServer(rtsp_handle_);rtsp_handle_ = 0;Log.e(TAG, "創建rtsp server端口失敗! 請檢查端口是否重復或者端口不在范圍內!");}//String user_name = "admin";//String password = "12345";//libPublisher.SetRtspServerUserNamePassword(rtsp_handle_, user_name, password);if (libPublisher.StartRtspServer(rtsp_handle_, 0) == 0) {Log.i(TAG, "啟動rtsp server 成功!");} else {libPublisher.CloseRtspServer(rtsp_handle_);rtsp_handle_ = 0;Log.e(TAG, "啟動rtsp server失敗! 請檢查設置的端口是否被占用!");}btnRtspService.setText("停止RTSP服務");btnRtspPublisher.setEnabled(true);isRTSPServiceRunning = true;}}}//發布/停止RTSP流class ButtonRtspPublisherListener implements OnClickListener {public void onClick(View v) {if (isRTSPPublisherRunning) {stopRtspPublisher();if (!isPushingRtmp) {ConfigControlEnable(true);}btnRtspPublisher.setText("發布RTSP流");btnGetRtspSessionNumbers.setEnabled(false);btnRtspService.setEnabled(true);isRTSPPublisherRunning = false;return;}Log.i(TAG, "onClick start rtsp publisher..");if (!isPushingRtmp) {InitPusherAndSetConfig();}if (publisherHandle == 0) {Log.e(TAG, "Start rtsp publisher, publisherHandle is null..");return;}String rtsp_stream_name = "stream1";libPublisher.SetRtspStreamName(publisherHandle, rtsp_stream_name);libPublisher.ClearRtspStreamServer(publisherHandle);libPublisher.AddRtspStreamServer(publisherHandle, rtsp_handle_, 0);if (libPublisher.StartRtspStream(publisherHandle, 0) != 0) {Log.e(TAG, "調用發布rtsp流接口失敗!");return;}if (!isPushingRtmp) {if (pushType == 0 || pushType == 1) {CheckInitAudioRecorder(); //enable pure video publisher..}ConfigControlEnable(false);}btnRtspPublisher.setText("停止RTSP流");btnGetRtspSessionNumbers.setEnabled(true);btnRtspService.setEnabled(false);isRTSPPublisherRunning = true;}};

RTMP推送和輕量級RTSP服務，可以在一個實例里面處理，所以推送參數的初始化，只需要調用一次即可。

private void InitPusherAndSetConfig() {Log.i(TAG, "videoWidth: " + pushVideoWidth + " videoHeight: " + pushVideoHeight+ " pushType:" + pushType);int audio_opt = 1;int video_opt = 1;if ( pushType == 1 ){video_opt = 0;}else if (pushType == 2 ){audio_opt = 0;}publisherHandle = libPublisher.SmartPublisherOpen(curContext, audio_opt, video_opt,pushVideoWidth, pushVideoHeight);if ( publisherHandle == 0 ){return;}if(videoEncodeType == 1){int h264HWKbps = setHardwareEncoderKbps(true, pushVideoWidth,pushVideoHeight);Log.i(TAG, "h264HWKbps: " + h264HWKbps);int isSupportH264HWEncoder = libPublisher.SetSmartPublisherVideoHWEncoder(publisherHandle, h264HWKbps);if (isSupportH264HWEncoder == 0) {Log.i(TAG, "Great, it supports h.264 hardware encoder!");}}else if (videoEncodeType == 2){int hevcHWKbps = setHardwareEncoderKbps(false, pushVideoWidth,pushVideoHeight);Log.i(TAG, "hevcHWKbps: " + hevcHWKbps);int isSupportHevcHWEncoder = libPublisher.SetSmartPublisherVideoHevcHWEncoder(publisherHandle, hevcHWKbps);if (isSupportHevcHWEncoder == 0) {Log.i(TAG, "Great, it supports hevc hardware encoder!");}}if(is_sw_vbr_mode){int is_enable_vbr = 1;int video_quality = CalVideoQuality(pushVideoWidth,pushVideoHeight, true);int vbr_max_bitrate = CalVbrMaxKBitRate(pushVideoWidth,pushVideoHeight);libPublisher.SmartPublisherSetSwVBRMode(publisherHandle, is_enable_vbr, video_quality, vbr_max_bitrate);}libPublisher.SetSmartPublisherEventCallbackV2(publisherHandle, new EventHandePublisherV2());//如果想和時間顯示在同一行，請去掉'\n'String watermarkText = "大牛直播(daniulive)\n\n";String path = pushLogoPath;if( pushWatemarkType == 0 ){if ( isPushWritelogoFileSuccess )libPublisher.SmartPublisherSetPictureWatermark(publisherHandle, path, WATERMARK.WATERMARK_POSITION_TOPRIGHT, 160, 160, 10, 10);}else if( pushWatemarkType == 1 ){if ( isPushWritelogoFileSuccess )libPublisher.SmartPublisherSetPictureWatermark(publisherHandle, path, WATERMARK.WATERMARK_POSITION_TOPRIGHT, 160, 160, 10, 10);libPublisher.SmartPublisherSetTextWatermark(publisherHandle, watermarkText, 1, WATERMARK.WATERMARK_FONTSIZE_BIG, WATERMARK.WATERMARK_POSITION_BOTTOMRIGHT, 10, 10);//libPublisher.SmartPublisherSetTextWatermarkFontFileName("/system/fonts/DroidSansFallback.ttf");//libPublisher.SmartPublisherSetTextWatermarkFontFileName("/sdcard/DroidSansFallback.ttf");}else if(pushWatemarkType == 2){libPublisher.SmartPublisherSetTextWatermark(publisherHandle, watermarkText, 1, WATERMARK.WATERMARK_FONTSIZE_BIG, WATERMARK.WATERMARK_POSITION_BOTTOMRIGHT, 10, 10);//libPublisher.SmartPublisherSetTextWatermarkFontFileName("/system/fonts/DroidSansFallback.ttf");}else{Log.i(TAG, "no watermark settings..");}//endif ( !is_push_speex ){// set AAC encoderlibPublisher.SmartPublisherSetAudioCodecType(publisherHandle, 1);}else{// set Speex encoderlibPublisher.SmartPublisherSetAudioCodecType(publisherHandle, 2);libPublisher.SmartPublisherSetSpeexEncoderQuality(publisherHandle, 8);}libPublisher.SmartPublisherSetNoiseSuppression(publisherHandle, is_push_noise_suppression?1:0);libPublisher.SmartPublisherSetAGC(publisherHandle, is_push_agc?1:0);libPublisher.SmartPublisherSetEchoCancellation(publisherHandle, 1, echoCancelDelay);libPublisher.SmartPublisherSetAudioMix(publisherHandle, is_audio_mix_?1:0);libPublisher.SmartPublisherSetInputAudioVolume(publisherHandle, 0 , mic_audio_volume_);if ( is_audio_mix_ ){libPublisher.SmartPublisherSetInputAudioVolume(publisherHandle, 1 , mix_audio_volume_);}libPublisher.SmartPublisherSetClippingMode(publisherHandle, 0);libPublisher.SmartPublisherSetSWVideoEncoderProfile(publisherHandle, push_sw_video_encoder_profile);//libPublisher.SetRtmpPublishingType(0);//libPublisher.SmartPublisherSetGopInterval(publisherHandle, 18*3);//libPublisher.SmartPublisherSetFPS(publisherHandle, 18);libPublisher.SmartPublisherSetSWVideoEncoderSpeed(publisherHandle, sw_video_encoder_speed);//libPublisher.SmartPublisherSetSWVideoBitRate(600, 1200);}

相關封裝：

//停止rtmp推送private void stopPush() {if(!isPushingRtmp){return;}if ( !isRTSPPublisherRunning) {if (audioRecord_ != null) {Log.i(TAG, "stopPush, call audioRecord_.StopRecording..");audioRecord_.Stop();if (audioRecordCallback_ != null) {audioRecord_.RemoveCallback(audioRecordCallback_);audioRecordCallback_ = null;}audioRecord_ = null;}}if (libPublisher != null) {libPublisher.SmartPublisherStopPublisher(publisherHandle);}if (!isRTSPPublisherRunning) {if (publisherHandle != 0) {if (libPublisher != null) {libPublisher.SmartPublisherClose(publisherHandle);publisherHandle = 0;}}}}//停止發布RTSP流private void stopRtspPublisher() {if(!isRTSPPublisherRunning){return;}if (!isPushingRtmp) {if (audioRecord_ != null) {Log.i(TAG, "stopRtspPublisher, call audioRecord_.StopRecording..");audioRecord_.Stop();if (audioRecordCallback_ != null) {audioRecord_.RemoveCallback(audioRecordCallback_);audioRecordCallback_ = null;}audioRecord_ = null;}}if (libPublisher != null) {libPublisher.StopRtspStream(publisherHandle);}if (!isPushingRtmp) {if (publisherHandle != 0) {if (libPublisher != null) {libPublisher.SmartPublisherClose(publisherHandle);publisherHandle = 0;}}}}//停止RTSP服務private void stopRtspService() {if(!isRTSPServiceRunning){return;}if (libPublisher != null && rtsp_handle_ != 0) {libPublisher.StopRtspServer(rtsp_handle_);libPublisher.CloseRtspServer(rtsp_handle_);rtsp_handle_ = 0;}}

傳遞采集到的視頻數據，攝像頭數據采集，也可選用camera2的接口，對焦和體驗更好：

@Overridepublic void onPreviewFrame(byte[] data, Camera camera) {pushFrameCount++;if ( pushFrameCount % 3000 == 0 ){Log.i("OnPre", "gc+");System.gc();Log.i("OnPre", "gc-");}if (data == null) {Parameters params = camera.getParameters();Size size = params.getPreviewSize();int bufferSize = (((size.width|0x1f)+1) * size.height * ImageFormat.getBitsPerPixel(params.getPreviewFormat())) / 8;camera.addCallbackBuffer(new byte[bufferSize]);} else {if(isPushingRtmp || isRTSPPublisherRunning){libPublisher.SmartPublisherOnCaptureVideoData(publisherHandle, data, data.length, pushCurrentCameraType, currentPushOrigentation);}camera.addCallbackBuffer(data);}}

如果內網環境下，用輕量級RTSP服務的話，需判斷對方有沒有播放自己的流數據的話，可以通過獲取RTSP會話數來判斷是否鏈接。

//當前RTSP會話數彈出框private void PopRtspSessionNumberDialog(int session_numbers) {final EditText inputUrlTxt = new EditText(this);inputUrlTxt.setFocusable(true);inputUrlTxt.setEnabled(false);String session_numbers_tag = "RTSP服務當前客戶會話數: " + session_numbers;inputUrlTxt.setText(session_numbers_tag);AlertDialog.Builder builderUrl = new AlertDialog.Builder(this);builderUrl.setTitle("內置RTSP服務").setView(inputUrlTxt).setNegativeButton("確定", null);builderUrl.show();}//獲取RTSP會話數class ButtonGetRtspSessionNumbersListener implements OnClickListener {public void onClick(View v) {if (libPublisher != null && rtsp_handle_ != 0) {int session_numbers = libPublisher.GetRtspServerClientSessionNumbers(rtsp_handle_);Log.i(TAG, "GetRtspSessionNumbers: " + session_numbers);PopRtspSessionNumberDialog(session_numbers);}}};

總結

Android平臺的一對一互動，除了WebRTC外，在保證低延遲的前提下，RTMP或RTSP技術方案也是非常不錯的選擇。

總結

以上是生活随笔為你收集整理的Android平台基于RTMP或RTSP的一对一音视频互动技术方案探讨的全部內容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網站內容還不錯，歡迎將生活随笔推薦給好友。