Creating Frame Processor Plugins

Creating a Frame Processor Plugin with Expo Modules

The Frame Processor Plugin API is built to be as extensible as possible, allowing you to create custom Frame Processor Plugins. In this guide, we will create a custom Face Detector Plugin that can be used from JavaScript, packaged as a local Expo Module.

info

This guide is specifically for projects using Expo with Continuous Native Generation (CNG). If you're using a bare React Native project, refer to the iOS or Android guides instead.

Step 1: Create the Expo Module

Create a local Expo Module using the official CLI:

npx create-expo-module@latest --local face-detector-frame-processor

Step 2: iOS Implementation

Configure the Podspec

First, add VisionCamera as a dependency to link the required headers.

Edit ios/FaceDetectorFrameProcessor.podspec:

Pod::Spec.new do |s|
  # ... standard setup ...
  s.dependency "ExpoModulesCore"
  s.dependency "VisionCamera"  # Required for Frame Processor headers
end

Implement the Frame Processor

Create ios/FaceDetectorFrameProcessor.m:

#import <VisionCamera/FrameProcessorPlugin.h>
#import <VisionCamera/FrameProcessorPluginRegistry.h>
#import <VisionCamera/VisionCameraProxyHolder.h>
#import <VisionCamera/Frame.h>

@interface FaceDetectorFrameProcessorPlugin : FrameProcessorPlugin

@end

@implementation FaceDetectorFrameProcessorPlugin

- (instancetype _Nonnull)initWithProxy:(VisionCameraProxyHolder*)proxy
                           withOptions:(NSDictionary* _Nullable)options {
  self = [super initWithProxy:proxy withOptions:options];
  return self;
}

- (id _Nullable)callback:(Frame* _Nonnull)frame
           withArguments:(NSDictionary* _Nullable)arguments {
  // code goes here
  return nil;
}

VISION_EXPORT_FRAME_PROCESSOR(FaceDetectorFrameProcessorPlugin, detectFaces)

@end

note

The above example uses Objective-C. If you prefer to write your Frame Processor Plugin in Swift, follow the Swift implementation guide in the iOS Frame Processor Plugin documentation, and adapt it for use within your Expo Module.

Step 3: Android Implementation

Configure Gradle Dependencies

Android requires explicit dependencies for VisionCamera.

Edit android/build.gradle:

// ... standard setup ...

dependencies {
  implementation project(':react-native-vision-camera')
}

Implement the Frame Processor Plugin

Create android/src/main/java/expo/modules/facedetectorframeprocessor/FaceDetectorFrameProcessorPlugin.java:

package expo.modules.facedetectorframeprocessor;

import androidx.annotation.NonNull;
import androidx.annotation.Nullable;

import com.mrousavy.camera.frameprocessors.Frame;
import com.mrousavy.camera.frameprocessors.FrameProcessorPlugin;
import com.mrousavy.camera.frameprocessors.VisionCameraProxy;

import java.util.Map;

public class FaceDetectorFrameProcessorPlugin extends FrameProcessorPlugin {
    public FaceDetectorFrameProcessorPlugin(VisionCameraProxy proxy, @Nullable Map<String, Object> options) {
        super();
    }

    @Nullable
    @Override
    public Object callback(@NonNull Frame frame, @Nullable Map<String, Object> arguments) {
       // code goes here
       return null;
    }
}

Register frame processor plugin with Expo Module

The Expo Module acts as a "loader" that registers the Frame Processor Plugin at startup.

Create android/src/main/java/expo/modules/facedetectorframeprocessor/FaceDetectorFrameProcessorModule.kt:

package expo.modules.facedetectorframeprocessor

import expo.modules.kotlin.modules.Module
import expo.modules.kotlin.modules.ModuleDefinition
import com.mrousavy.camera.frameprocessors.FrameProcessorPluginRegistry
import com.mrousavy.camera.frameprocessors.VisionCameraProxy

class FaceDetectorFrameProcessorModule : Module() {
  init {
    FrameProcessorPluginRegistry.addFrameProcessorPlugin(
      "detectFaces"
    ) { proxy: VisionCameraProxy?, options: Map<String?, Any?>? ->
      FaceDetectorFrameProcessorPlugin(
        proxy,
        options
      )
    }
  }

  override fun definition() = ModuleDefinition {
    Name("FaceDetectorFrameProcessor")
  }
}

info

The Expo Module uses the init block to register the Frame Processor Plugin. This ensures the plugin is available before any JavaScript code tries to use it.

Step 4: Configure Autolinking

Update expo-module.config.json to register the native modules for both platforms:

{
  "platforms": [
    "apple",
    "android"
  ],
  "apple": {
    "modules": [
      "FaceDetectorFrameProcessorModule"
    ]
  },
  "android": {
    "modules": [
      "expo.modules.facedetectorframeprocessor.FaceDetectorFrameProcessorModule"
    ]
  }
}

🚀 Next section: Finish creating your Frame Processor Plugin