Skip to main content

Creating Frame Processor Plugins

Creating a Frame Processor Plugin for Android

The Frame Processor Plugin API is built to be as extensible as possible, which allows you to create custom Frame Processor Plugins. In this guide we will create a custom Face Detector Plugin which can be used from JS.

Android Frame Processor Plugins can be written in either Java, Kotlin or C++ (JNI).

Mostly automatic setup

  1. Run Vision Camera Plugin Builder CLI
npx vision-camera-plugin-builder@latest android
info

The CLI will ask you for the path to project's Android Manifest file, name of the plugin (e.g. FaceDetectorFrameProcessorPlugin), name of the exposed method (e.g. detectFaces) and language you want to use for plugin development (Java or Kotlin). For reference see the CLI's docs.

  1. Register the package in MainApplication.java
        @Override
protected List<ReactPackage> getPackages() {
@SuppressWarnings("UnnecessaryLocalVariable")
List<ReactPackage> packages = new PackageList(this).getPackages();
// ...
packages.add(new FaceDetectorFrameProcessorPluginPackage()); // <- add
return packages;
}

Manual setup

  1. Open your Project in Android Studio
  2. Create a Java source file, for the Face Detector Plugin this will be called FaceDetectorFrameProcessorPlugin.java.
  3. Add the following code:
import androidx.annotation.NonNull;
import androidx.annotation.Nullable;
import com.mrousavy.camera.frameprocessor.Frame;
import com.mrousavy.camera.frameprocessor.FrameProcessorPlugin;
import com.mrousavy.camera.frameprocessor.VisionCameraProxy;

public class FaceDetectorFrameProcessorPlugin extends FrameProcessorPlugin {
FaceDetectorFrameProcessorPlugin(@NonNull VisionCameraProxy proxy, @Nullable Map<String, Object> options) {}

@Nullable
@Override
public Object callback(@NonNull Frame frame, @Nullable Map<String, Object> arguments) {
// code goes here
return null;
}
}
  1. Implement your Frame Processing. See the Example Plugin (Java) for reference.
  2. Create a new Java file which registers the Frame Processor Plugin in a React Package, for the Face Detector plugin this file will be called FaceDetectorFrameProcessorPluginPackage.java:
import androidx.annotation.NonNull;
import com.facebook.react.ReactPackage;
import com.facebook.react.bridge.NativeModule;
import com.facebook.react.bridge.ReactApplicationContext;
import com.facebook.react.uimanager.ViewManager;
import com.mrousavy.camera.frameprocessor.FrameProcessorPlugin;
import com.mrousavy.camera.frameprocessor.FrameProcessorPluginRegistry;

public class FaceDetectorFrameProcessorPluginPackage implements ReactPackage {
static {
FrameProcessorPluginRegistry.addFrameProcessorPlugin("detectFaces", FaceDetectorFrameProcessorPlugin::new);
}

@NonNull
@Override
public List<NativeModule> createNativeModules(@NonNull ReactApplicationContext reactContext) {
return Collections.emptyList();
}

@NonNull
@Override
public List<ViewManager> createViewManagers(@NonNull ReactApplicationContext reactContext) {
return Collections.emptyList();
}
}
note

The Frame Processor Plugin will be exposed to JS through the VisionCameraProxy object. In this case, it would be VisionCameraProxy.initFrameProcessorPlugin("detectFaces").

  1. Register the package in MainApplication.java
        @Override
protected List<ReactPackage> getPackages() {
@SuppressWarnings("UnnecessaryLocalVariable")
List<ReactPackage> packages = new PackageList(this).getPackages();
// ...
packages.add(new FaceDetectorFrameProcessorPluginPackage()); // <- add
return packages;
}

🚀 Next section: Finish creating your Frame Processor Plugin (or add iOS support to your Frame Processor Plugin)