Skip to main content Scroll Top

Segment Anything 3 (default)

C++ (Windows, Linux, MacOS / CUDA and Metal accelerated) port of https://github.com/facebookresearch/sam3.

Example Input & Outputs

Inputs Outputs
Input
Image Input
Input
Car Detections

Demo Code

 1#include "blace_ai.h"
 2#include <opencv2/opencv.hpp>
 3
 4// include the models you want to use
 5#include "sam3_v1_default_v6_ALL_export_version_v26.h"
 6
 7using namespace blace;
 8int main() {
 9  workload_management::BlaceWorld blace;
10
11  // load image into op
12  auto exe_path = util::getPathToExe();
13  std::filesystem::path photo_path = exe_path / "street.jpg";
14  auto world_tensor_orig =
15      CONSTRUCT_OP(ops::FromImageFileOp(photo_path.string()));
16
17  // construct model inference arguments
18  ml_core::InferenceArgsCollection infer_args;
19  infer_args.inference_args.backends = {
20      ml_core::TORCHSCRIPT_CUDA_FP16, ml_core::TORCHSCRIPT_MPS_FP16,
21      ml_core::TORCHSCRIPT_CUDA_FP32, ml_core::TORCHSCRIPT_MPS_FP32,
22      ml_core::ONNX_DML_FP32,         ml_core::TORCHSCRIPT_CPU_FP32};
23
24  blace::ops::OpP text_op = CONSTRUCT_OP(blace::ops::FromTextOp("car"));
25  blace::ops::OpP threshold_op = CONSTRUCT_OP(blace::ops::FromFloatOp(0.5));
26
27  // returns all mattes with shape [c, h, w]
28  auto matte = sam3_v1_default_v6_ALL_export_version_v26_run(
29      world_tensor_orig, text_op, threshold_op, 0, infer_args,
30      util::getPathToExe().string());
31
32  // sum all mattes along channel dimension
33  matte =
34      CONSTRUCT_OP(blace::ops::SumOp(matte, std::vector<int64_t>({0}), true));
35
36  // write result to image file
37  auto out_file = exe_path / "car_matte.png";
38  matte = CONSTRUCT_OP(ops::SaveImageOp(matte, out_file.string()));
39
40  // construct evaluator and evaluate
41  computation_graph::GraphEvaluator evaluator(matte);
42  auto eval_result = evaluator.evaluateToRawMemory();
43
44  return 0;
45}

Tested on version v1.1.1 of blace.ai sdk. Might also work on newer or older releases (check if release notes of blace.ai state breaking changes).

Quickstart

  1. Download blace.ai SDK and unzip. In the bootstrap script build_run_demos.ps1 (Windows) or build_run_demos.sh (Linux/MacOS) set the BLACE_AI_CMAKE_DIR environment variable to the cmake folder inside the unzipped SDK, e.g. export BLACE_AI_CMAKE_DIR="<unzip_folder>/package/cmake".
  2. Download the model payload(s) (.bin files) from below and place in the same folder as the bootstrapper scripts.
  3. Then run the bootstrap script with
    powershell build_run_demo.ps1 (Windows)
    sh build_run_demo.sh (Linux and MacOS).
    This will build and execute the demo.

Supported Backends

Torchscript CPU Torchscript CUDA FP16 * Torchscript CUDA FP32 * Torchscript MPS FP16 * Torchscript MPS FP32 * ONNX CPU FP32 ONNX DirectML FP32 *
✅ ✅ ✅ ✅ ✅ ❌ ❌

(*: Hardware Accelerated)

Artifacts

Torchscript Payload Demo Project Header

License