Libsnout

This is a rust implementation of Project Babble's baballonia face tracking sofware. It's designed to be a library; easy to integrate in a variety of frontend projects. However it can also be used as a CLI application through snout-cli.

• Building:
  • Building and running the cli
  • Installing on NixOS
• Configuring:
  • See CONFIGURING.md
• Usage:
  • Tracking
  • Training an eye model
  • Troubleshooting
• Remote Control:
  • See REMOTE.MD

Required dependencies

Libsnout requires the following build dependencies (in the form of fedora package names):

• llvm
• llvm-devel
• onnxruntime
• onnxruntime-devel
• rust

Building and running the CLI

Clone the repository,

git clone https://github.com/Darksecond/libsnout.git

and then build the program.

cd libsnout
cargo build --release -p snout-cli

The snout-cli executable will be located under target/release/

snout-cli can either be executed from this directory, like so:

target/release/snout-cli

or snout-cli can be added to your $PATH and executed like so:

snout-cli

These docs will assume that snout-cli is in your $PATH. If this is not the case, replace snout-cli in the following commands with the path to snout-cli. Like shown above.

Help on how to use the cli tool can be obtained with:

snout-cli help

Installing on NixOS

Add libsnout to your flake.nix inputs:

  libsnout.url = "github:Darksecond/libsnout";

Either use the package directly or add it to your overlays:

nixpkgs.overlay = [
  (final: prev: {
    snout-cli = libsnout.packages."${pkgs.stdenv.hostPlatform.system}".default;
  })
];

environment.systemPackages = with pkgs; [
  snout-cli
];

Tracking

Before being able to use Libsnout for face/eye tracking and training models, one must configure it. See CONFIGURING.md for information on how to do so.

Libsnout comes with a working face tracking model. It's the same as in the baballonia repository, but ran through onnxsim.

Once you have set up your configuration file to point to your cameras, and set the output OSC destination to the correct values for your program of choice. You can start tracking with the following command:

snout-cli track

This will start recording, along with sending data to the OSC endpoint specified in the configuration file.

Training an eye model

Eye models can be trained with the following command:

snout-cli train <user_cal.bin> <output.onnx>

the <user_cal.bin> file generated by baballonia can be found in the installation folder of the baballonia software. Next to the executable. The resulting <output.onnx> can then be used in the configuration file, for the corresponding eye.

Troubleshooting

A camera frame can be captured and written to a file with the following command to help with debugging tracking issues, along with aligning your face:

snout-cli capture <SOURCE> <OUTPUT.jpeg>

<SOURCE> can be any of the following camera sources left-eye, right-eye, face,

<OUTPUT.jpeg> will be the name of the file that the camera frame gets written to.

License

Right now it's licensed under the same license as Baballonia from Project Babble is, considering this is a derivative work.