Home-Assistant custom component for image classification (tag detection) using Machinebox.io Tagbox. The component adds an image_processing entity, where the state of the entity is the most likely tag in the image above a given confidence threshold. The attribute tags contains all data returned by the API, whilst the attribute matched_tags contains only tags matched with a confidence greater than a confidence threshold (in %). The event tagbox.detect_tag is fired for each tag above the confidence threshold, and the event data contains the fields name of tag, confidence of detection, and the entity_id that fired the event.

Place the custom_components folder in your configuration directory (or add its contents to an existing custom_components folder).

Add to your HA config:

image_processing:
  - platform: tagbox
    ip_address: localhost
    port: 8080
    confidence: 50
    source:
      - entity_id: camera.local_file

Configuration variables:

• ip_address: the ip of your Tagbox instance
• port: the port of your Tagbox instance
• confidence: (Optional, default 80) The threshold in % above which tags appear in matched_tags and tagbox.detect_tag events are fired.
• source: must be a camera.

Tagbox

Get/update Tagbox from Dockerhub by running:

docker pull machinebox/tagbox

Run Tagbox with:

MB_KEY="INSERT-YOUR-KEY-HERE"
docker run -p 8080:8080 -e "MB_KEY=$MB_KEY" machinebox/tagbox

To limit tagbox to only custom tags, add to the command -e MB_TAGBOX_ONLY_CUSTOM_TAGS=true. Tagbox will be then only return and calculate custom tags, saving some compute resources.

docker run -p 8080:8080 -e "MB_KEY=$MB_KEY" -e MB_TAGBOX_ONLY_CUSTOM_TAGS=true machinebox/tagbox

You should see a message on startup pretrained tags are disabled, only custom tags will be returned.

Default tags

Tagbox is trained on https://storage.googleapis.com/openimages/web/index.html with the default tags listed in tagbox_classes.csv

Teaching tagbox

I've written a script to allow teaching tagbox from folders of images, the script is at https://github.com/robmarkcole/tagbox_python

Limiting computation

Image-classifier components process the image from a camera at a fixed period given by the scan_interval. This leads to excessive computation if the image on the camera hasn't changed (for example if you are using a local file camera to display an image captured by a motion triggered system and this doesn't change often). The default scan_interval is 10 seconds. You can override this by adding to your config scan_interval: 10000 (setting the interval to 10,000 seconds), and then call the scan service when you actually want to process a camera image. So in my setup, I use an automation to call scan when a new motion triggered image has been saved and displayed on my local file camera.

Local file camera

Note that for development I am using a file camera.

camera:
  - platform: local_file
    file_path: /images/thebeatles.jpg