Using MovementBasedSegmenter with custom neural network

Yves · May 28, 2019, 2:50pm

Hi all

We are currently developing an application where we need to track multiple rectangles with different colors. The MovementBasedSegmenter seems to fit this use case with the option of filtering object contours and responding with the object location and id.
How do we go about creating our custom neural network? Is the structure in terms of entry points and respond methods the same as a custom watcher as described in the QRCodeDetector here?

Additionally, is there already an application with which we can send screenshots from lampix to a different device in order to train/test the network?

vfintinari · May 29, 2019, 12:44pm

Hi @Yves,

The good news is that we are currently working on a trainer app and server, which you will be able to use to automatically train a new custom neural network and also deploy it to your Lampix. Please bare with us, as we are currently testing this system.

In the meantime, you could try taking pictures with a smartphone, train your custom neural network (we are using Keras) and deploy it using the ‘:8888/apps’ endpoint. Once this is done and the neural network shows up in Control Panel, you could develop your application using the MovementBasedSegmenter and specify the name of the neural network.

Let us know if we can further help you.

Yves · June 17, 2019, 8:32am

We have created, trained and added the custom neural network ‘color-classifier’ to our device. However, we get the following error:
IndexError: tuple index out of range

Any ideas on how to fix this? Also, is there any documentation on how the custom model is loaded and used?

vfintinari · June 18, 2019, 8:17am

@Yves, Could you please provide the ‘.json’ file of your trained network? It seems that the architecture might not be compatible.

Yves · June 19, 2019, 6:14am

@vfintinari sure, here you go. Is there any documentation on how the architecture should be or how the NeuralNetworkClassifier or MovementBasedSegmenter use the provided custom network?

{“class_name”: “Sequential”, “keras_version”: “2.2.2”, “config”: [{“class_name”: “Conv2D”, “config”: {“kernel_initializer”: {“class_name”: “VarianceScaling”, “config”: {“distribution”: “uniform”, “scale”: 1.0, “seed”: null, “mode”: “fan_avg”}}, “name”: “conv2d_1”, “kernel_constraint”: null, “bias_regularizer”: null, “bias_constraint”: null, “dtype”: “float32”, “activation”: “linear”, “trainable”: true, “data_format”: “channels_last”, “filters”: 32, “padding”: “same”, “strides”: [1, 1], “dilation_rate”: [1, 1], “kernel_regularizer”: null, “bias_initializer”: {“class_name”: “Zeros”, “config”: {}}, “batch_input_shape”: [null, 64, 64, 3], “use_bias”: true, “activity_regularizer”: null, “kernel_size”: [3, 3]}}, {“class_name”: “Activation”, “config”: {“activation”: “relu”, “trainable”: true, “name”: “activation_1”}}, {“class_name”: “BatchNormalization”, “config”: {“beta_constraint”: null, “gamma_initializer”: {“class_name”: “Ones”, “config”: {}}, “moving_mean_initializer”: {“class_name”: “Zeros”, “config”: {}}, “name”: “batch_normalization_1”, “epsilon”: 0.001, “trainable”: true, “moving_variance_initializer”: {“class_name”: “Ones”, “config”: {}}, “beta_initializer”: {“class_name”: “Zeros”, “config”: {}}, “scale”: true, “axis”: -1, “gamma_constraint”: null, “gamma_regularizer”: null, “beta_regularizer”: null, “momentum”: 0.99, “center”: true}}, {“class_name”: “MaxPooling2D”, “config”: {“name”: “max_pooling2d_1”, “trainable”: true, “data_format”: “channels_last”, “pool_size”: [2, 2], “padding”: “valid”, “strides”: [2, 2]}}, {“class_name”: “Dropout”, “config”: {“rate”: 0.25, “noise_shape”: null, “trainable”: true, “seed”: null, “name”: “dropout_1”}}, {“class_name”: “Conv2D”, “config”: {“kernel_constraint”: null, “kernel_initializer”: {“class_name”: “VarianceScaling”, “config”: {“distribution”: “uniform”, “scale”: 1.0, “seed”: null, “mode”: “fan_avg”}}, “name”: “conv2d_2”, “bias_regularizer”: null, “bias_constraint”: null, “activation”: “linear”, “trainable”: true, “data_format”: “channels_last”, “padding”: “same”, “strides”: [1, 1], “dilation_rate”: [1, 1], “kernel_regularizer”: null, “filters”: 64, “bias_initializer”: {“class_name”: “Zeros”, “config”: {}}, “use_bias”: true, “activity_regularizer”: null, “kernel_size”: [3, 3]}}, {“class_name”: “Activation”, “config”: {“activation”: “relu”, “trainable”: true, “name”: “activation_2”}}, {“class_name”: “BatchNormalization”, “config”: {“beta_constraint”: null, “gamma_initializer”: {“class_name”: “Ones”, “config”: {}}, “moving_mean_initializer”: {“class_name”: “Zeros”, “config”: {}}, “name”: “batch_normalization_2”, “epsilon”: 0.001, “trainable”: true, “moving_variance_initializer”: {“class_name”: “Ones”, “config”: {}}, “beta_initializer”: {“class_name”: “Zeros”, “config”: {}}, “scale”: true, “axis”: -1, “gamma_constraint”: null, “gamma_regularizer”: null, “beta_regularizer”: null, “momentum”: 0.99, “center”: true}}, {“class_name”: “Conv2D”, “config”: {“kernel_constraint”: null, “kernel_initializer”: {“class_name”: “VarianceScaling”, “config”: {“distribution”: “uniform”, “scale”: 1.0, “seed”: null, “mode”: “fan_avg”}}, “name”: “conv2d_3”, “bias_regularizer”: null, “bias_constraint”: null, “activation”: “linear”, “trainable”: true, “data_format”: “channels_last”, “padding”: “same”, “strides”: [1, 1], “dilation_rate”: [1, 1], “kernel_regularizer”: null, “filters”: 64, “bias_initializer”: {“class_name”: “Zeros”, “config”: {}}, “use_bias”: true, “activity_regularizer”: null, “kernel_size”: [3, 3]}}, {“class_name”: “Activation”, “config”: {“activation”: “relu”, “trainable”: true, “name”: “activation_3”}}, {“class_name”: “BatchNormalization”, “config”: {“beta_constraint”: null, “gamma_initializer”: {“class_name”: “Ones”, “config”: {}}, “moving_mean_initializer”: {“class_name”: “Zeros”, “config”: {}}, “name”: “batch_normalization_3”, “epsilon”: 0.001, “trainable”: true, “moving_variance_initializer”: {“class_name”: “Ones”, “config”: {}}, “beta_initializer”: {“class_name”: “Zeros”, “config”: {}}, “scale”: true, “axis”: -1, “gamma_constraint”: null, “gamma_regularizer”: null, “beta_regularizer”: null, “momentum”: 0.99, “center”: true}}, {“class_name”: “MaxPooling2D”, “config”: {“name”: “max_pooling2d_2”, “trainable”: true, “data_format”: “channels_last”, “pool_size”: [2, 2], “padding”: “valid”, “strides”: [2, 2]}}, {“class_name”: “Dropout”, “config”: {“rate”: 0.25, “noise_shape”: null, “trainable”: true, “seed”: null, “name”: “dropout_2”}}, {“class_name”: “Conv2D”, “config”: {“kernel_constraint”: null, “kernel_initializer”: {“class_name”: “VarianceScaling”, “config”: {“distribution”: “uniform”, “scale”: 1.0, “seed”: null, “mode”: “fan_avg”}}, “name”: “conv2d_4”, “bias_regularizer”: null, “bias_constraint”: null, “activation”: “linear”, “trainable”: true, “data_format”: “channels_last”, “padding”: “same”, “strides”: [1, 1], “dilation_rate”: [1, 1], “kernel_regularizer”: null, “filters”: 128, “bias_initializer”: {“class_name”: “Zeros”, “config”: {}}, “use_bias”: true, “activity_regularizer”: null, “kernel_size”: [3, 3]}}, {“class_name”: “Activation”, “config”: {“activation”: “relu”, “trainable”: true, “name”: “activation_4”}}, {“class_name”: “BatchNormalization”, “config”: {“beta_constraint”: null, “gamma_initializer”: {“class_name”: “Ones”, “config”: {}}, “moving_mean_initializer”: {“class_name”: “Zeros”, “config”: {}}, “name”: “batch_normalization_4”, “epsilon”: 0.001, “trainable”: true, “moving_variance_initializer”: {“class_name”: “Ones”, “config”: {}}, “beta_initializer”: {“class_name”: “Zeros”, “config”: {}}, “scale”: true, “axis”: -1, “gamma_constraint”: null, “gamma_regularizer”: null, “beta_regularizer”: null, “momentum”: 0.99, “center”: true}}, {“class_name”: “Conv2D”, “config”: {“kernel_constraint”: null, “kernel_initializer”: {“class_name”: “VarianceScaling”, “config”: {“distribution”: “uniform”, “scale”: 1.0, “seed”: null, “mode”: “fan_avg”}}, “name”: “conv2d_5”, “bias_regularizer”: null, “bias_constraint”: null, “activation”: “linear”, “trainable”: true, “data_format”: “channels_last”, “padding”: “same”, “strides”: [1, 1], “dilation_rate”: [1, 1], “kernel_regularizer”: null, “filters”: 128, “bias_initializer”: {“class_name”: “Zeros”, “config”: {}}, “use_bias”: true, “activity_regularizer”: null, “kernel_size”: [3, 3]}}, {“class_name”: “Activation”, “config”: {“activation”: “relu”, “trainable”: true, “name”: “activation_5”}}, {“class_name”: “BatchNormalization”, “config”: {“beta_constraint”: null, “gamma_initializer”: {“class_name”: “Ones”, “config”: {}}, “moving_mean_initializer”: {“class_name”: “Zeros”, “config”: {}}, “name”: “batch_normalization_5”, “epsilon”: 0.001, “trainable”: true, “moving_variance_initializer”: {“class_name”: “Ones”, “config”: {}}, “beta_initializer”: {“class_name”: “Zeros”, “config”: {}}, “scale”: true, “axis”: -1, “gamma_constraint”: null, “gamma_regularizer”: null, “beta_regularizer”: null, “momentum”: 0.99, “center”: true}}, {“class_name”: “Conv2D”, “config”: {“kernel_constraint”: null, “kernel_initializer”: {“class_name”: “VarianceScaling”, “config”: {“distribution”: “uniform”, “scale”: 1.0, “seed”: null, “mode”: “fan_avg”}}, “name”: “conv2d_6”, “bias_regularizer”: null, “bias_constraint”: null, “activation”: “linear”, “trainable”: true, “data_format”: “channels_last”, “padding”: “same”, “strides”: [1, 1], “dilation_rate”: [1, 1], “kernel_regularizer”: null, “filters”: 128, “bias_initializer”: {“class_name”: “Zeros”, “config”: {}}, “use_bias”: true, “activity_regularizer”: null, “kernel_size”: [3, 3]}}, {“class_name”: “Activation”, “config”: {“activation”: “relu”, “trainable”: true, “name”: “activation_6”}}, {“class_name”: “BatchNormalization”, “config”: {“beta_constraint”: null, “gamma_initializer”: {“class_name”: “Ones”, “config”: {}}, “moving_mean_initializer”: {“class_name”: “Zeros”, “config”: {}}, “name”: “batch_normalization_6”, “epsilon”: 0.001, “trainable”: true, “moving_variance_initializer”: {“class_name”: “Ones”, “config”: {}}, “beta_initializer”: {“class_name”: “Zeros”, “config”: {}}, “scale”: true, “axis”: -1, “gamma_constraint”: null, “gamma_regularizer”: null, “beta_regularizer”: null, “momentum”: 0.99, “center”: true}}, {“class_name”: “MaxPooling2D”, “config”: {“name”: “max_pooling2d_3”, “trainable”: true, “data_format”: “channels_last”, “pool_size”: [2, 2], “padding”: “valid”, “strides”: [2, 2]}}, {“class_name”: “Dropout”, “config”: {“rate”: 0.25, “noise_shape”: null, “trainable”: true, “seed”: null, “name”: “dropout_3”}}, {“class_name”: “Flatten”, “config”: {“trainable”: true, “name”: “flatten_1”, “data_format”: “channels_last”}}, {“class_name”: “Dense”, “config”: {“kernel_initializer”: {“class_name”: “VarianceScaling”, “config”: {“distribution”: “uniform”, “scale”: 1.0, “seed”: null, “mode”: “fan_avg”}}, “name”: “dense_1”, “kernel_constraint”: null, “bias_regularizer”: null, “bias_constraint”: null, “activation”: “linear”, “trainable”: true, “kernel_regularizer”: null, “bias_initializer”: {“class_name”: “Zeros”, “config”: {}}, “units”: 512, “use_bias”: true, “activity_regularizer”: null}}, {“class_name”: “Activation”, “config”: {“activation”: “relu”, “trainable”: true, “name”: “activation_7”}}, {“class_name”: “BatchNormalization”, “config”: {“beta_constraint”: null, “gamma_initializer”: {“class_name”: “Ones”, “config”: {}}, “moving_mean_initializer”: {“class_name”: “Zeros”, “config”: {}}, “name”: “batch_normalization_7”, “epsilon”: 0.001, “trainable”: true, “moving_variance_initializer”: {“class_name”: “Ones”, “config”: {}}, “beta_initializer”: {“class_name”: “Zeros”, “config”: {}}, “scale”: true, “axis”: -1, “gamma_constraint”: null, “gamma_regularizer”: null, “beta_regularizer”: null, “momentum”: 0.99, “center”: true}}, {“class_name”: “Dropout”, “config”: {“rate”: 0.5, “noise_shape”: null, “trainable”: true, “seed”: null, “name”: “dropout_4”}}, {“class_name”: “Dense”, “config”: {“kernel_initializer”: {“class_name”: “VarianceScaling”, “config”: {“distribution”: “uniform”, “scale”: 1.0, “seed”: null, “mode”: “fan_avg”}}, “name”: “dense_2”, “kernel_constraint”: null, “bias_regularizer”: null, “bias_constraint”: null, “activation”: “linear”, “trainable”: true, “kernel_regularizer”: null, “bias_initializer”: {“class_name”: “Zeros”, “config”: {}}, “units”: 9, “use_bias”: true, “activity_regularizer”: null}}, {“class_name”: “Activation”, “config”: {“activation”: “softmax”, “trainable”: true, “name”: “activation_8”}}], “backend”: “tensorflow”}

Yves · June 19, 2019, 6:52am

@vfintinari curiously after redeploying the app and retrying today, it seems to work! Thanks for the help.

Yves · June 19, 2019, 9:40am

Additionally, @vfintinari, there seems to be a typo on the param filter_thresh on MovementBasedSegmenter

What image is handed to the model by the MovementBasedSegmenter? Is it the whole watcher area or is it a cutout of the contours it finds?

vfintinari · June 20, 2019, 5:54am

Hi @Yves,
Glad it worked!

It really is a typo. I will make sure that it is fixed. Thank you for letting us know.
The image of the whole watcher area is forwarded to the model.

Yves · June 20, 2019, 1:00pm

The image of the whole watcher area is forwarded to the model.

So basically if I have a already classified square on the table, then add a new one, the MBS first detects the new square and computes its boundaries, verifies that it is indeed a new object and then calls the custom model with the image of the whole watcher area. I would then have to do the same process in my custom network to get the boundaries of the new object and then classify it.

Is this assumption correct? And if so, wouldn’t it make more sense to just call the custom model with just a cutout of the new object?

vsirbu · June 24, 2019, 2:31pm

Hi @Yves,

Here are some points to clarify the way MBS works:

It detects the contour.
It then sends to the neural network only a crop of the detected contour. This could be done in two ways:

a) based on the size of the input of your neural network, cutting from the center of the contour half the size of your nn input size;

b) based on a desired region of interest size. This could be done by providing a file named <name_of_you_nn>.config.json when you load your neural network, which contains the parameter region_of_interest_size. In this case the value of region_of_interest_size will be taken into account when the crop is being made and then resized to the size of your nn.

An example of this <name_of_you_nn>.config.json:
```
{
  "architecture": "",
  "region_of_interest_size": {
  "width": 175,
  "height": 175
  },
  "filters": {}
}
```
This will result in a crop that has a width and height of 175 pixels.

If you wish to take into consideration the 60 cm Lampix configuration and the 90 cm configuration you can provide parameters height_1 and height_2, respectively. An example:
```
{
  "architecture": "",
  "region_of_interest_size": {
    "height_1":{
      "width": 250,
      "height": 250
    },
    "height_2":{
      "width": 100,
      "height": 100
    }
  },
  "filters": {}
}
```
In other words, at 60 cm your crop will have 250 pixels in width and height and when placed at 90 cm 100 pixels.

This is not officially documented but shall be after a future release.