Using MovementBasedSegmenter with custom neural network

Hi all

We are currently developing an application where we need to track multiple rectangles with different colors. The MovementBasedSegmenter seems to fit this use case with the option of filtering object contours and responding with the object location and id.
How do we go about creating our custom neural network? Is the structure in terms of entry points and respond methods the same as a custom watcher as described in the QRCodeDetector here?

Additionally, is there already an application with which we can send screenshots from lampix to a different device in order to train/test the network?

Hi @Yves,

The good news is that we are currently working on a trainer app and server, which you will be able to use to automatically train a new custom neural network and also deploy it to your Lampix. Please bare with us, as we are currently testing this system.

In the meantime, you could try taking pictures with a smartphone, train your custom neural network (we are using Keras) and deploy it using the ā€˜:8888/apps’ endpoint. Once this is done and the neural network shows up in Control Panel, you could develop your application using the MovementBasedSegmenter and specify the name of the neural network.

Let us know if we can further help you. :slight_smile:

We have created, trained and added the custom neural network ā€˜color-classifier’ to our device. However, we get the following error:
IndexError: tuple index out of range

Any ideas on how to fix this? Also, is there any documentation on how the custom model is loaded and used?

@Yves, Could you please provide the ā€˜.json’ file of your trained network? It seems that the architecture might not be compatible.

@vfintinari sure, here you go. Is there any documentation on how the architecture should be or how the NeuralNetworkClassifier or MovementBasedSegmenter use the provided custom network?

{ā€œclass_nameā€: ā€œSequentialā€, ā€œkeras_versionā€: ā€œ2.2.2ā€, ā€œconfigā€: [{ā€œclass_nameā€: ā€œConv2Dā€, ā€œconfigā€: {ā€œkernel_initializerā€: {ā€œclass_nameā€: ā€œVarianceScalingā€, ā€œconfigā€: {ā€œdistributionā€: ā€œuniformā€, ā€œscaleā€: 1.0, ā€œseedā€: null, ā€œmodeā€: ā€œfan_avgā€}}, ā€œnameā€: ā€œconv2d_1ā€, ā€œkernel_constraintā€: null, ā€œbias_regularizerā€: null, ā€œbias_constraintā€: null, ā€œdtypeā€: ā€œfloat32ā€, ā€œactivationā€: ā€œlinearā€, ā€œtrainableā€: true, ā€œdata_formatā€: ā€œchannels_lastā€, ā€œfiltersā€: 32, ā€œpaddingā€: ā€œsameā€, ā€œstridesā€: [1, 1], ā€œdilation_rateā€: [1, 1], ā€œkernel_regularizerā€: null, ā€œbias_initializerā€: {ā€œclass_nameā€: ā€œZerosā€, ā€œconfigā€: {}}, ā€œbatch_input_shapeā€: [null, 64, 64, 3], ā€œuse_biasā€: true, ā€œactivity_regularizerā€: null, ā€œkernel_sizeā€: [3, 3]}}, {ā€œclass_nameā€: ā€œActivationā€, ā€œconfigā€: {ā€œactivationā€: ā€œreluā€, ā€œtrainableā€: true, ā€œnameā€: ā€œactivation_1ā€}}, {ā€œclass_nameā€: ā€œBatchNormalizationā€, ā€œconfigā€: {ā€œbeta_constraintā€: null, ā€œgamma_initializerā€: {ā€œclass_nameā€: ā€œOnesā€, ā€œconfigā€: {}}, ā€œmoving_mean_initializerā€: {ā€œclass_nameā€: ā€œZerosā€, ā€œconfigā€: {}}, ā€œnameā€: ā€œbatch_normalization_1ā€, ā€œepsilonā€: 0.001, ā€œtrainableā€: true, ā€œmoving_variance_initializerā€: {ā€œclass_nameā€: ā€œOnesā€, ā€œconfigā€: {}}, ā€œbeta_initializerā€: {ā€œclass_nameā€: ā€œZerosā€, ā€œconfigā€: {}}, ā€œscaleā€: true, ā€œaxisā€: -1, ā€œgamma_constraintā€: null, ā€œgamma_regularizerā€: null, ā€œbeta_regularizerā€: null, ā€œmomentumā€: 0.99, ā€œcenterā€: true}}, {ā€œclass_nameā€: ā€œMaxPooling2Dā€, ā€œconfigā€: {ā€œnameā€: ā€œmax_pooling2d_1ā€, ā€œtrainableā€: true, ā€œdata_formatā€: ā€œchannels_lastā€, ā€œpool_sizeā€: [2, 2], ā€œpaddingā€: ā€œvalidā€, ā€œstridesā€: [2, 2]}}, {ā€œclass_nameā€: ā€œDropoutā€, ā€œconfigā€: {ā€œrateā€: 0.25, ā€œnoise_shapeā€: null, ā€œtrainableā€: true, ā€œseedā€: null, ā€œnameā€: ā€œdropout_1ā€}}, {ā€œclass_nameā€: ā€œConv2Dā€, ā€œconfigā€: {ā€œkernel_constraintā€: null, ā€œkernel_initializerā€: {ā€œclass_nameā€: ā€œVarianceScalingā€, ā€œconfigā€: {ā€œdistributionā€: ā€œuniformā€, ā€œscaleā€: 1.0, ā€œseedā€: null, ā€œmodeā€: ā€œfan_avgā€}}, ā€œnameā€: ā€œconv2d_2ā€, ā€œbias_regularizerā€: null, ā€œbias_constraintā€: null, ā€œactivationā€: ā€œlinearā€, ā€œtrainableā€: true, ā€œdata_formatā€: ā€œchannels_lastā€, ā€œpaddingā€: ā€œsameā€, ā€œstridesā€: [1, 1], ā€œdilation_rateā€: [1, 1], ā€œkernel_regularizerā€: null, ā€œfiltersā€: 64, ā€œbias_initializerā€: {ā€œclass_nameā€: ā€œZerosā€, ā€œconfigā€: {}}, ā€œuse_biasā€: true, ā€œactivity_regularizerā€: null, ā€œkernel_sizeā€: [3, 3]}}, {ā€œclass_nameā€: ā€œActivationā€, ā€œconfigā€: {ā€œactivationā€: ā€œreluā€, ā€œtrainableā€: true, ā€œnameā€: ā€œactivation_2ā€}}, {ā€œclass_nameā€: ā€œBatchNormalizationā€, ā€œconfigā€: {ā€œbeta_constraintā€: null, ā€œgamma_initializerā€: {ā€œclass_nameā€: ā€œOnesā€, ā€œconfigā€: {}}, ā€œmoving_mean_initializerā€: {ā€œclass_nameā€: ā€œZerosā€, ā€œconfigā€: {}}, ā€œnameā€: ā€œbatch_normalization_2ā€, ā€œepsilonā€: 0.001, ā€œtrainableā€: true, ā€œmoving_variance_initializerā€: {ā€œclass_nameā€: ā€œOnesā€, ā€œconfigā€: {}}, ā€œbeta_initializerā€: {ā€œclass_nameā€: ā€œZerosā€, ā€œconfigā€: {}}, ā€œscaleā€: true, ā€œaxisā€: -1, ā€œgamma_constraintā€: null, ā€œgamma_regularizerā€: null, ā€œbeta_regularizerā€: null, ā€œmomentumā€: 0.99, ā€œcenterā€: true}}, {ā€œclass_nameā€: ā€œConv2Dā€, ā€œconfigā€: {ā€œkernel_constraintā€: null, ā€œkernel_initializerā€: {ā€œclass_nameā€: ā€œVarianceScalingā€, ā€œconfigā€: {ā€œdistributionā€: ā€œuniformā€, ā€œscaleā€: 1.0, ā€œseedā€: null, ā€œmodeā€: ā€œfan_avgā€}}, ā€œnameā€: ā€œconv2d_3ā€, ā€œbias_regularizerā€: null, ā€œbias_constraintā€: null, ā€œactivationā€: ā€œlinearā€, ā€œtrainableā€: true, ā€œdata_formatā€: ā€œchannels_lastā€, ā€œpaddingā€: ā€œsameā€, ā€œstridesā€: [1, 1], ā€œdilation_rateā€: [1, 1], ā€œkernel_regularizerā€: null, ā€œfiltersā€: 64, ā€œbias_initializerā€: {ā€œclass_nameā€: ā€œZerosā€, ā€œconfigā€: {}}, ā€œuse_biasā€: true, ā€œactivity_regularizerā€: null, ā€œkernel_sizeā€: [3, 3]}}, {ā€œclass_nameā€: ā€œActivationā€, ā€œconfigā€: {ā€œactivationā€: ā€œreluā€, ā€œtrainableā€: true, ā€œnameā€: ā€œactivation_3ā€}}, {ā€œclass_nameā€: ā€œBatchNormalizationā€, ā€œconfigā€: {ā€œbeta_constraintā€: null, ā€œgamma_initializerā€: {ā€œclass_nameā€: ā€œOnesā€, ā€œconfigā€: {}}, ā€œmoving_mean_initializerā€: {ā€œclass_nameā€: ā€œZerosā€, ā€œconfigā€: {}}, ā€œnameā€: ā€œbatch_normalization_3ā€, ā€œepsilonā€: 0.001, ā€œtrainableā€: true, ā€œmoving_variance_initializerā€: {ā€œclass_nameā€: ā€œOnesā€, ā€œconfigā€: {}}, ā€œbeta_initializerā€: {ā€œclass_nameā€: ā€œZerosā€, ā€œconfigā€: {}}, ā€œscaleā€: true, ā€œaxisā€: -1, ā€œgamma_constraintā€: null, ā€œgamma_regularizerā€: null, ā€œbeta_regularizerā€: null, ā€œmomentumā€: 0.99, ā€œcenterā€: true}}, {ā€œclass_nameā€: ā€œMaxPooling2Dā€, ā€œconfigā€: {ā€œnameā€: ā€œmax_pooling2d_2ā€, ā€œtrainableā€: true, ā€œdata_formatā€: ā€œchannels_lastā€, ā€œpool_sizeā€: [2, 2], ā€œpaddingā€: ā€œvalidā€, ā€œstridesā€: [2, 2]}}, {ā€œclass_nameā€: ā€œDropoutā€, ā€œconfigā€: {ā€œrateā€: 0.25, ā€œnoise_shapeā€: null, ā€œtrainableā€: true, ā€œseedā€: null, ā€œnameā€: ā€œdropout_2ā€}}, {ā€œclass_nameā€: ā€œConv2Dā€, ā€œconfigā€: {ā€œkernel_constraintā€: null, ā€œkernel_initializerā€: {ā€œclass_nameā€: ā€œVarianceScalingā€, ā€œconfigā€: {ā€œdistributionā€: ā€œuniformā€, ā€œscaleā€: 1.0, ā€œseedā€: null, ā€œmodeā€: ā€œfan_avgā€}}, ā€œnameā€: ā€œconv2d_4ā€, ā€œbias_regularizerā€: null, ā€œbias_constraintā€: null, ā€œactivationā€: ā€œlinearā€, ā€œtrainableā€: true, ā€œdata_formatā€: ā€œchannels_lastā€, ā€œpaddingā€: ā€œsameā€, ā€œstridesā€: [1, 1], ā€œdilation_rateā€: [1, 1], ā€œkernel_regularizerā€: null, ā€œfiltersā€: 128, ā€œbias_initializerā€: {ā€œclass_nameā€: ā€œZerosā€, ā€œconfigā€: {}}, ā€œuse_biasā€: true, ā€œactivity_regularizerā€: null, ā€œkernel_sizeā€: [3, 3]}}, {ā€œclass_nameā€: ā€œActivationā€, ā€œconfigā€: {ā€œactivationā€: ā€œreluā€, ā€œtrainableā€: true, ā€œnameā€: ā€œactivation_4ā€}}, {ā€œclass_nameā€: ā€œBatchNormalizationā€, ā€œconfigā€: {ā€œbeta_constraintā€: null, ā€œgamma_initializerā€: {ā€œclass_nameā€: ā€œOnesā€, ā€œconfigā€: {}}, ā€œmoving_mean_initializerā€: {ā€œclass_nameā€: ā€œZerosā€, ā€œconfigā€: {}}, ā€œnameā€: ā€œbatch_normalization_4ā€, ā€œepsilonā€: 0.001, ā€œtrainableā€: true, ā€œmoving_variance_initializerā€: {ā€œclass_nameā€: ā€œOnesā€, ā€œconfigā€: {}}, ā€œbeta_initializerā€: {ā€œclass_nameā€: ā€œZerosā€, ā€œconfigā€: {}}, ā€œscaleā€: true, ā€œaxisā€: -1, ā€œgamma_constraintā€: null, ā€œgamma_regularizerā€: null, ā€œbeta_regularizerā€: null, ā€œmomentumā€: 0.99, ā€œcenterā€: true}}, {ā€œclass_nameā€: ā€œConv2Dā€, ā€œconfigā€: {ā€œkernel_constraintā€: null, ā€œkernel_initializerā€: {ā€œclass_nameā€: ā€œVarianceScalingā€, ā€œconfigā€: {ā€œdistributionā€: ā€œuniformā€, ā€œscaleā€: 1.0, ā€œseedā€: null, ā€œmodeā€: ā€œfan_avgā€}}, ā€œnameā€: ā€œconv2d_5ā€, ā€œbias_regularizerā€: null, ā€œbias_constraintā€: null, ā€œactivationā€: ā€œlinearā€, ā€œtrainableā€: true, ā€œdata_formatā€: ā€œchannels_lastā€, ā€œpaddingā€: ā€œsameā€, ā€œstridesā€: [1, 1], ā€œdilation_rateā€: [1, 1], ā€œkernel_regularizerā€: null, ā€œfiltersā€: 128, ā€œbias_initializerā€: {ā€œclass_nameā€: ā€œZerosā€, ā€œconfigā€: {}}, ā€œuse_biasā€: true, ā€œactivity_regularizerā€: null, ā€œkernel_sizeā€: [3, 3]}}, {ā€œclass_nameā€: ā€œActivationā€, ā€œconfigā€: {ā€œactivationā€: ā€œreluā€, ā€œtrainableā€: true, ā€œnameā€: ā€œactivation_5ā€}}, {ā€œclass_nameā€: ā€œBatchNormalizationā€, ā€œconfigā€: {ā€œbeta_constraintā€: null, ā€œgamma_initializerā€: {ā€œclass_nameā€: ā€œOnesā€, ā€œconfigā€: {}}, ā€œmoving_mean_initializerā€: {ā€œclass_nameā€: ā€œZerosā€, ā€œconfigā€: {}}, ā€œnameā€: ā€œbatch_normalization_5ā€, ā€œepsilonā€: 0.001, ā€œtrainableā€: true, ā€œmoving_variance_initializerā€: {ā€œclass_nameā€: ā€œOnesā€, ā€œconfigā€: {}}, ā€œbeta_initializerā€: {ā€œclass_nameā€: ā€œZerosā€, ā€œconfigā€: {}}, ā€œscaleā€: true, ā€œaxisā€: -1, ā€œgamma_constraintā€: null, ā€œgamma_regularizerā€: null, ā€œbeta_regularizerā€: null, ā€œmomentumā€: 0.99, ā€œcenterā€: true}}, {ā€œclass_nameā€: ā€œConv2Dā€, ā€œconfigā€: {ā€œkernel_constraintā€: null, ā€œkernel_initializerā€: {ā€œclass_nameā€: ā€œVarianceScalingā€, ā€œconfigā€: {ā€œdistributionā€: ā€œuniformā€, ā€œscaleā€: 1.0, ā€œseedā€: null, ā€œmodeā€: ā€œfan_avgā€}}, ā€œnameā€: ā€œconv2d_6ā€, ā€œbias_regularizerā€: null, ā€œbias_constraintā€: null, ā€œactivationā€: ā€œlinearā€, ā€œtrainableā€: true, ā€œdata_formatā€: ā€œchannels_lastā€, ā€œpaddingā€: ā€œsameā€, ā€œstridesā€: [1, 1], ā€œdilation_rateā€: [1, 1], ā€œkernel_regularizerā€: null, ā€œfiltersā€: 128, ā€œbias_initializerā€: {ā€œclass_nameā€: ā€œZerosā€, ā€œconfigā€: {}}, ā€œuse_biasā€: true, ā€œactivity_regularizerā€: null, ā€œkernel_sizeā€: [3, 3]}}, {ā€œclass_nameā€: ā€œActivationā€, ā€œconfigā€: {ā€œactivationā€: ā€œreluā€, ā€œtrainableā€: true, ā€œnameā€: ā€œactivation_6ā€}}, {ā€œclass_nameā€: ā€œBatchNormalizationā€, ā€œconfigā€: {ā€œbeta_constraintā€: null, ā€œgamma_initializerā€: {ā€œclass_nameā€: ā€œOnesā€, ā€œconfigā€: {}}, ā€œmoving_mean_initializerā€: {ā€œclass_nameā€: ā€œZerosā€, ā€œconfigā€: {}}, ā€œnameā€: ā€œbatch_normalization_6ā€, ā€œepsilonā€: 0.001, ā€œtrainableā€: true, ā€œmoving_variance_initializerā€: {ā€œclass_nameā€: ā€œOnesā€, ā€œconfigā€: {}}, ā€œbeta_initializerā€: {ā€œclass_nameā€: ā€œZerosā€, ā€œconfigā€: {}}, ā€œscaleā€: true, ā€œaxisā€: -1, ā€œgamma_constraintā€: null, ā€œgamma_regularizerā€: null, ā€œbeta_regularizerā€: null, ā€œmomentumā€: 0.99, ā€œcenterā€: true}}, {ā€œclass_nameā€: ā€œMaxPooling2Dā€, ā€œconfigā€: {ā€œnameā€: ā€œmax_pooling2d_3ā€, ā€œtrainableā€: true, ā€œdata_formatā€: ā€œchannels_lastā€, ā€œpool_sizeā€: [2, 2], ā€œpaddingā€: ā€œvalidā€, ā€œstridesā€: [2, 2]}}, {ā€œclass_nameā€: ā€œDropoutā€, ā€œconfigā€: {ā€œrateā€: 0.25, ā€œnoise_shapeā€: null, ā€œtrainableā€: true, ā€œseedā€: null, ā€œnameā€: ā€œdropout_3ā€}}, {ā€œclass_nameā€: ā€œFlattenā€, ā€œconfigā€: {ā€œtrainableā€: true, ā€œnameā€: ā€œflatten_1ā€, ā€œdata_formatā€: ā€œchannels_lastā€}}, {ā€œclass_nameā€: ā€œDenseā€, ā€œconfigā€: {ā€œkernel_initializerā€: {ā€œclass_nameā€: ā€œVarianceScalingā€, ā€œconfigā€: {ā€œdistributionā€: ā€œuniformā€, ā€œscaleā€: 1.0, ā€œseedā€: null, ā€œmodeā€: ā€œfan_avgā€}}, ā€œnameā€: ā€œdense_1ā€, ā€œkernel_constraintā€: null, ā€œbias_regularizerā€: null, ā€œbias_constraintā€: null, ā€œactivationā€: ā€œlinearā€, ā€œtrainableā€: true, ā€œkernel_regularizerā€: null, ā€œbias_initializerā€: {ā€œclass_nameā€: ā€œZerosā€, ā€œconfigā€: {}}, ā€œunitsā€: 512, ā€œuse_biasā€: true, ā€œactivity_regularizerā€: null}}, {ā€œclass_nameā€: ā€œActivationā€, ā€œconfigā€: {ā€œactivationā€: ā€œreluā€, ā€œtrainableā€: true, ā€œnameā€: ā€œactivation_7ā€}}, {ā€œclass_nameā€: ā€œBatchNormalizationā€, ā€œconfigā€: {ā€œbeta_constraintā€: null, ā€œgamma_initializerā€: {ā€œclass_nameā€: ā€œOnesā€, ā€œconfigā€: {}}, ā€œmoving_mean_initializerā€: {ā€œclass_nameā€: ā€œZerosā€, ā€œconfigā€: {}}, ā€œnameā€: ā€œbatch_normalization_7ā€, ā€œepsilonā€: 0.001, ā€œtrainableā€: true, ā€œmoving_variance_initializerā€: {ā€œclass_nameā€: ā€œOnesā€, ā€œconfigā€: {}}, ā€œbeta_initializerā€: {ā€œclass_nameā€: ā€œZerosā€, ā€œconfigā€: {}}, ā€œscaleā€: true, ā€œaxisā€: -1, ā€œgamma_constraintā€: null, ā€œgamma_regularizerā€: null, ā€œbeta_regularizerā€: null, ā€œmomentumā€: 0.99, ā€œcenterā€: true}}, {ā€œclass_nameā€: ā€œDropoutā€, ā€œconfigā€: {ā€œrateā€: 0.5, ā€œnoise_shapeā€: null, ā€œtrainableā€: true, ā€œseedā€: null, ā€œnameā€: ā€œdropout_4ā€}}, {ā€œclass_nameā€: ā€œDenseā€, ā€œconfigā€: {ā€œkernel_initializerā€: {ā€œclass_nameā€: ā€œVarianceScalingā€, ā€œconfigā€: {ā€œdistributionā€: ā€œuniformā€, ā€œscaleā€: 1.0, ā€œseedā€: null, ā€œmodeā€: ā€œfan_avgā€}}, ā€œnameā€: ā€œdense_2ā€, ā€œkernel_constraintā€: null, ā€œbias_regularizerā€: null, ā€œbias_constraintā€: null, ā€œactivationā€: ā€œlinearā€, ā€œtrainableā€: true, ā€œkernel_regularizerā€: null, ā€œbias_initializerā€: {ā€œclass_nameā€: ā€œZerosā€, ā€œconfigā€: {}}, ā€œunitsā€: 9, ā€œuse_biasā€: true, ā€œactivity_regularizerā€: null}}, {ā€œclass_nameā€: ā€œActivationā€, ā€œconfigā€: {ā€œactivationā€: ā€œsoftmaxā€, ā€œtrainableā€: true, ā€œnameā€: ā€œactivation_8ā€}}], ā€œbackendā€: ā€œtensorflowā€}

@vfintinari curiously after redeploying the app and retrying today, it seems to work! Thanks for the help.

1 Like

Additionally, @vfintinari, there seems to be a typo on the param filter_thresh on MovementBasedSegmenter

What image is handed to the model by the MovementBasedSegmenter? Is it the whole watcher area or is it a cutout of the contours it finds?

Hi @Yves,
Glad it worked!

  1. It really is a typo. I will make sure that it is fixed. Thank you for letting us know.

  2. The image of the whole watcher area is forwarded to the model.

The image of the whole watcher area is forwarded to the model.

So basically if I have a already classified square on the table, then add a new one, the MBS first detects the new square and computes its boundaries, verifies that it is indeed a new object and then calls the custom model with the image of the whole watcher area. I would then have to do the same process in my custom network to get the boundaries of the new object and then classify it.

Is this assumption correct? And if so, wouldn’t it make more sense to just call the custom model with just a cutout of the new object?

Hi @Yves,

Here are some points to clarify the way MBS works:

  1. It detects the contour.

  2. It then sends to the neural network only a crop of the detected contour. This could be done in two ways:

    a) based on the size of the input of your neural network, cutting from the center of the contour half the size of your nn input size;

    b) based on a desired region of interest size. This could be done by providing a file named <name_of_you_nn>.config.json when you load your neural network, which contains the parameter region_of_interest_size. In this case the value of region_of_interest_size will be taken into account when the crop is being made and then resized to the size of your nn.

    An example of this <name_of_you_nn>.config.json:

    {
      "architecture": "",
      "region_of_interest_size": {
      "width": 175,
      "height": 175
      },
      "filters": {}
    }
    

    This will result in a crop that has a width and height of 175 pixels.

    If you wish to take into consideration the 60 cm Lampix configuration and the 90 cm configuration you can provide parameters height_1 and height_2, respectively. An example:

    {
      "architecture": "",
      "region_of_interest_size": {
        "height_1":{
          "width": 250,
          "height": 250
        },
        "height_2":{
          "width": 100,
          "height": 100
        }
      },
      "filters": {}
    }
    

    In other words, at 60 cm your crop will have 250 pixels in width and height and when placed at 90 cm 100 pixels.

    This is not officially documented but shall be after a future release.

1 Like