Skip to content

Evaluator

Description

Evaluator component performs analysis of the trained model. The component uses the validation dataset. The component is used in creating Training and Inference pipelines. The Robotika Evaluator component is the inheritor of the Evaluator TFx [1] component with an extended UX.

Inputs
  1. Label key – is the name of the label in the prediction label in the current dataset. The field is mandatory.
  2. Metric name – the name of the metric used for evluation. User can identify a different metric from the one determined in the trainer component. (mandatory field)
  3. Lower and the upper values – Robotika checks whether the value of the metric is within the interval and bless the model forward. Otherwise, the model would not be blessed.
  4. Change direction – Robotika compares the performance of the model with the previously blessed model and bless the current model if the performance is improved according to the indicated “change direction”. Usually this parameter is used when the training is performed on the same data set with different models.
  5. Change value – Robotika compares the performance of the model with the previously blessed model. If the performance metric is better than previously blessed model by at least the indicated value, the model will be blessed.

The label key and metrics are are mandatory. You can indicate the upper and lower band for the performance of the algorithm.

How to use

First, the label key and metrics should be indicated.

The upper and lower band values are optional. Usually, the values are indicated as internal validation of the model. For example, if the value is below some chosen threshold value, it indicated that the model most likely do not perform properly and returns invalid outcomes. On the other hand, if the performance metric is too high, it is a good indication that something could be wrong along the pipeline.

The change direction and change value could be used in a situation when we want to compare several models for the identical data sets.

References

[1] TFX transform component by Google. https://www.tensorflow.org/tfx/guide/transform