The authors of the paper "TETRIS: Towards Exploring the Robustness of Interactive Segmentation", discuss the challenge of evaluating interactive segmentation techniques that apply user inputs to refine a selection mask iteratively. Despite the widespread use of click-based interactions in segmentation, there is a lack of understanding of actual user clicking behavior and its impact on model quality. Traditional methods of evaluating interactive segmentation may not accurately reflect real-world use, potentially overestimating model robustness.

What is the problem?

There are several techniques for emulating user clicks in modern interactive segmentation benchmarks. Most of them generate clicks as follows:

  1. The click is positioned in the center of the object
  2. A region with the largest error from the previous interaction round is selected
  3. The next click is positioned at a point that is farthest from the edges of this region
  4. Steps 1–3 are repeated until the desired IoU with the ground truth has been achieved

This strategy was selected in the paper as a baseline, and the authors show that this baseline has drawbacks, due to which the current interactive segmentation benchmarks may not fully demonstrate the robustness of the models.

Robustness analysis

In my opinion, one of the main advantages of this paper is the practical demonstration of the issue. For example, the authors conducted a large-scale study with real users to prove their strong thesis about the weak robustness of modern image segmentation algorithms and the limitations of current benchmarks.

These graphics demonstrate the spread of modern interactive segmentation methods quality (IoU between predicted and ground truth masks) depending on the position of the click. Here, "baseline click" is the click, which is positioned in the center of the object.

After conducting their research with the participation of almost 2,000 people, the authors came to the following conclusion: real people very rarely place clicks in the center of objects, as suggested by existing benchmarks. State-of-the-art interactive segmentation models are extremely sensitive to the positions of user clicks. However, current benchmarks do not take this into account and do not reflect the real quality of the methods.

Proposed method

The obvious solution to the problem noted above is to iterate over all possible click positions for a more complete assessment of interactive segmentation methods. But this approach is very time-consuming. Therefore, the authors propose to limit the search area using a white-box adversarial attack.

The scheme of the proposed adversarial attack.

Evaluation protocol

The proposed adversarial attack consists of the following steps:

  1. Start from the baseline click position
  2. Optimize the click position through gradient updates to minimize or maximize IoU with the ground truth mask
  3. Repeat step 1 until convergence

Click positions obtained by minimizing IoU are included in the minimizing trajectory, while those obtained by maximizing IoU – maximizing trajectory.

Metrics

Together with their adversarial attack, the authors propose two metrics for evaluating interactive segmentation methods:

  • IoU/BIoU-Min/Max — the area under the minimizing/maximizing trajectory curve
  • IoU/BIoU-D — the difference between the area under curves of maximizing and minimizing trajectories
Left: the standard IoU/BIoU-AuC score. Right: the proposed IoU-Min/Max and IoU/BIoU-D robustness scores.

TETRIS benchmark

Also, the authors introduce the new TETRIS benchmark with 2,000 high-resolution images manually labeled with fine segmentation masks to measure the models' robustness more comprehensively.

This dataset includes images of the following objects: people, transportation, wildlife, objects, domestic animals, food, architecture, plants, and statues.

Evaluation

The authors evaluate the robustness of the modern interactive segmentation models using new metrics on the proposed TETRIS benchmark, along with standard datasets like GrabCut, Berkeley, DAVIS, and COCO-MVal.

The quality of the various models measured on the TETRIS benchmark and well-known datasets.

Conclusion

The TETRIS evaluation framework and insights from real user clicks can significantly inform the development of more robust interactive segmentation models for production environments. By understanding and anticipating user behavior, developers can design interfaces and models that better suit real-world use, potentially enhancing user satisfaction and efficacy of the models in applications such as image editing, medical image analysis, and object selection.

Share this post