PeR-ViS: Person Retrieval in Video Surveillance using Semantic Description

Parshwa Shah

Arpit Garg

Vandit Gajjar

School of Engineering and Applied Science, Ahmedabad University | School of Computer Science, The University of Adelaide IEEE/CVF Winter Conference on Applications of Computer Vision Workshops (WACVw) The 3rd International Workshop on Human Activity Detection in multi-camera, Continuous, long-duration Video (HADCV'21), 2021

[Paper]

[Code]

Abstract

A person is usually characterized by descriptors like age, gender, height, cloth type, pattern, color, etc. Such descriptors are known as attributes and/or soft-biometrics. They link the semantic gap between a person’s description and retrieval in video surveillance. Retrieving a specific person with the query of semantic description has an important application in video surveillance. Using computer vision to fully automate the person retrieval task has been gathering interest within the research community. However, the Current, trend mainly focuses on retrieving persons with image-based queries, which have major limitations for practical usage. Instead of using an image query, in this paper, we study the problem of person retrieval in video surveillance with a semantic description. To solve this problem, we develop a deep learning-based cascade filtering approach (PeR-ViS), which uses Mask R-CNN [1] (person detection and instance segmentation) and DenseNet-161 [2] (soft-biometric classification). On the standard person retrieval dataset of SoftBioSearch [3], we achieve 0.566 Average IoU and 0.792 %w IoU > 0.4, surpassing the current state-of-the-art by a large margin. We hope our simple, reproducible, and effective approach will help ease future research in the domain of person retrieval in video surveillance. The source code and pretrained weights available here.

Approach

Paper

P. Shah, A. Garg, and V. Gajjar. PeR-ViS: Person Retrieval in Video Surveillance using Semantic Description. In IEEE/CVF WACVw 2021. [bibtex]

Acknowledgements

We would like to thank anonymous reviewers for providing us their valuable feedback on our paper. We would like to express our deep gratitude to Dr. Mehul S. Raval, Dr. Hiren Galiyawala and Mr. Kenil Shah for providing useful comments and discussion. We would also like to thank Ms. Ayesha Gurani, Mr. Viraj Mavani and Mr. Yash Khandhediya for their help with manuscript

References

[1] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision, pages 2961–2969, 2017.
[2] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4700–4708, 2017.
[3] Simon Denman, Michael Halstead, Clinton Fookes, and Sridha Sridharan. Searching for people using semantic soft biometric descriptions. Pattern Recognition Letters, 68:306–315, 2015.