top of page

What If a Robot Sees You in the Form of Your Shadow?  [ongoing project]

Many robots use a camera as an input to monitor a user’s state or intention. However, when entering a personal space like home, having a camera can pose risks to the user’s privacy and reduce comfort level in interaction. Inspired by the previous work of ShadowSense,  a semi-transparent material could filter the user’s high-fidelity images into the form of their shadows. This provides a privacy-maintaining alternative to camera-based interaction with social robots. Thus, it is possible to infer what the user is doing from shadows, detecting important interaction events such as accidental falls in the scenario of elderly assistance. 

By physically covering the robot's eyes with a translucent material, as some users already do with their laptop cameras, a robot can still make use of some interaction data in the form of users' shadows instead of high-fidelity images. Thus, the robot may still infer the users' activities without seeing their full appearance, such as when capturing the users' changing clothes while not invading their privacy.

Screen Shot 2023-02-06 at
Screen Shot 2023-02-06 at 4.43.19 PM.png
Screen Shot 2023-02-06 at 4.43.19 PM.png

We test several computer vision and deep learning models to detect users' interaction-related states from users' shadow videos/images, including users' positions and activities. 

The first algorithm is based on OpenPose and was augmented with a LSTM-RNN network, trained on labeled shadow image sequences for activity detection of basic gestures (standing, waving, pointing to the direction).


The second algorithm tracks the user's position in the shadow images with bounding boxes. We use the pre-trained EfficientNet-v2 model which was retrained with our shadow image dataset that was collected using the robot's eye camera with varying backgrounds, lighting conditions, and viewing angles. The resulting algorithm is able to perform user tracking in real-time and thus allowing the robot to respond to the users' positions in various human-robot interaction scenarios. For example, it allows the robot to gaze at the user during interaction while eliminating the discomfort of recording their high-fidelity data, especially in a confidential setting. 

Test videos: A robot wearing privacy-protected glasses gazes at the user, tracks the positions with bounding boxes (as displayed on the PC screen), and uses its antenna to respond to the user.


We will provide more technical details and results once the work gets accepted for publication!

bottom of page