About Zhenyi He

This is Zhenyi He, a PhD student in NYU Future Reality Lab(FRL) advised by Professor Ken Perlin. Currently, I am working on Virtual Reality (VR) and Augmented Reality(AR) especially in Human-Computer Interaction (HCI) area.

Before I joined FRL, I was in Digital Art Lab in Shanghai Jiao Tong University and advised by Professor Xubo Yang.

Also, I had collaboration with Professor Xiaojuan Ma from HKUST, Professor Christof Lutteroth from University of Bath, and Google research scientist Ruofei Du.

Welcome to my reality. I’d appreciate it if you enjoy any second of my work. Contact me if your interests collide with mine. Leave comments or email me: zhenyi.he@nyu.edu.


CollaboVR: A Reconfigurable Framework for Creative Collaboration in Virtual Reality

Writing or sketching on whiteboards is an essential part of collaborative discussions in business meetings, reading groups, design sessions, and interviews. However, prior work in collaborative virtual reality (VR) systems have rarely explored the design space of multi-user layouts and interaction modes with virtual whiteboards. In this paper, we present CollaboVR, a reconfigurable framework for both co-located and geographically dispersed multi-user communication in VR. Our system unleashes users’ creativity by sharing freehand drawings, converting 2D sketches into 3D models, and generating procedural animations in real-time. To minimize the computational expense for VR clients, we leverage a cloud architecture in which the computational expensive application (Chalktalk) is hosted directly on the servers, with results being simultaneously streamed to clients. We have explored three custom layouts — integrated, mirrored, and projective — to reduce visual clutter, increase eye contact, or adapt different use cases. To evaluate CollaboVR, we conducted a within-subject user study with 12 participants. Our findings reveal that users appreciate the custom configurations and real-time interactions provided by CollaboVR. We will open source CollaboVR to facilitate future research and development of natural user interfaces and real-time collaborative systems in virtual and augmented reality.

The paper is conditionally accepted by ISMAR 2020 and invited by I3D 2020 poster session.

TapGazer: Touch Typing with In-place Finger Tapping and Gaze-directed Word Completion (Ongoing)

Present a text entry system in which users type by tapping their fingers in place, as though on a QWERTY keyboard, without needing to look at their hands on any available surface, based on the anticipation of lightweight eye-glass enabling eye-tracking scenarios. Users are available to reach 60 word per minute.


Inside-out Tracking Full-body Avatar(Ongoing)

We provide a novel machine-learning approach to reconstruct human body from only inside-out tracking devices.

Avatar Representation in VR Video Conferencing (Intern Project)

We implement a 2D video image representation and a 3D avatar representation for comparison. To solve the uncanny feeling of 2D video image in VR, we designed no-change, cropped, segmented, frame, and portal. We applied social presence index framework for an on-going user study.

Exploring Configuration of Mixed Reality Spaces for Communication

Zhenyi He, Karl Toby Rosenberg, and Ken Perlin. In Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems (CHI EA ’19).

We present our MR multi-user workstation system to investigate how different configurations of participants and content affect communication between people in the same immersive environment. We designed and implemented side-by-side, mirrored face-to-face and eyes-free configurations in our multi-user MR environment.

video1 video2

Handwriting in VR (Intern Project)

We propose a handwriting approach with haptic feedback in VR for surface writing. Users can either hold a regular pen or conduct a pinch gesture in VR to perform writing tasks. Gestures are detected by the head mounted cameras with learning algorithms and surfaces are detected with markers.

Manifest the Invisible: Design for Situational Awareness(SA) of Physical Environments in VR

Zhenyi He, Fengyuan Zhu, Ken Perlin, Xiaojuan Ma, Arxiv 2018.

Five representational fidelities (indexical, symbolic, and iconic with three emotions, positive, neutral, and negative) for displaying ambient entities in VR based on the existing aesthetic theory are designed, and the efficacy of these designs are explored in terms of raising users’ SA while preventing breaks in virtual presence.


Chalktalk (A series of work)

Perlin, K., He, Z., & Zhu, F. (2018). Chalktalk VR/AR. International SERIES On Information Systems And Management In Creative EMedia (CreMedia), (2017/2), 30-31.

Talk as a representative of Ken in Global Cre8 summit

Chalktalk VR/AR is a paradigm for creating drawings in the context of a face to face brainstorming session that is happening with the support of VR or AR. Participants draw their ideas in the form of simple sketched simulation elements, which can appear to be floating in the air between participants. Those elements are then recognized by a simple AI recognition system, and can be interactively incorporated by participants into an emerging simulation that builds more complex simulations by linking together these simulation elements in the course of the discussion.

paper1 paper2 video


HoloKit, the low cost open source mixed reality experience, which includes the HeadKit cardboard headset and TrackKit software.
With your smartphone and Mixed Reality apps, HoloKit provides you access to the world of Mixed Reality right in your hands, affordably.
I helped create several demos for the release.

PhyShare: Sharing Physical Interaction in Virtual Reality

Zhenyi He, Fengyuan Zhu, and Ken Perlin. PhyShare: Sharing Physical Interaction in Virtual Reality. In Adjunct Publication of the 30th Annual ACM Symposium on User Interface Software and Technology (UIST ’17).

A new approach for interaction in VR using robots as proxies for haptic feedback. This approach allows VR users to have the experience of sharing and manipulating tangible physical objects with remote collaborators.

full pdf video1 video2


Holojam is a shared immersive experience. Each participant puts on a lightweight
wireless motion-tracked GearVR headset as well as strap-on wrist and ankle markers. These devices allow them to see everyone else as an avatar, walk around the physical world, and interact with real physical objects. Participants see the physical world around them, but it is visually transformed. People can draw in the air and collaborate by freely mixing physical and virtual objects.

Object Creation Based on Free Virtual Grid with ARGlass

Zhenyi He and Xubo Yang. 2014. Object creation based on free virtual grid with ARGlass. In SIGGRAPH Asia 2014 Posters (SA ’14).

A homemade AR eyeglasses is created by cameras, leap motion and Oculus VR and
a new approach called Free Virtual Grid is designed to help create 3D models. User could rotate the free virtual grid and scale the square size to moderate the orientation and precision.

Exhibition Experience

  • The outpost in SIGGRAPH 2020
  • ChalktalkVR in Oculus Education in Oculus Connect 6, 2019
  • ChalktalkVR in Facebook, 2019
  • CAVE in SIGGRAPH 2018
  • HoloDoodle in SIGGRAPH 2017
  • Holokit in AR in Action, 2017
  • Chalktalk in Global Cre8 Summit in Shenzhen, 2017

Professional Experience

Software Engineer Intern @ Facebook AI Face Tracking.
Full-time intern, June 2019 – August 2019

Research Intern @ Oculus Pittsburgh.
Full-time intern, June 2018 – August 2018,
Part-time intern, October 2018 – December 2018.

Academic Service (Program Committee)

  • Associate Chair (AC) for CHI2020 Late Breaking Work

Academic Service (Reviewer)

  • CHI 2019, 2020
  • UIST 2019, 2020
  • IEEE VR 2019
  • ICMI 2020
  • VRST 2018, 2020
  • Eurohaptics 2020
  • ISS 2020

Teaching Assistant

  • Computer Graphics 9/2017 — 12/2017 and 9/2016 — 12/2016
  • Human Computer Interaction and Interface 2/2012 — 6/2012 and 2/2013 — 6/2013
    Received Outstanding Teaching Assistant Award
  • Programming and Data Structure 9/2012 — 1/2013 and 2/2013 — 6/2013

Awards and Honors

  • The 2017 Snap Inc. Research Fellowship (top 12), 2017.
  • Merit Students from the university(top 3 in dept), 2014.
  • Outstanding League Leader, 2013.
  • Outstanding Teaching Assistant, 2012.
  • Shanghai Outstanding Undergraduate Student(top 6 in dept), 2012.
  • Outstanding League Member, 2011.
  • Merit Students from the university, 2010.
  • Advanced Individual from the university, 2010.
  • Third Grade of CASC Airplane Scholarship(top 10 in dept), 2013.
  • People’s Scholarship Grade B from the university, 2011.