About Zhenyi He

This is Zhenyi He, a PhD student in NYU Future Reality Lab(FRL) advised by Professor Ken Perlin. Currently, I am working on Virtual Reality (VR) and Augmented Reality(AR) especially in Human-Computer Interaction (HCI) area.

Before I joined FRL, I was in Digital Art Lab in Shanghai Jiao Tong University and advised by Professor Xubo Yang.

Also, I had collaboration with Professor Xiaojuan Ma from HKUST and Professor Christof Lutteroth from University of Bath.

Welcome to my reality. I’d appreciate it if you enjoy any second of my work. Contact me if your interests collide with mine. Leave comments or email me: zh719@nyu.edu.


Gaze-assisted typing system (Ongoing)

Instead of locating where is the keyboard and where are the fingers, we provide a new typing system which helps save that effort for typing in VR/AR/Reality.

Inside-out Tracking Full-body Avatar(Ongoing)

We provide a novel machine-learning approach to reconstruct human body from only inside-out tracking devices.

Avatar Representation in VR Video Conferencing (Intern Project)

We implement a 2D video image representation and a 3D avatar representation for comparison. To solve the uncanny feeling of 2D video image in VR, we designed no-change, cropped, segmented, frame, and portal. We applied social presence index framework for an on-going user study.

Exploring Configuration of Mixed Reality Spaces for Communication

Zhenyi He, Karl Toby Rosenberg, and Ken Perlin. In Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems (CHI EA ’19).

We present our MR multi-user workstation system to investigate how different configurations of participants and content affect communication between people in the same immersive environment. We designed and implemented side-by-side, mirrored face-to-face and eyes-free configurations in our multi-user MR environment.

video1 video2

Handwriting in VR (Intern Project)

We propose a handwriting approach with haptic feedback in VR for surface writing. Users can either hold a regular pen or conduct a pinch gesture in VR to perform writing tasks. Gestures are detected by the head mounted cameras with learning algorithms and surfaces are detected with markers.

Manifest the Invisible: Design for Situational Awareness(SA) of Physical Environments in VR

Zhenyi He, Fengyuan Zhu, Ken Perlin, Xiaojuan Ma, Arxiv 2018.

Five representational fidelities (indexical, symbolic, and iconic with three emotions, positive, neutral, and negative) for displaying ambient entities in VR based on the existing aesthetic theory are designed, and the efficacy of these designs are explored in terms of raising users’ SA while preventing breaks in virtual presence.


Chalktalk (A series of work)

Perlin, K., He, Z., & Zhu, F. (2018). Chalktalk VR/AR. International SERIES On Information Systems And Management In Creative EMedia (CreMedia), (2017/2), 30-31.

Talk as a representative of Ken in Global Cre8 summit

Chalktalk VR/AR is a paradigm for creating drawings in the context of a face to face brainstorming session that is happening with the support of VR or AR. Participants draw their ideas in the form of simple sketched simulation elements, which can appear to be floating in the air between participants. Those elements are then recognized by a simple AI recognition system, and can be interactively incorporated by participants into an emerging simulation that builds more complex simulations by linking together these simulation elements in the course of the discussion.

paper1 paper2 video


HoloKit, the low cost open source mixed reality experience, which includes the HeadKit cardboard headset and TrackKit software.
With your smartphone and Mixed Reality apps, HoloKit provides you access to the world of Mixed Reality right in your hands, affordably.
I helped create several demos for the release.

PhyShare: Sharing Physical Interaction in Virtual Reality

Zhenyi He, Fengyuan Zhu, and Ken Perlin. PhyShare: Sharing Physical Interaction in Virtual Reality. In Adjunct Publication of the 30th Annual ACM Symposium on User Interface Software and Technology (UIST ’17).

A new approach for interaction in VR using robots as proxies for haptic feedback. This approach allows VR users to have the experience of sharing and manipulating tangible physical objects with remote collaborators.

full pdf video1 video2


Holojam is a shared immersive experience. Each participant puts on a lightweight
wireless motion-tracked GearVR headset as well as strap-on wrist and ankle markers. These devices allow them to see everyone else as an avatar, walk around the physical world, and interact with real physical objects. Participants see the physical world around them, but it is visually transformed. People can draw in the air and collaborate by freely mixing physical and virtual objects.

Object Creation Based on Free Virtual Grid with ARGlass

Zhenyi He and Xubo Yang. 2014. Object creation based on free virtual grid with ARGlass. In SIGGRAPH Asia 2014 Posters (SA ’14).

A homemade AR eyeglasses is created by cameras, leap motion and Oculus VR and
a new approach called Free Virtual Grid is designed to help create 3D models. User could rotate the free virtual grid and scale the square size to moderate the orientation and precision.

Professional Experience

Software Engineer Intern @ Facebook AI Face Tracking.
Full-time intern, June 2019 – August 2019

Research Intern @ Oculus Pittsburgh.
Full-time intern, June 2018 – August 2018,
Part-time intern, October 2018 – December 2018.

Academic Service (Program Committee)

  • Associate Chair (AC) for CHI2020 Late Breaking Work

Academic Service (Reviewer)

  • CHI 2020
  • UIST 2019
  • IEEE VR 2019
  • CHI 2019
  • VRST 2018

Teaching Assistant

  • Computer Graphics 9/2017 — 12/2017 and 9/2016 — 12/2016
  • Human Computer Interaction and Interface 2/2012 — 6/2012 and 2/2013 — 6/2013
    Received Outstanding Teaching Assistant Award
  • Programming and Data Structure 9/2012 — 1/2013 and 2/2013 — 6/2013

Awards and Honors

  • The 2017 Snap Inc. Research Fellowship (top 12), 2017.
  • Merit Students from the university(top 3 in dept), 2014.
  • Outstanding League Leader, 2013.
  • Outstanding Teaching Assistant, 2012.
  • Shanghai Outstanding Undergraduate Student(top 6 in dept), 2012.
  • Outstanding League Member, 2011.
  • Merit Students from the university, 2010.
  • Advanced Individual from the university, 2010.
  • Third Grade of CASC Airplane Scholarship(top 10 in dept), 2013.
  • People’s Scholarship Grade B from the university, 2011.