Why This Tutorial?

Emerging platforms such as Augmented Reality (AR), Virtual Reality (VR), and autonomous machines, while are of a computing nature, intimately interact with both the environment and humans. They must be built, from the ground up, with principled considerations of three main components: imaging, computer systems, and human perception.

The goal of this tutorial is to set up CS researchers for the exciting era of human-centered visual computing, where we can both 1) leverage human perception for enhancing computing and imaging systems, and 2) develope imaging and computational tools that advance our understanding of human perceptions and, ultimately, augment human perception.

To that end, this tutorial will 1) teach the basics of visual computing from first principles, including imaging (e.g., optics, image sensors) visual content, computing on visual content, displays, and human perception and cognition, and 2) discuss exciting research opportunities for the systems and architecture community.

The tutorial is not tied to a specific product or hardware platform. Instead, we focus on teaching the fundamental principles so that participants can walk away being able to design new algorithms, building new applications, and engineering systems and hardware for those algorithms and applications.

Agenda

• Basic visual neuroscience
• Spatial/temporal/color vision
• Light/dark/chromatic adaptation
• Basics of AR/VR displays
• Display power consumption and visual quality

• Fundamental principles of imaging
• Camera optics
• Modern CMOS image sensor architecture
• Camera image-signal processing and hardware

• Computational image sensors: co-designing image sensors and computer architectures
• Human vision-guided AR/VR
• Aumenting human vision using AR/VR

Reference Materials

• MICRO 2023 iteration of this tutorial
• ASPLOS 2023 iteration of this tutorial
• Mobile Visual Computing, CSC 292 taught at University of Rochester
• Interactive Tutorials on Color Science

• BlissCam: Boosting Eye Tracking Efficiency with Learned In-Sensor Sparse Sampling, ISCA 2024
• Exploiting Human Color Discrimination for Memory- and Energy-Efficient Image Encoding in Virtual Reality, ASPLOS 2024
• CamJ: An Energy Modeling and System-Level Exploration Framework for In-Sensor Visual Computing, ISCA 2023
• Real-Time Gaze Tracking with Event-Driven Eye Segmentation, IEEE VR 2022
• Color-Perception-Guided Display Power Reduction for Virtual Reality, SIGGRAPH Asia 2022
• Euphrates: Algorithm-SoC Co-Design for Low-Power Mobile Continuous Vision, ISCA 2018