Visual Computing for Computer Architects

First Principles and Research Challenges

Joint Tutorial and Workshop Co-located with ISCA 2024

Merged with the EVGA Workshop

9:00 AM — 17:30 PM, June 30, 2024 (Alerce)

University of Rochester
University of Michigan
University of Toronto

Why This Event?

Emerging platforms such as Augmented Reality (AR), Virtual Reality (VR), and autonomous machines, while are of a computing nature, intimately interact with both the environment and humans. They must be built, from the ground up, with principled considerations of three main components: imaging, computer systems, and human perception.

The goal of this tutorial is to set up CS researchers for the exciting era of human-centered visual computing, where we can both 1) leverage human perception for enhancing computing and imaging systems, and 2) develope imaging and computational tools that advance our understanding of human perceptions and, ultimately, augment human perception.

To that end, this tutorial will 1) teach the basics of visual computing from first principles, including imaging (e.g., optics, image sensors) visual content, computing on visual content, displays, and human perception and cognition, and 2) discuss exciting research opportunities for the systems and architecture community.

The tutorial is not tied to a specific product or hardware platform. Instead, we focus on teaching the fundamental principles so that participants can walk away being able to design new algorithms, building new applications, and engineering systems and hardware for those algorithms and applications.

Tutorial (9:00 AM — 12:30 AM)

Introduction (9:00 AM — 9:05 AM)
Human Visual Systems (9:05 AM — 10:00 AM)
  • Basic visual neuroscience
  • Spatial/temporal/color vision
  • Light/dark/chromatic adaptation
  • Basics of AR/VR displays
  • Display power consumption and visual quality
Morning Coffee Break (10:00 AM — 10:20 AM)
Camera Imaging (10:20 AM — 11:05 AM)
  • Fundamental principles of imaging
  • Camera optics
  • Modern CMOS image sensor architecture
  • Camera image-signal processing and hardware
  • Basics of 3d reconstruction and novel view synthesis
  • Implicit neural representations, neural radiance fields, and gaussian splatting
  • Accelerating neural rendering
Sensor-Processor Architectures (11:50 AM — 12:30 AM)
  • Cost breakdown of the visual computing stack
  • Survey of contemporary sensor-processor architectures
  • New logic schemes and unconventional architectures for tighter sensor-processor integration

Lunch Break (12:30 PM — 14:00 PM)

Workshop (14:00 PM — 17:30 PM)

14:00 PM
DaCapo: An Acceleration Solution Enabling Adaptive AI on Autonmous Systems
Jongse Park (KAIST)
14:20 PM
Spatial Locality for Robot-World Collision Detection Acceleration
Deval Shah (UBC/AMD)
14:40 PM
Bridging Real-Time Robotics and Computer Architecture
Mohammad Bakhshalipour (CMU/Nvidia)
Afternoon Coffee Break (15:00 PM — 15:20 PM)
15:20 PM
Cicero: Addressing Algorithmic and Architectural Bottlenecks in Neural Rendering by Radiance Warping and Memory Optimizations
Feng Yu (SJTU)
15:40 PM
ML Workloads in AR/VR and their implication to the ML system design
Hyoukjun Kwon (UC Irvine)
16:00 PM
Efficient Neural Light Fields (ENeLF) for Mobile Devices
Austin Peng (Georgia Tech)
16:20 PM
3D Dataset Distillation: Condensing 3D Datasets for Enhanced Data Efficiency in 3D Reconstruction
TBD (Georgia Tech)
16:40 PM
The EarlyBird Gets the WORM: Heuristically Accelerating EarlyBird Convergence
Adithya Vasudev (Georgia Tech)
17:00 PM
Geode: A Zero-shot Geospatial Question-Answering Agent with Explicit Reasoning and Precise Spatio-Temporal Retrieval
TBD (Georgia Tech)

Reference Materials