Alexander (Sasha) Sax

I'm a senior research scientist at Meta FAIR in Menlo Park. My research focuses on building multimodal foundation models to enable Embodied Agents that can perceive, act in, and communicate about the physical world.

I did my PhD at UC Berkeley, where I was advised by Jitendra Malik and Amir Zamir (EPFL). In the summer of 2022, I interned at FAIR with Georgia Gkioxari. I got my MS and BS from Stanford University, where I was advised by Silvio Savarese and graduated with an Erdős number of 3.

Email  /  CV  /  Scholar  /  Github

Research

A lot of my work is around large-scale training for predictive and generative models using supervised learning and RL. It often involves developing multimodal 3D datasets, simulators, and techniques for efficient pretraining and finetuning. Here are some nice honors:

• CVPR Best Paper Award Nomination (X-Task Consistency, 2020)
• CVPR Best Paper Award (Taskonomy, 2018)
• NVIDIA Pioneering Research Award (Gibson Environment, 2018)
• Stanford University Distinction in Research (Computational Evidence for Structure in the Space of Tasks, 2018)
• Winner CVPR Habitat Embodied Agents Challenge (Mid-Level Representations, RGB track, 2019)
• Outstanding Reviewer (ICLR '24, CVPR '22)

A few papers are highlighted. Equal contribution is indicated by * and listed in alphabetical order.

An open-vocabulary benchmark for Embodied Question Answering (EQA) with a new dataset across 180+ real environments. GPT-4V at 48.5% lags behind human performance (85.9%). For questions that require complex spatial understanding, VLMs in December 2024 are nearly "blind".

Annotating a large-scale dataset of 3D scanned and artist-generated scenes, using Blender. Transformer models trained for monocular surface normal estimation achieved performance comparable to human annotators on the OASIS benchmark, and depth models outperformed MidasV2.

A method for creating a robust and diverse ensemble of predictive networks where the average pairwise disagreement provides a measure of uncertainty. The calibration can be supervised with a lightweight post-training step.

Introducing perceptual consistency losses for large-scale distillation; using learned priors from pretrained networks. We find that monocular priors from depth and surface normal estimation yield sharper details and better generalization than LPIPS.

Adding a lightweight side network to modulate a large pretrained network is surprisingly competitive for network adaptation and lifelong learning. We show results in vision, language, and robotics using supervised learning and behavior cloning.

In data regimes where visual domain randomization gets near-zero training performance, policies using pretrained image representations can saturate the benchmark. Pretrained representations yield qualitatively better generalization to new environments, including sim-to-real transfer.

Training end-to-end policies with pretrained image representations requires 10x less data to achieve a given performance vs. RL from raw pixels.

A fully computational approach for modeling the structure across 26 fundamental vision tasks. Exploiting this structure with a transfer learning curriculum reduces overall supervision by ~2/3.

A rendering and physics simulator to bridge the gap between large-scale simulation and real-world environments. The simulator features a database of thousands of real spaces and uses a neural network for infilling, like an early form of DLSS.

A large-scale indoor dataset providing mutually registered 2D, 2.5D and 3D modalities with instance-level semantic annotations. Covers over 6,000m² across 6 large-scale indoor areas with 70,000+ RGB images and corresponding depths, surface normals, semantic annotations, and global XYZ coordinates.

Misc.

Thanks Jon Barron for the clean website.
Last updated December 2024.