I am a Machine Learning PhD student at Georgia Tech advised by Zsolt Kira and Dhruv Batra. I am interested in developing generalist agents that complete complex tasks in the physical and digital world. My research investigates how to train generalist multimodal LLM agents with reinforcement learning and supervised training in addition to using simulation to generate large-scale embodied experience for training.
I am interning at Apple working with Alexander Toshev since Spring 2023. I interned at Meta AI from Summer to Winter 2022 and again in Summer to Winter 2021 working with Akshara Rai, Ruta Desai, and Franziska Meier. I completed my undergrad in Computer Science at the University of Southern California with a minor in Mathematics, where I worked with Joseph Lim. My email is "andrewszot 1 @ gmail com" (no spaces).
Selected Papers
From Multimodal LLMs to Generalist Embodied Agents: Methods and Lessons
Andrew Szot, Bogdan Mazoure, Omar Attia, Aleksei Timofeev, Harsh Agrawal, Devon Hjelm, Zhe Gan, Zsolt Kira, Alexander Toshev
Andrew Szot, Bogdan Mazoure, Omar Attia, Aleksei Timofeev, Harsh Agrawal, Devon Hjelm, Zhe Gan, Zsolt Kira, Alexander Toshev
Preprint, 2024
[paper]
[code/model coming soon!]
Grounding Multimodal Large Language Models in Actions
Andrew Szot, Bogdan Mazoure, Harsh Agrawal, Devon Hjelm, Zsolt Kira, Alexander Toshev
Andrew Szot, Bogdan Mazoure, Harsh Agrawal, Devon Hjelm, Zsolt Kira, Alexander Toshev
NeurIPS 2024
[paper]
Large Language Models as Generalizable Policies for Embodied Tasks
Andrew Szot, Max Schwarzer, Harsh Agrawal, Bogdan Mazoure, Walter Talbott Katherine Metcalf, Natalie Mackraz, Devon Hjelm, Alexander Toshev
Andrew Szot, Max Schwarzer, Harsh Agrawal, Bogdan Mazoure, Walter Talbott Katherine Metcalf, Natalie Mackraz, Devon Hjelm, Alexander Toshev
Habitat 2.0: Training Home Assistants to Rearrange their Habitat
Andrew Szot, Alex Clegg, Eric Undersander, Erik Wijmans, Yili Zhao, John Turner, Noah Maestre, Mustafa Mukadam, Devendra Chaplot, Oleksandr Maksymets, Aaron Gokaslan, Vladimir Vondrus, Sameer Dharur, Franziska Meier, Wojciech Galuba, Angel Chang, Zsolt Kira, Vladlen Koltun, Jitendra Malik, Manolis Savva, Dhruv Batra
Andrew Szot, Alex Clegg, Eric Undersander, Erik Wijmans, Yili Zhao, John Turner, Noah Maestre, Mustafa Mukadam, Devendra Chaplot, Oleksandr Maksymets, Aaron Gokaslan, Vladimir Vondrus, Sameer Dharur, Franziska Meier, Wojciech Galuba, Angel Chang, Zsolt Kira, Vladlen Koltun, Jitendra Malik, Manolis Savva, Dhruv Batra
All Research
From Multimodal LLMs to Generalist Embodied Agents: Methods and Lessons
Andrew Szot, Bogdan Mazoure, Omar Attia, Aleksei Timofeev, Harsh Agrawal, Devon Hjelm, Zhe Gan, Zsolt Kira, Alexander Toshev
Andrew Szot, Bogdan Mazoure, Omar Attia, Aleksei Timofeev, Harsh Agrawal, Devon Hjelm, Zhe Gan, Zsolt Kira, Alexander Toshev
Preprint, 2024
[paper]
[code/model coming soon!]
ReLIC: A Recipe for 64k Steps of In-Context Reinforcement Learning for Embodied AI
Ahmad Elawady, Gunjan Chhablani, Ram Ramrakhya, Karmesh Yadav, Dhruv Batra, Zsolt Kira, Andrew Szot
Ahmad Elawady, Gunjan Chhablani, Ram Ramrakhya, Karmesh Yadav, Dhruv Batra, Zsolt Kira, Andrew Szot
Reinforcement Learning via Auxiliary Task Distillation
Abhinav Narayan Harish, Larry Heck, Josiah P. Hanna, Zsolt Kira, Andrew Szot
Abhinav Narayan Harish, Larry Heck, Josiah P. Hanna, Zsolt Kira, Andrew Szot
Grounding Multimodal Large Language Models in Actions
Andrew Szot, Bogdan Mazoure, Harsh Agrawal, Devon Hjelm, Zsolt Kira, Alexander Toshev
Andrew Szot, Bogdan Mazoure, Harsh Agrawal, Devon Hjelm, Zsolt Kira, Alexander Toshev
NeurIPS 2024
[paper]
Large Language Models as Generalizable Policies for Embodied Tasks
Andrew Szot, Max Schwarzer, Harsh Agrawal, Bogdan Mazoure, Walter Talbott Katherine Metcalf, Natalie Mackraz, Devon Hjelm, Alexander Toshev
Andrew Szot, Max Schwarzer, Harsh Agrawal, Bogdan Mazoure, Walter Talbott Katherine Metcalf, Natalie Mackraz, Devon Hjelm, Alexander Toshev
Habitat 3.0: A Co-Habitat for Humans, Avatars and Robots
Xavier Puig*, Eric Undersander*, Andrew Szot*, Mikael Dallaire Cote*, Tsung-Yen Yang*, Ruslan Partsey*, Ruta Desai*, Alexander William Clegg, Michal Hlavac, So Yeon Min, Vladimír Vondruš, Theophile Gervet, Vincent-Pierre Berges, John M. Turner, Oleksandr Maksymets, Zsolt Kira, Mrinal Kalakrishnan, Jitendra Malik, Devendra Singh Chaplot, Unnat Jain, Dhruv Batra, Akshara Rai, Roozbeh Mottaghi (* - equal contribution)
Xavier Puig*, Eric Undersander*, Andrew Szot*, Mikael Dallaire Cote*, Tsung-Yen Yang*, Ruslan Partsey*, Ruta Desai*, Alexander William Clegg, Michal Hlavac, So Yeon Min, Vladimír Vondruš, Theophile Gervet, Vincent-Pierre Berges, John M. Turner, Oleksandr Maksymets, Zsolt Kira, Mrinal Kalakrishnan, Jitendra Malik, Devendra Singh Chaplot, Unnat Jain, Dhruv Batra, Akshara Rai, Roozbeh Mottaghi (* - equal contribution)
Skill Transformer: A Monolithic Policy for Mobile Manipulation
Haytham Huang, Dhruv Batra, Akshara Rai, Andrew Szot
Haytham Huang, Dhruv Batra, Akshara Rai, Andrew Szot
An Extensible, Data-Oriented Architecture for High-Performance, Many-World Simulation
Brennan Shacklett, Luc Guy Rosenzweig, Zhiqiang Xie, Bidipta Sarkar, Andrew Szot, Erik Wijmans, Vladlen Koltun, Dhruv Batra, Kayvon Fatahalian
Brennan Shacklett, Luc Guy Rosenzweig, Zhiqiang Xie, Bidipta Sarkar, Andrew Szot, Erik Wijmans, Vladlen Koltun, Dhruv Batra, Kayvon Fatahalian
Adaptive Coordination in Social Embodied Rearrangement
Andrew Szot, Unnat Jain, Dhruv Batra, Zsolt Kira, Ruta Desai, Akshara Rai
Andrew Szot, Unnat Jain, Dhruv Batra, Zsolt Kira, Ruta Desai, Akshara Rai
Galactic: Scaling End-to-End Reinforcement Learning for Rearrangement at 100k Steps-Per-Second
Vincent-Pierre Berges*, Andrew Szot*, Devendra Singh Chaplot, Aaron Gokaslan, Roozbeh Mottaghi, Dhruv Batra, Eric Undersander (* - equal contribution)
Vincent-Pierre Berges*, Andrew Szot*, Devendra Singh Chaplot, Aaron Gokaslan, Roozbeh Mottaghi, Dhruv Batra, Eric Undersander (* - equal contribution)
BC-IRL: Learning Generalizable Reward Functions from Demonstrations
Andrew Szot, Amy Zhang, Dhruv Batra, Zsolt Kira, Franziska Meier
Andrew Szot, Amy Zhang, Dhruv Batra, Zsolt Kira, Franziska Meier
Retrospectives on the Embodied AI Workshop
Matt Deitke, Dhruv Batra, Yonatan Bisk, Tommaso Campari, Angel X. Chang, Devendra Singh Chaplot, Changan Chen, Claudia Pérez D'Arpino, Kiana Ehsani, Ali Farhadi, Li Fei-Fei, Anthony Francis, Chuang Gan, Kristen Grauman, David Hall, Winson Han, Unnat Jain, Aniruddha Kembhavi, Jacob Krantz, Stefan Lee, Chengshu Li, Sagnik Majumder, Oleksandr Maksymets, Roberto Martín-Martín, Roozbeh Mottaghi, Sonia Raychaudhuri, Mike Roberts, Silvio Savarese, Manolis Savva, Mohit Shridhar, Niko Sünderhauf, Andrew Szot, Ben Talbot, Joshua B. Tenenbaum, Jesse Thomason, Alexander Toshev, Joanne Truong, Luca Weihs, Jiajun Wu
Matt Deitke, Dhruv Batra, Yonatan Bisk, Tommaso Campari, Angel X. Chang, Devendra Singh Chaplot, Changan Chen, Claudia Pérez D'Arpino, Kiana Ehsani, Ali Farhadi, Li Fei-Fei, Anthony Francis, Chuang Gan, Kristen Grauman, David Hall, Winson Han, Unnat Jain, Aniruddha Kembhavi, Jacob Krantz, Stefan Lee, Chengshu Li, Sagnik Majumder, Oleksandr Maksymets, Roberto Martín-Martín, Roozbeh Mottaghi, Sonia Raychaudhuri, Mike Roberts, Silvio Savarese, Manolis Savva, Mohit Shridhar, Niko Sünderhauf, Andrew Szot, Ben Talbot, Joshua B. Tenenbaum, Jesse Thomason, Alexander Toshev, Joanne Truong, Luca Weihs, Jiajun Wu
Preprint 2022
[paper]
Housekeep: Tidying Virtual Households using Commonsense Reasoning
Yash Kant, Arun Ramachandran, Sriram Yenamandra, Igor Gilitschenski, Dhruv Batra, Andrew Szot*, Harsh Agrawal* (* - equal advising)
Yash Kant, Arun Ramachandran, Sriram Yenamandra, Igor Gilitschenski, Dhruv Batra, Andrew Szot*, Harsh Agrawal* (* - equal advising)
Habitat 2.0: Training Home Assistants to Rearrange their Habitat
Andrew Szot, Alex Clegg, Eric Undersander, Erik Wijmans, Yili Zhao, John Turner, Noah Maestre, Mustafa Mukadam, Devendra Chaplot, Oleksandr Maksymets, Aaron Gokaslan, Vladimir Vondrus, Sameer Dharur, Franziska Meier, Wojciech Galuba, Angel Chang, Zsolt Kira, Vladlen Koltun, Jitendra Malik, Manolis Savva, Dhruv Batra
Andrew Szot, Alex Clegg, Eric Undersander, Erik Wijmans, Yili Zhao, John Turner, Noah Maestre, Mustafa Mukadam, Devendra Chaplot, Oleksandr Maksymets, Aaron Gokaslan, Vladimir Vondrus, Sameer Dharur, Franziska Meier, Wojciech Galuba, Angel Chang, Zsolt Kira, Vladlen Koltun, Jitendra Malik, Manolis Savva, Dhruv Batra
Goal-Driven Imitation Learning from Observation by Inferring
Goal Proximity
Youngwoon Lee*, Andrew Szot*, Shao-hua Sun and Joseph J. Lim (* - equal contribution)
Youngwoon Lee*, Andrew Szot*, Shao-hua Sun and Joseph J. Lim (* - equal contribution)
Experience
-
Apple Machine Learning Research:
With Alexander Toshev.
Spring 2023 - Present
-
Meta AI:
With Akshara Rai and Ruta Desai.
Zero-shot coordination in multi-agent RL (ICML 2023).
Summer 2022 - Winter 2022
-
Facebook AI Research:
With Franziska Meier.
Developed a new inverse reinforcement learning method for learning generalizable reward functions (ICLR 2023).
Summer 2021 - Winter 2021
-
USC CLVR Lab:
With Professor Joseph Lim.
Researched imitation from observation (NeurIPS 2021), generalization to new actions in reinforcement learning (ICML 2020), and benchmarks for tool usage and physical reasoning in RL.
Fall 2018 - Summer 2020 (Summer 2019 included)
-
USC MCL:
With Professor C.-C. Jay Kuo.
Researched weakly supervised CNNs using clustering approaches to pre-train.
Fall 2016 - Winter 2017
-
Airbnb:
Deployed a model to production that predicted and took automated action against account takeover fraud using Scikit-learn, XGBoost, Hive and Presto, resulting in a 50% reduction in account takeover fraud.
Summer 2018
-
eBay:
Delivered a data analytics platform using Scala, Spark, Hadoop and Druid to process petabytes of eBay metrics.
Summer 2017
Awards
- NVIDIA Graduate Fellowship Finalist (2023)
- Georgia Tech Institute-Wide Outstanding Online Teaching Assistant of the Year (2021)
- NSF GRFP Honorable Mention. (2021)
- USC Viterbi Fellowship: Research funding given to the top 20 engineering students per class. (2016-2020)
- USC Trustee Scholarship: Full tuition scholarship. (2016-2020)
- USC Provost Fellowship: Merit based fellowship for summer research funding. (2019)
Service and Teaching
- Reviewer: NeurIPS [2022,2023,2024], ICLR [2022,2024,2025], ICML [2023,2024], CVPR 2022, ICCV 2021, IROS 2021
- Lead organizer of Habitat Rearrangement Challenge at NeurIPS 2022.
- Organizer of Embodied AI Workshop at CVPR 2022.
- Graduate Deep Learning Class Teaching Assistant (CS 7643, won outstanding TA award). See the homework assignment I created for the class. (Fall 2021)