Prospects

We emphasized how the See stage is not merely about the passive acquisition of data, but about the active process of extracting meaningful information from the environment. Much of this discussion appeared in Part II, where we began by examining the fundamental properties and limitations of common sensing modalities, both proprioceptive and exteroceptive, and by establishing the physical and geometric principles that govern their measurements. Building on this foundation, we explored how raw sensor data can be transformed into structured and informative representations. We first surveyed classical approaches, including filtering techniques, feature detection, and methods for extracting geometric information about the environment. We then turned to modern, learning-based techniques, examining how deep neural network architectures enable end-to-end learning of rich representations directly from raw sensory inputs, including images as well as point-based and voxel-based representations of 3D geometry. Finally, we discussed higher-level visual understanding tasks such as object detection and segmentation, in which algorithms must interpret scenes by identifying object instances and reasoning about their spatial relationships.

Taken together, the See phase is not simply about sensing, but about perception, where the robot must transform raw sensory signals into the rich, semantic representations that are necessary for it to think and act intelligently.

Once the See stage has produced a structured and semantically meaningful representation of the world, the Think stage is where that representation is transformed into actionable intent. This is the stage in which the robot reasons about where it is with respect to its environment, its goals, and how its actions will shape future outcomes. In Part III, we discussed how this process typically begins with robot localization and mapping, where local sensor measurements are synthesized into a coherent global estimate of the robot’s state and a map of the environment. Building on this foundation, Part IV introduced techniques for high-level decision-making. Specifically, we discussed approaches to sequential decision-making, where the robot determines, loosely speaking, what to do, ultimately defining the objectives that guide subsequent stages of planning and control. To address this problem, we explored methods rooted in optimal control, such as dynamic programming, and extended this perspective to modern learning-based approaches, such as reinforcement learning and imitation learning. Across these frameworks, we emphasized different but complementary strategies for reasoning under uncertainty, balancing model-based formulations with data-driven approximations.

Viewed holistically, the Think phase emerges as a structured process that transforms information about the environment and abstract goals into concrete plans and intentions that can be executed by the robot.

The final stage of the cycle, Act, is where high-level decisions are translated into physical motion. While the Think stage determines what the robot should do, the Act stage determines how those decisions are realized by a dynamical system subject to physical constraints. In Part I, we developed the tools required to bridge this gap. We began with modeling the robot as a dynamical system, introducing the representations and notation needed to describe its motion and constraints. Building on this foundation, we discussed how desired behaviors are converted into feasible trajectories through motion planning and trajectory optimization. Motion planning algorithms produce geometrically feasible, collision-free paths, while trajectory optimization refines these paths into time-parameterized motions that respect dynamics, actuator limits, and task requirements. Execution, however, is never merely the blind following of precomputed motor commands. The real world is inherently uncertain, subject to external disturbances, sensor noise, actuator limitations, and inevitable inaccuracies in the robot’s internal dynamic models. For this reason, execution must be grounded in the principle of feedback control. Rather than assuming that a planned trajectory will unfold exactly as predicted, feedback control continuously compares the robot’s measured state against its desired state and corrects deviations in real time. Our discussion spanned a spectrum of closed-loop control strategies, from classical PID to optimization-based methods, such as Linear Quadratic Regulators (LQR) and Model Predictive Control (MPC), which explicitly reason about system dynamics, performance objectives, and constraints.

Viewed in the context of the full See-Think-Act cycle, the Act phase closes the loop between intention and reality. It is where plans meet physical embodiment, and where the robot’s ability to adapt and respond to the unpredictable nature of the real world is put to the test.

Over the past few years, artificial intelligence has undergone a dramatic shift driven by large-scale foundation models. Citing the seminal work of Bommasani et al. (2021)¹1. Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S. et al. “On the Opportunities and Risks of Foundation Models.” arXiv preprint arXiv:2108.07258, 2021., we define foundation models as:

“[Foundation models are] models that are trained on broad data (generally using self-supervision at scale) that can be adapted (e.g., fine-tuned) to a wide range of downstream tasks.”

Importantly, the term “foundation model” is not meant to refer to a specific architecture. Rather, it describes a class of models characterized by scale, broad pretraining, and adaptability. Well-known examples include large language models (LLMs) ²2. Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I. “Language Models are Unsupervised Multitask Learners.” 2019. ³3. Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., Amodei, D. “Language Models are Few-Shot Learners.” In Advances in Neural Information Processing Systems, 1877–1901, 2020., which underpin systems such as ChatGPT, as well as vision-language models (VLMs) ⁴4. Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., Sutskever, I. “Learning Transferable Visual Models From Natural Language Supervision.” In Proceedings of the 38th International Conference on Machine Learning, 8748–8763, 2021., video generation models ⁵5. Ho, J., Chan, W., Saharia, C., Whang, J., Gao, R., Gritsenko, A., Kingma, D. P., Poole, B., Norouzi, M., Fleet, D. J., Salimans, T. “Imagen Video: High Definition Video Generation with Diffusion Models.” arXiv preprint arXiv:2210.02303, 2022., and other large-scale architectures ⁶6. Rives, A., Meier, J., Sercu, T., Goyal, S., Lin, Z., Liu, J., Guo, D., Ott, M., Zitnick, C. L., Ma, J., Fergus, R. “Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences.” Proceedings of the National Academy of Sciences 118(15), 2021.. Beyond their impact on purely digital applications, foundation models are increasingly being integrated into physical robotic systems.

Their growing relevance in robotics stems from several key properties. First, foundation models provide broad priors about the world. Trained on large and diverse datasets, they encode a wealth of information about the structure of language, visual concepts, and physical interactions, enabling improved generalization in settings with limited task-specific data.

Second, foundation models are inherently multimodal. Many modern architectures jointly process text, images, video, audio, and structured data. Robotics, by its very nature, is a multimodal domain where robots must integrate information from diverse sensory modalities such as vision, proprioception, tactile sensing, and language-based instructions. Multimodal foundation models offer a unified representational framework in which these heterogeneous data streams can be fused and reasoned over coherently.

Third, foundation models provide a powerful and flexible interface for human-robot interaction. Language-conditioned policies enable users to specify goals in natural language rather than low-level commands. This dramatically lowers the barrier between human intent and robotic execution, enabling more intuitive forms of supervision, correction, and collaboration.

Within this broader paradigm, several emerging instantiations of foundation models are particularly influential in robotics:

A closely related development is the emergence of reasoning models within embodied systems ¹⁰10. Huang, W., Xia, F., Xiao, T., Chan, H., Liang, J., Florence, P., Zeng, A., Tompson, J., Mordatch, I., Chebotar, Y., Sermanet, P., Jackson, T., Brown, N., Luu, L., Levine, S., Hausman, K. “Inner Monologue: Embodied Reasoning through Planning with Language Models.” In Proceedings of The 6th Conference on Robot Learning, 1769–1782, 2023. ¹¹11. NVIDIA. “Alpamayo-R1: Bridging Reasoning and Action Prediction for Generalizable Autonomous Driving in the Long Tail.” arXiv preprint arXiv:2511.00088, 2025. ¹²12. Zawalski, M., Chen, W., Pertsch, K., Mees, O., Finn, C., Levine, S. “Robotic Control via Embodied Chain-of-Thought Reasoning.” In Proceedings of The 8th Conference on Robot Learning, 2024.. Beyond mapping observations directly to actions, these architectures allocate explicit computation to intermediate deliberation, generating structured reasoning traces—often in natural language—that attempt to make causal relationships, counterfactuals, and task constraints explicit before committing to specific actions. Inspired by advances in large language models where inference-time reasoning improves robustness and accuracy, reasoning-enabled robotic systems treat deliberation itself as a tunable resource, where more complex or safety-critical situations can trigger deeper chains of thought, self-reflection, or verification steps. In embodied contexts, however, reasoning cannot remain purely textual. It must be grounded in geometry, dynamics, and physical feasibility, and must remain consistent with the actions ultimately executed. The central challenge, therefore, is not merely to produce articulate explanations, but to ensure alignment between internal rationale and external behavior, so that reasoning becomes a functional component of autonomy rather than a post-hoc narrative.

This richer predictive interface is particularly valuable in robotics, where data is a fundamental bottleneck. Robot interaction data is expensive to collect, embodiment-specific, and limited in scale. In contrast, visual data—especially video—is abundant and diverse. World models provide a pathway for leveraging this disparity by learning general physical and semantic structure from large-scale, largely passive observations. In doing so, they enable a partial decoupling between world understanding and control, allowing robots to acquire broad knowledge about how environments behave without requiring equivalent amounts of task-specific interaction data.

At a high level, world models equip robots with the ability to predict and imagine. Given a current situation and, potentially, a proposed action sequence, a world model can forecast how the environment and the robot within it may evolve. This enables planning through the evaluation of imagined futures, where a robot may compare alternative outcomes, select actions that lead toward desirable states, or optimize behavior by reasoning directly over predicted trajectories.

A second major advantage is that world models can support the generation of realistic synthetic data. Because they model visual, spatial, and physical regularities, world models can be used to synthesize new scenarios, including rare edge cases or safety-critical situations that are difficult to collect in practice. In this way, world models can augment traditional simulation, providing a data-driven substrate for training and evaluating autonomous systems in a broader range of conditions, while maintaining high visual fidelity.

The precise form that such world models should take remains an open research question. Some approaches focus on predicting high-dimensional sensory observations such as future video frames conditioned on actions ¹⁵15. Google DeepMind. Genie 3: A New Frontier for World Models. Research blog, 2025.. Others learn compact latent representations of the environment and predict future states within this lower-dimensional space rather than at the raw pixel level ¹⁶16. Assran, M., Bardes, A., Fan, D., Garrido, Q., Howes, R., Komeili, M., Muckley, M., Rizvi, A., Roberts, C., Sinha, K. et al. “V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning.” arXiv preprint arXiv:2506.09985, 2025.. Additional work explores structured representations that encode objects, physical interactions, or causal relationships in an effort to improve generalization and data efficiency ¹⁷17. Locatello, F., Weissenborn, D., Unterthiner, T., Mahendran, A., Neumann, G., Rätsch, M., Kohli, P., Botvinick, M., Bachem, O. “Object-Centric Learning with Slot Attention.” In Advances in Neural Information Processing Systems 33, 11525–11538, 2020..

In this sense, foundation models are not merely new building blocks within the autonomy stack, but are beginning to redefine the abstractions and workflows used to construct it.

In parallel, progress in simulation has accelerated dramatically, expanding both its fidelity and its role within the autonomy development pipeline. Advances in physics-based engines, high-fidelity rendering, differentiable simulators, and large-scale synthetic data generation now enable training and evaluation in increasingly realistic environments ¹⁸18. Makoviychuk, V., Wawrzyniak, L., Guo, Y., Lu, M., Storey, K., Macklin, M., Hoeller, D., Rudin, N., Allshire, A., Handa, A. et al. “Isaac Gym: High Performance GPU Based Physics Simulation For Robot Learning.” In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, 2021.. At the same time, the emergence of generative simulation has begun to complement traditional simulators by producing realistic, diverse, and often open-ended scenarios directly from data.

Simulation has long been an essential tool in robotics, but it is increasingly becoming a central substrate for large-scale training, evaluation, and red-teaming of autonomous systems.

As autonomy stacks incorporate more data-driven components, the question of safety becomes increasingly critical. Data-driven models can hallucinate, extrapolate poorly outside their training distribution, or produce internally inconsistent reasoning traces. In physical systems, such failures carry real-world consequences.

Physical AI safety is therefore becoming a central topic. This includes research on detecting and mitigating failure modes of data-driven models, developing robust training and evaluation protocols, and combining data-driven learning with traditional control-theoretic safety guarantees ¹⁹19. Sinha, R., Elhafsi, A., Agia, C., Foutter, M., Schmerling, E., Pavone, M. “Real-Time Anomaly Detection and Reactive Planning with Large Language Models.” In Proceedings of Robotics: Science and Systems, 2024. ²⁰20. Sagawa, S., Koh, P. W., Hashimoto, T. B., Liang, P. “Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization.” In Proceedings of the International Conference on Learning Representations, 2020. ²¹21. Hoffman, J., Tzeng, E., Park, T., Zhu, J.-Y., Isola, P., Saenko, K., Efros, A. A., Darrell, T. “CyCADA: Cycle-Consistent Adversarial Domain Adaptation.” In Proceedings of the 35th International Conference on Machine Learning, 1994–2003, 2018. ²²22. Taylor, A., Singletary, A., Yue, Y., Ames, A. “Learning for Safety-Critical Control with Control Barrier Functions.” In Learning for Dynamics and Control, 708–717, 2020.. At the systems level, emerging frameworks such as NVIDIA’s Halos platform aim to provide end-to-end safety validation pipelines for autonomous systems, integrating simulation, scenario generation, and formal evaluation to stress-test models under diverse and safety-critical conditions ²³23. NVIDIA. NVIDIA Halos: Autonomous Vehicle Safety. 2024.. Such efforts highlight the growing recognition that safety must be addressed holistically, spanning model design, data curation, evaluation, and deployment.