PhD Position F/M Prediction of human whole-body motion for anticipation in human-robot interaction using multimodal, contextual models and generative AI

The position is funded by the PEPR O2R, a national French program to advance research in robotics which reunites several French laboratories in robotics, AI, and Social and Human Sciences.

In this context, the LARSEN team in INRIA is collaborating with CEA, to advance the anticipation skills of robots interacting with humans. One goal is to estimate the human intention of motion using the sensors onboard the robot (cameras, IMUs, microphones, etc), considering the contextual information extracted from vision or audio processing.

The successful candidate will join our dynamic research team and contribute to cutting-edge advancements in robotics and will be jointly supervised by INRIA (Serena Ivaldi and Jean-Baptiste Mouret) and CEA (Bertrand Luvison).

The PhD thesis will be physically based in Nancy, but the student will spend a short visiting period in CEA too.

The PhD candidate will participate in the exchange activities of PEPR, including hackathons, national conferences, and summer schools.

The student will also have the opportunity to perform the research using laboratory robotic assets including the omniTiago++ robot, the humanoid bipedal robot TALOS.

The objective of this Ph.D. is to propose new algorithms to accurately predict the future whole-body motion of a human, which is critical for human-robot collaboration. This has a direct application in social robots co-existing or interacting with humans in their environment, in exoskeletons, and in the direct teleoperation of robots with different degrees of anthropomorphism.

The thesis will target several advances in human motion prediction, such as predicting the intention of motion at different time scales, with different families of models, adding physical priors to guarantee the prediction follows the model of the human and physics constraints (e.g., the human is balanced, with the feet on the floor), and incorporating external sources of information to improve the prediction while taking the environment into account (objects nearby, payloads, obstacles from RGB-D sensors, etc.).

The level of physical interaction in this thesis will be minimal, as the focus will be particularly on situations when the onset of action must be detected to ensure a prompt anticipatory action from the robot. As an example, handovers or collaborative lifting/carrying will be studied.

The software and methods produced by Ph.D. will be a basic “building block" for many interaction scenarios addressed in the PEPR project: developing open software that can be re-used in other projects will be of paramount importance.

• Write code to perform experiments with robots
• Write software documentation
• Write an annual activity report
• Conduct experiments in the real environment, with real robots: Tiago++
• Write scientific articles and reports
• Participate to national and European projects events (meetings, hackathons) when appropriate

• Experience with machine learning, vision, or real robots is necessary.
• Experience with LLMs, visual models, generative AI, diffusion models is a plus.
• Familiarity with robotic platforms and simulation environments (e.g., ROS2, Gazebo, MuJoCo, CasADi, Eigen, Pinocchio, Nvidia Isaac GYM) is a plus.
• Proficiency in programming languages such as C/C++ and/or Python is required.
• Ability to work independently and as part of a team.
• Good communication and writing skills in English.

• Subsidized meals
• Partial reimbursement of public transport costs
• Leave: 7 weeks of annual leave + 10 extra days off due to RTT (statutory reduction in working hours) + possibility of exceptional leave (sick children, moving home, etc.)
• Possibility of teleworking (after 6 months of employment) and flexible organization of working hours
• Professional equipment available (videoconferencing, loan of computer equipment, etc.)
• Social, cultural and sports events and activities
• Access to vocational training
• Social security coverage

2100€ gross/month the 1st year