Building a context-aware robotic OR assistant
How ImFusion helped turn an ambitious concept for robotic surgical assistance into an integrated, preclinical demonstrator
Project overview
SASHA-OR (Situation Aware Sterile Handling Arm for the OR) developed an intelligent robotic assistance system that operates in the sterile field at the operating table. The project combined computer vision, collaborative robotics, and voice interaction to autonomously manage, deliver, and retrieve surgical instruments during laparoscopic procedures.
- Project type: academic-industry consortium
- Clinical field: minimally invasive surgery / operating room automation
- Consortium partners: Munich Institute of Innovative Technology in Medicine (MITI) at TUM, Baxter International Inc.
- Expertise applied: 3D computer vision / AI / robotics / motion planning / system integration
- Project duration: 42 months
- 48 ImFusion person months invested
- 7 laparoscopic instruments digitised for the demonstrator
- 1 integrated system evaluated with surgeons
A robot in the operating room must do more than recognise an instrument
It must understand which object is needed, where it is, how to grasp and orient it, when to act. And how to move safely around people.
The challenge
The consortium set out to develop a robotic assistant that could handle surgical instruments at the sterile operating table. To work safely alongside clinical staff, the system needed to recognise each instrument, grasp it correctly, and respond to changes in the operating room.
- Detect and classify similar-looking instruments using 3D cameras
- Estimate each instrument’s exact position and orientation for grasping
- Plan collision-free robot movements in a changing environment
- Coordinate sensors, robot control, voice recognition, and operating-room systems
Our approach
SASHA-OR went beyond recognising surgical instruments or controlling a robot as separate tasks. In the project, we connected 3D perception, AI, and robotic control in one integrated demonstrator that could respond to the situation at the sterile operating table.
- Trained AI models with synthetic data
- Combined AI-based segmentation with 3D pose estimation
- Used a robot-mounted camera for better viewpoints
- Integrated voice control, motion planning, and robotics
The impact
Operating rooms depend on highly qualified staff, while many hospitals face persistent shortages. SASHA-OR explored how robotic assistance could reduce pressure on surgical teams, support available operating-room capacity, and allow specialists to focus on tasks requiring clinical judgement.
- Demonstrated robotic instrument delivery and retrieval
- Tested the complete system with surgeons
- Reduced technical risk through early integration
- Created reusable components for future robotics projects
Our process
Model the domain
We began with a structured model of the clinical domain. Surgical instruments were catalogued using an ontology that combined 3D geometry with semantic information such as function, orientation and use during an operation. This created a shared foundation for perception, synthetic data generation and robotic grasp planning.
Prototyping and feasiblity study
We digitised seven laparoscopic instruments for a cholecystectomy scenario and used these models to develop and test the perception pipeline. Synthetic training data was generated from the instrument models. We also conducted extensive feasibility studies to select suitable 3D sensors.
Develop computer vision methods
We developed computer-vision methods for instrument detection, classification and 6D pose estimation, including synthetic-data pipelines and mesh-to-point-cloud registration.
Integrate, test and simplify
We led the integration of the Zivid vision sensors, the Franka robot and ROS communication services. We integrated a digital twin with ROS MoveIt for collision detection and motion planning, connected the voice-recognition component developed by project partners, and supported the context-aware orchestration of the overall system.
Prof. Dirk Wilhelm
Chair, Research Group Minimally invasive Interdisciplinary Therapeutical Interventions (MITI)
Working with ImFusion was straightforward and highly collaborative. Their unique combination of expertise in 3D vision, AI, medical robotics, and system integration was essential to bringing the different parts of SASHA-OR together in one working demonstrator.
When an application spans imaging, AI, robotics and a domain specific workflow, the value lies in making the entire chain work as one testable system
ImFusion technology
ImFusion contributed the 3D vision and robotics backbone of the demonstrator. We developed methods for instrument recognition and pose estimation, integrated cameras and robot control, and connected perception and motion planning through a shared software architecture. The ImFusion SDK and Suite ran on the project workstation and orchestrated context-aware functions.
ImFusion expertise
ImFusion's role went beyond delivering an isolated vision algorithm. We connected domain models, sensors, AI, planning and robot interfaces into an architecture that could be tested with clinical experts. To achieve that we brought together expertise in 3D computer vision, AI, synthetic data, sensor integration, medical robotics, motion planning, and the integration of complex software systems.
We offer two types of introductory calls so we can make sure we’re speaking to the right audience.
Product Demo
In a product demo, we show you how the ImFusion Suite, SDK, and ImFusion Labels support medical imaging and data annotation workflows. We focus on the capabilities relevant to your application and discuss directly how our technology could fit into your product or support your use case.
- See integration points mapped to your specific tech stack
- Identify bottlenecks before committing development resources
- Validate data format compatibility and transformation requirements
- Get immediate answers from engineers who built the system
Discovery Call
In a discovery call, we learn about your clinical or technical challenge and discuss how ImFusion’s expertise and technology could help make your idea a reality. Together, we identify promising first steps and determine where we can add the most value.
- Identify technical risks and regulatory considerations early
- Align engineering, clinical, and business stakeholders quickly
- Get a realistic timeline and resource estimate
- Leave with actionable next steps, not vague promises