This project aims to build a functional human-in-the-loop tracking system that allows a filmmaker to control an industrial robot instinctively. By capturing physical movement with a motion-capture camera system, the project will transform a high-tech laboratory into a 'digital atelier', enabling the robot to serve as a direct extension of the performer for audience-facing and virtual production applications.
This project is directly aligned with one of the main topics of the EPSRC UDLA research internship call, 'AI, Digitalisation, and Data', as it transforms human physical movement by AI digitalisation into a structured, high-value digital asset that a machine/robot can interpret and execute. It drives innovation in creative and filmmaking industries; it can also have wider applications in healthcare sectors, such as remote surgery.
Project objectives:
1. Seamless Motion Translation (The "Mirror" Effect)
- To create a "live-link" where the robot mimics human arm movements with no perceptible lag.
- Develop a software bridge that streams coordinates from the motion capture area directly to the robot's end-effector, ensuring that when the artist 'moves' in the air, the robot mirrors the stroke on a canvas in real-time.
2. Gesture-to-Stroke Precision
- To capture and replicate the subtlety of artistic and performance movements.
- Use the high-fidelity camera data to detect changes in velocity and acceleration. The objective is to ensure the robot doesn't just move from point A to B, but replicates the speed of a flicked wrist or the slowness of a careful detail.
3. Adaptive Workspace Mapping
- To allow a performer to perform large body-scale movements.
- Create a "scaling" feature where a large body movement by the human can be scaled down for intricate miniature work – for example, for scale virtual production, animation, character development, game design, or stop-motion and rostrum camera applications.
4. Safety-First Collaborative Environment
- To ensure the performer and robot can work in the same room safely.
- Program 'Virtual Boundary Walls' If the artist moves their arm too far or toward a prohibited area (like the camera stands), the system will automatically constrain the robot's movement to prevent physical collisions or equipment damage.
5. Performance Validation via "Physical Output"
- To prove the system works by producing an audience-facing short performance.