A Framework to Enable Human-Drone Interaction Through Natural Hand Gesture Control

Design Considerations for Stability and Efficiency in Unmanned Aerial Vehicle
Explore the docs »

View Demo · Report Bug · Request Feature

Table of Contents

1. About The Project
   • Built With
2. Getting Started
   • Prerequisites
   • Installation
3. Usage
4. Contact
5. Acknowledgments

About The Project

This project introduces a novel framework to enhance human-drone interaction through natural hand gesture controls. Using a Leap Motion Controller, this system enables intuitive drone operation by translating hand gestures into flight commands. The integration of ROS2, PX4, and a companion computer allows for a seamless connection between the Ground Control Station (GCS) and the drone. The system is validated in a SITL Gazebo simulation environment to refine and test the approach before real-world deployment.

(back to top)

Built With

• 
• 
• 
• 
• 
• 
• 

(back to top)

Getting Started

To get a local copy up and running follow these simple example steps.

Prerequisites

• Software:
  • Ubuntu 22.04
  • ROS2 Humble
  • Python 3.10
  • PX4 Autopilot
  • Micro XRCE-DDS
  • px4_msgs
  • QGroundControl
  • Gemini Ultraleap Hand Tracking Software
  • Gemini LeapC API Python Bindings
• Hardware:
  • Ultraleap Leap Motion Controller Camera
  • PX4 Flight Controller (e.g. Holybro Pixhawk 6X) *
  • Power Module (e.g. Holybro PM03D) *
  • Drone Frame (e.g. F450) *
  • Motors + ESCs *
  • Companion Computer with Antenna *
  • RX + TX *

* Optional for simulating prototype locally

Installation

1. Ubuntu 22.04 System Update:

sudo apt update && apt full-upgrade -y
sudo reboot

2. Install ROS2 Humble:

Follow the steps here.

3. Install Python 3.10:

sudo apt install software-properties-common -y
sudo add-apt-repository ppa:deadsnakes/ppa
sudo apt update
sudo apt install python3.10 python3.10-venv python3.10-dev

4. Create and activate a Python virtual environment:

5. Install Gemini LeapC API:

cd ~
git clone https://github.com/ultraleap/leapc-python-bindings.git
cd leapc-python-bindings
pip install -r requirements.txt
python -m build leapc-cffi
pip install leapc-cffi/dist/leapc_cffi-0.0.1.tar.gz
pip install -e leapc-python-api

6. Install PX4 Autopilot:

git clone https://github.com/PX4/PX4-Autopilot.git --recursive -b release/1.14
bash ./PX4-Autopilot/Tools/setup/ubuntu.sh

Restart your computer before continuing.

7. Install PX4 Python Dependencies:

Install Python dependencies as mentioned in the PX4 Docs with this command:

pip3 install --user -U empy pyros-genmsg setuptools
pip3 install kconfiglib
pip install --user jsonschema
pip install --user jinja2

8. Build Micro XRCE-DDS:

git clone https://github.com/eProsima/Micro-XRCE-DDS-Agent.git
cd Micro-XRCE-DDS-Agent
mkdir build && cd build
cmake ..
make
sudo make install
sudo ldconfig /usr/local/lib/

9. Setup Workspace:

mkdir -p ~/hand_controlled_drone_ws/src
cd ~/hand_controlled_drone_ws/src

10. Clone in Packages:

git clone https://github.com/PX4/px4_msgs.git -b release/1.14

git clone https://github.com/yuriachermann/Hand-Controlled-PX4-Drone.git

11. Building the Workspace:

source /opt/ros/humble/setup.bash
cd ..
colcon build

source install/setup.bash

(back to top)

Usage

Launch the simulation with the following command:

ros2 launch px4_offboard hand_gesture_control.launch.py

This will open the Gazebo simulator, RVIZ, and start the necessary ROS2 nodes. Once the system is initialized, you can use hand gestures to control the drone's movement in the simulation.

The controls are as follows:

• Move Hand Up: Throttle Up
• Move Hand Down: Throttle Down
• Turn Hand CCW: Yaw Left
• Turn Hand CW: Yaw Right
• Turn Hand Forward: Pitch Forward
• Turn Hand Backward: Pitch Backward
• Turn Hand Left: Roll Left
• Turn Hand Left: Roll Right

(back to top)

Contact

Yuri Winche Achermann - [email protected]

(back to top)

Acknowledgments

• Jaeyoung Lim - px4-offboard
• Braden Wagstaff - ROS2_PX4_Offboard_Example
• Harry Mills - leapc-python-bindings

(back to top)