Human Machine Interface or HMI is a system comprising of hardware and software that helps in communication and exchange of information between the user (human operator) and the machine. We normally use LED Indicators, Switches, Touch Screens and LCD Displays as a part of HMI devices. Another way to communicate with machines like Robots or Computers is with the help of Hand Gestures. Instead of using a keyboard, mouse or joystick, we can use our hand gestures to control certain functions of a computer like play/pause a video, move left/right in a photo slide show, scroll up/down in a web page and many more. In this project, we have implemented a simple Arduino based hand gesture control where you can control few functions of your web browser like switching between tabs, scrolling up and down in web pages, shift between tasks (applications), play or pause a video and increase or decrease the volume (in VLC Player) with the help of hand gestures.
Before Proceeding…
The project Arduino based Hand Gesture Control of Computer is implemented using Python. So, before proceeding with this project, I suggest you to work on this simple project CONTROLLING ARDUINO’S ON-BOARD LED WITH PYTHON. In this project, you can find the basics of how to use the Arduino with Python, installing Python on your computer, setting up the Serial Library (important for communicating with Arduino) and the project codes. So, I assume that you have already installed Python and pySerial (library for communicating with serial ports) and also worked on the basic project of blinking Arduino’s LED with Python.
Principle behind the Project
The principle behind the Arduino based Hand Gesture Control of Computer is actually very simple. All you have to do is use two Ultrasonic Sensors with Arduino, place your hand in front of the Ultrasonic Sensor and calculate the distance between the hand and the sensor. Using this information, relevant actions in the computer can be performed. The position of the Ultrasonic Sensors is very important. Place the two Ultrasonic Sensors on the top of a laptop screen at either end. The distance information from Arduino is collected by a Python Program and a special library called PyAutoGUI will convert the data into keyboard click actions.
Circuit Diagram
The circuit diagram of Arduino part of the project is shown in the following image. It consists of an Arduino UNO board and two Ultrasonic Sensors and you can power up all these components from the laptop’s USB Port.
Components Required
Arduino UNO x 1 [Buy Here] Ultrasonic Sensors x 2 USB Cable (for Arduino) Few Connecting Wires A Laptop with internet connection
Design of the Project
The design of the circuit is very simple, but the setup of the components is very important. The Trigger and Echo Pins of the first Ultrasonic Sensor (that is placed on the left of the screen) are connected to Pins 11 and 10 of the Arduino. For the second Ultrasonic Sensor, the Trigger and Echo Pins are connected to Pins 6 and 5 of the Arduino. Now, coming to the placement of the Sensors, place both the Ultrasonic Sensors on top of the Laptop screen, one at the left end and the other at right. You can use double sided tape to hold the sensors onto the screen.
Coming to Arduino, place it on the back of the laptop screen. Connect the wires from Arduino to Trigger and Echo Pins of the individual sensors. Now, we are ready for programming the Arduino.
Programming Your Arduino to Detect Gestures
The important part of this project is to write a program for Arduino such that it converts the distances measured by both the sensors into the appropriate commands for controlling certain actions. We have already seen a project called PORTABLE ULTRASONIC RANGE METER, where you can measure the distance of an object placed in front of an Ultrasonic Sensor with the help of Arduino. A similar concept is used here to measure the distance of your hand in front of both the Ultrasonic Sensors in this project. The fun part starts after calculating the distance. The hand gestures in front of the Ultrasonic sensors can be calibrated so that they can perform five different tasks on your computer. Before taking a look at the gestures, let us first see the tasks that we can accomplish.
Switch to Next Tab in a Web Browser Switch to Next Tab in a Web Browser Scroll Down in a Web Page Scroll Up in a Web Page Switch between two Tasks (Chrome and VLC Player) Play/Pause Video in VLC Player Increase Volume Decrease Volume
The following are the 5 different hand gestures or actions that I’ve programmed for demonstration purpose. Gesture 1: Place your hand in front of the Right Ultrasonic Sensor at a distance (between 15CM to 35CM) for a small duration and move your hand away from the sensor. This gesture will Scroll Down the Web Page or Decrease the Volume. Gesture 2: Place your hand in front of the Right Ultrasonic Sensor at a distance (between 15CM to 35CM) for a small duration and move your hand towards the sensor. This gesture will Scroll up the Web Page or Increase the Volume. Gesture 3: Swipe your hand in front of the Right Ultrasonic Sensor. This gesture will move to the Next Tab. Gesture 4: Swipe your hand in front of the Left Ultrasonic Sensor. This gesture will move to the Previous Tab or Play/Pause the Video. Gesture 5: Swipe your hand across both the sensors (Left Sensor first). This action will Switch between Tasks. Based on the above mentioned gesture, the following Arduino Program has been written.
Arduino Code
If you observe in the Arduino Code, the gesture mentioned above have been converted into 5 Commands that are sent to the Serial Port. Using these 5 commands, you can write a Python Program to control certain Keyboard Functions in order to achieve the required task.
Python Programming for the Project
Writing Python Program for Arduino based Hand Gesture Control is very simple. You just need to read the Serial data from Arduino and invoke certain keyboard key presses. In order to achieve this, you have to install a special Python Module called PyAutoGUI.
Installing PyAutoGUI
The following steps will guide you through the installation of PyAutoGUI on Windows Computers. The module PyAutoGUI will help you to programmatically control the mouse and keyboard. With the help of PyAutoGUI, we can write a Python Program to mimic the actions of mouse like left click, right click, scroll, etc. and keyboard like keypress, enter text, multiple key press, etc. without physically doing them. Let us install PyAutoGUI. If you remember in the previous project, where we controlled an LED on Arduino using Python, we have installed Python in the directory “C:\Python27”. Open Command Prompt with Administrator privileges and change to the directory where you have installed Python (in my case, it is C:\Python27).
If you have installed the latest version of Python, then pip (a tool for installing packages in Python) will already be installed. To check if pip is installed or not, type the following command. pip -V You should upgrade to the latest package of pip using the following command. If pip is already in its latest version, then ignore this step. python -m pip install -U pip Or python -m pip install –upgrade pip
After upgrading pip, you can proceed to install PyAutoGUI. In order to install PyAutoGUI, type the following command. python -m pip install pyautogui Or pip install pyautogui
Python Code
If everything goes well till now, you can proceed to write the Python Code. If you observe the Arduino Code given above, the Arduino sends out five different texts or commands through Serial Port upon detecting appropriate hand gestures. These commands are
Next Previous Down Up Change
Using these commands along with few functions in PyAutoGUI (like hotkey, scroll, keyDown, press and keyUp), you can write a simple Python Code that will execute the following tasks of keyboard and mouse.
Data = “next” – – > Action = Ctrl+PgDn Data = “previous” – – > Action = Ctrl+PgUp Data = “down” – – > Action = Down Arrow Data = “up” – – > Action = Up Arrow Data = “change” – – > Action = Alt+Tab
The Python Code for Arduino based Hand Gesture Control of Computer is given below.
NOTE: We have used Google Chrome as Web Browser and VLC Player as Media Player. Also, we modified the hotkeys of VLC Player to suit our Python Program. The modifications are as follows.
Keypress = Up Arrow – – > Action = Increase Volume Keypress = Down Arrow – – > Action = Decrease Volume Keypress = Ctrl+PgUp – – > Action = Play/Pause
Application of Arduino based Hand Gesture Control of Computer
In this project, we have implemented Arduino based Hand Gesture Control of Your Computer, where few hand gestures made in front of the computer will perform certain tasks in the computer without using mouse or keyboard. Such Gesture based Control of Computers is already present and a company called Leap Motion has been implementing such technology in computers. This type of hand gesture control of computers can be used for VR (Virtual Reality), AR (Augmented Reality), 3D Design, Reading Sign Language, etc.
do you have the same code for Python 3.4 ? This is the error which i am getting,please help me to sort out this, hope you will reply soon. regards puneet !!! ? This is the error which i am getting,please help me to sort out this, hope you will reply soon. If someone else is facing the same problem the try debugging the same code after closing the serial monitor in Arduino IDE I have Elegoo, and the name of the port is different, so, this project can work with Elegoo? Comment * Name * Email * Website
Δ
