In this guide you can learn about
What is SignTrack
SignTrack is a sign language transcriber that analyzes, processes, and recognizes sign language in real-time with remarkable accuracy and efficiency. It takes a people-centered approach to computing, aiming to make computers more accessible and inclusive for everyone. By seamlessly converting sign language into written language, SignTrack breaks down communication barriers and enables meaningful connections.
How it works
SignTrack harnesses the power of a state-of-the-art LSTM (Long Short-Term Memory) model, which leverages sequence-based data for predictions. This allows SignTrack to detect entire phrases and capture dynamic movements within sign language gestures. To enhance precision and efficiency, SignTrack focuses its training on key hand and pose landmarks. These essential landmarks are extracted using MediaPipe, a robust framework for pose estimation. By concentrating on these key landmarks using MediaPipe, SignTrack ensures accurate recognition across various skin shades.
Showcase
Data Collection
We prioritize making data collection user-friendly, even for non-coders. Our interface is easy to navigate, ensuring a seamless experience.
With its recent updates, SignTrack DataCollect has undergone significant improvements, resulting in a seamlessly streamlined experience. The enhanced reliability and smooth functionality of the platform empower developers, allowing them to effortlessly create high-quality training datasets and build custom models that precisely cater to their unique requirements.
Now, before saving a sequence in the dataset, SignTrack employs a verification process to enasure that hands are visible within the frames. This approach eliminates unnecessary data and enhances the overall quality and accuracy of the collected dataset.
Another standout feature is image flipping, generating data that simulates signing with the opposite hand. This enables unparalleled accuracy and versatility, ensuring precise predictions for both hands across all signing variations.
Privacy is a top priority. SignTrack maintains privacy by excluding personal information and unique biometric data from the dataset.
SignTrack DataCollect offers a comprehensive solution: user-friendly interface, robust data collection capabilities and enables advanced augmentation techniques. SignTrack empowers developers to create accurate and inclusive sign language recognition models tailored to their application requirements.
Model Training
Training a neural network can become confusing. Everything in SignTrack is automated, with the power of AutoTrain+. AutoTrain+ automatically sets training parameters, such as the number of epochs and data split, forming a training process that requires minimal or no adjustment from the user. AutoTrain+ ensures that you get the most out of your dataset by setting all the training parameters for you while automatically saving the best-performing model. It takes care of the training process, making it efficient and effective. During training, AutoTrain+ continuously checks for overtraining and reverts to a previous version of the model if needed. This helps prevent the model from overfitting and ensures optimal performance.
In addition to AutoTrain+, SignTrack Revision 1 introduces the new InSpace Engine, a revolutionary form of data augmentation. The InSpace Engine allows for the simulation of different camera angles, providing more diverse training data. This augmentation technique enhances the model's ability to generalize and recognize sign language gestures accurately, even from different perspectives.
What's more, AutoTrain+ has been redefined to make for an even faster tracking experience. FastTrack+ has been implemented right through the model training process. Utilizing FastTrack+ during inference significantly improves the model's ability to handle quicker signing speeds, making SignTrack a robust and versatile tool for real-time sign language recognition.
SignTrack Main
All the new updates form a unique and refreshing experience. SignTrack Revision 1 can make up to 6 times faster predictions than our previous version with less lag between signs, made possible by the reimagined FastTrack+ and InSpace Engine.
With FastTrack+, we utilize hand landmarks to analyze frames before and after the current one, enabling us to add contextual information to each frame. By detecting and comparing the hand landmarks in the previous and next frames, FastTrack+ intelligently calculates the differences between them. This process allows us to extract valuable insights and enhance the understanding of sign language gestures by incorporating temporal context.
With the InSpace Engine, we redefine what's possible in sign language recognition, empowering users to communicate naturally and effortlessly, irrespective of camera positioning or distance. This innovative technology represents a major leap forward, revolutionizing the accuracy and usability of sign language recognition systems.
SignTrack offers an uninterrupted experience through innovative optimizations. We have designed SignTrack to utilize resources only when necessary, ensuring efficient usage.
For instance, the SignTrack model is intelligently triggered to make predictions only when hands are visible within the scene. This intelligent activation conserves resources and optimizes performance, resulting in a smooth and responsive user experience.
But that's not all. We have taken SignTrack a step further. SignTrack Revision 1 can now understand when the user has completed a word, enabling it to immediately begin detection for the next one. This seamless transition between words ensures a fluid and natural interaction, making sign language recognition with SignTrack even more intuitive and efficient.
Dataset
SignTrack is bundled with a comprehensive dataset that includes 25 American Sign Language (ASL) words and phrases. This dataset serves as a solid foundation for testing. However, if you find that the model's performance does not meet your expectations, SignTrack provides a convenient solution through DataCollect.
Remember, the goal is to train SignTrack to accurately interpret sign language in real-world scenarios. So, capturing data from different angles and positions enhances the model's ability to handle variations commonly encountered in everyday signing.
Illustration
Each illustration has been created from scratch, ensuring a unique and tailored visual representation. In the assets directory, you will find SVG versions of these illustrations, providing flexibility and the ability to customize them according to your preferences. Feel free to modify and tinker with the illustrations using popular open-source illustration tools such as Inkscape.
Dependances
This project has been developed using:
- Python: 3.7
- Tensorflow: 2.5
- OpenCV: 4.1.2.30
- Scikit-Learn
- Matplotlib
- Mediapipe
- Cvzone
- Sentencepiece
- Transformers
Setup
SignTrack uses python poetry to make installation a breeze. Dealing with TensorFlow can often be difficult. We made sure that this is not the case with SignTrack.
Windows installation
-
Install Python 3.7
-
Install Python Poetry
-
Before installing Tensorflow you also have to install Visual C++ Redistributable for Visual Studio 2015
-
Open the location where SignTrack is downloaded on your terminal
-
Run the command: poetry install
-
✨Now a virtualenv has been created containing the required dependances✨
Linux & MacOS installation
-
Install Python Poetry
-
Open the location where SignTrack is downloaded on your terminal
-
Run the command: poetry install
-
✨Now a virtualenv has been created containing the required dependances✨
To run SignTrack( Applicable on all Operating Systems )
- Navigate to the SignTrack main script (signtrack/main)
- Open on terminal
- Type and run on terminal poetry shell
- Type and run on terminal python main.py
Troubleshoting
OpenCV errors
Try changing camera input
You can simply try to change the camera input selection in the first lines of code either on DataCollect or main file in the code.
-
Find this line of code cap = cv2.VideoCapture(0)
-
Change the default value zero to another one, like 1 or 2
Reinstall OpenCV
If that does not work try reinstalling OpenCV
-
Type poetry shell on your terminal on the SignTrack directory
-
Then type pip uninstall opencv-python
-
Finally type pip install opencv-python==4.1.2.30
Failling to install a package
Try installing manually from the poetry env
-
Type poetry shell on your terminal on the SignTrack directory
-
Type pip install and continue with typing the failed to install package, making sure that it is the matching version
Credits
Creating this amazing project would not have been possible without the help of:
- My mentor, Barbara Pongračić, for helping me focus on what is important.
- My friend, Fotini Tsiami, for assisting me throught the illustration designing proccess.
- DuckDuckGo and YouTube tutorials, well, for obvious reasons.