Setting Up Learning Mode

After choosing UDP as the optimal communication protocol for my sign language learning game, I've finally successfully implemented the full communication pipeline between the Python hand detection system and the Unity game interface. In this blog, I will go through on how I build my learning mode!

Python Side Implementation

On the Python side, I use TensorFlow Lite with a pre-trained model to detect hand signs from webcam input. When a sign is detected with confidence, I extract the letter and confidence score, then send this information to Unity:

Unity Side Implementation

For the Unity side, I used the UDP receiver script provided in the YouTube tutorial and customized it to fit my specific needs. The script already contained most of the functionality I needed, which saved me significant development time. Thank you so much Mr. Murtaza! My customizations focused on:

1. Adapting the script to work with my specific JSON data format
2. Adding the event system to notify game components about new detections:
3. Implementing the thread-safe queue system to safely pass data from the UDP thread to Unity's main thread

Game System of Learning Mode

In learning mode, I want my game system to teach users to make sign language gestures through guided and interactive lessons.

Core Components

The learning mode consists of several key components:

1. UDPReceiver: Handles incoming hand sign detections from Python (based on the tutorial's code)
2. LearningModeManager: Controls the learning flow and game logic
3. UI Components: Display the current letter, instructions, feedback, and camera feed

Flow of Learning Mode

1. Initialization:

• The system displays the webcam feed in Unity so users can see their hands
• A sequence of letters to learn is pre-loaded (currently A-C in my implementation)
• The LearningModeManager subscribes to the UDPReceiver's OnLetterDetected event
  

2. Learning Flow:

• A letter is presented along with a demonstration image showing the correct hand sign
• The user attempts to form the correct hand sign on the camera view
• Python detects the gesture and sends the recognized letter to Unity

3. Feedback system:

• When the correct letter is detected:
• Visual feedback: The letter flashes green
• Audio feedback: A success sound plays
• Character feedback: The guide congratulates the user
• The character then explicitly asks the user to lower their hand before continuing
• After the user lowers their hand, the system advances to the next letter
• For incorrect signs, the character provides helpful tips and encouragement

4. Completion:

• The system generates a personalized certificate with the user's name

Key Codes

Character Guide System

One of the key features that I want to put is the character guide. This will serve as a virtual instructor who provides personalized feedback. I implemented this with an image of their profile character and speech bubble system:

The character provides various types of feedback, which I store in arrays for variety:

Handling Letter Detection

The core functionality for handling letter detection is in the HandleLetterDetection method, which is triggered by the UDP event system:

Completion

After the user completed their learning, they will get a certificate of achievements with their name on it!

Key Technical Challenges

1. Python Stability Issues: I encountered random crashes in the Python detection script that required error handling and restart mechanisms.

2. Hypersensitive Detection: The TensorFlow model is extremely sensitive..It immediately tries to detect and classify any hand that appears in the camera frame.

Addressing the Hypersensitive Detection

To address the hypersensitivity issue of the TensorFlow model, I had an idea to implemented a simple but effective solution. After correctly signing a letter, the character asks the user to lower their hand. This creates a natural pause between letter detections and prevents the system from immediately trying to detect the next letter while the user's hand is still visible. After a sufficient pause (5 seconds), we move to the next letter. This problem is helped by the UI flow.

Setting Up User Interface

1. The Classroom 

My first concept featured a green chalkboard-inspired background with a brown-bordered frame. This created a familiar classroom feeling with the sign letter at the top, hand illustration on the left, and user's camera feed on the right. A character mascot in the corner served as a teacher figure.

2. The Book

Next, I tried a book-inspired design with a beige background and bookmark tabs. I aim to let users learn from a book like the old days. I envisioned this with a flipping book animation or adding a swipe gesture to go to the next letter.

3. Card Design (Final)

After testing both approaches, I settled on a cleaner card-based design that maintained the app's yellow grid background. The sign illustration appeared on a card with a subtle gradient, while the user's camera feed was clearly framed. Instructions appeared in a clean white container at the bottom, and the character mascot evolved from teacher to supportive guide.

Why the Final Design Worked Best

The simpler card design proved superior for several reasons:

• It maintained visual consistency with the rest of the app
• The cleaner layout reduced distractions and kept focus on learning
• Instructions were more readable in the dedicated text area
• The character felt more like a helpful companion than a formal instructor
• The design adapted better to different screen sizes

This shows how educational interfaces often benefit from simplicity and clarity over thematic design elements. I think by removing unnecessary visual metaphors, the final design better served the primary goal.

Moving Forward

With this foundation in place, I'm now planning to start developing duel mode which allows two players will compete to sign the correct letter first with a fun and competitive element.