Setting Up Duel Game

The Duel Mode Concept

1. Two players stand side-by-side in front of the camera

2. The game displays a random letter to sign

3. The first player to correctly form the sign wins the round

4. Players compete until one reaches the target score (6 points)

This is simple concept when I planned it out. But to implement it, it presented some complex technical challenges, particularly in detecting and distinguishing between two players' hand signs simultaneously.

Enhanced Python Detection System

To support both Learning Mode and Duel Mode, I completely redesigned my Python detection script to dynamically adjust based on the current game mode. I don't think having two python scripts is effective. So, I had an idea of implementing listener thread that receives scene information from Unity and automatically switches detection modes. 

Player Separation in Duel Mode

The most critical part of the Duel Mode detection system is distinguishing between two players standing side-by- side. I implemented this by dividing the camera view down the middle (with a line, of course)

Draw center line:

Bounding box for each detected hand

This simple but effective approach assigns hands to players based on which side of the screen they appear on. Each detection includes a player ID that's sent to Unity with the detection data. In the image I can only show one hand (because I needed my other hand to screenshot) but it will do and the same thing on the other side but different color and ID.

Unity Duel Mode Implementation

Game State Management

The DuelGame script tracks the current state of the competition:

Running a Round Sequence

Each round follows a structured sequence to ensure fair play:

Detection Window Processing

To ensure fair competition, I implemented a "detection window" system. This system ensures that even if one player is slightly faster, both players have a fair chance to complete their signs. Just in case they did the wrong sign, you still got chance!

IEnumerator ProcessDetectionWindow()

{

    // Mark detection window as active

    detectionWindowActive = true;

    

    // Initial detection timestamps

    float p1Time = float.MaxValue;

    float p2Time = float.MaxValue;

    

    // Store starting time of detection window

    float startTime = Time.time;

    bool p1Detected = false;

    bool p2Detected = false;

    

    // Continuously monitor for the duration of the window

    while (Time.time < startTime + detectionWindowDuration)

    {

        // Check both players every frame during the window

        if (udpReceiver.player1Detection != null && 

            udpReceiver.player1Detection.letter == duelLetters[currentLetterIndex] &&

            udpReceiver.player1Detection.confidence >= udpReceiver.minimumConfidence &&

            !p1Detected)

        {

            p1Time = Time.time;

            p1Detected = true;

        }

        

        if (udpReceiver.player2Detection != null && 

            udpReceiver.player2Detection.letter == duelLetters[currentLetterIndex] &&

            udpReceiver.player2Detection.confidence >= udpReceiver.minimumConfidence &&

            !p2Detected)

        {

            p2Time = Time.time;

            p2Detected = true;

        }

        

        // If both detected, we can stop early

        if (p1Detected && p2Detected)

            break;

            

        yield return null;

    }

    

    // Determine winner

    if (p1Detected && p2Detected)

    {

        // Both detected - winner is whoever was faster

        int winner = (p1Time <= p2Time) ? 1 : 2;

        AwardPointToPlayer(winner);

    }

    else if (p1Detected)

    {

        // Only Player 1 detected

        AwardPointToPlayer(1);

    }

    else if (p2Detected)

    {

        // Only Player 2 detected

        AwardPointToPlayer(2);

    }

    else

    {

        // No detections

        detectionWindowActive = false;

        roundActive = true; // Resume the round

    }

}

*Code is too long to screenshot

User Testing and the Evolution of Score Tracking

One of the most rewarding aspects of developing this mode was the early user testing sessions. I invited friends to try the game, and watching them compete while giving real-time feedback created a fun, collaborative atmosphere that significantly improved the final product.

Initially, I used simple numerical counters to track each player's score:

During one testing session, a participant made a suggestion that changed the entire feel of the game. As two players were neck-and-neck at 4-5 points, one of them said, "It would be cool if we could see the score filling up rather than just numbers."

This sparked an energetic discussion where we all chimed in with ideas. We sketched different visual representations and the consensus landed on a block-based system where each point would add a visual block to the player's score container.

I implemented this suggestion:

New Experience

This experience highlighted the value of not just collecting user feedback but actually involving testers in the design conversation. The collaborative atmosphere of these testing sessions produced ideas I never would have considered on my own.

Enhanced UDP Communication

To handle the complexities of two-player detection, I significantly upgraded the UDP communication system. The UDPReceiver class maintains separate detection data for each player:

It also dynamically adjusts to scene changes and game mode switches:

Technical Challenges

1. Detection Reliability in a Two-Player Setup

Perhaps the biggest challenge was ensuring reliable detection for two people standing side-by-side. Our TensorFlow model was originally trained for single-hand detection, and now it needed to accurately differentiate between two players' hands simultaneously.

The first problem was that the model would sometimes confuse hands when they crossed the center line or when one player reached into the other player's space. To solve this, I implemented a stricter player assignment system where hands were tracked continuously rather than just being assigned based on position in each frame. This prevented sudden player identity switches when hands moved around.

2. Ensuring Fair Competition

A major issue I discovered during early testing was that players would often raise their hands before the other player was ready. Since our TensorFlow model is extremely sensitive, it would sometimes award points unfairly when one player happened to have their hand in view first, even if they weren't actually signing correctly yet.

For example, if Player 1 raised their hand early while Player 2 was still getting ready, the system might occasionally misinterpret Player 1's hand as the correct sign and award them the point, even though Player 2 might have actually formed the correct sign first once both were ready.

How I address this:

First, I added a clear countdown sequence with "3, 2, 1, GO!" displayed prominently before each round starts. This gave both players equal preparation time and a clear signal for when to begin forming signs.

Second, I implemented a "Lower your hands" prompt between rounds. After each round, the system displays a message asking players to lower their hands, then waits for a short period before starting the next round. This prevents the hypersensitive detection system from carrying over detections between rounds.

Most importantly, I created a "detection window" system. When either player makes a correct sign, instead of immediately awarding them the point, the system opens a brief window (about 1 second) to allow the other player to complete their sign as well. If both players make the correct sign within this window, the point goes to whoever was faster.

Insights

Implementing the Duel Mode was challenging but rewarding. The final system creates an engaging competitive experience that encourages players to master their sign language skills while having fun.

Some key insights from this development process:

1. Unified Detection System: Creating a single Python script that handles both modes eliminated the need to switch between scripts and provided a more integrated experience.
2. Collaborative Design: The interactive user testing sessions transformed the development process from a solo endeavor into a collaborative design experience. The excitement and engagement of testers not only provided valuable feedback but also generated innovative ideas I wouldn't have considered on my own.
3. Fair Competition: The detection window system ensures fair play even with slight timing differences.
4. Visual Feedback: The block-based scoring system suggested by users created a more engaging visual representation of progress.

The Duel Mode adds an exciting competitive element to our sign language learning application, making practice more engaging and motivating users to improve their signing speed and accuracy. The positive reactions during testing sessions confirmed that turning learning into a social, competitive experience can significantly increase engagement and enjoyment.