Learn to build a web-based, gesture-controlled Breakout game from scratch using JavaScript, HTML Canvas, and MediaPipe for hand tracking. A fun, step-by-step AI project for developers of all levels.
Tired of the same old keyboard and mouse? What if you could control a game with a simple wave of your hand? It sounds like science fiction, but today, you’ll learn that it’s not only possible but surprisingly accessible.
In this tutorial, we’ll build a complete, web-based version of the classic game Breakout, but with a modern AI twist. You’ll control the paddle with your hand’s position, start and pause the game with hand gestures, and learn the core principles of building an AI-powered interactive experience.
The Customer Need: An Intuitive, “Magical” Experience
Before we write a single line of code, let’s define our goal. Our user wants more than just a game; they want a fun, interactive experience that feels futuristic and intuitive. They want to move beyond traditional controls and experience the “magic” of AI firsthand.
Our mission is to take this need and turn it into a real product. We’ll do this by building the game in three distinct, easy-to-follow parts:
- Part 1: The Foundation: Build a classic Breakout game controlled by the mouse.
- Part 2: The AI Magic: Integrate webcam-based hand tracking to control the paddle.
- Part 3: The Polish: Add gesture commands (start/pause) and game design improvements to make it a truly great experience.
Ready? Let’s get started.
Part 1: The Foundation (Mouse-Controlled Breakout)
Every great project starts with a solid base. We’ll first build a fully functional Breakout game that you can play with your mouse. This lets us perfect the game logic before adding the complexity of AI.
Step 1: The HTML Structure
All we need is a single HTML file. Create a file named breakout.html and add the following boilerplate. This sets up our game canvas, which is the digital easel where we’ll draw our game.
<!DOCTYPE html>
<html>
<head>
<title>Gesture-Controlled Breakout</title>
<style>
body { margin: 0; overflow: hidden; background-color: #000; }
canvas { display: block; }
</style>
</head>
<body>
<canvas id="gameCanvas"></canvas>
<script>
// All our JavaScript will go here!
</script>
</body>
</html>Step 2: Game Logic and Drawing
Now, let’s add the JavaScript logic inside the <script> tag. We’ll start by setting up the canvas and defining our game elements—the ball, the paddle, and the bricks.
We’ll create functions to draw each element and a main “game loop” that updates the screen on every frame using requestAnimationFrame.
Here is the core code for a functional, mouse-controlled Breakout game.
const canvas = document.getElementById('gameCanvas');
const ctx = canvas.getContext('2d');
// Game elements
let ball, paddle, bricks, score, lives;
function initGame() {
canvas.width = window.innerWidth;
canvas.height = window.innerHeight;
score = 0;
lives = 3;
paddle = {
h: 20,
w: canvas.width / 6.5,
x: (canvas.width - (canvas.width / 6.5)) / 2
};
ball = {
radius: 12,
x: canvas.width / 2,
y: canvas.height - 100,
dx: 4, // Speed on x-axis
dy: -4 // Speed on y-axis
};
// Brick setup will be in the final code
}
function drawPaddle() { /* Draws the paddle */ }
function drawBall() { /* Draws the ball */ }
function drawBricks() { /* Draws the bricks */ }
function collisionDetection() { /* Checks for collisions */ }
function updateGame() { /* Updates game logic */ }
function mainLoop() {
ctx.clearRect(0, 0, canvas.width, canvas.height);
// ... drawing and update calls ...
requestAnimationFrame(mainLoop);
}
// Mouse control
document.addEventListener('mousemove', (e) => {
paddle.x = e.clientX - paddle.w / 2;
});
initGame();
mainLoop();(For brevity, the full drawing and collision logic isn’t written out here, but it is included in the final code at the end of the tutorial.)
At this point, you would have a playable game using just the mouse. This is a crucial checkpoint.
Part 2: The AI Magic (Adding Gesture Control)
Now for the exciting part. We’re going to give our game eyes. To do this, we’ll use Google’s MediaPipe, a powerful and free framework that does the heavy lifting of computer vision for us. It can find hands in a video feed in real-time, right in the browser.
Step 3: Setting Up MediaPipe and the Camera
First, add a <video> element to your HTML (it won’t be visible to the user) and include the MediaPipe library scripts from their CDN. This is like importing a superpower into our project.
Add the <video> element inside the <body> tag, before the canvas:
<video class="input_video"></video>Add the <script> tags inside the <head> tag:
<script src="https://cdn.jsdelivr.net/npm/@mediapipe/camera_utils/camera_utils.js" crossorigin="anonymous"></script>
<script src="https://cdn.jsdelivr.net/npm/@mediapipe/control_utils/control_utils.js" crossorigin="anonymous"></script>
<script src="https://cdn.jsdelivr.net/npm/@mediapipe/drawing_utils/drawing_utils.js" crossorigin="anonymous"></script>
<script src="https://cdn.jsdelivr.net/npm/@mediapipe/hands/hands.js" crossorigin="anonymous"></script>Next, in our JavaScript, we’ll write the code to initialize MediaPipe and connect it to our webcam.
const videoElement = document.querySelector('.input_video');
const hands = new Hands({ locateFile: (file) => `https://cdn.jsdelivr.net/npm/@mediapipe/hands/${file}` });
hands.setOptions({
maxNumHands: 1,
modelComplexity: 1,
minDetectionConfidence: 0.5,
minTrackingConfidence: 0.5
});
hands.onResults(onResults); // We will define onResults next
const camera = new Camera(videoElement, {
onFrame: async () => await hands.send({ image: videoElement }),
width: 1280,
height: 720
});
camera.start();Step 4: Connecting Your Hand to the Paddle
When MediaPipe finds a hand, it calls the onResults function and gives us the coordinates of 21 key landmarks.
We only need one of these points to control our paddle: landmark 8, the tip of the index finger. Inside our onResults function, we’ll get the position of this landmark and use it to guide the paddle.
function onResults(results) {
if (results.multiHandLandmarks && results.multiHandLandmarks.length > 0) {
const landmarks = results.multiHandLandmarks[0];
// The camera feed is a mirror, so we flip the x-coordinate for intuitive control
const handX = (1 - landmarks[8].x) * canvas.width;
// Set the paddle's target position
targetPaddleX = handX - (paddle.w / 2);
}
}Crucial Detail: Your webcam shows you a mirror image. If you move your hand right, its image moves left on the screen. To make the controls feel natural, we have to flip the coordinate by subtracting it from 1 (
1 - landmarks.x).
With this, you now have a gesture-controlled game! But we’re not done. A working project isn’t the same as a great product.
Part 3: The Polish (A Masterclass in Game Feel)
Now, let’s add the features that elevate this from a tech demo to a genuinely fun game.
Step 5: Game States and Gesture Commands
We need a way to start and pause the game. We’ll create a simple state machine and write a robust detectGestures function. This function will check if the user is making a “Thumbs-Up” (to start/restart) or a “Peace Sign” (to pause/resume).
The logic is simple and intuitive:
- Thumbs-Up: The thumb is extended, and the other four fingers are curled.
- Peace Sign: The index and middle fingers are extended, and the ring and pinky fingers are curled.
We’ll add a handleGestures function that listens for these flags and changes the gameState accordingly, with a “cooldown” to prevent it from firing too quickly.
Step 6: Improving the Game Feel
To truly meet our user’s need for a “magical” experience, we’ll add three key design improvements:
- Smooth Paddle Movement: Instead of having the paddle instantly jump to your hand’s position, we’ll make it glide smoothly. This is done with a technique called linear interpolation, and it makes the controls feel responsive and natural.
// In the updateGame() function paddle.x += (targetPaddleX - paddle.x) * 0.2; - Dynamic Difficulty: A game that never gets harder is boring. We’ll make the ball’s speed increase slightly every time you successfully hit it with the paddle. This rewards skilled play and keeps the player engaged.
- Visual Polish: We’ll replace the harsh black background with a subtle dark blue gradient, make the ball glow, and add instant visual feedback to confirm when a gesture is recognized.
The Final Code
Here is the complete, single-file code for our finished product. It includes all the logic, AI integration, and design polish we discussed. Create a file named breakout.html, paste this code in, and open it in your browser.
<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8" />
<title>Gesture-Controlled Breakout</title>
<style>
body { margin: 0; overflow: hidden; background-color: #000; }
canvas { display: block; cursor: pointer; }
.input_video { display: none; }
</style>
</head>
<body>
<video class="input_video"></video>
<canvas id="gameCanvas"></canvas>
<script src="https://cdn.jsdelivr.net/npm/@mediapipe/camera_utils/camera_utils.js" crossorigin="anonymous"></script>
<script src="https://cdn.jsdelivr.net/npm/@mediapipe/control_utils/control_utils.js" crossorigin="anonymous"></script>
<script src="https://cdn.jsdelivr.net/npm/@mediapipe/drawing_utils/drawing_utils.js" crossorigin="anonymous"></script>
<script src="https://cdn.jsdelivr.net/npm/@mediapipe/hands/hands.js" crossorigin="anonymous"></script>
<script>
const canvas = document.getElementById('gameCanvas');
const ctx = canvas.getContext('2d');
let gameState = 'start';
let lives, score, ball, paddle, bricks;
let gestureCooldown = 0;
let isThumbUp = false;
let isPeaceSign = false;
const brickRowCount = 5;
const brickColumnCount = 8;
const brickColors = ['#e63946', '#f1faee', '#a8dadc', '#457b9d', '#1d3557'];
let targetPaddleX;
function initGame() {
lives = 3;
score = 0;
const paddleHeight = 20;
const paddleWidth = canvas.width / 6.5;
paddle = { h: paddleHeight, w: paddleWidth, x: (canvas.width - paddleWidth) / 2 };
targetPaddleX = paddle.x;
const ballRadius = 12;
ball = { radius: ballRadius, x: canvas.width / 2, y: canvas.height - 100, dx: 4 * (Math.random() > 0.5 ? 1 : -1), dy: -4, speed: 5.5 };
bricks = [];
const brickHeight = 30;
const brickPadding = 15;
const brickOffsetTop = 60;
const brickOffsetLeft = 30;
const availableWidth = canvas.width - (2 * brickOffsetLeft);
const brickWidth = (availableWidth - (brickColumnCount - 1) * brickPadding) / brickColumnCount;
for (let c = 0; c < brickColumnCount; c++) {
bricks[c] = [];
for (let r = 0; r < brickRowCount; r++) {
const brickX = (c * (brickWidth + brickPadding)) + brickOffsetLeft;
const brickY = (r * (brickHeight + brickPadding)) + brickOffsetTop;
bricks[c][r] = { x: brickX, y: brickY, w: brickWidth, h: brickHeight, status: 1, color: brickColors[r % brickColors.length] };
}
}
}
function resizeCanvas() { canvas.width = window.innerWidth; canvas.height = window.innerHeight; if (gameState !== 'start') { initGame(); }}
window.addEventListener('resize', resizeCanvas);
function detectGestures(landmarks) {
if (!landmarks) { isThumbUp = false; isPeaceSign = false; return; }
const fingersExtended = [];
fingersExtended.push(landmarks[4].y < landmarks[3].y);
fingersExtended.push(landmarks[8].y < landmarks[6].y);
fingersExtended.push(landmarks[12].y < landmarks[10].y);
fingersExtended.push(landmarks[16].y < landmarks[14].y);
fingersExtended.push(landmarks[20].y < landmarks[18].y);
isThumbUp = fingersExtended[0] && !fingersExtended[1] && !fingersExtended[2] && !fingersExtended[3] && !fingersExtended[4];
isPeaceSign = fingersExtended[1] && fingersExtended[2] && !fingersExtended[3] && !fingersExtended[4];
}
function handleGestures() {
if (gestureCooldown > 0) { gestureCooldown--; return; }
const GESTURE_COOLDOWN_FRAMES = 60;
if ((gameState === 'instructions' || gameState === 'gameOver') && isThumbUp) { initGame(); gameState = 'playing'; gestureCooldown = GESTURE_COOLDOWN_FRAMES; }
else if (gameState === 'playing' && isPeaceSign) { gameState = 'paused'; gestureCooldown = GESTURE_COOLDOWN_FRAMES; }
else if (gameState === 'paused' && isPeaceSign) { gameState = 'playing'; gestureCooldown = GESTURE_COOLDOWN_FRAMES; }
}
function drawBackground() { const gradient = ctx.createRadialGradient(canvas.width / 2, canvas.height / 2, 5, canvas.width / 2, canvas.height / 2, canvas.width); gradient.addColorStop(0, '#023e8a'); gradient.addColorStop(1, '#000428'); ctx.fillStyle = gradient; ctx.fillRect(0, 0, canvas.width, canvas.height); }
function drawPaddle() { if (!paddle) return; ctx.beginPath(); ctx.rect(paddle.x, canvas.height - paddle.h, paddle.w, paddle.h); ctx.fillStyle = '#ade8f4'; ctx.fill(); ctx.closePath(); }
function drawBall() { if (!ball) return; ctx.beginPath(); ctx.arc(ball.x, ball.y, ball.radius, 0, Math.PI * 2); ctx.fillStyle = '#ffffff'; ctx.shadowColor = '#ffffff'; ctx.shadowBlur = 15; ctx.fill(); ctx.closePath(); ctx.shadowBlur = 0; }
function drawBricks() { if (!bricks) return; for (let c = 0; c < brickColumnCount; c++) { for (let r = 0; r < brickRowCount; r++) { if (bricks[c][r].status === 1) { const b = bricks[c][r]; ctx.beginPath(); ctx.rect(b.x, b.y, b.w, b.h); ctx.fillStyle = b.color; ctx.fill(); ctx.closePath(); }}}}
function drawUI() { ctx.fillStyle = '#FFF'; ctx.font = '22px Arial'; ctx.textAlign = 'left'; ctx.fillText('Score: ' + score, 15, 35); ctx.textAlign = 'right'; ctx.fillText('Lives: ' + lives, canvas.width - 15, 35); if (gestureCooldown > 0) { ctx.font = '40px Arial'; ctx.textAlign = 'left'; if (gameState === 'playing' || gameState === 'paused') { ctx.fillText(isPeaceSign ? '✌️' : '👍', 15, 80); }}}
function drawMessageScreen(title, subtitle, titleColor = 'white') { ctx.fillStyle = 'rgba(0, 0, 0, 0.7)'; ctx.fillRect(0, 0, canvas.width, canvas.height); ctx.fillStyle = titleColor; ctx.font = 'bold 60px Arial'; ctx.textAlign = 'center'; ctx.fillText(title, canvas.width / 2, canvas.height / 2 - 50); ctx.fillStyle = 'white'; ctx.font = '30px Arial'; ctx.fillText(subtitle, canvas.width / 2, canvas.height / 2 + 20); }
function collisionDetection() { for (let c = 0; c < brickColumnCount; c++) { for (let r = 0; r < brickRowCount; r++) { const b = bricks[c][r]; if (b.status === 1 && ball.x > b.x && ball.x < b.x + b.w && ball.y > b.y && ball.y < b.y + b.h) { ball.dy = -ball.dy; b.status = 0; score++; if (score === brickRowCount * brickColumnCount) { gameState = 'gameOver'; }}}}}
function updateGame() { paddle.x += (targetPaddleX - paddle.x) * 0.2; if (ball.x + ball.dx > canvas.width - ball.radius || ball.x + ball.dx < ball.radius) { ball.dx = -ball.dx; } if (ball.y + ball.dy < ball.radius) { ball.dy = -ball.dy; } else if (ball.y + ball.dy > canvas.height - ball.radius) { if (ball.x > paddle.x && ball.x < paddle.x + paddle.w) { let collidePoint = (ball.x - (paddle.x + paddle.w / 2)) / (paddle.w / 2); let angle = collidePoint * (Math.PI / 3); ball.speed += 0.15; ball.dx = ball.speed * Math.sin(angle); ball.dy = -ball.speed * Math.cos(angle); } else { lives--; if (!lives) { gameState = 'gameOver'; } else { ball.speed = 5.5; ball.x = canvas.width / 2; ball.y = canvas.height - 100; paddle.x = (canvas.width - paddle.w) / 2; targetPaddleX = paddle.x; }}} ball.x += ball.dx; ball.y += ball.dy; }
function mainLoop() { drawBackground(); if (gameState === 'start') { drawMessageScreen('Gesture Breakout', 'Click Anywhere to Begin'); } else if (gameState === 'instructions') { drawMessageScreen('Setup Complete!', 'Show a THUMBS-UP to Start'); } else if (gameState === 'playing' || gameState === 'paused') { drawBricks(); drawBall(); drawPaddle(); drawUI(); if (gameState === 'playing') { collisionDetection(); updateGame(); } else { drawMessageScreen('Paused', 'Show a PEACE SIGN to Resume'); } } else if (gameState === 'gameOver') { const isWin = score === brickRowCount * brickColumnCount; const title = isWin ? 'YOU WIN!' : 'GAME OVER'; const color = isWin ? '#70e000' : '#d00000'; drawMessageScreen(title, 'Show a THUMBS-UP to Restart', color); } requestAnimationFrame(mainLoop); }
const videoElement = document.querySelector('.input_video');
const hands = new Hands({ locateFile: (file) => `https://cdn.jsdelivr.net/npm/@mediapipe/hands/${file}` });
hands.setOptions({ maxNumHands: 1, modelComplexity: 1, minDetectionConfidence: 0.5, minTrackingConfidence: 0.5 });
hands.onResults(onResults);
const camera = new Camera(videoElement, { onFrame: async () => await hands.send({ image: videoElement }), width: 1280, height: 720 });
camera.start();
function onResults(results) { if (results.multiHandLandmarks && results.multiHandLandmarks.length > 0) { const landmarks = results.multiHandLandmarks[0]; detectGestures(landmarks); handleGestures(); targetPaddleX = ((1 - landmarks[8].x) * canvas.width) - (paddle.w / 2); } else { isThumbUp = false; isPeaceSign = false; }}
canvas.addEventListener('click', function() { if (gameState === 'start') { gameState = 'instructions'; initGame(); }});
resizeCanvas();
mainLoop();
</script>
</body>
</html>Conclusion: You Did It!
Congratulations! You’ve successfully built a complete, gesture-controlled game from the ground up. You took a user need for a futuristic, interactive experience and delivered a polished product that brings that vision to life.
More importantly, you’ve learned the fundamental workflow of modern AI web development: start with a solid foundation, integrate a powerful AI library like MediaPipe, and polish the final product with a keen eye for user experience.
Now, the real fun begins. What will you build next? You could use these same principles to control presentation slides, a virtual musical instrument, or even a drone. The possibilities are endless.
Try out the Gesture-controlled Breakout Game online.
Share your creations and ideas in the comments below!
Leave a Reply