Running Docker on Windows WSL2/Ubuntu (Part 6) – Jellyfin with GPU Acceleration

Introduction: Why Jellyfin?

When it comes to media servers, three names dominate the scene: Plex, Emby, and Jellyfin.

Plex is polished but partially paywalled—premium features like advanced transcodes or mobile sync often sit behind subscriptions. And Emby.. I have no idea why it comes with paid plan.

Jellyfin, on the other hand, is 100% free and open-source. It includes hardware-accelerated transcoding at no cost.

👉 What is transcoding?

Transcoding is the process of converting video on the fly so that it plays smoothly on your device (for example, shrinking a 4K HEVC file into a 1080p H.264 stream for your phone). On Synology NAS, this is usually disabled because of weak CPUs. But on a desktop PC with Nvidia or AMD GPU, you can finally enjoy buttery-smooth playback with no stuttering.

That’s why for our WSL2 + Docker Engine setup, Jellyfin is the perfect choice.

Choosing the Right Networking Mode

Since Jellyfin isn’t just about streaming via a web browser—it also integrates with TV apps, consoles, and DLNA clients—the best way to unlock its full potential is by running it in host networking mode.

• Host Mode: Jellyfin binds directly to your PC’s IP, just like a native application. Required for DLNA, UPnP, and full LAN discovery.
• Bridge/Edge Mode: Works fine for web-only access through Nginx Proxy Manager (NPM), but limits advanced features.

👉 For this guide, we’ll use host mode to get the most out of Jellyfin.

Step 1: 🚀 Docker Compose for Jellyfin

If you have an Nvidia graphics card, Docker can pass it through to Jellyfin using the NVIDIA Container Toolkit.

- In earlier tutorials, you may see references to nvidia-docker2 or nvidia-container-runtime. These older packages are now deprecated.

- The current recommended way is to use nvidia-container-toolkit, which merges functionality and is actively maintained.

- For Docker 19.03+, you can also use the --gpus option together with this toolkit.

1-1. Install NVIDIA Container Toolkit (NVIDIA ONLY!)

On your Ubuntu WSL (or Linux host), run the following commands:

# Add NVIDIA package repositories
curl -s -L https://nvidia.github.io/libnvidia-container/gpgkey | sudo apt-key add -
distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
curl -s -L https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.list | \
  sudo tee /etc/apt/sources.list.d/nvidia-container.list

# Update package lists
sudo apt-get update

# Install the toolkit
sudo apt-get install -y nvidia-container-toolkit

# Restart Docker
sudo systemctl restart docker

✔ What this does:

• Adds NVIDIA’s repository to apt
• Installs the nvidia-container-toolkit package
• Configures Docker so containers can access your GPU
• Restarts Docker so the new runtime is available

👉 At this point, you can test if Docker detects your GPU:

docker run --rm --gpus all nvidia/cuda:12.2.0-base nvidia-smi

If successful, you’ll see your GPU details inside the container.

1-2. Docker Compose (NVIDIA GPU)

services:
  jellyfin:
    image: jellyfin/jellyfin:latest
    container_name: jellyfin
    network_mode: "host"
    runtime: nvidia
    environment:
      - NVIDIA_VISIBLE_DEVICES=all
      - NVIDIA_DRIVER_CAPABILITIES=all
      - TZ=America/New_York   # change to your timezone
    volumes:
      - ./config:/config
      - ./cache:/cache
      - /mnt/media/movies:/media/movies
      - /mnt/media/tv:/media/tv
    restart: unless-stopped

🔹 Key Points

• Host Networking → Full LAN integration (DLNA, Chromecast, etc.)
• NVIDIA Runtime & Env Vars → Enable GPU transcoding (decoding, encoding, CUDA)
• Persistent Volumes → Keep config, cache, and media libraries safe
• Restart Policy → Auto-restart unless manually stopped

👉 Summary:
With the NVIDIA Container Toolkit installed and this Compose file, Jellyfin will run with GPU acceleration, in Host mode, with persistent storage. This gives you smooth transcoding, seamless device discovery, and a setup that behaves like a true home media appliance.

1-3. Docker Compose (Intel or AMD GPU)

If you’re running Intel iGPU (Quick Sync) or AMD Radeon, you can also take advantage of hardware acceleration in Jellyfin. The setup is similar to the Nvidia version, but instead of runtime: nvidia and NVIDIA-specific environment variables, you’ll pass through your GPU using /dev/dri.

services:
  jellyfin:
    image: jellyfin/jellyfin:latest
    container_name: jellyfin
    network_mode: "host"
    devices:
      - /dev/dri:/dev/dri   # Intel/AMD GPU passthrough
    environment:
      - TZ=Asia/Seoul   # adjust timezone
    volumes:
      - ./config:/config
      - ./cache:/cache
      - /mnt/media/movies:/media/movies
      - /mnt/media/tv:/media/tv
    restart: unless-stopped

🔹 Key Differences from Nvidia Setup

GPU Passthrough

devices:
  - /dev/dri:/dev/dri

• /dev/dri is the Direct Rendering Interface used by Intel iGPUs and AMD GPUs.
• This gives the container direct access to your GPU for hardware-accelerated video decoding/encoding.
• Works out of the box on most Linux kernels without needing extra runtime configs like Nvidia.

No NVIDIA-Specific Env Vars

• Since you’re not using the NVIDIA runtime, you don’t need:
  • runtime: nvidia
  • NVIDIA_VISIBLE_DEVICES
  • NVIDIA_DRIVER_CAPABILITIES
• Intel/AMD acceleration relies entirely on /dev/dri being available.

1-4. Notes for Intel & AMD Users

• Intel Quick Sync (QSV) → Supported in Jellyfin for H.264, HEVC, VP9, and AV1 depending on CPU generation.
• AMD VA-API → Works via /dev/dri as well, with hardware acceleration for H.264/HEVC/VP9.
• Make sure you install the proper drivers on your host system (Mesa for AMD, Intel Media Driver for Intel).
• In Jellyfin’s Admin Dashboard → Playback → Hardware Acceleration, you’ll see VAAPI as the option to enable.

👉 Summary:
The Intel/AMD Compose file is simpler: just mount /dev/dri into the container. No extra runtime or environment tweaks are required. With proper drivers installed on your host, Jellyfin will detect your GPU and use VAAPI (Video Acceleration API) for hardware transcoding.

1-5. Deploying Jellyfin

Once your docker-compose.yml is ready:

docker compose up -d

Jellyfin will start on port 8096 by default.

Step 2: Reverse Proxy with Nginx Proxy Manager (NPM)

Since Jellyfin is running in host mode, its web UI is exposed at:

http://<your-wsl-ip>:8096

Yes, you could open this port on your router and stream media from anywhere. But that’s not why we installed Nginx Proxy Manager (NPM). The whole point is to hide messy port numbers behind a clean, SSL-secured subdomain.

2-1. Create a Proxy Host for Jellyfin

1. Log into Nginx Proxy Manager.
2. Go to Hosts → Proxy Hosts → Add Proxy Host.
3. Fill in the details:
   • Domain Names: jellyfin.wody.duckdns.org (or your domain)
   • Forward Hostname/IP: your WSL LAN IP (e.g., 192.168.0.121)
   • Forward Port: 8096
   • Scheme: http
4. Under SSL:
   • Select your Let’s Encrypt certificate (DuckDNS or your personal domain).
   • Enable Force SSL and HTTP/2.
   • Keep HSTS off until you’re sure everything works.
5. In the Advanced tab:
   • ✅ Enable Websockets Support (checkbox).
   • Paste in the configuration below.

2-2. Advanced Configuration for Jellyfin

# --- Long-running streaming sessions ---
proxy_http_version 1.1;

# Preserve client information
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;

# Extend timeouts for movies, transcoding, or paused sessions
proxy_connect_timeout 60s;
proxy_send_timeout    3600s;
proxy_read_timeout    3600s;
send_timeout          3600s;
keepalive_timeout     120s;

# Disable buffering & caching for smoother playback
proxy_buffering off;
proxy_request_buffering off;
proxy_cache off;

# (Optional) Allow large uploads
# client_max_body_size 200m;

Why This Matters

• WebSocket Support: Enabled via the checkbox in NPM → ensures remote control, live progress, and Chromecast work.
• Timeouts: Prevents streams from cutting off during long movies or when pausing.
• Headers: Ensures Jellyfin sees the correct client IP and protocol.
• Buffering Off: Reduces playback latency and makes seeking snappier.

👉 From now on, Jellyfin is available at:

https://jellyfin.wody.duckdns.org — no more ugly port numbers.

Step 3: First-Time Setup

When you visit Jellyfin for the first time, you’ll be guided through:

• Creating an admin account (username + password)
• Adding your media libraries (Movies, TV, Music, etc.)
• Choosing preferred metadata sources (TMDB, TVDB, MusicBrainz, etc.)

Once finished, you’ll land on the Jellyfin dashboard and your content will start indexing.

3-1. Hardware Transcoding Setup

Now comes the fun part: enabling GPU transcoding.

Inside Jellyfin Web UI:

1. Go to Dashboard → Playback → Transcoding.
2. Enable your GPU type:
   • Nvidia NVENC
   • Intel Quick Sync
   • AMD VAAPI
3. Recommended options:
   • ✅ Enable hardware acceleration
   • ✅ Enable tone mapping (for HDR → SDR)
   • ✅ Allow hardware decoding for H.264, HEVC, VP9, AV1
   • ✅ Allow hardware encoding (H.264, HEVC, AV1 if supported)
4. Save and restart Jellyfin.

With this, your desktop GPU will handle the heavy lifting, making even 4K HDR transcodes smooth.

Final Thoughts

At this point, you’ve turned your WSL2-powered home server into a full media hub:

• Jellyfin serving your library
• Nginx Proxy Manager cleaning up access with SSL subdomains
• GPU-powered transcoding delivering smooth playback to any device

Unlike Plex or Emby, this setup is 100% free, flexible, and future-proof.

Your NAS may have struggled with transcoding, but your desktop PC is more than ready.

Next Up

In the next part of this series, we’ll dive into Nextcloud—an open-source alternative to Dropbox and Google Drive.

With Nextcloud, you can:

• 🔄 Sync files between your server and desktop just like Dropbox
• 🌍 Access documents anywhere through a secure web interface
• 👥 Collaborate with shared folders, calendars, and contacts
• 📱 Automatic mobile photo backup — upload pictures and videos from your phone (iOS/Android) directly to your server, just like Google Photos
• 🖼 Photos app to browse, organize, and share your media library

In short, Nextcloud turns your home server into a private cloud storage + photo backup hub that keeps you in full control of your data.

👉 Stay tuned for Part 7: Installing Nextcloud for Private Cloud and Photo Backup.