RuntimeError: CUDA error: device-side assert triggered #14236

Search before asking

YOLOv8 Component

Train

!yolo task=detect mode=train model=yolov8n.pt data=/content/Custom-Data-YOLOv8-Iranian-license-plates/Plate_Detection/custom.yaml epochs=25 imgsz=640

RuntimeError: CUDA error: device-side assert triggered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.

change code to:

import os
os.environ['CUDA_LAUNCH_BLOCKING'] = '1'

!export CUDA_LAUNCH_BLOCKING=1

RuntimeError: CUDA error: device-side assert triggered
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

Environment

Ultralytics YOLOv8.2.50 🚀 Python-3.10.12 torch-2.3.0+cu121 CUDA:0 (Tesla T4, 15102MiB)
Setup complete ✅ (2 CPUs, 12.7 GB RAM, 34.5/78.2 GB disk)

Minimal Reproducible Example

No response

Additional

No response

Are you willing to submit a PR?