so vlc was glitchy af for me i couldn't do anything.
so after trying to miserably debug it, i thought "meh lemme just try to use it using Xwayland"
so i enabled the X flag for vlc and disabled the wayland flag
now everything works perfectly like magic. no glitchy videos anymore
vlc is glitchy? are you using wayland? use Xwayland
Message
I use Plasma on this machine. I have both Xorg and Wayland installed on this system and SDDM lets me select which I want on graphical login. I also experienced some unexpected video playback behavior when I switched from Xorg to Wayland. I also have vlc but my most of the time, I use smplayer (with mpv) as my go-to video player.
Under Xorg, my preferred video output (the --vo=X argument to mpv or mplayer) is "gpu-next." It gives good performance and behaves as expected.
Smplayer has an option (under the "Video" tab of General in the settings) called "Wayland support." This tries to prevent video playback from happening in a second window as it tends to do under Wayland. Apparently, the intended design of smplayer is that the mpv/etc video output stays in the one smplayer window -- that way, unless you use full-screen mode, all the usual GUI controls (stop, pause, reverse, fast-forward buttons) are right there and available, along with the usual progress bar for that video. Some VO options like dmabuf-wayland and wlshm will appear in a second window regardless of this setting, but it works for most.
If I disable that "Wayland support" setting, the gpu-next VO option opens a second window but otherwise performs as well as it did on Xorg. If I enable the "Wayland support" setting, everything stays in a single window but the performance with gpu-next is terrible. The frame rate is drastically reduced. What works well for me under Wayland is to use the old "xv" VO driver with "Wayland support" enabled. Based on the very low (single-digit) CPU usage as reported by htop, regardless of the codec or resolution of the videos played, I assume this is still using the GPU.
It sounds like you found something similar. My media-video/mpv package is compiled with the "wayland" USE flag. It's also got the "xv" USE flag, which, according to the output of "equery uses mpv", is described as: "Add in optional support for the Xvideo extension (an X API for video playback)."
Under Xorg, my preferred video output (the --vo=X argument to mpv or mplayer) is "gpu-next." It gives good performance and behaves as expected.
Smplayer has an option (under the "Video" tab of General in the settings) called "Wayland support." This tries to prevent video playback from happening in a second window as it tends to do under Wayland. Apparently, the intended design of smplayer is that the mpv/etc video output stays in the one smplayer window -- that way, unless you use full-screen mode, all the usual GUI controls (stop, pause, reverse, fast-forward buttons) are right there and available, along with the usual progress bar for that video. Some VO options like dmabuf-wayland and wlshm will appear in a second window regardless of this setting, but it works for most.
If I disable that "Wayland support" setting, the gpu-next VO option opens a second window but otherwise performs as well as it did on Xorg. If I enable the "Wayland support" setting, everything stays in a single window but the performance with gpu-next is terrible. The frame rate is drastically reduced. What works well for me under Wayland is to use the old "xv" VO driver with "Wayland support" enabled. Based on the very low (single-digit) CPU usage as reported by htop, regardless of the codec or resolution of the videos played, I assume this is still using the GPU.
It sounds like you found something similar. My media-video/mpv package is compiled with the "wayland" USE flag. It's also got the "xv" USE flag, which, according to the output of "equery uses mpv", is described as: "Add in optional support for the Xvideo extension (an X API for video playback)."