RPi4B usb lag + jitter

[szatox]

I've been using RPi4B with on-board wifi and USB modem-pretending-to-be-ethernet as a router. It mostly works, but keeps corrupting the FS on pendrive hosting the system, and the internet, even though reasonably fast, is not very good for audio. Apparently, my voice is very choppy for the people on the other side, even though I can hear them just fine.
So I've ran some tests and noticed that the ping to the USB dongle itself is in range of 10-30ms when it's plugged into RPi and well under 1 ms when plugged into a PC.
It does add quite a lot to the lag, I've had pings as good as 30-40 ms with this dongle, and 20ms jitter certainly doesn't help.
I plug both, pendrive and the modem into USB3 ports.
The load is next to nothing, and the link is not overloaded, I've seen it handle way more traffic.

I wonder if anyone else notices similar issue with this (or similar) boards?
Now, this thing is not running gentoo yet, so technically not supported, but I still wonder if it's a configuration issue in raspbian or a hardware one. Or maybe if it's by design? I'm going to get back to some of my projects and I have to decide where to put my energy first... You know, It's nice to see some return on investment.