Since nobody answers, I'll post my half solution here, but before that, I'd like to ask you in foreword: please if you find a better solution then mine, don't forget to post it, because I'm having a hard time figuring out this topic too.
First of all, I got to ask if you enabled the 64 bit on raspbian.
If not, it can be done by
sudo nano /boot/config.txt
Go to the very end of the file and add this line:
arm_64bit=1
then reboot.
You can find more information about what you just did here. Maybe it's not related but it's worth a try and might solve your issue.
If you're still stuck, here it goes how I solved my similar issue on my raspi4, but I'd be happier if I could solve it more straightforward:
I had virtual machines (I could even run three of them, although didn't run anything on them. The biggest load I think was when I had two machines, one with a minecraft server running and the other some small tasks like bittorrent, a webserver and maybe some other small things.) on a headless rpi4 with 4 gigs of ram and a minimal (xorg) 64bit ubuntu on it (the one which the rpi-imager also includes among the "other OS" section). So I can tell that probably it's possible what you're trying to do. I didn't dare to put graphical desktops neither on the host or the client. What I did was installing cockpit on it by
sudo apt-get install cockpit
It uses kvm, although I didn't manage to figure out what commands this interface executes.
point is, that with cockpit you get a web interface to your server where you can install VM-s. If I remember well it automatically started the webserver which could be reached on the 9090 port through a webbrowser.
More about cockpit here
The setup for installation is quite comfortable (automatically grabs the selected image and basically just point and click), the catch though is that at the installation process the serial console is quite buggy I did debian installs, when the TUI fell apart or seemed to be not respondend, I moved around with the arrow keys and blindly hit enter. It's not too bad though, 90% of the time TUI stayed together. You also gotta be patient, I remember the first time I did it, I almost gave up because it takes for a while for the console to wake up. So wait, and if you got unpatient play around with the arrow key, that might bring the console to life.