I could bet it’s not about memory assigned. Because I thought so too and gave it over 1,5GB, but that didn’t resolve it. The only thing that helped was (and still is) to shutdown and start sys-usb again (not only to restart in my case).
Which makes me think it’s some kind of memory leak, actually. The same happens when I switch keyboard layout, but this time to whole Qubes.