So I finally trained a LoRA that's actually remotely useful and somewhat OK for the most part! Nice!
That being said, this is a very unstable and experimental LoRA so not everything will work out of the box unfortunately. But I tried my best at least.