Dear Friends,
1. Introduction
As the 3rd article of my LoRA making series, I want to introduce the subject "multiple LoRAs into One LoRA". There are two approaches here: 1. Merging the existing LoRAs into one. 2. Training a LoRA with multiple sub-LoRAs from scratch. We cannot do #1 on TA trainer. So I focus on #2.
Assume that you want to make 3 girl LoRAs and 1 Combined LoRA from the 3 girl LoRAs. Then, you can make a structure as follows:
ComboChar/
├── character_A/
│ ├── trigger word: "charA"
│ └── images: 30
├── character_B/
│ ├── trigger word: "charB"
│ └── images: 40
├── character_C/
│ ├── trigger word: "charC"
│ └── images: 35
2. Trigger Words order
ComboChar is the trigger word for the combined LoRA. Each sub LoRA has its own trigger word like charA, charB and charC. You prepare for the image sets of 30 + 40 +35 =105 and train them all together. You need to put the trigger words ComboChar at the beginning of each image's caption and then the trigger word of the sub LoRA next to it. So each image has captions staring with two trigger words now.
During the training, make sure that Shuffle captions = True and Keep N tokens = 2. Then during the training, the first two trigger words are not used to train the corresponding image. They are used for the generation of related LoRAs during image generation after LoRA creation is done later.
Of course, you can add additional phrases related to each sub-LoRA at the end of the caption of each image as you did before. They are useful to guide the training process and image generation for later. The order of your captions are randomly shuffled during training if you set Shuffle cations = True, which is recommended.
3. Training and Showcases
Unfortunately, during the LoRA training process, you can generate sample images for only one of the LoRA types you are training. For the complete test, you'll have to generate images for the candidate epochs after the training is over. Then, make adjustments by re-training if needed.
I created some LoRAs as Multi-LoRA before. They are RPG Booster, Yosemite, Yellowstone, and Winter Resort for various base models like SDXL, SD3, Hunyuan and Flux. I want to show two great examples of Multi-LoRA created by Art-Intelligent too. You can go to see the project pages for more details about how the trigger words are used in the prompt to trigger the sub-LoRA you want each time.
That's it. I wonder if you want to create a Multi-LoRA or not. If you want to create one, good luck and have fun~ 🤗😉
https://tensor.art/models/763237064822154516/EK-RPG-Booster-Flux-1.0
https://tensor.art/models/693818522029238092/Yellowstone-XL-1.0
https://tensor.art/models/692404635975183575/Yosemite-XL-2.0
https://tensor.art/models/691867885322291970/Winter-Resort-XL-1.0
https://tensor.art/models/690313480823300547/AI-Italian-Medieval-Architecture-XL-v1.0
https://tensor.art/models/796784405830987317/AI-TRENDY-ART-MAKER-FLUX
