REPRINT
Kohaku-XL Zeta
DiT is not all you need
join us: https://discord.gg/tPBsKDyRR5
Highlights
Resume from Kohaku-XL-Epsilon rev2
More stable, long/detailed prompt is not a requirement now.
Better fidelity on style and character, support more style.
CCIP metric surpass Sanae XL anime. have over 2200 character with CCIP score > 0.9 in 3700 character set.
Trained on both danbooru tags and natural language, better ability on nl caption.
Trained on combined dataset, not only danbooru
danbooru (7.6M images, last id 7832883, 2024/07/10)
pixiv (filtered from 2.6M special set, will release the url set)
pvc figure (around 30k images, internal source)
realbooru (around 90k images, for regularization)
8.46M images in total
Since the model is trained on both kind of caption, the ctx length limit is extended to 300.
Usage (PLEASE READ THIS SECTION)
Recommended Generation Settings
resolution: 1024x1024 or similar pixel count
cfg scale: 3.5~6.5
sampler/scheduler:
Euler (A) / any scheduler
DPM++ series / exponential scheduler
for other sampler, I personally recommend exponential scheduler.
step: 12~50
Prompt Gen
DTG series prompt gen can still be used on KXL zeta. A brand new prompt gen for cooperating both tag and nl caption is under developing.
Prompt Format
As same as Kohaku XL Epsilon or Delta, but you can replace "general tags" with "natural language caption". You can also put both together.
Special Tags
Quality tags: masterpiece, best quality, great quality, good quality, normal quality, low quality, worst quality
Rating tags: safe, sensitive, nsfw, explicit
Date tags: newest, recent, mid, early, old
Rating tags
General: safe
Sensitive: sensitive
Questionable: nsfw
Explicit: nsfw, explicit
Dataset
For better ability on some certain concepts, I use full danbooru dataset instead of filterd one. Than use crawled Pixiv dataset (from 3~5 tag with popularity sort) as addon dataset. Since Pixiv's search system only allow 5000 page per tag so there is not much meaningful image, and some of them are duplicated with danbooru set(but since I want to reinforce these concept I directly ignore the duplication)
As same as kxl eps rev2, I add realbooru and pvc figure images for more flexibility on concept/style.
Training
Hardware: Quad RTX 3090s
Dataset
Num Images: 8,468,798
Resolution: 1024x1024
Min Bucket Resolution: 256
Max Bucket Resolution: 4096
Caption Tag Dropout: 0.2
Caption Group Dropout: 0.2 (for dropping tag/nl caption entirely)
Training
Batch Size: 4
Grad Accumulation Step: 32
Equivalent Batch Size: 512
Total Epoch: 1
Total Steps: 16548
Training Time: 430 hours (wall time)
Mixed Precision: FP16
Optimizer
Optimizer: Lion8bit
Learning Rate: 1e-5 for UNet / TE training disabled
LR Scheduler: Constant (with warmup)
Warmup Steps: 100
Weight Decay: 0.1
Betas: 0.9, 0.95
Diffusion
Min SNR Gamma: 5
Debiased Estimation Loss: Enabled
IP Noise Gamma: 0.05
Why do you still use SDXL but not any Brand New DiT-Based Models?
Unless any one give me reasonable compute resources or any team release efficient enough DiT or I will not train any DiT-based anime base model. But if you give me 8xH100 for an year, I can even train lot of DiT from scratch (If you want)
License
Fair-AI-public-1.0-sd
