( 7 ) My Second Note on Experiences, About "2boys" ( 7 )

There are also a few other things I’ve noticed when it comes to tags.

In Illustrious 2.0, even elements in the image that weren’t explicitly tagged can still be picked up and fitted into the model.

And when you later prompt those untagged elements,

the AI will tend to generate something close to the corresponding part of the training image.

But figuring out which ones it learned that way?

That’s pretty much impossible.

Also, similar or conceptually related tags tend to affect each other.

This becomes even more obvious when training clothing-related features.

For example:

Say you only tag an outfit as “sneakers” during training—

then during generation, even if you use tags like “footwear” or “shoes,”

the model still ends up generating the “sneakers” from the training data.

To deal with the chaos that this kind of thing can cause in “2boys” generation,

I recommend looking up the exact clothing item’s tag on Danbooru,

and then assigning it a dedicated custom name—

like “someone blue t-shirt,”

instead of vague general terms like “blue shirt” or “short sleeves.”

The same thing I mentioned in the last paragraph also applies here.

When training a “1boy,” you can afford to describe clothing in more detail—

all the little design elements, accessories, etc.—

breaking them down into pieces helps ensure they don’t bleed into other outfits during generation.

But when training “2boys,”

detailed tagging of clothing tends to result in absolute chaos during generation.

Because some tags get “locked,”

and others remain “replaceable,”

and you still have no idea which are which.

So, if you want to make sure clothing appears on the correct boy in a 2boys LoRA,

you need to tag as little as possible.

Of course, the tradeoff is that when generating other outfits,

the original outfit’s features will often stick around.

Now, here’s another fun thing about clothes—

Illustrious 2.0 doesn’t just ignore what you tag—

sometimes it tries to “replace” similar-looking content in the image

based on what the tag says.

For instance:

if the image shows a “tank top”,

but the tag says “vest”,

then the model might literally rewrite the tank top into a vest in the generated image—

like conjuring up something totally different out of nowhere.

One reason this happens is that when using auto-tagging,

a single item might get labeled with several different but similar tags.

If you don’t manually check and clean these up,

this ambiguity will likely introduce chaos later on during generation.

To avoid that, it’s still best to stick with Danbooru-standard tags when training on Illustrious 2.0.

Then again… you can also use this behavior on purpose.

Like a little... “concept swapping.”

Here’s a not-so-safe-for-Tensor example:

you replace “pexxs” with “small pexxs” in your training data.

The AI learns that “normal” = “small.”

Then, when you generate with just the word “pexxs,”

the model ends up producing large ones,

as if it’s trying to fill in the gap left by “normal.”

Yes—they do actually come out bigger than if you had just labeled them as “pexxs” in the first place.

Sigh… you know what I mean.

Trying to avoid getting banned on Tensor here 😅

But!

I’m saying “but” again...

this kind of thing only works on certain tags,

so you can’t reliably use it to swap concepts.

There’s no way to know in advance which ones will work.

Also, regarding the whole “custom naming” trick I mentioned earlier—

it doesn’t always work either.

Some tags, even when you give them a unique name,

won’t actually be learned by the model.

But if you swap them out for a class token,

suddenly the AI gets it and learns it properly.

Again, you just can’t tell in advance which tags behave this way.

Sometimes a LoRA turns out looking off,

or is missing details,

or no matter how much you tweak it, it just doesn’t feel well-fitted.

It’s very likely the issues described in this section are to blame—

but you’ll have no idea which tags are actually causing the mess.

So if this kind of thing happens to you,

you’ve got two options:

either comb through your dataset carefully…

or just accept your fate and move on.

So, following my approach,

you’ll probably want to stick to characters with simpler designs.

I haven’t yet figured out a reliable method to maintain generation quality for more complex designs.

For now, the only option is to increase batch size and go for overfitting,

which helps improve things somewhat.

Let me repeat one more time:

As long as you have enough training images, most of these problems go away.

You don’t need to worry about all this or follow the methods in this article.

This whole workflow is only for situations where you’re short on data.

Basically, that’s the full extent of my approach and methodology for creating a “2boys LoRA”.

If the first result you train doesn’t come out well, you can use the flaws described earlier to roughly determine what’s missing.

In general, you can improve it by retraining with adjustments like the ones below:

1、 Most importantly, increase the number of “2boys” images in your training set.

For example, if each single character has 20 images, but your “2boys” images are several times more than that—or even over 100—

then even if you throw them all into the same folder with the same repeat value,

you can still get pretty good results.

But if the situation is reversed, and “2boys” images are scarce,

that’s when the methods in this article actually start to become effective.

Here’s something extra I noticed during testing:

Putting different characters into separate folders always seems to give better results than putting them all into one folder.

I honestly don’t know why.

Also note: folder names can actually be used as trigger words.

After some A/B testing, I found that for “1boy” LoRA training,

you can use the folder name as the character’s trigger word,

and skip writing it in the tag file entirely.

But for “2boys” training, don’t do that.

Give the folder a name unrelated to any trigger words,

and whatever you do, don’t make the folder name overlap with any trigger words in the tags.

2、Adjust the repeat values.

Based on the results of your generations, either increase or decrease the steps for single or double characters.

3、Figure out which features can be locked-in using tags,

and try re-tagging and retraining.

4、Try switching to a different random seed.

Honestly, random seeds are part science, part superstition.

No matter how much you tweak steps 1–3,

you’ll often find that loss curves for the same dataset under the same seed look roughly the same.

So changing the seed might make it better—or worse. Who knows.

5、Try slightly tweaking the optimizer parameters,

or switch to a completely different optimizer and see what happens.

6、That’s as far as my personal methods go.

The rest is up to your own experiments.

One last thing to add is a special case: regularization.

While training single-character LoRAs,

I tried adding a large number of “2boys” images into the regularization set

and enabling regularization during training.

In theory, this kind of setup should reduce feature blending when combining LoRAs later on.

But whether it was due to the images I used, or the tags, or the parameters,

the results weren’t good—

so I abandoned that method.

Let me wrap things up with a little side note:

A friend of mine jokingly asked—

“Why are you going through all this effort to build an AI model?

Wouldn’t it be easier to just draw it yourself? Or commission an artist?”

Well... sure, if I could draw,

or if I were filthy rich,

then yeah, I wouldn’t even need to consider AI or commissions, would I?

But honestly, the rise of AI-generated images and the practical use of LoRA

have opened the door for people who can’t draw to still create doujin and fanart content.

And that’s kinda amazing.

Ok, that basically wraps up the discussion on “2boys LoRA”.