patrik 🐈

INTRODUCTION: WHAT IS LoRA EXTRACTION?Have you ever found an incredible fine-tuned model like Juggernaut Z and wished you could take just the essence of its style to apply to a faster model like Z-Image Turbo? This is exactly where LoRA Extraction using SVD (Singular Value Decomposition) comes into play.In simple terms, LoRA extraction works by calculating the mathematical difference between two full model checkpoints: LoRA (Difference) = Tuned Model (Juggernaut) minus Base Model (Z-Image Original)The computer compares both models layer by layer, isolates the unique style and details, and compresses that difference into a lightweight LoRA file of around 171 MB to 340 MB.THE GOOGLE COLAB PROBLEM: OUT OF MEMORY (OOM) AND CPU FREEZESMany creators run into issues when trying to perform LoRA extraction on the free tier of Google Colab. There are two main limitations:Out of Memory (OOM) Crashes: Standard extraction scripts load both full-size model checkpoints (about 11 GB each) into RAM at the same time. The free Colab RAM (typically 12-15 GB) fills up instantly, causing the session to crash.CPU Freezes on SVD Calculation: Performing a full SVD on massive weight matrices (such as the QKV layers which can be 11520 x 3840) on a free CPU is painfully slow. The extraction process will appear frozen or stuck for hours.SOLUTION: STREAMING WORKFLOW AND LOW-RANK SVDTo bypass these hardware limits, we optimized the extraction process using two key technical approaches:Layer-by-Layer Streaming Method Instead of loading 22 GB of model checkpoints into RAM, the script lazily opens the files using the safetensors library. It only reads one weight matrix of a single layer at any given moment, calculates the difference, and immediately purges it from memory using garbage collection and CUDA cache clearing before moving to the next layer. This keeps system memory consumption under 1 GB throughout the entire process.Utilizing Low-Rank SVD (torch.svd_lowrank) Instead of using standard SVD functions that calculate thousands of unused singular values, we switched to a Low-Rank / Randomized SVD approach. This function only focuses on computing singular values up to our target rank dimension (such as 32 or 96). It reduces the calculation time by over 95 percent, making the extraction process nearly instantaneous on a GPU and highly stable even on a CPU.UNDERSTANDING THE KEY PARAMETERS: RANK, ALPHA, AND CLAMPINGWhen experimenting with different extraction variants, three parameters play a vital role in your final image quality:The Scaling Ratio (Alpha / Rank) Inside your generator UI, the LoRA'sn strength is multiplied by this ratio: Scaling Factor = Alpha divided by Rank.32/16 Variant (Ratio 0.5): Delivers a highly stable, subtle style enhancement, minimizing the risk of color artifacts. 32/32 Variant (Ratio 1.0): Provides a more dominant, instantly visible style at the default slider weight of 1.0. 64/32 Variant (Ratio 0.5): Offers double the detail capacity for complex prompts with smooth, stable style transitions.Clamping (clamp_quantile) Calculating the difference between models can sometimes produce extreme outlier numbers (numerical spikes). These spikes can destabilize the model, causing NaN errors, broken visual artifacts, or solid black images. Clamping acts as a guardrail to cut off these spikes. Setting this to 0.98 or 0.99 helps ensure visual stability in your extracted LoRA.CONCLUSIONSVD extraction is an incredible method to streamline your AI assets. By combining data streaming and low-rank SVD algorithms, you can safely bypass Google Colab's strict hardware limitations. You can now easily pack the unique artistic traits of massive models into highly portable LoRAs, ready to enhance your future creative work!SCRIPThttps://rentry.org/vtqtso5f

🚀 Fix ModelScope Flux Klein LoRA Training for Tensor.Art UploadsIf you trained a Flux LoRA on ModelScope (Diffusers format) and are facing a "Deployment Error" or "Invalid Model Structure" on Tensor.Art, this script is the definitive fix.This script was built upon an initial convert.py provided by the Tensor.Art developer team. Their script provided the essential foundation for layer renaming and basic tensor merging logic, which allowed us to understand the core requirements of the platform.❓ The Problem & EvolutionWhile the initial developer script is excellent for standard model renaming, we discovered a specific challenge with LoRA adapters. ModelScope outputs split Q, K, and V tensors, but simply "stacking" them doesn't satisfy the Flux architecture's requirement for a 12288 output dimension in the Fused QKV layer.🛠️ The Solution: Block Diagonal MergingTo solve this, we evolved the initial logic. Through intensive "vibe coding" with Gemini and KIMI, we implemented a Block Diagonal Strategy. This method:Merges the split Q, K, and V adapters from the Diffusers format.Expands the Rank (e.g., from 64 to 192) using a sparse diagonal matrix.Preserves 100% of your training quality by mathematically isolating the weights, ensuring the output perfectly matches the required 12288 dimension.📝 How to use this script:Step 1: Install Dependencies & Conversion Script (Block Diagonal)Run this cell to set up the environment (Torch & Safetensors) & Conversion Script (Block Diagonal)Step 2: Clone RepositoryEnter your ModelScope repo_id (e.g., username/model_name). If your model is private, paste your access_token; otherwise, leave it blank. Run to download.Step 3: The "Block Diagonal" Fix & Metadata InjectionCopy the path of the downloaded model into input_file and set a path for the final_file. Enter your metadata. Run this cell to automatically perform the Matrix Merging and save the fixed file.Step 4: Download & UploadDownload the resulting file and upload it to Tensor.Art.🔗 Script Link: https://colab.research.google.com/github/sevunk/fixing_layer_name/blob/main/merge_qkv_flukklein_diff.ipynbSpecial thanks to the tensor.art developer, community and intensive Gemini and KIMI debugging for cracking the Flux matrix dimension requirements!Final Hope:Let's hope the developer of Tensor.Art update their backends to support Diffusers-style Flux LoRAs natively soon, so we don't have to manually convert them anymore! :D Until then, let this script do the heavy lifting for you.

Complete Tutorial: Scraping Image Captions from Tensor.ArtThe goal of this tutorial is to automatically grab all the captions from your image dataset on Tensor.Art and save them into individual .txt files for each image, ready to be used for LoRA training.This process is divided into two main parts:Part 1: Extracting all unique captions from the web page into a single text file using JavaScript.Part 2: Splitting that single text file into many separate .txt files using PowerShell.Part 1: Extracting All Captions from the WebsiteIn this section, we will copy all unique captions from the web page to your clipboard.Step 1: Prepare the Web PageOpen your Chrome browser and navigate to your Tensor.Art dataset page containing the images.CRUCIAL STEP: Slowly scroll down the page until ALL of the images in your dataset (e.g., all 63 images) have appeared and loaded on the screen. If you don't do this, the script will only capture captions from the visible images.Step 2: Open the Developer Tools ConsoleOnce all images are loaded, press the F12 key on your keyboard to open the Developer Tools.In the Developer Tools window that appears, click on the "Console" tab.Step 3: Run the JavaScript ScriptCopy the entire code block below:// 1. Grab ALL <p> elements inside the caption divs. const allCaptionPTags = document.querySelectorAll('.train-model-assets-image-tags p'); // 2. Create an empty array to hold the texts. let duplicatedCaptionsList = []; // 3. Loop through each element, CLEAN the text, then add it to the list. allCaptionPTags.forEach(pTag => { // Get the raw text const rawText = pTag.innerText; // CLEAN THE TEXT: Replace all sequences of whitespace with a single space, // and then remove leading/trailing spaces. const cleanedText = rawText.replace(/\s+/g, ' ').trim(); // Push the cleaned text into the list. duplicatedCaptionsList.push(cleanedText); }); // 4. Create a 'Set' from the list of cleaned text to automatically remove duplicates. const uniqueCaptions = [...new Set(duplicatedCaptionsList)]; // 5. Join the unique captions into one large text block, separated by new lines. const finalText = uniqueCaptions.join('\n'); // 6. Copy the result directly to the clipboard. copy(finalText); // 7. Display a confirmation message with the correct count. console.log(`Total cleanup successful! Exactly ${uniqueCaptions.length} unique captions have been copied to your clipboard.`);Return to the Console window in your browser, then paste the code.Press Enter.You will see a confirmation message in the console stating the number of unique captions that were successfully copied, for example: Total cleanup successful! Exactly 63 unique captions have been copied to your clipboard.Step 4: Save the Results to a Text FileCreate a new folder on your computer to store your dataset. For example: D:\LoraTraining.Open the Notepad application.Press Ctrl + V to paste all the copied captions.Click File > Save As....Navigate to the folder you just created (e.g., D:\LoraTraining).Save the file with the name e.g., caption.txt.You now have a single file containing all unique captions, each on a new line.Part 2: Splitting the caption.txt File into Individual FilesIn this section, we will use PowerShell (a built-in tool in Windows) to automatically create one .txt file for each line of text in caption.txt.Step 1: Open PowerShell in the Working FolderOpen the folder where you saved caption.txt (e.g., D:\LoraTraining).Inside the folder (not on a file), hold down the Shift key on your keyboard and right-click on an empty space.Select the "Open PowerShell window here" or "Open in Terminal" option from the context menu.Step 2: Run the PowerShell ScriptA blue (PowerShell) or black (Terminal) window will appear. Copy the entire code block below:# 1. Define the input file name and the output file format $inputFile = "caption.txt" # Customize with your file name. $outputPrefix = "image" # The result will be image_1.txt, image_2.txt, etc. # 2. Read all lines from the caption.txt file $captions = Get-Content $inputFile # 3. Create a counter $i = 1 # 4. Loop through each caption line foreach ($line in $captions) { # Make sure the line is not empty if ($line.Trim() -ne "") { # Create the new file name, e.g., image_1.txt $outputFile = "${outputPrefix}_${i}.txt" # Write the line's content to the new file Set-Content -Path $outputFile -Value $line # Increment the counter $i++ } } # 5. Display a completion message Write-Host "Done! Successfully created $($i-1) .txt files." Paste the code into the PowerShell window.Press Enter.Step 3: Verify the ResultInstantly, your D:\LoraTraining folder will be populated with many new files: image_1.txt, image_2.txt, image_3.txt, ..., all the way to image_63.txt. Each of these files contains its corresponding single-line caption.