FLUX – an intro
Last update: August 31, 2024 (guide updated to v.3.3)
FLUX came out on August the 1st, it was unexpected.
It was incredible. Everybody was on it, testing, writing about it, trying to understand how it works and how to make it more usable like SD 1.5 or SDXL.
At the beginning it seemed like it was impossible to have LoRA’s, Controlnets and make it lighter to run on older (or smaller) computers.
Then someone decided to try…
“Everyone knew it was impossible, until a fool who didn’t know came along and did it.” — Albert Einstein.
So, LoRA’s arrived, Controlnets too and also small and light checkpoints, in place of the 24Gb Unet files, that could be run on pc that did not have a Rtx 4090 with 64Gb Ram on it. Flux was available to everyone, and you could do anything with it!
I never used ComfyUI before, but on the 1st of August that was the only way to try FLUX. I had to install Comfy and started to learn how to use it. Now I would never go back!
I started to create a nice and easy workflow for FLUX, and then I added little things like LoRA’s manager, img2img prompt generator, upscaler, facedetailer… in few days it was a huge workflow. This post wants to be a short guide to the last version (v.3.0) of my workflow. At the end of this post you can find what files you need to run this workflow and the links for downloading them.
Tenofas FLUX workflow v.3.3
My complete ComfyUI workflow looks like this:
You have several groups of nodes, with different colors that indicate different activities in the workflow. I will go into details later on. For now I want to give a quick overview of the workflow. The yellow nodes at the bottom left are just instructions and links to the files you need to run the workflow. Then you have the blue grup, this is the core group of the workflow, where FLUX is loaded and you can set up its parameters. Following it, there is a red group, these are the switches and selectors for the various groups: here you will choose what kind of prompt you want to use (will write more on this later) and if you want to activate the Facedetailer (if you are not generating portraits of human beings you can just turn this off as it won’t work and it will waste time and memory) and the Upscaler (same thing: if you are not planning to Upscale your images keep this turned off). Next to it there is a gray-blue group with a strange name: “Latent space magic dimension” this is where all the magic of AI image generation takes place. On the top of the workflow, there are three green groups. These are the prompts options: the first is the classic txt2img prompt, just write your prompt and chose Input 1 in the red selector; the second is a img2img prompt generator that use Florence 2 model to convert the uploaded image to a text prompt (Input 2 on the prompt selector); last one is the LLM prompt generator, just write a short instruction or just a few keywords, and the LLM model will generate a colloquial-english prompt (Input 3 on the prompt selector).
In the middle, at the bottom, of the workflow there is a large group of nodes, this is the FaceDetailer. It uses SD1.5 (checkpoint and lora) to enhance the skin details of the image. If you are generating a portrait of a human being it will recognize where the face is in the image and apply the Lora skin detailer. If the image will not have any human face, just turn this off.
Then we have a large yellow node with a high res image, this node will allow you to compare (with a slider) the original FLUX generated image (even with LoRA’s) with the one that has been modified by FaceDetailer. It’s very useful if you are looking for the most realistic portrait output.
Last group, the large on the right, is the Ultimate SD upscaler. This is a very precise upscaler, and it can be quite slow as it works in many upscaling passes to mantain and enhance all the details of the image. It’s the best upscaler around, on my opinion.
Let’s see each part of the workflow in more detail.
The FLUX model loader and settings
My workflow uses the original model files released on August 1st by Black Forest Lab (the developer of FLUX), I use the Dev version, but you could also run the Schnell one. This group is the core of the workflow for generating FLUX images.
You can set the image size in the node “Image size and batch size”, the sizes are preset on the SDXL standard for better results. You can set the Flux Guidance (usually set at 3.5), the Sampler/Scheduler (there are a few suggestion you can try), the Steps and the Seed (it’s set to Random seed, but you can select Fixed if you want to). In the middle, at the top of the group, you can select the LoRA’s you want to use, set each LoRA’s strength, and turn on/off each one.
Warning: the first time you start a generation the workflow needs to load the Unet and Clip files, so it will take a few minutes “stuck” at the first nodes. The speed depends on which Unet weight and Clip you are using and on the Vram/Ram your computer is running.
Available Prompts
This workflow allows you to use different prompting methods. The first is the classic txt2img prompt: you just write a description of what your image should look like, and generate the image. Remember to use descriptive text, as FLUX clips understand very well human language and will give you better results. You can also try to write prompts in other languages, for example italian works fine. To select this prompt method you will have to set “Input 1” in the red Prompt Seletor.
The second method is a img2img prompt generator using Florence 2 model. Just upload an image and the model will generate a text describing in detail the image you uploaded. This method can be used by choosing Input 2 in the Prompt Selector. There is a “Text Find and Replace” node that allow you to modify the output text as Florence 2 model will start describing the image with “The image is a…”, you can change the “find” and “replace” string as you like. Leave “more_detailed_caption” for a verbose prompt output.
The third method (for those of you who can’t write in good english or are just lazy) uses a LLM model (Groq is suggested as it’s free at the moment, but you could also use OpenAI). You will need an API key (free) and it has to be saved in Comfy by using a specific node (TaraApiKeySaver) that you can remove once you save the key, just open the node, insert the key, lauch a generation and you are done. You must write your instructions in the bottom-left node, it can be a brief description of the image you want or just a few keywords. You can try to change the LLM settings (temperature can be set between 0.0 and 2.0, never above 2.0 or you will get errors). To use this prompt method you need to set the selector to Input 3. More about these LLM nodes on Github: Tara – ComfyUI Node for LLM Integration.
My suggestion for LLM settings: use mixtral-8x7b model on Groq, temperature 1.0, max tokens 300. In my tests these gave the best results.
The fourth method is just for batch-generation. You just need a .txt file with all the prompts you want to generate, load it on this module and start generating. It will automatically create one image for each prompt you added in the .txt file. So if you want to create a batch of one hundred image, just write 100 prompts in a simple .txt file, load it and click on Queue. Now you can go do something else, while ComfyUI will generate all 100 images. The .txt file needs to have the prompts in this format:
positive: xxxx xxxx xxxx xxx xxxx
negative: (leave empty, this workflow does not use negative prompt)
——–
positive: xxxx xxxx xxxx xxx xxxx
negative: (leave empty)
——–
…and so on.
Workflow Control Center
In the bright red group you have the Workflow Control Center, where you can choose which one of the three prompt system you want to work with and the On/Off switches for the img2img and LLM prompt generators, the Face Detailer and the Ultimate SD Upscaler. So, if you turn all the switches off you will have a plain and simple FLUX model with LoRA’s txt2img generator.
Face Detailer
This is the most complex group. It check the image FLUX generated, recognize the face in it and applies more skin details following the insctructions in the positive and negative prompts (green and red nodes). It uses a SD 1.5 checkpoint and LoRA that have proven to be among the best for photo realism.
Compare the FLUX and FaceDetailer outputs
If FaceDetailer was active and the image has a face in it, you can compare the original FLUX output with the image that was modified by FaceDetailer. Just move the mouse over the image, and a slider (left to right) will appear, showing you all the differences between the two images. Remember, for FaceDetailer to work, you need an image with a human face. The above bulldog did not activate the FaceDetailer… but he is still soo cute!!
The Upscaler
The Ultimate SD upscaler comes in two steps. First you need to preview the images FLUX and FaceDetailer generated, the workflow will pause here, waiting for you instructions: 1. cancel the queue, 2. select 1 or 2 images and continue to the upscaler. You can send one or both the images to the upscaler. Once you send one or two images and press on “Continue” you will start the Upscale process
Here you can set the parameters for the Upscale. You can play with the settings, but beware, it could make the upscale process extremely long. Upscale by 2 means the image will double width-height sizes, so a 1024×1024 image will be upscaled to 2048×2048. You can change the Steps, the tile width/height (but it’s better to leave it close to the original image size) and the other settings if you want.
The FaceSwap module
This module uses ReActor nodes to manage a simple faceswap. You just upload an image (a high res portait) of the source face you want to swap in the Flux generated image, and start the generation. You can compare the original image with the resulting image with a slider.
The Apply LUT module
You can use LUT (Look UP Tables) to give your image an analogic-film look. Just downlad the LUT you need (here are some free good ones). The LUT files must be saved in the following folder: ComfyUI/custom_nodes/ComfyUI_essentials/luts/
Image Overaly Text module
This is mostly useful for testing and comparing results. This module will just add a simple text over the image you generate (you can modify text settings as you wish) this way, for example, you can keep track of what settings you used to generate the image.
Where to download the workflow from
I have my workflow available for download from two websites:
1. CIVITAI (you need to be logged to Civitai to download it)
2. OpenArt.ai
Model, Lora and other files you will need
This workflow was designed around the original FLUX model released by the Black Forest Lab team. You will need the following files to use the workflow:
1) UNET – Dev or Schnell version (each one is around 24Gb, Dev gives better results, Schnell is the “turbo” version). These files must be saved in /models/unet/ folder.
Dev download Link: https://huggingface.co/black-forest-labs/FLUX.1-dev/resolve/main/flux1-dev.safetensors?download=true
Schnell download Link:
https://huggingface.co/black-forest-labs/FLUX.1-schnell/resolve/main/flux1-schnell.safetensors?download=true
2) Clip – you need these clip files (use fp16 for better results, fp8 if you have low Vram/Ram). These files must be saved in /models/clip/ folder.
3) VAE – last but not least the VAE file (must be saved in /models/vae/ folder):
https://huggingface.co/black-forest-labs/FLUX.1-dev/resolve/main/ae.safetensors?download=true
FaceDetailer uses a SD 1.5 checkpoint and LoRA to get more realistic results, as FLUX’s LoRA’s are still just a few and not perfectly trained yet.
These are the files you need for the mask detector:
1) Epicrealism SD 1.5 checkpoint (goes in folder /models/checkpoints/ )
https://huggingface.co/Justin-Choo/epiCRealism-Natural_Sin_RC1_VAE/resolve/main/epicrealism_naturalSinRC1VAE.safetensors?download=true
2)More details SD 1.5 LoRA (goes in folder /models/loras/ )
https://civitai.com/models/82098/add-more-details-detail-enhancer-tweaker-lora
3) sam_vit_b_01ec64.pth (goes in folder /models/ultralytics/bbox/ )
https://huggingface.co/datasets/Gourieff/ReActor/blob/main/models/sams/sam_vit_b_01ec64.pth
4) face_yolov8n_v2.pt (goes in folder /models/ultralytics/bbox/ )
https://huggingface.co/Bingsu/adetailer/blob/main/face_yolov8n_v2.pt
5) eyeful_v2-paired.pt (goes in folder /models/ultralytics/bbox/ )
https://civitai.com/models/178518/eyeful-or-robust-eye-detection-for-adetailer-comfyui
You can use different SD 1.5 or SDXL checkpoint/lora too if you want.
Last, but not least, for the Upscaler I use the 4x_NMKD-Siax_200k.pth model (goes in folder /models/upscale_models/) that you can download here: https://huggingface.co/gemasai/4x_NMKD-Siax_200k/tree/main or https://civitai.com/models/147641/nmkd-siax-cx
I hope you will enjoy this workflow. Leave a message if you have any question, request or hint. Thanks!
Leave a Reply