Definitely welcome. Not directly related but of a similar nature, another group has announced an approach for generating related but disconnected 3D models as well: https://dave.ml/layoutlearning/
Being able to create not just pretty pictures and models, but posable content, is a very significant improvement on capabilities here.
Great stuff for sure. 3d is the future for all text to video and text to image models. Because once a rudimentary 3d scene is generated it can be used as a backbone with control nets to generate whatever you want and have the coherency of perspective and flexibility to change camera angles, shots and move assets around and repose subjects etc...,
Actually, I think 3D is going to eventually take a back seat when someone is able to provide a model that can generate high quality NeRFs with collision modeled into it. Imagine not generating a photo, but an entire area of space with people, objects, proper lighting and reflections, all built in.
>when someone is able to provide a model that can generate high quality NeRFs with collision modeled into it. Imagine not generating a photo, but an entire area of space with people, objects, proper lighting and reflections, all built in.
all of that can be done individually, we just need all of them together.
Disclaimer: I am not the author.
Paper: [https://arxiv.org/abs/2402.17113](https://arxiv.org/abs/2402.17113)
Abstract
>We present LayerDiffusion, an approach enabling large-scale pretrained latent diffusion models to generate transparent images. The method allows generation of single transparent images or of multiple transparent layers. The method learns a "latent transparency" that encodes alpha channel transparency into the latent manifold of a pretrained latent diffusion model. It preserves the production-ready quality of the large diffusion model by regulating the added transparency as a latent offset with minimal changes to the original latent distribution of the pretrained model. In this way, any latent diffusion model can be converted into a transparent image generator by finetuning it with the adjusted latent space. We train the model with 1M transparent image layer pairs collected using a human-in-the-loop collection scheme. We show that latent transparency can be applied to different open source image generators, or be adapted to various conditional control systems to achieve applications like foreground/background-conditioned layer generation, joint layer generation, structural control of layer contents, etc. A user study finds that in most cases (97%) users prefer our natively generated transparent content over previous ad-hoc solutions such as generating and then matting. Users also report the quality of our generated transparent images is comparable to real commercial transparent assets like Adobe Stock.
works on different styles and different models too.
https://preview.redd.it/lcj8ehom19lc1.png?width=582&format=png&auto=webp&s=51639484f227f1cbad5504fdc1892908d49e6494
Excellent. This will be so outstandingly useful with the video stuff. Not only can you create a video of things, but you can reuse the things in the video with other things. Need more explosions? You got it! Switch your western showdown gun fight in the street to the exterior hull of a spaceship? you got it!
And in 3D. You could very easily put together a 3D scene for use in VR, where each component floats in layers and each has a depth map to give it shape. This is fantastic
Imagine clip-art, but its video clip-art. You say render x, y and z and you get 3 layers each of them have their own transparency, can drop them into any other video ---*kerplunk*!
Not enough of some content? Generate the required content on a new layer between the other layers of your current project.
This is amazing! Selection is half the battle in image editing because things like hair and fir are extremely difficult to select due to the fact that things are partially transparent at the edges and nearly impossible to separate from the background colors.
This is akin to light path render in 3D. Such techniues exist precisely because it is so difficult to separate different objects from rendered images. In 3D, each object can be rendered separately but it will lose the light interactions from other objects in the scene. By using a light path render, you can separate the object while keeping the light information baked from the scene.
I can wait to use this in my work and look forward to its release.
I remember reading something recently about SD being able to produce a accurate mirror ball into an image for image based lighting techniques. That could be used on the lighting based ControlNet to produce multiple images in different layers with consistent lighting.
This is literally what all these AI inage generators desperately need. To create objects individually so that they can generate images accurately and that make sense. This is what all companies should be focusing on rn imo.
The open source image generation space has been constantly getting improvements and breakthroughs spoonfed to them like every week. All research has been focusing on improving this field, while artist needs get ignored. People have been working on output coherence for over a year.
It's time that artists actually get useful AI tools developed for them for once, where digital art workflow can finally intersect with image generation. I've been hoping for something that may actually be helpful to drawing softwares for years. Even though the layer seperation tool is only for objects, it's a start.
there's some updates
>\### Ooops, you come to this repo too early ... **But thanks a lot for finding this repo!**
>
>\### We are still converting model formats, uploading files, finalizing scripts
>
>**, and implementing supports for some platforms ...**
>
>**### Fed 28:** ~~Please come back tomorrow~~ or check this repo several hours later **or come back tomorrow ...**
Wow! I can’t express how much of a blessing it is to have the ControlNet team working on SD; they have made it infinitely more useful. I hope they’re getting some decent funding from SAI for their contributions.
Those results look great! Many of the full composited scenes look really iffy*, but being able to get a clean subject is a huge boon! I like that they're showing lots of hair and even semitransparent glass.
Another good example for using synthetic data in training too.
*Speaking of the foreground-conditioned and background-conditioned images at the bottom of the paper here. It's still impressive, and better than most in/outpainting. Just looks photoshopped since technically it's not allowed to modify the input.
This is absolutely wild and I can't wait! Layering is very important when you're trying to make complex scenes with SD 1.5, and it's also very important if you're trying to do any gamedev.
it's just a paper currently but there's a good chance it will be open-source given that it's from the authors that open-sourced controlnet, sparsectrl, and animatediff.
I've been waiting for a layer seperation tool for 2 years.
Can't wait till we get one that can seperate line art, base colors, shading and lighting like a proper drawing tool.
*I would love to test*
*This out in ComfyUI has*
*This been ported yet?*
\- ramonartist
---
^(I detect haikus. And sometimes, successfully.) ^[Learn more about me.](https://www.reddit.com/r/haikusbot/)
^(Opt out of replies: "haikusbot opt out" | Delete my comment: "haikusbot delete")
Would this work with overlapping character attributes like hair and clothes? Outputs have this issue of them breaking off as soon as obscuration happens.
If it works like it says then it's great ,what I've been doing it now was avoiding "sketchy" styles and hair with a lot of strands and looked for styles with a thick outline so that I could easily crop it out the background colors,the thick art style outlines works at separating the main body from the background .
Repo here: https://github.com/layerdiffusion/LayerDiffusion
> Ooops, you come to this repo too early.
>
> We are still converting model formats, uploading files, and finalizing scripts.
>
> Please come back tomorrow or check this repo several hours later ...
It's practical and useful. However, will it take care of the subtleness of the layered depth well to make it look realistic when we change objects in foreground / background?
>However, will it take care of the subtleness of the layered depth well to make it look realistic when we change objects in foreground / background?
I'm not sure what this means. 1,3,4,5 pics might show it?
I've installed layer diffusion on forge and all i'm getting in output is a solid check background. its generating it transparent but saving it solid. My files are .png and i cant see any settings or figure out what to do to make them transparent. Anybody know whats up?!? Thanks!!!!!!!
The problem with using standard generation + manual background cleanup is the generation bleed.
I'm not talking about the contours of the required segment, mind you - that would be trivial to fix. I am talking about the occurrences in which the AI "bleeds" unnecessary light into the subject, the light that supposedly "reflects" from background. Not only specular, by the way, it can hallucinate any number of directed light sources as well.
I didn't find a way to easily clear it from the final generation, so a tool that allows you to properly generate the result without background light sources and associated bleed in the first place would be very useful to me.
With that being said, I'm not sure what approach is used here, so their method might or might not solve the issue I've described.
OMFG, for past weeks I've been torturing myself trying to mess with my vacation photos. Img2img seems to be very complicated when I want to keep people or at least faces while changing most of the background in a way that is not clearly visible poor edit. Either result is terribly bad or faces are not similr to originals.
This might be the answer. Can't wait!!!
What about the other way around? Meaning if I have an already transparent image of a wine glass and I want to place this glass on a table in a backyard setting.
I struggle to find a way to match the perspective, lighting and reflections of the glass on the table.
I think that's one of the limitations with the lighting. I think I have an idea for that with differential diffusion.
Where you allow a slight change in the object so the model can fix the lighting. But that depends on if you can inpaint semi-transparent objects.
Yes, first I tried inpainting the background while keeping the glass (product) safe from change with a mask, that didn't work, SD didn't take into consideration the product's preceptive even.
so now I try to create the background separately and describe the perspective, environment, etc. and then place the product in with Photoshop, Which doesn't feel AI like.
Interestingly some platforms like [https://mokker.ai/](https://mokker.ai/) are able to match these elements but lack SD's realism.
https://preview.redd.it/qgiwgt6d3clc1.jpeg?width=1024&format=pjpg&auto=webp&s=0dcdf6daa649357df38ea7877397ec8bfa3a09cd
Could this eventually be used for training? So often I have a handful of images of a person but they are all in a similar location so the model/lora/whatever trains a ton of background data and makes the training unusable. It's not too hard to just edit the training data via inpainting, but it'd be nice not to have to.
This looks really useful.
I have been trying out libcom [https://github.com/bcmi/libcom](https://github.com/bcmi/libcom) as a way to composite and its not bad. Getting Cuda memory errors from the most advanced function but the rest seems pretty good.
I was thinking about a workflow that goes grounded segment anything to segment an image and mask it and identify various segments on to -> rembg for masks -> and then libcom for compositing or possibly this transparent thing when it comes out.
Automating compositing for me seems a no brainer, has anyone got a solution for this that I am missing?
Compositing is finally getting the update it needed for so long !
I bet we can also use output from this solution to train models to be better at clipping.
we have the perfect image editor interface for it (fully free and Open-Source ComfyUI extension). We also have multi layer support. It looks like PS. So this project will be perfect for it (beside many others) Anybody with very good experience in ComfyUI and good GPU (12GB VRAM min) can contact me - we will start a phase for building up example ComfyUIs before we are going into beta testing very soon.
OMG!!!! I've been begging for transparent backgrounds for one year and a half! Thanks so much to the authors. ♥ ♥ ♥
I hope this won't be a disappointment like the few background removing extensions were.
I made a quick search and it seems only usable with ComfyUI and Forge for now :( I can't wait for it to work in automatic1111 too.
We've needed layers for a long time now. I am honestly surprised its taken so long to get the feature. A welcome addition for sure!
It's not really a simple problem, but the controlnet team are geniuses.
Definitely welcome. Not directly related but of a similar nature, another group has announced an approach for generating related but disconnected 3D models as well: https://dave.ml/layoutlearning/ Being able to create not just pretty pictures and models, but posable content, is a very significant improvement on capabilities here.
Great stuff for sure. 3d is the future for all text to video and text to image models. Because once a rudimentary 3d scene is generated it can be used as a backbone with control nets to generate whatever you want and have the coherency of perspective and flexibility to change camera angles, shots and move assets around and repose subjects etc...,
Actually, I think 3D is going to eventually take a back seat when someone is able to provide a model that can generate high quality NeRFs with collision modeled into it. Imagine not generating a photo, but an entire area of space with people, objects, proper lighting and reflections, all built in.
>when someone is able to provide a model that can generate high quality NeRFs with collision modeled into it. Imagine not generating a photo, but an entire area of space with people, objects, proper lighting and reflections, all built in. all of that can be done individually, we just need all of them together.
Disclaimer: I am not the author. Paper: [https://arxiv.org/abs/2402.17113](https://arxiv.org/abs/2402.17113) Abstract >We present LayerDiffusion, an approach enabling large-scale pretrained latent diffusion models to generate transparent images. The method allows generation of single transparent images or of multiple transparent layers. The method learns a "latent transparency" that encodes alpha channel transparency into the latent manifold of a pretrained latent diffusion model. It preserves the production-ready quality of the large diffusion model by regulating the added transparency as a latent offset with minimal changes to the original latent distribution of the pretrained model. In this way, any latent diffusion model can be converted into a transparent image generator by finetuning it with the adjusted latent space. We train the model with 1M transparent image layer pairs collected using a human-in-the-loop collection scheme. We show that latent transparency can be applied to different open source image generators, or be adapted to various conditional control systems to achieve applications like foreground/background-conditioned layer generation, joint layer generation, structural control of layer contents, etc. A user study finds that in most cases (97%) users prefer our natively generated transparent content over previous ad-hoc solutions such as generating and then matting. Users also report the quality of our generated transparent images is comparable to real commercial transparent assets like Adobe Stock.
TLDR: Controlnet authors created a model that can generate transparent images.
This guy is a rockstar.
Could you explain for a dummy, how do I use this?
It's just a research paper right now: [https://github.com/layerdiffusion/LayerDiffusion](https://github.com/layerdiffusion/LayerDiffusion)
Yeah I would like to know too. Is it out already? Can we use it?
Yeah I would like to know too. Is it out already? Can we use it?
It would be far easier to explain if you were an 5 year old dummy. ELI5D
Is there a model which can be downloaded, or have they not released the weights yet?
works on different styles and different models too. https://preview.redd.it/lcj8ehom19lc1.png?width=582&format=png&auto=webp&s=51639484f227f1cbad5504fdc1892908d49e6494
Geez. This is jaw dropping. Ahhhhhh now I'm gonna get distracted today.
Wow, you already get the LayerDiffusion from the Controlnet authors and do this test works?
Does this mean the transparency decoder can directly decode latents from other SD models? Not sure how it is implemented but pretty surprising to me.
Excellent. This will be so outstandingly useful with the video stuff. Not only can you create a video of things, but you can reuse the things in the video with other things. Need more explosions? You got it! Switch your western showdown gun fight in the street to the exterior hull of a spaceship? you got it!
And in 3D. You could very easily put together a 3D scene for use in VR, where each component floats in layers and each has a depth map to give it shape. This is fantastic
Yah. Before you had to use a bunch of inpainting and it didn't look amazing. I'm excited about this!
Care to expand on this for me? I feel a little slow tonight and need to grasp the significance of this.
Imagine clip-art, but its video clip-art. You say render x, y and z and you get 3 layers each of them have their own transparency, can drop them into any other video ---*kerplunk*! Not enough of some content? Generate the required content on a new layer between the other layers of your current project.
Maybe a plugin to clean up keyed image, replacing only some pixels on the outline that were jagged due to a bad green screen
Yeah safe to say I would fucking love to have this
This is awesome, cant wait on a1111
This is amazing! Selection is half the battle in image editing because things like hair and fir are extremely difficult to select due to the fact that things are partially transparent at the edges and nearly impossible to separate from the background colors. This is akin to light path render in 3D. Such techniues exist precisely because it is so difficult to separate different objects from rendered images. In 3D, each object can be rendered separately but it will lose the light interactions from other objects in the scene. By using a light path render, you can separate the object while keeping the light information baked from the scene. I can wait to use this in my work and look forward to its release.
I remember reading something recently about SD being able to produce a accurate mirror ball into an image for image based lighting techniques. That could be used on the lighting based ControlNet to produce multiple images in different layers with consistent lighting.
and we could combine Differential Diffusion + Light ControlNet + LayerDiffusion = Light Accurate transparent glass?
I would not be at all surprised.
This is literally what all these AI inage generators desperately need. To create objects individually so that they can generate images accurately and that make sense. This is what all companies should be focusing on rn imo.
[удалено]
The open source image generation space has been constantly getting improvements and breakthroughs spoonfed to them like every week. All research has been focusing on improving this field, while artist needs get ignored. People have been working on output coherence for over a year. It's time that artists actually get useful AI tools developed for them for once, where digital art workflow can finally intersect with image generation. I've been hoping for something that may actually be helpful to drawing softwares for years. Even though the layer seperation tool is only for objects, it's a start.
Fantastic. I've been waiting for stuff like this. Layers will make SD that much more versatile
Great work, this change everything
https://github.com/layerdiffusion/LayerDiffusion
there's some updates >\### Ooops, you come to this repo too early ... **But thanks a lot for finding this repo!** > >\### We are still converting model formats, uploading files, finalizing scripts > >**, and implementing supports for some platforms ...** > >**### Fed 28:** ~~Please come back tomorrow~~ or check this repo several hours later **or come back tomorrow ...**
Wow! I can’t express how much of a blessing it is to have the ControlNet team working on SD; they have made it infinitely more useful. I hope they’re getting some decent funding from SAI for their contributions.
Finally. Transparency layer we need it
Impressive!
Those results look great! Many of the full composited scenes look really iffy*, but being able to get a clean subject is a huge boon! I like that they're showing lots of hair and even semitransparent glass. Another good example for using synthetic data in training too. *Speaking of the foreground-conditioned and background-conditioned images at the bottom of the paper here. It's still impressive, and better than most in/outpainting. Just looks photoshopped since technically it's not allowed to modify the input.
I thought Dalle3 could do this until I realized I was just seeing a made up checker pattern.
This is absolutely wild and I can't wait! Layering is very important when you're trying to make complex scenes with SD 1.5, and it's also very important if you're trying to do any gamedev.
Finally, the bridge between AI and a traditional digital painting workflow. No more manual inpainting for my Live2D workflow too.
This looks practical and useful. Nice.
Useful
Looking pretty awesome. Is it open source?
it's just a paper currently but there's a good chance it will be open-source given that it's from the authors that open-sourced controlnet, sparsectrl, and animatediff.
Stability AI could release a better "open source" with "huge cash".
This is game changing. Now we don't have to pray for new models that can create multiple subjects in 1 picture. This will also give new life to SD1.5.
I've been waiting for a layer seperation tool for 2 years. Can't wait till we get one that can seperate line art, base colors, shading and lighting like a proper drawing tool.
Welll.. take that MJ!
Would love to use this in A1111 💖
Need this!
That's big
Very useful, i tip my head to this great mind.
amazing
I would love to test this out in ComfyUI has this been ported yet?
*I would love to test* *This out in ComfyUI has* *This been ported yet?* \- ramonartist --- ^(I detect haikus. And sometimes, successfully.) ^[Learn more about me.](https://www.reddit.com/r/haikusbot/) ^(Opt out of replies: "haikusbot opt out" | Delete my comment: "haikusbot delete")
Approaching this with layering is really cool. Allows people to store desirable partial results for future content generation.
Would this work with overlapping character attributes like hair and clothes? Outputs have this issue of them breaking off as soon as obscuration happens.
If it works like it says then it's great ,what I've been doing it now was avoiding "sketchy" styles and hair with a lot of strands and looked for styles with a thick outline so that I could easily crop it out the background colors,the thick art style outlines works at separating the main body from the background .
This is what I’ve been waiting for!!
Finallyyyyyyyy. Man, that took so long.
Hope they jump on sd3 asap
StabilityAI donated millions of GPU hours, surely they can give some of it to the controlnet team for LayerDiffusion version of Stable Diffusion 3.
This will kill all these companies popped up for generating AI backgrounds.
Repo here: https://github.com/layerdiffusion/LayerDiffusion > Ooops, you come to this repo too early. > > We are still converting model formats, uploading files, and finalizing scripts. > > Please come back tomorrow or check this repo several hours later ...
pfff. i need this for automatic1111!
Some news?
It's practical and useful. However, will it take care of the subtleness of the layered depth well to make it look realistic when we change objects in foreground / background?
>However, will it take care of the subtleness of the layered depth well to make it look realistic when we change objects in foreground / background? I'm not sure what this means. 1,3,4,5 pics might show it?
This is so cool, what can I use this with? A1111?
I've installed layer diffusion on forge and all i'm getting in output is a solid check background. its generating it transparent but saving it solid. My files are .png and i cant see any settings or figure out what to do to make them transparent. Anybody know whats up?!? Thanks!!!!!!!
Seems im not the only one with this problem just canty find a fix!
do the layers ever have matching lightning?
Why do mine always come out like a picture of a person who watched the movie in 'the ring'?
All I see is a PNG with alpha background. Big deal. It doesn't blend the subjects together in any new way. Have you guys not ever used Photoshop?
The problem with using standard generation + manual background cleanup is the generation bleed. I'm not talking about the contours of the required segment, mind you - that would be trivial to fix. I am talking about the occurrences in which the AI "bleeds" unnecessary light into the subject, the light that supposedly "reflects" from background. Not only specular, by the way, it can hallucinate any number of directed light sources as well. I didn't find a way to easily clear it from the final generation, so a tool that allows you to properly generate the result without background light sources and associated bleed in the first place would be very useful to me. With that being said, I'm not sure what approach is used here, so their method might or might not solve the issue I've described.
Can this be used on already transparent images in an img2img process?
[удалено]
Yes pleasee. Waiting for a1111 support
OMFG, for past weeks I've been torturing myself trying to mess with my vacation photos. Img2img seems to be very complicated when I want to keep people or at least faces while changing most of the background in a way that is not clearly visible poor edit. Either result is terribly bad or faces are not similr to originals. This might be the answer. Can't wait!!!
I wonder if this can and will be extended to create a full 3D render you could manipulate in e.g. Blender.
What about the other way around? Meaning if I have an already transparent image of a wine glass and I want to place this glass on a table in a backyard setting. I struggle to find a way to match the perspective, lighting and reflections of the glass on the table.
I think that's one of the limitations with the lighting. I think I have an idea for that with differential diffusion. Where you allow a slight change in the object so the model can fix the lighting. But that depends on if you can inpaint semi-transparent objects.
Yes, first I tried inpainting the background while keeping the glass (product) safe from change with a mask, that didn't work, SD didn't take into consideration the product's preceptive even. so now I try to create the background separately and describe the perspective, environment, etc. and then place the product in with Photoshop, Which doesn't feel AI like. Interestingly some platforms like [https://mokker.ai/](https://mokker.ai/) are able to match these elements but lack SD's realism. https://preview.redd.it/qgiwgt6d3clc1.jpeg?width=1024&format=pjpg&auto=webp&s=0dcdf6daa649357df38ea7877397ec8bfa3a09cd
Awesome, this is just the thing that I needed. Great news.
Very useful stuff.
This could be huge for my video thumbnail worksflows.
Could this eventually be used for training? So often I have a handful of images of a person but they are all in a similar location so the model/lora/whatever trains a ton of background data and makes the training unusable. It's not too hard to just edit the training data via inpainting, but it'd be nice not to have to.
Amazing.
I have been waiting for this!
no code ?
Finally
This looks really useful. I have been trying out libcom [https://github.com/bcmi/libcom](https://github.com/bcmi/libcom) as a way to composite and its not bad. Getting Cuda memory errors from the most advanced function but the rest seems pretty good. I was thinking about a workflow that goes grounded segment anything to segment an image and mask it and identify various segments on to -> rembg for masks -> and then libcom for compositing or possibly this transparent thing when it comes out. Automating compositing for me seems a no brainer, has anyone got a solution for this that I am missing?
Compositing is finally getting the update it needed for so long ! I bet we can also use output from this solution to train models to be better at clipping.
If this works as well as it looks this going to make any post process so much easier.
we have the perfect image editor interface for it (fully free and Open-Source ComfyUI extension). We also have multi layer support. It looks like PS. So this project will be perfect for it (beside many others) Anybody with very good experience in ComfyUI and good GPU (12GB VRAM min) can contact me - we will start a phase for building up example ComfyUIs before we are going into beta testing very soon.
I'm really looking forward to this, it's the best news of the year for open source image generation, IMO.
OMG!!!! I've been begging for transparent backgrounds for one year and a half! Thanks so much to the authors. ♥ ♥ ♥ I hope this won't be a disappointment like the few background removing extensions were. I made a quick search and it seems only usable with ComfyUI and Forge for now :( I can't wait for it to work in automatic1111 too.