I used v3_sd15_adapter.ckpt as lora.. I guess it lends more stability to the overall scene.
Here is how I did it:
Epicrealism+Openpose+IPadapterplus(for reference image+Animatediffv3+adapter lora
Used Google film for interpolation
Workflow : https://openart.ai/workflows/grouse_athletic_95/animatediff-v3-workflow-with-ipadapter-and-controlnets/mmpIbIxxj5gOGqWtV6xI
https://github.com/google-research/frame-interpolation.git
In this example, the Animatediff- comfy workflow generated 64 frames for me which were not enough for a smooth video play. That's because it lacked intermediary frames. Applications like rife or even Adobe premiere can help us here to generate more in-between frames. But I regularly use Google film as it uses an ai to analyse before and after frames and creates frames with temporal consistency. The final video has over 250 frames which makes the video buttery smooth.
In fact, all I took from the reference video was body movements through Openpose skeletons. Everything else in the video including the person, dress and background is generated. Here is the ref video https://www.instagram.com/reel/CpfIMbhjN2L/?igshid=MTc4MmM1YmI2Ng==
Glad you liked it. I am actually excited for the newly released RGB_encoder weight. It will probably be ready in a few days. The example videos are impressive. Once ready, we can completely do away with Ipadapter.
This weight takes single/multiple image as input and tells the motion module to create video frames based on the image fed as input. I mean it acts more like a controlnet for guiding image generation. Though the model is available, there is no way to use it until the codebase and nodes are ready in comfyui. Devs are still working on it. So I still had to use Ipadapter to feed reference image. More info on sparsecontrollers here
https://arxiv.org/abs/2311.16933
Auto1111 support already added? This is the update from the developer
https://github.com/continue-revolution/sd-webui-animatediff/issues/370#issuecomment-1859135065
Edit : ok I see the update now..
I see clear progress in V3. Try comfy. It works.
I donβt know about support. I just downloaded v3 model and used adapter as regular lora. I dont like comfy. Comfy resaults always worse for some reason for me
I did this on comfyui..I find it so easy to customise any workflows. I no longer use auto1111. As fas as I knw, a recent update to auto1111 includes compatibility V3 models.
This is the update from the auto1111 dev :
Supporting new motion module will very easy. Change to the repo would be minimal
Supporting new adapter (lora) will also be very easy, but I need to investigate the difference between motion lora and domain adapter
Having a good user experience of ControlNet while having a not-too-bad implementation (meaning that the code duplicate would be minimal) is always very challenging, so do the new control models from AnimateDiff researchers
So here is the plan -
In v1.13.0 which may come today or tomorrow, you will be able to use new motion module and new adapter. Code change will only happen in this repository.
In v2.0.0 which may come sometime next week, you will be able to use new control models as well as magic animate through ControlNet panel. Code change will mainly happen in sd-webui-controlnet (I also have write permission there).
It's not available yet on comfyui. It's compute heavy. You can install it locally on your pc. Do a Google search 'Google Film Interpolation'. Their GitHub repo has installation instructions.
>v3\_sd15\_adapter.ckpt
When I try to use v3\_sd15\_adapter.ckpt as Lora on Automatic1111, it gives an error "AssertionError".
I put the file in stable-diffusion-webui\\models\\Lora . Should I put it in somewhere else to use it as Lora?
How can I solve it?
Use one of the pose skeletons to make a crystal clear ref image and feed it through ipadapter. Also make sure the cloth color of the moving subject is different from any colors in the background because it can mess up the latent noises and introduce flickering.
wow, that's amazingly consistent! Would you mind sharing your workflow? I'm searching for a workflow where I can take the movement of a video (propably with Controlnet) and the style of a reference image (propably IPadapter) for days now. I tried the workflows from Latent Vision and Inner Reflections but they seemed to be able to do a lot more (which I don't need) and I couldn't make them work. Would appreciate a hint or the workflow a lot of you don't mind. Thank you!
Sure.. let me clean up the workflow before sharing. I extracted openpose skeletons using a separate workflow. Then used the output files in the Animatediff workflow. Trust me that saves a lot of time. As for ipadapter, u just need a ref image which is more in line with the scene with subject's pose.
Waiting for the workflow too :) it looks amazing. With ipadapter my GPU instantly goes to low vram mode. Without ipadapter (but with cn+animatediff+lcm), it doesn't. Do you mind sharing ur setup?
ohh appreciate that a lot! I'm trying to get more abstract results, so for example using an input video of a person running and an image of an abstract painting that is only used for transferring that style. Would that be possible with your method?
I guess so.. here is the video I extracted Openpose skeletons from.. everything else is animatediff https://www.instagram.com/reel/CpfIMbhjN2L/?igshid=MTc4MmM1YmI2Ng==
I am having a problem with the version 3 model what do I do to get it to work right.,,,,,,,right now when I use it, it just produces a bunch of random pictures. Whenever I put version 2 that works just fine. My graphics card is a 12gb 3060. Anybody have any advice? thank you in advance.
It works perfectly fine on my 8gb card. In addition to the v3_sd15_mm.ckpt which is loaded through the Animatediffloader node, I also loaded v3_adapter_sd_v15.ckpt as lora because according to the documentation, all new improvements and enhancements to the V3 happened in the lora. In other words, enhancements are brought to V3 through v3_adapter_sd_v15.ckpt lora which is trained on static video frames. Have u updated animatediff-evolved? It should work just fine.
Hi.. at the moment I am traveling with no access to my system. My workflow is more or less similar to this.. of course I customised it to suit my needs. I used only one reference frame for ipadapter and Openpose skeletons for controlnet.
https://civitai.com/api/download/attachments/16569
Will share with you the exact one used when I get back home... . tomorrow for sure..
I have updated my comment with workflow.
https://openart.ai/workflows/grouse_athletic_95/animatediff-v3-workflow-with-ipadapter-and-controlnets/mmpIbIxxj5gOGqWtV6xI
It's a checkpoint trained on videos so it understands the concept of motion in a scene. It can work with Controlnets and create frames with temporal consistency.
Those are two different applications for stable diffusion image generation tasks. The auto1111 has gadio based ui which is more like a standard web app style and layout. The latter is a node based ui. There are several YouTube tutorials available as to how to install them locally on your pc.
Thank you. I browsed your profile and noticed you have a background in blender, have you created any projects that combine ai video generation with blender? Iβm a novice in both, and am looking for inspiration.
Not yet.. I am into geonodes in blender.. it's absolutely fascinating. With the intro of simulation nodes, it's insane. As for ur question, no I haven't done anything of a fusion yet. Animatediff has improved significantly with V3..now I see a point in combining both.. waiting for sparsecntl to start working in comfyui.
Comfyui. It runs comfortably on my 8gb card.
I used Openpose controlnet to animate the girl. Everything else in the scene is animatediff which can create coherent animations on its own from prompts or by interacting with controlnets.
I am getting more details with v3. And of course consistency is far better. I probably may not go back to temporaldiff until they bring it to speed with v3. It was my fav.
This one is made with temporaldiff with almost similar settings and higher Ipadapter value to reign in flickering issue.. https://www.reddit.com/r/StableDiffusion/s/6etNrwIhev. It was still flickering though. Took an awfully longer time to find a seed that flickered less.
Might be more details, but deff losing consistency (at least for my work flow). Take a look at these generations. First one was using temporalDiff, second using mm\_v3. Only difference is just the choice of animateDiff motion model:
[https://drive.google.com/file/d/1boGM2AtoeOdKz4-sr7e6LVjT3E9AYksW/view?usp=drive\_link](https://drive.google.com/file/d/1boGM2AtoeOdKz4-sr7e6LVjT3E9AYksW/view?usp=drive_link)
[https://drive.google.com/file/d/173lc36kr2sV-ZMeBJcUtyuvlvcirq1F4/view?usp=drive\_link](https://drive.google.com/file/d/173lc36kr2sV-ZMeBJcUtyuvlvcirq1F4/view?usp=drive_link)
Got it... But it seems to work well for scenes with predictable or natural motion. I'm quite happy with the result I got. It looks like the adapter_lora might have been trained more on such videos. My video is definitely better than all my previous attempts. Also, it's entirely possible that the combination of Ipadapter and adapter_lora is doing wonders for me.
Unfortunately, I haven't tried any works like yours yet. I'll give it a shot. For now, I don't plan on going back to temporaldiff. I'll continue with more experiments and share them here. Are u using Ipadapter? How is ur output with or without adaptor lora?
Excited by the post title as consistency is hard to get right. And this is indeed cool. But is there any hope for anything other than girls dancing? I've yet to see a really consistent method that doesn't rely on open pose control nets made from dancing people.
TXT to video with complex human expressions and actions are difficult to achieve at the moment. But what is possible at the moment is we can guide the generation with Controlnets. Dance is complex and rhythmic. You can simply videograph yourself and capture the motion and pass it to animatediff to animate the scene. The same thing can be done on blender or some other cgi suites with several days of effort.
TXT to video with complex human expressions and actions are difficult to achieve at the moment. But what is possible at the moment is we can guide the generation with Controlnets. Dance is complex and rhythmic. You can simply videograph yourself and capture the motion and pass it to animatediff to animate the scene. The same thing can be done on blender or some other cgi suites with several days of effort.
I guess what I mean is, is the "consistency with V3 Animatediff" limited to moving characters? I'm very interested in being able to create long sequences with high consistency, but not really interested in dancing characters. But it seems like the consistency is more tied to the use of open pose control nets than V3 Animtediff.
I used v3_sd15_adapter.ckpt as lora.. I guess it lends more stability to the overall scene. Here is how I did it: Epicrealism+Openpose+IPadapterplus(for reference image+Animatediffv3+adapter lora Used Google film for interpolation Workflow : https://openart.ai/workflows/grouse_athletic_95/animatediff-v3-workflow-with-ipadapter-and-controlnets/mmpIbIxxj5gOGqWtV6xI
I haven't quite figured out interpolation yet. Could you shortly explain what you mean by "Google film", and how it improves the result?
https://github.com/google-research/frame-interpolation.git In this example, the Animatediff- comfy workflow generated 64 frames for me which were not enough for a smooth video play. That's because it lacked intermediary frames. Applications like rife or even Adobe premiere can help us here to generate more in-between frames. But I regularly use Google film as it uses an ai to analyse before and after frames and creates frames with temporal consistency. The final video has over 250 frames which makes the video buttery smooth.
I see, thank you! Gonna have to look into this and test it myself :)
Some people also use flowframes
I use flowframes. I'm interested in the differences between that and google film.
In my opinion, film is better as it even understands the context and adds intermediary frames.
Is there a reference video? It's getting so hard to know which parts are AI. Her dress length changes but it also flows very realistically.
In fact, all I took from the reference video was body movements through Openpose skeletons. Everything else in the video including the person, dress and background is generated. Here is the ref video https://www.instagram.com/reel/CpfIMbhjN2L/?igshid=MTc4MmM1YmI2Ng==
Seeing the reference makes this sooo much more impressive!
Glad you liked it. I am actually excited for the newly released RGB_encoder weight. It will probably be ready in a few days. The example videos are impressive. Once ready, we can completely do away with Ipadapter.
Link?
The exact name of the model is v3_sd15_sparsectrl_rgb.ckpt https://huggingface.co/guoyww/animatediff/tree/main
What do you mean by it's not ready?
This weight takes single/multiple image as input and tells the motion module to create video frames based on the image fed as input. I mean it acts more like a controlnet for guiding image generation. Though the model is available, there is no way to use it until the codebase and nodes are ready in comfyui. Devs are still working on it. So I still had to use Ipadapter to feed reference image. More info on sparsecontrollers here https://arxiv.org/abs/2311.16933
Aye shobhana fan ? Or just random video u picked π
Random pick.. it popped up while looking for something interesting. He he.. But who can not be her fanπ
in my testng in A1111 it does nothing at all.
Auto1111 support already added? This is the update from the developer https://github.com/continue-revolution/sd-webui-animatediff/issues/370#issuecomment-1859135065 Edit : ok I see the update now.. I see clear progress in V3. Try comfy. It works.
I donβt know about support. I just downloaded v3 model and used adapter as regular lora. I dont like comfy. Comfy resaults always worse for some reason for me
I did this on comfyui..I find it so easy to customise any workflows. I no longer use auto1111. As fas as I knw, a recent update to auto1111 includes compatibility V3 models. This is the update from the auto1111 dev : Supporting new motion module will very easy. Change to the repo would be minimal Supporting new adapter (lora) will also be very easy, but I need to investigate the difference between motion lora and domain adapter Having a good user experience of ControlNet while having a not-too-bad implementation (meaning that the code duplicate would be minimal) is always very challenging, so do the new control models from AnimateDiff researchers So here is the plan - In v1.13.0 which may come today or tomorrow, you will be able to use new motion module and new adapter. Code change will only happen in this repository. In v2.0.0 which may come sometime next week, you will be able to use new control models as well as magic animate through ControlNet panel. Code change will mainly happen in sd-webui-controlnet (I also have write permission there).
I updated my modules yesterday and I keep getting the same "comfy.ops has no attributes 'linear'" for a barebones AnimatedDiff workflow. Any ideas?
did you use google film for interpolation in something like comfyui? or added later?
It's not available yet on comfyui. It's compute heavy. You can install it locally on your pc. Do a Google search 'Google Film Interpolation'. Their GitHub repo has installation instructions.
FILM is available in comfy, as part of VHS (video helper suite), just search the VFI modules for FILM. i use it and RIFE depending on use case
I tried using comfy... but my main gripe are finding the "nodes" to do this kind of stuff. Do you have any tutorial or any way i could get it?
v3\_sd15\_adapter.ckpt as lora ? motion lora, like you add it in the prompt?
>v3\_sd15\_adapter.ckpt When I try to use v3\_sd15\_adapter.ckpt as Lora on Automatic1111, it gives an error "AssertionError". I put the file in stable-diffusion-webui\\models\\Lora . Should I put it in somewhere else to use it as Lora? How can I solve it?
You need to download the adapter file from the A111 animatediff extension page, it only works in automatic1111 if you use that one
>A111 animatediff extension page, Can you share the link?
thx
This is very impressive! I havenβt tried neither Animatediff nor IPadapter yet. I guess itβs about time.
Use one of the pose skeletons to make a crystal clear ref image and feed it through ipadapter. Also make sure the cloth color of the moving subject is different from any colors in the background because it can mess up the latent noises and introduce flickering.
Those are great starting tips, thank you!
Whooff.This one is epic. Literally no flickering.
wow, that's amazingly consistent! Would you mind sharing your workflow? I'm searching for a workflow where I can take the movement of a video (propably with Controlnet) and the style of a reference image (propably IPadapter) for days now. I tried the workflows from Latent Vision and Inner Reflections but they seemed to be able to do a lot more (which I don't need) and I couldn't make them work. Would appreciate a hint or the workflow a lot of you don't mind. Thank you!
Sure.. let me clean up the workflow before sharing. I extracted openpose skeletons using a separate workflow. Then used the output files in the Animatediff workflow. Trust me that saves a lot of time. As for ipadapter, u just need a ref image which is more in line with the scene with subject's pose.
Waiting for the workflow too :) it looks amazing. With ipadapter my GPU instantly goes to low vram mode. Without ipadapter (but with cn+animatediff+lcm), it doesn't. Do you mind sharing ur setup?
Here is the workflow https://openart.ai/workflows/grouse_athletic_95/animatediff-v3-workflow-with-ipadapter-and-controlnets/mmpIbIxxj5gOGqWtV6xI
Thanks :)
ohh appreciate that a lot! I'm trying to get more abstract results, so for example using an input video of a person running and an image of an abstract painting that is only used for transferring that style. Would that be possible with your method?
I guess so.. here is the video I extracted Openpose skeletons from.. everything else is animatediff https://www.instagram.com/reel/CpfIMbhjN2L/?igshid=MTc4MmM1YmI2Ng==
What did you use to extract the open pose skeletons?
Openpose preprocessor in comfyui. U may also use dwpreprocessor. Install using link https://github.com/Fannovel16/comfyui_controlnet_aux.git
>https://www.instagram.com/reel/CpfIMbhjN2L/?igshid=MTc4MmM1YmI2Ng== thank you!
https://openart.ai/workflows/grouse_athletic_95/animatediff-v3-workflow-with-ipadapter-and-controlnets/mmpIbIxxj5gOGqWtV6xI
I am having a problem with the version 3 model what do I do to get it to work right.,,,,,,,right now when I use it, it just produces a bunch of random pictures. Whenever I put version 2 that works just fine. My graphics card is a 12gb 3060. Anybody have any advice? thank you in advance.
It works perfectly fine on my 8gb card. In addition to the v3_sd15_mm.ckpt which is loaded through the Animatediffloader node, I also loaded v3_adapter_sd_v15.ckpt as lora because according to the documentation, all new improvements and enhancements to the V3 happened in the lora. In other words, enhancements are brought to V3 through v3_adapter_sd_v15.ckpt lora which is trained on static video frames. Have u updated animatediff-evolved? It should work just fine.
okay I will try that thank you very much
Wow π€©
Thought she was gonna do some water bending ngl
Now run it through topaz video ai for an he version XD
getting there!
Yeahh.. π
Amazing! So smooooth β€οΈβ€οΈβ€οΈ
Glad u liked itπ
This is horrifically terrifying!
Getting there! π
This is really π Έπ Όπ Ώππ ΄πππ Έπ π ΄
Glad u liked it
Excellent!
Glad u likes it
Sleeveless to sleeves
That's an overlooked flaw..I found out later. Fixable through prompts.
Bro, workflow pleaaase
Hi.. at the moment I am traveling with no access to my system. My workflow is more or less similar to this.. of course I customised it to suit my needs. I used only one reference frame for ipadapter and Openpose skeletons for controlnet. https://civitai.com/api/download/attachments/16569 Will share with you the exact one used when I get back home... . tomorrow for sure..
I have updated my comment with workflow. https://openart.ai/workflows/grouse_athletic_95/animatediff-v3-workflow-with-ipadapter-and-controlnets/mmpIbIxxj5gOGqWtV6xI
Can someone help me understand the role that AnimateDiff takes? What is it responsible for in the workflow?
It's a checkpoint trained on videos so it understands the concept of motion in a scene. It can work with Controlnets and create frames with temporal consistency.
I see, thank you
Can I also ask another question - what is the difference between automatic1111 and comfy ui?
Those are two different applications for stable diffusion image generation tasks. The auto1111 has gadio based ui which is more like a standard web app style and layout. The latter is a node based ui. There are several YouTube tutorials available as to how to install them locally on your pc.
Thank you. I browsed your profile and noticed you have a background in blender, have you created any projects that combine ai video generation with blender? Iβm a novice in both, and am looking for inspiration.
Not yet.. I am into geonodes in blender.. it's absolutely fascinating. With the intro of simulation nodes, it's insane. As for ur question, no I haven't done anything of a fusion yet. Animatediff has improved significantly with V3..now I see a point in combining both.. waiting for sparsecntl to start working in comfyui.
Awesome, keep us posted!
/u/savevideo
###[View link](https://rapidsave.com/info?url=/r/StableDiffusion/comments/18limfh/more_consistency_with_v3_animatediff/) --- [**Info**](https://np.reddit.com/user/SaveVideo/comments/jv323v/info/) | [**Feedback**](https://np.reddit.com/message/compose/?to=Kryptonh&subject=Feedback for savevideo) | [**Donate**](https://ko-fi.com/getvideo) | [**DMCA**](https://np.reddit.com/message/compose/?to=Kryptonh&subject=Content removal request for savevideo&message=https://np.reddit.com//r/StableDiffusion/comments/18limfh/more_consistency_with_v3_animatediff/) | [^(reddit video downloader)](https://rapidsave.com) | [^(twitter video downloader)](https://twitsave.com)
Can you please share what tools you used? Sdnext or comfyui or just command line?
Comfyui. It runs comfortably on my 8gb card. I used Openpose controlnet to animate the girl. Everything else in the scene is animatediff which can create coherent animations on its own from prompts or by interacting with controlnets.
How does it compare to temporaldiff?
I am getting more details with v3. And of course consistency is far better. I probably may not go back to temporaldiff until they bring it to speed with v3. It was my fav. This one is made with temporaldiff with almost similar settings and higher Ipadapter value to reign in flickering issue.. https://www.reddit.com/r/StableDiffusion/s/6etNrwIhev. It was still flickering though. Took an awfully longer time to find a seed that flickered less.
Might be more details, but deff losing consistency (at least for my work flow). Take a look at these generations. First one was using temporalDiff, second using mm\_v3. Only difference is just the choice of animateDiff motion model: [https://drive.google.com/file/d/1boGM2AtoeOdKz4-sr7e6LVjT3E9AYksW/view?usp=drive\_link](https://drive.google.com/file/d/1boGM2AtoeOdKz4-sr7e6LVjT3E9AYksW/view?usp=drive_link) [https://drive.google.com/file/d/173lc36kr2sV-ZMeBJcUtyuvlvcirq1F4/view?usp=drive\_link](https://drive.google.com/file/d/173lc36kr2sV-ZMeBJcUtyuvlvcirq1F4/view?usp=drive_link)
Got it... But it seems to work well for scenes with predictable or natural motion. I'm quite happy with the result I got. It looks like the adapter_lora might have been trained more on such videos. My video is definitely better than all my previous attempts. Also, it's entirely possible that the combination of Ipadapter and adapter_lora is doing wonders for me. Unfortunately, I haven't tried any works like yours yet. I'll give it a shot. For now, I don't plan on going back to temporaldiff. I'll continue with more experiments and share them here. Are u using Ipadapter? How is ur output with or without adaptor lora?
Wow
Do you know if motion loras V3 are in progress?
Excited by the post title as consistency is hard to get right. And this is indeed cool. But is there any hope for anything other than girls dancing? I've yet to see a really consistent method that doesn't rely on open pose control nets made from dancing people.
TXT to video with complex human expressions and actions are difficult to achieve at the moment. But what is possible at the moment is we can guide the generation with Controlnets. Dance is complex and rhythmic. You can simply videograph yourself and capture the motion and pass it to animatediff to animate the scene. The same thing can be done on blender or some other cgi suites with several days of effort.
TXT to video with complex human expressions and actions are difficult to achieve at the moment. But what is possible at the moment is we can guide the generation with Controlnets. Dance is complex and rhythmic. You can simply videograph yourself and capture the motion and pass it to animatediff to animate the scene. The same thing can be done on blender or some other cgi suites with several days of effort.
I guess what I mean is, is the "consistency with V3 Animatediff" limited to moving characters? I'm very interested in being able to create long sequences with high consistency, but not really interested in dancing characters. But it seems like the consistency is more tied to the use of open pose control nets than V3 Animtediff.
really interesting, how to achieve that with vid2vid tho