I will be presenting our work improving generative adversarial imitation learning (GAIL) by incorporating a diffusion model as a discriminator at the Generative Models for Decision Making workshop at #ICLR2024 . Stop by our poster at 3 PM @ Lehar 3!

🚀 Time to upgrade your LoRA for free! Introducing DoRA #ICML2024 ! Outperforming LoRA across various models like LLaMA & LLaVA. Give it a quick try: github.com/NVlabs/DoRA
Thanks to Sebastian Raschka for the sharing! 🙌
#AI #MachineLearning #NVIDIAAI #PEFT ICML Conference

You can now train a DoRA with the 🧨diffusers advanced dreambooth training script 🤗

Option ❶ follow this colab ⬇️
colab.research.google.com/drive/134mt7bC…

Option ❷ if you already have a training setup with the advanced script, run these 3 lines of code before launching your train🚀

Thanks Jeremy Howard for applying our DoRA ICML Conference to model compression
Impressive to see 'QDoRA + LLaMA2' is clearly better than 'QDoLA + LLaMA3'!!

Feel free to try our official DoRA code: github.com/NVlabs/DoRA
Stay tuned for more details!

#ICML2024 #icml #peft #lora #AI

Thanks merve and Linoy Tsaban🎗️ for trying our DoRA ICML Conference for diffusion models, generating better details aligned with text prompts!!

Feel free to try the official code: github.com/NVlabs/DoRA
Stay tuned for more details!

#ICML2024 #icml #peft #lora #AI #ML

Thanks Leshem Choshen @LREC 🤖🤗 for sharing our DoRA ICML Conference, which could be the default replacement for LoRA!!
[TL;DR] DoRA outperforms LoRA with various backbones (e.g., LLaMA 1, 2, 3 & LLaVA, etc.)
[Code] github.com/NVlabs/DoRA
Stay tuned for more details!
#ICML2024 #icml #peft #lora #AI

I've tried DoRA (arxiv.org/abs/2402.09353) with SDXL using PEFT, these are quite detailed 🤩🌟
as usual trained on lego dataset I compiled, compare with previous sota pivotal tuning results below

[ #ICML2024 ]
Not satisfied with existing PEFT methods?
Try our DoRA ICML Conference, which outperforms LoRA with various backbones (e.g., LLaMA 1, 2, 3 & LLaVA, etc.)
[Code] github.com/NVlabs/DoRA
Stay tuned for more details!
NVIDIA AI #icml #peft #lora #AI
Thanks Sebastian Raschka for sharing!

DoRA [ #ICML2024 ] has been reproduced by top researchers and could be the default replacement for LoRA! Give it a quick try: github.com/NVlabs/DoRA

A great intro! Adding a cherry on top of the cake - DoRA will appear at #ICML2024 - stay tuned on more details!

Is DoRA going to dethrone LoRA? 👑

I've been training SDXL DoRAs for the last couple of days - and while most examples aren't as dramatically better as this one (trained on our huggy mascot) - I'm still starting to think it might 👀

DoRA explores the magnitude and direction and
surpasses LoRA quite significantly

This is done with an empirical finding that I can't wrap my head around
NVIDIA AI
arxiv.org/abs/2402.09353
Shih-Yang Sean Liu Chien-Yi Wang @ ICLR 🇦🇹 Hongxu (Danny) Yin Pavlo Molchanov Min-Hung (Steve) Chen

While everyone is talking about Sora, there's a potential successor to LoRA (low-rank adaptation) called DoRA. Here's a closer look at the 'DoRA: Weight-Decomposed Low-Rank Adaptation' paper: arxiv.org/abs/2402.09353

LoRA is probably the most widely used parameter-efficient

FSDP/QDoRA is just as memory efficient and scalable as FSDP/QLoRA, and critically is *also* as accurate for continued pre-training as full weight training.

If you haven't read the DoRA paper yet, now's a good time! (It's a clear and well written paper.)
arxiv.org/abs/2402.09353