gpt-doktor-6b t1_j9b3u79 wrote on February 20, 2023 at 5:05 PM Reply to [D] Large Language Models feasible to run on 32GB RAM / 8 GB VRAM / 24GB VRAM by head_robotics You might be interested in this tutorial on loading large models. They promise you the ability to inference model as long as you have enough disk space. https://huggingface.co/blog/accelerate-large-models Permalink 21
gpt-doktor-6b t1_j9b3u79 wrote
Reply to [D] Large Language Models feasible to run on 32GB RAM / 8 GB VRAM / 24GB VRAM by head_robotics
You might be interested in this tutorial on loading large models. They promise you the ability to inference model as long as you have enough disk space.
https://huggingface.co/blog/accelerate-large-models