Tensor (aka model parallel) parallel with model checkpointing works better than FSDP (though they can be used in conjunction) from my experience. FSDP is easier to work with though.

This isn't correct, at least in the US. AI-generated material is not considered copyrightable unless there has been significant human involvement.

https://www.federalregister.gov/documents/2023/03/16/2023-05321/copyright-registration-guidance-works-containing-material-generated-by-artificial-intelligence

If you need to pad your paper, that means there hasn't been enough original research done.

Anyone else tired of papers that obscure a simple concept with endless paragraphs of verbose gibberish? This 17 page could be a few sentences.

Tl;DR the authors wrote prompts to tell GPT-4 to fix code given some unit tests and the output of the broken code. It performs better than GPT-4 that doesn't have access to the output of the code execution.

https://github.com/noahshinn024/reflexion-human-eval/blob/main/reflexion.py#L7-L12

The value that nanoGPT offers is that it is a self-contained (minimal dependencies), easy to understand code. This repo is essentially a wrapper for huggingface's models, dataset and accelerator, which is not very useful for didactic purposes.

Also, Paul actually has trained and works closely with ML models, unlike Eliezer, who does not understand how deep learning works .

Note: graphs comparing GPUs are not actual benchmarks but theoretical results. Nvidia likes to arbitrarily add restraints to their non-datacenter GPUs, so its not clear what the real-word performance is.

Google Magenta has done a bunch of research in this area. This one based on Perceiver is pretty promising, but currently only trained on piano.

10Gbps is more than sufficient, data loading from the internet is not the bottleneck. Most likely you'll have the data already stored on the machine itself. Btw why did you remove the post?

Are you looking to do distributed training across machines? Otherwise the NIC seems complete overkill.

What models have you tried? Wonder what the gaps between CUDA and ROCm are.

The model hasn't reached convergence, and/or the train dataset was too small.

My unsubstantiated hypothesis: BLOOM is severely undertrained, so most neurons aren't contributing at all to the final result compared to OPT-175.

Is that calculation taking into account memory (RAM/SSD) access latencies?

So it's calculating nearest neighbor compared to all of the images in the index every time a new search is done? Might be slow past say, 1,000 images.

How do you do the top-k neighbor search in iOS? Is there a library for it?

TorchBench (https://github.com/pytorch/benchmark) is used by PyTorch core developers to test performance across a wide variety of models. I've never used it personally though.

This benchmark is not representative of real models, making the comparison invalid. The model has ~5,000 parameters, while the smallest resnet (18) has 10 million parameters. You're essentially just comparing the overhead of PyTorch and CUDA, which isn't saying anything about the actual performance of the different GPUs.

Arc GPU only has 16GB, it would be worth giving it a shot if it had 24GB+ like the 3090/4090 does imo.

Which company has the best after-parties?

/r/machinelearning is more mainstream than LW and is less of a community. It's easy to bully weirdos.

LessWrong is a silly place where people take themselves too seriously, but its pretty cringy to have a subreddit dedicated to make fun of those people.

You'll have to try it out and see what works best. By the way, PyTorch has some beta support for mobile deployment: https://pytorch.org/mobile/home/

tl;dr a thinly disguised ad comparing a zero-shot model with a fine-tuned model, of course the fine tuned model is going to be better. the lack of intellectual honesty really encourages me to try snorkel

also, /u/bradenjh good job pretending that you have no affiliation with the company

Yes, the post talks about research, not commercial use.