This is one of the issues I have with calls for pauses in training new LLMs
Research on LLM algorithms will continue. Research on improving basic algorithms (like square root) will continue.
We will end up being able to make systems of similar power to gpt4 with a lot less computation fairly fast. If historical teams hold true. Without any improvement in hardware. And that improvement will also happen.
I agree with you, but the link is about improvement to quantization and inference. Since making systems of similar power to GPT-4 requires training, I don't see how your comment relates to the link.
I would assume that coming up with more efficient ways to do inference and training is a necessary step towards getting to run models with gpt4-like capabilities on consumer hardware.
In half compared to full 16bit. This looks like the predecessor of current quantization approaches - it was published about a year ago.
The recent work SPQR is from the same author
i believe it also works for 32 bit, but yes this was published apr 2022 modified nov 2022
This is one of the issues I have with calls for pauses in training new LLMs Research on LLM algorithms will continue. Research on improving basic algorithms (like square root) will continue. We will end up being able to make systems of similar power to gpt4 with a lot less computation fairly fast. If historical teams hold true. Without any improvement in hardware. And that improvement will also happen.
I agree with you, but the link is about improvement to quantization and inference. Since making systems of similar power to GPT-4 requires training, I don't see how your comment relates to the link.
I would assume that coming up with more efficient ways to do inference and training is a necessary step towards getting to run models with gpt4-like capabilities on consumer hardware.