ChatGPT is OpenAI’s endeavor in optimizing a language model for dialogues.
It is interesting to understand some of the key facts of this immensely viral language model which is released in the form of a chatbot.
Fact #1 It costs approximately $100,000 each day to run ChatGPT.
OpenAI which is funded by Microsoft is spending a fortune to keep this model available for public use.
A single chatGPT query uses 8 NVIDIA A100 GPUs
Fact #2 ChatGPT is trained on data only till December 2020.
Yes. Feels quite odd that this knowledge machine does not know anything that happened after December 2020. When asked about its counterpart Google’s LaMDA (released May 2021) — ChatGPT responds that it is aware of only BERT language model released by Google.
Fact #3 ChatGPT is the 5th model released by OpenAI in terms of finetuning their GPT model series
GPT was released in Jun 2018
GTP-2 was released in Feb 2019
GPT-3 was released in May 2020
GPT-3.5 was released in Jan 2022
ChatGPT was released in Nov 2022
Fact #4 GPT-3 model is finetuned with RLHF (Reinforcement learning using human feedback) and then trained on dialogues and rules to avoid inappropriate answers. That gave us ChatGPT
GPT3.5 — released in Jan 2022 is also called instructGPT
Fact #5 Is ChatGPT the largest language model till date?
No. There are numerous language models released which are trained on larger datasets. However ChatGPT is a model finetuned for dialogues.
Below — A representation of various models in terms of the number of parameters (in Billion) they are trained on (Till Oct 2022). ChatGPT is based on GPT-3 which is trained on 175 B parameters
Fact #6: Language models are build and released on a large scale in recent years. In 2022 alone 52 language models were released
The significant increase in language models can be attributed to the encouraging results shown by OpenAI’s GPT model released in 2020.
Fact #7 The training data for GPT-3 on which ChatGPT is finetuned from consisted of –
410B tokens from Web data
55B tokens from online books
12B tokens from books
3B tokens from Wikipedia.
Around 480 tokens were fed to formulate 175B parameters to train this model.
Fact #8 ChatGPT can lie (unintentionally??)
On account of huge amount of unreliable data fed to train this model, ChatGPT often provides incorrect answers to our questions.
The dialogue model although make it sound very convincing.
If you correct ChatGPT, it quite humbly accepts its mistake of providing misinformation.
Fact # 9 GPT-4 is coming soon
OpenAI plan to launch GPT-4 soon. The possibility is that this model will be trained on a gigantic dataset formed from the web and also from the interactions people had with chatGPT.
It is quite fitting that the huge interactions chatGPT has with people will generate humongous dialogue training data which OpenAI can use to train a new language model/ With this data, the model will hopefully be more finetuned and enhanced on account of some finetuning techniques (which openAI has not disclosed so far)
Thanks