The reliance on supervised learning limited their use on datasets that were not well-annotated, and also made it prohibitively expensive and time-consuming to train extremely large language models. Prior to transformer-based architectures, the best-performing neural NLP ( natural language processing) models commonly employed supervised learning from large amounts of manually-labeled data. Also around that time, in 2018, OpenAI published its article entitled 'Improving Language Understanding by Generative Pre-Training,' in which it introduced the first generative pre-trained transformer (GPT) system (' GPT-1'). That development led to the emergence of large language models like BERT in 2018 and XLNet in 2019, which were pre-trained transformers (PT) but not designed to be generative (they were 'encoder-only'). Generative pretraining (GP) was a long-established concept in machine learning applications, but the transformer architecture was not available until 2017 when it was invented by employees at Google. Also, companies in different industries have developed task-specific GPTs in their respective fields, such as Salesforce's 'EinsteinGPT' (for CRM) and Bloomberg's 'BloombergGPT' (for finance). For example, other GPT foundation models include a series of models created by EleutherAI, and recently seven models created by Cerebras. The term 'GPT' is also used in the names and descriptions of such models developed by others. Such models have been the basis for their more task-specific GPT systems, including models fine-tuned for instruction following-which in turn power the ChatGPT chatbot service. The most recent of these, GPT-4, was released in March 2023. Each of these was significantly more capable than the previous, due to increased size (number of trainable parameters) and training.

OpenAI has released very influential GPT foundation models that have been sequentially numbered, to comprise its 'GPT-n' series. As of 2023, most LLMs have these characteristics and are sometimes referred to broadly as GPTs. GPT models are artificial neural networks that are based on the transformer architecture, pre-trained on large data sets of unlabelled text, and able to generate novel human-like content. The first GPT was introduced in 2018 by OpenAI.

Generative pre-trained transformers ( GPT) are a type of large language model (LLM) and a prominent framework for generative artificial intelligence.

Transformers: Age of Extinction instal