Specialist
Executive at a cognitive intelligence company
Agenda
- Pre-trained model types, technology development history from GPT-1 (generative pre-trained transformer-1) to GPT-4, application scenarios, plus China’s and overseas industry development comparison
- Pre-trained models’ application in the financial industry, client demand, usage
- Domestic players’ business models, technological capabilities and development status quo
Questions
1.
Could you talk about the differences and similarities among the pre-training methods for natural language processing (NLP) voice models such as Bidirectional Encoder Representations from Transformers (BERT), Embeddings from Language Models (ELMo) and generative pre-trained transformer (GPT)? Which is the current mainstream? Do you think the trend will change greatly in the future?
2.
In June 2018, OpenAI introduced GPT and has been gradually upgrading it. Could you talk about the technology development history from GPT-1 to GPT-4 and the main iteration route? Which aspects have been upgraded?
3.
As you said earlier, GPT’s upgrade mainly relies on increased parameters. GPT-3 is trained on around 300 billion tokens. It is said that GPT-4 is trained on about 16 trillion tokens. Lately, some domestic companies have claimed that they are developing models similar to GPT by training the models on a huge amount of data. Do you think they can manage to copy OpenAI’s success? Or are there some technological barriers that hinder domestic companies?
4.
Users probably will be charged for using GPT-4 because the costs of GPT-4 including the costs of answering questions and computing power are relatively high. Leaders of some domestic companies think that small companies probably do not have many opportunities in the GPT segment. Do you agree? What do you think will be the future charging model? How profitable will it be?
5.
OpenAI has made GPT-2 open-source. Will GPT-3 become open-source after GPT-4 is launched? Does OpenAI make a previous product open-source after it launches a new product?
6.
Some domestic companies including Baidu and Alibaba are going to launch products similar to ChatGPT. They say that they are conducting R&D and may soon launch the products. How many of those products are based on GPT-2 or other early-stage open-source models? What do you think of the competencies of different players? Could you predict the future trend?
7.
Could you elaborate on the main application scenarios of GPT models or other PTMs in China? What are those application scenarios’ market sizes or growth rates over the past few years?
8.
What are the main application scenarios of these GPT models or other PTMs in the financial industry? For example, what are the possible output types generated by PTMs? What is the outputs’ value to downstream clients? Could you explain the cost advantages of PTMs?
9.
Are the large-scale PTMs that have gone into service in China mainly based on BERT models or other technical solutions?
10.
When it comes to parameters, do BERT models have pretty small parameters than GPT models in applications in the financial industry?
11.
Which application scenarios would be given priority in GPT model applications? Recently, Google Bard gave inaccurate answers to questions, which resulted in dramatic falls in stock prices. This warns us of the current problems of GPT models as well. Which technologies do you think can help to avoid such risks, say, improving the quality of trained materials?
12.
In which other application scenarios do you think GPT models have great potential? You mentioned some emerging application scenarios that don’t use the BERT models. Which development directions do you think are worthy of attention?
13.
Microsoft has recently integrated ChatGPT into its products. Will domestic companies follow this path? Are there any other paths?
14.
You mentioned that ChatGPT may bring huge development opportunities to large companies. What do you think are the opportunities and countermeasures of small and medium-sized start-ups?
15.
Is there anything about the industry or new trends we haven’t covered yet?
