Has ChatGPT become stupid?
A topic that has been fueling debate on social media for months has finally received academic analysis. The research team from Stanford University and the University of California, Berkeley compared the GPT model’s answers to the same question in March and June, and found that the GPT’s ability to solve math problems and generate code in June has declined.
One speculation on this is that strict regulatory measures hurt the output quality of large models. On July 14, the U.S. Federal Trade Commission (FTC) sent a letter of inquiry to OpenAI, the owner of ChatGPT, asking the company to respond to whether there were unfair or deceptive practices in its operations. This is by far the strongest regulation AI technology companies have faced in the United States.
Various “correct” requirements for model training and output content will affect the performance of ChatGPT-children who are too obedient will appear to lack creativity, not to mention Large Language Models (LLMs). However, in terms of time, some users have reported that GPT has become “stupid” from April to May. At that time, OpenAI and the US government were still in the honeymoon period, and regulation was at least not the only reason for its performance decline.
Another more reasonable speculation is that OpenAI actively reduces the performance of GPT. OpenAI CEO Sam Altman (Sam Altman) said in early June that there is an extreme shortage of GPUs that provide computing power for large models, and “it would be better if fewer people used Chat GPT.” The “OpenAI Profit and Loss Analysis” report just released by Founder Securities provides evidence: If the accuracy of GPT-3.5 drops to 95.7%, the cost will drop by about 25%. Of course, OpenAI categorically denies the speculation of actively reducing performance, and only said that it will further study the reasons why GPT may become stupid.
As a product that ushers in a new era of AI, the changes in ChatGP T more or less reflect the changes in the field of intelligence—the honeymoon period between artificial intelligence and the public is over. The government will no longer turn a blind eye to allow AI technology companies to grow wildly, and consumers who have lost their sense of novelty will also start to use critical eyes to judge whether it is worth an extra expenditure.
And the shortage of supervision and computing power, the two major problems that make ChatGPT stupid, are also affecting every AI technology company.
Regulatory and computing power challenges persist
From a regulatory point of view, after realizing the potential of generative AI, the world has accelerated the introduction of laws and regulations. This has a far greater impact on startups than on technology giants. After all, technology giants with capital advantages and first-mover advantages can better absorb rising compliance costs.
On July 21, the White House announced that the government had reached a voluntary agreement with seven AI technology companies, which promised to “maintain strong security measures.” These seven companies are Amazon, Google, Meta, Inflection, Anthropic, Microsoft, and OpenAI. They are either technology giants themselves, or the main investors behind them are technology giants. The “big” of the big model corresponds to the “big” of the big company. big”.
Unlike the U.S. government, which prioritizes voluntary agreements with tech giants, the Chinese government’s regulation is more proactive. On July 13, seven departments including the Cyberspace Administration of China jointly issued the “Interim Measures for the Management of Generative Artificial Intelligence Services” (hereinafter referred to as the “Interim Measures”), requiring technology companies to file generative AI algorithms and ensuring that the content generated by AI is “transparent”. “”Reliable” and cannot be “false and harmful”. What generative AI should and cannot do, the “Interim Measures” basically made this requirement clear, but still did not mention the issue of open commercialization of generative AI.
From the perspective of another major problem—shortage of computing power, it is still a curse on AI technology companies in the short term, but in the long run, the relationship between supply and demand will return to balance.
On June 21, Intel announced that it would spin off its chip manufacturing business. Over the past three decades, Intel has been pursuing a vertically integrated strategy, taking care of everything from chip design, manufacturing to sales. It has excellent chip manufacturing capabilities, but only makes chips of its own design.
After the split, Intel’s chip manufacturing department will open up its production capacity, which means that in addition to TSMC and Samsung Electronics, chip design companies such as Nvidia and AMD will usher in a third partner with the ability to manufacture high-performance AI chips. Not only that, Intel and Samsung Electronics announced plans to expand production capacity almost at the same time. In the long run, the supply of computing power will become more and more abundant.
However, for Chinese companies, computing power risk may be a long-term problem. At the end of June, the “Wall Street Journal” stated that the Biden administration is considering implementing stricter restrictions on AI chips exported to China: direct export of chips to Chinese customers must be approved; at the same time, it is prohibited to provide computing power to Chinese AI companies in the form of cloud services . These two measures, if implemented, would all but eliminate Chinese companies’ access to high-performance computing power.
In response, China’s Ministry of Commerce and the General Administration of Customs announced on July 3 that they would impose export controls on gallium and germanium, two of the most important materials for manufacturing semiconductors. China’s exports of gallium and germanium currently account for 80% of global gallium and germanium exports. Immediately after July 17, the US Semiconductor Industry Association (SIA) issued a statement on the official website of the possible restrictions on the United States, saying, “The White House has repeatedly adopted overly broad, vague, and sometimes unilateral restrictions that may weaken the United States. competitiveness in the semiconductor industry, disrupting supply chains and causing significant market uncertainty.”
A Preliminary Study on the Business Model
It is uncertain when the policy will allow generative AI to be open for commercial use, and it is facing that the supply of chips may be interrupted at any time, but Chinese technology companies have not stopped in the dilemma. At the 2023 World Artificial Intelligence Conference (WAIC), which ended on July 8, major companies such as Alibaba, Huawei, Tencent, and Baidu all set up stands to display their “big models”, with a total of more than 30 models.
In the venue of the Shanghai World Expo Center, almost every company emphasized that its model can serve corporate customers in different industries. The most frequently heard words are finance, media, government, retail, etc., but it is rare to hear which company Talk about how to make large models serve individual users.
It is difficult to get individual users to pay, at least in the case of ChatGPT. Data from the app analysis platform data.ai shows that in the first month after the launch of the ChatGPT iOS app, there were only about 1,500 new paid users ($20/month) per day, and the monthly active payment rate (number of monthly paid users ÷ monthly active number of users) about 1.6%. In contrast, the payment rates of Spotify, Slack, and Evernote are 46%, 30%, and 4% respectively. Even considering that some users choose to pay through web channels and are not included in the statistics, ChatGPT’s figures are still not optimistic.
Still, Microsoft, the world’s largest enterprise software company, looks confident in selling generative AI products. On July 18, Microsoft announced at the Inspire conference that the company will give priority to providing Microsoft 365 Copilot services to enterprise customers, priced at $30 per month. This price is more expensive than most people expected. Compared with Microsoft 365 (formerly known as Office 365, including Word, Excel, PowerPoint, Outlook, Teams and other office software) the price of $12.5 per month for the business standard edition includes The full total price of Copilot (generative AI assistant) function is close to 3 times the original one.
However, for enterprises, Copilot is cost-effective even if it only improves single-digit efficiency. After all, the cost of hiring employees with a monthly salary of 20,000 yuan is enough to buy 100 sets of generative AI assistants. Microsoft reached an agreement with KPMG a week before announcing the pricing. KPMG will spend a total of US$2 billion on Microsoft’s artificial intelligence and cloud computing services over the next five years. The accounting firm expects the investment to create 1.2 billion billion dollar return. According to its 2022 revenue calculation, it is equivalent to these “AI employees” who will bear 7% of the new performance every year.
Compared with Microsoft’s series of progress in the commercialization of generative AI, the commercialization of OpenAI has been difficult. Not only is the payment rate not optimistic, but the number of users is also facing growth bottlenecks. Statistics website Similarweb said ChatGPT’s website visits and application software downloads declined for the first time in June.
Silicon Valley Tech Giants Awaken
As the relationship between the two parties gradually undergoes subtle changes, the honeymoon period between Microsoft and OpenAI will also end.
OpenAI CEO Sam Altman has repeatedly mentioned that ChatGPT will be a “super assistant”, which is undoubtedly reminiscent of the concept “Copilot” that Microsoft has repeatedly promoted – an intelligent co-pilot, which is actually another way of saying a super assistant. And OpenAI did what it said. After launching the “function call” function in June, it launched the “custom instruction” function in July. The former allows ChatGPT to connect to software or data other than the model, so as to give more accurate answers when answering specific questions; the latter allows users to control the style of ChatGPT’s reply content, improving personalization capabilities.
Naturally, Microsoft will not put all its eggs in the OpenAI basket. At the Inspire conference on July 18, Microsoft also announced its cooperation with Meta to add the new Llama 2 model developed by it to the AI model catalog of Azure cloud services, providing customers who want to call large models with GPT Another option for . Llama 2 is the second-generation large language model developed by Meta. Meta claims that its performance is only slightly inferior to GPT-4 and PaLM 2 (they are the most advanced large models of OpenAI and Google respectively). The key is that this model is free for commercial use.
In addition to the familiar old faces, there are also heavyweight new players entering the field of intelligence in July.
On July 12, Musk announced the establishment of a new artificial intelligence company-xAI. He convened a group of artificial intelligence scientists with a mathematical background, determined to be a general artificial intelligence that can help humans solve complex problems in mathematics, physics and other disciplines. Musk said the team will keep a smaller size, closer to a research institution.
On July 19, Bloomberg reported that Apple has been secretly developing the chat robot AppleGPT since 2022, and the team managers report directly to CEO Cook. When Apple actually launches this app, Siri, which has been with users in the iPhone for many years but can only forecast the weather, will become a veritable smart assistant.
During the honeymoon period, even if ChatGPT replies with stupid answers, users will still blindly like it. But after the honeymoon period is over, everything gradually returns to basic business logic. AI technology companies need to walk among the government, users, competitors, and partners to find a balance of interests for all parties. No relationship is unbreakable, not even Microsoft and OpenAI.
Attachment: The following are the news details of important progress in the field of artificial intelligence in regulation, models and chips in July 2023.
01. A boot of domestic supervision has landed: “Interim Measures for the Management of Generative Artificial Intelligence Services” was released on
July 13, and seven departments including the State Cyberspace Administration, the National Development and Reform Commission, the Ministry of Education, the Ministry of Science and Technology, the Ministry of Industry and Information Technology, the Ministry of Public Security, and the State Administration of Radio, Film and Television Jointly issued the “Interim Measures for the Management of Generative Artificial Intelligence Services” (hereinafter referred to as the “Interim Measures”), which for the first time made clear regulations on the research and development and services of generative AI. Compared with the “Generative Artificial Intelligence Service Management Measures (Draft for Comment)” in April, the official version of the “Interim Measures” clarified the dialectical principles for generative AI supervision for the first time, requiring “equal emphasis on development and safety, promotion of innovation and law-based Governance” and the implementation of “inclusive prudence and classification and hierarchical supervision”. The “Interim Measures” will come into effect on August 15.
02. US Federal Trade Commission: The first official investigation into OpenAI
On July 14, the US Federal Trade Commission (FTC) sent a 20-page letter of inquiry to OpenAI, raising issues including model training, privacy and prompt word risks, API 49 questions including plug-ins, personal information, etc., including more than 200 small questions. This is the first time that U.S. regulators have officially launched a review of the potential risks of generative AI, and it is also the first investigation OpenAI has faced. Talk about the relationship. At that time, most of the questions raised by the senators were not sharp, and more often they asked Altman for advice on artificial intelligence technology and opinions on the management methods of artificial intelligence like ChatGPT.
03. The Biden administration: Establish an “AI development consensus” with 7 technology giants
On July 21, the White House announced that the administration had reached a voluntary agreement with seven generative AI companies to develop guidelines designed to ensure the safe development of the technology. These companies promise to fulfill the three basic principles of “safety, security and trust” in the process of artificial intelligence technology development, and take a number of measures to ensure AI security, including internal and external security tests and development of watermark systems before releasing products Help users identify AI-generated content and reduce the risk of fraud and misleading. It is said that OpenAI also specially drafted an internal policy memorandum, which shows that the company supports the idea of ”requiring any institution that wants to develop artificial intelligence systems to first obtain government permission.” The company’s CEO Altman testified at the Senate hearing in May. Similar views have been expressed above.
04. Musk established another company xAI: to do general artificial intelligence
On July 12, Musk announced on Twitter that he officially founded the artificial intelligence company xAI. In Musk’s eyes, xAI is an artificial intelligence company, but it is not limited to artificial intelligence, but has a broader goal-“understanding the universe”. This is another new company founded by Musk after the airline SpaceX in 2002, the new energy vehicle company Tesla in 2003, and the brain-computer interface company Neuralink in 2016. He said that xAI will cooperate deeply with the other two companies he owns, Twitter and Tesla, and use the high-quality content of the former and the computing power of the latter to train the model. Musk criticized ChatGPT and other large models for creating lies to the public in April. He believes that political correctness cannot be considered when building AI, because artificial intelligence must be allowed to say what it really believes, even if it will attract criticism, but it is the pursuit of truth And the only way to ensure that AI does not threaten humanity.
05. Microsoft’s generative AI pricing for Microsoft 365: $30 per person per month On
July 19, Microsoft announced Microsoft 365 Copilot—a generation tool specifically designed for Microsoft 365 with the name “Office 365”, including Word and Excel. Pricing and rollout of the AI Assistant—based on OpenAI’s most advanced model, GPT-4. Microsoft is pricing the product at $30 per person per month, and it’s only being sold to enterprise customers for now. Many people think that the price is too expensive. GitHub Copilot, an AI programming assistant owned by Microsoft, only charges $19 per month, and the advanced version of ChatGPT is only $20 per month. However, after the meeting, Microsoft’s stock price rose by nearly 4%, reflecting the capital market’s confidence in the attractiveness of Microsoft’s office suite.
06. Apple’s entry: AppleGPT has been tested internally
Bloomberg reported on July 19 that Apple has created an artificial intelligence framework called “Ajax”. The framework was created in 2022 with the goal of unifying Apple’s development in machine learning. The system is built on GoogleJAX, Google Cloud’s machine learning framework (which may be where the weird name “Ajax” comes from). Based on the new “Ajax” framework, Apple has improved many products such as search, Siri, maps, etc., and developed a ChatGPT-like chat robot AppleGPT based on this framework. Apple plans to hold a major artificial intelligence-related event next year, when it could unveil the product, which has been more than a year behind rivals.
07. Intel: plans to spin off the chip manufacturing business
According to a Reuters report on June 21, the chip giant Intel announced an organizational restructuring, and its manufacturing business will operate independently in the future and be responsible for its own profits and losses. The spin-off will help Intel focus more on chip design. After the spin-off, according to Intel CFO David Zinsner (David Zinsner) on the investor conference call, Intel’s chip design department will be separated from the manufacturing business. Build a “customer-supplier” relationship. Based on this new model, Intel will become the world’s second largest foundry next year, with manufacturing revenue exceeding $20 billion (TSMC’s sales target is $85 billion). However, Intel did not give a specific execution timetable. After the news was released, Intel’s stock price closed down 6%, the biggest one-day drop since January.
08. The Biden administration: preparing to upgrade restrictions on AI chips in China
According to a report from the “Wall Street Journal” on June 27, the Biden administration is considering imposing new restrictions on AI chips exported to China. The U.S. Department of Commerce may stop chip manufacturers such as Nvidia from exporting chips directly to customers in countries such as China. Such exports must be approved. In addition, the U.S. government is also considering restricting the provision of cloud services to Chinese artificial intelligence companies. The U.S. government hopes to prevent circumvention of the chip export ban in this way.
But U.S. chip companies have objected to the restrictions. On July 15, executives from the three major U.S. chip giants—Intel, Qualcomm, and Nvidia—jointly went to Washington to lobby against the Biden administration’s expansion of restrictions on the sale of certain chips and semiconductor manufacturing equipment to China. Two days later, the US Semiconductor Industry Association also called for no further escalation of restrictions on chip sales to China, as the restrictions would “disrupt supply chains, cause significant market uncertainty, and prompt China to continue to escalate retaliatory actions.”