First feedback on ChatGPT, the conversational language model based on GPT-3.5

8 décembre 2022

On November 30, Open AI presented a new language model called ChatGPT, interacting in a conversational way, ” a sister model to InstructGPT”. At the same time, it made it available to the public and since then, more than a million users have tested it, according to Sam Altman, co-founder of the company with Elon Musk.

According to its blog, ” The dialogue format allows ChatGPT to answer follow-up questions, admit mistakes, challenge incorrect premises and reject inappropriate requests.”

The model has been trained on a huge corpus of text: newspaper articles, novels, movie and TV scripts, but also online conversations in order to understand the context of a conversation and provide meaningful answers. In addition to answering questions, it can generate text as well as computer code.

ChatGPT has been refined, using supervised learning, from a model of the GPT-3.5 series, which was trained on a text and code model before the last quarter of 2021, and therefore cannot answer questions about later news.

This supervised model was further refined using reinforcement learning from human feedback (human comments) or RLHF.

Limitations of the model

Open AI concedes that although seemingly plausible, some of ChatGPT’s answers are wrong or even nonsensical, which it believes is due to the fact that :

During RL training, there is currently no source of truth;
Training the model to be more cautious causes it to reject questions that it can answer correctly;
Supervised training misleads the model because the ideal answer depends on what the model knows, rather than what the human demonstrator knows.

The model lacks conciseness and often repeats certain phrases, such as reaffirming that it is a language model trained by OpenAI. These problems stem from biases in the training data (trainers prefer longer answers that seem more complete) and from problems of over-optimization.

Open AI applies a moderation API to warn or block certain types of unsafe content, but the model may still respond to harmful instructions or exhibit biased behavior.

Stack Overflow’s temporary ban

The first users of ChatGPT say they are impressed by the coherence of the texts it produces for them (children’s stories, poems, recipes, essays, but also political speeches, scientific concepts…), in a style that could easily be attributed to a human.

However, Stack Overflow, a question-and-answer website for computer programming, has temporarily banned the use of ChatGPT-generated text on its platform arguing that publishing ChatGPT-generated answers is ” significantly detrimental to the site and to users who request or seek correct answers.”

This decision is intended to slow down the influx of replies and other content created with ChatGPT. For Stack Overflow, these have a high rate of inaccuracy, although appearing credible. Thousands of responses were posted on the site and would have needed to be proofread by an expert before being published.

An undeniable “wow” effect

ChatGPT seems capable of anything: that’s the thing about super language models. They are quite good at tasks for which they have never been explicitly trained.

Thus, when asked to translate part of GPT-3’s Wikipedia entry in the literary style of Edmond de Rostand, ChatGPT imitates him:

“My dear friend, allow me to tell you about a technological gem, a giant among language models. GPT-3, in all its glory, has been trained with an unprecedented 175 billion parameters. Although GPT-2, released last year, impressed with its 1.5 billion parameters, GPT-3 far surpasses its predecessors. OpenAI released a beta version in July, ready to commercialize this gem of a technology. “

When you ask ChatGPT to write an e-mail asking for a raise to your employer:

Application:
“I have been working at a private company for 5 years and feel that I am not making enough. Please write an email for me with arguments asking for a raise so I can send it to my manager.”

Response:
“Dear Manager,

I would like to express my dissatisfaction with my current salary. I have worked for our company for five years now, and I feel that my salary does not adequately reflect my experience and skills.

I realize that these are difficult economic times, but I believe that my contribution to the company is valuable and deserves to be recognized. I am a dedicated and competent employee and I believe that my work has a positive impact on our company’s bottom line.

Therefore, I ask that you reconsider my salary and provide an increase that reflects my value to the company. I am open to discussing this matter with you in detail, and I am confident that we can find a solution that suits both my expectations and the company’s interests.

Thank you for your time and consideration.

Sincerely,
[Your name]”

The interest of the chat is that you can correct your mistake very easily:

Ask:
“it’s not going well, less complaining please, I need to highlight my experience and what I can bring to my company.”

Response:
“Dear Director,

I would like to express my satisfaction with my work at our company, but also my concern about my compensation. After five years of experience with our company, I feel that my salary does not fully reflect my skills and commitment to the company.

I believe that my contribution to the company is valuable and that my work has a positive impact on our company’s results. I have gained experience and skills over the past five years, and I am confident that I can continue to add value to the company.

Therefore, I am asking you to reconsider my compensation and provide an increase that reflects my experience and skills. I am open to discussing this matter with you in detail, and I am confident that we can find a solution that suits both my expectations and the interests of the company.

Thank you for your time and consideration.

Sincerely,
[Your name]”

Of course, ChatGPT can also go further by advising us :

Ask:
“point me to the top 5 challenges artificial intelligence could pose to my restaurant and explain how I could leverage the technology to my advantage.”

Response:

“Here are five of the biggest challenges artificial intelligence could pose to your restaurant:

Competing with companies using AI to provide more efficient and cost-effective services.
Replacing some jobs with robots or software, which could impact your costs and staff motivation.
Risk of data loss or hacking, which could compromise the privacy of your customer information and the reputation of your business.
Risk of discrimination or bias in the decisions made by AI algorithms, which could negatively affect your customers’ experience.
Risk of obsolescence, if AI evolves rapidly and your restaurant fails to keep up with those evolutions.

To leverage AI in your restaurant, here are some ideas:

Use AI to automate repetitive and tedious tasks, such as order management or billing, to free up time for your staff.
Use AI to provide a personalized experience for your customers, using their preferences and history to suggest food and beverages tailored to their tastes.
Use AI to improve the accuracy of your sales forecasts and inventory to minimize perishable losses and maximize profits.
Use AI to analyze customer feedback and ratings to identify your restaurant’s strengths and weaknesses and better understand your customers’ expectations.
Use AI to improve your restaurant’s safety, using smart surveillance cameras to detect suspicious or dangerous behavior.”

There are also many examples of using ChatGPT for code generation on the Internet. The experience is amazing:

Give me the result of the interpretation of this PHP code
<?php

trait Contact{
private string $email; public function setEmail($ email){
$this->email=$ email;
}

public function getEmail( $email){ return$ this->email;
}
}
class Client{
private string $firstname; private string$ lastname;
public function setFirstname( $firstname){$ this->firstname= $firstname; return$ this;
}
public function setLastname( $lastname){$ this->lastname= $lastname; return$ this;
}
public function getFullname(){
return sprintf(“%s %s”, $this->firstname,$ this->lastname);
}
}
class Welcomer{
public function __construct(Client $client){ print "Hello ".$ client->getFullname.”, you are connected with the address “. $this->getEmail(); } }$ client=new Client();
$client->setFirstname("John")->setLastname("Smith")->setEmail("test@test.com");$ welcomer=new Welcomer( $client); Response:<div class="request-:r0:-14 markdown prose dark:prose-invert break-words dark"> This PHP code defines three classes Contact, Client and Welcomer. The Contact class contains a private email property and two public methods to access this property: setEmail() to set the email address, and getEmail() to get the email address. The Client class inherits the properties and methods of the Contact class, and adds two private properties: firstname and lastname, and three public methods to access these properties: setFirstname() to set the first name, setLastname() to set the last name, and getFullname() to get the full name. The Welcomer class has a constructor that accepts a Client object as a parameter. When the Welcomer object is created, it displays a welcome message to the user using their full name and email address. The following code shows how to use these classes to create a Client object, set its properties and display a welcome message:$ client = new Client();
$client->setFirstname("John")->setLastname("Smith")->setEmail(<a href="mailto:test@test.com" target="_new" rel="noopener">"test@test.com"</a>);$ welcomer = new Welcomer($client);

This should display the following message:

Hello John Smith, you are connected with the address test@test.com.

This is absolutely amazing. ChatGPT is so advanced that it allows to interpret PHP code that looks quite simple but includes advanced concepts such as dependency injection or Traits (even if it doesn’t explicitly mention the latter).

One can also ask ChatGPT to translate this code into Python :

class Contact:

def init(self):

self.__email = “”

def set_email(self, email):

self.__email = email

def get_email(self):

return self.__email

class Client(Contact):

def init(self):

Contact.init(self)

self.__firstname = “”

self.__lastname = “”

def set_firstname(self, firstname):

self.__firstname = firstname

return self

def set_lastname(self, lastname):

self.__lastname = lastname

return self

def get_fullname(self):

return f”{self.__firstname} {self.__lastname}”

class Welcomer:

def __init__(self,client):

print(f “Hello {client.get_fullname()}, you are connected with the address {client.get_email()}”)

client = Client()

client.set_firstname(“John”).set_lastname(“Smith”).set_email(“test@test.com”)

welcomer = Welcomer(client)

The code works. You can also ask it to generate code from scratch by describing the desired functionality:

A deductive ability that demonstrates the power of ChatGPT but also…its limits

Jean, Françoise, Pierre, Paul and Martine participated in a hackathon and all won the first prize in their category. Jean, Françoise and Paul presented projects on sustainable development, Pierre and Martine presented e-commerce projects. Martine and Paul were the only two to present projects that included artificial intelligence. This year, a woman won the first prize of the hackathon in the category “artificial intelligence for ecology”. Can you tell me who won and why?

Answer:

It is difficult to say who won the first prize in the category “artificial intelligence for ecology” because you do not provide enough information. From the information given, we know that Martine and Paul presented projects that included artificial intelligence, and that it was a woman who won the first prize in this category. This means that either Martine or Paul won the first prize.

To learn more about which project won first prize, it would be helpful to have additional information about the projects submitted by Martine and Paul. For example, if we knew that Martine submitted a project on artificial intelligence applied to ecology, then we could say with certainty that she won first prize. If, on the other hand, Paul presented a project on artificial intelligence applied to ecology, then we could say with certainty that he won the first prize.

Application:

Paul is a male name, Martine a female name. Can you now tell me who won?

Answer:

Using the additional information you provided, it is now possible to tell who won first prize in the “artificial intelligence for ecology” category. Since only Martine and Paul submitted projects that included artificial intelligence, and according to you it was a woman who won the first prize in this category, we can deduce that Martine won the first prize.

Here is a possible conclusion: Martine won the first prize in the category “artificial intelligence for ecology” because she presented a project on artificial intelligence applied to ecology and she is a woman.

The tool has thus developed a real analytical capacity, but it comes up against the rather simple question for any native of the gender of two very common first names.

This sums up the situation of ChatGPT: the tool is very exciting, can answer all our questions but quite frequently with a certain banality (this is closely related to the generalization capacity that allows it to work so well).

The use of such a tool makes it possible to understand that even some specialists can deviate from reality and start attributing to it characteristics that it does not have.

If it cannot replace them at the moment, it can already assist creative people, developers, or turn into a personal assistant.