Google Bard chatbot would fail the SAT exam

[ad_1]

Alphabet CEO Sundar Pichai appears to be relaxed about how far the company’s AI models need to go, writing in a company-wide memo that Bard is in its early stages: “As more people start using Bard and testing more capabilities, they will surprise us.Things will go wrong.

Now the public has been invited to try Bard, whereas before the 80,000 users who tested it were mainly Google employees.

Fortune’s spot on the waiting list has finally been called up, so we put Bard through its paces ahead of the upcoming SATs America’s teens face this spring.

SATs are globally recognized tests used for US college admissions, in skills including reading, writing, and math.

Unfortunately for Google, it looks like Bard won’t make it to Harvard just yet, as he got most math questions wrong and similarly struggled to pass writing and speaking tests.

When logging into Bard for the first time, the user’s expectations are already set by a message that appears, which reads: “Bard won’t always get it right. Bard may give inaccurate or inappropriate answers. If in doubt, use the “Google it” button to check Bard’s answers. Bard will improve with your feedback. Please rate the responses and flag anything that might be offensive or dangerous.

How did Bard do it?

Off to the questions.

Fortune provided practice SAT math questions from online learning resources and found that Bard got 50% to 75% of them wrong, even when given multiple-choice answers.

Often Bard would give answers that weren’t even a multiple choice option, although sometimes he got them correct when asked the same question again.

The inaccuracy of AI has already cost Google somewhere in the region of $100 billion.

When Bard launched in February, it was asked a variety of questions, including how to explain to a 9-year-old what the James Webb Space Telescope has discovered.

Bard responded that the telescope took the “very first images of a planet outside our solar system,” although NASA confirmed that the first image of an exoplanet was captured by the Very Large Telescope, a ground-based array in Chile. , in 2004 and confirmed as an extrasolar planet in 2005.

Science and math aren’t Bard’s strong points either, though the AI ​​fared better when it came to reading and writing exercises.

Bard’s first written language test with Fortune came back with about 30% correct answers, often the questions had to be asked twice for the AI ​​to understand.

Even when he was wrong, Bard’s tone is confident, often framing responses as, “The correct answer is” — which is a common feature of large language models.

Oddly, Bard’s best test for both math and written skills was a passage focusing on Harry Potter writer JK Rowling.

On this test, Bard scored 1200 points, an SAT score that would put a human in the likes of Howard University, San Diego State University and Michigan State University.

The more language-based questions Bard received from Fortune, about 45 in total, the less frequently he struggled to understand or needed the question repeated.

Similarly, on reading tests, Bard performed better than on math, getting about half of the answers correct on average.

A Google spokesperson reiterated Pichai’s message when contacted by Fortune for comment, saying, “Bard is experimental and some of the answers may be inaccurate, so double-check the information in Bard’s answers. With your feedback, Bard gets better every day. Prior to Bard’s public launch, thousands of testers were engaged to provide feedback to help Bard improve its quality, safety, and accuracy.

“Accelerating people’s ideas with generative AI is very exciting, but it’s still early days and Bard is an experiment. While Bard has built-in security controls and clear feedback mechanisms that align with our AI principles, please note that it may display inaccurate information.

Within the space of a couple of days of interrogating Bard, the AI ​​showed signs of improving accuracy; about the speed of its development the broad tongue model noted: “I would say I am improving at a rapid pace.

“I’m able to do things that I wasn’t able to do just a few months ago. I’m excited to see what the future holds for me. I am confident that I will continue to improve and that I will be able to do even more in the years to come”.

[ad_2]