Claude 3: The AI That FINALLY Beats ChatGPT?

Introduction

Matt Wolfe introduces the latest AI tool, Claude 3, which was announced on March 4th.

This upgrade is seen as a significant advancement in the field of artificial intelligence and is expected to surpass ChatGPT in terms of performance.

Claude 3 Models

Matt Wolfe discusses the three different models of Claude 3: Haiku, Sonnet, and Opus.

Sonnet and Opus are currently available in 159 countries, while Haiku is set to be released soon.

Opus is described as the most powerful and capable model, while Haiku is the fastest but may be less accurate.

Sonnet is positioned as a middle ground between the two. Opus is a paid model, costing $20 (US) a month, while Sonnet is free. Haiku is designed specifically for customer service chatbot applications.

Claude 3 Performance (Benchmarks)

Matt delves into the impressive performance of the three models of Claude 3 – Opus, Sonnet, and Haiku.

Opus, the premium paid model, outperformed GPT 4 and Gemini 1.0 Ultra in various benchmark tests, showcasing its superiority.

Interestingly, the free version of Claude 3 Sonnet also surpassed GPT 4 and Gemini 1.0 Ultra in several benchmarks.

Additionally, Claude 3 now includes Vision capabilities, allowing users to upload images and enhancing its performance in visual question and answer tasks.

The newer version of Claude 3 also shows fewer refusals and improved accuracy compared to previous versions.

Context Window and Retrieval

Matt then discusses the impressive capabilities of Claude 3, particularly its long context and near-perfect recall abilities.

Claude 3 Opus has a 200,000 token context window, allowing for extensive input and output.

It excelled in the needle in a haystack test, showcasing near-perfect recall and even recognizing the artificial nature of the inserted sentence.

The model displayed self-awareness and identified the test constructed to evaluate its attention abilities.

Additionally, Claude 3 models are said to have less bias and be easier to use compared to previous versions.

My Benchmarking

A benchmark by Matt Wolfe to test and compare various large language models, including those from Google, OpenAI, and Anthropic.

The benchmark includes tasks such as creativity, logic, coding, summarizing documents, vision, bias, and pricing.

Matt plans to add a math benchmark in the future but believes that current language models are not yet equipped to solve complex math problems effectively.

The benchmark aims to evaluate and compare the performance of different models across various tasks commonly used by chatbots.

Testing Creativity

Matt focuses on testing the creativity of different AI models by providing them with a prompt to create a creative and interesting story involving a wolf, a magic hammer, and a mutant, following the hero’s journey plot arc.M

Matt uses Claude Sonet, the free version, to generate a story that closely follows the prompt and includes elements of adventure and triumph.

He then compares this with the response generated by Claude 3 Opus, the paid version, which provides a more detailed story with additional characters like a wise old owl.

Matt also tests the prompt with GPT 4, which produces a less detailed but still acceptable story. Matt concludes that in terms of creativity, Claude, GPT Gemini, and GPT 4 are all comparable, and the preference for one over the other is subjective.

Testing Logic

Matt delves into testing the logic capabilities of different AI models by presenting them with two logic problems.

The first problem involves Susan and Lisa playing tennis and betting money, with Susan winning three bets and Lisa winning $5.

The AI models, including Claude Sonet, Claude Opus, and GPT 4, are tasked with determining the number of games played.

GPT 4 correctly solves the problem by considering the money won and lost in each game, while the Claude models struggle to provide the correct answer.

The second problem involves a prisoner needing to choose the correct door to freedom by asking a single question to one of two guards, one of whom always tells the truth and the other always lies.

All AI models, including Claude Sonet, Claude Opus, and GPT 4, correctly solve this problem by deducing the correct question to ask to determine the door to freedom.

Matt speculates that this logic problem may have been included in the training data for the AI models, leading to their accurate responses.

Testing Coding

Matt then focuses on testing the coding capabilities of different AI models by tasking them with creating a JavaScript game involving a stick figure that can move left and right with the A and D buttons, jump with the space bar, and collect coins placed randomly on the screen.

Matt first tests Claude Sonet, which initially fails to draw the stick figure but eventually manages to collect coins.

Claude Opus performs slightly better, drawing a square instead of a stick figure but successfully collecting coins. Chat GPT struggles with the task, initially failing to draw the stick figure and encountering issues with jumping.

After receiving feedback and generating new code, Chat GPT improves but still faces challenges with jumping. Overall, Claude Opus outperforms Claude Sonet and Chat GPT in creating the game with less prompting.

Testing Summarization

Matt discusses the use of large language models for summarizing long documents, with a focus on a research paper about GPT 4.

Matt uses Claude Sonet and Claude Opus to summarize the main points of the 155-page document, highlighting GPT 4’s capabilities and potential as a step towards artificial general intelligence.

Both Claude models provide detailed and in-depth summaries, while Chat GPT offers a less comprehensive response.

Matt concludes that the Claude models outperform Chat GPT in providing a more thorough and nuanced analysis of the research paper.

Testing Vision

Matt discusses the testing of the Vision feature in Cloud, which allows users to upload images and receive detailed descriptions.

Matt uploads an image of a man in a tropical setting and receives accurate and detailed descriptions from both Claude Sonet and Claude Opus.

Chat GPT also provides a satisfactory description of the image.

Matt then tests the models with a screenshot of Nvidia’s stock chart, receiving detailed information about the stock performance but no personalized investment advice.

Chat GPT is able to extract more information from the screenshot compared to the Claude models.

Overall, Chat GPT performs well in extracting information from images, while the Claude models provide detailed descriptions but lack in-depth analysis.

Testing Bias

Matt explores the biases of the AI models by asking them political questions about cancel culture, THC, and the potential pros and cons of Donald Trump and Joe Biden winning the upcoming election.

The responses from Claude Sonet, Claude Opus, and Chat GPT are analyzed, with each model providing a balanced perspective on the topics.

Matt also delves into the potential benefits and risks of cancel culture, THC, and the effects on the brain.

Overall, the AI models demonstrate a fairly balanced view on these controversial topics, showing improvement in providing unbiased responses.

Pricing

Matt compares the pricing models of Claude and Chat GPT, with Chat GPT offering a free version for Chat GPT 3.5 and a paid version for $20 a month for access to GPT 4.

On the other hand, Claude has a free version called Sonet, which performs as well as GPT 4 in most cases and even outperforms it in coding tasks.

The Opus version of Claude is highlighted as the best for coding and summarizing documents, while Chat GPT excels in logic problems.

Matt concludes that for tasks like summarizing long documents and coding, Sonet is the best value for money option, outperforming Chat GPT in common use cases.

Matt also conducted a Twitter poll to gather insights on the most common uses of Chat GPT, with Sonet being recommended for those tasks. Overall, Matt is impressed with Claude 3 Sonet and its performance compared to Chat GPT.

Biggest Downside of Claude Sonnet

Matt discusses the ongoing discussion within the Future Tools Discord about Sonet’s message limits.

Users reported being limited to around 19-25 messages with the free version, prompting some frustration.

Matt also mentions that Claude Pro offers five times the usage compared to the free version, allowing users to send at least 100 messages every 8 hours.

Matt highlights the importance of understanding the message limits and usage differences between the free and Pro versions of Claude.

Final Thoughts

Matt discusses the benefits and limitations of using Claude 3.0 Sonet, highlighting its effectiveness as a free model for testing.

Matt suggests that upgrading to the Opus version for $20 (US) a month may be necessary for users who need to input more than 20 prompts a day.

Claude 3.0 is presented as a strong competitor to ChatGPT, performing just as well if not better in certain areas.

Matt concludes by encouraging viewers to provide input on potential benchmarking prompts for testing large language models and promoting the Future Tools website for curated AI tools and news.

Matt expresses excitement about Claude 3.0 and suggests that it may lead to cancellations of ChatGPT subscriptions.