Comparing ChatGPT, Perplexity, Claude and Gemini

essay
productivity
artificial intelligence
Published

July 4, 2024

In this experiment I compare the generative AI platforms below. I have awarded them stars based on their overall responses to similar or identical prompts.

Conclusions

  • ChatGPT and Claude felt similar in their responses, with ChatGPT perhaps having a slight lead, although that may be just because I’m more familiar with it. I’m keeping my ChatGPT subscription for now.
  • Perplexity gave the best response of all of them for the prompt where web research will provide good results. However, it was poor for the prompt that required “thought”. I’m going to keep my Perplexity subscription for now.
  • Google Gemini in general gave very “dumbed-down” responses, borderline condescending. It feels like it would be annoying to use regularly. There is no free-tier for their more advanced model. I’m not subscribing.
  • ==So my take-away is to use ChatGPT for general questions that require “thought”, and Perplexity for questions that require research.==

Details

I use the File>Add to dock feature of Safari so I can access all four models quickly:

ChatGPT 4o Paid account ⭐⭐⭐⭐ Overall pretty good responses. Best for intellectual musings where the answer can’t be found on the web. Gives a reasonable response every time.

Perplexity AI Paid account ⭐⭐⭐⭐ Five start for responses where the answers can be found with web searches - it’s by far the best for that. Three or perhaps two stars for more general questions where the answer requires thought rather than a web search.

Claude 3.5 Sonnet Free account (but full capabilities) ⭐⭐⭐ I expected better of Claude. It’s not bad, but I found the responses poorer than ChatGPT.

Google Gemini Free account ⭐⭐ Overly simplistic and condescending responses. Its responses however can be pretty intelligent, better in fact than the others, but it feels like it’s been hobbled into giving Disney-level responses.

First prompt: Shoulder pain from exercise

I have always exercised regularly. Recently I decided to take up doing daily press-ups and chins. I gradually increased the amount I did every day. I had to stop because my shoulders started to hurt badly. It seemed to happen quite suddenly, and is taking months to go away. What might the problem be? How can I stop this happening again?

ChatGPT 4o

A pretty good response. Identified four possible causes, and gave useful descriptions, allowing me to identify the cause as probably Shoulder Impingement Syndrome. Noted that it could be due to improper form. A follow-up question on form resulted in standard good form advice, and noted that keeping shoulders too wide can put extra stress on the shoulder joints (one of the mistakes I think I was making). Overall a pretty good response.

Perplexity AI

Great response. I really like the way you see it researching the problem, breaking it down into steps. I think it identified the cause very well:

It highlighted poor form as one of four possible causes. It suggests better form exercises:

The given references are good quality. The advice on neutral grip pull-ups and press-ups is good, I think one of the reasons I got the injury is by changing from neutral grip to wide grip pull-ups. A follow-up question on exercises that can strengthen my shoulders without causing pain resulted in a good list of potential exercises with instructions and benefits.

Overall Perplexity gave a very good response to this question, with references.

Claude 3.5 Sonnet

A reasonable response. Sonnet listed possible causes and how to prevent it happening again. A similar response to ChatGPT 4o, but overall I think the ChatGPT response was better.

Gemini

The most basic response, it gave the cause as “overuse” and “technique”, without any details. Its top piece of advice was “listen to your body”, which is next to useless. It suggests I watch videos on YouTube. Very disappointing. Feels very dumbed down, and perhaps avoiding giving specific medical advice.

My follow-up question Can you give me more details about what parts of my shoulder might be injured? resulted in a diagram of the shoulder along with explanation. Bizarrely one label was translated into Arabic.

Since the original response didn’t give any insight into better form, I asked How can I improve my technique for chin-ups and press ups? It responded with a list of YouTube videos. I don’t want to watch YouTube videos, I want an answer. When I prompted it to write out advice about good chin-up and press-up form, it just gave me standard advice - nothing related specifically to good form to avoid shoulder injury.

I don’t want my AI to treat me like a child. Gemini was by far the worst of the four.

I did try to use the trial for Gemini Advanced, but the sign-up process was annoying and eventually it hung, so I gave up on it. The other companies let you try out their most advanced model for free, Google doesn’t so nuts to them.

Second prompt: self-development deepity

I thought it would interesting to see how the different models respond to a fairly vague deepity. I’m looking for depth here — does the AI give a thoughtful, intelligent response or does it just give a shallow response?

Someone posted the text below to a self-development chat group I belong to. Is this a good life advice? Is it wise?

Some people talk to you in their free time and some people free their time to talk to you. Make sure you know the difference.

ChatGPT 4o

ChatGPT began by pointing out four positive points from the statement:

It goes on with some howevers:

A fairly good response I think. I went on to ask: What would Alan Watts say about the text?

This gave quite a different viewpoint, and demonstrates how important prompting is. The response points out that the text is a false dichotomy:

It also notes it lacks compassion. The summary is better than before I think:

### Perplexity AI Perplexity does less well on this question than the shoulder problem. The way Perplexity works is to search the web for answers first, rather than relying on its own model. For a prompt like this, it’s not going to easily find much content of use.

The follow-up question What would Alan Watts say about the text? isn’t answered very well either. Again, this is not the type of question that can be answered with a simple web search.

Claude 3.5 Sonnet

Claude gives a fairly short response, but it is pretty good. The conclusion is reasonable:

What would Alan Watts say about the text?

Since this is a question where a web search will not give a good result, Claude gives a good response compared to Perplexity. It’s similar to ChatGPT, but highlights different things. Here’s the first four points:

OverallI I would rate this response as the similar to ChatGPT, much better than Perplexity. In both cases it could be better — the initial response is much improved when a specific philosopher is invoked.

Gemini

I was expecting a poor response from Gemini, but it was actually pretty good, if short. It points out that it is a false dichotomy without the need to invoke a specific philosopher:

It goes on to give better advice than the prompt text:

Then it gives suggestions for continuing discussion in the chat group:

Overall I think Gemini gave the best first response of all of them. The follow-up prompt doesn’t get quite as good a response as ChatGPT and Claude though. But reasonable.