At the announcement post, OpenAI made some bold claims about GPT-5. Including:
ChatGPT is now designed to think deeply when you need it to.
As a coding collaborator, GPT‑5 tackles complex tasks end-to-end and delivers more readily usable code, better design, and is more effective at debugging.
Create clearer, more compelling messaging for everything from stories to speeches and beyond.
Our best model yet for health-related questions, providing more precise and reliable responses while acting as more of a proactive thought partner.
Our most reliable model yet. It’s less prone to hallucinations and pretending to know things.
The last two points seemed particularly sus to me. Health answers? I still wouldn’t trust that. “Less prone to hallucinations”? That’s a big claim. Hallucinations have consistently been one of the biggest issues with LLM models. Even though it’s not as bad as the ChatGPT 3.5 days, it’s not clear if this problem will ever be fully solved without a paradigm shift.
During the week, I experimented with GPT-5. My first impressions on Friday weren’t too positive. Sam Altman tweeted that “the autoswitcher broke” and promised improvements.
GPT-5 did feel better the following week, but still not as good as it was hyped to be. My favorite model until now was OpenAI o3, and even ChatGPT-5 Thinking often didn’t seem as good as I remembered it. Not that I was able to compare, since access to previous models was removed altogether! I eventually got 4o back (after massive backlash from people who formed deep bondings with 4o), but I didn’t get o3 on ChatGPT Team and so was not able to compare.
After a few days, I learned to use GPT-5 better. Because of the auto-router, GPT-5 benefits from more prompt engineering (“think deeply”). This felt like a step back, since previous models got good at knowing what I want. Or I just already learnt how to use them. A few months ago I was confused with ChatGPT offering so many confusingly named models (what’s better, o3 or o4-mini high?). However, I eventually learned to use them and now missed the choice being taken away from me. Even though, from a user perspective, a model that knows how to choose for you to give you the best answer should be a better option. On the other hand, I suspect GPT-5’s auto-router is actually a cost-cutting measure in disguise, behind the scenes too often opting to use cheaper models even though they give noticeably worse answers.
The results I’ve been getting with ChatGPT 5 have been really inconsistent. Some answers are great, others are dumb. I guess that’s par for the course for AI but I was expecting an improvement, and this doesn’t feel like it. The jumps between ChatGPT 3.5 to 4, or 4o to o3 felt more considerable to me.
As for the claim of reduced hallucinations? This has not been my experience. I caught ChatGPT 5 lying in many occasions. It’s hard to compare if it’s any worse than previous models (again because I lost access to them), but it does sometimes feel like it.
Even ChatGPT 5 Thinking hallucinates. In one instance, I was asking ChatGPT 5 (Auto) how to configure a certain GitHub setting for an Organization. ChatGPT 5 confidently answered that it’s impossible. I then switched model to ChatGPT 5 Thinking to see if I would get a different answer. After several minutes of “thinking”, ChatGPT 5 Thinking confidently answered that it’s possible and even gave me exact instructions. Except, the instructions were impossible to follow because the answer was entirely hallucinated. In this case, ChatGPT 5 was more correct than ChatGPT 5 Thinking. The setting didn’t exist (even though both I and ChatGPT 5 Thinking wish that it did).
A few hours after GPT-5 was announced, Cursor announced the release of Cursor CLI plus free GPT-5 credits for one week. My 1 month Claude Pro subscription was just ending, so I decided to use Cursor CLI with GPT-5 for the week to experiment with both (compared to Claude Code with Sonnet 4).
Cursor CLI is clearly inspired by Claude Code. I don’t mind the rip-off personally since I like Claude Code. In Claude Code with Sonnet 4 the agent is far more transparent about what it is doing and tends to consult more; it even shows a checklist of the tasks the agent plans and executes. That clarity is missing in Cursor CLI for now: it explains less, simply makes changes, and sometimes it’s not clear why - though you can always stop it and ask questions.
Another thing missing in Cursor CLI is support for MCP, even though regular Cursor already has solid MCP support. But Cursor CLI came out less than a week ago. I assume they will improve it over time.
Aside from those gaps, I got decent results with Cursor CLI. The quality felt comparable to Claude Code, and the interface is almost a complete copy.
After the GPT-5 free credits ended for me, I decided to go back to Claude Code for now (I resubscribed for 1 month of Claude Pro). While Cursor CLI might improve in the future, for now it’s not as good as Claude Code. I also worry that the CLI might be an afterthought for Cursor.
At my current client, the only approved AI tool is Microsoft 365 Copilot (not GitHub Copilot). I had subpar results with it in the past, so I was glad that it was now updated to use GPT-5.
This was also a good way to experiment with GPT-5 for free. Even without an account, Microsoft Copilot offers a generous amount of GPT-5 requests (you have to remember to enable GPT-5 every time you start a new chat).
Still, the experience of using GPT-5 in Microsoft Copilot feels different from using it in ChatGPT, despite claiming to use the same model. I suspect the infamous auto-router likes to give Microsoft Copilot the cheaper models more often than not, unless you prompt it specifically not to. Even when prompting heavily, I still got considerably faster results than ChatGPT 5 Thinking. Perhaps the Azure backend is more optimized here or maybe Microsoft Copilot rarely gets routed to the best models by GPT-5.
Regardless, I did feel an improvement in the answers compared to the previous Microsoft Copilot models (“Quick response” and “Think Deeper”, which I believe are based on some variation of GPT-4). Even so, Microsoft Copilot is still limited in other ways (compared to ChatGPT), such as a small context window.
Overall conclusion, Microsoft Copilot is usable for basic work but far from my preference. I wouldn’t use it unless I had no other choice (which is the case at the current client).
Overall a disappointment, but still useful. I will continue to use it, particularly with ChatGPT 5 Thinking.
OpenAI has addressed some of the negative feedback already and will no doubt continue to improve GPT-5.
Featured image by Igor Omilaev on Unsplash.
]]>It doesn’t just “think”; it also does web research and cross-checks sources to reach a conclusion. Other ChatGPT models can browse too, but o3 digs deeper and pulls more sources (when I read its “thoughts,” it said it tries to fetch at least 10 sources).
This is similar to what Deep Research does, which makes sense because ChatGPT’s DR used the o3 model even before it launched. However, DR returns essay-length answers (and is limited to 10 uses per month on ChatGPT Plus), which isn’t always practical. o3 gives answers closer in length to the other ChatGPT models. There are Plus usage limits, but I had to use it quite a lot before hitting them.
The model “thinks” for several minutes before responding. Usually it’s worth the wait, except for simple questions another model could answer faster. For complex questions, o3 is often noticeably better. I tried tough coding prompts that 4o struggled with (confident answers with hallucinations), then asked o3 and got much better results. For bigger tasks I sometimes had to tweak the prompt a few times, but in most cases o3 eventually delivered (unlike 4o).
The model isn’t perfect. There are still hallucinations and mistakes. Neverthless, in my experience fewer than other models I’ve tried.
AI moves so fast that it’s hard to keep up. Last month o3 was probably the best model around, and now people say Gemini 2.5 has overtaken it. It takes time to use a new model enough to really understand its strengths and weaknesses.
I also played a bit with ChatGPT 4.5 and 4.1. I haven’t used them much yet and so far I’m less impressed.
I haven’t tried o4 or o4-mini-high yet. Assuming o3 is better, I’d rather wait a few minutes for deeper reasoning. For simpler questions I still default to 4o.
Featured image by Levart_Photographer on Unsplash.
]]>