Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

It's absolutely unconscionable that Gemini Ultra got memory-holed. I can't trust anything that Google says about benchmarks.

It seemingly existed only so in December 2023, Gemini ~= GPT-4. (April 2023 version) (on paper) ("32-shot CoT" vs. 5-shot GPT-4)



Anyone who uses both products regularly will tell you that Gemini Advanced is far behind GPT-4 and Claude 3 Opus.

Pretending that they have a model internally that's on par but they're not releasing it is a very "my girlfriend goes to another school" move and makes no sense if they're a business that's actually trying to compete.


Gemini 1.5 Pro is on-par with GPT-4 and Opus in LMSys, and you can go try it for yourself in LMSys, and it's coming soon to Gemini Advanced (announced at I/O). Seems like GPT4O puts OpenAI in front again on LMSys


Gemini Ultra is 1.0 with 8k window. This is 1.5 with 1m window. Your feeling is based on incorrect assumption.


And?

You're replying to a comment that points out Gemini Ultra was never released, wasn't mentioned today, and it's the only model Google's benchmarking at GPT-4 level. They didn't say anything about feelings or context window.


Gemini Ultra has been available for people to try via Gemini Advanced (formerly Bard) for a few months


It says it may fall back to a worse model under load and there is no way to tell which you are getting. I think chatgpt has at times done something similar though.


> You're replying to a comment that points out Gemini Ultra was never released

What are you even talking about? How do you know it's memory-holed if you haven't used it? The API is not GA, but the model can be used through the chatbot subscription. GP is talking about their lack of trust on Google's claim of 1M context token, not GPT-4 level reasoning. If you're expect GPT-4 level performance with cost-efficient models, that's another problem.


Idk why you're so aggro, they're right, I meant the GPT-4 level reasoning




Consider applying for YC's Fall 2026 batch! Applications are open till July 27.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: