We realize this will not be received well by some, but we hope it will educate the rest of you to think twice before you push that button.
Although the concept of artificial intelligence (AI) has been in play for centuries (we had chess-playing machines over 100 years ago), and the mid-1950’s (80 years ago!) is generally considered the official start of AI as a field, it seems everyone has become an expert in the past few years.
Further, with free services like ChatGPT available to anyone at any time, it is hard not to be lazy and use AI to formulate what you had previously used your brain for.
How is the accuracy?
- Due to the small sample size, this is not a scientifically defensible study, but our team has tested a few of the Large Language Models (LLMs) with fifteen questions, every three months, for the past 18 months. Of the 90 questions asked, the LLMs got six right (7%). We’ve been told we have been too hard on the LLMs and needed to spend more time training them. This could be true.
- However, per the attached, a far better study (1,000 standard questions – 500 multiple-choice and 500 free-response on simple topics like high school math and high school world history – to 14 LLMs) produced accuracy of 33% – 85%.
I don’t know about you, but I would not want to send anything to a prospect or client that was, at best, 85% accurate. That remaining 15% is what separates the winners from the losers.
But is there a bigger tradeoff than accuracy?
Yes. Just like your brain using 20% of the oxygen you breathe in, these LLMs consume a huge amount of energy to think.
- For instance, one study shows ChatGPT uses the same amount of energy as 33,000 homes in order to process your daily questions.
- In 18 months, the LLMs could consume the electricity equivalent to a country the size of the Netherlands.
- As for accuracy, the larger the energy consumption, the higher the accuracy.
Net, for every data inquiry, you have a choice:
- Use your brain and, although it will take you longer, you will become smarter, and your accuracy will probably be higher.
- Or, to be sensitive to the environment, use LLMs that consume lesser amounts of energy….and get mostly wrong answers.
- Or, environment-be-damned, use LLMs that consume greater amounts of energy…and still fall short of what you should want in terms of accuracy.
We are not promoting a return to living in caves and rubbing sticks together.
That said, you now know there is a sizable cost, in both accuracy and environmental impact, attached to that ‘free’ information.
Hopefully, this knowledge will cause you to not automatically use AI as a first option every time.