I’ve been working with Grok from xAI. It seems to be better than most of the others I’ve tried. It has done a good job of helping me debug complex systems.
I wouldn’t trust it to provide instructions to pour piss out of a boot.
I was working with it for the configuration of the Amanda backup system. I’m integrating it with my Ceph cluster. While it did an “ok” job of configuring Ceph, for Amanda it was entirely wrong.
Even though it asked, and I provided version numbers, it gave instructions for a configuration method that hasn’t been used in years.
I still don’t have a working Amanda configuration, but I have ideas on how to get there.
Because I’m not a great writer, I use every tool at my disposal to write better. I asked Grok to analyze my article about Charlie.
It insisted that Charlie was still alive and that I was writing some “wish-fulfillment” fictional scenario. I told it to prove that Charlie had been murdered. It did its thing and told me that Charlie had indeed been killed on the 10th.
In the very next prompt, it again insisted that it was a “fictional fact” and that it was merely “alleged” that Charlie had been killed.
One of the things to know about AIs is that the more they need to analyze, the longer it takes and the more “expensive” it becomes. This means that they have a memory or token limit.
This leads to article truncation when it is asked to retrieve web pages.
It told me my article ended abruptly and that it was willing to help write the ending. When it was instructed to print my original article, it showed me the truncated version. When I told it to fetch the complete article, that it had to be a complete <div>, it said it had fetched the article, then proceeded to write its version of the end of the article.
When asked to provide Obama’s statement on the death of Charlie, it showed me three or for articles that only short quoted Obama, leaving out all the context.
AI is an amazing tool. It is still at the verify then verify again stage.
Story Time
I was working with a client on a Magento site. This client was a “digital agency.” They specialized in throwing up WordPress sites for $1500 or so.
They could do this by outsourcing most of the work to an Indian firm.
The CEO was up to give us a talk on using Indian outsourcing. He was very proud of himself for figuring out that these Indian tech firms will claim they can do any technical thing, even if they have no idea what they are doing. You only know if they are any good at the task after you’ve invested in the firm.
He had solved this by going to India and personally investigating a dozen different Indian tech firms before deciding on the one that actually knew how to do WordPress sites.
He had great success using them.
He had asked this same firm, “Do you do Magento?” They had answered in the affirmative. I was the unhappy recipient of their “product”.
My question to that CEO: “Did you ask them if they can do a tech thing? “Yes.” “Do Indian tech firms always say ‘yes’ to that question?” “Yes.” “Did you violate your own rule about trusting Indian tech firms?” “Yes.”
My point is that you can find an AI that does a fantastic job on a task. They might produce great results every day for a month. That doesn’t mean you can trust the next answer it gives. It is just as likely to make it up or lie to you as give you a good, correct answer.