One of the biggest questions / challenges / opportunities is detecting the Bot. So we thought we'd give it a go.....
AI jackpot
AI has hit the jackpot. ChatGPT’s creator, OpenAI (At the time of writing), is valued at $29,000,000,000. Not bad for a company that expects to make one-twenty-ninth of that in revenue … in 2024. While Google’s Bard knocked a cool $160bn off the value of the company with just one small ‘hallucination’, as invented ‘facts’ are called in the AI chatbot world.
Don’t get us wrong, ChatGPT and its kin are remarkable achievements. They’re great fun to play with too and starting to prove very useful, if a bit immature as products.
But as for stealing our jobs, putting a generation of knowledge workers out on the street? Or helping cheats pass degrees without all the bothersome ’learning’ bit? We should probably take an in-depth look at that.
People are rightly concerned that AI can be used to “fake” human effort. It seems even easier to cheat than the old method of lifting content off webpages and pasting it into essays.
To counter this cheating, an arms race has started. Universities now habitually employ plagiarism detection software to stop old style cheating. At the same time, a new breed of AI writing detectors has started to spawn.
We have created our own: Find the real writer: Bot or Not
It began as an experiment and blossomed into a fascinating product. We reasoned, possibly unjustifiably, it must be possible to detect the difference between a soulless machine writing and a real flesh and blood human. And you can, to a reasonable degree of confidence, but it was quite challenging.
After all, the AI has been taught with mind-bending quantities of actual human writing on which to base its own writing. Not to mention, writing is really really really really complex.
We had to consider what makes ChatGPT and other language-processing AI models “good” and how that “good” is measured. There must be some metric for evaluation?
After digging around, we discovered several measures that usually get mentioned:
- perplexity
- cross-entropy
- bits-per-character
We will go into what these measures in depth in another blog, possibly by asking an AI tool to write it for us. We’ll also blog about how we built the app (spoiler alert: iteratively). And we’ll write about how we tested it (and retested and tested it again), thinking about all the ways humans can try and write robotically. 🧐
Right now, please go and play with our AI detector, or you can watch our live stream, where we discuss the tool and ChatGPT generally. We’d love to hear your thoughts.
