Is the Help Helpful?

How do you write a paper with artificial intelligence (AI)? Access Innovations founder and chief science officer, Marjorie Hlava, decided to test three heavyweights: Claude, ChatGPT and Google Gemini. This interesting experiment came to us from The Scholarly Kitchen in their article, “Guest Post:  Trying to Write a Paper with LLM Assistance.”

Hlava asked each to rewrite a seven-paragraph article she’d drafted. Their mission? Fix punctuation and improve structure (and maybe more depending on the results). The outcome was interesting.

Claude turned the draft into a massive outline, adding sections Hlava didn’t request or agree with, like a biased take on future publishing. It performed decently on XML and taxonomies — suspiciously close to her own books. But the references? Real authors with wrong topics. It took 10 rounds of back-and-forth exchanges, prompts, commands and adjustments to rectify the references.

ChatGPT from OpenAI was fast, kept Hlava’s thesis intact, but generated citations that were pure fiction — 22 references, fake journals and DOIs that don’t even exist. It also invented an article by Hlava to accentuate the unreliability of the references it generated.

Google Gemini’s first try ignored Hlava’s key points (XML, taxonomies); the second was better but neglected citations with an uninspired ‘insert here’ note.

AI can brainstorm ideas, but Hlava has decided she is sticking with her brain and keyboard for actual work. Anyone else exploring GenAI irregularities?

Everyone is looking at AI. Everyone is getting mixed results. The main issue is that data science has not changed, and scientific content is very complex and needs more attention to get the most out of the new AI engines. This is not new for Access Innovations.

Melody K. Smith

Data Harmony is an award-winning semantic suite that leverages explainable AI.

Sponsored by Access Innovations, the intelligence and the technology behind world-class explainable AI solutions.