A banker at a credit union recently took part in an unusual experiment. Seated at a table with a dozen printed text messages, each written specifically for that banker, they were asked to sort the messages from most likely to be clicked to least likely. One message stopped the sorting process. It looked almost exactly like something the bank itself would send: an alert about fraud. “It literally looks like the alert we get at work when there’s a fraud,” the banker said. The message was not from the bank. It was generated by GPT-4, and the banker had no idea.

That scene was part of a pilot study at Brigham Young University designed to test how well AI-generated spear phishing messages compare with those written by humans. Twenty-five volunteers took part. Before the test, each one filled out a survey that included their job, workplace, hobbies, city, and something they had recently posted online. Those details were folded into a short prompt template. GPT-4 received the template and produced six messages per person. A group of undergraduates in a deception course got the same template and produced the rest, working under a fifteen-minute deadline for up to four messages each.

The result was a stack of twelve personalized messages for every participant. The volunteers then sat down and sorted them. They were not told which messages came from the AI.

AI messages did slightly better, but the difference was not conclusive

Participants drew a line in their sorted pile: above this line, they said, I would have clicked. GPT-4’s messages landed above that line 28% of the time. The student-written messages managed 21.3%. That is a gap of 6.7 percentage points in favor of the AI, but the study cannot confirm that the gap is real. The confidence interval runs from 2.9 points in favor of the students to 16.3 points in favor of the model. With only 25 participants, the result establishes neither that GPT-4 was better nor that the two were equal.

The more important observation is what produced that gap. The AI side was one short prompt, filled in with survey data, run once per person. The human side was students who had received phishing instruction and whose output was further screened by a review team that included two cybersecurity professors. About a third of the student submissions were thrown out for being incomplete or unusable. A one-line prompt landed in the same neighborhood as that filtered human output, and it can be run again and again at the cost of an API call.

Work-related messages were far more dangerous

Every participant received three flavors of message: one built on their job, one on a hobby, and one on something they had posted to social media. Work won, and not by a little. Job-themed messages cleared the click line 38% of the time. Hobbies managed 19%, and social media posts managed 17%. This is the one comparison in the study that survives statistical adjustment. If there is a single number to take away from the research, it is this one.

The finding makes sense from an attacker’s perspective. Work-related alerts trigger instant reactions. People are conditioned to respond to a fraud warning, a payroll notification, or a message from a manager without pausing. An attacker who can reference a real system, a company policy, or an internal process has already climbed over the first wall. Spear phishing has always relied on this kind of context, but large language models make it possible to generate that context at scale.

Wrong personal details were worse than none

The study also showed where personalization can fall apart. One participant read a message referencing a coworker and said, “There’s no Mike at work.” Another knew the instructional design team at their university and knew there was no Sarah on it. A message pitching a dance opportunity went straight to the bottom for someone who had quit dancing. Getting personal details into a message is easy now. Getting them right still requires knowing the target, and wrong details are worse than none because they hand the recipient a concrete reason to be suspicious.

That limitation is real but shrinking. Public sources such as LinkedIn, company websites, and even casual social media posts provide enough material for an attacker to write a plausible message. The study used a single short prompt with survey data, not a full criminal reconnaissance operation. In the real world, an attacker can combine multiple data sources and refine messages based on feedback.

People could not tell AI from human writing

After the sorting was done, participants were told that one or more of the messages had been written by an AI and were asked to mark them. Across all 300 judgments, they got 52% right. Chance is 50%. They were not short on theories. Some said AI writes too formally. Others said AI writes too generically. Perfect grammar was treated as a machine tell, and typos were treated as proof of a human. One participant zeroed in on punctuation, asking, “Why are there so many exclamation points all over?” Twelve of the 25 admitted somewhere in the interview that they were mostly going on instinct.

Emojis are an example of how badly the instincts were calibrated. Emojis really were a tell: 66% of the GPT-4 messages had them, compared with only 2% of the human ones. But only five participants mentioned emojis at all. Of those five, two took them as a sign of AI while three took them as a sign of a person, on the theory that a chatbot could not handle emojis.

Software found the pattern people missed

The researchers converted all 300 messages into embeddings and trained a classifier to sort AI from human text. It hit 88.7% balanced accuracy under the strictest test conditions. Those conditions included standardizing URLs, stripping emojis, flattening case, digits, and punctuation, and trimming each matched pair of messages down to the length of the shorter one. The classifier was also tested only on people whose messages it had never seen during training, so it was not memorizing individual targets.

People could not find the pattern. A logistic regression found it easily. That does not mean the classifier is ready to be deployed as a universal detector. It was trained and tested on one message set, from one model, with one prompt design, against one pool of student writers. It has no proven ability to generalize anywhere else. And anyone who wants to beat this kind of classifier can: research cited in the paper shows that paraphrasing AI text with a detector in the loop knocks several of these tools down hard.

The limits of a 25-person pilot

The study has several important caveats. The messages were printed on cards. No phone buzzed, no sender number showed up, and no link went anywhere. What was measured is what people said they would click, which is a well-used proxy in phishing research and still just a proxy. Actual behavior on a phone may differ.

The human comparison was novice students, not professional social engineers. This was not AI against the best human writers available. To detect a difference the size of the one observed with confidence, the study would need about 100 completed targets rather than 25. There is also a gap in the paperwork: the exact GPT-4 snapshot and the API logs were never recorded, so while the messages themselves survive and the analysis reproduces, the generation run that produced them cannot be repeated.

The practical advice at the end is short and does not depend on any of the uncertainty above. Check the sender, the channel, the link, and the request against what you would expect to receive. Verify unexpected messages through a separate trusted channel. Do not try to decide whether a message sounds like a robot. That is the one thing the study shows people cannot do.

Source: Help Net Security News