What Makes AI Role-Play Practice Real?

A recent study of AI conversation training produced two findings that belong together.

The participants got better.

The simulator was also frequently unrealistic.

That combination is more useful than a clean success story because it gets closer to the real question facing anyone building or buying AI role-play training. An AI counterpart does not have to be perfect to create useful practice. But some of its imperfections attack the training mechanism itself.

The study, published in Science Communication in September 2026, involved 37 science students and early-career researchers. Each participant held two voice conversations with an AI simulator called DiaLogic. Between the conversations, the system gave them feedback on their performance.

Independent expert evaluation found statistically significant improvement in overall dialogic performance and in three areas: content, interpersonal rapport, and perspective-taking and listening.

The sample was small. There was no control group. The second conversation followed immediately after the first and used the same general scenario. We therefore cannot say that the AI caused durable improvement, or that the gains would transfer to a real conversation days later.

But the study gives us something that many discussions about AI training lack: observed performance before and after a practice-feedback-repeat cycle, assessed by people other than the AI itself.

Then come the failure modes.

They may be even more instructive.

DiaLogic was instructed to behave as a curious layperson with no prior scientific knowledge. That role mattered because the student was supposed to practise explaining specialised research to a non-expert.

In 18 of the 74 analysed transcripts, or 24 percent, the simulator broke that assumption by understanding specialist terminology that a layperson should not reasonably have understood.

One example involved the term “Marangoni flow.” Instead of behaving like a non-expert, the AI recognised the concept and began discussing the underlying physics.

That sounds like a small model error.

For training, it is structural.

If you are practising how to explain something to somebody who does not understand it, and your counterpart quietly understands it anyway, the exercise stops testing the intended skill.

The same problem appears immediately in negotiation.

Suppose a buyer is practising with the commercial director of a specialist supplier. The supplier is meant to know its own manufacturing process but have limited visibility into the buyer's internal cost target.

If the AI suddenly knows the target, it is no longer playing that supplier.

Or imagine a supplier is supposed to be under capacity pressure and unwilling to accept additional volume without a commitment. Halfway through the conversation, it forgets that constraint and starts offering capacity freely.

The conversation might still sound fluent. The role-play has failed.

Realism in simulation therefore depends on more than natural language. The counterpart has to know the right things, not know the wrong things, care about the right things and continue doing so after twenty turns of conversation.

The researchers found another problem at much larger scale.

Five participants explicitly complained that DiaLogic sounded overly enthusiastic and praised them too readily. When the transcripts were examined, repeated enthusiastic compliments appeared in 67 of the 74 conversations.

That is 90.5 percent.

Anyone who uses general-purpose language models regularly will recognise the behaviour. AI assistants are usually designed to be cooperative. They acknowledge, encourage and move the conversation forward.

Those are good characteristics for an assistant.

They can be destructive characteristics for a simulation.

A supplier negotiating a renewal should not congratulate the buyer for every argument. An irritated stakeholder should not become warm because the user has produced a well-structured sentence. A difficult employee conversation loses its value if the simulated employee constantly signals that the manager is doing wonderfully.

The user needs enough resistance to discover what they actually do when cooperation is not immediately available.

There is a subtle point here. Real counterparts are not hostile all the time either. A simulator that opposes everything mechanically is just another bad role-play.

The objective is credible behaviour.

If an argument would genuinely shift the counterpart's position, it should. If it would not, praise should not substitute for movement.

Participants did not complete a simulation, receive a report and leave.

They spoke once, received feedback, reflected on it and immediately spoke again.

The conversations lasted only about five to eight minutes each.

That sequence matters.

A great deal of professional training ends just when practice should begin. Someone performs an exercise, receives feedback and goes back to work. The feedback may be accurate, but the learner never has to convert it into a different behaviour while the original experience is still fresh.

The DiaLogic study built the second attempt directly into the intervention.

We cannot isolate which part produced the measured improvement. It may have been the feedback, the repetition, increased familiarity with the scenario, the simulation itself or some combination of them. The absence of a control group matters here.

Still, the design points toward a sensible training principle: feedback becomes more useful when there is an immediate opportunity to act on it.

A buyer who learns that they conceded too quickly after a supplier invoked capacity risk should not merely read that observation. They should hear the capacity argument again and try a different response.

That second attempt is where information has a chance to become behaviour.

AI training systems are often judged by the quality of the analysis at the end.

That is understandable. Feedback is visible. It can be scored, summarised and compared.

But the conversation generating that feedback comes first.

If the counterpart accepts an unrealistic outcome, the resulting evaluation is based on a distorted negotiation. If it feeds the learner information the real counterpart would never reveal, the learner is practising with hidden assistance. If it praises every move, the user receives social signals that would be absent in the real conversation.

The feedback may describe the transcript accurately and still reinforce the wrong behaviour.

For negotiation training, this is especially important because so much of the skill lies in adapting to resistance.

A buyer should learn what happens after they challenge the supplier's cost basis. Does the supplier produce evidence? Escalate to supply risk? Change the subject? Offer a trade? Hold firm?

The quality of the next move determines the quality of the practice.

It is tempting to solve this with a better prompt.

“Act as a tough CFO.”

“Do not give in easily.”

“Challenge the user.”

That helps. It does not fully define a role.

A useful negotiation counterpart needs a position underneath the personality.

What outcome is commercially acceptable? Which variables can move? Which cannot? How important is the account? What happens if there is no agreement? Which arguments does the counterpart believe? What information are they allowed to reveal? How does their behaviour change as the negotiation progresses?

“Tough” is a speaking style.

Those other elements create a negotiating position.

This is one reason AI role-play becomes more difficult as the conversation gets longer. The system has to generate natural speech while maintaining constraints established much earlier. A persuasive learner will also actively try to change the counterpart's position, which is exactly what should happen in a negotiation.

The simulator therefore has to distinguish between legitimate persuasion and simply abandoning the role.

That is a much harder problem than sounding human.

Ten participants in the study, 27 percent, mentioned the lack of physical presence and nonverbal signals such as facial expression, gesture and eye contact as a limitation.

That criticism is fair.

A voice simulation does not reproduce every part of sitting across a table from someone. Even very good audio cannot show a counterpart leaning back after an aggressive anchor or exchanging a glance with a colleague before answering.

That does not make voice practice useless. It defines its boundary.

Training should resemble the target situation closely enough to exercise the behaviour that matters. For a phone negotiation or remote supplier call, voice may reproduce much of the interaction channel. For a complex in-person executive negotiation, there are additional signals it cannot currently provide.

Simulation should not claim fidelity it does not have.

The tempting headline is that researchers have shown AI communication training works.

That goes further than the evidence.

Thirty-seven participants improved between two closely spaced conversations after receiving AI feedback. That is promising evidence from an exploratory study. It is not yet proof of durable learning, transfer to real-world performance or superiority over good human coaching.

The more defensible lesson is also more useful.

AI conversation practice can produce measurable changes over a very short practice-feedback-repeat cycle. At the same time, generic LLM behaviour can undermine the realism of that practice through excessive agreeableness and failure to maintain the assigned role.

Both findings should influence how AI training is designed.

For Voice2Evolve, this is why the useful unit is not simply a voice model pretending to be a supplier. The counterpart needs a defined persona and commercial position, the scenario needs constraints, and the analysis needs to be grounded in the conversation that actually occurred. Repetition then gives the user somewhere to apply the feedback.

The aim is not to make the AI endlessly difficult.

It is to make the resistance credible.

A good sparring partner should change its mind when you give it a reason to change its mind. It should hold its ground when you do not. It should not know what it has no reason to know. And it should never congratulate you merely for taking part.

That is when role-play starts becoming practice.

Yacobson, E., Baram-Tsabari, A., & Amir, O. (2026). Scaling Dialogic Science Communication Training With Large Language Models. Science Communication. Primary study