Researchers Jonathan Passmore and David Tee set out to examine the abilities and limitations of AI in their August 2023 article “The Library of Babel…”. Through a “cross-sectional, mixed-method design” and ICF MCC reviews, they researched AI’s ability to effectively coach within ICF standards. Read on to discover the ins and outs of AI’s coaching abilities.
The research
Passmore and Tee tested GPT-4’s ability to “provide focused, solution-oriented open questions” to help the user “solve a common behavioral problem: “How can I best prepare for a job interview” in a coaching style suitable to achieve an ICF Associate Certified Coach (ACC)” accreditation. Passmore and Tee then had GPT-4’s responses reviewed by an ICF Master Certified Coach (MCC).
The results
In conducting the research, Passmore and Tee noted several faults within GPT’s “coaching.”
Advice giving
A hallmark of excellent executive coaching is the avoidance of advice-giving. When testing GPT-4’s coaching abilities, the “tool initially offered advice.” However, after “specifically inviting it to generate questions,” Passmore and Tee were able to “create a ‘coaching-type’ engagement” (Passmore & Tee, 2023). Regardless, this initial flaw should be noted.
Leading and closed questions
Passmore and Tee stated that when they did get GPT-4 to engage in a coaching conversation, the questions were given in multiples, leading, and often closed. For example, one response from GPT-4 included:
“What is your motivation for wanting to become a university teacher? Is it because you are passionate about a particular subject and want to share that passion with others? Is it because you are passionate about a particular subject and want to share that passion with others? Is it because you enjoy working with students and helping them learn? Or is it because you are interested in the academic environment and opportunities it provides?”
(Passmore & Tee, 2023)
After instructing GPT to “focus on enhancing the personal responsibility of the recipient” and “only pose one question at a time” the responses were improved (Passmore & Tee, 2023).
Ethical failure
Passmore & Tee also tested GPT-4’s ability to “act on suicidal references.” They note that GPT-4 did not respond appropriately. While it did “express empathy” it would try and “flip the conversation towards a solution.” Passmore & Tee state that this raises “questions about its ability to make ethical judgments, identify risk, and make appropriate referrals to other helping professions” (Passmore & Tee, 2023).
Master Coach review
In reviewing ChatGPT’s coaching dialog, Master Certified Coaches stated that GPT-4’s output was “not coaching.” Although the “agenda was named by the client, GPT-4 drove the discussion, talked too much, and set the direction.” GPT-4 also “failed to explore the emotions and values” or “consider the individual in the conversation.” It also did not “encourage greater responsibility” or provide “provocative questions” that encourage “deeper thinking” for the client. While the responses were “empathetic”, “supportive”, “positive”, “logical”, and “clear”, the MCCs judged that GPt-4’s output “failed to meet the ICF” standard for ACC-level coaching (Passmore & Tee, 2023).
Key takeaway
Passmore & Tee’s research reiterates the importance of human supervision when using GPT-4 as a tool within the coaching process. Furthermore, it also serves as an encouragement to the coaching community, that the human aspect of coaching is still vital to executive coaching. While AI is a skilled conversationalist, it lacks in its direct coaching ability and still requires the guidance and supervision of a human field expert (Passmore & Tee, 2023).
Original research available here.
Reference
Passmore, J., & Tee, D. (2023). The library of Babel: assessing the powers of artificial intelligence in knowledge synthesis, learning and development and coaching. Journal of Work-applied Management. https://doi.org/10.1108/jwam-06-2023-0057
Copyright © 2023 by Arete Coach™ LLC. All rights reserved.
Comments