BACHELOR'S THESIS · GENERATIVE AI & CUSTOMER FEEDBACK

Human or Computer? A Qualitative Analysis of AI in Online Customer Feedback Management

A qualitative study investigating how people distinguish human-written from AI-generated responses to online reviews, which characteristics shape those judgments, and where generative AI can support customer feedback management.

CORE IDEA

Human.
AI.
Indistinguishable?

Role

Independent Research

Context

Bachelor's Thesis · UZH

Method

Qualitative Content Analysis

Data

504 Participants · 4,000+ Evaluations

01 / CONTEXT

Customer responses are becoming easier to automate — but authenticity still matters.

Online reviews have become an important part of how companies understand and respond to customers. At the same time, writing individual responses to large amounts of feedback requires substantial time and resources.

Generative AI offers a way to automate parts of this process. The central question is therefore not only whether AI can generate a response, but whether that response appears appropriate, personal and human to the people reading it.

RESEARCH QUESTIONS

Which indicators of authorship can be identified in responses to online reviews, and which quality characteristics can be derived from them?

How can generative AI help make online customer feedback management more efficient?

02 / MY ROLE

An independent qualitative research project.

I conducted the thesis independently at the Institute of Informatics at the University of Zurich within the field of Business Information Systems.

The work focused on analysing existing survey data from the ReAdvisor research project. I prepared the data for qualitative analysis, developed and applied a coding system in MAXQDA, aggregated the resulting codes and interpreted the findings through both authorship judgments and communication theory.

Supervisor

Dzmitry Katsiuba

Research Context

ReAdvisor Project

University

University of Zurich

DATASET

Thousands of judgments about who wrote the response.

504

survey participants

4,000+

evaluated review-response pairs

~1,300

unique review-response pairs

850+

distinct reviews

Participants evaluated review-response pairs and judged, among other things, whether the response had been written by a human, a computer, or through a combination of human and computer involvement. Their open-ended explanations of these judgments formed the central qualitative material for my analysis.

03 / RESPONSE CREATION

Four ways of creating a customer response.

The ReAdvisor survey compared responses produced under different levels of human and AI involvement.

Human only

A person wrote the response manually without AI support.

Human + AI tools

The response was written by a person supported by tools such as sentiment analysis and quality checks.

AI + Human

AI generated the initial response and a person could adapt it afterward.

G-AI

AI only

The response was generated entirely by AI, including variants using a dedicated model and GPT-3.

04 / METHOD

Turning open-ended responses into structured insights.

Prepare

I reorganised the German and English survey data into MAXQDA-compatible datasets containing participant, review, response and author-setting information.

Code

Reviews, response texts and open-ended authorship explanations were manually coded using a qualitative code system in MAXQDA.

Aggregate

Individual codes were combined through a bottom-up approach to derive broader quality characteristics and recurring patterns.

Interpret

The resulting characteristics were compared with the actual authorship of each response and later mapped to Grice's conversational maxims.

AUTHORSHIP SIGNALS

What made a response feel human — or artificial?

HUMAN

Personal and context-aware

Individualisation

Appropriate reaction to the review

Natural or detailed content

Positive and suitable tone

Direct reference to specific details

COMPUTER

Standardised and impersonal

Generic or automated wording

Impersonal responses

Irrelevant or unnecessary information

Advertising

Standard phrases and formulaic language

230

of 517

WHEN AI LOOKS HUMAN

GPT-3 was frequently mistaken for a human author.

In 230 of the 517 evaluated GPT-3 review-response pairs, participants selected a human as the author. This made GPT-3 one of the most striking results in the analysis: characteristics that participants associated with human writing were already being reproduced convincingly by generative AI.

05 / QUALITY

Human-like writing still depended on relevance and context.

REVIEW REFERENCE

GPT-3 showed a comparatively strong ability to refer directly to the content of the review, while the dedicated AI model performed considerably worse on this dimension.

HALLUCINATIONS

The dedicated AI model produced substantially more hallucinations in the analysed responses than GPT-3, highlighting the importance of response accuracy.

LANGUAGE

GPT-3 still showed recognisable weaknesses such as standard phrases, formulaic wording and limited contextual knowledge.

COMMUNICATION FRAMEWORK

From quality characteristics to Grice's maxims.

To interpret the findings from a communication perspective, I mapped the derived quality characteristics to four principles of effective communication.

Quality

Truthfulness, content quality and avoiding incorrect information.

Quantity

Providing enough information without unnecessary or excessive content.

Relevance

Individualisation, appropriate content and direct reference to the customer's review.

Manner

Clear language, appropriate tone, structure and style.

KEY FINDINGS

AI could already imitate many characteristics people associated with human responses.

Personalisation signals humanity

Individualisation and appropriateness were among the most important characteristics participants associated with human authorship.

GPT-3 performed surprisingly well

It frequently produced responses that participants believed were human and compared favourably with the dedicated AI model across several analysed dimensions.

Professional humans still mattered

Professional authors remained stronger in several areas, particularly where experience, contextual knowledge and deliberate personalisation were important.

IMPLICATION

Efficiency should not come at the cost of authenticity.

The results suggest that generative AI has substantial potential to support companies in managing online customer feedback. However, successful use depends on maintaining relevant, personalised and contextually appropriate communication rather than simply generating text at scale.

06 / LIMITATIONS

A snapshot of generative AI at a fast-moving point in time.

The participant sample represented a broad demographic range but was still limited in size and geography.

Participants did not have full contextual knowledge about the companies or real situations behind the review-response pairs, which affected their ability to verify some claims.

Generative AI was developing rapidly even while the thesis was being written, meaning newer models could produce different results.

07 / OUTCOME

Generative AI showed clear potential for customer feedback management.

The analysis identified concrete characteristics people use when judging whether an online response feels human or computer-generated, including tone, language, individualisation, relevance and appropriateness.

It also showed that generative AI — particularly GPT-3 in this dataset — could already generate convincingly human-like responses, while still leaving room for improvement compared with experienced professional authors.

PERSONAL TAKEAWAY

The thesis gave me practical experience in qualitative content analysis, systematic coding, working with large survey datasets and translating unstructured user feedback into structured findings about technology, communication and user perception.

BACK TO SELECTED WORK

View projects→