Measuring Agency Quality: From Perception to Metric – Dr. Florian Bresler

Why quality cannot be measured directly

A communication service emerges in the exchange with the client and cannot be checked at the end of a line like a component. The QMC model therefore treats communication quality as a latent variable: it is not directly visible but inferred from several statements that describe its prerequisites (enablers) and its effect (success). This approach is common in marketing and service research because satisfaction, trust and attitude cannot be observed directly either. What matters then is that the statements are chosen and tested with care.

Step 1: Categories from the literature

From 31 studies on the prerequisites of good communication and 24 studies on measuring success, the dissertation derived 26 categories: four per enabler and three or four per success measure. The measurement thus stands on a broad professional foundation and not on the opinion of individual practitioners.

Step 2: Preliminary study with practitioners

Agency executives rated the relevance of each category on a scale from 0 to 3. Categories clearly below the overall mean of 1.70 were dropped, mostly because of overlaps or lack of clarity. Employee motivation, for example, was dropped as a success measure because employee topics already sit on the enabler side of the model and would otherwise be counted twice. Six categories were dropped, 20 measures remained. The preliminary study thus works like a filter: it keeps categories out of the questionnaire that practitioners consider unclear or counted twice. The financial indicators stayed in the model despite concerns, although respondents described them as speculative and measurable only in the long term.

Step 3: Survey and structural equation model

209 complete questionnaires from clients were evaluated, each on a completed project worth more than 100,000 euros. Twelve statements concerned the enablers and were rated from 1 (poor performance) to 5 (top performance); eight concerned success in the client's own company, from 1 (no agreement) to 5 (maximum agreement).

A structural equation model estimates how strongly each prerequisite explains communication quality and how strongly quality in turn explains success. In the study, a path with p < 0.10 counts as significant. Because the initial model was only partly confirmed, it was simplified in three steps down to the final model. What dropped out is a result in itself. For each factor, one statement serves as the reference against which the others are weighted; this is why some paths have no significance value of their own.

What means do not reveal

All 20 statements have means between 3.14 and 3.80. The level of a rating says nothing about whether it explains quality. Customer loyalty received the highest mean and still dropped out of the model. Knowledge management was rated better than any leadership trait and remained without a significant relationship to quality. Anyone who wants to measure agency quality therefore needs both: the rating of individual statements and the question of which of them actually explain the overall judgment.

What can be adopted for your own agency

A structural equation model needs a large sample and is too elaborate for everyday agency work. The principle can nevertheless be scaled down. The following hints are a transfer, not a finding of the study:

• Collect judgments from clients instead of internal self-assessments, about a specific, completed project.
• Phrase a few statements on a uniform scale from 1 to 5, one each on leadership, team, collaboration and trust.
• Additionally ask for an overall judgment and check over several projects which statement is most closely related to it.
• Discuss the results in a learning session and record measures for the next project.

Two limits remain: all values describe perception, not balance sheet or measurement data, and satisfaction judgments tend to be friendly. More telling than the absolute value is therefore the comparison over time and between statements.