Contents
	Introduction
	Basic Assumptions
	How Utility Analysis Builds on These Assumptions
	What Is Needed to Complete a Utility Analysis
	Method
	Getting Ready for the Analysis
	Doing the Analysis
	Following Up the Analysis
	Learning More About Utility Analysis
	Example of a Utility Analysis
	Measuring Productivity and Determining Its Monetary Value
	Establishing the Dollar Value of Productivity
	Establishing the Dollar Value of Productivity Improvement
	Measuring the Course's Affect on COTR Productivity
	Determining the Course's Utility
	Assessing Return on Investment
	How Productivity Was Improved
	References
	About the Author
	Feedback Please Download PDF of Article

Introduction

Utility analysis is a quantitative method that estimates the dollar value of benefits generated by an intervention based on the improvement it produces in worker productivity. Utility analysis provides managers information they can use to evaluate the financial impact of an intervention, including computing a return on their investment in implementing it.

The concept of utility was originally introduced by Brogden (1949) and Brogden and Taylor (1950) and further developed by Cronbach & Gleser (1965). The concept has been researched and extended by Cascio (1982); Schmidt, Hunter, and Pearlman (1982); and Reilly and Smither (1983), among others. It was introduced as a method for evaluating the organizational benefits of using systematic procedures (e.g., proficiency tests) to improve the selection of personnel but extends naturally to evaluating any intervention that attempts to improve human performance.

Basic Assumptions

The first assumption of utility analysis is that human performers generate results that have monetary value to the organizations that employ them. This assumption is also the basis on which people claim compensation for the work they do.

The second assumption of utility analysis is that human performers differ in the degree to which they produce results even when they hold the same position and operate within like circumstances. Thus, salespersons selling the same product line at the same store on the same shift will show a variation in success over time with a few doing extraordinarily well, a few doing unusually poorly, and most selling around the average amount for all salespersons. This assumption is broadly supported in common experience and in research. It is, for example, the basis on which some performers demand and receive premium compensation.

The direct implication of these assumptions is that the level of results produced by performers in their jobs have different monetary consequences for the organizations that employ them. Performers are differentially productive and the productivity of performers tends to be distributed normally (Exhibit 1).

How Utility Analysis Builds on These Assumptions

The approach of utility analysis asserts that the utility of any intervention can be valued by determining how far up the productivity distribution the intervention moves the performer. The distance the performer is moved is translated into a productivity gain and the dollar value of that productivity gain is what is termed the utility (U$) of the intervention.

What Is Needed to Complete a Utility Analysis

In completing the analysis, the performer needs to generate the following:

• A method for measuring role productivity,
• A way to assign monetary value to role productivity,
• The distribution of productivity among performers of the role,
• The dollar value of a one standard deviation difference in role productivity (SD$), and
• A method to measure the intervention's impact on role productivity.

With these elements of information, the analyst can compute the utility of the intervention in dollars.

To accomplish the analysis, the analyst must be skilled in the methods of quantitative analysis in general and utility analysis in specific. This person needs to be aware of the variety of ways one can measure human productivity, determine its monetary value, and gauge the affects of interventions on participant performance.

Given that there are a variety of methods for computing utility, the exact resources needed for the task will depend on the method the analyst selects. The least set of resources anyone will need are:

• Access to the people who will be using the results of the study to make decisions;
• The identity of the intervention whose utility you will measure;
• A subject matter expert who is knowledgeable of the intervention;
• A description of each affected role including its duties, outputs, and success criteria;
• The compensation scale for each affected role; and
• A subject matter expert who is knowledgeable of the role(s) affected by the intervention.

Method

Getting Ready for the Analysis

1. Understand the people whose decision-making the study will support.
Tip: You need to meet the people who will use your study's findings in order to understand what information they are seeking, what decisions they will use the information to make, and any issues or concerns they may have about the study. You should also alert them to your ongoing need for their feedback on the methods you will propose for accomplishing the study. Assure them that you will guarantee that the methods you propose satisfy the professional criteria, but that their feedback is needed to ensure that the methods are also credible in their eyes and the eyes of anyone with whom they will share the results.

2. Learn about the intervention you will assess.
Tip: Identify the intervention whose utility you will measure and contact the subject matter expert who is knowledgeable about it. Learn about the intervention's purpose, target population, content, operations, cost, and any metrics used to measure its implementation and effects. Also, uncover what the thinking is about how the intervention affects the productivity of the performers it targets. With these facts, you can determine what information needed for the analysis exists and what information you will need to develop.

3. Learn about the role(s) whose productivity is affected by the intervention.
Tip: Obtain a description of each affected role. Contact the subject matter expert who is knowledgeable of each role. Learn each role's purpose, duties, outputs, and success criteria. You also need to understand how the role is valued from a compensation perspective. For example, is compensation linked to output or is it paid as a salary? You will want to understand, as well, how the company values the output of each job. If the output is sold, is it valued by cost or price? And you need to uncover how much responsibility each role has for the outputs its performers produce. Finally, for each job that is salaried, obtain its compensation scale and the average salary paid to its incumbents. If salaries are not normally distributed, you may need to obtain either the median or modal salary instead of the mean.

4. Determine how to measure the productivity of the performers of each role.
Tip: You will need to develop a productivity measure and a method for determining the status of each role incumbent on the measure. You will need to use your understanding of each affected role and the assistance of its subject matter expert. The subject matter expert will have to approve the method of measurement you devise, otherwise your approach to measuring productivity will not have credibility in the workplace.
In devising the productivity measure, it is preferable to base the measure on production of correct outputs—for example, the total amount of sales generated less returns or the number of welds made per unit of time less the number of welds that fail inspection. Outputs are the tangible contributions a role makes to an enterprise and measuring the quantity, quality, and complexity of outputs generated by performers is usually a measure of productivity that is readily accepted.
Sometimes, however, a workplace will not accept a measure of productivity that is tied to outputs. In these situations, you still need a way to measure how well the role is performed. Sometimes supervisor ratings of successful performance are used or multirater approaches that use rating of supervisors, peers, and subordinates (when appropriate).
If the workplace will not agree that different performers achieve different levels of success or that the level of a performer's success in performing the role can be measured, then the utility analysis cannot be done.
Once you have devised a measure of productivity, plan how you will gather information about the status of role incumbents on the measure. Your method must be feasible meaning that its cost must be reasonable, its result credible, and its burden on participants acceptable.

5. Determine how to value role productivity in dollars.
Tip: The method you choose will be determined by how you measure productivity. If you use a method that calibrates outputs produced, then you will assign monetary value based on the dollar value of the outputs. If the job produces an interim output, some component of a larger final product, then determine the component's contribution to the total product and determine the value of the role's output by adjusting the value of the final output. Material outputs can be valued based on cost or sales price. Service outputs that are used in-house (e.g., a marketing plan, a processed personnel action) can be valued using market pricing—that is, what it would cost to purchase the service from external sources.
If you are not using a measure of productivity that is tied to output, then you can use the typical salary paid for the job (i.e., mean, median, or mode). Salary is acknowledged as reflecting the value a role contributes to a company.

6. Decide how to measure the affect of the intervention on role productivity.
Tip: Basically, you need to find a mathematical bridge that relates participation in the intervention and change in role productivity. There are very many ways to accomplish this. One way is to use a control group comparison. Here, you identify two sets of people who are comparable in all important ways except that one set went through the intervention and the other did not. You compare the differences in productivity of these two sets of people. If the intervention was effective, the people who went through it will have higher productivity scores and the difference between the groups will represent the intervention's impact on productivity. Another way is to use correlational methods to associate some indicator of participation or benefit from the intervention with scores on role productivity. Be sure that the information with which you are working satisfies the requirements of the statistical method you use and that your approach makes sense to the people who will use the results of the analysis. Your solution needs to satisfy both professional standards and credibility to provide benefit.

7. Create a plan for the utility analysis.
Tip: Be sure your plan documents how you will produce each of the information elements needed to accomplish the utility analysis. Include in it any decision rules you will apply in making judgments. For example, if you are also computing a return on investment ratio, what rule will you apply to decide if the ratio is positive? Will 1.0 be sufficient? Will the ratio need to be 2.0 or higher? In a professionally conducted analysis, all decision rules must be documented prior to the study.

Doing the Analysis

1. Determine the productivity of performers.
Tip. Execute your plan for measuring the productivity of current role incumbents.

2. Determine the dollar value of a one standard deviation difference in role productivity (SD$).
Tip. Distribute the productivity scores you gather. Confirm the distribution is essentially normal and compute its mean and standard deviation. If the distribution is not normal, use a transformation method (e.g., z-transformation) to normalize it. Apply your method for valuing role productivity. Derive the dollar value of productivity achieved by average performers and the dollar value of a one standard deviation difference in productivity (SD$).

3. Compute the effects on performer productivity associated with the performer's participation in the intervention being evaluated.
Tip. Apply your method for measuring the affect of the intervention on productivity. Determine how many standard deviations of change in worker productivity the intervention produces (SD).

4. Compute the dollar value of productivity improvements generated by the intervention.
Tip. The dollar value of productivity improvements generated by the intervention is the intervention's utility (U$). To compute utility, multiply the number of standard deviations of change the intervention produces in worker productivity (SD) and the dollar value of a one standard deviation difference in productivity (SD$) (SD x SD$ = U$).

Following Up the Analysis

1. Add context to the findings.
Tip. Statistical methods are systematic and, when properly applied, produce reliable results. They do not, however, guarantee meaningful results. Sometimes quantitative relationships are found for which there is no reasonable explanation. One reason this occurs is that you rarely can control all the possible factors that may influence whether a found relationship is valid. Sometimes what your data says is causing the effect is not, rather some other intervening factor which you have not identified or controlled may create the appearance of a relationship that does not actually exist. One way to eliminate this possibility is through the use of controls during the process of relating participation in the intervention and changes in productivity. Another way is to explore whether the content of the intervention or the experiences of its participants suggest a meaningful mechanism for the effects your analysis detects. Study the intervention to see if any aspects of it suggest such a mechanism. Gather or review existing accounts of the actual experience of people participating in the intervention. Their experiences will help provide qualitative information that may either suggest a lack of reasonableness to the quantitative findings or provide a bases for making the findings understandable. In analyzing the intervention itself, we look for content or activities that previous research supports as reliably affecting human performance. With respect to gathering participant experiences, our typical approach is to use focus groups in which we gather the participants' observations of what happened, whether in their viewpoint it affected their performance, and how it affected their performance.

2. Report the results of the analysis.
Tip. Be sure to describe the methods you used to generate your findings and the rationale for each. After you draft your report, obtain feedback on it from the subject matter experts of the role(s) affected by the intervention and the intervention itself. This feedback may identify issues you need to address and assist you in improving the communication of your results.

Learning More About Utility Analysis

Read the example of a utility analysis we completed to evaluate the monetary return of a training intervention. Use this example to clarify how the steps are performed. Also, study each of the references listed at the end of this article. They provide a sound introduction to the various methods of utility analysis one may employ.

Example of a Utility Analysis

We were asked to evaluate a contracts management course offered on a fee-for-service basis by the human resource department of a government agency. The course trained contract officer's technical representatives (COTRs) in how to specify requirements, build a request for quote or proposals, evaluate bidders, select and contract with the best supplier, manage contract performance, and ensure the delivery of the needed products or services on time, at cost, and to specifications. One of the questions being asked was whether the course returned a monetary value greater than its cost. We proposed a utility analysis as the means to assess the monetary benefits produced by the course and a return on investment analysis to determine the ratio of benefits received to the cost expended. Prior to these evaluations, we determined that the content offered by the course was relevant to the COTR role and that the course participants did demonstrate increased proficiency in their performance as a result of completing the course.

Measuring Productivity and Determining Its Monetary Value

With the role identified, we studied the job it accomplished by reviewing its tasks, outputs, and performance expectations. No measure of productivity existed—yet the means for deriving a measure appeared evident. First, the COTR role had a defined output and criterion for judging success. COTRs were responsible for successfully satisfying a product or service need within their agency through contracting. Successful satisfaction of the need meant the timely delivery of products and services that met technical specifications and the accomplishment of these ends at the cost specified. Second, there was a monetary value associated with the output. The dollar value of every contract a COTR managed was systematically determined. Third, there was a logical way to relate the monetary value of the role's output and its success criterion. A COTR realized the value of a contract to the degree that the contract was concluded on time, at cost, and to specifications. Conversely, to the degree it was not concluded on time, at cost, and to specifications, monetary value was lost.

While the basic logic was sound, conversations with incumbents and supervisors quickly revealed that while the COTR was responsible for the contract, sometimes he or she was not free to exercise complete control over its contents or the decision-making associated with it. Therefore, some amount of the value of the contract was outside the control of the COTR and its realization or loss should not be credited to the performer. We also learned that contracts sometimes yielded benefits greater than their face value and that this could be the result of the COTR's forward thinking, selection of the means for accomplishing the contract, speed of execution, and other factors.

To measure role productivity, we developed and tested the COTR Productivity Rating Form. This form measured the degree to which each COTR brings in his or her assigned contracts at cost, on time, and to specification. It also calibrated the importance of each of these factors for the contracts managed and the typical degree of control over each contract the COTR has. The different component measures were converted into an overall productivity score. This score was a percentage that represented the degree to which each COTR realized the value of the contracts he or she manages on a yearly basis.

The COTR Productivity Rating Form was sent to the 266 supervisors of COTRs randomly selected so that the ratings would reflect the status of COTRs in the general population. One hundred and thirty (130) responses were received (48.9% response rate). The response level provided estimates of productivity that were accurate to +/- 5% at a 95% level of confidence.

Establishing the Dollar Value of Productivity

We completed three steps to determine the dollar value of improved productivity. First, we distributed the productivity scores achieved by COTRs to determine their average level of productivity and the productivity levels of performers at the 15th and 85th percentiles. Exhibit 2 depicts the distribution of COTR productivity. The average performer is 81.65% successful in extracting the controllable value from the contracts he or she manages. In contrast, the exemplary COTR realized 94.04% of the value from the contracts he or she manages and the poor performing COTR extracted only 69.26%.

To calibrate the value of productivity in dollars, the study used the median face value of contracts managed by COTRs during one year as modified by the control the COTR has over the outcome of the contracts. The degree to which a COTR brings in his or her assigned acquisitions at cost, on time, and to specifications determines how much of the controllable dollar value of those contracts is realized. The median face value of contracts fulfilled per year by COTRs was $500,000. Corrected for the degree of control COTRs have over outcomes, as perceived by their supervisors, the median potential single year benefit a 100% productive COTR produces is $397,525. By multiplying the average actual productivity of COTRs (81.65%) against the controllable dollar value of the contracts a COTR manages on a yearly basis ($397,525), the study estimated the dollar benefits generated by the average performing COTR at $324,583.13. Poor performing COTRs— that is, performers achieving at or below the 15th percentile of all COTRs—generated only $274,344.10 of value each year. Exemplary performing COTRs, defined as incumbents whose productivity was at or above the 85th percentile of all COTRs, generated $373,895.44 of value.

Establishing the Dollar Value of Productivity Improvement

To determine the monetary value of improvement in productivity, the study computed the dollar value of one standard deviation in change (SD$) in role productivity. The SD$ for the current distribution of performers is $49,239.04. This means that if some intervention advanced the productivity of a COTR by one standard deviation, that COTR would generate $49,239.04 in additional benefits to the agency each year.

Measuring the Course's Affect on COTR Productivity

The correlation between COTR's job proficiency and productivity ratings served as the mathematical bridge for estimating the course's impact on performer productivity. The elements required to use this bridge were the amount of proficiency change produced by the course, the regression coefficient (beta) relating job proficiency scores to productivity ratings, and the standard deviation of productivity scores. Applying these elements, the course advances COTRs upward in productivity by .1547 standard deviations (Exhibit 3).

	Exhibit 3. Measuring the Course's Impact on COTR Productivity
	Change in Proficiency Produced by the Course	Regression Coefficient (Beta)	Change in Productivity Produced by Increased Proficiency	Standard Deviation (SD) Difference in Productivity Scores	Change in Productivity Expressed in Standard Deviation Units
	1.02	1.81	1.84%	11.916%	0.1547

Determining the Course's Utility

As stated above, utility is the dollar value of the increased productivity of a single COTR that is generated by the course. To determine the utility of the course, the study translated the distance the course advanced COTRs along the productivity continuum into dollars. As reported, the course advanced COTRs .1547 standard deviations up the productivity continuum. We previously determined that one standard deviation change in productivity has a monetary value of $49,239.04. Multiplying this amount by the .1547 provides us the course's utility (.1547 x $49,239.04 = $7,617.27). This figure ($7,616.18) is the dollar value of the improvement in productivity evidenced by each COTR as a result of training (Exhibit 4).

	Exhibit 4. Computing the Utility (U$) of Contracts Management Training
	Change in Productivity Expressed in Standard Deviation Units	Dollar Value of a 1 Standard Deviation Change (SD$) in Productivity	Dollar Value of Productivity Improvement Produced by COTR Course
	0.1547	$49,239.04	$7,616.18

Assessing Return on Investment

The return on investment was computed using the method of dividing the dollar value of the productivity benefits generated by the course by the cost of participating in the course (benefit to cost ratio). In this study a desirable ROI was defined as any value greater than 1. The study determined the per student cost for completing the COTR course. It added the fee charged to departments for each COTR taking the course with the cost of lost opportunity associated with the COTRs not performing their regular job during the 10-day period of the instruction. This fee ($700) included all expenses associated with the course. The cost of lost opportunity was computed by dividing the salary of the typical COTR who participated in the course (GS-14, Step 1) by the number of hours that define full time employment in the Government (2,087). This per hour cost is then multiplied by the 80 hours that the COTR is off the job. The opportunity cost per student was $2,385.63. The total cost for participating in the course was computed as $3,005.63 per COTR.

The ROI for the course was 2.53 ($7,616,18/$3,005.63) for one year of COTR performance following completion of the course. This means that for every dollar invested in completing the course, the sponsoring department receives $2.53 in benefits the first year. Any reasonable assessment of return should recognize that the benefits of the course extended forward. Given the general stability of the content the course teaches, a three year period for return on investment was considered conservative. Within 3 years, the total productivity improvement benefit is $22,848.54 and the ROI is 7.60— meaning, for every dollar spent, $7.60 in agency benefits is generated (Exhibit 4).

How Productivity Was Improved

The completion of the two focus group discussions with COTRs who completed the contracts management course provided insight into the course's mechanism of impact. Participants uniformly confirmed their experience of benefit from the course. They listed 17 ways their performance was improved by what they learned. One major element they emphasized was that the course provided a cognitive map of the contracting process that allowed them to see ahead, to plan and prepare, and feel more confident in the conduct of their role. As well, the course equipped them to produce the products required by the role and to know how to judge the adequacy of each product. Also stressed was the learning about the various players in the contracting process, their responsibilities, the importance of communicating with them, and the importance of creating a teamed effort. Equally important, the course participants felt they grasped the principles that ensured the integrity of the contracting process and that they were able to see how these principles apply in different contracting situations. Finally, participants also reported the training coursebook provided with the course served as a continuing learning resource that they turned to as they encountered new contracting experiences.

References and Suggested Readings

Bernstein, Allen L. (1966) A handbook of statistical solutions for the behavioral sciences. New York: Holt, Rinehart and Winston.

Brogden, H.E. & Taylor, E.K. (1950) The dollar criterion: applying cost accounting concepts to criterion selection. Personnel Psychology, 3, 133-154.

Burke, Michael J. & Frederick, James T. (1986) A comparison of economic utility estimates for alternatives SDy estimation procedures. Journal of Applied Psychology, 71, 334-339.

Byron, James & Vitalo, Raphael L. (1991) Quality improvement through exemplar-based productivity analysis. American Productivity & Quality Center, Brief #82, March 1991.

Cascio,W. (1982) Applied Psychology in personnel management. Reston, VA: Reston Publishing Company (see Chapter 7, Utility: the concept and its measurement).

Cronbach, L.J. & Gleser, G.C. Psychological tests and personnel decisions. (2nd ed.), Urbana: University of Illinois, 1965.

McDaniel, Michael; Schmidt, Frank L.; & Hunter, John E. (1987) Job experience as a determinant of job performance. Paper presented at the 95th Annual Convention of the American Psychological Association, August 1987.

Myers, Jerome L. (1966) Fundamentals of experimental design. Boston: Allyn and Bacon, Inc.

Reilly, Richard R. & Smither, James W. (1983) An examination of two alternative techniques to estimate the standard deviation of job performance in dollars. Journal of Applied Psychology, 70, 651-661.

Rossi, Peter H.; Freeman, Howard E.; & Wright, Sonia R. (1979) Evaluation: a systematic approach. Beverly Hills: Sage Publications.

Schmidt, F.L.; Hunter, J.E.; & Pearlman, K. (1982) Assessing the economic impact of personnel programs on workforce productivity. Personnel Psychology, 35, 333-347.

U.S. Department of Health Education and Welfare (1975) A practical guide to measuring project impact on student achievement. Washington, DC: U.S. Government Printing Office (Stock Number 017-080--1400-2).

Published April 2004, Revised 2022

Download PDF of Article

Help Us Provide You Better Content.
Tell us your thoughts about this article.
Be sure to name the article in your feedback.