Best Practices in Employment Selection

In 2002, a major North American financial services company implemented the Craft Personality Questionnaire (CPQ) within its selection program for sales representatives. Approximately one year after implementation, the company conducted a study to determine the validity and utility of the selection program. The program utilized by the company, and hosted on a platform offered by CraftSystems, Inc., included the Craft Personality Questionnaire (CPQ) and a biodata questionnaire developed specifically for the company.

Developed by Dr. Larry Craft, a behavioral scientist with over 25 years of experience in the employment testing industry, the CPQ measures eight primary and numerous secondary and interactive traits. These psychological traits are referred to as "personality scales" when used in pre-employment selection and post-hire employee development and coaching. A brief description of the CPQ'S eight personality scales is provided in table 1 on the following page.

The biodata questionnaire utilized by the company was developed by consultants and researchers in order to evaluate candidates' probability of success for a career in financial services sales. The survey was locally validated through numerous studies and was determined to offer a valid and reliable evaluation of candidates' background history characteristics that relate to a track record of success. This biodata instrument assesses candidates on a variety of factors, including: occupational background, employment stability, belief in financial services product ownership, educational level, natural market and sales experience.

The company had been using the CPQ and biodata questionnaire as individual hurdles with independent cut scores. However, they were interested in seeing the benefit of requiring both the CPQ and the biodata questionnaire in selection. Together, the two test results were computed into a composite score that provided ratings of low, moderate or high to managers during the selection process. The effectiveness of the composite score was then studied. The analysis discussed in this case includes a comparison of performance based on the composite scores of sales representatives hired over the course of the year.

Due to the multiple hurdle approach to selection that was in place, once the composite score was computed it was determined that a pool of incumbents would not have passed had the CPQ and biodata questionnaire been required and evaluated in the form of a composite score. This provided a complete range of test results for the study, which allowed comparisons to be made between passing and non-passing incumbents on the composite score. The study hypothesized that requiring both tests and using the composite score would strongly increase the validity of the selection process. It was also hypothesized that there would be a considerable difference in sales by those incumbents earning a passing composite score, as compared to those incumbents who did not receive a passing composite score. In response to the hypotheses, the sales productivity of incumbents was measured at 6-month and 1-year intervals.

Data Analysis
In this section, hypotheses are addressed as research questions through an analysis of the data collected by the company.

Research Question 1: What level of validity was offered by the CPQ when used in a formal selection program for financial services sales representatives? The financial services company implemented the CPQ in 2002 to assess the drive and motivation of candidates in relation to preferred norms established by CraftSystems. To answer the first research question, the company provided a sample of 314 field sales representatives, which included a mix of genders, as well as a mix of Caucasians and People of Color, who took the CPQ in 2002-2003 as part of the selection process.

The CPQ was shown in previous research involving other financial services companies to be predictive of commission earnings and premiums. The criterion of commissions earned was of particular interest in the present study as it provided the best overall indicator of sales performance. Also, commissions earned was found by the company to be a strong predictor of voluntary turnover. According to a database of hires over a 4-year period, there were moderate to strong correlations between commissions earned during the first (.137), second (.169) and third (.253) years and whether the person remained at the organization. For the purposes of this study, commissions were measured at two points within the archival records; at six months and then at the end of the first year of employment. Only participants who had a full six months and/or a full year of employment were included in this study.

The validation study was predictive in nature, since the CPQ was administered as part of the selection process and then production (based on commissions earned) was tracked over time. Commission information was analyzed using regression, with the CPQ score as the predictor for six month commissions and first year commissions independently. The criterion ranged from $0.00 to $19,401.94 for commissions earned at six months, and from $0.00 to $44,577.22 for commissions earned at one year. The sample sizes, means and standard deviations for these criteria are outlined in table 2.

The CPQ scores were entered into the regression equation predicting commissions earned at six months and commissions earned at one year separately. The validation results were reviewed with regard to R, F, and significance level. The results are outlined in table 3.

The results indicated that the CPQ was statistically significant in predicting commissions at both six months and one year of employment. These findings were consistent with those of other studies within the financial services industry, in which validity coefficients ranging from .191 to .322 were confirmed.

Research Question 2: What level of validity was offered by using the CPQ and biodata questionnaire together in a composite score for the selection of sales representatives? As this study sought to identify the validity associated with the composite score (including both the CPQ and the biodata questionnaire), the data analysis described above was replicated for the biodata questionnaire alone. The analysis provided evidence of predictive validity for the biodata questionnaire in predicting commissions earned at six months and one year (r=.204, p<.01 and r=.185, p<.05 respectively).

Having established that both the CPQ and biodata questionnaire offered predictive validity as independent assessments, an analysis was conducted using the overall composite score. The composite score offered evidence of predictive validity through its correlation to commissions earned during both six month and one year periods (r=.289, p<.01 and r=.273, p<.01 respectively). The data analysis confirmed the validity associated with each assessment, as well as the improved overall validity derived by combining the CPQ and biodata questionnaire assessments into a composite score. Specifically, the composite score resulted in a 42% improvement in overall validity compared with using just the biodata questionnaire alone during the selection process.

Research Question 3: What financial impact (utility) was offered to the company after utilizing the selection program for one year? To determine the utility of the selection system, the company analyzed performance differences among incumbents based on their composite scores at the point of selection. The composite score was based on scores from the CPQ and biodata questionnaire, the validity of which was established previously in this case study. The company calculated the commissions earned and the premiums sold for the incumbents who would have passed the composite scores, and those who would not have passed.

The approximate pass rates for the multiple hurdle approach was 55%. This was matched for the composite score and a "cut" was made to approximate a 55% pass rate. With this cut score, approximately 29% of sales representatives hired, for a total of 594 incumbents, would not have passed the composite score. These 594 incumbents were indicated in the following graph as the "not pass" category. The remaining 71% of sales representatives, for a total of 1,475 incumbents were labeled as the "pass" category.

With an average difference of $3,875 in first-year commissions, it is apparent that a significant difference in commissions earned was found between the failing and passing groups. Specifically, sales representatives earning passing composite scores performed at a level that was 47% higher than that of their non-passing peers.

Of interest to the calculation of utility, was the overall impact of the composite score on premiums generated by the sales representatives.

The information presented by the adjacent graph indicated a significant difference in premiums generated based on the composite scores. From this graph, it is evident that sales representatives earning passing composite scores, generated $6,657 more life premiums during the first year of employment. Based on these findings, sales representatives passing the composite scores generated as much as 43% more life premiums than their non-passing counterparts.

To quantify the specific financial impact associated with these results, it was necessary to multiply the difference in premiums mentioned above by the number of sales representatives who would not have passed the composite score, and thus would not have been hired. In doing so, a dollar amount representing the opportunity cost of not using a composite score previously was provided, thus rendering an estimate of the program's utility. The utility calculation is described below:

Conclusion
The company utilized the CPQ and biodata questionnaire as components in a structured selection program hosted on a platform designed and maintained by CraftSystems. The validity associated with each assessment was confirmed in this study, and an improved level of validity was reported when the assessments were combined in a composite scoring routine. Simply put, in this case the "whole was greater than the sum of its parts," as a 42% increase in validity was found by using the assessments together. This finding confirms the value of multifaceted selection programs as a means of capitalizing on information provided from more than one assessment, each focusing on a separate facet of employee compatibility. The most informed, and therefore valid, selection decisions were proven in this case to originate from combining factors such as drive and motivation with factors such as experience and successful track records.

Finally, this study reported a substantial utility associated with the selection program. Specifically, the estimated financial impact experienced by the company approached $4,000,000 in the first year based on higher revenue gained from improved selection practices. These improved practices were derived specifically from selecting sales representatives with the preferred drive and motivation as well as the desired sales experience and successful track records, which are known to provide a quick-start in an industry otherwise characterized by high turnover in the first year of employment. Given the costs associated with the CPQ and the biodata questionnaire, a conservatively estimated return on investment (ROI) would surpass a 2,000% return.