The problem of the validity of judgments of the various traits was considered in a more detailed way by Norsworthy, from whose account of her inquiry we have already quoted. She chose the traits enumerated by Cattell, and performed several experiments to determine the reliability of judgments of this sort. Thus she had five intimate acquaintances independently grade a sixth person for her possession of these twenty-four traits, on two different occasions several weeks apart.
Two things were clearly shown. In the first place the individual judges, in their second trials, did not diverge far from their first ratings. In the second place the double judgments of the five different judges did not diverge far from each other. These two facts "prove that the ratings do stand for some actual quantitative value and are not subject to mere chance. The validity of the judgments, in the sense of their correspondence with the actual character of X is then only a matter of the impartiality of the group of judges."
Similar results were found in the judgments of nine members of a college society by five of their comrades, and in the judgments of a teacher by two hundred college students. It was apparent also that judges differ from one another in the general accuracy of their gradings. Some of them agree closely with the consensus of opinion, while others depart, in varying degrees, from the average or correct estimate. It was also seen that, in estimating certain individuals, judges with presumably equal acquaintance with those being judged agreed closely with one another. Other persons had produced quite different impressions on the different judges and this was revealed in the greater divergence of the grades assigned to such persons.
As in the case of Cattell's results, figures are presented showing the degree of divergence among the judges in estimating the different traits. In the table on page 139 these figures are given, as shown in the records of five judges in one of Norsworthy's experiments, and the records of the twelve judges in Cattell's investigation. The average variability or degree of divergence for all the twenty-four traits is taken as the standard and each trait compared with this standard. A variability of one hundred thus indicates the average amount of disagreement. Figures smaller than one hundred indicate that the agreement was closer than average, and figures larger than one hundred indicate that here the judges disagreed by more than the average amount.