The methods of examination include: direct measurement, ranking, comparison.

With **direct measurements** by an expert method, the value of physical quantities or quality indicators is determined immediately in established units (SI units, points, standard hours, etc.)

Such measurements are taken on a ratio scale, an interval scale, or an order scale.

You can directly measure the weighting coefficients, the measurement is carried out on a scale of intervals.

The value of these coefficients is calculated by the formula:

_{}_{}

where *n* is the number of experts,

*m* is the number of weighted indicators,

*G i, j* – weight coefficient, rank;

*j* -th indicator in points, given by the i -th expert.

Direct measurement by the expert method is complex and places high demands on experts.

**Ranking** consists in arranging measurement objects or indicators in order of their preference for importance or weight. The place occupied in such an arrangement is called a rank. The higher the rank, the more desirable the object.

**Pairwise comparison** is a method of conducting an examination, when the measured values are compared with each other in pairs and for each pair the results of the comparison are expressed in the form of “more – less” or “better – worse”. Then the measured values are ranked (placed in ranks).

**Examples**

**Example 1** Determine the degree of agreement between the opinions of five experts, the results of ranking seven objects of expertise are shown in Table 1.

Table 1 – The results of the examination to determine the measure of consistency of opinions of experts

Examination object number, j |
Expert assessment, i |
Sum of ranks | Deviation from the arithmetic mean | Square of deviations from the arithmetic mean | ||||

1st | 2nd | 3rd | 4th | 5th | ||||

-5 | ||||||||

-thirteen | ||||||||

_{} |
_{} |

Decision:

1) calculate the arithmetic mean of the ranks:

_{}

2) calculate the sum of squared deviations from the arithmetic mean (S):

S=630

3) calculate the concordance coefficient (W):

_{}

Answer: The degree of agreement between experts is satisfactory.

If the degree of agreement between experts’ opinions is unsatisfactory, measures are taken to improve it (training, discussion of results, analysis of errors).

**Example 2. “Ranking”.** The results of the assessment by 5 experts of 7 objects (by rank) are shown in Table 1.

According to the sum of ranks, the seventh object is the best, the fourth in quality is the second, then the sixth, first, second, third and fifth. If the ranking is carried out in order to determine the weight coefficients g _{i} for seven objects, then they are calculated using the above formula:

_{}

Then the weight coefficients of the objects have the following values:

_{}_{}

similarly:

_{}_{}_{}

_{}_{}_{}

where_{}

_{}

**Example 3 “Pairwise matching”.** Table 2 presents the results of tasting food products, indicated by numbers from 1 to 6. The preference for the *i* -th product over the *j* -th corresponds to 1, the opposite ratio – 0. The equivalence of the products corresponds to the X sign. Arrange the products by quality on an order scale.

Table 2 – The results of the examination of the quality of products by the method of pairwise comparison

i – number | j – number | ||||||

Total | |||||||

X | |||||||

X | |||||||

X | |||||||

X | |||||||

X | |||||||

X |

Solution: The ranked row is as follows: No. 4; No. 5; No. 6; No. 2; No. 1; No. 3.

**Example 4** Table 3 presents the results of a pairwise comparison of the skills of the singers who performed at the vocal competition. The advantage of the *i* -th soloist over *j* – m is indicated (1), *j* – th over *i* – m, respectively (-1), equivalent performances are indicated (0). Determine the results of the competition.

Table 3 – The results of the examination of the skill of singers by the method of pairwise comparison

j – number | |||||||

i – number | Total | ||||||

-one | |||||||

-one | -one | -one | |||||

-one | -one | -one | |||||

-one | -one | -one | -one | -one | -5 | ||

-one | -one | -one |

Tables 2 and 3 are redundant. With pairwise matching, there is enough data, on one side of the diagonal. Preference is expressed by indicating the number of the preferred object as shown in Table 4, modified from Table 2.

Decision: The winner of the competition was the singer who performed third, the second place was taken by the soloist who performed first. The third, fourth and fifth places were shared between the vocalists who performed second, fourth and sixth. The last place went to the singer, who performed fifth.

Table 4 – The results of the examination by the method of pairwise comparison

Examination object number, j |
Total | ||||||

X | |||||||

X | |||||||

X | |||||||

X | |||||||

X | |||||||

X |

Solution: The ranked row is as follows: No. 4; No. 5; No. 6; No. 2; No. 1; No. 3.

**Example 5** Suppose that 5 experts participate in the examination of 6 objects, and all experts expressed their opinion in the same way, i.e. the opinion of each of the five experts is presented in Table 4. Determine the weight of each object and build a ranked series.

Decision:

The score of the *j* -th object, determined by the *i* -th expert, is calculated by the formula:

_{}

where: *Fi,j* is the frequency of preference by the *i* -th expert of the *j* -th object,

*C* – the total number of judgments of one expert,

*m* is the number of objects of expertise.

_{}

1. Determine the frequencies of preferences by each expert of the *j* -th object ( *Fi,j* ), i.e. how many times the expert preferred the *j* -th object to five others:

_{}_{}_{}

_{}_{}_{}

2. Calculate the total number of judgments of each expert (С):

_{}

3. Calculate the score or weight of each object of examination ( *Gi)* according to the general opinion of five experts:

_{}

_{}

_{}

_{}

_{}

_{}

_{}

4. We determine the sum of the points of all objects of examination, according to five experts:

_{}

Therefore, the values of *Gj* obtained in paragraph 3 are considered as normalized and are used as weight coefficients.

5. Based on the weight coefficients, we write down the ranked number of objects of expertise: (as the score decreases)

#3 #1 #2 #6 #5 #4

**Example #6.**

Table 5 shows the opinions of 4 experts on 5 objects of expertise. The examination of objects was carried out according to one quality indicator.

1. Based on the sum of the ranks of each object of examination, build a ranked series, which is the result of multiple measurements;

2. Determine the weight of the members of the ranked series;

3. Determine the degree of agreement between the opinions of 4 experts by calculating the concordance coefficient.

Table 5

Expert opinion | Initial data |

Opinion 1 expert | Q4 > Q3 > Q2 > Q5 > Q1 |

Opinion 2 experts | Q1 |

Opinion 3 experts | Q2 < Q1 < Q5 < Q3 < Q4 |

Opinion 4 experts | Q4 > Q3 > Q5 > Q1 > Q2 |

Calculation procedure:

1. The given initial data must be presented in the form of inequalities of the same meaning. In this example, the opinion of the 3rd expert should be rewritten in the form Q4 > Q3 > Q5 > Q1 > Q2, and the opinion of the 4th expert – in this form Q2 > Q1 > Q5 > Q3 > Q4.

2. Find the sum of the ranks of each object of expertise.

Table 6

Examination object number | Expert assessment | Sum of ranks | Deviation from the arithmetic mean | The square of the deviation from the arithmetic mean | |||

1st | 2nd | 3rd | 4th | ||||

-one | |||||||

-one | |||||||

3. Arithmetic mean of ranks:

4. Determine the total number of judgments of one expert, related to the number of objects of expertise “m”:

5. Calculate the weight of the members of the ranked series:

6. Calculate the concordance coefficient:

**Conclusion:** with the obtained concordance coefficient (W = 0), there is no degree of agreement among experts.

## Be First to Comment