Sample

In this study, we used a multistage sampling design to select the samples. In the first stage, we used cluster sampling to choose one tertiary public hospital within each district of Beijing, which resulted in six hospitals being included in our survey. Each of the participating hospitals had 800 to 1500 clinic beds and 1200 to 3500 employees. In the second stage, 5% of physicians were sampled within each of the participating hospitals using stratified random sampling and physicians were stratified based on the profession. A total of 900 physicians participated in the questionnaire survey.

Questionnaire development

The selection of job evaluation factors for the questionnaire was based on a literature review, focused interviews with physicians and the Delphi method. First, through a literature analysis, we retained the 16 factors from the NHS JE system and consolidated another eight factors from four job evaluation models designed by governments or consulting firms [12,13,25,26]. Second, we identified an additional three factors through focused interviews with 20 physicians. Table 1 shows the lists of sources of job evaluation factors for the questionnaire.

Table 1

Lists of sources of job evaluation factors.

Number	Factors	M1	M2	M3	M4	M5		FI	D
1	Professional knowledge	√	√	√	√	√		√	√
2	Training	√				√		√	√
3	Experience	√		√		√		√	√
4	Communication and relationship skills	√	√	√	√	√		√	√
5	Planning and organizational skills	√		√		√		√	√
6	Analytical and judgemental skills	√		√		√		√	√
7	Physical skills	√		√		√		√	√
8	Responsibilities for patient/client care	√						√	√
9	Responsibilities for policy and service development and implementation	√	√	√		√		√	√
10	Responsibilities for financial and physical resources	√		√				√	√
11	Responsibilities for human resources	√			√	√		√	√
12	Responsibilities for information resources	√				√			√
13	Responsibilities for research and development	√						√	√
14	Freedom to act	√	√	√	√				√
15	Emotional effort	√				√		√	√
16	Mental effort	√						√	√
17	Physical effort	√	√					√	√
18	Working conditions	√	√	√	√	√		√	√
19	Innovation skills			√	√	√		√	√
20	Task relevance		√	√	√			√	√
21	Task complexity		√	√	√	√		√	√
22	Emergency response skills					√		√	√
23	Temporal characteristics					√		√	√
24	Knowledge updates					√		√	√
25	Awareness of quality and safety						√		√
26	Knowledge diversity							√
27	Work urgency							√

Note: National Health Service Job Evaluation scheme (M1) [14], Model by the U.S. Office of Personnel Management (M2) [13], Hay’s Job Evaluation Model (M3) [25], International Position Evaluation (IPE) (M4) [12], Model by Jinan Kangjia Hospital Management Consultants Ltd., (M5) [26], Focused Interviews (FI), Delphi method (D).

We screened and determined the factors using the two-round Delphi method with a panel of 15 experts. These experts included senior attending physicians and experts on hospital management and human resources. All of the experts were professors and had at least a Master’s degree. Most of the experts were 40 to 60 years old and had more than 20 years of work experience. We received 13 valid responses in the first round of the questionnaire and 12 valid responses in the second round. The result showed that the expert authority coefficient for all factors in the two rounds was higher than 0.8, suggesting that the result of the experts’ evaluation had high reliability. The Kendall coordination coefficient for all factors in the two rounds was statistically significant (p<0.05), which indicated that the result had high validity. After two rounds of Delphi, two factors, Knowledge Diversity and Work Urgency, were removed, and 25 factors were ultimately retained. The expert group used the Delphi process to derive a detailed definition of the following factors: Analytical and Judgemental Skills, Task Complexity, Responsibilities for Policy and Service Development and Implementation, and Knowledge Updates.

As shown in S1 and S2 Tables, the final version of the questionnaire contained 25 factors. The questionnaire asked participants to rate the importance of each factor on a 5-point Likert scale, ranging from “very important” (5 points) to “very unimportant” (1 point).

Reliability and validity of the questionnaire

The questionnaire was pretested to assess its reliability and validity. A total of 50 physicians from a tertiary general hospital in Beijing participated in the preliminary survey. A total of 48 questionnaires were returned, with 45 valid responses. In the retest survey, 30 out of the 50 physicians accepted the retest administration within a two-week time span. The results showed that the overall Cronbach’s alpha of the questionnaire was 0.90, which implies satisfactory reliability. The test-retest reliability coefficient was 0.87, suggesting good retest reliability.

With the data from the preliminary survey, we applied Bartlett’s Test of Sphericity and the Kaiser-Meyer-Olkin (KMO) test to measure how suited the data are for factor analysis. The results showed that the Bartlett’s Test of Sphericity was significant at p< 0.05, and the KMO value was 0.879, which indicated that the data were appropriate for the purpose of factor analysis. We then used a principal component analysis to obtain the initial factor solutions. Five principal components with Eigenvalues greater than one were extracted, and they cumulatively explained 60.42% of the variance. Hence, the results indicated that the questionnaire had good structural validity.

Data processing and statistical analysis

After receiving the completed questionnaires, a preprocessing step was applied to remove incomplete or invalid data. The exclusion criteria were as follows: 1) there were more than three factors unanswered; 2) all the factors were answered the same; and 3) the answers displayed an obvious pattern. In addition, after all the data had been entered, 5% of the data were drawn for rechecking to ensure the accuracy of data entry.

We performed a principal component analysis with a varimax rotation for feature extraction and dimensionality reduction. The factors with Eigenvalues greater than one were retained, and the factors with factor loadings less than 0.5 were removed from the analysis. The reliability of the job evaluation system was assessed by calculating Cronbach’s alpha and analysing the correlation between factors from the same and different dimensions. The validity was assessed via a structural validity test on the job evaluation factors. The data were processed with Microsoft Excel 2015 and analysed with SPSS 21.0.

Ethics approval and consent to participate

This study has received ethics approval from Tsinghua Shenzhen International Graduate School Medical Ethics Committee. The questionnaire surveys were anonymous and voluntary, and written informed consent had been obtained from all study participants and experts.

Characteristics of respondents

In this survey, a total of 900 questionnaires were distributed, and 759 were completed, which resulted in a response rate of 84.3%. A total of 693 questionnaires were regarded as valid, with a validity rate of 91.3%. Table 2 shows the characteristics of the study samples. Overall, 51.9% of the respondents were male, and 48.9% were aged between 30 and 39 years old. In terms of education, 51.7% had a Master’s degree. Attending physicians made up 47.8% of the sample. The majority of the respondents were general staff (70.7%) and had an annual salary of between 10,000 and 20,000 dollars (34.4%). In addition, most of the respondents worked in the surgery department (33.8%), followed by the internal medicine department (25.8%). There was no significant difference in the demographic characteristics between the exploratory samples and the confirmatory samples (p>0.05), which indicated that the responses had good consistency.

Table 2

Demographic characteristics of respondents (N = 693).

Variable	Items	Total (N = 693)		Exploratory Samples (N = 348)		Confirmatory Sample (N = 345)		p-value
		N	%	N	%	N	%
Gender	Male	360	51.9	186	53.4	174	50.4	0.473
	Female	323	46.6	158	45.4	165	47.8
	Missing	10	1.4	4	1.1	6	1.7
Age (years)	< 30	101	14.6	49	14.1	52	15.1	0.597
	30–39	339	48.9	177	50.9	162	47.0
	40–49	197	28.4	92	26.4	105	30.4
	≥ 50	56	8.1	30	8.6	26	7.5
Education level	Bachelor’s or below	134	19.3	68	19.5	66	19.1	0.212
	Master’s	358	51.7	180	51.8	178	51.6
	PhD	201	29.0	100	28.7	101	29.3
Title	Attending	331	47.8	167	48.0	164	47.5	0.927
	Junior attending	232	33.5	118	33.9	114	33.1
	Senior attending	130	18.7	63	18.1	67	19.4
Level of position	Head of department	71	10.2	32	9.2	39	11.3	0.332
	Group leader	132	19.1	70	20.1	62	18.0
	General staff	490	70.7	246	70.7	244	70.7
Working time (years)	≤5	128	18.5	60	17.2	68	19.7	0.454
	6–10	156	22.5	88	25.3	68	19.7
	11–15	155	22.4	78	22.4	77	22.3
	16–20	112	16.2	52	14.9	60	17.4
	>20	142	20.5	70	20.1	72	20.9
Annual income (dollars)	≤10,000	186	26.9	98	28.2	88	25.7
	10,000–20,000	238	34.4	116	33.3	122	35.6
	20,001–30,000	143	20.7	75	21.6	68	19.8
	>30,000	124	17.9	59	17.0	65	19.0
Departments	Internal medicine	179	25.8	89	25.6	90	26.1	0.683
	Surgery	234	33.8	117	33.6	117	33.9
	Gynaecology and paediatrics	93	13.4	42	12.1	51	14.8
	Emergency	85	12.3	43	12.4	42	12.2
	Others	102	14.7	57	16.4	45	13.0

Descriptive statistics of the job evaluation factor questionnaire

As shown in Table 3, the average scores of the 25 factors ranged from 3.8 to 4.7, with a mean value of 4.2. The factor that received the highest score was Professional Knowledge (4.7±0.60), and the factors that received the lowest scores were Responsibility for Financial and Physical Resources (3.8±0.99) and Responsibility for Information Resources (3.8±0.87). The standard deviations were between 0.60 and 0.99, suggesting that all factors had a low degree of dispersion.

Table 3

Descriptive statistics of the job evaluation factor questionnaire (N = 693).

Factors	Minimum	Maximum	Mean	Standard deviation
Professional knowledge	4	5	4.7	0.60
Training	3	5	4.2	0.75
Experience	3	5	4.6	0.62
Communication and relationship skills	3	5	4.4	0.65
Planning and organizational skills	3	5	4.2	0.78
Analytical and judgemental skills	3	5	4.4	0.66
Physical skills	3	5	4.5	0.67
Responsibilities for patient/client care	3	5	4.2	0.82
Responsibilities for policy and service development and implementation	2	5	3.9	0.94
Responsibilities for financial and physical resources	1	5	3.8	0.99
Responsibilities for human resources	2	5	3.9	0.93
Responsibilities for information resources	1	5	3.8	0.87
Responsibilities for research and development	2	5	3.9	0.86
Freedom to act	3	5	4.2	0.82
Emotional effort	3	5	4.4	0.73
Mental effort	3	5	4.3	0.76
Physical effort	2	5	4.1	0.86
Working conditions	3	5	4.4	0.77
Innovation skills	2	5	4.1	0.84
Task relevance	2	5	4.1	0.77
Task complexity	2	5	4.1	0.81
Emergency response skills	3	5	4.4	0.71
Temporal characteristics	2	5	4.1	0.78
Knowledge updates	3	5	4.5	0.66
Awareness of quality and safety	3	5	4.4	0.78
Mean			4.2	0.77

Factor analysis and dimensions

Bartlett’s Test of Sphericity was significant at p< 0.05, and the KMO value was 0.939, which supported the appropriateness of the principal component analysis technique. The principal components with Eigenvalues greater than one were retained. Five principal components were extracted from the principal component analysis and were defined as the five dimensions of the job evaluation system. The five components cumulatively explained 66.26% of the total variance, and the result was acceptable. A varimax rotation was used to transform the components into factors that were more clearly interpretable. Factors with factor loadings greater than 0.5 were extracted. All 25 factors had factor loadings above 0.5, and thus, all were retained in this study. Considering the content of the factors under each dimension, we named the five dimensions Task Characteristics (four factors), Knowledge (five factors), Responsibility (five factors), Effort/Environment (four factors) and Skills (seven factors). Table 4 demonstrates the factor loadings for each factor. The factor loadings of the factors in some dimensions were all above 0.7. For example, the five factors of the Responsibility dimension had factor loadings between 0.775 and 0.879, and the four factors of the Effort/Environment dimension had factor loadings between 0.734 and 0.843.

Table 4

Factor loadings by principal component analysis (N = 693).

Factor	Component
	Task Characteristics	Knowledge	Responsibility	Effort/Environment	Skills
Task complexity	0.890	-	-	-	-
Task relevance	0.823	-	-	-	-
Temporal characteristics	0.804	-	-	-	-
Freedom to act	0.673	-	-	-	-
Professional knowledge	-	0.828	-	-	-
Knowledge updates	-	0.827	-	-	-
Physical skills	-	0.664	-	-	-
Experience	-	0.606	-	-	-
Awareness of quality and safety	-	0.563	-	-	-
Responsibilities for policy and service development and implementation	-	-	0.879	-	-
Responsibilities for financial resources	-	-	0.875	-	-
Responsibilities for human resources	-	-	0.857	-	-
Responsibilities for information resources	-	-	0.810	-	-
Responsibilities for research and development	-	-	0.775	-	-
Emotional effort	-	-	-	0.843	-
Mental effort	-	-	-	0.839	-
Physical effort	-	-	-	0.830	-
Working conditions	-	-	-	0.734	-
Planning and organizational skills	-	-	-	-	0.810
Communication and relationship skills	-	-	-	-	0.764
Analytical and judgemental skills	-	-	-	-	0.750
Emergency response skills	-	-	-	-	0.733
Innovation skills	-	-	-	-	0.715
Training	-	-	-	-	0.696
Responsibilities for patient/client care	-	-	-	-	0.687

Reliability and validity analysis of the CN-PJE

The Cronbach’s alpha coefficients of the five dimensions ranged from 0.841 to 0.909, which indicated a high level of internal consistency. The Cronbach’s alphas were then calculated for each factor under the dimensions. As shown in Table 5, the removal of any factor would result in a lower Cronbach’s alpha. In terms of the corrected factor-total correlation, all correlation coefficients met the requirements, which suggested that the system’s factors and dimensions had good reliability. Moreover, the correlation coefficients between factors under the same dimension were all greater than 0.376, which indicated that the factors had good consistency, as shown in S3 Table.

Table 5

Reliability analysis of CN-PJE (N = 693).

Factor	Corrected factor-total correlation	Cronbach's Alpha if factor deleted	Cronbach's Alpha
Task complexity	0.870	0.882	0.907
Task relevance	0.823	0.888
Temporal characteristics	0.804	0.895
Freedom to act	0.673	0.906
Professional knowledge	0.588	0.814	0.841
Knowledge updates	0.706	0.789
Physical skills	0.681	0.810
Experience	0.562	0.818
Awareness of quality and safety	0.531	0.834
Responsibilities for policy and service development and implementation	0.855	0.882	0.909
Responsibilities for financial and physical resources	0.847	0.889
Responsibilities for human resources	0.801	0.890
Responsibilities for information resources	0.768	0.892
Responsibilities for research and development	0.703	0.905
Emotional effort	0.795	0.860	0.885
Mental effort	0.746	0.846
Physical effort	0.739	0.855
Working conditions	0.699	0.874
Planning and organizational skills	0.715	0.850	0.881
Communication and relationship skills	0.709	0.867
Analytical and judgemental skills	0.717	0.858
Emergency response skills	0.656	0.861
Innovation skills	0.606	0.863
Training	0.597	0.869
Responsibilities for patient/client care	0.562	0.864

Table 6 displays the inter-correlations of the five dimensions. The correlation coefficients between the dimensions were greater than 0.488, suggesting good consistency between the dimensions. Knowledge and Skills (r = 0.718) were the most correlated, while Knowledge and Responsibility (r = 0.488) were the least correlated.

Table 6

Inter-correlations of the five dimensions (N = 693).

Dimension (Mean/Standard Deviation)	Task Characteristics	Knowledge	Responsibility	Effort/Environment	Skills
Task Characteristics (4.1/0.66)	1	-	-	-	-
Knowledge (4.5/0.50)	0.570***	1	-	-	-
Responsibility (3.9/0.75)	0.608***	0.488***	1	-	-
Effort/Environment (4.2/0.65)	0.703***	0.524***	0.576***	1	-
Skills (4.3/0.57)	0.618***	0.718***	0.592***	0.496***	1

***denotes statistical significance at p<0.001.

The above results of the factor analysis show that the system has good structural validity. The content validity is well established by building on the NHS JE system and other well-established job evaluation systems with a two-round Delphi method. Therefore, our results show that the CN-PJE has both good reliability and validity.

The feature of CN-PJE

The survey results indicate that the CN-PJE can reflect the characteristics of physician posts. First, physicians are usually confronted with a number of challenges, including complicated clinical practices, varied clinical demands of patients, high risks, and heavy workloads [27]. These characteristics are reflected by the dimensions of Task Characteristics (four factors) and Effort/Environment (four factors). These factors should play an important role in determining a physician’s salary. Second, a physician is a knowledge-intensive position with high requirements for knowledge and skills [28,29]. These features are reflected by the dimensions of Knowledge (five factors) and Skills (seven factors). Third, physicians need to shoulder multiple responsibilities in hospitals [30], which can be captured by the Responsibility dimension (five factors). In addition to patient care, physicians also engage in management responsibilities, which involve policy development and implementation, financial and physical resources, human resources and information resources. Responsibility for research and development is an integral part of the work of physicians, especially for those in tertiary hospitals. Usually, research achievements can affect the salary of physicians in Chinese public hospitals. Furthermore, more responsibilities require more skills. Hospitals also put great value on physicians’ skills in problem solving, social interaction, administration and innovation. The survey suggests that physicians also attach importance to these skills and responsibilities.

The factors we developed in this study conform to the latest trends in job evaluation. To adapt to changes in the internal and external environments of hospitals, the job evaluation factors must reflect the organizational strategy and pay attention to the performance, contribution and development of employees [4]. Furthermore, job evaluation systems pay attention not only to fairness but also to flexibility [31]. These trends are embodied in this study. In conclusion, the CN-PJE can reasonably represent the relative values of physicians.

The comparative analysis of two job evaluation systems

There are similarities and differences between the NHS JE and the CN-PJE, as shown in Table 7. Compared with the NHS JE system (five dimensions and 16 factors), the CN-PJE consists of similar dimensions but more factors (five dimensions and 25 factors). The factors of the Effort/Environment dimension are exactly the same, and the factors of Responsibility are almost the same as those of the NHS JE system. The results suggest that these dimensions and factors are reasonable.

Table 7

The dimensions and factors of CN-PJE and NHS JE.

Dimensions	CN-PJE	NHS JE
Task Characteristics	Freedom to act	Freedom to act
	Task complexity	Newly added factor
	Task relevance	Newly added factor
	Temporal characteristics	Newly added factor
Knowledge		Knowledge, training and experience
	Professional knowledge	Split out from Knowledge, training and experience
	Experience	Split out from Knowledge, training and experience
	Physical skills	From Skills dimension
	Knowledge update	Newly added factor
	Awareness of quality and safety	Newly added factor
Responsibility	Responsibility for policy and service development and implementation	Responsibility for policy and service development and implementation
	Responsibility for financial resources	Responsibility for financial resources
	Responsibility for human resources	Responsibility for human resources
	Responsibility for information resources	Responsibility for information resources
	Responsibility for research and development	Responsibility for research and development
		Responsibility for patient/client care
Effort/Environment	Emotional effort	Emotional effort
	Mental effort	Mental effort
	Physical effort	Physical effort
	Working conditions	Working conditions
Skills	Planning and organization skills	Planning and organization skills
	Communication and relationship skills	Communication and relationship skills
	Analytical and judgemental skills	Analytical and judgemental skills
	Emergency response skills	Physical skills
		Newly added factor
	Innovation skills	Newly added factor
	Training	Split out from Knowledge, training and experience
	Responsibility for patient/client care	From Responsibility dimension

In terms of differences, these two systems apply to different posts in healthcare organizations. The NHS JE scheme is used to determine the pay bands for all staff employed in the NHS, with the exception of doctors, dentists and some senior managers, who have separate pay contracts [14,15,32]. In contrast, the job evaluation factors in this study are specifically developed for physicians. Although the NHS JE system does not apply to physicians, it provides a rational framework for this study because most evaluation factors are common to all job positions. Moreover, significant effort has been made in this study to take into account the different aspects of physician posts so that the evaluation factors are adopted to the particular characteristics of physicians.

There are some differences in the factors of some dimensions. First, there are different numbers of factors under the Knowledge dimension. In the NHS JE system, there is only one factor under the Knowledge dimension to measure all the forms of knowledge required to fulfil the job responsibilities. This factor is the most heavily weighted factor (24%) and usually can make a difference between one pay band and the next [14]. In our study, we identified five factors to measure different forms of knowledge, specifically professional knowledge, experience, physical skills, awareness of quality and safety, and knowledge updates. Previous studies have suggested that having only one factor to measure all forms of knowledge might result in inaccurate and inconsistent outcomes [11,18]. Indeed, in addition to abundant professional knowledge, strong practical skills and rich experience [28], physicians need to have an awareness of quality and safety, given that patient safety is one of the most important components of health care delivery. In addition, physicians also need to constantly improve themselves through learning and updating new knowledge and techniques. Therefore, it is essential to define factors to measure each of these areas of knowledge.

Second, this study identified three new factors in the Task Characteristics dimension. Task complexity and Task relevance are mentioned by many other models, including the General Schedule of the U.S. Office of Personnel Management [13], Hay’s Job Evaluation model [25] and the International Position Evaluation model [12]. The amount of working time and duty periods vary from one post to another. Consequently, it is important to take into account the temporal characteristics. As suggested by our survey results, these new factors are highly accepted by physicians. In brief, these five factors can fully reflect the post characteristics of physicians.

Third, under the Skills dimension, Emergency response skills and Innovation skills are factors that are not included in the NHS JE system but are identified in our system. These factors can clearly reflect the skills required for physicians to address complicated medical services and the diversified clinical demands of patients. Training is from the Knowledge dimension, and Responsibility for patient/client care is from the Skills dimension of the NHS JE system. These two factors are the basis of other skills. Overall, the factors in this study are significantly correlated and can fully capture the characteristics of physicians.

The application and implication of CN-PJE

The point-factor job evaluation is a systematic method for realizing internal equity within public organizations [11,33]. The current Post-Performance Payment system in Chinese public hospitals fails to consider internal relativities across different posts and results in internal inequity, job dissatisfaction and physicians’ profit driven incentive [20,22,23]. The CN-PJE system can be used in salary plans and to ensure internal equity for physicians. As the CN-PJE system is built on survey data from physicians, its design can better reflect what values physicians consider to be fair and will thus be better accepted by physicians. The CN-PJE has been adopted by two public hospitals in Beijing and have been used in their internal compensation programmes. Therefore, physician job evaluation can provide an important basis for the development of physician compensation plans. In addition, it can be combined with a variety of performance appraisal methods, such as the balanced scorecard and key performance indicators, to develop a reasonable salary system for physicians [34,35]. Furthermore, based on China’s explorations, the methods adopted by this study will be applicable to the localization of NHS JE and the establishment of job evaluation factors for other positions and in other countries.

Limitations

Several limitations of this study should be mentioned. First, the samples for this study had a limited geographical range. As a result, this study can only represent the situation in Beijing and it can be viewed as a preliminary study. Further studies conducted in China’s different regions will reflect China’s whole situation and allow greater generalization. Second, the established CN-PJE has only been used in two hospitals in Beijing. The applicability of the system needs further verification through practice. Finally, this study was conducted in tertiary hospitals. Future studies can test the CN-PJE in other different levels of hospitals.