We tested the general acceptance of the proposed framework through a two-phased operation test.
Phase One—Group Demonstration and Evaluation
The purpose of this phase-one evaluation was to test the operations of the competency-based needs assessment model and the prototype system from a practical standpoint. We looked for experienced HR practitioners as our study sample because they were most qualified to judge whether the system was valid to use in the field as a needs assessment tool. We carefully selected a class of experienced managers who were registered in a MIS (Management of Information Systems) course offered through the HRM program in a major university in Taiwan. These managers were studying for their master’s degree in a graduate program designed specifically for experienced human resource managers and professionals. We conducted a Q&A session toward the end of the MIS course to ensure that the study sample had adequate knowledge of the use of IS in human resource management.
Thirty-one managers participated in the study, and most of whom had experiences in training related jobs. The complete process of a needs assessment using the prototype system was demonstrated in the MIS class. The demonstration took 40 minutes followed by a 20-minute Q&A. Then the class was asked to fill out a questionnaire. The questionnaire contains seven questions using a Likert-type scale of 1 to 5 (1 means strongly disagree and 5 means strongly agree).
Table 4 presents the results of the descriptive statistics of mean and standard deviation (S.D.). Respondents showed positive reactions toward each of the seven measures. The concept of a competency-based training needs assessment received most favorable ratings, followed by the use of gap analysis to assess these needs. Respondents were also impressed with the ability of the system to shorten operation time required of a competency-based
training needs assessment.
Table 4 Results of Validity Measures
Questionnaire Items Mean* S.D.
1. Accept the concept of a competency-based training needs assessment
4.23 0.88
2. Accept the concept of using gap analysis to assess training needs 3.84 0.90 3. The system effectively assist the planning phase of a
competency-based training
3.32 0.94
4. The system simplifies the process of a competency-based training needs assessment
3.48 0.93
5. The system shortens operation time required to conduct a competency-based training needs assessment
3.71 0.97
6. The system effectively expedites the transmission of training needs assessment information
3.65 0.91
7. The system makes analysis results of employees’competence gaps more accessible to HRD professionals
3.48 1.06
*Average score ranging from 1 strongly disagree to 5 strongly agree
Phase Two—Individual Tryout Sessions
The purpose of this phase-two evaluation is to test the functionality and usability from a more realistic operation of the prototype system. To this end, we conducted one-on-one interviews with three training and development (T&D) managers after they had experienced the operation of the prototype system hands-on.
The three T&D managers were from different industry backgrounds. Manager A was a T&D manager for a major department store and had 8 years of working experience in HR-related functions. Manager B worked as a T&D supervisor for a government agency for more than 10 years. Manager C was in charge of training for an area hospital and had 3 years of HR-related experience. Each manager received a 10-minute introduction of the competency-based training needs assessment model, then worked through each of the five modules of the prototype system to complete a hypothetical needs assessment process like the
use case. The hands-on experience took 15 to 20 minutes, followed by a 30-minute semi-structured interview. Interview questions included the following: (See table 7 for a complete list of interview questions.)
1.
Effectiveness, time-saving and accuracy of the questionnaire management module2.
Effectiveness and accuracy of the needs assessment module to calculate competence gaps3.
Whether the prototype system was easy to follow, straight-forward and user-friendly4.
Consistency of layout, appropriate size and amount of texts, appropriate use of graphics5.
Content accuracy, flow of information and legibility of the prototype systemIn general, all three managers agreed that the Web-based prototype system provides a good model to assess employees’training needs, reinforces the concept of competency-based training, and simplifies the analysis stage of a planning process for training. While the training managers were positive about the effectiveness and the process improvement prospects of the system, they also stressed the importance of customization to realize these benefits. The extent of customization included company organization and communication channel, a company-wise competency databank, training curriculum, process linkage to training course registration, and administration, etc. Concerns of employees mistaking a training needs analysis to performance evaluation were raised. Though not a system problem, one of the managers-Manager C-suggested adding this message to the survey: “For development purpose only, and will not be used in performance evaluation”to minimize the possibility of employees sending falsified information.
In terms of the functionality of the system, the managers suggested the following improvements:
Provide multiple methods for data analysis or allow exporting of data to other statistical program for further data manipulation.
Send an automatic confirmation when the survey is received by the intended employee.
Provide better tracking mechanism, such as status on send, reply and follow-up.
Provide better navigation functions, such as backtrack, system maps and on-line instructions.
Provide options to setup the order of how respondents fill out the question items.
Remind respondents of missing answers.
Table 7 Interview Findings
Interview Questions Results Improvement Suggestions
1. Do you agree with the concepts imbedded in the prototype system?
Positive
2. Will the system facilitate the use of
competence-based training needs assessment in your company?
Positive - Customization is required prior to implementation.
3. Will the system help save time in planning employee training?
Positive
4. Will the system simplify the assessment process for employee training?
Positive - Provide choices of analytical tools
5. Is the system effective in assisting a
competency-based training needs assessment?
Positive
6. Effectiveness of questionnaire management module: Design? Dissemination? Collection?
Positive - Automatic confirmation when message is received by the receiver.
- Provide stats on send, reply and follow-up status.
7. Effectiveness of needs assessment module in quantifying employees’competence gaps?
Positive - Stress “the survey is for
developmental purpose only, and will not be used in performance evaluation”
8. Is the prototype system easy to follow and user-friendly?
Positive - Provide system maps and on-line instructions.
9. Aesthetic appeals of the screen display:
Consistency? Appropriate placement and size of text and graphic?
Positive
10. Legibility, accuracy and flow of contents in the system?
Positive - Provide option to setup the order of how respondents fill out the
We have presented an integrated framework of a Web-based training needs assessment system to effectively and efficiently assist organizations in their pursuit of competitive core competencies. Neither the competence-based training approach nor the needs assessment method is new to seasoned HR professionals. The method of using survey questionnaires to uncover skill gaps in the organization has been around for decades. Competence-based theories, or more specifically, building organization core competencies for sustained competitive advantage, have been the topic for discussion since the 80’s. But these concepts are rarely implemented in the real world (Selmer, 2000) because the process required is both tedious, and time-consuming. (Gilley & Eggland, 1989; Goldstein, 1993; Guo, 1987)
This study contributes to the HR practice in several ways. First, it qualitatively confirms that HRD professionals accept a competence-based approach to build company’s training curriculum. Second, it observes a lack of effective tools to help HRD professionals in the task of training needs assessment. Third, it shows that HRD professionals do recognize the power of Web technology in helping them become more efficient. Lastly, A Web-based training