personality.cn The Chinese Personality
at Work Research Project
University of Queensland, Australia, Dr. Graham Tyler & PsyAsia
International
Identity Questionnaire Research Results - A synopsis
No reproduction without permission.
NB: This research study was not part of the
original University of Queensland Research. It does however form
part of the Chinese Personality at Work Research Project and was
conducted in association with Quest Partnership, UK.
Introduction to the Study and Outline of the
Phases
In September 2008, Quest Partnership Ltd, PsyAsia
International, and the Hong Kong Institute of Vocational Education
(HKIVE) embarked on a project to translate the Identity Self-Perception
Questionnaire from English into traditional Chinese. The reason
for translating the questionnaire was to produce an occupationally
focused personality questionnaire that could be used in China and
Hong Kong SAR. At the same time, Quest were also producing a new
Careers Report for the Identity system. This enabled the volunteer
students to gain useful feedback on their questionnaire. The project
was headed up by Max Choi of Quest Partnership Ltd and Dr. Graham
Tyler of PsyAsia International. Max Choi is an Occupational Psychologist
with BPS chartered status and has substantial experience in designing
and validating tests. Graham Tyler is a registered psychologist
and has a PhD based on psychometric assessment and validating tools
for predicting performance at work in Asia.
The research was split into several stages:
• Translations – involving the translation
and back-translation of Identity into Simplified and Traditional
Chinese by professional staff at HKIVE.
• Pilot Study – using the translated
Identity questionnaire.
• Phase 1 Testing – a sample of participants
at HKIVE completed the Chinese Identity questionnaire.
• Phase 2 Re-testing – participants
were asked to complete the questionnaire for a second time one month
later i.e. re-testing to determine the reliability of the questionnaire
items.
• Data Cleansing – first to identify
and remove ‘rogue’ answer sheets from students who did
not complete the questionnaire seriously.
• Data Analysis & Results– analysis
of the data and understanding the results.
• Producing Norms and Building this into the New
Career Focus Report – norms were produced based on
these Hong Kong students. This norm group was used for the new Career
Focus Report which is now available for the Hong Kong education
sector.
• Translation into Simplified Chinese –
the project to translate the Identity Questionnaire into simplified
Chinese and have it available online was completed in December 2009.
Translations
In September 2008, the questionnaire was translated
into both Traditional and Simplified Chinese by 4 individuals at
the HKIVE who hold the British Psychological Society's Level A and
B Certificates of Competence in Occupational Testing. This process
was supervised by Dr. Graham Tyler, who has a good understanding
of principles behind item construction. The translated questionnaire
was sent to the test publishers (Quest Partnership Ltd) in the UK
for evaluation and further refinement, working with Chinese natives
now resident in the UK.
The translated questionnaire was then back-translated
into English by lecturers in the English language department at
the HKIVE. Independent back-translation provides the quality check
of how effective the translation has been. The back-translation
was checked against the original version of the questionnaire to
ensure it retained its overall theme and meaning. A few items achieved
poor back-translations and these were reviewed and improved and
back-translated again to check that the translation had improved.
The traditional Chinese translation took precedence on the basis
that this would be evaluated first and then simplified Chinese would
follow at a later date.
Pilot Sample
20 students at VTC completed the translated traditional
Chinese questionnaire. They also completed a form which collected
their feedback on items that they did not fully understand or where
they felt the wording could be improved. This feedback was analysed
and a few minor improvements were made for the next phase.
Phase 1 Testing
In October 2008, a large sample of 800+ Chinese
students at HKIVE completed the Traditional Chinese Identity Questionnaire.
Most of these administrations were conducted under standardised
test administration conditions during classes. The final sample
after data cleansing consisted of 421 students.
Phase 2 Testing
One month later many of the Chinese students from
the Phase 1 testing were invited to complete the questionnaire again.
The test-retest study is based on 206 students who completed the
questionnaire again. Most of these administrations were conducted
under standardised test administration conditions during classes.
Students were entered into a monetary prize draw
as an incentive to take part in the research. Also, students received
a Career Focus Report from their completed questionnaire.
Data Cleansing
Identifying ‘Rogue’ Responses
We placed stringent requirements on the data that
could be used. It was evident that a proportion of the student responses
were not usable. This may be as a consequence of asking the students
to complete the questionnaire as part of class work. So although
they were volunteers, the request during class time may have resulted
in some slightly ‘reluctant’ volunteers. Also, others
may have become bored after starting the questionnaire and may not
have taken the whole questionnaire seriously, unlike real candidates
applying for jobs. So a small minority will complete the questionnaire
in a non-serious manner. Only a few rogue answer sheets can be visually
identified (e.g. students who have put in the same response for
the whole column or making neat zig-zag patterns on the answer sheet).
So we needed to employ more sophisticated techniques to identify
other ‘rogue’ respondents in order to remove these from
our sample before conducting further analysis on the data.
Removing Answer Sheets with Too Many ‘3’ Responses
The instructions for completing the questionnaire clearly states
that 3 should be used sparingly. But for this Chinese student sample,
the mean number of ‘3’s chosen was 30.4, with a Standard
Deviation (SD) of 32. For our UK sample however, the mean number
of ‘3’s chosen was 9.85, with a SD of 15. It was decided
that participants who responded with over 71 unsure ‘3’
responses would be removed from the sample i.e. this means that
they are putting down ‘3’ to over a third of their questionnaire
items – which is much too high. A caveat to this however is
that given the “middle-way” philosophy in the East,
it can generally be anticipated that central tendency responding
will be higher in China than in the West.
Removing Answer Sheets with Random Responses
We employed two established methods to detect answer sheets which
were being completed randomly i.e. the True Response Inconsistency
(TRIN) and the Variable Response Inconsistency (VRIN) methods. Both
methods are based on paired items which are highly associated in
that knowing an individual’s response to one item will provide
a very high level of prediction of their response to the other item.
Therefore, when a person scores below a certain threshold with many
paired items, we can be confident that their responses to the questionnaire
have been random.
Data Analysis and Results
Test Re-Test Reliability
At Phase 2, students completed the Identity Questionnaire again
about one month after the Phase 1 original completion of the questionnaire;
we were able to conduct a Test-Retest analysis. This allows us to
look into the stability or reliability of the questionnaire over
time.
The final sample size for the test-retest was 206
after all the data cleansing procedures were conducted. Overall
the vast majority of Identity scales were reliable. A small number
of scales were below the benchmark of .70. However we need to be
reminded that we are dealing with a translated questionnaire so
we would expect some loss of reliability compared to the original
questionnaire. So the original English Identity questionnaire sets
the upper limits.
The original English Identity single scale test-retest
coefficients ranged from .77 to .92 (based on a test-retest sample
of 121). For the translated traditional Chinese questionnaire the
test-retest coefficients ranged from .58 to .87. Seven of the 36
Identity questionnaire scales reported less than ideal test-retest
coefficients:
Interestingly, it might be argued that these scales
are less meaningful to this student sample and different results
are likely to be obtained in a business sample.
Internal Consistency Reliability
Another method to determine reliability is to look at internal consistency
of each scale to see how well items within a scale correspond with
one another. From this analysis we identified nine scales at a lower
range of reliability coefficients than our ideal of 0.7:
Combining the two methods of establishing reliability
it was useful to see if there were any scales that would have both
low test-retest and low internal consistency reliability. The following
2 scales had lower reliabilities than ideal:
• Empathy
• Adaptability
We will be collecting more data so with more extensive
use of the tool with participants who will be completing the questionnaire
for non-research purposes we do expect the reliabilities to improve.
Study Results: Comparisons with UK Data
The results for this group of Hong Kong students were compared
against the UK working population and also against a group of UK
A Level applicants and Final Year Students for a Design & Technology
course at a UK university.
The group of Hong Kong students compared to the other groups tended
to be slightly lower on the following scales:
However, it is not possible to determine exactly why these differences
are found as there are a range of variables as to how the groups
differ from each other e.g. motivational aspects as the students
were volunteers rather than real job applicants; age differences;
cultural and educational experience differences; work experience
differences.
Producing Norms & Developing the Career Focus Report
A set of Hong Kong student norms has been established
(N= 421) and more data will be added to this at a later date when
it becomes available.
At the same time as this research Quest Partnership
also developed a new Career Focus Report for Identity and participating
students were provided with a report. This new report has been developed
with educational clients in mind but can be used by other clients
supporting individuals with career guidance. Currently, the report
can be normed against the UK working population and the Hong Kong
students.
Translation into Simplified Chinese
The project then made traditional Chinese available as an online
solution for clients with a view to collect on-going norms data
and to work with any clients who can support with validation studies.
In December 2009 the simplified Chinese version was also made available
online.
