Home Home | About Us | Sitemap | Contact  
  • Info For
  • Professionals
  • Students
  • Educators
  • Media
  • Search
    Powered By Google

Playing Around


by Robin Stanton Gerrow

Gamified Assessments Prove Valid Alternatives to Traditional Psychometric Measures

Cognitive testing tools have been digitized over time from the traditional pen-and-paper versions, but are generally still the same old questions and quizzes. With gamified testing—which has been found to be effective across generations—scoring is quicker, and research shows the test-takers are more engaged and motivated.

But how do those games compare in their ability to measure cognitive ability—an important predictor of job and career success—to the tried-and-tested traditional delivery methods?

In “Validating a Machine Learning Algorithm for Measuring Cognitive Ability Using Games,” which will be presented in April at the 2018 Conference of the Society for Industrial and Organizational Psychology in Chicago, Illinois, lead researcher Sonia-Cristina Codreanu and her team approached that question by examining the validity of six digital games designed to measure cognitive ability in the general population, and using machine learning algorithms in the scoring process. 

Crucially, they wanted to verify that game-based assessments are legitimate alternatives to evaluate cognitive ability—and they did. The research showed the gameplay scores to be highly correlated with traditional cognitive ability measures, thus indicating that they can be used as valid measurements in both research and practice.

Gamification is part of the trend toward integration of AI and machine learning into human resource processes, as noted in SIOP’s 2018 Top 10 Workplace Trends.

“Cognitive games present several key advantages when compared to questionnaire-based assessments of cognitive ability,” said Codreanu, doctoral candidate at University College London and a business psychologist with MindX. “Games create a more engaging and motivating test taking experience."

“Participants generally report higher enjoyment and a heightened sense of flow when taking gamified assessments,” she said. “Test takers also receive continuous feedback on their performance as they progress through various levels of difficulty, thus creating a more motivating testing experience.

Importantly, the reframing of assessments as games can lower anxiety levels in test takers, reducing stereotype threat and minimizing adverse impact.”

She collaborated on the research with Franziska Leutner, PhD, a teaching fellow in psychology at University College London and business psychologist with MindX, and Adriano Soares Koshiyama, a doctoral student in computer science, also at University College London, and data scientist with MindX.

“The first step is collecting data for each test taker’s gameplay behavior,” Codreanu said. “Given the richness of games, we collect a multitude of data points during gameplay, including final level reached and accuracy rates. The relationship between gameplay and cognitive ability is then modeled using different statistical methods and the performance of these different models is compared. The model that yields the highest concurrent validity with cognitive ability in a test sample is then selected and used as a subsequent scoring algorithm for the games. “

The research was conducted in the summer and fall of 2017. In the study of 328 participants, with an age range of 17 to 54 years, the six games were timed at 20.5 minutes, with the overall cognitive ability score available at the 30-minute mark, showing the potential to significantly decrease assessment time, as well as opportunities to increase the quality of data gathered.

Gamification of cognitive testing can complement the use of AI and machine learning in the talent analytics function of an organization by creating new evaluative tools for hiring. As noted in "Trends and Practices in Talent Analytics," by Jasmit Kaur and Alexis A. Fink, this type of work falls squarely into the category of organizational research examining broad human resource issues across companies, by helping organizations quickly identify qualified candidates and improving the staffing process.

And, while the initial thought might be that this type of testing would appeal most to millennials, Codreanu said the games make the test-taking process more engaging across generations.

“The games are designed to be intuitive and easy to understand, thus making them appealing among many different groups of individuals,” she said. “Previous gaming experience or technological skill shouldn’t offer any advantage in terms of performance on these games, given that they don’t require any motor skill coordination or other skills that are typically more developed among video game players. The games are simply intended to provide a more engaging and motivating testing environment for a cognitive ability assessment. “

In low unemployment environments, gamified assessments are one way that employers can quickly, and accurately, find the right person for the right job. According to Jennifer Alsever in Fortune, “Predictive algorithms and machine learning are fast emerging as tools to identify the best candidates.” While AI is ubiquitous in the recruitment stage, companies are now using it to create tools that will assess candidates based on the skills they will need daily in their job—and continued gamification of those assessment tools is picking up steam.

“Given the range of employment opportunities that exist for job seekers these days, improved candidate experience for psychometric tests used in recruitment and selection has become a must for employers,” Codreanu said. “Assessments have to be short and predictive of future performance. Game-based assessments can offer valid, shorter and more engaging alternatives to traditional, questionnaire-based assessments. As game-based assessments allow collecting higher quantities of data during shorter testing times they are well suited to using AI. “

While the games investigated by Codreanu and the team were designed for measuring cognitive ability as a predictor of future job performance, further research is being done to gauge the use of games in other aspects of the hiring process, such as job and organizational fit.

“The current games are designed to measure ability, and as such are focused on the measurement of domain-general traits that work across different jobs, industries, or seniority levels,” she said. “Cognitive ability has been proved to be the best predictor of job performance across many different jobs, functions and industries. Nevertheless, job and organization fit are also of paramount importance and a comprehensive prediction of job performance should not only include cognitive ability, but also other more job- and organization-specific traits and abilities.

Game-based assessments are very versatile and can be designed to measure a range of constructs. Despite the main focus in past research being on cognitive ability, there is emerging research focusing on adapting gameplay to measure personality and values. “

This presentation is part of the SIOP 2018 Annual Conference Poster Session on Friday, April 20 at 4 p.m.