iGoogle. The Fraud Triangle: Understanding threats to test security. Posted by Julie Delazyn There can be lots of worry and planning around assessment security, but there’s not a one-size-fits-all solution.
The Fraud Triangle described by criminologist Donald Cressey provides a useful lens for identifying security threats and understanding how to deal with them effectively. The diagram below shows the key issues in fraud. In order for someone to commit fraud (e.g. cheat at a test), he or she must have Motivation, Opportunity and Rationalization. Motivation: Understanding the stakes of assessments is key to determining motivation and therefore the appropriate level of security for it. What is the Angoff Method? Posted by Julie Delazyn When creating tests that define levels of competency as they relate to performance, it’s essential to use a reliable method for establishing defensible pass/fail scores.
One of these is the Angoff Method, which uses a focus-group approach for this process. This method has a strong track record and is widely accepted by testing professionals and courts. Subject-matter experts (SMEs) review each test question and then predict how many minimally-qualified candidates would answer the item correctly. The average of the judges’ predictions for test questions is used to calculate the passing percentage (cut score) for a test. Basing cut scores on empirical data instead of choosing arbitrary passing scores helps test developers produce legally defensible tests that meet the Standards for Educational and Psychological Testing.
Topic based feedback goes to the ball. In talking with some of our customers last week, I was reminded how valuable it can be to offer participants topic based feedback.
Obviously, everyone wants to know whether they’ve passed or failed a test. And most people look at their feedback on questions they got wrong, to understand how to improve. But you can get a single question wrong for many different reasons including misunderstanding the question, making a mistake or having a tiny gap in knowledge. However, if you score weakly in a topic area, it very likely means you have a weakness in that area that needs addressing. In many ways, topic feedback is the Cinderella in the feedback world. S: 12 tips to writing good test questions. Posted by Joan Phaup Writing effective questions takes time and practice.
Whether your goal is to measure knowledge and skills, survey opinions and attitudes or enhance a learning experience, poorly worded questions can adversely affect the quality of the results. I’ve gleaned the following tips for writing and reviewing questions from Questionmark’s learning resources: 1. Keep stems and statements as short as possible and use clear, concise language. 2. How many items are needed for each topic in an assessment? How PwC decide. I really enjoyed last week’s Questionmark Users Conference in Los Angeles, where I learned a great deal from Questionmark users.
One strong session was on best practice in diagnostic assessments, by Sean Farrell and Lenka Hennessy from PwC (PricewaterhouseCoopers). PwC prioritize the development of their people – they’ve been awarded #1 in Training Magazine’s top 125 for the past 3 years – and part of this is their use of diagnostic assessments. White Paper Assessment Validity and Reliability. Posted by Julie Chazyn Assessments are not all created equal…Those that are both reliable and valid are the superior ones that support learning and measure knowledge most effectively.
But how can authors make sure they are producing valid, reliable assessments? I picked up some tips about this in revisiting the Questionmark White Paper, Assessments through the Learning Process. So, what is a reliable assessment? One that works consistently. And what is a valid assessment? If an assessment is valid, it looks like the job, and the content aligns with the tasks of the job in the eyes of job experts. What makes a good diagnostic question? Posted by John Kleeman What makes a good diagnostic question?
First, it should be almost impossible for someone to get the right answer for the wrong reason: A participant will only get the question right if he/she has the right idea about whatever the instructor wants them to be able to know, understand or do. Second, wrong answers should be interpretable: If a participant chooses a particular wrong response, the instructor should be able to guess why the person has done so and what misconception he/she has. So suggests Dr. Ten assessment types that can help mitigate risk. Posted by Julie Delazyn Mitigating risk –- most notably the risk of non-compliance — is a key component of success in the financial services industry. Other risks abound, too, such as losing customers and/or good employees. If employees don’t understand and follow the processes that organizations put in place to mitigate risk and maintain compliance, the risk of non-compliance increases – and a business is less likely to succeed. Online assessments do a lot to help ensure that employees know the right procedures and follow them.
Here are 10 assessment types that play essential roles here: (1) Internal exams -– check your employees are competent Some companies administer internal competency exams annually –- and do so more frequently. . (2) Knowledge checks – confirm learning and document understanding Running knowledge checks or post-course tests (also called Level 2s) right after training helps you find out whether the training has been understood. Making sure assessment video and audio are accessible to participants. Posted by Noel Thethy One of the benefits of an assessment management system like Questionmark’s is its ability to include rich and interactive media within questions.
But how do you cope with this content to ensure it is accessible to those with needs? Video and audio content When including this type of content in questions, you should provide alternative means for consuming the information.