Ratings
To generate insights and ultimately arrive at confident results, users can rate decision options. Instead of rating options with a single, all-encompassing (and therefore also quite meaningless) metric, users of PrioMind can rate options against a set of more specific criteria.
Breaking ratings down into better defined criteria removes some of the guesswork and subjectivity. It delivers a more robust basis for facilitating alignment and, ultimately, for determining an option's overall score.
EXAMPLE
Picture this: You're asked to rate two cars on a scale of 1 to 10. Sounds simple, right? But wait – what exactly are you rating? Style? Speed? Fuel efficiency? Suddenly, it's not so straightforward.
This is where PrioMind shines. Instead of a vague "overall" rating, we break it down into specific criteria, each focusing on a different aspect of the decision. The result? Ratings that are more informed, more precise, and ultimately more useful for making that big decision.
Default Criteria Set
Say hello to SOURC·E®, PrioMind's secret sauce for stellar decision-making. This powerhouse set of criteria is available to all users, whether you're on a free plan or living the premium life. Let's break down this acronym:
- S for "Strategy contribution"
- O for "Opportunity and potential"
- U for "Urgency"
- R for "Risk mitigation"
- C for "Capability fit"
- E for "Evaluation confidence", a meta criterion
The set covers a wide range of use cases and should be sufficient to reach decisions of high quality. The meaning of the criteria is explained in more detail in the docs' criteria section.
SOURC·E® is based on a triangular sequence with the values 1, 3, and 6 to represent ‘low', ‘medium' and ‘high' ratings. This scale is very simple and yet ensures that high ratings get an unproportionally higher score and really stand out.
Meta Criteria
Meta criteria encourage you to take a step back and to "rate your rating." They are like the volume knob on your decision-making amplifier.
Take "Evaluation confidence" from our SOURC·E® set, for example. At a high level, it actively boosts or dampens your other ratings by:
- rewarding high ratings of the other criteria
- penalizing low ratings of other criteria
EXAMPLE
Let's see the effect of a meta criterion in action – here with our default criteria set and its meta criterion "Evaluation confidence":
| User | (Non-meta) Criteria Ratings | Meta Criterion Rating "Evaluation Confidence" | User Score |
|---|---|---|---|
| A | all 6 | 6 | 150/150 |
| B | all 6 | 1 | 75/150 |
| C | all 1 | 1 | 50/150 |
| D | all 1 | 6 | 0/150 |
"Evaluation confidence" as meta criterion is pivotal as it differentiates between informed assessments and mere opinions. It addresses the German wisdom "viel Meinung für wenig Ahnung" (loosely, "a lot of opinion for little knowledge"), helping to identify where ratings are backed by solid understanding versus where they're based on less substantiated views.
This meta-criterion not only enhances the accuracy of decisions by weighing ratings based on confidence levels, but also pinpoints areas requiring further expertise or research. Ultimately, it serves as a built-in quality check, significantly boosting the robustness and reliability of the entire decision-making process.
Collective Confidence Level
For each option, PrioMind calculates the standard deviation across all ratings to determine how confident the group of users is about a given option. The confidence level is given as percentage and impacts an option's score.
EXAMPLE
Let's say ten users rated an option with an average of 100/150. However, PrioMind also calculated the standard deviation of the ten ratings, showing that the collective confidence level sits at only 65%. This will result in an overall score of 65/150.
The options with the lowest confidence are shown on the decision's page as Top-Diverging Positions to encourage discussions and increasing a group's confidence level on the given option.
PrioMind further facilitates these conversations by showing on an option's page which users have the most notable diverging viewpoints on which criterion.
Overall Score
The average of all ratings by all users, weighted with the collective confidence level, results in an option's overall score. The score is given in absolute points, i.e., something like 128.9/150.
The score is then used to determine the Top-Scoring Options.
Diverging Positions
On each option that has been rated at least once and hasn't been discarded, PrioMind shows detailed diverging positions. If the collective confidence level of an option is low, we recommend addressing these diverging positions.
PrioMind calculates the largest deltas amongst all of an option's ratings – or the largest "distance" between the positions of any users on a specific criterion. Large deltas are a possible signal that someone might know something relevant the others don't. We consider deltas of 20% and higher to be relevant for discussion.
To address the largest deltas, engage your team and advisors in a conversation. Get the discussion going by asking the "position leaders" to interactively give their reasoning behind their rating of the respective criterion.
Addressing diverging positions costs time and effort. Be sensible about which positions to address. Start with the options that PrioMind indentifies as top-diverging positions and address the largest deltas within those options first.
WHEN ARE DIVERGING POSITIONS ADDRESSED ENOUGH?
You've successfully addressed diverging positions if knowledge gaps were filled and people can reevaluate their ratings under the light of new information.
Emphasis is on ‘can:' Even after a valuable knowledge exchange, people might keep their diverging positions. Typically, one can observe some alignment effect, but there are occasions where deltas remain high even after several exchanges. That's OK.
The important thing is to identify the diverging positions and give your group the fair chance to converge on a more aligned position. The recognition and the fair chance are the important things here, not necessarily a converged position.