The gig economy has been steadily rising over the years and remote work is also being adopted by many companies. Approximately 9% of American workers participated in gig work in 2021 and bolstered the $350-billion gig economy. Freelance work has been gaining traction as around 70% of freelancers reported improved work-life balance and 84% feel that their job gives them purpose.
Many companies and global talent networks that employ gig workers use algorithmic ranking or scoring systems to better manage freelancers. While these systems can be a great way to gain more visibility and insight into remote and fractional workers, poorly executed ranking systems can end up doing more harm than good. How do algorithmic ranking systems affect freelancers, and how exactly can they build – or even destroy – trust?
A recent study conducted by Gigster and Stanford University [1] researched the effect of algorithmic ranking systems on gig workers. The study examined Gigster’s internal ranking system called the “karma system,” which is intended to help with familiarity of workers and reduce uncertainty among the 800 freelancers in the Gigster Talent Network. Here are the key findings.
With work, collaboration and delivery of output all done online, freelancers generate data in the system as they go. This data is collected by the system, analyzed and evaluated according to certain criteria set forth by the company or organization. This process of automated ranking of datasets aligned with a criteria is called algorithmic ranking.
At Gigster, everyone receives their initial karma score when they join the network, which is based on their interview performance during the vetting process.
After the initial scoring, bots in Gigster’s collaboration platform will collect data from project teams’ chat channel or discussion boards and calculate algorithmic ranking scores – even announcing karma awards or deductions in real time.
There are two ways for freelancers to increase their karma score. The first is to deliver milestones on time and achieve high customer satisfaction scores as part of a team project. When a team fails to deliver on time or receives lower customer satisfaction ratings, each member loses an amount of karma.
The second way is through “peer karma” from other members of the network. New members can perform favors or well-defined subtasks from existing projects such as vetting a document, doing research for other members etc. In the same way, freelancers can ask for karma subtractions for peers showing transgressive behaviors.
The study reported high levels of participation among freelancers under the karma system, with active engagement in network relationships, but showed a decline in participation after the study was conducted.
Initially, freelancers were confronted with feelings of uncertainty and frustration with the karma system and its interventions. Having a bot that monitored project channels – and at times suddenly reported karma awards or deductions – felt like an intrusion and an uninvited chat participant looking at your every move.
The algorithmic properties of the system elicited feelings of discomfort and initial distrust towards the system. There was also resistance in the freelancers' day to day interactions with the system.
Programmed automated algorithms also tend to be definite without context. For instance, a team’s score was deducted as they were labeled ‘late delivery’ of the project. However, the system failed to see that the team delivered the project on time, and was only late to mark it completed. These types of scenarios were met with resistance and frustration from the freelancers.
Algorithmic ranking systems can also be opaque and perceived as coercive by freelancers. Research from Northwestern University [2] found that companies do not disclose the criteria and calculation of scores for these systems to avoid users’ attempts to work around the system and artificially boost their scores. However, lack of insight into the ranking system can make awards and deductions feel arbitrary and unfair.
Algorithmic systems can be limiting for freelancers and make them feel powerless and put inside an “invisible iron cage”. For example, one team lost karma scores based on a “false read” of data or systems error, and received warning messages from a chatbot, which became a burden to the team while working. These instances can subsequently erode trust in the system.
As the network operated on the karma system, freelancers later on realized the value of the system in the vetting and identification of the 800 freelancers in the platform. The system began to be a resource that reduces uncertainty among fellow freelancers and helped them to navigate relationships with some base data and insight into each other.
With the gamification nature of the system of “leveling up”, or awarding karma scores based on merit, it amplified positive emotional entertainment, engagement, and pro-social behaviors in the organization.
“These behaviors decreased sharply when the karma system was discontinued and replaced by a non-algorithmic evaluation system that did not include a public ranking,” the study stated.
In contrast, adding greater transparency into the algorithmic ranking system allowed freelancers to verify that it was consistent and accurate with its analysis, calculations, and overall activities. This then improved freelancers’ perceptions of procedural justice within the organization, which in turn built trust and engagement towards the system.
In this sense, freelancers felt that the karma system made decisions based on clear rules with equality among all workers, contrasting this to “biased” and authoritative human managers.
The peer karma function was also received positively as freelancers felt that their opinions matter in shaping the organization. Peer karma allows freelancers to ask the system to award or deduct scores from their co-workers, depending on their positive or negative performance and behaviors.
The formation of relationships was made faster in the karma system as workers are compelled to interact with one another and help each other out to access jobs and gain karma scores.
More experienced members were also open and willing to help new members, creating a space of trust and kindness for everyone. Finding a mentor is more natural and more achievable as people help each other navigate through the system. New team members can also use the leaderboard to easily approach more experienced members or mentors in the platform if they have questions.
Moreover, freelancers bonded over karma. They felt and expressed shared emotions as they collectively interacted with it. This provided consistent and immediate feedback on the system. The quick formation of relationships within the system meant that there was an increase in trust and engagement among the members of the network.
Algorithmic ranking systems are a great way to manage and engage freelancers of global gig platforms and networks such as Gigster - when done correctly. The most important insight we took from the study was that people like objective scoring IF they know the rules and understand how they’re being graded and where they stand in the group. When the rules aren’t clear, the system can have a negative impact on trust and engagement.
You can check out the full Stanford study [3] to learn more about Gigster’s algorithmic ranking system and how we use it to better assemble and manage development teams and achieve greater speed and quality.
1. Lix, K. and Valentine, M. (2020). When a Bot Scores your Karma: Algorithmic Ranking Systems as Uncertainty Reducers in Platform Gig Work. View Study.
2. Rahman, H. (2021). Gig Workers Are Increasingly Rated by Opaque Algorithms. It’s Making Them Paranoid. Kellogg Insight. View Resource.
