Tech Ranking

Designing the interface for a new 'check-your-rank' service for tech company Reputationaire

OVERVIEW

The Problem

'Tech Ranking' is a new algorithm-based service created by tech company, Reputationaire. It takes a developer's GitHub or StackOverflow profile and calculates how they rank compared to others on that platform. Recruiters can check an applicant's rank by entering their profile name, as provided on their CV. Only problem?

  • Cool concept. But would recruiters and employers actually find it useful?
  • Would developers use it to check their rank?

The Goal

Design the interface for Tech Ranking that will

  • help tech recruiters in their recruitment process
  • encourage developers to check their own rank

My Role

Team Lead
UX Research
Landing Page

Remote Team

Peggy Wei
Franchette Fabila
‍Jeremy Sariaatmadja
Luisa Fernanda

Timeline

2 Weeks | June 2020

The Solution

  • One landing page. Two different users.
  • We gave CONTEXT to the rank and helped recruiters understand the rank as an indicator for both technical skill AND transferable skills.
  • We tried to gamify the experience for developers using a badge system. It didn't work. They wanted to know HOW the rank is calculated.
  • As a service, especially for recruiters, we recommended improvements
Go to Hi-Fi Prototype ↓

OUR PROCESS

1

Research

2

Define

3

Ideate

4

Prototype & Testing

Research |

What is Tech Ranking? 

Great question. We didn't know either. To understand the brief, we needed to start with some good old desk research, and delved into the world of developers.

Tech Ranking calculates a developer's rank on GitHub and Stack Overflow.
‍
Github Icon
GitHub is an online code sharing platform. Anyone can use it to share their own projects, or contribute to others. There are 44+ million users around the world.
Stack Overflow Icon
Stack Overflow is a question and answer site for developers. Think of it as Quora or Yahoo Answers but for programming. There are 50+ million visits per MONTH.
With more questions than we had started out with, we headed to our discovery workshop with the client, thinking we'd just be meeting with one person - the developer-turned-Founder of the company.

Instead, the video call was joined by another 3 people from his team. It was a party.
Our Team
Client Team
UX Designer
UX Designer
UX Designer
UX Designer
CEO/Developer
Developer
Marketing guy
Developer
I facilitated the session, covering topics from a design, business and developer perspective. Here, my secret weapon came in handy. Having programmed many psychology experiments in the past and used GitHub and Stack Overflow myself, this enabled me to communicate much more effectively with the client team.

We discovered Tech Ranking was intended to:
  • help developers lacking in local work experience prove their skills when applying for jobs. Developers who may not have local work experience, but have done a tonne of work on GitHub or Stack Overflow.
  • help recruiters or employers, especially those without a technical background, quickly assess an applicant's technical skills and engagement in the community.
The developers had already started on building something. We could immediately tell that a recruiter, especially without a technical background, would not know what the rank indicated.
Checking my own rank on Github with Tech Ranking. Not bad for an amateur!
These main takeaways became our starting point:
  • We had two clear user archetypes: developers and recruiters.
  • This service would help both of them with applying or recruiting for jobs

Quantitative Research

To understand the users, we investigated the two archetypes with two separate surveys.

Developers

51

Survey Responses
Goal: Gauge demographics.
Find out how many developers use GitHub and Stack Overflow when applying for jobs.
For the developer survey, I took a closer look at patterns in the data using Excel.

Recruiters

20

Survey Responses
Goal: Gauge demographics.
Find out how many recruiters consider an applicant's GitHub or Stack Overflow profile.
For the recruiter survey, Google Forms sufficed. Thanks Google!

Qualitative Research

We dug deeper into their experiences around applying or recruiting for jobs.

Developers

10

Interviews
Goal: Find out how developers are engaging with GitHub and Stack Overflow.
Pain points around landing a job.

Recruiters

8

Interviews
Goal: Find out how recruiters recruit, specifically for developer positions.
Their needs, goals and pain points.
It was a whole new world to us. In fact, it was TWO whole new worlds.
Clustering insights from recruiter interviews.

What We Found

From the quantitative and qualitative data, we found two groups of developers - those local to Australia and those who migrated to Australia.

Do DEVELOPERS use GitHub and Stack Overflow? Absolutely!

90.2%

Use BOTH GitHub and Stack Overflow

But do they use them for job applications? Nope.

70%

Locals

77%

Migrants
Use NEITHER GitHub or Stack Overflow to increase their chances of getting hired

Why not? For locals starting out in the industry, it's simply not having the skills yet.

"It's not easy...normally you have to devote a lot of time into understanding what's happening before you can make your contributions" 

For migrants with more years in the industry, they thought it didn't accurately show a developer's skill.

"You might have a GitHub but that doesn't show you really have the skills for it. The same goes for Stack Overflow"

Do RECRUITERS look at an applicant's GitHub and Stack Overflow? Yes, most do!

55%

Recruiters will look at an applicant's GitHub/Stack Overflow when screening
"They're additional data points...kind of nice-to-haves. If they're on these platforms I know they're passionate about technology" 

Do they understand it? Well...

"As a tech recruiter, if you don't have that deep knowledge about the language like Java, Python etc... it can get confusing!" 

Define |

Three Personas Emerged From Our Research

Dave the Developer

  • Recent Computer Science graduate looking for a job
  • Lacks work experience and personal projects to show on his CV 
  • Does not contribute to projects on GitHub or answer questions on Stack Overflow

Madhu the Migrant Developer

  • Developer with 5+ years experience in the industry
  • Recently moved to Australia and is looking for a job
  • CV is getting overlooked because he lacks local work experience and recruiters don't understand his profile
  • Contributes to projects on GitHub but doesn't use it for job applications

Rob the Recruiter

  • Tech recruiter with 10 years recruiting experience
  • Doesn't have a strong technical background
  • GitHub and StackOverflow are 'nice-to-haves' - secondary information to previous work experience and education
The team then split in half to map out Dave's and Rob's journeys. I was on Team Rob!
We decided Team Dave would focus on Dave (instead of Madhu) because this persona made up the most in our quantitative data.

To understand Rob's recruitment process, I mapped it out on a Journey Map

Creating Rob's Journey Map on Miro. Many post-its were moved around before we settled.
This was DIFFICULT. We learnt from our interviews that every recruiter has a different recruitment strategy. My team member and I couldn't settle on one order of events.
Clarity came when we joined with the other half of the team. We decided Rob's journey would reflect the process of recruiting for a graduate position, with Dave as one of the applicants.

With the whole team, we brought together Rob's and Dave's journeys

Rob's journey (blue) was integrated into Dave's journey (yellow)

And finally, Madhu's journey was also merged to complete the full picture

To help us explain the complicated interaction between Dave, Madhu and Rob's journeys, we opted for a visual that would help us tell the story when we presented to the client.
Journey map for Dave, Rob and Madhu

Key Insights

We boiled down our research into these key insights:
  • For Rob, technical skills is important but matching an applicant's attitudes, personality and SOFT SKILLS is equally as important
  • Dave and Madhu were NOT using GitHub or Stack Overflow when applying for jobs
  • (Developers don't like LinkedIn. There's a larger community on Twitter)

Some Concerns

From our interviews with developers, we learnt there were some concerns with the algorithm as a service. I communicated these issues to the engineers on the client team who then sought to improve the algorithm.
  • Work that was PRIVATE on GitHub/Stack Overflow didn't get taken into account by the rank. (In one case, I ranked higher than an experienced developer. Now THAT'S a problem!)
  • If you were really sneaky, you could 'fake' a good profile which wouldn't accurately reflect an applicant's skill or contributions to the community

Ideate |

Designing for Dave + Madhu + Rob

We pulled out the most important HMWs from the journey maps and ideated from them.
One for the developers, Dave and Madhu, and one for our recruiter, Rob.

HMW show or confirm a developer's soft skills with Tech Ranking?

HMW help Rob understand the value of Tech Ranking to aid in the selection stage?

We ideated with Crazy 8s, placed the ideas on an MVP Matrix, and voted for the best ideas.

We integrated the winning ideas into the user flow.

Winning Idea #2. Landing page

Creating the User Flow

We started with a task flow.

One landing page.
Two users.

Then sketched the wireframes.

Prototype & Testing |

2

Iterations

14

User Tests

3

Engineers
collaborated with!

Landing Page

I wrote the copy for the landing page and mocked up this first screen while my team members worked on the other screens.
The biggest constraint here was the algorithm being confidential. We needed to explain how the rank gets calculated, without actually explaining it. Fun times.
‍
I ran the copy by the client, who made some edits, and we got the okay! 

We ran user testing on 5 Developers + 3 Recruiters

SUCCESS!

88%

Understood what the web app does and navigated to the profile page with no issues

Developer's Flow (Mid-Fidelity)

Recruiter's Flow (Mid-Fidelity)

Graphing the Rank

To help users understand the rank, we tried to show it visually.

It was a BAD IDEA. People were EVEN MORE confused.

"It seems counterintuitive that it starts at 100% and goes down to 1%"

So we stuck with showing the rank as a single number.

From insights, we knew recruiters often needed to match applicants by programming language e.g. Java, React etc., however, after conferring with the engineers on the client team, we discovered this wasn't possible.

Also, users wanted more CONTEXT on the rank

75%

Users were asking how the rank is obtained
So we pivoted.

We gave context to the rank by highlighting the soft skills that it could indicate and added an info icon to explain how the rank is calculated on the profile.

The Result?

We validated the prototype on 3 Developers + 2 Recruiters and averaged their ratings.

8.4/10

Usability

8.3/10

Readability

8/10

Overall Experience

However there was also room for improvement.
‍
"Knowing my rank is good, knowing how to improve my rank is better"
- Developer
"It looks like a tech job board such as Indeed etc..."
- Recruiter

High-Fidelity Prototype

Developer's User Flow

Recruiter's User Flow

Back To Top ↑

OUR IMPACT

"I'm impressed by how much you've done in just 2 weeks. There's a lot of information and data, super valuable data which we're going to use elsewhere in our business."
- CEO
"In the case that they contribute only once to a highly starred repo - thanks for bringing this to our attention! ...We appreciate feedback like this so we can constantly improve"
- Engineer

REFLECTION

What I Learnt

  • COLLABORATING WITH ENGINEERS. This project was unique in that we worked with engineers (the client) to build a product for engineers. I learnt a lot from both working with the client and interviewing users and definitely have a better grasp on 'speaking their language' now.
  • ALGORITHM AS A SERVICE. Data science, AI, machine learning etc. is just going to get bigger and bigger. Giving users some understanding of how it works and what it's doing is a must and this is where UX/UI is extremely valuable.
  • DESIGNING FOR TWO. Bringing the two archetypes into the one product was a challenge. We learnt to prototype and test their flows SEPARATELY. Sounds obvious, but it's easy to forget when you're in the middle of a sprint.

What I Could Have Done Better

  • Defined scope of project for timeline. Designing for either the developers or recruiters is a sprint within itself. I felt that we could have delivered a better result if we only focused on ONE of the archetypes in the 2 weeks.
  • Set expectations on Day 1. We could have made it clearer that we weren't there to brand their product, we were there to DESIGN a product for their users. We were asked if we could provide a logo/name/colours on Day 2 (which we took on because we love giving ourselves more work) but in the end, it took away from the HCD work we were doing.