About the company
Torre.co is a job matching network startup focused primarily on remote jobs, with over 1.5M users. It has a powerful matching algorithm that uses 121 different factors to determine if a person’s genome is a good fit for a job and vice versa.
The problem
One of the most critical metrics in Torre is applications for jobs and one of the most efficient channels for applications was automated job alerts.
Although many users liked receiving job suggestions through standard (automated) job alerts, many others considered them too impersonal.
Hypothesis
If we can suggest full-time and internship opportunities to job seekers in a way that feels human and personal, a significant amount will be more likely to engage with the posts.
Understanding the opportunity
After many conversations with users about the product, we were bullish on building a secondary job alerts system that could leverage our algorithm in a way that felt a lot more personal and premium.
In essence, we wanted to build a “Senior Recruiter” email bot that would send job posts to users and automate around 90% of our one-on-one conversations with job seekers. The interactions had to be carefully crafted to simulate our real recruiters.
For context, the following screenshot shows what job alerts looked like at the time. Sufficiently straightforward for some, a bit cold for others.
In order to build a bot that could pass for a human, there were a few things to take into consideration:
- To ensure accuracy, the bot should only send jobs to users who have enough information filled out on their professional genomes for our algorithm to match them.
- The bot should help users with incomplete professional genomes to complete them.
- The bot should avoid spamming users at all costs.
- Since the most important metric was submitted applications, the bot should be able to guide users to complete their applications if it determines they got stuck at some point.
- The bot should only work during working hours.
Design
From the start, I was aware that this project would require little to no UI design. This experience would happen in conversations between our email bot, job seekers, and real recruiters from Torre. However, in order to send most of the emails, we would have to set up triggers linked to several of our core interactions on the platform. Plus, we would have to create endpoints and integrations with Hubspot for recruiters to chime in when it was necessary.
The deliverable for the engineering team would be a BPM detailing all possible interactions and a document that included every email with 30+ variations for each one to better emulate a human writer.
Building the BPM took me about three weeks of full-time work and many back-and-forth conversations with the CEO. We started out with a very general idea of what we needed but making the bot believable required much more attention to detail than we first estimated.
For example: As I reviewed one of the first versions of the full journey, I realized in order to prevent spamming, the bot would have to ask the user about their frequency preferences (in a non-robotic way). Also, our algorithms manager requested a way to get some feedback about the matches so they could be improved.
In order to achieve this, I came up with a system of auto-replies to the recently sent email. Depending on many factors, after sending a job post email, the recruiter bot would send a reply to that same email with a note.
Some of those reply emails were:
- Ask the user if they should adjust their parameters and send them other types of jobs.
- Ask the user if they would prefer if they sent them jobs with a higher/lower frequency.
- Ask the user if they want to start receiving automated job alerts (in addition to the ones being sent by their recruiter).
Another very important addition was the “Book me for a call” line at the end of each email. Before we decided to add it, I had a strong feeling that something was missing. Even with all the versions of emails, the interaction still felt robotic. The [real] possibility of speaking face to face with María would add the final touch to humanize the interaction.
Of course, this addition wasn’t as simple as a line of text. We had to coordinate with our team of recruiters for them to keep a Calendly and take as many calls as possible. Luckily, the link was rarely used because communication by email was very fast.
In summary, since I can’t publish the BPM, here’s what the bot would do according to the flow:
- Screen people by genome completion.
- Inform users of their missing information in order to activate their dedicated recruiter privileges.
- Send automated job suggestions.
- Change frequency of emails on demand.
- Send feedback reports back to us to adjust the algorithms.
- Offer assistance when a user didn’t complete their application.
- Offer tips and a call with a recruiter after an application had been submitted to improve their chances.
- A set of reminder emails for all of the previously listed emails sent with carefully set timers to avoid spam and give it a human touch.
Prototype
This project could not be tested with a regular prototype. It required a guided one-on-one experience in which testers would go through the entire flow of a new Torre user.
For it, I set up a very basic presentation with Figma that would allow me to show different emails to the user through the video call depending on their response to the previous question.
Usability and value test
I personally tested the prototype with 10 users through video calls, walking them through the journey and asking their thoughts after every hypothetical interaction.
I would purposely avoid mentioning the recruiter would be a bot during the test so I could ask about it at the end.
After each test was finished I would ask general questions about their opinions of the service, the help being offered, and the tone of the emails. Finally I would ask them two questions that they should answer numerically (1= not at all 10 = completely). The first one was about their overall rating of the experience and the second was to what degree they suspected they were talking to a bot.
The results were overwhelmingly positive:
- All 10 users were excited about the idea of getting that level of attention from a dedicated recruiter.
- Six users asked about the cost and found it hard to believe it was free.
- One user was concerned about being spammed but still liked the human touch
- All users ranked the experience an eight or higher
- Only one user said they had suspected it could be a bot but didn’t find that problematic. The others said it didn’t cross their minds.
Usability (again) test
The development took a couple of months. The BPM was extensive, the system touched on many different areas of the platform, and there was little room for error. As the project manager and designer, I was there whenever necessary to answer questions and guide the developer through the details.
A few weeks after launch, once we had solved the major bugs and the flow seemed to be running smoothly, we conducted a usability test with the real product.
We randomly selected 8 users who had received emails from the recruiter bot and asked them about their impressions.
- 4 users had already submitted applications surging from this channel
- 4 of them had no recollection of receiving an email from María.
Excerpts from the results:
“In terms of usability, the majority of testers understand what the role of a recruiter advisor is. For those who remember communication with the RA, it's easy to interact with her, and the communication is clear.”
“In terms of value, most of them highlight the benefit of having a RA. They feel grateful. Also, most of them would recommend the service, and some would like to keep working with the RA to properly refer it to others.”
Analysis
One month after the product was fully launched, we analyzed the quantitative impact it had on the platform. Results were very promising:
- Our hypothesis was proven right. Users were now slightly more likely to apply to jobs sent by their recruiter bot than from regular job alerts.
- We also saw an increase in mutual matches (companies liking the candidate who applied) vs regular job alerts.
- The open rates for the recruiter bot emails were 30% higher on average than regular job alerts.
A year after the product was launched it has become the main channel for applications on the platform. The bot has beat regular job alerts in open rate by a 13% difference and in applications by 16%.
Hundreds of people have been hired thanks to the recruiter bot and it has continued to grow with new functionalities. María (the bot) constantly receives praise for the attention she has given to our users. Here are some of those comments:
★★★★★ 4.7
“I love Torre with all my soul. Since I signed up, Maria and Alexandra were there, sending me job posts and supporting me through the hiring process. Today I started my new job thanks to Torre.”
Giselle Data Scientist
★★★★ 4.0
“Many thanks to you, María, and all the Torre team. I applied for many good opportunities with you and I just started a full-time job. Thanks again for all your help!”
Alejandro Software Developer
★★★★★ 5.0
“Hello María, good morning! I just wanted to email you to let you know that thanks to Torre, your efforts, and everyone on the team who reached out to me, I’ve been hired by Torre. I wanted to thank you for every single email you sent me and for keeping in touch with me to achieve this goal. Hope to see you around in meetings very soon!
Sofía Account Manager
★★★★★ 5+
“Hi María, I am seriously so impressed with your assistance. Excellent service. Where can I rate you? You deserve a 10”
José DevOps Engineer
