Producing faster, more detailed feedback for students to improve educational outcomes
The Multiverse platform helps companies identify, close, and prevent skills gaps through personalised, on-the-job learning. When Anna Wang joined Multiverse as their head of AI, she looked for opportunities to leverage LLMs to solve learning experience challenges from the ground up. One of those problems: automating feedback for students, so they could receive high-quality, highly detailed feedback nearly instantly.
While developing the feature, Anna felt strongly that humans need to be kept in-the-loop to develop the AI ethically. Initially, to evaluate the output of Multiverse’s AI model, product managers relied on managing spreadsheets sent around to subject matter experts like educators, assessment designers, and engineering. The spreadsheets were critical in ensuring the quality of the AI model prior to launch, but the reliance on these documents created a logistical and collaboration bottleneck.
Systematizing releases with humans and LLMs
Seeing her team’s velocity slow down due to spreadsheets, Anna searched for a better solution and chose Gentrace to address Multiverse’s challenges because of its UI and support for both human and LLM-grading.
With Gentrace, Multiverse has:
- Involved nontechnical subject matter experts including education coaches, product managers, and designers in the AI evaluation process
- Created a high-quality release process where LLM evaluation is used for local development and human grading prior to release
- Setup a strong regression testing system so that future capabilities don’t make existing functionality worse
Anna X. Wang, Head of AI at Multiverse, said:
“Gentrace allows our ML engineers to work cohesively with other engineering teams, product managers, and coaches. Combining AI and human evaluation really helps us move faster and be more confident in our deployment of AI to benefit our customers and learners.”
Our partnership with Multiverse
We look forward to continuing our partnership with Multiverse.
Multiverse is a tech company that identifies, closes and prevents skills gaps, through personalised, on-the-job learning. It partners with over 1,500 companies in the US & UK to deliver skills that enable business transformation, and economic opportunity for individuals. Its focus areas include AI, tech and data skills, for people of any age and any career stage. Multiverse learners have driven $2bn + ROI for their employers, using the skills they've learnt to improve productivity and measurable performance. In June 2022, it announced a $220 million Series D funding round co-led by StepStone Group, Lightspeed Venture Partners and General Catalyst. With a post-money valuation of $1.7 billion, the round makes the company the UK's first EdTech unicorn. For more information, please visit https://www.multiverse.io