How virtual beauty try-on tech has evolved, and where it’s going next

In the last few years, beauty AR and AI tech has become increasingly realistic (remember the buzz over the TikTok Bold Glamour filter earlier this year?). But it hasn’t always been that way, particularly in the virtual beauty try-on space.

Wayne Liu, chief growth officer at Perfect Corp, which partners with brands like Benefit, E.l.f., and Mac on virtual try-ons, told Retail Brew people were “suspicious” about how well the tech would work back in 2015.

“In the early days of our company, we did lots of more evangelist-type of work. We tried to educate the customer and try to get them to trust and believe,” Liu said.

• Its tech was making its way through different test markets abroad when Covid completely accelerated it, Liu said, and the tech has since become “more like table stakes” for beauty brands.

While virtual try-on tech is still a developing concept in other categories like apparel, innovators in the beauty space have worked to improve the customer experience to be as close to the real thing as possible, working to match the sheen of a lip gloss or the nail polish on a moving hand to make it an increasingly essential service for beauty brands.

“[In apparel], it’s just one of the tools that you use as part of your merchandising, whereas with beauty, it seems like it has a lot more potential to actually be one of a major part of the online beauty shopping experience,” Laura Kennedy, CBInsights principal analyst, told Retail Brew.

Nobody’s perfect: When Perfect Corp first began, it was focused largely on AR, but the addition of AI machine learning and shape mapping technology helped the company feature a more inclusive range of skin tones, Liu said. And while it previously relied on identifying “landmarks” of the face for try-ons, it’s now using physical-based rendering, which gives a “more accurate understanding” of the light around an object while it’s moving.

Liu said foundation was “quite challenging,” and is the reason it started involving AI into the tech. The company conducted panels with groups of around 50 consumers to get a better understanding of skin types and skin tones.

• Mascara wasn’t easy either, he admitted. Eyelashes are teeny tiny aspects of the face, and can be even more differentiated than face shapes.
• Its most popular try-on product, lipstick, is also getting “complicated,” Liu said. It’s not just about color, but how a product glitters or reflects on the lips.

Now, it says its technology has boosted sales and time consumers spend on brands’ websites, and lowered brands’ return rates. Across its 500+ brand and retailer partners, it’s also creating more individualized experiences, like a brow try-on for Benefit Cosmetics and a Looks feature for Mac that lets consumers try on a full makeup look using a combination of products.

“The engine is pretty much developed by us, but the customer experience is heavily impacted by the feedback of the customer,” Liu said.

Snap to it: Snapchat, too, continues to double down on virtual try-on within different beauty categories, last week announcing new try-on lens partnerships, one with Supergoop to support a new line of Lipshades, and another with OPI for a range of nail polish colors. The OPI lens introduced new Snap AR capabilities using segmentation tech and advanced nail tracking to ensure the color stays put even as the fingers move around.

“We’re able to do so much more across different categories…the tech that [the OPI lens] uses, it allows you to really recognize the nail, do different textures and lighting. So it really is not just a flat color right now on your nails; you’re able to get more dimension,” Michele Nevitt, head of US CPG, health, and beauty at Snapchat, told Retail Brew.

And these lenses don’t just create fun snaps to send to friends. In a partnership with Ulta Beauty, consumers tried on Ulta products 30+ million times, driving $6 million in incremental sales. Beauty brands that work with Snapchat on these lenses are “building an immersive experience, especially with connecting with Gen Z”—a move that “is really what’s going to drive brand loyalty, brand affinity,” Nevitt said.

What’s next: Liu said “phase two” of virtual try-ons will focus on personalization. Mac’s Trend Try-on Tutorials allow customers to scan a QR code, pick a trend (we went with the Washed Denim Look), and watch as a phantom eyeshadow brush paints light blue shadow on your lids, gel eyeliner across your waterline, and lipglass on your lips that, yes, adjusts to the changing light reflection as you move your face.

Nevitt said Snapchat hopes to grow its AR experiences in beauty categories like skin care and fragrance—though we’re not quite in a place where consumers will be able to smell perfume through the phone (yet).