Facebook is launching a “universal product recognition model” which is set to transform they way consumers purchase goods. It looks towards a future where the products in images on the site can be identified, classified and then purchased in one place. This concept os “social first” purchasing may not be new, with influencers using the swipe up function on Instagram stories to enable their followers to purchase what they are wearing via an affiliate link. But with Facebook taking the technology to another level with the intention of making possible to identify clothing from photos, rather than requiring affiliate links.

In the paper GrokNet: Unified Computer Vision Model Trunk and Embeddings For Commerce it is set out that by using a variety of datasets across several commerce verticals it became possible for the AI to identify various items from clothing to furniture. The system is trained on the input provided by users and the annotations included in their posts. Everytime you posted a photo of you wearing a new #dress or sitting on your favourite #sofa the AI would use these tags to “learn” the difference between the two.

The report states:

For fashion products, we label data for 68 attribute types across 9 object categories (818 total), with attribute types varying from high-level concepts (style, color, material) to fine-grained details (dress waistline, shirt embellishments).

Data collection

For the fashion category, 40,000 public posts tagged with fashion products were used. annotators were asked to validate the tags and draw tight bounding boxes around the product for the post’s image and an image against a neutral background. The Facebook team didn’t only rely on user generated content. It also took 2000 mobile photos of furniture in stores and included user-annotated product IDs.

These methods of data collection highlights two interesting issues – First, the Terms and Conditions users sign up to when joining and posting on facebook may lead some to question the extent of the licence they are granting facebook. Second the photos taken in store may cause those working on the High-street to find it unfair that without there consent their products have contributed to the machine learning of a system that will place further pressure on their margins.

Facebook licence terms

The permissions you give us…

to provide our services, we need you to give us some legal permissions (known as a ‘licence’) to use this content. This is solely for the purposes of providing and improving our Products and services as described in Section 1 above.Specifically, when you share, post or upload content that is covered by intellectual property rights on or in connection with our Products, you grant us a non-exclusive, transferable, sub-licensable, royalty-free and worldwide licence to host, use, distribute, modify, run, copy, publicly perform or display, translate and create derivative works of your content (consistent with your privacy and application settings).

https://en-gb.facebook.com/legal/terms

The licence above is extensive and enables facebook to “use” the images you post for a variety of purposes which are not specified. With this licence, facebook will use the photos with accurate tags to train an AI system to recognise and correctly identify products. This is a good example of how facebook is a data company. It uses the vast data sets available to it (not only personal data) to train AI systems and with vast repositories of images, it is well placed to train a variety of systems.

The taking of photos in store could be more problematic for facebook, particularly when building an AI system to identify luxury products. Many luxury stores and concessions in department stores have restrictions on taking photos, particularly of displays. Following the development of this technology, luxury brands (and hughstreet stores in general) may want to consider whether photos of products should be allowed in store at all. Enforcing this step is another matter.

From the input, facebook will enable users to create listings from taking a photo. But there is also the possibility that this functionality could be used in the opposite direction, with an individual taking a photo of something they would like to buy (like the chairs in your favourite cafe) and then being directed to a seller of the same chairs in the marketplace.

The competitors

Echo Look and Amazon Fire Phone. Amazon has already drawn attention for its forays into the fashion industry, particularly with issues of counterfeits arising. In the recent decision in Coty v Amazon it was held that Amazon was not liable for counterfeit products stored in its warehouse because, despite orders being ‘fulfilled by Amazon’ Amazon was not the active seller.

Amazon has attempted to join the luxury fashion industry for some time and recently announced a partnership with Vogue and the Council of Fashion Designers of America to support the industry during the time of COVID-19. The platform is also enabling consumers to purchase spring collections from some designers. It will be interesting to see if Amazon will use the images (and labelling) of these products to develop a similar system to facebook’s. Given the machine learning of the Amazon Fire Phone, it has a very good tech basis to pivot the technology to fashion.

It is worth noting that AI in the fashion industry is not new. Many people will have already seen the use of AI in the online marketplace with it being deployed on eBay to assist sellers with listings and suggesting pricing. Consumers may want to familiarise themselves with the terms to which they sign up, but the terms are unlikely to indicate the actual uses of the content posted. By using user generated content in this way, facebook is demonstrating the value the platform has in the field of non-personal data as well as personal data.