- Part of Etsy’s annual hackathon, which in 2023 was focused on GenAI
- Goal was to generate taxonomy-aware alt-text for product images
- At the time, APIs in this regard were much more limited
- Secondary goal was to demonstrate that appropriate models could be self-hosted
- At the time, leadership did not believe that this was possible
- We decided to use BLIP-2 with OPT-2.7B (link)
- We got relevant, if only curt, responses out of the box
- What was more interesting is that, for popular products, the model was very specific descriptions
- We realized that either BLIP-2 or OPT must have been trained on Etsy listings
- While we didn’t have time to do any fine-tuning, we were able to get it deployed on Etsy’s dev Kubernetes cluster
- The project won the hackathon in the accessibility category
- Leadership was also very excited about the deployment, especially as to that point, nobody had deployed a PyTorch model
- If they ask about this, can talk about how severe siloing had caused a lot of teams to start treating things as black boxes