I work at Spotify on ML. I spent quite sometime working on our production vector model pipelines.
We have three components:
1. Training data: Scalding pipelines (could be any large scale production framework) to preprocess training data and reduce it to a few hundred gigs.
2. Run latent vector models (Matrix factorization, Embedding models etc) to generate item vectors. This stage is designed to be non specific to framework. We use typed avro objects to communicate with the previous stage.
3. Production deployment of models. Essentially glue code that ensures that models are versioned, atomic (user, music vectors are all from the same batch) and deployed into music recommendation and realtime frameworks.
We have three components:
1. Training data: Scalding pipelines (could be any large scale production framework) to preprocess training data and reduce it to a few hundred gigs.
2. Run latent vector models (Matrix factorization, Embedding models etc) to generate item vectors. This stage is designed to be non specific to framework. We use typed avro objects to communicate with the previous stage.
3. Production deployment of models. Essentially glue code that ensures that models are versioned, atomic (user, music vectors are all from the same batch) and deployed into music recommendation and realtime frameworks.
Read here[1] more.
[1] https://labs.spotify.com/2016/08/07/commodity-music-ml-servi...