Bighead is internal machine learning system at Airbnb.

Situation in 2016-Q4:

• machine learning used for search rankings, smart pricing and fraud detection
• it took up on average 10 weeks to build a model
• built with Aerosolve, Spark and Scala
• no support for the latest tooling like TensorFlow, Torch or sklearn
• no consistency between machine learning workflows
• new teams struggle using machine learning
• build-and-forget -problems all around

Situation in 2018-Q1:

• machine learning used for classifying lists, room type classification, experience ranking, personalization, host availability, business travel classifier, making listing a space easier, customer service ticket routing
• everything standardized and automated using Bighead

The main focus of Bighead is to:

1. remove incidental complexity by providing generic, reusable solutions
2. simplify the workflow
3. provide tools, libraries and environments for machine learning
4. sharing feature data and model components inside the company
5. make it easy to do the right thing e.g. consistent training/streaming/scoring logic

Bighead architecture:

1. Zipline: data management framework, define features, data quality monitoring
2. ML Automator: offline training and inference, periodic training, alert on score changes, uses Airflow to orchestrate these tasks
3. Bighead Library: provide standard transformations for NLP and images, provide visualizations for the data, pass metadata about the original data samples, simple serializations and deserialization
4. Bighead Service: supplies single source of truth to track model history, make model training reproducible, keeping evaluation metrics in a single place, model health data, model version (model code and docker image), model artifact (weights learned while training), models are always wrapped with a lightweight model API to integrate with other services
5. Bighead UI: deployment rollback, review changes, model health metrics, alerts, split traffic to two or more models
6. DeepThought: online inference service, allow data scientists to launch new models in production, share data transformations with training and inference, median response time 4ms
7. Redspot: multi-tenant Jupyter Notebook environment, JupyterHub with remote instances running containerized notebooks, persistent notebooks in EFS

Sources

• Airbnb meetup presentation about the system