For the past several years, Python is taking the top spots among the most popular and most loved programming language. According to the 2020 StackOverflow Developer Survey - If we look at technologies that developers report that they do not use but want to learn, Python takes the top spot for the fourth year in a row.
Sifting through thousands of engineering blog posts from leading tech companies, we've found interesting use cases and real life applications of various Python libraries and frameworks.
Python is an indispensable tool for data science and machine learning, so most companies use it in that context these days. It's also a popular tool as a glue language in systems engineering, data engineering pipelines and API development.
If you're curious to learn which company runs the biggest deployment of Django in the world or what popular Apache projects started as Airbnb hackathon projects, read on.
Python at Netflix
In the 2019 article on the Netflix engineering blog, we're told about the major use cases for Python at Netflix.
They use it heavily in service monitoring and management, capacity and failover engineering, big data orchestration.
Personalization systems at Netflix are said to deliver $1 billion of business value a year. Python is an indispensable tool for running these systems, with its entire universe of data science and machine learning tools - Jupyter Notebooks, TensorFlow, Keras, PyTorch, XGBoost, numpy, scipy, sklearn, cvxpy).
Aside from the standard data science and machine learning toolse, Netflix engineers share insights into concrete projects and tools built in Python. Here are several notable examples as featured on the Netflix engineering blog:
- Automation as a Service — Introducing Scriptflask - a tool built to ease creation and scaling of scripts for automated testing
- Introducing Bolt: On Instance Diagnostic and Remediation Platform - a platform providing a simple way to automate common diagnostics and remediation tasks on AWS instances.
- Open-Sourcing Metaflow, a Human-Centric Framework for Data Science - described in the article as a "human-centric framework for data science", Metaflow is a Python library that makes it quick and easy to build and manage real-life data science projects.
To discover more applications and write-ups about Python at Netflix, check out this python at netflix on Blogboard.
Python at Airbnb
In about a dozen great articles on applications of Machine Learning at Airbnb, Airbnb engineers have started more than a few open source projects. Some of the notable examples are Apache Airflow and Apache Superset, both very popular data science tools and both written primarily in Python.
Today, Airflow is a piece of technology every data engineer should know. Airflow was introduced back in 2015 in an article titled Airflow: a workflow management platform. It is a Python framework for workflow management, commonly used by data engineers for data transformation pipelines (commonly known as ETL pipelines). As the article states:
Much like English is the language of business, Python has firmly established itself as the language of data. Airflow is written in pythonesque Python from the ground up.
Being a popular tools, other companies have written about their experiences with it - check the articles in this list to learn how Airflow is used at companies such as Slack, Lyft, Pandora and PayPal.
Apache Superset is a platform for interactive data exploration and visualization. It originally started as an internal hackathon project at Airbnb and was released under the name Caravel. Today it's a top level Apache project and is adopted at companies such as Dropbox. It's backend is pure Python (Flask, Celery, SQLAlchemy, Pandas).
More Python use cases at Airbnb - "python at airbnb".
blogboard.io - Engineering blogs from top tech companies. Search, discover, follow.
Python at Instagram
According to a 2016 article, Web Service Efficiency at Instagram with Python, Instagram is (unsurprisingly?) the world's largest deployment of the Django web framework. The article describes the tooling built at Instagram for service monitoring and performance optimization.
In another Instagram Engineering article, Static Analysis at Scale: An Instagram Story, Benjamin Woodruff describes the Instagram backend as follows:
Instagram Server is entirely Python powered.
Well, mostly. There’s also some Cython, and our dependencies include a fair amount of C++ code exposed to Python as C extensions.
Our server app is a monolith, one big codebase of several million lines and a few thousand Django endpoints , all loaded up and served together. A few services have been split out of the monolith, but we don’t have any plans to aggressively break it up.
Being such a large Python deployment, engineers at Instagram have worked extensively on performance and utilities around the basic features of the language.
Various other articles, listed here, discuss topics such as garbage collection optimization, static analysis and module management at scale.
Python at Robinhood
As of 2020 Robinhood.com counts more than 10 million users on its platform. Being a stock trading platform, real time stream data processing is the primary source of engineering challenges.
One of the applications of Python at Robinhood is stream analysis for use cases such as fraud detection, ad tracking, event logging and news aggregation. For this purpose, engineers at Robinhood have developed and open sourced Faust - a stream processing library for Python 3.
Going back to Apache Airflow, a project originating at Airbnb, in Why Robinhood uses Airflow Vineed Goel from the Robinhood engineering team compares Airflow to its alternatives. The primary use case considered is batch processing of big data. Dating back to 2017, the article lists frameworks less commonly used today. One of them, Luigi by Spotify, is also a well known Python project.
Further, in How We Built a Better News System, Arpan Shah outlines how at Robinhood they use Faust and Airflow together to deliver newsfeeds to their users.
If you're curious for more check out Python at Robinhood on Blogboard.
At blogboard.io we collect official engineering blogs and provide a search engine on top of thousands of articles. The collection spans several years and hundreds of companies.
In practice, this means a simple search term like "python" will give you stories about real-life use cases of Python at leading tech companies.
And yes, the backend for blogboard.io is written in Python (Flask).