How to Detect Anomalies in High Cardinality Dimensions and Make Them Actionable

Shankar Vedaraman, Manager, Data Science & Engineering
Chris Colburn, Senior Analytics Engineer
Netflix

Agenda 

*** Bring ID (e.g. Driver’s License) for eBay Security ***  
6:30 Doors Open, Food & Networking  
7:00 Presentation  
*** Please arrive by 7 PM due to Security ***

Event Details

Anomaly detection is the process of identifying data points that do not conform to normal behavior, and it is used ubiquitously at Netflix.  For example, real-time systems detect, and raise, outliers when internal systems do not meet some service level agreement.  In data warehousing applications, traditional outlier detection methods (e.g. some number of standard deviations) will work for low cardinality dimensions that are normally distributed, but typically dimensions of interest are neither normally distributed nor have low cardinality.  In these settings the number of false positives/negatives create an unnecessary overhead and limit the end-user’s ability to respond.

In this lecture, we present a case study at Netflix where we deployed a variant of the Singular Value Decomposition for anomaly detection in high cardinality dimensions.  We then wrapped this in a Business Intelligence tool to present actionable insights for business use.

We will then discuss a specific application centered on payment processing.  With more than 50 Million customers worldwide, Netflix has to ensure that the payment methods provided by customers do not fail due to processing problems in the payment network. A typical payment transaction goes through at least 4 external participants (issuers, acquirers, payment gateways, processors, etc…) in addition to Netflix’s systems.  The wide array of banks that customers use to pay for Netflix creates this high cardinality dimension, and the complexity of the payment transaction necessitates the need for a different solution than the common methods mentioned above.  We will also present the decoupled architecture in the cloud that enables us to provide a highly performing, scalable solution.

Speakers Bios

Shankar Vedaraman

Shankar leads the Payment Analytics Data science and Engineering team at Netflix. His team is responsible for providing analytical solutions for Payments, Fraud and Retail gift analytics. The solutions include data engineering, BI engineering and analytical story telling. Shankar is highly passionate about creating data products that utilize the power of data science for better business benefits. 

Prior to Netflix, Shankar was to Rovi Corporation where he led a Business Intelligence team to provide analytical solutions for various business units. Shankar has a Masters in Computer Science and is currently pursuing a Masters in Predictive Analytics program from Northwestern university to master the science behind data science. 

Chris Colburn

As part of the Product Analytics team at Netflix, Chris works to provide his business partners with the data, and insights, that they need to make good business decisions.  Sometimes this can be done with a simple SQL query, sometimes this means writing custom code for big data, and sometimes utilizing data science techniques. Ultimately Chris hopes to create reusable tools that will enable other engineers/analysts at Netflix to do their jobs better (with the belief that this will yield a superior product).

Prior to Netflix, Chris was with Teradata-Aster, where he supported pre-sales and professional service engagement help customers resolve their business problems using data science. Chris has a PhD in Engineering in with five years of research experience and two years of predictive modeling consulting experience

Event page provided by ACM
https://www.sfbayacm.org/event/anomaly-detection-payment-processing-netflix