How I Used Deep Learning to Create New Yorker Captions

No computer has ever won the New Yorker Cartoon Caption Contest, and for good reason: it’s hard. The New Yorker magazine runs a weekly contest where readers submit captions for an un-captioned cartoon like the one below. The winning caption gets published in the magazine.

Example image from a previous New Yorker Caption Contest. Image from NextML. CC BY 4.0.

People have created New Yorker caption generators, but none actually use the image of the cartoon to generate the caption. Instead, they use previous captions to predict what a new one would be like [1].

Going directly from the cartoon image to a caption is hard for a few reasons:

  • Cartoons usually depict situations and…


Hands-on Tutorials

A Pytorch tutorial

Photo by Joshua Earle on Unsplash

Autoencoders are unsupervised neural networks used for representation learning. They create a low-dimensional representation of the original input data. The learned low-dimensional representation is then used as input to downstream models. While autoencoders are effective, training autoencoders is hard. They often get stuck in local minima and produce representations that are not very useful.


This article was originally written February 28, 2017.

Let’s say you want to teach a computer to read handwritten digits. You might give it a bunch of rules to tell it what to do. For example, an oval is most likely a 0. Another approach you might try is “machine learning.” Give a computer a bunch of examples of each digit to study so that it can learn its own rules. This latter method has worked surprisingly well. …


Originally Written for Princeton Innovation Science Magazine in Spring 2015

Photo by Luke van Zyl on Unsplash

It’s a cold winter day; you’re on your way to class when all of a sudden you get a text. You take out your phone, but with your gloves on, your phone’s touchscreen won’t react. You begrudgingly take off your glove to reply to your friend, wishing that you had bought one of those fancy “touchscreen-friendly” gloves instead. The question then arises: What makes “touchscreen-friendly” gloves different from your “normal” ones? Why do some screens need special gloves when others don’t?

To answer these questions, we must look at a…


Photo by Isaac Smith on Unsplash

Over the past two months I worked on my first Kaggle competition. Kaggle is a platform where data scientists compete to come up with the best algorithms to solve real world problems. The goal of this particular competition (M5 Forecasting) was to predict June 2016 Walmart sales based on the past five years of data (January 2011-May 2016). To make it more challenging, we needed to predict the sales for each item in each store.¹ There were 5,000 teams participating in the competition for $50,000 in rewards for the top 5. I finished at #7.

Here are some things I…

Eugene Tang

Data Scientist and Software Engineer. Interested in seeing how technology and data science can help improve the world.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store