Five Lessons from Kaggle's Event Recommendation Engine Challenge

Nothing could have prepared me for the pleasant surprise awaiting me last Thursday's morning. Just a few hours earlier I've submitted my final predictions on Kaggle's Event Recommendation Engine Challenge (in short - the goal is to predict which events will be interesting to a given user). The public leader board was frozen a week earlier, and at the time I was at the 11th place, after falling six places in a couple of days. Moreover, during the board-freeze session, I didn't find any feature or modeling technique that improved significantly my score on my own validation set. I was hoping to be in the top 10%, but woke up to find that I won the third place, reaching my best ranking ever.

I strive to learn from each competition I participate in, and this one is no different. However, my take-aways from this challenge don't involve a new algorithm or feature selection insights. Rather, they are lessons about handling data on Kaggle challenges and in real life :). But first, the boring stuff:

• I used pandas, which I really wanted to try out (great, but I'm missing some of R's data exploration methods), scikit-learn and iPython notebook (fantastic!).
• I've treated the challenge as a classification problem, and ranked the events by their predicted probability to be classified as interesting.
• My model is basically an average between a Random Forest and a Gradient Bosting Classifier, with a sprinkle of Naive Bayes on the events' descriptions.
• The top feature for me (as judged by the random forest), was the time between when the user was presented with the event and the event's date, or delta for short. Next comes the ratio of invited users that decided not to attend an event (surprisingly, the higher the ratio, the more interesting is the event),  the total number of attendees, and the estimated distance between the user and the event (more on this later).

Lesson One: Over Fitting is a Bitch

As you can see in the graph below, the public score was a very bad predictor for the private (and final) score.

Luckily, I've trusted more my five-fold validation averaged score than the scores I got on the public leader board. But boy, was that frustrating. A few times during the competition I've submitted my predictions after fixing some bug or improving the reliability of some feature just to get a lower public score. Even worst, in the last couple of days before the board freeze, I was unable to improve my public score and was watching new-comers getting far better scores than mine. I was exiled from the warm comfort of the fourth place to the 11th place. It wasn't easy to ignore the public score and not to optimize against it, but luckily I did so.

Lesson Two: Don't Believe Random Forest's Hype

It was my first time successfully employing a random forest classifier as the main predictive mode (usually, logistic regression works better for me), and I believed all the hype about random forests being better at avoiding over fitting. However, the telltale this isn't the case here was observing that adding features to the model sometimes decreased my validation score. 

I've combatted that behavior by limiting the trees in the random forest to a certain maximum depth and a minimum number of samples at each leaf, and averaging with the GB classifier (which I haven't tweaked).

Knowing what I know now, I would have also dropped some features that I've added to the model just because they seemed relevant and I believed Overkill Analytics' approach of adding as many features as possible to a random forest, and let it sort them out.

Lesson Three: There's Always Some Data Leakage

Midway through the competition I've discovered a feature with a very strong predictive power. Turns out that in many cases you could guess whether a user in the train set is interested in a specific event just by looking at the timestamp when he observed that event. If a given user observed several events at timestamp X, and another one at timestamp Y>X (even if those are just a few seconds apart), that other "later" event was probably marked as interesting. 

Obviously this was some sort of a bug in the train set, and once I've proved that it was exploitable by reaching the third spot, I've (foolishly, since I didn't get any "finders fee" :)) alerted Kaggle about it. Fortunately, I got back to third place without such dirty tricks.

However, in the way the train and test sets were assembled, there was some leakage that could (and should) be exploited. First, for each user we had a list of about six events, from which exactly one was marked as interesting. That is, we are given a lot of information in comparison of the classical classification problem. I've exploited that by having a feature that compared the delta of each event displayed to a user against the delta of the earliest event. This proved to have more predictive power than the number of friends a user had in the event.

Moreover, we can assume that startup behind the competition already propose only relevant events to its users. For example, users from Indonesia were rarely presented with events happening in the US. They probably do this by examining IP address which they chose not to share in the train set.

This led to my first attempt and still most successful at creating a "distance between user and event" feature; I simply calculated the median of the locations of the events presented to the user, and for each event calculated its distance from that median, as though the median was a substitute to the user's location. This worked better than deriving the user coordinates by looking at events occurring at the same city as the one found in the user's profile, and averaging between them.

Lesson Four: Always (Always!) Use Random Seeds

Some of my features are derived from graphs, such that a pagerank score for events according to their attendance graph (where events share an edge with a weight that depends on the Jaccard similarity between the users that attended them), or, like many other did, events and users clusters. Sadly, because my hardware isn't powerful enough, I had to prune the graphs and I did so by deleting edges at random.

Big mistake. Now I cannot recreate my final submission, and after all the work I put into it, I may not see the prize money. I was smart enough to set a random seed for my random forest, but too lazy or stupid to do that when creating the graph. Gah!

Lesson Five: Things that didn't Work

I've tried a lot of things during this competition, most of them didn't work, or at least I couldn't prove them to be working. However, they are good things to consider in future competitions:

• I've tried using ELO rating as an extra event-rating technique over the classification ones. It showed some (a lot!) of promise in the public test set, but failed in my 5 fold validation sets.
• I've tried calculating the "distance" between a user and an event by considering a graph with edges between users according to their friendship status, and between users and events according to their attendance. This was a complete failure.
• Naive Bayes (or more precisely, Multinomial Naive Bayes) on the event's descriptions was very disappointing. I then splited the users according to geographies and had a different NB classifier for each, which made this technique only slightly disappointing. 
• Page-ranks of events (and even more so, page-ranks of the events organizers) didn't contribute much.
• I've tried filling up missing users and events locations by using an iterative approach; I guessed the location of each user by averaging the coordinates of the events  she attended, and the location of each event by averaging the coordinates of the users who attended it. I did this repeatedly until I failed to discover the locations of any new events or users. This actually made my performance worse.

Epilogue

Wow, did you really read this very long blog post? If so, you may be interested in following me on Twitter. Hope you enjoyed it, and feel free to comment, especially if you notice any grammatical or spelling mistakes :).