For many structured data problems. It turns out that creating brand new training examples is difficult, but there's something else you could do which is to take existing training examples and figure out if there are additional useful features you can add to it. Let's take a look at an example. Let me use an example of restaurant recommendations where if you're running an app that has to recommend restaurants to users that may be interested in checking out certain restaurants. One way to do this would be to have a set of features for each user of each person, and a set of features for each restaurant that then get fed into some learning algorithms, say a neural network and then your network, whose job it is to predict whether or not this is a good recommendation, whether to recommend this restaurant to that person. In this particular example, which is a real example, error analysis showed that the system was unfortunately frequently recommending to vegetarians restaurants that only had meat options. There were users, they were pretty clearly vegetarian based on what they had ordered before and the system was still sending to them maybe a hot new restaurant that they recommended because there's a hot new restaurant, but it didn't have good vegetarian options. So this wasn't a good experience for anyone and there was a strong desire to change this. Now, I didn't know how to synthesize new examples of users or new examples of restaurants because this application had a fixed pool of users and there are only so many restaurants. So rather than trying to use data augmentation to create brand new people or restaurants to feed to the training set, I thought it was more fruitful to see if there were features to add to either the person inputs or to the restaurant inputs. Specifically one feature you can consider adding is a feature that indicates whether this person appears to be vegetarian. And this doesn't need to be a binary value feature 0 1. It could be soft features such as a percentage of fruit order that was vegetarian or some other measures of how likely they seem to be vegetarian. And a feature to add on the restaurant side would be. Does this restaurant have vegetarian options or good vegetarian options based on the menu. For structure data problems, usually you have a fixed set of users or a fixed set of restaurants or fixed set of products, making it hard to use data augmentation or collect new data from new users that you don't have yet on restaurants that may or may not exist. Instead, adding features, can be a more fruitful way to improve the performance of the algorithm to fix problems like this one, identify through error analysis. Additional features like these, can be hand coded or they could in turn be generated by some learning algorithm, such as having a learning average home, try to read the menu and classify meals as vegetarian or not, or having people code this manually could also work depending on your application. Some other food delivery examples, we found that there were some users that would only ever order a tea and coffee and some users are only ever order pizza. So if the product team wants to improve the experience of these users, a machine learning team might ask what are the additional features we can add to detect who are the people that only order tea or coffee or who are the people that only ever or the pizza and enrich the user features. So as the hope the learning algorithm make better recommendations for restaurants that these users may be interested in. Over the last several years, there's been a trend in product recommendations of a shift from collaborative filtering approaches to what content based filtering approaches. Collaborative filtering approaches is loosely an approach that looks at the user, tries to figure out who is similar to that user and then recommends things to you that people like you also liked. In contrast, a content based filtering approach will tend to look at you as a person and look at the description of the restaurant or look at the menu of the restaurants and look at other information about the restaurant, to see if that restaurant is a good match for you or not. The advantage of content based filtering is that even if there's a new restaurant or a new product that hardly anyone else has liked by actually looking at the description of the restaurant, rather than just looking at who else like the restaurants, you can more quickly make good recommendations. This is sometimes also called the Cold Start Problem. How do you recommend a brand new product that almost no one else has purchased or like or dislike so far? And one of the ways to do that is to make sure that you capture good features for the things that you might want to recommend. Unlike collaborative filtering, which requires a bunch of people to look at the product and decide if they like it or not, before it can decide whether a new user should be recommended the same product. So data iteration for structured data problems may look like this. You start out with some model, train the model and then carry out error analysis. Error analysis can be harder on structured data problems if there is no good baseline such as human level performance to compare to, and human level performance is hard for structured data because it's really difficult for people to recommend good restaurants even to each other. But I found that error analysis can discover ideas for improvement, so can user feedback and so can benchmarking to competitors. But through these methods, if you can identify a academy or a certain type of tag associated your data that you want to drive improvement, then you may be able to go back to select some features to add, such as features to figure out who's vegetarian and what restaurants have good vegetarian options that would help you to improve your model. And because the specific application may have only a finite list of users and restaurants, the users and restaurants you have maybe all the data you have, which is why adding features to the examples. You have maybe a more fruitful approach compared to trying to come up with new users or new restaurants. And of course I think features are a form of data to which is why this form of data iteration where error analysis helps you decide how to modify the features. That can be an efficient way as well of improving your learning algorithm's performance. I know that many years ago before the rise of deep Learning, part of the hope for deep learning was that you don't have to hand design features anymore. I think that has for the most part come true for unstructured data problems. So I used to hand design features for images. I just don't do that anymore. Let the learning I won't figure it out. But even with the rise of modern deep learning, if your dataset size isn't massive, there is still designing of features driven by error analysis that can be useful for many applications today. The larger data set, the more likely it is that a pure end-to-end deep learning algorithm can work. But for anyone other than the largest tech companies and sometimes even them for some applications, designing features, especially for structured data problems can still be a very important driver of performance improvements. Maybe just don't do that for unstructured data nearly as much because learning algorithms are very good at learning features automatically for images, audio and for text maybe. But for structured data, it's okay to go in and work on the features.
