When starting work on a machine learning project, one of the most useful first step to take is to establish a baseline and is usually only after you've established a baseline level of performance that you can then have tools to efficiently improve on that baseline level. Let's dive into some best practices for quickly establishing that base. Let me use the speech recognition example. Let's say you've established that there are four major categories of speech in your data. Clear speech, which is when someone speaks without much background noise. Speech with car noise in the background as if they were in a car when they use your speech recognition system. Speech with people noise in the background so that they're outdoors with other people's out in the background or speech on a low bandwidth connection, what it sounds like if you're using a cell phone with a very bad cell phone connection. If your accuracy on these four categories of speeches, 94, 89, 87, and 70 percent accuracy, you might be tempted to say, well, it does worse on low bandwidth audio, so let's focus our attention on that. But before leaping to that conclusion, it'd be useful to establish a baseline level of performance on all four of these categories. You can do that by asking some human transcriptionists to label your data and measuring their accuracy. What is human level performance on these four categories of speech? In this example, we find that if we can improve our performance on clear speech up to human level performance, looks like there's a potential for a one percent improvement there. If we can raise our performance up to human level performance on audio of car noise in the background, maybe four percent improvement, two percent improvement and slightly zero percent improvement on low bandwidth audio. Whereas we had previously said without the human level of performance, we may have thought working on low bandwidth audio was most promising. With this analysis, we realized that maybe the low bandwidth audio was so garbled. Even people, humans can't recognize what was said and it may not be that fruitful to work on that. Instead, it may be more fruitful to focus our attention on improving speech recognition with car noise in the background. In this example, using human level performance, which are sometimes abbreviated to HLP, Human Level Performance, gives you a point of comparison or a baseline that helps you decide where to focus your efforts on car noise data rather than on low bandwidth data. It turns out the best practices for establishing a baseline are quite different, depending on whether you're working on unstructured or structured data. Unstructured data refers to data sets like images, maybe pictures of cats or audio, like our speech recognition example or natural language, like text from restaurant reviews. Unstructured data tends to be data that humans are very good at interpreting. In fact, humans evolve to be very good at understanding images and audio and maybe language as well. Because humans are so good at unstructured data tasks, measuring human level performance or HLP, is often a good way to establish a baseline if you are working on unstructured data. In contrast, structured data are the giant databases or the giant Excel spreadsheets you might have, such as if you run an eCom website, the data showing which user purchased at what time and for what price, that will be stored in a giant database. This type of data stored in a giant Excel spreadsheet or some more robust database would be an example of structured data or your product and inventory data that would also be stored as structured data. Because humans are not as good at looking at data like this to make predictions. We certainly didn't evolve to look at giant spreadsheets. Human level performance is usually a less useful baseline for structured data applications. I find that machine learning developments best practice is quite different, depending on whether you're working on an unstructured data or structured data problem. Keeping in mind this difference, let's take a look at some ways to establish baselines for both of these types of problems. We've already talked about human level performance as a baseline, particularly for unstructured data problems. Another way to establish a baseline is to do a literature search for state-of-the-art or look at open source results to see what others reports they are able to accomplish on this type of problem. For example, if you're building a speech recognition system and others report a certain level of accuracy on data that's similar to yours, then that may give you a starting point. Using open-source, you may also consider coming out with a quick-and-dirty implementation. Now, this is going to the system, but just a quick-and-dirty implementation that could start to give you a sense of what may be possible. Finally, if you already have a machine learning system running for your application, then the performance of your previous system, performance of your older system can also help you establish a baseline that you can then aspire to improve on. What a baseline system or a baseline level of performance does is it helps to indicate what might be possible. In some cases, such as if you're using human level performance, especially on unstructured data problems, this baseline can also give you a sense of what is the irreducible error or what is Bayes error. In other words, what is the best that anyone could possibly hope for in terms of performance on this problem, such as helping us realize that maybe the low bandwidth audio is so bad that is just not possible to have more than 70 percent accuracy, as in our earlier example. By helping us to get a very rough sense of what might be possible, it can help us be much more efficient in terms of prioritizing what to work on. Sometimes I've seen some business teams push a machine learning team to guarantee that the learning algorithm will be 80 percent accurate or 90 percent or 99 percent accurate before the machine learning team has even had a chance to establish a rough baseline. This, unfortunately, puts the machine learning team in a very difficult position. If you are in that position, I would urge you to consider pushing back and asking for time to establish a rough baseline level of performance before giving a more firm prediction about how accurate the machine learning system can eventually get to be. It helps you to make your case, feel free to tell them that I asked you to do so. I think establishing that baseline first will help set you and your team up better for long-term success. Now to tell us about the importance of baseline, there are few additional tips I want to share with you about how to get started quickly on the machine learning project. Let's go on to the next video to take a look at some of these tips.
