I am delighted to have with me here today, Kathy McKeown, who is the Henry and Gertrude Rothschild Professor of Computer Science at Columbia University. Where she is also the founding director of the Institute for Data Sciences and Engineering. She is also an Amazon Scholar and is well known for the work that she's done over many years on text summarization and many other topics in NLP. Welcome, Kathy, and thanks for joining me. >> Thanks Andrew, and thanks for having me. >> So today you lead a large group doing NLP research, but your journey to becoming an AI researcher and an NLP searcher has been an unusual one. If I remember correctly, you actually majored in comparative literature when you were in undergrad, even though you're also very mathematically oriented at the time. So tell us you story of how you became an NLP researcher. >> Yeah, so when I started out at Brown I didn't know what I wanted to major in. So I took courses both in math and comparative literature. And as I went on, I became more interested in comparative literature. Probably in part because of the teachers who I had who really influenced me. It was only as I came near the end of my time at Brown that and when I graduated, I took a job as a programmer, which I found actually very boring. And I thought if I was going to have to be working 40 hours a week, I wanted to be doing something that I enjoyed. And it was then that a friend of mine who was a linguistics major at Brown told me about computational linguistics. And so I spent a lot of the year in the library reading about AI and natural language processing. And when I applied for Graduate School the following year I I knew that was what I wanted to do. Because it gave me a way to bring together my interest in language and in math. >> So that's fascinating, so as a comparative literature major, you spent a lot of time in the Brown University Library reading about computational linguistics and NLP. Today, with a lot of learners, maybe some watching this video that may not yet be a NLP researcher or AI engineer wanting to break into the field. So I'd love to hear more about what your experience was like reading so much. Were you doing it by yourself? Did you have a study group or what was that like? >> I was doing it entirely by myself. I really had no guidance in terms of what to look at. I guess this friend that I had made a few suggestions. And then I traced references when I first began reading. I would follow up on references to go further. Yeah, and when I first entered Graduate School, and I had essentially switched fields, I found it very frightening. I was sure that I was and impostor, that I didn't know enough. And before long they would find out that I really shouldn't be there after all. Yeah, but that's something you overcome with time and you learn that it's not the case and people value your input. >> That's really inspiring. Thank you for sharing that. Would you have any advice for someone maybe today that is trying to do this themselves and wondering if they know enough, or are good enough, or should be in the field? It sounds like you got through that and you've been incredibly successful. But what would you say to someone today maybe looking to follow in your footsteps and wondering if this is very for them or if they'll make it? >> So I I guess I have a couple pieces of advice. I do think reaching out to people and talking to people is useful. Until I got to Graduate School I wasn't in an environment where I had people to talk to. So I do think it's really helpful, especially to talk to your peers about what they're doing and what they're interested in. When you pick problems to work on. And I guess, especially in today's world of deep learning and neural nets, I would advise choosing problems that are different from what everybody else works on. Sort of strike out in a different direction, choose something new, a new task, and take off from there. >> And I think I would love to come back to the different problems stopping within. And when I speak with learners from around the world, I do hear from some that they feel lonely or isolated. They're kind of out somewhere or maybe not living in the major tech hub, and they sometimes feel like they're doing this by themselves. So I find it actually really inspiring that you were able to do that by yourself in a library in Brown University. I don't know if you have any other thoughts to offer learners that may feel like they're somewhere in a company or in a city just trying to do this by themselves. >> I'm not sure I do have a lot more to say about it. I guess read what you enjoy about, if you can be part of a reading group, an online reading group, that would be helpful. There are a lot of reading groups now, and that's a good way to get sort of insights. There are online videos and course experiences, like yours. And I think that's a way to find out what's going on and get in touch with what people are doing. So I think today, the online environment can help people get connected and hear what's going on. I was lucky, I mean, I was really lucky, [LAUGH] I applied to Penn. I didn't know that at the time it was the best place in natural language processing, and that was totally luck. So I don't know that I would recommend doing it blind again today, I think getting advice is great. >> Everyone that's been successful has had many elements of luck. But the preparation makes you ready to take advantage of when good luck falls into your lap. >> Yeah. >> Thank you for sharing that, that was really inspiring to hear about your early days as a developing researcher. And today you lead a large group at Columbia University doing very interdisciplinary work and doing a lot of work on summarization and other topics. So tell us a bit more about your current work and which excites [INAUDIBLE]. >> So I mean, summarization has really been the bulk of my work over the most recent years. We've done work on summarization of all kinds of different genres, from personal novels to emails. One thread of research in summarization that I'm particularly excited about is work that I've done with researchers at Amazon and which was published at ACL. And this is work on summarization of novel chapters. And it's very new, no one has been working on this task, it's very challenging, very different from summarization. So the chapters are much longer than the news articles on which most current work in summarization today is done. And that is a challenge for current neural models. And a big problem is that there is an extreme amount of paraphrasing between the input, which is 19th century novels, and the output, which is a summary written in today's language. None of the current models can handle the kind of paraphrasing that we see there. And that, in general, is a topic that I'm really interested in is a sort of very abstractive summarization. Where the sentences use different words than the input document, where the syntactic structure is different. And that is very different from the vast Majority of work today which is done on summarization of news. And it's done on summarization of news because that's where the data is. So some of the other areas that I'm looking at are summarization of personal narratives that you find online where the personal narratives are very informal language and the summary is more formal. Summarization of debates. In past work, we've also done summarization of email. Which has some of those same characteristics. >> Why did you choose to work on the novel summarization task? >> Well, so we had done work on novels even earlier. I would say in 2010, when one of my students who was very interested in creative writing, and I really thought he should do a PhD. And so to convince him to stay, we came to a topic that he would be happy with, which was analysis and generation of creative language. And I felt then that my work came full circle, that we collaborated then with a professor in comparative literature. So I acme back to my roots in comparative literature and that was a lot of fun. So, when I first went to Amazon, I knew because of Kindle and online on Amazon, they have a lot of novels. And I thought, what would be more fun than being able to summarize novels? >> Sounds like a fun project, I read a lot on my Kindle, so maybe your work will be a feature on Kindle some day. One aspect of your work that stood out as well is that you're known for doing highly interdisciplinary work. So rather than focusing narrowly on NLP research, your work spans AI, and the media where I know the Columbia University is a great journalism school. So wonderful journalists you work with there or the application of NLP to social networks. I think you work with medical problems. So tell us a bit about how you think about interdisciplinary work because you've done more of it I think than most NLP researchers have. >> Yeah, I really enjoy interdisciplinary work. I think it's my most favorite kind of research to do. And in part we get a really different perspective on research in the world when we talk to people in other fields. It takes us out of our sort of technical, narrow field. >> And so earlier, you alluded to picking research topics that are novel and I think your research portfolio has certainly touched on a lot of problems that very few others are working on. So can you say more about that? So how do you pick research projects you work on and how do you advise others to pick topics to work on? >> I think it's important to pick a task that matters. So that for me is one thing to look at. For example, most of the work in tech summarization today is done on summarization of what's called single document summarization of news. So take one news article in and generate a summary of that news article. And the reason for that is because that's where the data is. There's a huge amount of data that has been pulled together from first the CNN Daily Mail Corpus, and later New York Times, and there are a number of other corpora as well. The problem is that's not really a task that we need. We've known for a long time that the lead of the news article can help people pretty well in serving as a summary of the document. And in fact, for years it was hard to beat. The lead people just worked on, that wasn't a problem that people worked on in the early years of summarization. >> The lead being the first sentence or the first- >> The first couple of sentences in the news article. So, Yeah, I mean, people work on a problem like that because that's where the data is. We have leaderboards. People are competitive. They like to be The leaderboard, but I would question does that one or even half a point in Rouge, which the automated network used to score them, really make a difference? If you look at the output you can see that actually the summaries are quite similar and either one of them might be fine. So I prefer to go in directions that people haven't gone in before and to choose a task where if you solve it, it's going to help people. It's going to be a useful application that you've developed. So this is why I have done things like summarization of personal narrative, which we did in the course of disaster. So that we could summarize, think of having a browsing view of summaries of what people have experienced after they've lived through a disaster. Or the current work on summarization of novels where it'd be helpful to have a summary of an input chapter. I'd like to go in a different direction, in part because I want to solve the task that matters. But I also like to go in a different direction because in this day in age of deep learning where results come so fast, everybody works on the same problem trying to beat the previous state of the art. It can be hard to be the first one to get there, and if you go in a different direction, nobody else is working on, you are going to be the first one to come to solution. And that's what I like to do in my research. Overtime I like to be first on a problem. I see, cool yeah. And I feel like, for myself, I have a lot of respect for people that could push that extra half points of performance on the leaderboard because hopefully that advances the whole field. And this all shifts. I also have a lot of respect for people with your creativity in the inside, to charter the new problem that no one else has thought of, and advances the whole field in a different direction. I think the field of AI NLP is broad enough. I think it's actually not a bad thing if we have lots of people working on lots of different things, including standardized benchmarks. And a bunch of new things. >> Yeah, sure, I feel that there are not as many people who want to go in that new direction and it does take some sort of guts to do it. Because the first thing that happens when you submit a paper is there is no benchmark. There is no baseline of prior work and reviewers have a very hard time dealing with that. How can they judge whether it's really a good step forward. Whereas if you can show on a leaderboard that you've improved by a certain amount and you stay within the traditional trajectory, it's easier to judge. >> Yeah, I'm with you on that. Actually, I was recently chatting with one of my friends Sharon Joe who mentioned that sometimes the way benchmarks and metrics are established is that some researcher publishing a paper publishes something using some metric. Maybe a good one, maybe an okay one. But to make sure that subsequent papers can compare to earlier work, then everyone, and more and more people end up using the same metric. More for historical reasons and it makes things comparable rather than because it is actually the most useful metric. It's funny how metrics get established in academia. >> Yeah, I mean that has happened in the summarization field. And I think also in machine translation where you want an automated metric because it's easier to develop a system to train over and over again. And yet everybody knows that the automated metrics that we currently have are really flawed. And but everyone keeps using them, because that's what we've always done. One of my most stark memories was I remember going to a the Information Retrieval Conference and attending a workshop in text summarization. And I remember being fascinated, but struck that about half of that workshop was on text summarization algorithms and the other half of that workshop was on how to develop metrics to evaluate. Text summarization especially, the development of automated metrics has been challenging. >> Yeah. >> Hey, so in terms of choosing. Lead topics to work on. One of the pieces of work that you've been doing that I thought was fascinating, was you were taking texts from the black community from Harlem, near I guess where you teach at Columbia University and analyzing that as well. Tell us about that. >> This is where I'm moving with my work with a researcher from social work. And we're also beginning to involve a linguist who works on African American vernacular. And what we're doing is we're looking at what people say, what kind of emotions they express in reaction to major events that are going on today. So, for example, in reaction to Black Lives Matter and in reaction to COVID-19. So this is work that we're just beginning. We've begun with developing an interface where people can post about their experiences with these events and how they're feeling. And I guess what we're hoping to do with that in part, so we have two directions to go. One on the natural language side is to be able to understand how people express different kinds of language. Sorry, different kinds of emotion and African American vernacular. And how that difference from how people express it in standard American English. And look at the difference in language and probably even the difference in content in terms of what's expressed. And this can help us in developing algorithms that are not biased as we move forward. Most of the work in natural language, all of the systems have been trained on language that comes from news like the Wall Street Journal. >> Yeah, that's great. If this type of work can help fight bias or build bridges between communities or just play some role in understanding and helping to advance the Black Lives Matter movement, that seems to be a wonderful thing to me. >> Yeah, I mean, we also want in that work to look at the impact of trauma. So it's a different kind of trauma and sometimes it's not your personal trauma, but the trauma of seeing what has happened to other people who are like you. So yeah, we want to look at how that is expressed in the different kinds of emotions, the intensity of emotion, and so forth. >> I find it really wonderful that NLP researchers, AI researchers can play an active role in some of these most important societal questions and issues of our time. It feels like the work we do as AI researchers, it could matter in these really important times. >> Yeah, I mean I think so. And I've sort of been trying to do that for a while. I think it really attract students to work with you, and often different kinds of students into the field to work with you. On our first work on with analyzing social media posts of gang involved youth, we didn't have funding. We did that entirely with undergraduates who were just totally amazing. In earlier work we were looking at being able to automatically generate updates about disaster as it unfolded. We did that after Hurricane Sandy hit New York. And again it was something that students came to me and they had seen this happen and they have seen their neighborhoods hurt. Or they lived through the uncertainty of it and they wanted to help. They wanted to know what can we do and that was at that point in time, this whistle pre-neural net we began developing systems that could automatically generate updates as an event unfolded. >> I think that you don't need a PhD, don't need a long publication record, but in undergrad spotting an opportunity with a desire to help. Can step in and start to work on systems that they can make a difference. >> Yes, they're really passionate about it and There are really good, the work that came out of that was really excellent. >> Yeah, thank you, so Kathy, this is great stuff. And switching tracks a bit, you've been working in NLP and associated areas for a long time. In fact, I saw that even way back in 1985 you written an early book on text generation before the modern neural text generation techniques were around. So you've been a leader in the field for a long time and seen a lot of things change. I'd love to hear your thoughts on how the field of NLP has evolved over these many years. >> Sure, so when I started, which I got my PhD in 82. so I spent those earlier years at Penn. And there were some characteristics of the field that were salient. So one of them is that there was a lot of interdisciplinary work. There was in developing NLP systems, we drew a lot on work from linguistics, from philosophy, from psychology, from cognitive science. And so when I was at Penn I interacted a lot with faculty from linguistics. Ellen Prince was one of the people or faculty from philosophy. We spent time in these interdisciplinary meetings. And I can remember walking across campus with my advisor to go from the computer science department to the psychology department, for example. I was influenced a lot and I have to mention this, although it's not exactly what you asked, I was influenced a lot by senior women at the time. If I look back to who was most influential in how I progressed in my early research. My advisor, of course, who was a male Aravan Joshi. But then also Bonnie Weber, who was there in computer science, Eva Hychova from Charles University at Prague who was a linguist. Barbara Gross who lives at Sanford at that time in the CFLI Institute. And Karen Spark Jones was very influential to me. She was from the field of information retrieval. And she and I spent a lot of time talking about summarization. So interdisciplinary is one main feature of that time. A second was drawing on theories from these other areas, so we drew on theories from linguistics. One main kind of theory that we looked at was the focus of attention and how that changed over the course of a discourse. And how that influenced how you made choices and how you realized text in language. So for example, did you use a pronoun or did you use a full noun phrase? What kind of syntactic structure did you use? You might use different syntactic structures to make a concept more prominent in the discourse. We also drew on work from philosophy. So we drew on work from theories from Cyril about intention, and work from Grice about conversational implicature. And so we looked at these theories and we looked at how we could embody them in our natural language approaches. >> It's great to hear about some of your early sources of inspiration. Much as I think today you will be a source of inspiration to many others. So you've seen a lot and see a lot in NLP, which continues to be a rapidly evolving field. So I'm actually curious, Kathy, what do you find most exciting in terms of emerging or exciting NLP technologies? >> For me, personally, some of the work that I've already talked about today on truly abstracted summarization that uses extreme paraphrasing. Work on analyzing the language from diverse community. So we've been looking at the black community, but I think there are other communities we could look at as well. I'm interested in looking at how you deal with bias and data. And another very important topic is being able to arrive at what I would call para linguistic meaning. So this pragmatics information about emotion, about intention, would be another important direction to go. And I also think more work on events, being able to understand what events have happened, and to be able to follow them. I also think about often if I look back. Is it okay, if I talk about this now. I look back? >> Okay. If I look back on my favorite technologies and papers, I can think of papers from thee points in time. The first would be older, and this was very early work in language generation, on how we pick the words in our sentence. And we thought then that it was a hard problem that constraints came from many different sources. And we wrote a paper called Floating Constraints on Lexical Choice. Where we looked at how information from different parts of language, from the discourse, from the lexicon, from syntax, from semantics, will influence what we chose. And we worked in two different domains. One was basketball and one with stock markets. And I give it the example of the floating constraint, where we want to express both the time at which something happened and the manner. In the first example we expressed time and the verb, and the manner and the adverbs. So Wall Street indexes open strongly, open is the time. And in the second weeks press manner in the verb and time and the propositional phrase. So stock index surged at the start of the trading day. And so we wanted to look at how we could control that choice. And I think control is something that's missing in language generation and summarization. Today using deep learning methods, how do we control what the output is and make sure it's true to what our intention is? In more recent work, my favorite is work on News Blaster. That's still about 15 years ago, but it feels recent to me. And that was where we took a real world problem. We did do some collaboration with journalists, and we've developed a testbed, where we could identify the events that happened during the day, and produce summary on each event. And then we also looked at how we could track that overtime. And this platform gave us a common sort of application, in which my students could address really hard research questions. And so that was where we looked at, did some of our first work on abstract and summarization. Looking at how we might compress sentences, how we could fuse phrases together, how we might reference, edit references, so that the summary was more coherent. And we also did work on multi lingual summarization. Yeah. >> Cool, Thank you. Lots of exciting, very distinct projects over the years. Do you have any wrap up? Did you have any lost thoughts? You can just say, do you have any final thoughts? >> Well, I guess I would just say that natural language is a really exciting field today. There's been a huge amount of progress with deep learning. We've seen dramatic increases in accuracy, but we still have a lot of directions to go. And I guess I would like to see more of the interdisciplinary work being brought back in. I'd like to see people looking at the data more and at their output more, rather than just numbers. But I think there are many exciting directions for people to work in, and I hope we'll see many people joining the field. >> Thank you, that was great. Yeah, I saw the hope that will have a lot more people join NLP and contribute to all of this exciting work. So thanks Kathy, it was cool >> Thanks you much for asking me, it was fun. >> For more interviews with NLP thought leaders, check out the DeepLearning.AI YouTube channel, or enroll in the NLP specialization on Coursera.
