J.P. Morgan

Distinguished Lecture

Series on AI

J.P. Morgan Distinguished Lecture Series on AI

[00:00:00.04] MANUELA VELOSO: Well, I'm sorry for the delay. We had some technical complications here. But I am very, very pleased to share with you that we are going to have Professor Roni Rosenfeld present us on-- he's going to share his screen, and is going to share his talk on really forecasting epidemics and pandemics. So let me just tell you about Professor Rosenfeld.

[00:00:26.28] He has studied math and physics at Tel Aviv University. And then he did a PhD and a master's in computer science at Carnegie Mellon University. I had the pleasure and honor of being his colleague in our PhD at Carnegie Mellon. And currently, Professor Rosenfeld is the head of the machine learning department at Carnegie Mellon.

[00:00:51.95] So after he finished his PhD, he became faculty, like I did. So his path and mine are very similar. The emphasis of Professor Rosenfeld's research has always been in machine learning, and also, in particular, also in speech and language technologies. And he has been part of the machine learning department and the Language Technology Institute for many years, probably more than 20 years.

[00:01:22.59] I'm very pleased to tell you that Professor Rosenfeld is going to share with us very compelling research on this problem of using machine learning on tracking and forecasting epidemics. And he has been doing this in several contexts before the COVID. But now, the interesting results that he's also going to share with us are the results that he got with this particular data and this particular crisis we are facing now. So it's a very, very timely conversation with Professor Rosenfeld.

[00:01:58.86] Just to finish, he, of course, has many awards. And please, look at his vitae that we shared. But he's a recipient of a very dear to our heart award, which is the Allen Newell Medal for Research Excellence, given by Carnegie Mellon in memory of Professor Allen Newell, who we both knew very well. And he's one of the founders of the field of AI.

[00:02:22.35] So Roni, it's a pleasure to have you with us. And please, share your thoughts with the group. And we all look very much forward to do this. So let's plan on a talk of the time you want, and then we have some time, until 11:30 AM, for asking questions to Professor Rosenfeld. Thanks a lot, Roni, and welcome to JPMorgan.

[00:02:50.47] MANUELA VELOSO: Yes, we are perfect.

[00:02:51.51] RONI ROSENFELD: OK. Thank you for this very, very generous introduction. I think my most noteworthy award is being an FOM, a friend of Manuela.

[00:03:04.10] [LAUGHTER]

[00:03:05.23] I don't think you shared that we arrived at Carnegie Mellon in the same year and have been through the PhD program together. So that's a really bonding experience.

[00:03:16.12] MANUELA VELOSO: [INAUDIBLE] do in slideshow?

[00:03:20.58] RONI ROSENFELD: I can.

[00:03:21.69] MANUELA VELOSO: Yes, I think it's better to-- there we go. Oops.

[00:03:27.30] MANUELA VELOSO: Yeah, there we go. Thank you.

[00:03:28.41] RONI ROSENFELD: That good?


[00:03:29.01] RONI ROSENFELD: OK. All right, so the title of my talk tries to convey a story. And the story is that we have been forecasting epidemics for eight years, but have been forecasting pandemics for only a few months now, doing it in a hurry and in a scramble. And I'll explain that a little more later, but this is why the pandemic part is sort of hastily added, for obvious reasons.

[00:03:59.19] For the last eight years, since 2012, we had a research group called the Delphi Research Group that my colleague, Ryan [INAUDIBLE], and I co-founded. And it was a typical academic research group. You know, a few PhD students, maybe a few masters students, vocational, undergrad. And we were working on forecasting flu, dengue, a few other diseases. And then I used to say that I'm not going to get involved in forecasting a pandemic because I don't think it can be done. And then the pandemic happened, and I was kind of forced to change my mind to find ways to do it anyway. So I will tell you the story in more details.

[00:04:42.35] The people you see in the line in the middle are the original members of the Delphi Research Group, as of January. And the people you see at the bottom are all the people who joined us since then to scramble to do like a little Manhattan Project for COVID. We had a tremendous number of volunteers. In fact, there were many more people who volunteered who we could not accommodate. This group is together about 30 some people. By now, we're close to 40, and we're still growing. Our biggest challenge is how to grow so fast without falling apart.

[00:05:29.88] I used to spend a good bit of time in my talks convincing people that epidemics are important, serious, have to be considered carefully and planned for. I don't need to do that anymore. I needed to explain to people why forecasting is important. I don't think I need to do that anymore, either. So I cut out a lot of the explanations, but there are a few things I still want to show you. And this is one of them. This is infectious disease mortality per year since 1900. The peak that you see here, do you all see my cursor?

[00:06:13.07] [INTERPOSING VOICES]

[00:06:14.20] RONI ROSENFELD: OK. The peak that you see here is, of course, the 1918 flu pandemic. The other thing to notice here, of course, is that there has been a persistent decline in mortality from infectious diseases over the century due to wonderful inventions like vaccines and public hygiene and other measures. This improvement kind of dropped to almost nothing starting in the '50s, '60s.

[00:06:41.59] And then was something very worrisome happened-- things started getting worse, starting in the 1980s. Mortality all over the world, including developed countries, started going up. And in fact, that trend continues-- this goes up to about 2000-- that trend continues on here. And of course, we don't know what we're going to see here. We know there's going to be a blip.

[00:07:01.94] It will not be as high as the 1918 blip. We're fairly confident of that. How high it's going to be is yet to be seen. But the take-home message here is that epidemics and infectious diseases are not only a problem now with COVID. Even when we manage to bring COVID under control, or even if it went away tomorrow, it is still a significant problem that kills many, many people.

[00:07:29.11] I mentioned epidemics and pandemics, so I should probably explain the difference between them, and why it makes a difference. If you look at the definition of epidemic in the dictionary, you will find many different definitions, but a common one is a rise in the prevalence of a disease or condition. Usually, we mean a rapid rise. Something happens over a period of weeks, sometimes days. In some epidemics, like HIV, it would be over years. So the pace can change, but usually we mean a dramatic rise in the prevalence of a disease.

[00:08:04.76] And if you look at the definition of pandemic, it says epidemic that spreads globally. That's where the pan part comes from. But that doesn't really capture what we mean by pandemic. It's really not what we use the word pandemic to mean. Really, what we mean in practice when we talk about pandemic versus epidemic is a pandemic is an epidemic driven by a new pathogen. A new virus, a new bacteria, a new parasite. Most often, a new virus.

[00:08:35.39] And the reason that its being new is important is twofold. One is there's very little known about it, so you cannot easily bound the potential damage from it. You cannot bound the risk. You don't know enough yet. You know, when flu comes around, even if it's a new strain of flu and it spreads, epidemic strain of flu, we kind of know flu. We know what it can do.

[00:09:01.90] Even the 1918 flu, which was quite unusual, we have a sense of the bounds on how bad it can get. When you have a brand-new pathogen, all bets are off. It could be a lot worse. Could be orders of magnitude worse. And there is nothing really in the science or in evolution that guarantees that it won't happen.

[00:09:26.63] The second important factor about a new pathogen is that the population has pretty much no pre-existing immunity against it. This is important because pre-existing immunity exists for almost all diseases that we have seen before. And even those that are pretty serious, like flu, the reason flu only affects about 10% of the US population every year is because of pre-existing immunity. And every couple decades, on average, we get a new type, or new subtype of flu that is a-- we call it a pandemic strain. And the pre-existing immunity against it is not as high as it is every year, but it's still in existence, which is why only maybe 30%, 40% of the population would get it.

[00:10:14.92] I'll give you an example. The 2009 H1N1 pandemic affected about a third, or touched or infected about a third of the US population and about a third of the world population. Higher among children. More than 50% among children. So even though it was a pandemic, it was somewhat modulated and mitigated by the fact that it was flu.

[00:10:42.67] When you get a completely new pathogen, you may have to burn through 60%, 70%, 80% of the population before pre-existing immunity brings it to the level of a sort of steady state, an endemic level. In the case of COVID, the inherent infectivity of COVID is such that, if we didn't do anything to protect ourselves against it, if we continued to behave as normal, it would blow through, but it will not go away until it infected about 75% of the population.

[00:11:18.64] Given that we're all much more careful now, and that we have implemented many sort of spontaneous measures-- we don't shake hands, we don't hug and kiss anymore, we don't get too close to people unnecessarily-- that number might go down to 50%, perhaps even 40%. So it's the novelty and in the pre-existing immunity that determines how many people it will have to burn through before it goes away.

[00:11:47.89] Benefits of epidemic forecasting or pandemic forecasting, I'll go through them quickly because you're probably familiar with many of them. There are multiple potential stakeholders and beneficiaries-- governments, health care industry, and individuals. I would add large organizations, as well. With governments, one of the most important things that a government can do is coordination and communication.

[00:12:17.63] If you think about CDC, we tend to think about it as the public health agency of the nation, but really, it isn't. Public health law in the US is such that the CDC has very little authority. The vast majority of authority, making decisions resides locally, within counties and states, sometimes within school districts. CDC has very little authority, perhaps none. But it still has a huge role to play. And that is the role of collecting information, analyzing it, sort of recommending various measures and procedures, and communicating them effectively to the local public health officials, to health care providers, and to individuals.

[00:13:08.62] The timing and focus of communication is extremely important. A large part of the CDC is devoted to communication strategy. A large part of what they do under normal conditions is communication. Beyond communication, there are antiviral policies. School closure decisions, as I mentioned, they are local. And in some diseases, active measures, like mosquito controls.

[00:13:32.77] I'll say something about mosquito controls. I know you may be eager to hear about COVID, but we're in this for the long term. We're thinking not just about COVID and flu, we're thinking about most infectious diseases around. Many infectious diseases in the world, those who contribute the most to mortality, are vector-borne. That means they're transmitted by mosquitoes. There used to be ways of controlling against them by spraying.

[00:13:57.31] Let's take dengue, for example. Spraying has become less and less effective. The only currently effective measure against dengue is a very human-intensive measure of going door to door looking for standing water, looking for holes in screens or lack of streets. You can do that. It works very well. But you cannot do large-scale on an entire city or entire country, not even entire large city. You have to do it locally. So forecasting where the outbreaks will happen is as important as forecasting when they will happen.

[00:14:37.70] If you think about health care providers, here, the benefit of forecasting is huge. At the height of a flu season, a typical flu season, about 15% of beds in hospitals are taken by flu patients. You probably know that hospitals in the US do not like to stand idle. The economics of health care is such that they need to stay almost fully occupied.

[00:15:06.05] So if you're going to get 15% of your beds all of a sudden taken by a rapidly moving epidemic, you want to know about it. You want to maybe cancel some elective surgery. We all know about that through COVID. But this is things they've been doing every year, except with very, very little notice. You want to worry about your staffing, about scheduling of vacations and repositioning of equipment. I used to have to spend a lot of time explaining these. I think all of these have now become household words.

[00:15:36.53] And then my favorite-- well, before I get to my favorite stakeholder, which is individuals, I should mention that large organizations like JPMorgan also have a stake in forecasts. Organizations that employ tens of thousands of people or hundreds of thousands of people, who move them around the globe, it's vital for their sort of internal intelligence to know what the health risks are and when are good times to do that. I'm also thinking of militaries, where it's crucial to know what problems may be expecting you in different places.

[00:16:12.58] And back to my favorite stakeholders are individuals. If you have a mother or grandmother who is old, maybe over 80 or even over 90, flu and, in general, infectious diseases, and COVID, are particularly dangerous to them. So again, imagine yourself post-COVID. You have a flu season coming, flu showing up every year. But a flu season may be very wide. It may be many, many months over the winter.

[00:16:48.72] But flu epidemic itself is actually quite narrow. It lasts maybe eight weeks or so. If you know the flu epidemic is arriving in your city in two weeks, you may want to advise your grandmother or mother-in-law, whatever it is, to stay put, not go to the mall. If she wanted to go travel, visit her sister in Cleveland, and you know Cleveland is in the middle of its peak now, or is about to be hit by a peak, maybe she wants to change her behavior. It may not affect you personally, but people who are in the high-risk groups would benefit greatly from having a heads up.

[00:17:30.69] I should mention the other high-risk groups. For flu, there are four typical high-risk groups-- the very old, the very young, children under six months old, pregnant women, and people with immunodeficiency. The high-risk groups for COVID-- you've probably heard them-- the very old, very similar profile, age-dependent profile to flu. The very young does not seem to be at particularly higher risk.

[00:17:57.87] Pregnant women also do not seem to be at a higher risk, although we've yet to see what happens to babies who are born to such women. That's still to be seen. And immunodeficiency is a factor in COVID, as are a few other things, especially cardiovascular diseases, obesity, diabetes, COPD, and so forth. So there's a whole range of high-risk conditions for corporate that do not seem to be implicated in flu.

[00:18:27.67] I'll tell you a little bit about our research group. We were founded eight years ago, as I mentioned. My colleague, Ryan [INAUDIBLE], who is also a member of our machinery department, as well as statistics, and I founded it in 2012. Our vision is to make epidemiological forecasting as useful as weather forecasting to governments, health care industry, other institutions, and the public. Our method of operation is by releasing all of our code as open source, releasing all the data that we process and create, to the extent allowed by our sources, releasing all of our tools, and most importantly, releasing all of our forecasts publicly in real time.

[00:19:15.74] I might want to skip this. This is a bit about the development of the research community and their realization that the epidemic forecasting is a reality. If you go back, I'll just mention a few things. If we go back to 2012, almost nobody was talking about forecasting epidemics. There was a tremendous amount of work on modeling epidemics. So that goes back more than 100 years now.

[00:19:42.94] There's a very, very rich theory, mathematical theory of epidemics. What people have not tried until this past decade is to actually create a technology, create something that forecasts and gives you statistical bounds and guarantees, and sort of putting it-- it's a little bit like studying the science of meteorology versus creating the technology of weather forecasting. We advocated for that back in 2012, wrote a white paper. We weren't the only ones. There was another group who did that out of Columbia.

[00:20:15.86] In 2013, the CDC got into the game, as well. Maybe we had something to do with it, maybe they were doing it on their own. But either way, they were a major mover. They announced an annual competition to forecast flu in the US. They were followed a year later by DARPA running a competition to forecast chikungunya. A year later, the White House Office of Science Technology Policy ran a competition to forecast dengue over five years.

[00:20:41.31] The DARPA and White House competitions were one-time events. The CDC became an annual event. The number of groups participating kept growing. The first competition, there were I think seven groups. By today, there were-- well, last year, there were close to 40. By today, we have many more because of COVID.

[00:20:59.50] The other thing that's worth mentioning here is, in 2019, the CDC designated two national centers for flu forecasting, which have now evolved to be national centers for forecasting in general because we've included COVID in it. Our group became one of these two centers. The other one is at the University of Massachusetts.

[00:21:21.23] Let me go for one more slide, and then stop for some questions. When we talk about epidemic forecasting, there are actually many, many types of flavors of forecasting. The first one is forecasting pandemics, namely those events that are new, based on a new pathogen, and as I mentioned, we know very little about them and their risk bounds are very, very broad. We don't know how to quantify them. And most importantly, we don't have much data about them. This is the kind of forecasting that I stayed away from very, very adamantly up until January of this year.

[00:22:04.29] Within the alternative paradigm of forecasting epidemics, now you're talking about forecasting something that is somewhat periodic. Not completely. Not completely regular, otherwise there wouldn't be anything to forecast, but something that repeats, and that you believe that past performance is indication of future performance. And that's where we put most of our effort in the last eight years.

[00:22:27.03] Flu falls into this category, dengue falls in this category, and many other diseases fall in this category. And within that category, you can think of forecasting across seasons, or perhaps the most important point with flu is what strain will dominate the next year so that you can formulate the vaccines appropriately. We may be in a similar situation with COVID if it mutates the way flu does. Or you can make forecasts within a season. And I'll focus here on forecasting within a season. That means the season has just started or is about to start.

[00:22:59.58] You may have a bit of information. Here, in this case, the season is maybe a third of the way in. You still don't necessarily see the epidemic wave, you just know that it may be coming. But you're tracking it, and you're trying to forecast when the wave will come, when it will start, how high it will get, how many people will it encompass when it's over, and when will it reach its worst time. So these are the kind of things you may want to ask.

[00:23:25.79] Within seasonal forecasting, you can break into nearcasting, forecasting the next few weeks, longer-term forecasting of the entire trajectory, but also, maybe particularly interestingly, is something we call nowcasting and backcasting. So nowcasting is simply the question of what is happening right now, and backcasting is the question of what happened in the last few weeks or since the beginning of the season.

[00:23:53.50] Now, it seems maybe strange that we would be struggling to understand what's happening now. This does not have a parallel in weather forecasting. In weather forecasting, it's very easy to know what's happening right now. Easy scientifically. It took some work, but now there is a grid all over the planet, a three-dimensional grid of measurement instruments that continuously send measurements from every point on the globe to central locations, where it's shared across all countries. So knowing the present is not a problem. We know the temperature and humidity and wind conditions and anything else that's relevant all over the planet in real time. And as a result, we also know what happened in the past.

[00:24:43.18] In weather forecasting, you know the past and the present. You are free to focus on the future. In epidemic forecasting, you don't. Understand and estimating what's happening right now and what happened in the past is still a very significant part of the problem. In fact, a lot of our work-- I would argue more than half of our work-- was on nowcasting and backcasting as a precondition for doing successful forecasting. That becomes even more critical in a pandemic. And I will get to that.

[00:25:16.60] So this picture tries to capture conceptually this notion that, yes, there is a lot of uncertainty about the future. We're measuring in the y-axis here sort of very vaguely the notion of uncertainty. And that uncertainty grows as you move further into the future, but even in the present and even in the past, there is uncertainty. In fact, it never ends. We don't know with certainty how many people have any particular disease in any particular city, and we probably never will.

[00:25:44.01] Let me take a quick break here and ask if there any clarification questions.

[00:25:51.78] MANUELA VELOSO: I think we are fine, Roni. We should go and take the questions at the end. OK? Thank you.

[00:25:55.93] RONI ROSENFELD: OK, sure.

[00:25:59.27] I mentioned the kind of targets that we're typically interested in-- how bad will this season be, its peak intensity, when will the peak intensity happen, how long, and so forth. Let me mention briefly to the more technical people in the crowd the kind of metrics that are often used in measuring how good your forecasts are. Roughly, they fall into two categories. One is point prediction. You're trying to, quote unquote, guess or estimate the most likely outcomes. And then when you measure, to the extent you can, that outcome, when the time has passed, you measure things like your absolute error or your squared error, and try to minimize those.

[00:26:39.14] A different kind of metric is applied to distributional predictions. That means we don't try to guess a single answer of what's most likely to happen, but we actually try to give you an entire distribution of how likely is each different outcome. And that's far more useful to governments and to other decision-makers than just a point estimate, because a point estimate does not convey to you your level of confidence.

[00:27:04.91] Two people can give you the same point estimate. One of them is extremely confident, the other one is not confident at all. If that point estimate turned out to be very, very accurate, then you could say that the person who was extremely confident gave you, in hindsight, you can say that they gave you a better answer. But of course, if it turned out to be off, the better answer was by the person who wasn't so confident. So conveying both what you think is going to happen and how likely it is to happen is extremely important for decision-making. And this is where most work on epidemic forecasting is focused.

[00:27:43.37] I'll tell you a little bit about common approaches to modeling, and therefore to forecasting epidemics. The first one is the one that you may have heard about most. It's called the mechanistic approach, also a compartmental approach. And the most famous example is the SIR model, or SEIR models. That means that you make very simple assumptions about what matters.

[00:28:14.15] You assume, for example, perfect mixing of a population. If any person is interacting with any other person with the same probability, which is, of course, not true, no matter what unit you're using-- state, city, neighborhood, family-- it's never correct, but it's the assumption that's being made-- and you keep track of how many people at any one time are infectious, how many people are still susceptible-- that means they could be infectious-- and how many people-- that's the remainder-- who are recovered or removed or resistant,

[00:28:46.65] This is the modeling method used by 95% of people who are doing modeling of COVID because it's very, very simple. It's easy to build and to track. It's intuitively very appealing. But it's not very good at making forecasts. The historical track record of the last eight years show that models that were based on SEIR [INAUDIBLE] relatively poorly relative to [INAUDIBLE]. Our group is focused not on [INAUDIBLE]

[00:29:29.01] I'll pause for a minute. It says my connection was not good. We stayed away from SIR models. We are incorporating them a little bit into corporate modeling for reasons I'll describe later when we get to pandemics. The almost opposite of the SIR models are the agent-based models.

[00:29:53.92] "Agent-based" here does not mean what it means elsewhere in AI. It does not mean agents of the type that, say, Manuela's work is famous for. All it means here is individual-level simulation.

[00:30:04.63] So it is also a mechanistic model, in the sense that it tries to model the way we believe the dynamics of epidemics work, namely, you have a simulation of many, many people. You try to build like a Sim world, a model of the world with millions of people. And they live in households. And they go to work and they go to school.

[00:30:27.35] And the households have certain distributions. Some of them are two adults, one children, two adults, two children, two adults, three children, one adult. You try to match the reality of the location you're modeling.

[00:30:41.71] Same with workplaces, commute times-- you really try to build a miniature real world. And then you "infect" some of these people, and you watch them go to school. And there is some probability they will transmit it to other kids, or that adults will transmit it. And there's transmission in the neighborhood. There is no limit to the details that you can put into such a model.

[00:31:03.64] MANUELA VELOSO: Oh, can I tell you something? I think that some people may leave at 11:00 because of other meetings. Then--

[00:31:10.99] [INTERPOSING VOICES]

[00:31:11.37] RONI ROSENFELD: OK.

[00:31:12.31] MANUELA VELOSO: --more. I thought that you-- I think that you want to give a demo. I don't know what that is, but--

[00:31:17.26] RONI ROSENFELD: I will move forward. I will move forward.

[00:31:19.44] MANUELA VELOSO: It might be better. I'm sorry for the beginning, but I think--

[00:31:22.20] RONI ROSENFELD: No problem.

[00:31:22.76] MANUELA VELOSO: It would be better.

[00:31:23.38] RONI ROSENFELD: Let me do that. All right, let me talk for one minute about now-casting, and then move to my demo, because the demo is a lot about the downcasting of COVID.

[00:31:46.74] This is a focus on the concept of now-casting that we described before. And what we're trying to convey here is that the number of people who have flu-like symptoms is here, and it's not measurable. And it can come from many different epidemics, for example, one of them might be flu. The other one might be COVID. Third one might be some other virus.

[00:32:09.63] But what we do have actually [INAUDIBLE] is a variety of other indicators that we can measure, from which we can try to infer what the actual number of people with flu-like or COVID-like symptoms are. So a lot of our work is on expanding this line, on creating what we call "sensors," which are measures of things that we can measure that we believe are strongly related to the latent number of people who have the infection, which is a prerequisite for doing the forecasting.

[00:32:44.28] Having said that, I will skip all the description of how we do now-casting, maybe just to show you the end result of how it works. The black line here is the ground truth, to the extent we're able to have it a year later or six months after the fact when all the measurements are in. The colored lines are the different sensors.

[00:33:09.20] You can see that any one sensor is quite noisy. When we apply our method of sensor fusion, we get a reconstruction of the original signal in real time. To emphasize how important this is in flu and in the future in COVID, the black line is only known well after the end of the season, six months later. The red line is known in real time. So we basically managed to cut six months off of the surveillance process.

[00:33:40.94] Let me move to the pandemic, as Manuela suggested. Our approach to forecasting has always been a machine learning approach. That means not a mechanistic approach.

[00:33:52.91] The last slide I showed you talked about two mechanistic approaches, and the next two were machine learning approaches, which basically means we don't trust our understanding of how the pandemic spreads. We rely on data instead. This is sort of a natural way of describing it.

[00:34:13.21] When you do that with flu, when you have at least 20 years worth of data, it works very well. Our methods have consistently beat all the other methods that are used, including those that used the mechanistic approaches, because we had 20 years worth of data to trade on. I mean, this is the story of machine learning-- when you have data, you typically do better than that a non-machine learning approach.

[00:34:37.12] The problem with a pandemic is that there's almost no training data. So under these circumstances, the mechanistic approaches have a huge advantage. They don't rely as much on data. They rely on stories. They rely on understanding.

[00:34:52.16] The way we compensated for that is by relying more on diversity of geographic locations rather than on time depth. We do not have much time depth at this time. At the time we started this, we had zero historical data on the pandemic, and by now, we have a few months worth of historical data.

[00:35:11.90] What we do have is a lot of goodwill. Because of the conditions, because of the overall both anxiety level and eagerness to help, we had a tremendous number of volunteers helping. Equally important, we had a tremendous amount of data providers coming forward and offering us their data.

[00:35:31.18] I cannot emphasize enough how critical data is to our work, and how difficult it is to get data that's relevant to epidemic and pandemic behavior. I have spent a good part of the last five years, a good part of my work professional time, seeking new data sources, and convincing the owners to release them to us, and negotiating the data use agreements. The other people who have a lot of goodwill at this time are lawyers. So negotiations that took a year or two before were finalized in a matter of a week or two. So this has been a tremendous sort of boost and complement for us.

[00:36:14.57] So here is our strategy in approaching the pandemic-- we could not build sensors in the traditional machine learning sense of sensors because we could not do supervised training of our different covariates or different sources of information. Remember, classic machine learning approach is you have historical data. You have the true labels for that data. You run some kind of mapping, either regression or something else, that trains your model to go from your covariates to the labels that you have for the history.

[00:36:51.26] Well, we're in a situation-- or we were in February, March-- in a situation where there is no ground truth. And we didn't expect to have ground truth for at least six months to a year. So we cannot do supervised training, supervised learning.

[00:37:05.45] What we decided to do instead is unsupervised learning, and build what are called "indicators" where we can show that they have significant information relative to the short-term future, because short-term, a week or two, we could wait. But we could not wait six months. So what I'm going to show you next is all the indicators that we developed that have been vetted and proven to be highly correlated on a time scale of one to two to three weeks, with the evolving pandemic.

[00:37:35.42] The second decision we made for forecasting a pandemic is to focus on what matters in focusing on the pandemic. The thing that matters the most is forecasting demand on hospitals, and on ICUs, and ventilators. That was the case in February, March and April. It's slightly less the case today. I think the bottleneck has been somewhat relieved, but it's still the case that most public health officials we talk to, the first thing they want is a forecast of demand on their hospitals.

[00:38:02.15] When you focus demand on hospitals, you have to realize that these hospitals do not serve just one particular local community. They tend to serve much larger community that spans many, many counties. For example, in southwestern Pennsylvania, they spanned an area of about 13 counties. These are known as "hospital referral regions." And that's the geographic level at which we do our forecasting.

[00:38:22.82] And then we focus on about a four-week timeframe. The reason we chose four weeks is because any longer than four weeks, we felt we could not validate and we could not realistically create accurate forecasts for. There was way too much unknown. Any less than four weeks, it becomes not so actionable.

[00:38:44.21] If you want to intervene in a pandemic, either by making the mitigation stricter or less strict, it will take two to three weeks for that to have an effect. So you need to have a little bit more of a heads-up time. And I already mentioned, we make up for lack of history with geographic diversity.

[00:39:06.74] And I will go to the demo now. So what you're seeing here-- I'll switch to states-- is our main now-casting and eventually forecasting site. Right now, this page focuses on now-casting and back-casting.

[00:39:28.74] So this is a list of indicators that we have developed from different data sources, some of them public, some of them not. Some of them are available at the state level. Some of them are available down to the county level. And some of them are available at metro level.

[00:39:46.98] This is the picture of the United States as of July 7. We can move it to more close to now, July 18. This happens to measure the percentage of daily doctor visits that are due to COVID-like symptoms. This is an important definition. This is technically known as "CLI," COVID-like illness.

[00:40:08.25] It doesn't mean they have COVID. It means they have symptoms that are consistent with COVID. It is not as specific, but it is a leading indicator. And we have demonstrated a very high correlation between this measure and confirmed cases a week and two weeks later and deaths two weeks later.

[00:40:31.95] If you want to see-- you guys are in New York, at least most of you are. You can zoom in on, first of all, New York State, and look at how this measure was tracked over time. You can load a few more weeks and look at the entire story of the epidemic in New York State, or we can switch to metropolitan area and look at the history, which is probably all too familiar and painful to you, of the New York, New Jersey metropolitan area.

[00:41:15.18] You can see here the wave of doctor's visits in May, much later than the wave that you are familiar with of deaths. You see there are some differences between symptomatic behavior and confirmed cases or confirmed deaths. You can switch to hospital admissions. Hospital admissions are a far more specific measure. Usually when you're admitted to hospital, either you already know that the person has been tested and confirmed as COVID positive or it becomes confirmed one way or the other within a day or two.

[00:41:56.39] Switch to county view. You can look at the behavior of individual counties. I will go to my favorite county, Allegheny County in Pittsburgh. You can see we've had a reasonable history up until recently. Some areas do not have enough data to form a robust estimate, so we lump them together within the state.

[00:42:36.40] I'll show you a few more of the indicators. One of the perhaps most exciting indicators we have, or a set of indicators, comes from a survey that Facebook enabled us to run. So Facebook agreed to put our survey on their feed, on people's feed throughout the country. It's opt-in. It's optional, but quite a few people opt in. We get, I would say, about 70,000 responses a day.

[00:43:08.55] This is the percentage of people who respond positively to the question, does anybody in your household currently have any COVID-like symptoms? Where we give them the list of COVID-like symptoms. You can see that percentages are quite low if you switch to state level. The percentages would typically be in the 1% to 2% level.

[00:43:33.13] You may think that's very small here. In Texas, you have 1.2. But if you change the question slightly, and you ask them, do you know anybody in your community who has COVID-like symptoms, the numbers jump to the range of 20 to 35. Both numbers are very informative, and they actually don't convey the exact same information. In other words, having both of these indicators gives you a higher correlation and higher predictive power than having just one of them.

[00:44:06.22] I don't know, but chances are if there are 300 people on the call, that several of you have seen these surveys. I hope you took them. They ask a lot more questions that we're still in the middle of analyzing.

[00:44:20.71] We also track people's mobility. This is from Safegraph data. That's a company that makes their data available.

[00:44:29.32] And that's available, again, at the metro area, county, and-- so this is a measure of how mobile people were at different times of pandemic. You can see very clearly the stay-home periods versus the more modern periods. What you're seeing here is a weekly cycle.

[00:44:57.09] Google search trend is another source of information that was very useful with flu, and is useful now. I know it's 11:00. Manuela, did you want me to talk a little bit about the forecasting itself? Most of our work has been on now-casting. We started doing our forecasting about-- we started releasing our forecasting to the CDC about two, three weeks ago. We waited about a month. We took about a month of internal forecasting to track our accuracy.

[00:45:26.27] MANUELA VELOSO: So I think very good. Thanks a lot, Roni. So this is amazing. And does the tool also provide an ability to see your forecasting, or it's basically an analysis-- or is it the analysis of the past data?

[00:45:42.29] RONI ROSENFELD: This website is past and current. This is up to date and updated daily. We have not released our forecasting publicly yet.

[00:45:55.08] The CDC asked us to give them our forecasting first. They're going to sort of vet it. And they're very, very sensitive to public display of forecasts. We will eventually make our forecast public, but not until we have, I would say, another three to four weeks of track record.

[00:46:14.09] So what we've done is, our first forecasts were made about five to six weeks ago. We went through a period of three to four weeks of vetting them internally, checking prospectively that they were accurate enough. Then about two weeks ago, we decided they're good enough for the CDC. We started passing them to the CDC. We submitted the last two weeks, and in about two more weeks, we will probably make them public.

[00:46:38.57] MANUELA VELOSO: Very nice. So let's thank Professor Rosenfeld, then open for questions now. For the people that can stay with us half an hour more, we can in fact then delve into the forecasting aspect to see the technical details for the forecasting. Otherwise, let me take questions from anyone who would like to first ask questions about general studies on epidemics, and then this particular kind of now-casting ability that Professor Rosenfeld made available. So should we open for questions? Shane, can you give directions on how people should ask questions please, or someone?

[00:47:25.51] AV TECHNICIAN: Ladies and gentlemen, please use the Raise Your Hand feature located at the bottom right of your screen, if you're using a computer, or in the top right if you're using an iPad. Your name will be called in the name it was received. We do have a question from Rambir. Please unmute yourself and go ahead.

[00:47:43.57] MANUELA VELOSO: Thank you.

[00:47:45.22] AUDIENCE: Hi, can you hear me?

[00:47:46.90] MANUELA VELOSO: Yes, Rambir, we can. Go ahead.

[00:47:49.83] AUDIENCE: Yeah. So first of all, thanks for organizing this. This was good, even though in between, I kept on getting messages, I mean work messages, but I tried to remain engaged. And this was a good presentation.

[00:48:10.58] My question is, I have not heard of this forecasting earlier in the group. So when you mentioned that a lot of others are using it, so until they tried, when the COVID was happening all around, between January and April or May, was there any government organizations using your forecasting to prepare themselves? And what do you think in future? Do you think your forecasting is-- I mean, is that good that in future, if there is similar epidemic, then you can let them know probably a week or two weeks in advance, and they can actually prepare better? Because this time, definitely, we were not prepared well.

[00:49:15.67] MANUELA VELOSO: Thanks.

[00:49:16.74] RONI ROSENFELD: Yeah, it's very easy to forecast. It's very hard to forecast accurately. So you've seen a lot of forecasts out. It's not hard to build an SIR model. In fact, there are probably hundreds of them online, and they can come online as early as February or March.

[00:49:36.98] AUDIENCE: Right.

[00:49:37.67] RONI ROSENFELD: That's not what we do. What we do is build a track record, and validate our forecasts, and give you not only a forecast, but also a confidence level. That takes time. We have been doing it for flu now for eight years.

[00:49:59.78] And to answer your question, the CDC has been using our forecast for flu. They tested it for the first three or four years in 2017. The head of the flu division, one of the largest divisions of the CDC, made a decision to start using our forecast for their operational use.

[00:50:17.69] So they started putting them on the website. And they started incorporating our forecast into their public--

[00:50:25.04] MANUELA VELOSO: I'm sorry, can I ask that everybody mutes themselves? Or Rambir, maybe it's you. We are listening to typing. Thank you.

[00:50:32.42] AUDIENCE: Yeah, I'm sorry.

[00:50:34.01] RONI ROSENFELD: They have started using our forecast in their operational work. They incorporated them into their public messaging. They incorporated them into the web.

[00:50:43.11] If you look at the web pages for flu forecasting for the CDC, you will find our forecast there. So we've been doing that for several years now. For three or four years, they've been using it regularly, and they will continue to use it.

[00:50:55.52] Now for the current situation, you can jump up and give a forecast in March or February, March or April. And by the time it's time to check it, probably it will not be remembered anymore. What's really important to do is to create a track record of forecast so that you don't have to rely on whether the model was right, or accurate, or appropriate. You rely on the track record instead.

[00:51:24.11] This is what we have been doing, slowly and surely. we're going to continue to do both COVID and flu in the next year, in the coming years. We're here for the long term. Our center was funded for five years and we'll probably be renewed for at least five after that. So our focus, in the case of COVID right now, is in validating the forecast and making up to four weeks ahead predictions. This is what we feel we can do for the first time a pathogen comes around.

[00:52:02.25] As far as being prepared, I don't think you can be prepared for when a pandemic shows up. But once it shows up, you can start providing very, very near-term estimates of it. I think a lot of things about being unprepared had to do with many other things, with protective equipment, with the testing, and so forth that I don't think we need to comment on here. Clearly, we could have done things better, but our focus is on creating sort of the long-term infrastructure and the long-term guidance at the local level.

[00:52:35.26] AUDIENCE: OK, so thanks for that. I just have a follow-up question. So one is that--

[00:52:41.71] MANUELA VELOSO: Rambir, sorry, maybe we'll give a chance to someone else to ask a question. And then you can come back.

[00:52:46.77] [INTERPOSING VOICES]

[00:52:47.01] AUDIENCE: OK, no problem.

[00:52:48.83] MANUELA VELOSO: Hold on a second, Rambir. Just a second. Are there any other questions, please?

[00:52:58.92] AV TECHNICIAN: Yes, Avishek, unmute yourself and go ahead.

[00:53:02.48] AUDIENCE: Hi. This is Avishek [INAUDIBLE]. I have a couple of questions. Basically, this seems more like now-casting, as you're saying, when I look at the indicators, it seems like there's a strong positive correlation between them. So how have you taken into consideration that factor? Because to me, it seems like it's giving same results for all the indicators and there's a strong positive correlation. And second, can you talk a little bit more about the unsupervised learning for any trend analysis that you're going to do for future predictions, like what algorithms you have used on unsupervised learning?

[00:53:42.03] RONI ROSENFELD: Sure. So the indicators are correlated because they're all indicative of the same future behavior. We selected them to have high correlation with week ahead and two weeks ahead measurements. So in that regard, they're correlated.

[00:53:58.44] But there's also a lot of differences between them. And in fact, one thing I didn't show you is we created a combined indicator that combines many of these, which is for people who are not interested in the detailed view of the different aspects of things, but just want to see how bad things are. And what we did there is basically principal component analysis, with some modifications because of the most recent data is not complete. It's subject to backfill or to revisions later on.

[00:54:39.21] That's one reason. Second reason is that the different measures are in different dimensions. But effectively, at the heart of what we did in combined measure, is the first principal component.

[00:54:54.75] As far as the differences between the different measures, I don't know how much you've looked at the site, but there are many places where they disagree. And that's OK. It tells you something, too.

[00:55:05.12] The main use we see for this map is for local health officials. Remember, public health is done locally. And by local, I mean county, usually.

[00:55:14.41] So public health county officials typically are interested in the different aspects of the pandemic as they happen in their location-- how many people are currently hospitalized, but also how many people are currently being tested, what is the positivity rate, how many people have symptoms but have not gone out to be tested, how many people have heightened awareness maybe because of COVID in their neighborhood and how many people know people in the neighborhood who have it and so forth. And these may very much disagree with each other.

[00:55:51.54] AUDIENCE: So how are you making sure that the results are not over-fitted when you are combining various indicators?

[00:55:57.77] RONI ROSENFELD: Because we have not combined them. So we're not combining for now-casting. We combine them for forecasting. So for now-casting, we did a principal component analysis, and we're giving you here the first component.

[00:56:10.25] For forecasting, we have employed two methods. One of them is a time-aligned regression. So we take the time series of cases and deaths in different locations, in different counties or in different states. We align them by their best fit of when things started picking up in each location.

[00:56:35.91] And then we use that with regularization, basically with L1 regularization, to make forecast of incident deaths one week ahead, two weeks ahead, three weeks ahead, and four weeks ahead. That's the essence of the method that we have so far vetted. We have another method that's being vetted now, and we intend to combine them.

[00:56:59.86] MANUELA VELOSO: OK, yes. Maybe you can also send me, Roni, a pointer to a paper that explains the technical details. Because many people in the call are basically interested on the technical detail of the actual now-casting.

[00:57:15.80] RONI ROSENFELD: Sure. I was aiming my talk at a broader audience, but I'm happy to point you where we are writing up a blog post with the technical details of the forecasting. And we have the details of the now-casting already online, so I'll point you to all of them.

[00:57:32.71] AUDIENCE: That would be great. Thank you so much.

[00:57:34.46] MANUELA VELOSO: Thanks a lot. Other questions?

[00:57:38.71] AV TECHNICIAN: Yes, Joseph, please unmute yourself and go ahead.

[00:57:44.67] AUDIENCE: My question was around the confirmed cases. One of the key indicators is about when we hit herd immunity, whether that's 50% or 75% depending on how much we isolate. Currently, the confirmed cases are around about 4 million, but reasonably, only a fraction of the real infected people are that 4 million. So I've heard estimates of roughly 10 to 1 ratio between confirmed and true cases. So do you have any indicators which provide insight into what the true number of cases that are out there, and how close we are to herd immunity?

[00:58:21.92] RONI ROSENFELD: Yeah, that's a wonderful question, thank you. We have that in the pipeline. We don't have it yet, because the only measure that we have that sheds light on that is antibody tests.

[00:58:33.95] And those are few and far between, but they're becoming more and more diverse. So as the CDC has a plan that has already started of doing periodic measurements of same populations, so measuring their antibody levels over time, we will mirror them here and relate them to-- yeah, this has been the biggest open question, the $64 billion question, or I would say trillion dollar question.

[00:59:02.51] What is the size of the iceberg underneath? And the number you quoted of around 10 is indeed a number that I am familiar with. There was a paper that just came out yesterday in the Lancet with some more information that showed significant variability between locations, so a very different multiplicative factor in New York City, say, as in some rural areas. But it all is around the order of magnitude of 10, maybe 6 to 20-something.

[00:59:33.64] Herd immunity-- New York might be close. So the consistent antibody measurements in New York are into 20-some percent. And remember that the antibody testing takes two weeks, so if you're measuring it today, it's the state of the population as it's been infected up until two weeks ago.

[00:59:53.84] And the measurements themselves were done in May. So we had 20-some percent already in late April. By now, it's probably somewhat higher.

[01:00:04.00] The question is what percentage would you need to get herd immunity? I mentioned earlier that we know if you behave the way you behaved a year ago, you would probably need about 70%, 75%. But nobody is behaving the way they behaved a year ago, so 40% might be sufficient.

[01:00:22.33] AUDIENCE: Thank you.

[01:00:25.01] MANUELA VELOSO: Thanks a lot. Another question? We have time for a few more questions.

[01:00:31.38] AV TECHNICIAN: Yes, [INAUDIBLE], please unmute yourself and go ahead.

[01:00:37.84] AUDIENCE: Hi, Roni. So I have a question, because when we are saying "unsupervised learning," so you must have used-- and you have also used time series analysis. So was there any possibility where you, while doing a correlation analysis between different factors, vector analysis-- I'm basically referring to vector analysis here-- so what was the exact deciding factor for you for which particular parameter or which particular dimension, you would consider under this integration

[01:01:14.92] [INTERPOSING VOICES]

[01:01:15.42] RONI ROSENFELD: Sure. Sure. So we use two steps. The first step was vetting of each information source. That was done in isolation, each source by itself.

[01:01:27.77] We considered many more. In fact, there are about three or four more that are coming in the next week. And there are more that we considered, worked on for a while, and decided to ditch. Our measure for that was correlation, one- and two-week-ahead correlation with cases and deaths. That was the first step.

[01:01:46.74] Second step is how do you combine them? And here, we have two subgroups, each working with slightly different methodology. The one that we have felt confident enough to put out and give the CDC starting two weeks ago uses L1 regularization.

[01:02:03.26] So we basically threw in all of the measures, all of the covariates, and used L1 regularization to choose among them. There's some variations among that. Some of the measures are more useful at the time shift, as you can imagine, than others. But ultimately, we didn't have to choose them. We basically use a regularizer to sort of push down on the number of covariates.

[01:02:34.93] AUDIENCE: OK, thank you, Roni.

[01:02:38.18] MANUELA VELOSO: Next question.

[01:02:41.23] AV TECHNICIAN: Charles, please unmute yourself and go ahead.

[01:02:44.63] AUDIENCE: Hi. This is Charles. Thanks, Dr. Rosenfeld, for your wonderful presentation. And I have a question regarding the now-casting, because I think one of the major challenges we're facing when dealing with COVID is this kind of super-long, a two-week long incubation period, plus the number of asymptomatic COVID patients. So from the scientist's perspective, that is saying the data we're looking at probably is like two weeks old and probably underestimated. So I was wondering if there is any method we can adapt to make sure our model can tolerate this type of gap and uncertainty, and provide real-time guidance to people right now.

[01:03:24.53] RONI ROSENFELD: That's a great point. We've been struggling with exactly this issue. You could imagine what's called in epidemiology a "severity pyramid."

[01:03:36.62] Imagine at the bottom of the pyramid is the whole population. A rung above it is people who are infected, whether or not they're symptomatic. A rung above that is the people who are symptomatic, a rung above that is people who chose to go to the doctor or the emergency department, so they're called "medically attended." And then above that is hospitalized, and above that is intubated, and above that is dead.

[01:03:59.63] This is called the "severity pyramid." And parallel with the severity pyramid is the time dimension, which is what you were talking about. When are people going through that? Of course, they start by being infected and then on average five and a half days later, most of them show symptoms. And then a few days later, some of them choose to go to the doctor, and some of them a few days later go to the hospital, and so forth.

[01:04:22.59] So you can imagine that the bottom of the pyramid is the leading indicators and the top of the pyramid is the lagging indicators. And what we have done very successfully with flu is combine all of them, where the leading indicators are, in some sense, the most informative, the most important for the reason you mentioned. But at the same time, they are less specific, because if somebody has symptoms, we still don't know if it's flu or in this case, we still don't know if it's COVID.

[01:04:49.20] So as you move up in the time, as you lose time, you gain specificity. So your question is focused on the early stages, on the early indicators. And we put a lot of effort into that, things like retail information, people buying thermometers.

[01:05:08.97] People searching the web for information is often the leading indicator. It doesn't solve the problem completely because we don't have any measurements of actual infection. The measurements we have start with symptoms. But it's still as ahead as we can get.

[01:05:27.39] If you are looking for measures of infectivity, you have to go and look at transmissibility. So these are these two measures. If you believe that people's mobility behavior is captured by Google and by Safegraph, how much time they spend at work versus how much time they spend at home, versus how much time they spend in places like malls and in local places-- if you believe that's strongly correlated with transmission, that may be by far your earliest leading indicator.

[01:05:59.11] So far, the data is very unclear about that. Part of the problem is that when people lock down, they still continue to transmit, except it's within the household. The way we ideally would deal with that is by having the cases broken down by was it likely a household transmission or a community transmission.

[01:06:22.35] Unfortunately, we don't have that data yet. I hope that in the future, that kind of data will be collected and made available. But it is a constant struggle, the timeliness.

[01:06:34.31] AUDIENCE: Thank you so much.

>[01:06:39.53] AV TECHNICIAN: Next, we have [INAUDIBLE]. Please unmute yourself and go ahead.

[01:06:43.94] AUDIENCE: Thank you. Thanks for the excellent lecture, Dr. Rosenfeld. My question is on your back-casting because backcasting is very popular and entrenched in finance.

[01:06:56.42] Given your better forecasting technology, what are you trying to do with the backcasting? Are you not trusting the data? And does it affect your now-casting? Thank you.

[01:07:07.49] RONI ROSENFELD: So by backcasting, I do not mean prospective or in-sample forecasting. What I mean is trying to estimate as accurately as possible how many people were infected in the particular location at a particular week in the past. There is no ground truth. That's our biggest problem.

[01:07:27.50] There is no ground truth. Nobody knows and nobody probably will ever know what the actual number is. The only way we can validate how good our estimates of past incidents is is by do they help us make better forecasts? That's the proof is in the pudding.

[01:07:46.76] Let me show you an example where there is ground truth. Flu is very not sexy these days because of COVID. But we've worked on flu for a long time.

[01:08:05.35] So this is our now-casting for flu. It's called the ILI-Nearby. ILI is influenza-like illness. It's been around for five years now. What I want to show you is one particular chart. It also speaks to the previous questions, I think Charles' comment on timeliness.

[01:08:37.79] It comes back, but it's not the one I wanted. So let me make it a little bigger. In flu, there is a very large surveillance network run by the CDC that collects information from doctors' offices-- not from all of them, but from a large representative sample-- and makes it available, but at a significant time delay.

[01:09:05.26] So information about flu during a particular week is first reported at the end of the following week. And then it's revised a week later, and a week later, and a week later, and it's continued to be revised throughout the season until well after the end of the flu season. In June, July, the CDC finalizes all of their reporting and measurements and determines what is the official ground truth for flu for each one of the weeks of the preceding season.

[01:09:33.29] Of course, by that point, it's not very actionable. It's way too late. It's good for historical purposes, for analysis, and it's not really, really ground truth, because we didn't really measure every person in the community, but it's much closer to ground truth than anything we have for COVID right now.

[01:09:50.38] So in this situation, we were able to do supervised training of the same models and the same kind of indicators. And what you're seeing here is the trade-off between accuracy and timeliness. The way you read this is the shaded week is the week in question, so everything here is about a specific week.

[01:10:10.01] And the x-axis is when you got the estimate. The first time the CDC has their own estimate is here at the square. And as I mentioned, it's available a full week after that week ended. And it's not very good.

[01:10:26.58] It's got a significant error relative to the final estimate, relative to the "ground truth." A week later, there is a revised estimate that the CDC gets that is significantly better. And then it keeps getting better and keeps getting better, but it doesn't get to zero until many, many weeks.

[01:10:44.07] What we were able to do for them is the following-- we were able to give them an estimate that's almost as good as their first estimate, but almost a week earlier, as soon as this week ended. We were able to give them an estimate before the week ended. This is basically a forecast. It's happening on the second day of that week.

[01:11:03.32] We were able to give them an estimate a week before that and the week before that. So these are true forecasts. And you can see the trade-off here. You can see how as time passes, you get a more and more accurate answer. What we were doing for them is basically expanding the time horizon backwards to when they get the estimate.

[01:11:22.67] Now, this, you could argue, is not nearly as good as this, which is true. But it is already quite useful to them. It is very useful to them, even with a 0.5%-- this is measured in percent, so this is about 0.05% error. It's still extremely useful for them to know it four weeks in advance or three weeks in advance relative to what they had before.

[01:11:44.36] We would like to do the same thing for COVID. The problem is not not having the data. It's not having the ground truth.

[01:11:54.30] AUDIENCE: Thank you.

[01:11:58.63] AV TECHNICIAN: Next, we have Alex. Alex, please unmute yourself and go ahead.

[01:12:02.31] MANUELA VELOSO: And this will be our last question. OK. Alex. Can we try the next one?

[01:12:18.69] AV TECHNICIAN: Yes, [INAUDIBLE], please unmute yourself and go ahead.

[01:12:23.57] MANUELA VELOSO: Thank you.

[01:12:25.30] AUDIENCE: Hi. Thank you, Professor, for this webinar. My question is for the fundamental corollary for predictions. So you said that the government didn't allow you to really put out predictions, but you do intend to [INAUDIBLE] the forecast. So if and when you do, for example, you would predict that four weeks from now, in your forecast, the cases would go down significantly.

[01:12:51.59] But what if this information is public, and people get overconfident, and they started going out? So the forecast would cancel itself. So do you have that kind of fear? And if you do, then what are the measures that you are taking?

[01:13:10.52] RONI ROSENFELD: We do have that kind of concern. Let me just clarify. I did not mean to say that the CDC did not allow us to forecast it.

[01:13:16.70] We are free to forecast whatever we want. It's just that they asked us to provide them the forecast ahead of time, for them to vet it with other forecasts, and to combine it into an ensemble that they use in their communications. We decided we do not want to release it publicly until it's been vetted through that process.

[01:13:38.24] To answer your question, the question of when forecasts interfere with, change reality, how do you deal with that, I call that a good problem to have. It used to be that you had to really convince people to listen to these forecasts. Now, we have the opposite problem, that they listen too much to these forecasts.

[01:13:59.21] They listen to forecasts that do not have a track record yet. People are so desperate to know what's happening to reduce the uncertainty and the anxiety, that they listen to forecasts. And in fact, it's very dangerous, I would say, or problematic to release forecasts, because it is almost guaranteed that they will be misinterpreted.

[01:14:22.32] For the reasons that you alluded to-- that all these forecasts are done under particular assumptions of what the public behavior will be and what the government decisions would be. And when the decisions or the public behavior deviate from that, the forecast is really meaningless. And yet people forget that.

[01:14:41.61] Even when you do a good job of putting the warning on the forecasts, and many, many sites did not do as good a job as they should have, but some did. And they said, this is only under these conditions. That's too much for people to remember and carry with them.

[01:14:56.13] People forget that. All they remember is the curve. So that's why releasing forecasts publicly in a time like this is really fraught with danger.

[01:15:06.69] Our goal has always been support to public health officials rather than the public. We do make what we do public when we can, because it's one of our target audiences in the long term. But for COVID, our decision was to focus on public health officials rather than the public for several reasons, but this is one of them. It's really-- you're asking for trouble if you release a forecast publicly.

[01:15:38.44] MANUELA VELOSO: So this is actually a perfect way to end, with a very kind of like inspiring comment and the very excellent questions. Let's thank our speaker again. And thank you very much, Roni. It was a pleasure listening to this, an amazing work. And I wish we could have-- maybe we'll have a follow-up with papers for the people interested in further technical details. Thank you very much, Roni.

[01:16:05.14] RONI ROSENFELD: Thank you, Manuela. Thank you all.

[01:16:06.89] MANUELA VELOSO: Bye-bye. Thank you. Bye-bye, bye-bye. I'll leave the--

[01:16:09.12] [INTERPOSING VOICES]

[01:16:09.28] AV TECHNICIAN: Thank you.

MILIND TAMBE: So let's just jump right into it. So with the current progress in AI and multiagent systems, it is now important to direct these advances toward societal benefits. And I'll focus on three areas, public safety and security, conservation, and public health

And in all of these areas where we're viewing these societal problems as multiagent systems, we focus on the key research challenge-- how to optimize our limited intervention resources when interacting with other agents in these domains.

So when we talk about public safety and security, we have the limited resources of the security resources. We have a large number of targets to protect limited security resources, how to schedule or plan or allocate these resources, taking into account the watchful adversary. We've contributed a new model called Stackelberg Security Games and contributed new algorithms that have been in use by security agencies in the United States and internationally.

With respect to conservation, the challenge is we have large conservation areas to protect limited range of resources. Here we have contributed a new model called Green Security Games. Concrete example is work we've done in Uganda for the past several years. Taking into account past poaching data, we are able to predict where poachers may set traps or snares, and for the past several years, then remove a large number of snares and even get poachers arrested.

Finally, with public health, the challenge is we have a limited number of social worker resources. Concrete example here is work we have done in homeless shelters in Los Angeles. Harnessing the social networks of homeless youth, we are able to show that our AI influence maximization algorithms are far more effective in spreading information about HIV compared to traditional approaches.

More recently, as Manuela mentioned, I've taken over as the director for AI for Social Good at the new AI Research Lab that Google established in Bangalore. So we held a workshop a couple of months ago, and out of that workshop came out six projects that we are actively pursuing. I'm not going to be able to describe them in a whole lot of depth, but each of them is with a different NGO. For example, ARMMAN is an NGO that focuses on the health of babies and young mothers. Khushi Baby is another one that also focuses on health of babies.

In conservation, Wildlife Conservation Trust, for example, focuses on human wildlife conflict, and others. And so we are eagerly pushing this forward, and hopefully we'll be able to start reporting on the results of these in the next few months to come.

But as you see all of these projects, there'll be three themes that will cut across them. First, multiagent systems reasoning-- that's my area of work-- in particular, use of game theory and social networks. But all of this is embedded within a data-to-deployment pipeline. By that I mean it's very important for us, as we work on AI for Social Good, to actually visit the sites where the work is being deployed, sometimes in remote communities, to understand the challenges firsthand, understand what kind of data are available.

This immersion gives us the kind of data that we want. Following that, a predictive model then makes predictions of which case is a high risk versus low risk. But we have limited resources, so we can't intervene on all high risk cases. So a prescriptive algorithm, then-- usually this multiagent systems reasoning algorithm-- tells us which cases to actually intervene on.

And following that, field test and deployment. And this is important because by testing things in the field, we can understand what are the weaknesses of our models and algorithms.

I wanted to make two quick points about this data-to-deployment pipeline, the first being field testing is important not only because it tells us what are weaknesses of our algorithms. But we are working on AI for social impact, and it is important to actually achieve social impact. We can't say it's AI for social impact and stop by writing a paper about it.

Secondly, we've always been told there's a lot of data. We all have big data. In all of these domains, there is not this big data, but rather a lack of data. And so we constantly have to fight sparsity of data, and you'll see that struggle throughout all of these projects.

Finally, all of this work is only possible because of the interdisciplinary partnerships with governmental and nongovernmental organizations. And to establish these, we've really patrolled with the US Coast Guard on their boats in this Port of New York, really patrolled with wildlife conservation agencies in forests in Malaysia and Cambodia. We've spent time in homeless shelters in Los Angeles so that we can understand user needs and understand challenges they truly face.

So with this, let me get into the outline for this presentation. So I am going to give you a quick overview of about 14, 15 years of the work we've done, starting with public safety and security. This is the earliest part of our work, and I'll go through this rather quickly, focusing more on the more recent work in conservation and public health.

And so all of the work has been published in top AI conferences, AMA, AAAIH and others. There you'll see a lot of simulation results and formal models and so forth. Here I'll focus more on the real world evaluation. If you are interested in following up on this more, there are these three books, each on one of these topics, which are edited books. If you buy these, then from the royalties I can take my students out to a nice dinner.

But to really acknowledge the contributions of our students and postdocs, I'll put pictures of the student who led the project in the top right-hand corner of the slide where the work will be shown.

So let's start with public safety and security. So this work started in 2007 when the chief of airport police at the time, Errol Southers, approached us to improve security at the LAX airport. His challenge was somebody driving a suicide truck into one of the terminals, as happened in Glasgow.

There are eight inbound roads into the airport, but not enough officers to have checkpoints on all roads at all times. Eight terminals, not enough dogs to be everywhere. So question is, where and when do you set up checkpoints? Where and when do you do canine patrols? The question for us was could we propose game theory as a way of optimizing these limited resources? Could we convince scientific advisory committees that this was the right approach? Could we convince our AI reviewers that this was the right approach, and could we convince the sergeants on the ground that this was the right approach?

But that's what we decided to do, and we built a new model called Stackelberg Security Games. I'm going to explain it using this two-by-two toy example. So here we have one police unit trying to protect two terminals, and there's one adversary. You could imagine that the defender, knowing Terminal 2 is more important, always trying to protect Terminal 2.

Now an adversary conducting surveillance will attack Terminal 1. The adversary gets a positive reward of 5. The police get a negative reward minus 5. If, as a result, the police were to switch and say we'll always protect Terminal 1, an adversary conducting surveillance will now attack Terminal 2. The adversary again gets a positive reward. The police again get a negative reward.

So any deterministic strategy the adversary could defeat. If the police were to use a randomized strategy, a mixed strategy, 60% of the time they're at Terminal 1, 40% at Terminal 2, an adversary conducting surveillance will only know that the police are there 60%, here 40%. What they'll do tomorrow remains unpredictable.

So these kinds of games are called Stackelberg games because defender commits first to a randomized strategy. The adversary observes and then responds. These are security games where they are played on targets, and the payoffs are based on estimates of losses that may occur in terms of lives lost, for example.

Now we are optimizing the limited use of limited resources. We are not guaranteeing 100% security because in the real world, there is no such thing. We're increasing the cost or uncertainty to an adversary in coming up with a plan of attack.

So this is how we built the ARMOR system at LAX. We start with the game metrics. The payoff's coming from loss of human lives and other estimates. This is fed into a mixed integer program that produces a mixed defender strategy. For example, the probability that there is a canine patrol at air terminals 2, 5, and 6 is 0.517. Probability there is a canine patrol at 8:00 AM at Terminals 3, 5, and 7 is 0.33. And then we sample from this distribution to generate an actual schedule.

So for example, at 8:00 AM, send Team 1 to Terminal 2, Team 3 to Terminal 5, Team 5 to Terminal 6. At 9:00 AM do something different. And this is the actual schedule that the police officers get, and then they execute this.

So I'll briefly go into the mixed integer program just to give you a sense of how this program was written. And so we are trying to maximize defender expected utility. RIJ here refers to the reward to the defender if the defender takes a strategy I and the adversary takes a Strategy J. XI is the probability with which we take a Strategy I. So X1 is the probability there's a dog on Terminal 1 and a dog on Terminal 2. X2 is the probability there's a dog on Terminal 1 and dog on Terminal 3.

So for every single combination of defender resources to targets, we generate a probability variable. This works for the small domain. It's hard to scale up, and I'll come to how we did that a little bit later, that scale up. We model the fact that the adversary gets to observe our mixed strategy, our randomized strategy, and then chooses a particular terminal to attack. And so this is adversary's best response to our mixed strategy.

So with this program written and everything in place-- and there's a lot more that went on for many months to make it all work, and I'm not going to go into some of those details-- ARMOR was operational at LAX airport. It's the first such application of computational game theory for operational security.

And there's a lot of news surrounding this event, and reports of weapons, captors going into the airport, like in January of 2009, all these weapons being captured going into the airport. So this led to floodgates opening in terms of other security agencies looking to use game theory to operationalize, to improve their operations.

But one challenge we faced was how to scale up these games to be able to solve large-scale games. So if you have 1,000 targets and 20 guards, for example, suddenly the game add 10 to the power of 41 different strategies, 10 to the power of 40 even different roles as shown here. And if you just fed it into our ARMOR program, the program would quietly die, running out of memory. So we needed a different approach.

And so this was done by using this iterative model. So you start with a small game matrix, and there's some iterative process that tells us what's the next best role to add, the next best defender strategy to act. And you iterate in this fashion until you converge to a global optimal, but without expanding 10 to the power of 41 strategies. So you only may expand 1,000 strategies and still achieve a global optimal.

So this is how we built a number of other applications. And so we started with ARMOR at the LAX airport, IRIS for the Federal Air Marshals. If you are on a US Air carrier-- American, Delta, United, et cetera-- whether there was an Air Marshal or not on your flight internationally may have been decided by this program.

PROTECT, which was to generate patrols in different ports-- New York, Los Angeles, Boston, so forth, including around the Staten Island Ferry, and the list goes on. Each of these ended up in Congress. So for example, in 2008 or '09, Errol Southers talked about ARMOR in the Congressional committee hearing.


LAX is safer today than it was 18 months ago. A team of researchers led by Dr. Milind Tambe worked with our department to develop ARMOR. This software randomizes our vehicle checkpoints along airport access roads and the deployment of our explosive detection canine teams throughout the airport.


MILIND TAMBE: And I guess that one disappeared. But anyway, other such hearings as well. But of course, you sort of put that in the paper. Oh, our work got cited in a Congressional Committee hearing. But Reviewer 2 is not impressed. And so usually that means you really have to go and understand things in more detail, do real world testing.

And so we've spent a lot of time trying to evaluate things in practice to show that security games approaches are superior in optimizing limited resources compared to human schedulers or other simple random approaches.

For example, we spent time trying to figure out-- fare evaders on trains in Los Angeles is a challenge. So LA Sheriffs Department does randomized checks. So we could use a game theoretic approach to do these checks versus their baseline strategy. And so we sent officers trying to look for fare evaders on trains for 21 days, sometimes using a game theoretic approach, sometimes using this baseline approach-- they didn't know which one was which-- and at the end of the period, try and understand who did a better job.

And our results were that the game theoretic approach were successful in capturing 60% more fare evaders compared to their baseline approach. And there's many other such real world evaluations that we did. Some came out naturally. For example, at the LAX Airport, from before ARMOR was deployed to after, the numbers of arrests jumped fivefold.

So all of this just to show that the security games are indeed more effective in optimizing limited resources compared to traditional approaches.

So this security games approach then led us into wildlife protection and conservation. So this came out of a trip. I was in Uganda. This is Murchison Falls National Park in Uganda. And there's wonderful wildlife there, and these are my pictures from the trip. But there's also threat to the wildlife, snares or traps that are used to maim and kill animals, and in fact, thousands of these traps that are used every year to kill these animals. As you know, illegal wildlife trade is a $5 or $6 billion industry.

And so how can we optimize the use of limited range of resources? So we've come up with this new model called Green Security Games. And so let's take the Queen Elizabeth National Park in Uganda. Now we divide it up into one kilometer by one kilometer grid square. And so now each grid square is a target. Where there's water, for example, more animals, therefore a more important target.

And we could imagine at first we would solve it just like we did with ARMOR with this mixed integer program. However, it doesn't work like that because we are not here working against a fully strategic adversary, but instead many multiple, boundedly rational poachers. So they're not going to provide this best response.

Instead, we are going to learn the adversaries' bounded rational response from past data. For each grid location I, we are going to, based on past ranger patrol frequency and different features, understand the probability of finding a snare in each grid cell. And so this function GI, then, can be used to optimize our range of patrols.

So we're trying to learn with what probability the rangers are going to attack each grid cell. We got 12 years of past poaching data from Uganda. This had like ranger patrol frequency in each grid cell, animal density, distance to roads, rivers, and so forth.

And now we faced one challenge in trying to predict these snares. When the rangers report they found a snare in a grid cell, yes, they found a snare. It's reported. When they say that they didn't find a snare, it could just be that they didn't walk enough in that grid cell. If they just had walked a little bit more, maybe they could have found that snare.

So when we have a positive instance, a snare found, we trust that. When we have a negative instance, we are not sure. And so we have some kind of a positive and unlabeled data set. And so to deal with that, we came up with the ensemble model, which used many different filter data sets.

So for example, one filter data set here says we're not going to trust any negative instances reported where the rangers have walked less than one kilometer in the grid cell, and then learn a classifier for that. In another, we are going to throw out all data where the rangers have reported a negative instance when they walked less than two kilometers per grid cell, and learn the classifier for that.

And so we learn a range of such classifiers, and then in the end, generate an ensemble so that, for example, when, in the test case, you have less than two kilometers walked, use two of these classifiers, and so forth.

So we generated this set of classifiers. We showed our results in the lab are excellent compared to traditional machine learning models, that we achieve higher accuracy in past poaching data. Paper's published, everything is excellent.

But our partners, Wildlife Conservation Society and Uganda Wildlife Authority, are not happy. They want to see an actual test on the ground. So 2016 we did this test. We said, OK, we are going to give you two new nine-square-kilometer areas which were infrequently patrolled where you haven't found snares before. So we're not asking you to go back to where you have found snares or traps before. And so these are the two areas, and these green dots are where we are asking them to patrol.

The red dots where previously snares were found. So we're not asking them to go back to where they've found these snares earlier. And this was done one month before a conference deadline so that if we find snares, we will be able to write a paper. And if there's no snares, there's no paper. So this is a high stakes game for everybody.

And rangers went out to patrol. So every day they would go out and send us an email what happened today. Initially there was nothing. But then came a sad news that they had found a poached elephant with its tusk cut off. So this is sad news. We were too late for this elephant. But at least we were being pointed in the right direction.

Then a whole elephant snare roll was found and removed. So poachers were active in the area. They were killing elephants. But before they could kill the next set of elephants, we were able to remove this elephant snare roll, and hopefully save lives of elephants. And then a whole set of antelope snares were found and removed.

So I had promised my students, because this was this high stakes test, that for every snare found, I would buy them a drink. At this point, they were going, OK, we don't want any more. So our paper shows our hit rate versus historical hit rate and so forth. The main point is that this just started generating confidence that these machine learning approaches are going to help us in trying to find these traps.

But this was a small test. So the next thing was two national parks in Uganda, six months. We made predictions over 24 separate areas, one in Queen Elizabeth, on in Murchison Falls National Park. The rangers were unaware as to which of these areas are high risk and which are low risk.

So we predicted some of these areas are high risk areas. More snares will be found. Some of these will be low risk areas. Less number of snares would be found. And for six months, the rangers went out, and the question was at the end, what would the results be?

And indeed, where we predicted that more snares would be found, high risk areas, more snares were found in Queen Elizabeth. In Murchison Falls there was a high, medium, and low differentiation, and again, our predictions [INAUDIBLE] to work out.

Now these are all areas which were not patrolled heavily, so we are basically going to newer areas and saying, this is how you're going to be able to find snares. So this has now generated further confidence that machine learning-based approaches could be the right approaches in order to try to find snares. But it is my firm belief that we have to continually stress test this technology.

And so to that end, we started working in [INAUDIBLE] Wildlife Sanctuary in Cambodia. So we were there last year, and these are my pictures. So this is with the ranger who was formerly a Meru soldier who has become a ranger. Other pictures from our trip, poachers being arrested. And again, we provided them our predictions on where larger number of snares will be found, high risk versus low risk.

And very gratifying to see all these traps and snares being found based on our predictions. And again, where we made predictions, high number of snares were indeed found. The numbers of snares captured jumped from 2018 fivefold. And so the head of WWF Zero Poaching effort, Rohit Singh, sent me this email saying I'm super excited with the results. Let's get going on to other countries, too. So this was like a really big shot in the arm saying we are going to move ahead now.

And indeed, what's happened now is a collaboration between SMART, which is all these different wildlife conservation agencies coming together, Microsoft's AI for Earth, and us, trying to put this PAWS software, which is the one we have generated. We're making it available to hundreds of national parks around the globe.

So we are doing final testing for this. And in April, we expect this to be released to parks worldwide. And hopefully, this will contribute to allowing us to finding a larger number of traps and to save wildlife. Our next steps there would be then to protect forests and fisheries and others.

So far I've talked to you about this pipeline where we use past data to make predictions. But let's now start looking at real time information. So this is from drones. And so this is work done with an NGO called Air Shepherd. And so they have drones that fly in South Africa that take infrared video at night. The video is sent to a van. There's a human being sitting there looking at that infrared video trying to find poachers in that video.

So you can see they're trying to find late at night if they're human figures who would be, then, poachers. It's very difficult. And so assist them, we developed a system called SPOT that would automatically detect poachers and animals in this scenario. And so SPOT was delivered. So it can be tested to see if there are animals and poachers in the video.

So as we start shipping this and automatically detecting wildlife and automatically detecting poachers, the idea is Air Shepherd will fly their drone. They'll spot the poachers. We'll spot poachers using our software. And now they'll call a ranger saying, come and arrest these poachers.

So what can we do? So our collaborators did something clever. So what they said is, let's do deceptive signaling. So they'll turn on lights at night as they fly their drone and fly towards the poacher to indicate to them rangers are on the way even though there's nobody there.

And so they'll fly-- and this is real life video-- and poachers will run away when they fly towards the poachers. Problem is that if you always try to do this, you always turn on the light and fly towards the poachers when there's nobody there, they'll figure out this is an empty threat. And so we have to be strategic in our deceptive signaling.

And this strategic signaling, security games with optimal deceptive signaling, was the PhD topic of my student [INAUDIBLE]. And so he came up with this idea where the defender knows the pure strategy and the mixed strategy, meaning the defender knows more than the adversary in these situations, and to use that informational advantage to come up with deceptive signaling.

So consider a situation where we have poachers who have been spotted. And now we know that the rangers are available with a probability of 0.3 with 0.7 chance there's no ranger. When the drone flies overhead, it not only knows the probability, but knows actually if there is a ranger or not.

The adversary only knows the probabilities, but doesn't know really what is going on, whether in reality, there is a poacher there or not. So we are going to truthfully signal to the poacher that there is a ranger whenever there's really a ranger there. So when there's a ranger, truthfully signal there is a ranger. When there is no danger with the probability of 0.7, then-- yeah. Yeah.


MILIND TAMBE: It's flying high. It's a fixed wing plane flying very high without sound. So it's hard to attack. But yeah, that would be a threat that we haven't dealt with yet. That would be a good problem to think through.

And so 0.3, a fraction of this time, we are going to now deceptively signal to the poachers that there is rangers coming when there are no rangers. So from the poachers' perspective, half of the time they know they are being lied to. But these probabilities are adjusted in a way so that even though they know that they're being deceived, it's in their best interest to run away, so essentially trying to put some credibility in the threat. And then with the remaining probability, there is no signal.

So it turns out that actually, adding this kind of signaling reduces the complexity of computing the Stackelberg equilibrium. So that's kind of an interesting and surprising computational result. So now, more recently, there's been an extension to this to say we had previously assumed that we have a perfect understanding, meaning we have perfect detection of when the poachers are there.

What if there is imperfection? So sometimes we have false negatives. We can't detect anybody, but maybe the poachers are present. And to deal with that, it turns out we can handle that because when there is a false negative, we can still do this kind of deceptive signaling because we know we didn't see anybody, but the poacher doesn't know whether they were seen or not. And so you use that information asymmetry to overcome some of these limitations.

So there's a recent paper that we wrote at AAAI this year, which investigates overcoming this uncertainty in our detection. There's a lot of Bollywood movies that I know that use this premise, but unfortunately, I can't go into them right now.

Let me now go to the big issue that I mentioned earlier, that is, not enough data. This is a common theme you'll see cutting across the talk. So what this means is we may not learn an accurate enough model of an adversary. And so that means there may be errors in planning patrols on targets.

And to handle that, we have a new approach on game-focused learning. The basic idea is that maximizing learning accuracy is not the same as maximizing decision quality. And so we want to learn to maximize decision quality directly.

So here's how it's supposed to work. So let's go back to our prediction phase. So we collect data. And normally, in the prediction phase in the previous approaches, in sort of a stage-by-stage approach, we would try to maximize accuracy of how valuable the adversaries consider each target to be. And this is shown by the blue shades on each target.

So we spend a lot of our effort essentially trying to get the adversary estimates as exactly right as possible, and then land patrol coverage. The problem, of course, is that in this patrol coverage, perhaps two of these targets have really a large effect on the defender expected utility. The other two targets actually don't.

And what this means is all of the effort spent trying to get the estimates of these other two targets right was wasteful. And so if you could only concentrate on the targets that really mattered and get their estimates right, then our patrols would work out more correctly. And so this is the idea of game-focused learning. So instead of trying to maximize accuracy of the estimates of values of targets of adversaries, we try to, in the learning phase, maximize defender expected utility.

And what this means now is that we get higher defender expected utility, higher decision quality in the end. So it's learning to directly improve decision quality rather than learning just to improve accuracy of what it is that we're trying to learn.

And so for those of you familiar in terms of gradient descent, normally in gradient descent, we are just trying to maximize accuracy in the learning phase. Now, we are going to go all the way to the decision quality, where our game was [INAUDIBLE] to generate a defender expected utility, and pass the gradients back all the way so that what we focus on our targets that matter. What we focus on are concentrating on our learning accuracy on objects that really matter.

So if we look at it in terms of how this impacts our decision quality, on the x-axis is some metric of amount of data available, amount of training data available. Orange line is one when we do stage-by-stage learning. The blue line is when we do this game-focused learning. And what we see is that, as with limited amounts of data, the blue line is higher, meaning we reach a higher decision quality with the same amount of data when we use this game-focused learning because it's focusing its attention directly on trying to maximize decision quality rather than just trying to learn to improve accuracy.

So I'll now switch to the final part of my talk, which is on public health. And so this is work done jointly with Professor Eric Rice of Social Work. And so this was work that we did in Los Angeles, where there's a very large number of people who sleep on the streets, 6,000 homeless youth who sleep on the streets of Los Angeles.

One big challenge is HIV prevention. The rates of HIV amongst homeless youth are 10 times the rate of the normal house populations. So homeless shelters tried to run campaigns to educate youth about HIV. You can't go and talk to each person. And so basic idea is you recruit a few peer leaders, and you educate them about HIV. And idea is that they'll talk to their friends, and their friends will talk to their friends. By word of mouth, information will spread.

Now this is a social network being used, but this is not Facebook. This is just actual people interacting with each other. This is, in computer science, the traditional problem of influence maximization. Given a social network, Graph G choose K peer leaders in order to maximize expected number of influence nodes. So trying to choose these k nodes in order to make sure that largest community of homeless youth gets informed about HIV.

The assumption here is that information spreads. We had this so-called independent cascade model. So basically, you take Youth A, inform him or her about HIV, and there's some probability 0.4 shown here that adjacent Youth B will be informed about HIV.

Immersion in this domain, though, showed us that it's difficult to estimate these probabilities. So we can model this as sampling from some distribution. So if a Youth C gets informed, we sample from some distribution, so what's the chance that Youth D will be informed? And we are unsure of the mean of this distribution, so we can also say we are uncertain about the mean and model this as being from within some interval.

So to solve this problem of influence maximization under uncertainty, we came up with this game theoretic algorithm where our algorithm is playing a game against nature. We are trying to choose a policy. We're trying to choose youth in order to spread information in maximum form. And nature is trying to choose parameter values to get our algorithm to perform as worse as possible.

And the payoff here is some version of regret. So essentially, you can imagine we choose some 12 youth in the network, and nature chooses parameter values so that this selection of [INAUDIBLE] youth is really going to cause us a lot of regret, meaning some other choice would have been better.

So unfortunately, there is a very, very large number of policies, and of course, nature can have a very large range of what kind of parameter values it can set. So this is, again, a massive game. And just like before, we can use iterative algorithms to solve this massive scale game and come up and converge to an equilibrium strategy that works even though it doesn't expand all of these different strategies. So we can solve these massive scale games and converge with an approximation guarantee.

There is a second problem we face, which is that we want to bring in 12 youth and educate them. But we can't bring them all in. We have capacity to only educate, let's say, four at a time. So we'll say, OK, we A,B,C,D, four youth to show up. But on the way to the session, this daylong session, Youth B may get arrested and their friend B Prime may show up. Instead of D, some other Youth E may show up. So we called A,B,C,D. We ended up with some other set of youth.

So when we call the next four youth, we have to take into account who actually showed up in the first round. This is planning under uncertainty. This is a problem that can be cast as a POMDP in AI parlance. And so essentially, this is sequential decision making under uncertainty of which youth are going to show up.

Solving these POMDPs is very difficult in general. But my student [INAUDIBLE], whose PhD thesis this was focused on, he figured out that there's groups of youth who go play basketball together, groups of youth who hang out on Venice Beach together. And so you can divide up this community into smaller subcommunities and solve for each subcommunity separately without significant loss in solution quality. And so with all of this, our HEALER program was built, and now it was time to actually test it.

So we did some initial pilot testing. So this one, HEALER, is our program. HEALER Plus Plus is a second one, a second version of our program. Degree Centrality is the more traditional approach. Bring in the most popular youth. And so having recruited these youth, we selected 12 peer leaders in each case. And our social work colleagues then brought them in according to this staged intervention that I mentioned and educated them about HIV prevention.

And now the question was after a month, who got more information about HIV. So we are interested in the non-peer leaders, the people that we did not bring in, and see how many of them got informed after a month. And the result was in the HEALER, in the AI cases, you can see that 75% of the non-peer leaders got informed. But with Degree Centrality, the more traditional approach, bring in the most popular youth, only 25% people got informed.

So this was promising. But there's one obstacle here. We start out with the understanding that we have the entire social network in hand. Now that means that our social work colleagues had to sit in the homeless shelter trying to do surveys, trying to figure out who's connected to who. And even though it's uncertain-- there's uncertainty that the report-- still, creating that social network is not easy.

So can we work with a small sample of this network? So that was the next task. Could we sample 18% of the youth, do surveys of those youth, and use that to figure out who are the influential youth from the community. And so this sampling algorithm we generated reported in AAAI 18, for example, and then showed that in simulation, it worked well.

But does it work well in practice? Again, we recruited 60 homeless youth, used that algorithm to select 12 peer leaders, and compared the results, and found that with the sampling algorithm, even though it only sampled 18% of the network, we were still able to achieve, in one month time period, the percentage of non-peer leaders, again, matched that of having the entire social network in hand.

So this is promising. We can actually sample a small fraction of the network and work with that to find peer leaders to do influence maximization. And so this then led us to our final test, which was with 900 homeless youth, that was conducted over a couple of years in Los Angeles. And at the end of the study, the idea is not only to find who got more information, but actually did the behavior change.

And so this was done using different metrics. There's a paper coming out about this. So for example, this is the sampling algorithm shown in blue. The orange line is the Degree Centrality arm, which is bringing the most popular youth. The control arm is without any intervention at all. And we can see that there is improvement in behavior based on the intervention that we generated here.


- A beautiful way to kind of like marry this tech world with the social service world, and how we can kind of go deeper and impact young people and elevate them.

- If this group became of a really big thing, it could really help out a lot of youth.


MILIND TAMBE: So more recent work is looking at reinforcement learning to try to achieve the same sampling approach. So essentially, instead of our hand-drawn algorithm, can we learn to sample? And we've shown that this actually leads to improvement in performance as well. This is in simulation.

The final five minutes or so, I'll spend on work that we've been recently doing on preventing tuberculosis. This is work being done in India. As you know, tuberculosis is a disease that kills maximum numbers of people around the world. In India alone, half a million people die every year due to TB. Three million are infected.

And there's two problems we are focusing on. One, non-adherence to TB treatment, and the second, active case finding using social networks. I'll just quickly talk to you about non-adherence TO TB treatments, and then we'll end the lecture.

So the challenge here is that patients are supposed to take their medicine for six months. But often, after a week, two weeks, or maybe a month, they'll drop off of treatment. This is not only bad for them, but it's also bad for others because now you've got drug-resistant bacteria.

And so to try to prevent people from dropping off, there's now this digital adherence tracking technology. So essentially, on this flap you can see some phone numbers. So every time you open, there's a phone number. It's a random phone number. You're supposed to call that phone number, and that way health workers know whether you have taken the medicine today or not.

So on the health worker side, there's a dashboard. And you can see for different patients. 6204 didn't call on day one but called another day. 6214 didn't call on day four and day seven, for example. And so you have times when people called and days when they didn't call.

And so the way this works is if, like, 6231 didn't call for three days, time to go to their home and knock on their door and say, hey, you haven't been taking your medicine. You have to take your medicine to continue this treatment. Our job is if we could predict in advance-- because each caseworker has a caseload of something like 120 patients-- if you could tell them in advance who's at higher risk of dropping off, then that would allow them to focus their attention on those patients.

So if we could predict in advance based on the phone call patterns that 6214 is the highest risk patient, then the health worker could only concentrate on that. And so that's what we've been trying to do. So we've got data, 15,000 patients, 1.5 million calls here. And you may get a pattern like this. Patient one didn't call on day one but called on other days. Patient two didn't call on day three and five, for example.

And if, out of these patient phone calls, if we can predict patient two and four are the one high risk of dropping off, then the health workers can concentrate on those. So based on this data, we're able to show, indeed, that our model increases the true positive rate, reduces the false positive rate.

And so this is something very promising. This is something we are hoping to do a trial this spring. And a memorandum of understanding and other kinds of agreements are being put into place.

So I'll skip some of this work. Bill [INAUDIBLE], who is a co-founder of [INAUDIBLE] and our colleague and collaborator on this work whose data we've been working with, was very gracious in saying that this work has a lot of potential to save lives.

I'll skip some of the remaining slides and go to the final point I want to make. So to summarize here, we've been looking at how to optimize our limited intervention resources. I looked at public safety and security, conservation, and public health as three challenge areas.

We looked at multiagent systems reasoning and data-to-deployment pipeline as some of the unifying themes, as well as the interdisciplinary partnerships that we've built. And there are research contributions here in models and in algorithms in terms of Stackelberg Security Games in game-focused learning, as well as beyond the models and algorithms and actually deploying these solution techniques.

I want to end by talking about a few lessons we have learned in doing this work on AI for Social Good. The most important is that it is possible to simultaneously advance AI research and focus on AI for social good.

Second is that you really have to have a whole data-to-deployment perspective. It's not just about improving algorithms and writing papers. It's important to step out of the lab and into the field in order to do this work. It's important to embrace interdisciplinary partnerships. Computer scientists alone cannot do this work. We have to collaborate with social work colleagues, with conservation scientists, and others.

A lack of data is the norm. And so basically, we have to assume there's going to be lack of data, and it should become part of project strategy. And for AI researchers, the appeal is that AI for social impact needs to be evaluated differently. If we just narrowly focus on algorithms, all we'll get are papers that focus on algorithms.

And so there, we need to expand our sense of how to evaluate this work to include social impact as a part of the evaluation. So with this I'll end, and we're very happy to collaborate here in Boston or in Bangalore to realize AI's is tremendous potential in improving society and in fighting social injustice. So thank you. Thank you for listening. Thank you.


INTERVIEWER: So we have just a couple of minutes for questions, just a couple of questions [INAUDIBLE].

AUDIENCE: Thank you. Was it examined at all, when the drones turn on the light when the rangers are actually arriving, does that impact apprehension rates of poachers, and then potentially impact the long-term effectiveness of anti-poaching efforts?

MILIND TAMBE: So the strategic deception strategy that I outlined is something that we haven't yet implemented. It's a proposal that we are putting forward rather than the one that they're using, which is not going to work in the long run. So it's something to be determined in the future.

AUDIENCE: So when you were talking about your approach, you mentioned the data immersion process is very important in assigning maximum values or predicting outcomes. Did you find out that any of your assumptions were incorrect or misguided? How hard is it for you to almost restart model, and how does that affect deployment?

MILIND TAMBE: So this almost always-- the first model is wrong, and then you have to go and-- so I'll give you a concrete example. We were working with our friends in Malaysia. So we had worked in Uganda to generate patrols for rangers there. And we used the same ideas to generate patrols for our friends in Malaysia.

And so we would generate patrol says that this doesn't work. This doesn't work. And shortest distance between two points in Malaysia is not a straight line. And so we flew from LA to Malaysia and started patrolling with them. And then we suddenly realized what was going on because in Uganda, people could just walk in straight lines. Here is a very hilly terrain, so they really had to walk along riverbeds, along ridge lines.

And so the whole patrol generation strategy had to be very different. And this we couldn't sit there in our labs, s just imagine that this is something that only became apparent once we visited that place. And so that's where often we'll build the first model. There's many, many, many examples. We generated patrols for Sheriff's officers in LA for checking people traveling without tickets on trains. And then patrols would get executed at about half and then stop.

So why is this happening? And then we sent students along, and said, you know, we didn't build in bathroom breaks in the schedules. And so after about half the time, basically, they take a bathroom break. And our schedule is highly optimized. They miss the next train. Doesn't work like that.

So I'll end by saying that we are the first group to have built bathroom breaks in Stackelberg equilibria in game theory.

AUDIENCE: Better outcomes?

MILIND TAMBE: Could you please repeat? Sorry, didn't hear.

AUDIENCE: So with the game theory that you're using, you're choosing, like you said, the model. Are you developing AI to improve the game that you actually choose better models, and try different ones, even that you wouldn't expect, to whatever situation you're looking at?

MILIND TAMBE: So then there is data, like in the poaching situation, indeed, we are learning from that data to continually improve the game. So basically, by learning diversity model we get a better definition of the game, which target areas are more attractive to the poachers. And they change over time. And so more data allows us to build a better model over time.

AUDIENCE: So is that you just changed the model, or is it the model that changes itself, taking that into account? We've built an algorithm that essentially, it just takes the data and says, OK, this is the new model of the adversary. But in a broader sense, the game is something we've defined ourselves, yeah.

INTERVIEWER: OK. One final question here. [INAUDIBLE] right here in the front. Thank you.

AUDIENCE: So looking at a couple different examples that you showed today, one of them involved LAX and essentially fixed sites, so terminals. Another involved the poachers, which included the poachers, and then they were kind of going after areas where these exotic animals would be.

One question I have is thinking about a different situation where, for instance, Marines, soldiers overseas, the adversary sometimes is focused on fixed sights, other times are focused on population centers, and other times are focused on the soldiers and Marines themselves. So there's a bit of interaction there where, as patrol routes change, these IEDs and targeted attacks start to move along with them. So there's a bit of a feedback loop. Has any AI research been done on that?

MILIND TAMBE: So the poacher trap setting and so forth follows a similar set of ideas, basically. So what happens is we'll patrol and they'll shift where they'll target. And then we get new data, and then we'll use that data to come up with better patrols, and they'll shift. And so that has some similarity to what you're saying. So it is possible to adapt to an adaptive adversary.

Adaptation here, though, is slow. It's not, perhaps, as fast as might be happening in a very urban setting where there's a lot of information flowing. But there are some similarities to what we've done in the past there.

INTERVIEWER: So I do see that I have a couple of more minutes. So why don't we take-- yeah.

AUDIENCE: Hi. So this question's less on the technical side of AI, but more like politically. So you do a lot of good work in social. How do local government officials approach it? Not the guys on the ground, but on a higher level. Are they open to it? Are they against it? What's the general reaction?

MILIND TAMBE: So universally, the way things begin is the first meeting is like, everybody's like, we don't need you. We know what to do. Maybe there's one person who's a champion, and the rest of the people are often not very receptive. Many cases, we have seen this, that the first meeting is like, you AI people, we don't need you. We know what to do.

And there's sort of a fear of what is AI bringing? Is it going to take things away, and so on and so on. And so we find that sort of building up this relationship, understanding what the challenges are, and then finally showing that we were building decision aids that can potentially make their tasks better, easier, and so forth leads us often that they themselves become champions of the work that we have done.

And so we ran a competition, AI versus human schedulers, for scheduling patrols on trains on a counter-terrorism exercise in LA. The whole thing began with OK, let's not do this, we don't need AI, to slowly, finally doing this competition.

And at the end of the competition, the AI system had performed better than the human schedulers. But the human scheduling, the leader of that team, went and hugged the postdoc who was doing the AI system, and just saying that this is a great system and he wants it.

And so it's a process. We find that usually it starts out a little bit antagonistically. But then slowly, we seem to kind of arrive at a good understanding. The misunderstanding about what we are doing, what AI is capable of, so forth, gets eliminated. I would say it doesn't always succeed. In some government agencies, we've knocked on their doors many times and it just doesn't work if people are just not moved by whatever. You know, AI, stay outside. And so that has also happened.

AUDIENCE: I'm curious about what the variables are that you used in the HIV prevention use case in selecting who the true peer leaders are, who the true influencers are, given that it wasn't digitally based. It was face-to-face interaction-based, and given that the social workers who theoretically are very embedded in that community and should know with a high level of accuracy who the true, pure leaders are, their choices performed very poorly compared to yours. So I'm wondering what the variables are, how you collected that data, how you validated that data, and PS, is that scalable.

MILIND TAMBE: And so that's an excellent question. So the thing is, we only looked at the network. We don't look at other demographic variables, gender, race, nothing. It's just the social network and where people are located strategically in the social network. And when we did the first pilot test, our colleagues in the homeless shelter were surprised as to who the algorithm had selected as the peer leaders because these were not the more popular youth. They were not in the center of the network.

They were in the corners of the network. But they were strategically positioned in the network. And so even though they were surprised that the algorithm picked them-- even the youth themselves were surprised-- why me? But it performed really well.

And so it seemed like what was going on previously was people were being picked to be peer leaders who were the more popular youth who were all in the center of the network. So they're all very good friends with each other. But that meant that the messages stayed within a small circle, and they didn't go to the other parts of the network.

And so that's where in some sense, we could say that we, as human beings, bring in our own biases. Here, the AI system brought in a more positive bias and allowed the messages to be spread more broadly.

So is this scalable? I guess with the sampling algorithm, we certainly believe it is scalable. We did this test with these 900 youth. So we are hopeful that this can be applied more broadly.

INTERVIEWER: OK. Let's please thank Professor MILIND TAMBE for a great talk. Thanks, all, for coming.



Manuela Veloso: ...to be here and to be able to introduce to Francesca Rossi. So, Francesca Rossi has a -- and you can see on the bio -- that she shares with all of us - has a very -- how do you say -- a deep experience in AI. So, she has been like a researcher in the field of AI for an eternity. And, she also has played a lot of important roles in the AI community. Like, chairing IJCAI -- which is the International Joint Conference on AI -- a very important conference on AI.

And, basically, also being part of AAAI. Another - an Association for the Advancement of AI. So, she’s a very strong presence in the AI world. And more recently - while she was a professor at (unintelligible) -- University of (Unintelligible). And more recently, she actually joined IBM. And, at IBM she has been leading the efforts on understanding this problem of AI and ethics.

Along these same lines, Francesca is also the - part of the founding members of an institution called Partnership on AI that combines Google, and IBM, and Microsoft, and Facebook, and these big tech companies towards having AI being used for the good of society. So, in some sense you are going to listen to a person that I’ve known for many years. And, that will bring to us the experience of knowing about the research and also the community in these new goals of trying to make AI and ethics together.

Francesca has many papers published -- many. She has many honors. She just published a book in Italian I understand about actually, like, getting to people to know what AI’s all about -- the general public. And so, there are many things that you can find more about Francesca. Also, about, like, five years ago - six years ago we were both here in New York - or I was in New York and Francesca was also, I guess, already at IBM. No. You were still in (Padova) and I was here on sabbatical.

And, we have actually -- what we call -- the first - I mean, the New York AI summit. And, we put together like 40 or some people on AI to discuss a lot of issues. So, she’s an excellent speaker. She has a lot of experience. So, let’s please welcome Francesca Rossi and thank her for being here with us.

Francesca Rossi: Thank you. And, I hope that Manuela will join the Partnership on AI. Hi, good morning. Thanks, Manuela, for inviting me here. I’m happy to speak to all of you. If you have any questions, you can stop me at any time. Or, you know, wait until the end -- whatever you like.

So, today I’m going to talk about issues around AI. So, AI -- as you know -- has many capabilities. But, it also has many limitations as in scientific discipline. And also, brings certain concerns and risks. Some of them are legitimate. Some are less in my view legitimate, but they get a lot of space in the media. So, that’s also why I published this book for non-experts. Because, as you know, the non-expert that don’t know anything and they just read about these very extreme views about AI on the media.

You know, like, AI is the most terrible thing that can happen. Or, it’s the most - you know, it can solve all the problems of the world. So, that’s the extreme views that they get. So, instead it’s very important that everybody understand the real capabilities, limitations, and risks and concerns about this technology.

So, today I’m going to talk a little bit about this - very little about capabilities and limitations because I’m sure you know about them. But, also about the concerns and risks. And also, what the community and people are doing in various ways to address these concerns. So, this is just a very, very brief one slide history of AI. Just to make sure that you understand my point of view on AI which is beyond machine learning and beyond deep learning of course.

AI started in the ‘50s and all the term AI has been used for the first time in 1956. And over time, of course, in the first few decades the more successful way of developing AI and deploying it were, you know, algorithms that were telling machines of how to solve problems. But, without the machine having to come up with these algorithms.

And then, in the ‘80s, already this started to be another approach which is the machine learning approach -- which is based on data -- where, you know, you tell - indirectly you tell a machine how to solve a problem by giving examples. Over (unintelligible), you know, by giving examples. So, you’re not telling exactly, you know, the steps to solve a problem.

And, these algorithms were already done in the ‘80s then they evolved, they improved. And, recently, the deep learning approach -- and I’m just writing that and singling it out. But, there are of course many other machine learning approaches. But, the deep learning one is the one that is most visible, popular, and successful in many applications.

But, you know, to give you the idea. That for me, you know, and also coming mostly from the kind of reasoning - kind of artificial intelligence too. For me, you know, there are these -- and I will tell you a little bit more about -- there are these two approaches to AI that more and more there is evidence that need to be combined, you know, that to really overcome the limitations.

So, of course, this - as I said, this machine learning approaches were already there. Not as improved enough but were already done in the ‘80s. But, of course, it did not work well because we did not have enough data to train these algorithms. Enough examples, you know, to make this algorithm learn how to solve a problem. And, even if we had enough data -- which we didn’t have -- we wouldn’t have had enough computing power to just look and handle this data. So, the coexistence now of a lot of data, a lot of computing power, and improved algorithms allows, you know, these machine learning approaches to be very, very successful.

And, the reason why we have now the data is that we produce the data, you know. We produce a lot of data. That was the videos that they watch on YouTube, all the likes, all the pictures. All the things that we share. And, we didn’t have ways to share them with everybody until recently. And so, with this coexistence of this capabilities then these algorithms have very successful.

For example, here there is this emerged net classification error. So, to classify correctly what is in an image -- and this is own data set that has been used for several years as a benchmark to test these algorithms. And, already in 2015 it was - I mean, less - it had the narrow rate which was less than that of a human being.

So, as I said before, you know, so there are these two main paradox. Which, if you think about it, they’re very obvious. You know, how do you teach somebody to solve a problem. You teach it either in two ways. If you know the recipe, you know, the steps to solve the problem. But, you write down this recipe and you code it into a system and then the system from now on knows how to solve the problem. And, this approach -- called the reasoning logical approach -- works well in those problems that are very clear and where the researcher knows exactly that by following these steps for sure you find the solution, or you find the optimal solution and so on.

For example, in the - your, whatever, Google Maps or Navigator that you use, you know, the algorithm is just that kind of algorithm. It’s a kind of reasoning logical (unintelligible).  Called (unintelligible) that is in every textbook of AI.  And it is just a sequence of steps and this machine follows those steps.

For sure we tell you it will obtain the optimal solution from the optimal root to go from where you are to where you want to go.  And the reason why you can do that and it works well is because there are no - there is no vagueness.

The info data is where you are, where you want to go, the maps.  Okay and this is abstracted as a graph which abstracts all the things but they are not relevant to the solution of the problem.  And then on these abstraction you can follow - you can use these algorithms to find the best route. 

Then of course you can say this is a very simplified way of doing it.  Probably the problem because of course you can take it or counter all real time traffic whatever.  Many other things that where you can insert data.  But the basic approach is the typical approach with the reasoning, methodology for AI works well.

Instead there are other problems where you cannot use this approach.  Of course you can use it but it doesn’t work well.  For example, image classification, image interpretation is one of them.

So if you want to build a system that recognizes whether an image is a face of a person.  How are you going to write the steps such that if you follow those steps for sure you have the right correct answer?  That the way the face can be in an image is so varied and so unexpected that you are not able to do that and covering all the possible cases.

So in that case you say, okay I cannot tell you the recipe.  Well I will give you a lot of examples.  I will give you a lot of pictures and I will tell you for each one of these pictures if it is the image of a person or not.

And then you give an algorithm to the machine that allows the machine to learn from these examples and to extrapolate and generalize in a way that it behaves well also in pictures that has never seen in the example.

So those are very obvious ways if you think about it of telling somebody whether it is a person or a machine how to solve the problem.  And as you know the kind of AI we have right now and we have so far is the narrow kind of AI.

So AI that can solve a specific problem very well even maybe better than a human being.  But it is not the kind of AI that can go from one problem to another or learning here and using what they learned over there and so on.

So this approach is typical - these two approaches are for generating the narrow and vertical kind of AI.  So that is why more and more people think that if you want to make AI a little bit broader make it learn horizontally from one problem to another one.  Probably what you need is a combination of these two approaches.

Okay this is another more visual thing, a way of looking at the fact that AI has many sub-disciplines as (unintelligible).  So I said there is machine learning and inside machine learning there is supervised, unsupervised, deep learning, (unintelligible) so many, many variants.  There is symbolic reasoning and knowledge representation, planning, scheduling.  There is a lot of sub-disciplines.

All of these inside the computer (unintelligible) and then of course there is robotics which in the media of course is the usual, you know, representation of AI.  But robotic of course in some sense it is true to make a robot behave, you know, in a way that it can achieve the goal that you have given the robot you need to embed into the robot AI algorithms and methods.

But of course robotics has to do with many other disciplines as well that have to do with the physical part of the robot.  So engineering, control and so on.  So it also goes outside of AI for all that issue - for all those issues.

So AI given all these capabilities that we of solving problems.  As you know is in many, many different applications in our everyday lives.  And here I just put pictures of some of them. 

But of course it is not just providing services to individuals like social media, like checking credit card.  But it also is helping many companies in improving their operations or in creating new way of doing things inside the companies.  Making it more effective, less costly but also really changing the way companies operate.

And given however all these capabilities and all these services that are given to us by AI.  But of course AI also has a lot of limitations.  In fact if you look at it from the application side it looks very successful, very advanced technology.  And it is in terms of applications.

But if you look at it from the discipline, scientific discipline point of view.  In some sense AI is still very primitive.  For example, especially in the machine learning area it is very primitive way that machines learn how to solve the problem from data.  Because what the machine learn is just how to - they learn and input/output relationship.

They learn that given this feature they have to say yes or no to whatever you ask about the picture.  And they learn very well with a very low error rate and so on. 

But they don’t really understand what they are doing and they don’t - they cannot do any other task and any other - they cannot respond to any other question on those pictures or on those pieces of text or on those videos and so on.

So and the reason why AI cannot do that is that it lacks fundamental reasoning capability that we have which is common sense reasoning.  So common sense reasoning is something that we use so much whenever we do things and whenever we interact with each.  And AI still we haven’t understood how to embed that into an AI system.

And again combining learning and reasoning is something that still is starting right now in the recent years.  But for the most part of the history of AI, these approaches were kind of parallel and not really talking to each other.

The third thing I will call natural language understanding.  So of course AI can interpret natural language like Google Translator with almost, you know, very high quality, very few errors.  But it is not understanding that language.  Or Alexa can respond to our questions, you know, our phone can respond to our vocal commands correctly but there is no real understanding of natural language.


And this is reason why you don’t see any dialogue system.  You only see a lot of question answering system.  But then the next question - yes go ahead.

Okay I will give you an example of lack of understanding so maybe we can - so basically if you understand a piece of text for example, of course you can translate it well if you know the other language of course.  But you could also, you should also be able if you understood that piece of text.  You should also be able to answer other question that I have about that text.

While Google Translate is very good at translating but doesn’t have any clue of what that text is here in this language or in the other language and so on.  Yes?

Man: (Unintelligible).

Woman: It could be.  It could be.  It could be.  It could be and there is always these Chinese room argument.  That you know maybe machines don’t need to understand in the way we feel we understand data or we understand text or images and so on.  Or maybe there is a different way which is mostly the input/output things that you observe of the machine.

But still there is a lot of limitation interpreting natural language that is - and the current state is very good for very specific tasks.  But again it is not horizontal.  The machine can handle text but it cannot really - it doesn’t know how to extract information from a piece of text for example.

But then it can reuse also for other tasks.  So more and more broad.  Of course there is no threshold to say okay now it understands and before it was not understanding.  So as you say it is a moving part and it is broader and broader.

Man: Like as far as answering questions about text I think that given Google’s speed I think we are - if we are not there yet we are pretty much there.  But even when that happens I won’t really feel like the computer understands.  And so I just kind of want to…

Woman: It could be but now we are at a very primitive stage I think.  A very primitive stage in my view.  In really being able to handle natural language.  And again the proof is that you cannot have a dialogue with the machine.

And I will give you an example of this lack of, you know, ability to resolve simple things that we resolve when interpreting natural language.  And of course learning from few examples.

Machine learning needs a lot of examples usually supervised machine learning.  And there are situations in which you don’t have a lot of examples.  Like think about a new problem in a company.  If it is a new problem you don’t have a lot of examples. 

So you need to learn how to solve that problem by learning some very general concepts or coding, mixing, coding or learning so very general concepts and properties. 

And then can the machine should use all this information, the domain knowledge, the knowledge of the expert in that field and some data to be able to solve the problem well.  So how do learn from few examples? 

And then of course is over robustness.  Robustness means that yes the machine learning approaches have very low error rate.  Sometimes even lower than human beings but they can be fooled in ways that are very, you know, shocking to us.  And I will show you some examples of that.  So how do we cope with that situation? 

Yes so here is a small example about what I mean that it lacks natural language understanding.  So there is this challenge (unintelligible) challenge it is called that requires the machine to solve a very simple (unintelligible) for our resolution.  Which for us human beings is very, very simple.

So all the - the challenge is like being able to resolve and to understand what a pronoun refers to in sentences that involve the same schema.  And the schema is what you see here in these examples.  So the box - for example, the box does not fit in the suitcase because it was too small or too large?

Now what does that it refer to?  Of course if you write small you know it refers to the suitcase and if you write large you know that it refers to the box.  Okay.  Simple enough resolution.  For us it is very, very simple to solve it.

The best teams from all over the world.  The best universities (unintelligible) accuracy is 60%.  Almost like flipping a coin okay?  So if machines cannot resolve that pronoun (unintelligible) how can they have a dialogue with us?  Because we have a lot of these things when talking to a machine.

So this is just one small way of understanding that still machines are really very primitive in being able to interpret language.

Okay so there are - besides the capabilities and the limitations there are also some risks.  And this comes - the whole area of AI ethics.  So AI ethics is an area that only very recently, you know, came about.  And because of the pervasive applications of AI to our society in many ways at scale.

So people start thinking okay but since technology is so impactful in our lives let’s think about, you know, how is it going to change the way we live?  The way we interact?  The way we work and so on.

So and whether the decisions making by this technology.  Decisions or recommendations the way human beings to make a decision.  Whether these decisions really do have the properties and the values that I would like to see and I expect from a human being now that I give the decision to a machine should I expect the same properties or even more or even less or whatever?

So the AI ethics area has really been picked up by many areas and many groups of people in academia, in companies, in other institutions, in the policy makers, the governments and so on.  Like in Europe for example, every one of the European countries has a national AI strategy.

In Dubai there is a minister for AI.  So even policy makers are really starting to think about this.  So and here you see just how much it grows over the years very recently.

So this is one slide where I just wrote what are the main concepts about - that one could include within AI ethics?  So the first group of concepts is about the properties that you desire of the technology once it is deployed in some applications.

So the first one is bias.  So you would like your machines once they are deployed to make decisions possibly or recommend human beings to make decisions.  And while doing that you would like the property of fairness.  Whatever that means in that specific context.  We know that there are many, many different definitions of fairness. But once you identify the correct one and possibly together with those that are going to be impacted by the decision and so on you would like the machine to make decisions following this property of fairness right? So it means that you should try to detect the machine at some bias and mitigate it, okay? And of course if you use reasoning, logical reasoning approach to where yes in some sense you are coding, you’re telling the machine exactly what to do so it’s explicit if you are coding some on fair way of doing making decision.

But if you are telling the machine how to solve a problem indirectly with examples, with data then these indirect way of telling the machine what to do can - you could inject bias into the machine so by choosing the training data sets or by any other decisions that you make in the design or the development the machine because you’re really not directly telling the machine what to do. So it’s easier to inject bias if you’re not careful.

But it’s not just fairness. One, fairness is just one of the properties that you would like the machine to have once it is deployed. There are many other values or guidelines or code of conducts or other properties that you would like the machine to have. So in general people talk about value alignment meeting that once it is deployed the machine makes decisions or gives recommendation, I want these values to be followed to be aware, the machine to be aware of them and to be followed. One of them is fairness, but there are also other ones. So just for example any business conduct guidelines that you expect from a human being in a certain profession to follow, so if you give this human being a decision support system of course that decision the person should be aware and should be able to follow those same business conduct guidelines just to give you an example.

The third one is the issue of the black box. So these machines may have all the values that you want, all the fairness of the properties that you want but if you cannot be convinced of what values they’re following or why they are making such recommendations or certain decisions then probably you’re not going to use them in the long run, you know, especially if they recommend you decisions that are not just like recommending the next book to buy which okay if you make a mistake it doesn’t really matter. I don’t need to know the motivation why you are recommending me this book. But if you want to make decision that have to do with giving somebody a loan or not or, you know, giving some social benefit to somebody or not or using a certain therapy for a patient or not then of course with this decision becomes very high stake you probably want to have some motivation and some explanation of why certain decisions are suggested by them. So a machine cannot be a black box.

And of course this is an obvious observation yes of course. Like human beings you should explain and also machine so (unintelligible) but unfortunately machine learning matters are kind of opaque in this respect. And so there is a lot of effort in the research community to try to embed explanation capabilities into machine learning approaches. Okay, so these are some properties of the technology.

Then of course there is the issue of data. So since machine learning is so based, fundamentally based on the availability of data then questions arise about what happens to my data? So I am happy to give my data to a machine that gives me a service because I understand that if I give my personal data the machine will give me a more personalized service which is better for me and not just the service which is the same for everybody. So maybe it’s not exactly tailored to me, but on the other hand I start asking the question of what happens to my data once I release them? Who is going to own that data, who is going to pass it to? What are they going to do with the data, just giving me the personalized service that I expect or are they going to use it for - in some other way to manipulate my preferences or, you know, other things?

So the issue of data are very important especially in some parts of the world. And so much for example that in Europe there is already a regulation, the General Data Protection Regulation, that that I mean it have to be question the effectiveness of the regulation but is a regulation that wants to all over Europe wants to give some right to the data subject which is us, you know, the people that are giving the data.

The next one is accountability. So we know that the sum of the statistical base machine learning-based approaches has always is more percentage or error, and this is intrinsic in the fact that they are statistically based. But so then we put them together in the loop with the person and so to mitigate this error and so on but still there could be some incorrect decisions.

So when (unintelligible) correct decision of impacting people in a negative way of harming people who should I ask for, I don’t know compensation or for redress or for something, so especially I don’t know in the judicial system or in the health care system or even in the financial system? So the whole area of accountability and responsibility has been is also from the legal point of view has been built around people making decisions and not about machines or machines plus people whatever, so there is a lot of discussion of how should it be changed in order to account for the fact that some of these decisions are made by machines or teams of machines and human beings.

The next batch of concern is about the impact on society. So the first one is the impact on jobs okay? Are some jobs going to disappear because they’re jobs that can be automated and can be done by AI methods? And even if a job is not going to disappear but probably in every job some tests are going to be automated completely. And the other tasks maybe are not going to automated, they’re going to remain to the human being. But the human being probably in those tasks has to work together with a machine. so it has to be retrained, it has to learn how to interpret what comes out of the machine to make his own decisions and so in general every job will be transformed. So how do we cope with this transformation, how do we help people to cope with this reformation?

When there was an Industrial Revolution things were happening because there were physical objects that were built in the factories. Things would happen with a certain time, a certain scale of time. And so we had the time, the society had the time to put together education, educational initiative for everybody so that everybody could take part of the integration. Now things are happening very fast so how do we retrain and help everybody being part of this reformation and take advantage of it?

And of course there is the impact on society meaning how we live, how we interact with each other, how we interact with machines, what happens with our children they grow up with thinking that Alexa is like their friend, you know, preferring to talk to Alexei with Siri than other kids on the phone or even, you know, the interactions, the addiction to the social media. So how is - that the transformation of our lives that we really want in the longer term or not?

And then there is this issue of the deep fake, you know? So yes right now it’s very good not only at interpreting images, videos and text but also at generating new content, new images, new videos and that in a way that is practically indistinguishable from real meaning generated by a human being or by a camera or whatever in terms in the case of an image.

So if we cannot distinguish what is a real image like a real face or a real person or who talks with which voice and whatever, if we cannot distinguish what is real and what is not real, all - many of our systems were on which society is really based on like for example the legal system, has to be rethought. Because in few, even now probably or in a few years we cannot, we will not be able to use an image as a proof that something really happened or a video as a proof that something really happened. So how are we going to deal with that in a legal system just to give you an example of something that is so fundamental to have to give rules for a collective society to work well if we cannot really distinguish what is real and what is not real?

The next one is all about the users of the technology. Also people say that AI is a dual, now dual use technology. In fact AI is many - the same technology the same algorithm, has many different possible uses. So it has to be a very conscious decisions about which users we want to have. And we cannot just say okay this is a technology as a new technology now people can do good and bad things like a hammer that you can use for good things or for bad things. We have to be more conscious about what are the possible uses but also what are the desirable uses and the non-acceptable uses of the technology. So as I think about the, you know, autonomous weapons or I think of about large at scale surveillance systems of society and so on.

And then there are these long-term concerns about the singularity and the super intelligence which has to do with the concern that as machines become more and more capable of solving problems and even more in horizontal way, you know, like in a broader way then not the way they have now then are we sure that we can stay in control of these machines?

And we - if we lose control of these machines because they have become so capable of finding resources that - by themselves, solving problems and some, how can we make sure that then we are not left like as a marginal species if you’re not in control of what they’re doing? And of course this has to do also with the issue that I mentioned at the beginning of value alignment. You know, more - the more machines become capable the more there is the need to make sure that they are aligned to our values because otherwise they can do very damaging things if they are not aligned to our values.

So let me give you also here I just say that many of these concerns and these initiatives to solve these concerns and to address them are around this notion of trust. So how can we build this technology and use it in a way that we can build a system of trust for the technology so that comes to properties that we want to assure in the technology but also trust to work the people that are building and delivering the technology and also trust to work the people that are going to regulate and to give rules to use this technology? So the issue of trust it seems very central to many of these initiatives.

So at IBM for example we focus on this IBM research but then also the business unit we focus on this idea of trust a lot. And we say well what does it take to trust the decision made by a machine beside the fact that we want very low error rate right, that’s yes? But besides that what do we need? Well we need that is fair, aligned to our values as we said that is easy to understand why it makes a certain decision that is robot and that is accountable meaning that it’s transparent the capabilities and limitation. So that’s why we have this four pillars around this vision for trust at AI.

So I’ll just give you an example of bias and how it can be easily injected even without wanting into a machine. So bias of course is a prejudice not in a human being, a prejudice that can lead us to make decisions that are not fair. And of course the reason why AI can be biased is because we train, we design the AI systems. And since human beings like you see here have many different kinds of biases for human beings, of course it is very easy for us to not even in an intended way but to inject some of these biases into the design and the development of the machine.

Here it give just an example of how it - the bias in the training data gets inherited by the decisions of the machines. So here I took just Google translate. I took two sentences in English, she’s a nurse, she’s a doctor. I got that translated into Turkish because by doing that the machine lost the gender information because Turkish doesn’t have the gender. You see that it’s just a big O for E for he or she, doesn’t have the he or the she. So then at that point I asked the machine to translate it back to a language, any language into English yes, that wants the gender.

So in that second step of translation the machine had to guess the gender because it had lost it okay? And where does the machine go to get the gender, well it goes into, well it extrapolates and generalize from the training data sets. So if the training data set has that most nurses are women and most doctors are men of course the machine will say well, you know, to be as correct, to minimize my error, I better say that the nurse is a woman and the doctor is a man.  So this means that, of course, this is not a high stakes decision.  Just, you know, hey you made a mistake in translating, but you can imagine.  So, in general in high stakes decisions scenarios, so what we put in the training data set can get, you know, inherited by the decision making of the machine.

So, for example, if you train a machine that has to decide whether a loan request is accepted or not, and you put together a lot of data meaning a loan request and then yes or no, whether it should be accepted or not.  So you are not careful about this data that you are collecting and, for example, all the loan request that you say should be accepted by chance, because of the data that you collected, they all come from women, and all the ones that you say you should be rejected they all come from men.  Well the machine will get that correlation between the gender of the person and whether it should be accepted or not.  You didn't see it probably.  You just said, okay, these are all the data I have, for examples, of loans that are request that should be accepted or not, but the machine will pick up that correlation, and then it will use it probably when making its own decisions.

So now, the question is, but all possible human beings are bias.  So now if we replace or we help human beings with machines, and made machines whereby, is it better this is worse?  Well, human beings, ys, they are bias.  We have a lot of bias as I showed you before, but of course is easier to detect and mitigate bias in a machine because you can analyze the training data set for example, or you can do some tests on the machine.  It's more difficult to detect it in and to mitigate it and to correct it in the mind of a person.  So in some sense it is better that the machine is bias rather than a human being.

On the other hand, a human being that is bias is one human being.  It's making a lot of decision, but it's just one human being.  It doesn't replicate automatically to many others.  While the machine, if it is bias, then everybody can buy that same machine.  So then every loan request, you know, systems are going to be bias.  So this is an issue of scale, of course.

Okay, value alignment, as I told you value alignment has to do with the fact that you want the machines to solve a specific problem, but at the same time you want it to do it while following some guidelines and some code of conduct.  Here is like a funny list of examples of where you need alignment.  They're mostly on their own domain, and they show that even if the developer didn't want the machines to do certain things, but using the machine learning approach, and just told the machine what to do by giving examples and so on, and other machine learning approaches.  But if you do that, of course, the good thing is that the machine has some freedom, some space for creativity to solve the problem in a way that maybe is even something that we could not come up with.

But on the other hand, the machine can also do things that you do not expect in a negative way.  So for example, here you see an elite school game playing agent that, of course, was supposed to learn to play well the game and therefore to go very high in the list of names and scores in all likelihood when you play a game, and it understands just half that list.  So it didn't learn to play the game at all, it just have that list because he understood that that was the final goal.  And of course, that's not what the programmer, the developer, wanted.

Or another one, a sorting program, that was sorting a list of numbers that was always out putting the empty list.  By definition, the empty list is always (unintelligible).  Or a game playing agent that was killing itself at the end of level one to avoid losing at level two because, of course, it knew that losing at level two or a higher level would have a very negative reward or signal or score, and many others.  So there is a very much longer list at that URL.

But so to tell you that sometimes, you know, this seems like, oh, how come it behave like that, you know?  But of course those programmers did not think that the machine could do something like that.  They thought that they had given the machine the information to solve a specific problem without doing these unusually things.  So it's important to be able to understand how to bound the freedom of the machine, not completely because you want the machine to be free to find good solutions, but to bound it in a way that does not do undesired things.  So that's the value alignment issue, the value alignment task.

Okay, explainability, as I said, is another important thing, you know.  So if you don't have explanations, probably a one track - the decision of the machine.  And explainability as compared to buyers and sellers has one additional complication.  There's, of course, the explanation has to be understood by whoever receives that explanation.  So for the same decision you may want to produce several different explanations.  Like in the healthcare system you want to use - when you have a decision support system that recommends a septum therapy.  You want to give a certain explanation to the doctor using the terminology, technical, that the doctor understand.  Technical in sense of healthcare terminology that the doctor understand.  You want to give a different explanation to the patient, a difference explanation to the developer that may use the definition as a debugging tool, okay, to understand why the machine has made that decision.

And you also may want in, again, very high stakes domains like healthcare, judicial system, and you may also want to give - need to give an explanation to the lead - to the whatever is going to judge that decision if somebody is asking for a redress, so to the regulatory body that is looking on those decisions.  So it's going to be that this additional problem and the issue of really being - as need to be tailored to the recipients of the explanation.  And of course, there are many, many approaches to explainability, but still I think we are very at the beginning of this.


So (unintelligible), as I said, it's easy - so machines are very small.  The data deployed are a very small error rate, but they can be fooled the easily.  What does it mean easily?  Fooled easily in the sense that - let's take images in image interpretation.  Even images like this, sir, just for a few pixels, one from the other one, which to us human beings, they appear exactly the same, okay?  Machines can interpret them in various different ways.

So, for example, the ones on top, only the images of an ostrich, only the one on the left is the original one and these are correctly identified as an ostrich.  The other ones, they look the same to us, but actually they're different for a few pixel here and there, and they are interpreted incorrectly by the machine.  One is a safe, one is a shoe shop, one is a vacuum, or whatever.

The same, I mean, this is kind of a funny one, but the second, the lower part of the (unintelligible), shows you something that is more concerning.  Because the images like the box called with A, that's the original imagine of a street with some pedestrians and so on, and the one to the right, the prediction, the B, is the correct interpretation.  So the machine correctly identified the pedestrians and the streets and the buildings and so on.

But then the one below A, so the C, there is a final example, is the same as the imagine, the original imagine, but with few pixels changed.  And now the last is exactly the same, you will still just see the pedestrians.  We still see the street.  We see that, but the machine does not see the pedestrians anymore. 

So, you see that, you know, you can imagine that this is a very area where people are very concerned.  For example, in this case for self-driving car but also in many other situations, and it's not yet clear how to solve this problem.  So, even though machines may have like, only 3% of error rate, but these errors are very strange because they make errors in a way in things that to us they're, you know, very easy, and so it's not easy to understand.  So far we didn't understand because, in fact, the community didn't understand how to solve this problem well.

The last issue is about this transparency meaning - okay, so suppose I take care of explainability, I take care of robustness, I take care of families and value alignment.  But then I deploy the system with all these properties, and then how do I make sure that it's used in the most appropriate way, and that whoever is using it, or deploying it if it's a company, it can understand what are the real capabilities, limitations, and risks of this piece of AI.  So it's very important to be transparent about these three things, the capabilities, the limitations, and the risks.

And the idea that we came up with at IBM about - there are many other similar ideas in other companies, as we call it the fact sheet.  So it's a list of questions that during design and development that needs to be answered about all the design choices that have been made.  What kind of bias did you detect?  What kind of mitigation algorithm for bias did you use?  What kind of databases did you use to test that?  What form of explain ability to do interact and so on.  All the different - and also what kind of error or landscape that machine has that it has a lot of false positives or lead a lot of false negatives?  Because, of course, this makes a difference when you want to understand what the appropriate or non-appropriate uses of an AI system.

For example, if you have phrase recognition system with a lot of false positives, all right, it's okay to use it if you're looking for a missing child.  Because the worst that can happen is that you stop more children than you should have stopped, but it's not okay to use it to incriminate people, of course.  So you see that the same piece of AI with the same capabilities and limitations than the error rate, it can be very appropriate to use it in a separate course, but not in another context.  So it's very important to have these transparencies not in the sense of releasing the data sets and the algorithm things that are almost always proprietary, but instead determine the terms of documenting the design choices that allow whoever is going to be on the receiving side of this AI system to understand the capabilities, the limitations that there is.

There are many things that are done in researchers, technological solutions to address bias, sustainability, security, or (unintelligible) and so on, but there are also many other tools that people use, like best practices, guidelines, certificate, standards, auditing systems, and so on and even laws and regulations.  And there are a lot of enablers like education, educating the students, educating the developers, educating the policymakers, and also dissemination and also governance.  What is the right governance approach for a country that wants to be able to use AI in its, for example, public sector or - and what is the right internal governance for a company?  What are the relevant issues?

What are the relevant AI issues for a specific company?  And how can that company address those AI issues within the company?  What are the pieces that he has to put in place?  An AI board maybe, or maybe a protocol so that the business units can tell the board what they are doing and what they are trying to deploy, and whether to understand whether that's risky or not in terms of the (unintelligible)?

How can research innovation, so like the technological solutions, be injected easily into the business units?  How can we make sure that the business units, once we have - a company has the right, for example, bias detection algorithm, how can we make sure that the business unit use it?  How can they have the right incentives to spend time in doing this thing?  So rather than just deploying it as fast as possible.

All these approaches, all these initiatives, it's important to understand if it goes back, well, it doesn't matter, that they are all multidisciplinary.  So the AI people alone, or technological people alone, cannot address, cannot even identify the issues by themselves.  So you need sociologists, you need philosophers, you need psychologist, you need people that are experts in - from outside looking at the technology and understanding what's the impact on society to be able to raise issues.  And then the AI people can find possibly the technological solution, but you need to be able to engage with all these aspects in other disciplines, as well as the policymakers.

Education, of course is very important.  Here is just a very initial part of a very long list of courses that there are in academia.  This is a crowd sources list of AI ethics and computer science ethics courses all over the world, mostly in the US, but also in other parts.  Everybody can add new courses if they know about another course, but it's very, very long.  So several weeks ago when I wrote - when I took this screenshot, it has more than 200 entries.  So probably now it has many more, but so you see that more and more academia feels that there is the need to educate AI and technological people, engineers, to know how to advance the capabilities of the AI, but at the same time, be aware of these issues around the ethics.

Okay, at IBM, and in many other companies, we do a lot of things that try to address these many concerns.  The ones at the top, you see these AI fairness critique, (unintelligible) robustness, AI explainability.  So there are three open source toolkits that everybody can use to add the code, the data sets, the algorithms, tutorials, demos, around these three fundamental issues around the properties of the technology.  One of them is fairness, one of them is robustness, and one of them is explainability, and there are DI's to help the research community to get together into an open source place where everybody can share knowledge and results about how to address that issue and to better understand it.

But then also we have, for example, the blue one in the middle is called, Everyday Active for AI, which is a booklet that we have built, and we have written for our developers, but it's available for everybody, to really remember in their everyday job all these main issues.  However, there is the possibility of unconsciously injecting bias and all the - and to remember to try to inject explainability wherever they can and thoughtful, really something to get our developers aware of the issues, and then the AI proxy that I already mentioned.  But there are also many other initiatives that we are involved externally that in a very more multidisciplinary way and each one with its own specificity, try to address this issue.  Somebody building certificate, somebody building guidelines, somebody build a list of principles.

So, for example, here in the top left there are Asilomar AI Principles by the Future of Life Institute, and those were made in - were written in 2017 by a very multidisciplinary group of people that started with more than 100 principles for the beneficial user AI, and then they voted the principles, and those 23 were the ones that unanimously people in that group decided.  And, for example, those are a very high-level principle.  They're not very specific and concrete, but they have been picked up, for example, the State of California has adopted formally these principles over the State of California as principles that every AI system should follow.

The one in the middle, IEEE, the book, a very big book that tries to help developers, the engineers.  IEEE is the Worldwide Association of Engineers.  So it tries to help engineers to know more about the issues and how to solve.  So possible recommendation on how to solve the AI issue.

The one in the top right is (unintelligible), the Partnership on AI, that was founded in 2017 by six companies, IBM, Facebook, Apple, Google, Amazon, and Microsoft, okay, those six companies in 2017, with the idea to have a shared place, a common place, to share best practices, to define best practices about the beneficial deployment in user AI.  So we got six companies in 2017, and now we are more than 100 partners of which less than half are companies and the rest are everybody else.   Basically UN agencies, academia, advocacy groups, ACLU, and other - yes, you have a question?

Woman: (Unintelligible).

Woman: Yes, so there are more - we expanded to a company.  You know, as I said, we are about 100 partners and the companies are much more than 60, or almost half of the hundred, and there are companies also in Europe, in Asia, like for example, there is Baidu, there is Saudi, and then several other ones in Europe, but the idea, the intention, is to have a very wide coverage geographically, but also coverage in terms of the roles of the company.  So, those that deliver AI, those that use AI, and those that are impacted by the AI solution.  So all of the sector, so in automotive, the healthcare, you know.  So we still are expanding to reach these coverages in these various dimensions, but one of them, of course, the geographical regions.

Then the one at the bottom, these two ones with the (unintelligible) are the two deliverables to report of the - it's called the High-Level Expert Group on AI that the European Commission put together.  This is an independent group of people that the European Commission nominated, 52 people, very multi-stakeholder, multi-disciplinary.  And in these - in 52 people, the AI experts are definitely less than 10, okay, and then there is everybody else, consumer rights advocates, lawyers, sociologists, philosophers, academia, and companies, businesspeople, and so on.

So those two reports are, one, the blue one, is the ethical guidelines for AI in Europe.  And there we start from human rights, we go to principles, and then we go to seven requirements that we think AI in Europe should have, and we call it Trustworthy AI.  So again, the notion of class accounts for it.  And these seven requirements are similar to the things that I told you.  That's should be the data issue treated in the right way, the fairness, explainability, the accountability, and so many others.

And then the green one is saying later on we published also the green one, and it says, well, okay, now that we know that we want to have trustworthy AI, now these are the recommendations that we give to the European Commission to make sure that it gives the right incentives and policies to support a trustworthy AI in Europe.  So the idea is that to help the policymakers understand how they can facilitate the development of that kind of AI in that part of the world.  And all though that think is in that part of the world, so not here, not in China, no, but I think that there are several elements and also in the content of the document that can be useful in other parts of the world.

Another thing that I wanted to mention is the one in the bottom left, which is the a new conference that we started, like, two years ago.  It's called AI Ethics and Society because scientific conferences also are very siloed, very multidisciplinary. 

So there are conferences in AI, of course, there are conferences on ethics, there are conference on others, but there was not one conference that could put together aspects of the (unintelligible) list, talking to each other, learning from each other, submitting papers that are evaluated as usual in computer science, you know, and a subset of them accepted and then presented, and then exchange opinion and so on.  So that one is being supported by AAAI, which is the worldwide artificial, and ACM, which is the worldwide Computer Science Association, and to really fill that gap and have a multidisciplinary AI ethics conference.

So these are the slides that I wanted to show, so I will stop here and wait for questions you have.  Yes?

Man: (Unintelligible).

Woman: Yes?

Man: Thank you, (unintelligible).

So I'm thinking back at the beginning of the talk about the main concern slide that summarizes a bit of the risk.  So I think there was bias, and I think all of the concerns, in fact, your whole talk applies to biological intelligence to human biases, and you mentioned it at some point humans are biased because of education, lack of data, and the sampling of trauma.

Woman: Yes.  Yes.

Man: They have questionable values, including elected people have questionable values, and I think all the least of the things, except (unintelligible), applies to humans, and no one is, kind of, unmatched with them at this point, I think.  So I'm wondering why some countries have a Minister of Artificial Intelligence and no country…

Woman: Just one.

Man: Yes, one.  Everyone is freaking out about option in intelligence in those subjects, and why is no one having those discussions, and the Minister of Biology Intelligence, which today is making all the decisions?  So my question isn't the fact that we are Christianizing artificial intelligence, but does it make it obviously better than human intelligence just because we are looking at those concerns now?

Woman: (Unintelligible).

Man: We should have had that for human experience hundreds of years ago.

Woman: Yes.  Well, I mean, in some sense, as I said, you know, in some sense, the systems that we put in place as a society with values means various laws, you know, systems in various parts of the world.  They are intended to address these concerns for human beings, like some lows.  Now, like for example, it is not illegal to discriminate when you hire somebody on gender or whatever, you know, age or whatever.  So there are laws, but all these laws they take into account the fact that it's a human being issues, right?

So, now that we are going to work more and more with machines, some of this you need to have the same process also about machines, okay, because machines are going to make decisions, make (unintelligible).  So, just like you have - you had concerns and you decided about the regulations to add to the concerns with human beings, you also now do the same with machines.

Also because even if machines are not going to be autonomous when making decisions, but however to work with human beings, if I am a doctor and the machine, I don't even know whether it is biased or not, whether it gives me random suggestions or not.  I mean, I need to understand why is it telling me something?  What values does it have?  Does it know that I have my Hippocratic Oath to follow?  And does it fall of that as well?  So if we don't scrutinize these machines even more than we scrutinize human beings, then the human beings are not going to adopt whatever good decisions the machines may make.

So in some sense when you want to build a team between the machine and the human being, you need to scrutinize more because when it is a team between two people, you know, I can maybe - I have never met you before, but I know that we share a lot.  We share, for example, the same experience of the world, the same kind of common-sense reasoning.  Maybe we don't share the values.  Maybe, you know, some values we don't share, but I know that there are many things that we share, the same kind of education, and the same - not the same knowledge but the same environment where we listen to things and so on.

So with the machine I don't have that.  The machine can be completely, you know, can come from Mars, is somebody from Mars, and I need to scrutinize it more than I need to scrutinize a new team member.  You know, of course, when a team member in his new, you asked more questions probably then at the beginning then later on, and the same maybe it would be with the machine, but the machine even more because you don't know what to share or what you don't share with the machine.  So I understand it is true that the bar for scrutinizing the machines is higher than for human beings, yes.


Man: Given the differences that exists geographically, politically, socially, and economically between many of the major players within the AI space…

Woman: Yes?  Yes?

Man: …what are your thoughts on whether we'll be able to reach some sort of global consensus in terms of deployment and utilization of AI?  And who do you think would enforce something like that?  Would it be the UN or some sort of other NGO?

Woman: Well, there are many worldwide initiatives.  For example, there is an initiative that piloted by Canada and France to build an international panel of AI similar to the international pioneers of climate change, which is worldwide.  Similar, but not really similar because the issues are different and the fundamental, you know, the knowledge is different, but the idea to build something worldwide that can be scrapped and be factual about the capabilities of these technologies and their risks.

It doesn't mean that there will be solutions that are applicable to the same solution everywhere in the world.  We know that different parts of the world have different priorities over the values.  Like in some parts of the world, individual privacy is much more important than stability of the collectivity.  In other parts of the world it's completely the opposite because, you know, it's very easy to judge very negatively what happens in China with AI, but my colleagues in China, they say, well for us, the individual privacy is not the most priority - the highest priority value.  So size and cohesion and harmony is more important.  So for them, all though they see it as attempt, you know, to regulate and to surveil and so on, but they don't see it as clear cut as I was to say, oh, my god, they're using - they're surveilling everybody, they're having this social squad and so on.

So we know that different parts of the world have different priorities over the value.  So probably the same exact solution will not be overall, but it's important that they all talk and discuss and factually they understand the same thing.  Because just like, you know, in the climate change, but possibly even more successfully than the climate change, probably.


Woman: Firstly, I'm ambivalent about self-driving cars and particularly given the example you used, I'm very interested to hear your take on the future and the accountability of it.

Woman: Yes, well for me, self-driving car - really self-driving cars level five, you know, in all of the autonomy are going to be deployed very very far in the future, okay, but in the meantime all the work on self-driving cars is advancing AI, many kinds of AI because self-driving car is really needs a lot of different kinds of AI.  It needs to look at the pedestrians, it needs to understand the videos, images, anticipate and predict what people will do.  So it needs a lot of advancement in a way.

So if all these technologies are going to be using many other things, with maybe not self-driving car, and they're going to be used more and more in assisting our human driving, you know.  And they are already used like that, you know, like to keep the lane, to see an obstacle that you didn't see at the last moment, and so on, to keep the distance with the next car as well.  So more and more already the overall goal, which is to - one of the goals of several goals, which is to save a lot of lives, is achieved by assisting us in helping us when we have distractions, when we fall asleep, or whatever, you know.  So in some sense we will see - I think that we will see a lot more of this in cars than maybe not just the high-end the cars but in every car, and this is already, you know, will achieve a lot.

And then the real full autonomy, another difficulty besides the legal accountability and such, another difficulty is the fact that when we drive, we use a lot of visual communication with other drivers, like at the stop sign, for example.  And so we need to understand how to do that with self-driving cars.  If self-driving cars are going to be in the streets together with human driven cars, and that transition is going to be difficult much more than a situation where there are only self-driving cars, and then they can communicate with each other in the, you know, in other ways, which are not visual, you know, and maybe they can, you know, regulate each other in the traffic and so on because of their ways of communicating and interacting.  So to me the level five autonomous self-driving cars are very far, but it doesn't mean that many of the goals of self-driving cars cannot be reached and even in the way to going there.


Man: I was wondering, when you spoke, you spoke about these sort of coalition's and panels to think about and discuss AI, and I just wonder how relevant these things will be because it seems, like from what I see from the media, it seems like society is so money driven, and the ethics now or so much like, if it's legal, I can do it.  I just wonder what relevance will these panels have, and also would you be able to provide some example, or examples, of a company, which voluntarily said, we won't do this AI related thing because we feel it doesn't correspond with our values or it's not ethical.

Woman: Yes.  So, okay.  So about the relevance of these initiatives, well, I mean, some of them are very visible.  Some of them, like the one put together by the European Commission, is going to - I think it's going to be relevant with the new European - well now there is a new European Commission that has been voted in the summer.  So the idea is that they are going to take some of our recommendation, the recommendation of the High-Level Expert Group, in a year, to really make sure.  And in Europe is the kind of place that this, of course, can happen easily because of this - compared to US, Europe is more regulatory based, no, rather than a bottom up innovation, and then we'll see if things go wrong or not, which are here - which has pros and cons, of course.  And so I think that that initiative is going to be relevant for European - the way Europe is going to deploy, develop, and use AI.

The Partnership on the A is a visible organization.  It has several meetings with the policymakers in Washington.  So it is being visible and, for example, one report that the Partnership of AI I recently published about 10 recommendations and 10 risks and concerns for using the AI industry in the US judicial system for assessing risk of recidivism, like the callback algorithm is one.  So that report has been very, very scrutinized and received and read and discussed by policymakers as well.

Then in terms of companies, well, you know, many companies.  I mean, at IBM the whole AI trust initiative inject some explainability and the bias detection mechanism into the - even the platform that we developed, and we deliver to our clients.  And of course, this takes time, which you cannot deploy something else.  So in some sense, resources are devoted and - oh, you can see other companies like Google or - no, Google you will see, you know, what - they had some use of technology that was not considered acceptable by the people in the - some people in the company, and the company changed its mind.

So more and more companies, in general, I think that they understand it's not just to protect the brand of the company, which means money.  If you just think that this is the only thing that company cares, which I don't think, but even if you think that money is the only thing that company cares about, to protect their business model and their brand, they understand that be following and being compliant to the law is not enough.  Because if your clients lose trust in you, then it's very difficult to regain that trust and you're going to lose your business value just because - even if you were compliant to the law.

So that thing is something that companies more aware of, and their clients are telling them, look, how are you going to, you know, you give me this thing, but did you check some bias?  Because I heard that, you know, that the bias can, you know, put me into trouble.  So, did you try for that?  You know, what kind of bias do you have that?  So more and more people are understanding in companies that the AI ethics, which is beyond being compliant to the law is useful for protecting the company.

So being compliant to the law is not enough.  Because if you lose trust, then it's very difficult to regain it.  Especially in an environment where there are many competitors.  There are many cases in which there are no competitors, like, social media, you know, what are - okay, some people go now to Facebook where, you know, who do you know, you know, if you like this and you use Facebook, you connect with others, you do many things you keep track of many.

But if you don't work with IBM or their other companies, again, do similar services to you as a company.  So it's very important in a competitive environment that you cannot lose trust because trust is a competitive advantage if you are trusted and there are other companies not trusted.  So you can lose it, then it's really very damaging for the company.

Man: You've given us a lot to think about and work towards.  So thank you so much, and once again a hand for (unintelligible).


Coordinator: And that concludes today's conference.  Thank you everyone for participating.  You may now disconnect.


MANUELA VELOSO: Thank you very much for coming to this distinguished lecture-- JP Morgan distinguished lecture on AI-- one more in our series, and a very special guest we have today. So Professor Jeannette Wing has many honors. In the invitation, you've seen all the honors she has. I'm not going to say all the boards in which she is-- all the honors that she has, but I will tell you one thing though that you should appreciate.

Professor Wing, she did not write in her biography in a little bio, but she actually introduced to the world the concept of computational thinking. She was a professor at Carnegie Mellon for many years. When I came to Carnegie Mellon, Professor Wing was actually in this faculty, and I was still PhD student. And then we were colleagues for a long time. And during that time, there was this-- from an educational point of view, this kind of thinking that math, and reading, and liberal arts education, it was not going to cover everything that people needed to know.

And we were in this computer science department. And then Jeannette one day did come up with this concept of the world, and our students, and the outreach-- I mean, and the K through 12-- everybody should start thinking computationally. And I believe you gave many talks. And she really wrote great articles. So the homework number one today is to really Google computational thinking, and you are going to see the tremendous impact that such concept has had, started by Jeannette.

So that's, I think, the thing that I wanted really to get across. And then Jeannette now is the head of-- the director of Data Science Center at-- in fact, of the inaugural director of the Data Science Center at Columbia University. She was the head of all research at Microsoft-- at the Microsoft Research, where in fact, I was invited to give a talk by Jeannette there. And we actually had a lot of interactions with Microsoft Research.

And before that also, it's important for you also to know that Professor Wing was extremely influential also by heading NSF-- an NSF division for funding efforts. And why is this so important? It's because she gained this perspective of everything that was going on, in terms of research in computer science, in AI, in systems, in programming language and verification. So she has this very, very large spectrum of knowledge and now focus in data science.

This is important for you to know. She's probably going to give a great talk on data for good, and introducing her science of Columbia, but at the end, if you have questions that cover this wide spectrum of Professor Wing, feel free to ask her because she's really a genius. Professor Wing, it's a pleasure to have you here as a friend and as a researcher here.


JEANNETTE WING: Well, that is some introduction. Dr. Veloso, is that what I call you?


JEANNETTE WING: Professor Veloso, OK. I've never heard Manuela call me Professor Wing. We do go back many years on-- she was still a graduate student at Carnegie Mellon, and I have watched her career just skyrocket. And now, she's at JP Morgan doing phenomenally, not surprisingly, and still coming up with really out-of-the-box thinking. So JP Morgan is really lucky to have her.

In fact, I would say, New York City is very lucky to have someone like Manuela here, because I have-- now that I'm in New York and at Columbia, I'm trying to promote the tech industry in New York. And to have someone like Manuela for me to point to and say, look, JP Morgan Stole her from Carnegie Mellon, is a really important message. So I'm going to talk about data for good, data science at Columbia University. But as-- I'll just say Manuela-- as Manuela said, because of my experience in industry at Microsoft Research and in government at the National Science Foundation, and of course, in academia at Carnegie Mellon and here at Columbia, I'm more than happy to answer questions on almost anything. So please feel free to ask.

So what I wanted to start out doing was putting in context how I think of data. So I'm going to talk about the data lifecycle. It starts on the very left with the generation of data. Now, in science, we've been generating lots and lots of data for many, many years. If you think about large scientific instruments like Large Hadron Collider, telescopes in Chile, neutrino detectors in the south pole, these one-off very expensive scientific instruments have been generating and continue to generate volumes and volumes of data.

And so the sciences have been dealing with lots and lots of data all the time, and I think the finance industry is the same. What's new to the universe is, more recently, that we, people, are generating lots and lots of data. Of course, we've always done that, but through our digital devices and our interaction with the digital world, we are generating lots and lots of data. It's these, if you will, big companies that are collecting all this data about us, to the point that others know more about us than we know about ourselves.

So generation of data, then we collect the data. We don't always collect all the data we generate. We process the data. Under processing, I include encryption, compression, but also the less sexy things of data wrangling, and data cleaning, and so on. Then we actually store the data in some medium. Then we usually store it in a way that we can retrieve it quickly, access it quickly. So that's where all the database data management aspects of data come into play.

Then there's data analysis, and I think this is what is-- it's AI and machine learning that people usually identify with data science. It's this data analysis phase, which really is where the machine learning and the AI are coming to fore. But it's not enough to spit out probabilities or is it yesses is or noes, or cats or dogs. One has to visualize these results, and that's where data visualization comes to play.

And it wasn't until I joined Columbia University and I talked to all my colleagues across the university-- in particular, the School of Journalism-- when I really came to appreciate that last step I put on my data lifecycle. The journalists call it storytelling. I call it interpretation. And that is really it's not enough to show a pie chart or a bar graph-- you really have to explain to the end user, what am I looking at? Tell a story about the data interpretation.

So I like to emphasize the privacy and ethical concern throughout this data lifecycle. I think it's very important from even the start of what data do we collect to what data do we analyze. What answers do we provide the end user? So let me also share with you a very succinct definition of data science, which I think this audience will especially appreciate. Data science is the study of extracting value from data. And there are two important words in this definition. The most important word is value, and value is-- and I define it deliberately to be-- value is subject to the interpretation of the reader.

So value to a scientist is discovering new knowledge, but value to a company is likely accrues to the bottom line. In fact, it's very likely calculable. And value to a policymaker is information so that the policy maker can make a decision about the local community. So the other important word here is extracting, because it takes a lot of work to get this value from the data. And now I want to share with you my three-part mission statement for the Data Science Institute.

The first is to advance the state of the art in data science. This is about pushing the frontiers of the field, about doing basic research, about doing basic long-term research, and inventing new techniques, and new discoveries, and new science. The second is transforming all fields, professions, and sectors through the application of data science. And this really speaks to the prevalence of AI, machine learning, data science that's affecting all fields, all sectors as we see it today. Everyone has data and data is everywhere. And data today is what feeds those very hungry machine learning algorithms. So with a lot of data, you can do a lot using these machine learning techniques.

And finally, ensure that responsible use of data to benefit society. This really speaks to both benefiting society using data-- and this is really tackling societal grand challenges like health care, energy, climate change, the UN sustainability goals, for instance-- but it's ensure the responsible use, which I inserted in my mission statement that I really want to emphasize. With a lot of concern that we read about every day in the news, about biased algorithms, or biased models, or biased data that we feed their algorithms produce biased models, I think it's very important that we technology people try to ensure that what we do will be non-discriminatory, be fair, and take into consideration these ethical concerns.

So I summarize this long-winded mission statement into my tagline data for good-- data to do good for society and also using data in a good manner. So what I wanted to do-- oh, let me just-- a few more facts about the Data Science Institute, and then I'll go through some research stories. So we actually, Columbia University, the Data Science Institute at the university level and a university-wide Institute.

So we have over 350 faculty now from 12 different schools across the university. Every single profession is represented. Every single discipline is represented. Everything-- every single sector is represented, from arts and sciences, architecture, business, dentistry, all the engineering disciplines, public policy, journalism, law, medicine, nursing, public health, and social work. So this is partly reflecting the breadth of the university.

We have a few centers that are thematic. I'm not going to belabor this, but just to say that we have some themes that pop out. Like business and financial analytics, we have a center there. We have a center in health analytics, computational social science, and so on. We have a robust master's in data science program at Columbia University. And this is just to show you how we define a minimum bar of what it means to be a data scientist.

I share this with you because in industry today, across the country, across the world, there's a lot of confusion about what is a data scientist, and there are a lot of titles out there. And so I thought, well, let me set a bar for what makes a data scientist. So we have a highly selective program and a very rigorous program. So there are six required courses-- three in computer science, three in statistics, and then the capstone course, which is where an industry person or affiliate comes in, brings industry data to a team of students. They work on that data set, and answer real-world questions driven by the industry affiliate. And of course, everyone gets a job.

So that's our program. I should mention we just started a PhD specialization in data science at Columbia. So if you're already enrolled in certain PhD programs, you can take these additional courses and get it-- a specialization. I wanted to mention a couple of things going on in the education space. There's a program called the Collaboratory that's joint between the Data Science Institute and Columbia Entrepreneurship.

And we have now run this program for a few years, which requires that professors from two or more different disciplines come together, co-design, and co-teach a new course where one of those professors is computer science, or data science, or applied math. And it's really to bring computational and data science expertise to some other domain.

And the one I want to highlight is not just a course, but an eight-course curriculum that was designed by the Business School, along with computer science and data science, to the point where now, 50% of the MBAs coming out of Columbia have had some exposure to data science. So this is phenomenal, and the only limiting factor here is capacity. So if we could, it would probably be 100% eventually.

So what this says to me is, first of all, students are always smart. They see the future. They know what they need to learn while they're in school in order to be a good productive employee in any discipline later. We have a robust industry affiliates program. I'm double checking that JP Morgan is there. Yes it is. And we even this past year expanded it to international companies, like three from China-- Alibaba, Baidu, and [INAUDIBLE] should be up there-- and we have a company from Brazil.

Very recently, we created a center with IBM specific to blockchain and data transparency. This was actually quite a big deal for Columbia, and it was-- it's going strong. We have three tracks-- one on research, one on education, and one on innovation-- which is really an accelerator incubator track, so startups can get nurtured and then spawned out of this center.

So now, what I wanted to do in my remaining time is just to share with you a few research stories to show the kind of work that's going on in Columbia and data science. And I'll just do one on advancing the state of the art, and a few in terms of the other two parts of the statement. I wanted to emphasize that unique to Columbia University, data science-- the foundations of data science builds on three pillars of strengths-- computer science, of course, statistics, of course, but also operations research.

So at Columbia, we have a very strong OR department in the School of Engineering and a very strong OR group in our Business School. In fact, they all work together. And so from afar, it's really a great strength of Columbia. And OR, a lot of it is optimization, and there's just so many similar techniques and interest. And OR, I think, as a field, is moving more into machine learning, data science, and so on.

So the one story I wanted to share with you, in terms of advancing state of the art, has to do with causal inference. Of course, causal inference has been of interest to statisticians for decades. It is the bread and butter of economics, political science, and so on. And I think the machine learning community has deliberately and rightly been shy to ever infer causality and all the patterns they recognize. They say, it's just a correlation.

But still what we want, decision makers and policymakers especially want to know, does this cause that? Does smoking cause cancer? That's the canonical example. So what I want to share with you is some new results by [INAUDIBLE] Wang and David [INAUDIBLE] on multiple causal inference. And it turns out that the classical causal inference problem is a univariate problem, where you just want to know about a single cause having an effect.

But it turns out that the multiple causal inference problem is actually more of a prevalent problem. And it turns out also to be an easier problem to solve with weaker assumptions. So let me frame it in terms of an example. So pretend I'm a movie director, and I want to choose actors for the movie I'm going to produce. And I want to know how much money am I going to make. I want to predict how much money I'm going to make.

So what happens to movie revenue, if I place a certain actor in my movie? And what I have it at my disposal is a little database that says, for this movie, for these actors, I made that amount of money. And then mathematically, in the statistics point of view of causality, we would frame this as solving this equation-- or solving this expression, estimating the potential outcomes of a set of actors in my movie [INAUDIBLE].

Or if you are a computer scientist and you learned causality in some course, or you read about it through Judea Pearl's book, The Book of Why, you might express it in terms of the [INAUDIBLE] notation. They're essentially equivalent. So understanding causality-- and this, by the way, is a multiple causes that-- the problem with understanding causality is that there can be many possible confounding factors that can influence the outcome.

And I want to account for those confounding factors, otherwise, I will over count some factors and undercount others. So this problem has many applications, whether it's in genetics-- what genes cause a particular trait-- the people I choose on my sports team-- I want to know how many points I'm going to score-- or prices in a supermarket-- how much money is going to be spent depends on the prices in the supermarket.

So in classical causal inference-- by the way, for the movie example, let me just give you some examples of some confounders, and I think that that will motivate why it's a difficult problem. So if I were making an action movie, the genre of the movie is likely to affect the revenue. Action movies make more than artsy movies. And even knowing that I'm going to make an action movie will likely affect who I'm going to choose to be in my movie.

And then who I might choose in a movie might affect who else I might choose in the movie. And whether the movie is a sequel or not is another confounding factor. So today, when we do classical causal inference, in terms of a single causal inferences, the approach is you think about all these confounders-- genre, sequel, other actors, and so on and so forth. That's a human task.

And then, assuming you have thought of all the confounders, then you can plug the numbers in and estimate the causal effects. That's the right-hand side of the equation. The problem is it's a big assumption and it's untestable. But we seem to be OK with that. But this is the way it is. So we make this assumption that we've been really smart, we thought of everything, and then we crank it out.

And we get our numbers. We base everything on what we just cranked out, probably forgetting that we made that big assumption. So in the new approach, under the assumption that we have multiple causes, the idea is to construct what's called a deconfounder. And the beauty of this approach-- there are two advantages.

One is there's a weaker assumption, and the other is one can test the model of the deconfounder that you're constructing against a goodness function. So it's more constructive. So the basic idea is to fit a local latent variable model to the assigned causes. Those are the observable. So think of a factor model. And then infer the latent variable for each data point. That's what the z hats sub i's are.

And it is a substitute confounder. And then instead of using the Ws from the previous slide, which are the confounders I thought of, use the Zs, which come out of the model that I've constructed. And then use the usual right-hand side construct for the substitute confounder in the causal inference. And the only assumption that we need to make is that there's no unobserved single cause confounder.

In the classical causal inference case, we had to assume we thought of all the unobserved confounders. So it's a weaker assumption. Moreover, once you construct this model, you can actually test it for goodness against some function that you define as good. And then there's a proof in the paper that shows is an unbiased inference. So this is actually, I think, quite a move forward in the grand scheme of causal inference. So if at all you're interested in this, there's a paper you can read.

So let me go back to movies. What does this mean? Once we construct a deconfounder based on that little database that I showed you-- this is a snapshot of a James Bond movie. Sean Connery used to play the role of James Bond, 007. I don't know if any of you are old enough to remember Sean Connery. You probably remember Roger Moore.

But anyway, with the deconfounder, Sean Connery, the person who played James Bond, his value goes up, whereas unfortunately, those actors who played the lesser roles, M and Q, their values go down. What this means is that, without the deconfounder, Sean Connery's value was underestimated, and M and Q's actors' values were overestimated. So the deconfounder corrects for that.

And then once you have this model, just as with any causal model, you can do this counterfactual reasoning-- what if this, what if that? OK, so that's one story, in terms of advancing the state of the art. I think causal inference is still a very hot topic. And it's always been for statistics, but it's really rearing its head in the AI and computer science community now.

Now, what about transforming all fields, professions, and sectors through the application of data science? What I wanted to do is run through a lot of little stories to show you the breadth of what's going on at Columbia University. And I'm going to start with some science stories-- in particular, biology. So this is where the big data-- and there's not sophisticated machine learning going on here. It's just the big data problem where you're using this DNA sequencing-- in particular, of the microbiome-- around pancreatic cancer tumor cells.

And what the scientists discovered is that the microbiome around the pancreatic cancer tumor cells were counteracting the effect of the chemotherapy used to treat the tumor. So this is not good. It basically says that chemotherapy's ineffective. But the scientists went one step further to show that, if you inject the tumor cells with an antibiotic, that antibiotic would counteract the effect that the microbiome, therefore making the chemotherapy treatment effective.

So all of that was done through just lots and lots of data. I'm very impressed by the astronomy community because they have so much data, and they've been capturing so much data from all these large telescopes on Earth, flying in the around the universe, giving us images of the universe. And why I'm so impressed by the astronomers is that they'll throw anything at this data. They're very courageous.

And so this group of faculty at Columbia, along with some computer scientists and data scientists, use convolutional neural networks to look at weak gravitational lens images coming from large telescopes that are flying around. And what they showed is that they were able to estimate the parameters of the Lambda Dark Matter Model, which is a model of the universe, far better than off-the-shelf statistical techniques.

Now, as a person who's witnessed the use of neural networks and deep learning over the past five years or so, and how it's exploded in its success and applications-- from image processing, to speech processing, to natural language translation, and everything-- it's exasperating to me that, yet again, we throw neural networks at this, and there's huge success. Because as a scientist, what's exasperating is we don't really know why these neural networks are so successful. But there you go.

For something completely different-- this is coming from our economics faculty-- they have been looking at online market-- labor markets-- in particular, Amazon Mechanical Turk, and other markets such as that. And what they've discovered using this technique called double machine learning is that these online labor markets do not behave as a regular free marketplace.

Rather, they behave like what's called a monopsony. So all of you probably know what monopsonies are, but I didn't when I read this paper. A monopoly is when you have one seller and multiple buyers. A monopsony is when you have one buyer and multiple sellers. And so these online labor markets are actually behave more like that. And it's counterintuitive.

An example of a piece of data that they were-- a piece of information they were able to generate from all the data they collected is here, where they show that high-reward tasks do not get picked up more quickly than similar low-reward task. The idea is that, if it were a normal marketplace, you would go after the high-reward task because you get more money. But that's not how this marketplace-- this labor marketplace behaves.

For something more, and perhaps related to your world, is finance and reinforcement learning. This is a colleague of mine in the Operations Research Department who has been looking at robo-advising. So this is all very familiar to you. You have a lot of money. You might have a personal financial advisor. And over time, that personal financial advisor learns your investor preferences, what your risk aversion is to investing in a certain portfolio of instruments.

And so what he does-- and I just wanted to show one equation-- basically using standard reinforcement learning, over basic eight or nine iterations of this formula, you learn what the investor's preferences are. And what he also goes to show is that actually the combination of human and machine still outweighs either the human or the machine.

Now, again, for something completely different, we have some very modern history faculty who are using standard machine learning techniques, like topic modeling, sentiment analysis, and so on to look-- to paw through documents. So one set of documents that one of the history professors has accumulated is the largest set of declassified documents.

So he has downloaded everything that the federal government produces every year. And one example of what he's done with that is to look at-- if he will looking at just the cablegrams that diplomats sent each other in the 1970s, he wanted to see if he could detect the anomalous events, which should be the interesting historical events.

So each black dot here represents one of those interesting events. And you can recognize some of them-- the evacuation of Saigon, the death of Mao Zedong, and so on. So this is really the History Department and the history faculty using machine learning, AI, and so on in doing their research.

It's actually quite interesting that the history faculty have said to me-- they're at this juncture in what to teach their next generation students, because there's all these techniques that you learn as a historian, but now all of a sudden there are these computational techniques, these machine learning techniques. Knowing that everything is now digitized, it's important for the history students to learn this material.

So now let me talk a little bit about data for good, responsible use of data. And I want to share with you an acronym that I already don't like anymore, but it reminds me of the principle that we need to subscribe to, in terms of using data for good-- so fairness, accountability, transparency, ethics, safety and security. And I call that faiths.

My sole contribution to this acronym is S for safety and security. Others have come up with [INAUDIBLE] and FATE and so on. But what I wanted to focus on is safety and security. So there's a system that actually, JP Morgan, thanks to Manuela's little program, is helping to fund-- some large larger efforts that we are exploring at Columbia University on looking at deep learning and using formal methods and programming language techniques to better understand these deep learning systems.

In this particular work, DeepXplore, they're using two techniques inspired by software engineering and programming languages to look at deep neural networks. And the one technique that I think-- I find very easy to understand is this notion of neuron coverage. So we know from programming languages and writing computer programs, when you're testing a program, you want to-- there's a notion of code coverage.

So if you want to test all the paths in your program to make sure that at least for those paths, you get the right answer. Inspired by that idea, why not use that idea for a notion of neuron coverage, where you want to tickle every node in your network and every edge in your network, and see what is germane, given the inputs to the output? So that's one idea.

And what they've found, using another idea called differential testing, was they took off-the-shelf state-of-the-art DNNs used for image processing, like ImageNet and so on, and they've basically found ways-- they found flaws in these classifiers. So in particular, let me give you an example.

In the first case, on the left is an image for which the classifier-- and now, think of this classifier as being in your car or your self-driving car-- the camera of your car-- so this classifier looks at this image and correctly says, veer to the left. And that's fine. What this DeepXplore does is it finds natural perturbations to input images in such a way that you can fool the classifier to do the wrong thing.

So in this case, a natural perturbation is to darken the image just slightly, which is a natural event because we don't always drive in daylight. And once you do that, then this classifier will actually say, veer to the right. You hit the guardrail, you fall down the cliff, and you die. And so that's why they call it fatal errors.

So another interest in data science, and computing more generally, and especially with EU GDPR, is privacy. And what this group did was combine the notion of differential privacy, which has been out there for over a decade now, with deep learning to-- rather than test a deep learning system to see if it will do the right thing, can we once and for all ensure that for all inputs or for a set of inputs, we will guarantee the classifier's robust to perturbations?

And so they use this notion of adding noise that we get from differential privacy to add a noise layer to the DNN. In particular, they found that adding the noise layer early on will give us this ability to prove once and for all that the output is robust to perturbations from the input. So I think this is very exciting work, and it really there again shows this combination of results from different parts of the field coming together.

So now, let me close with some stories about tackling societal grand challenges. I wanted to share this one because it's a New York City happening. This is a huge grant that the combination of many universities, including Rutgers and Columbia, NYU, receive from the National Science Foundation to basically put in a one square mile area in Harlem-- so right next to Columbia University-- a test bed that will be advanced, in terms of the kinds of antennas put in, the kinds of-- wireless test bed that people can play with.

And this is so that we can look at not just 5G protocols, but protocols beyond 5G, and not just the wireless protocols that we need, but the applications that will sit on top of that. So this is going to light up a lot of interesting research. And having it close to Columbia and working with the Harlem community has been really wonderful.

So in terms of climate science, as many of you probably know, Columbia University has an Earth Institute, which is a university level, university-wide institute. And within the Earth Institute is Lamont Doherty, which is probably the premier-- one of the premier climate science capitals in the world. And this work that I'm showing you here is actually an example of the kind of infrastructure needed for many scientists to share models, share data, share algorithms.

It's called Pangeo, another open-source platform, also partly funded by the National Science Foundation, but other agencies, as you can see. And it is already being used by climate scientists around the world to share their models, to share their data, to share their algorithms. And it's a really nice layered stack of software, and it's already being used to do meteorologic examples, hydrology.

And the example I wanted to show you is not being done by Pangeo, but this is a simulation done by NASA and JPL of the ocean currents flowing around the Earth. This is a simulation based on 2 petabytes of data. And 2 petabytes right now is not a lot, because the IPCC anticipates within the next 10 years of generating up to 100 petabytes. And there's no one system that can simulate all that data. And of course, when I look at that, I immediately want to zoom in and out. I want more the ocean-- more than the oceans. I want so much more, but this is state of the art.

OK, my last example is in the health care arena, and I wanted to use this example for a couple of reasons. First, Columbia economy is the coordinating center for this federated data set of electronic health records around the world. There are already 600 million patient records in this federated data set coming from 25 different countries, 80 different databases. What is phenomenal to me, as the IT person usually in the room, is that these records are all in the same format.

Yeah. But once you have this data set, as you all know, you can do things you would not be able to do in traditional medical science. So what I wanted to show you is a couple of examples of what results they were able to see from just looking at these patient records. No clinical trials-- just observational data.

So first, the other reason I wanted to show you this is to reinforce the point that data visualization is not enough. I have to tell you the stories behind these beautiful pictures. So first of all, they looked at three different diseases-- diabetes, hypertension, and depression, from left to right. Each circle of rings represents a single data set.

So on top of the topmost the middle column is CUMC. That's Columbia University Medical Center. So if you are treated for hypertension, you're in there. Now, what they were looking at is, for each disease, for each patient, the sequence of drugs given to that patient to treat that person for that disease.

And so first, you're given a drug, and that would represent the inner circle. So for instance, if I pick on that hypertension one up in CUMC, that drug represented by the orange color is the first drug of treatment. If that works, fine. But if that doesn't work, then I'm given a second drug, and that drug is represented by the second circle-- second ring around that circle.

And if that doesn't work, I'm given a third drug. If that doesn't m I'm given a four drug, and so on. And by the way, I'm only showing you four rings for each circle. This goes on for tens of rings, in some cases. So the first interesting observation the scientists made by looking at just this data is that, if I collapse all the rings and circles for hypertension across all the data sets that they collected, 1/4 of patients treated for hypertension are treated uniquely.

That's pretty astounding. So that says, Manuela, if you have hypertension and you're in that 1/4, and you say to me, Dr. Jeannette, is there anyone else being treated in the world like me, the answer's no. This is just an observation. Of course, this observation then asks more questions, like why is this? What's really going on?

So that's one interesting result from just looking at the data. The second is the lower left-hand corner here. This is diabetes. If I were to show you all the rings and circles for all the data sets that are represented, they would pretty much look like the top two, where this first drug-- the chartreuse drug-- works pretty well.

But you'll notice it doesn't for the lower left-hand corner, and that lower left-hand corner represents a Japanese medical clinic. And it turns out that the Japanese are predisposed against that chartreuse drug. This was not known until looking at the observational data. And so of course, this also raises interesting questions of why, and so on.

And I asked my colleague, well, does this hold for the Chinese and the Koreans? And it doesn't. So again, some interesting new science to discover what's really going on. So I will close now. Data for good, just remember that of my talk, and I'll be thankful. Thank you.


MANUELA VELOSO: We have time for a couple of question.

JEANNETTE WING: Don't be shy. Yes?



AUDIENCE: With the Data Science Institute, are there any upcoming programs or events that you're really looking forward to, or initiatives--

JEANNETTE WING: First of all, we have an annual event called Data Science Day. It's usually at the end of March or beginning of April. It's a huge event. This past year, we had over 800 registrants-- a very popular, event especially with our industry affiliates. I would absolutely encourage all of you, if you're interested, to look for that one-day event.

It's where we showcase a lot of the research going on across the entire university in data science. We have demos, and posters, and all sorts of cool things. So that's one event that, if there's just one event you were to attend, that would be it. But the other thing that we run routinely are smaller workshops or conferences that are more thematic. Most recently, we actually ran one on machine learning and finance. And we've done that one a few years in a row.

We also have done an event on business, and I can't remember the title, but it basically is business and journalism. And so we run a lot of these different sorts of things. If you're interested, I can make sure that you're on some mailing lists, and then you can hear about them, because we're continuously running lots of events. Uh-oh.

MANUELA VELOSO: So I have a very difficult question, and I'll take advantage of having you here to ask it. So with all your experience-- I forgot to say, Jeannette was actually the head of the Computer Science Department at Carnegie Mellon for many years-- the first woman, the first energetic [INAUDIBLE]. Anyway, with all your experience, suppose there is an oracle, someone that will answer one question the right way.

So in all the things, from a scientific point of view, from an organizational point of view, from a society point of view, what is the one question that takes your sleep at night, and you would like to have someone give the absolutely truth [INAUDIBLE]? Jeanette--


MANUELA VELOSO: Jeannette, I'm not letting you go without you answering.


MANUELA VELOSO: What were you expecting from me and you here?

JEANNETTE WING: You should have warned me.



MANUELA VELOSO: --what is that question? Don't tell me that it's like the traffic in New York or something. No. But what is the question--

JEANNETTE WING: No, what is that question?

MANUELA VELOSO: --that you really think? Is it an educational question? Is it what should people learn? What is the question? Should the machines be faster? What is the question? This oracle will give you the right answer, so you don't have to do research more about it. You are done. What is the question that you would like to be done with?

JEANNETTE WING: Why do humans behave the way they do?


MANUELA VELOSO: Very good question.

JEANNETTE WING: What do you think of that?

MANUELA VELOSO: I think it's a great question. In fact, the mystery are the humans indeed.

JEANNETTE WING: We are. We have this Neuroscience Institute at Columbia University. They study mind, brain, and behavior to show that it's not enough to understand how your neurons work-- it's all the layers on top. And then when I think about human behavior, that causes grief to computer scientists, because it's the presence of uncertainty that-- Manuela's as poor little robots are always trying to avoid the humans because they might get in the way.

And then the cars-- the cars are worried about pedestrians and bicyclists, because we are what we are. And it's not just that. When I think of my friends in public health, public health schools continue to exist because, despite best practices and good advice on what to eat, exercise, and so on, we don't do that.

MANUELA VELOSO: There are many examples here also in the financial world [INAUDIBLE]

JEANNETTE WING: Yeah, you should relate it to the financial world. But I think that the finance world is being slowly overtaken by machines.

MANUELA VELOSO: It's humans, I'm telling you. It's 250,000 human at JP Morgan.

JEANNETTE WING: Around the world.

MANUELA VELOSO: So any other more easier questions?

JEANNETTE WING: Easier questions please.

AUDIENCE: I'm not quite sure how this relates to anything, but recently, I was reading about China and the HIV-resistant babies created by the CRISPR. And even in machine learning, China has so much detail, rightly or wrongly corrected. And then your title is Data for Good, but good is relative, so I don't know. Where do you see China versus the US in the next 5 or 10 years in machine learning?

JEANNETTE WING: That's an excellent question-- so timely, because I just returned from a trip to Shenzhen, which is just an amazing, amazing little corner-- big corner-- bigger and growing corner of China. Though the Chinese are, I think, right now unstoppable. In AI, they have humongous amounts of data. They don't have a lot of regulation to get in the way. The citizens of the country kind of-- they know they're monitored for everything, so privacy is-- and the big companies, like Alibaba, Baidu, TenCent, and so on, they're just going full steam ahead.

And even the business models, the economic policy models-- China's very flexible, very adaptable, very experimental. So China is definitely a place, if you want to think competitively, with respect to the US, to watch. And they're not going to wait around for any other country to get their act together or to behave collegially or anything.

AUDIENCE: I just want to say I really relate to your fascination about how humans behave. My question actually that I wanted to ask before you said that was, how do you see data science a machine learning advancing neuroscience research, and just understanding how the brain works and all the mysteries?

JEANNETTE WING: It's a great question. And I'm glad I actually have an answer to that because fortunately, the Zuckerman Mind Brain Behavior Institute asked me to give a keynote recently, and I said, OK, I better relate it to neuroscience. And so I did a little homework. So the first thing, of course, as many of you know, is that deep learning itself was inspired by the way the brain is wired.

And I remember-- so Manuela probably remembers this too-- remember Jeff Hinton? The three of us were colleagues in the '80s together, and we used to kind of just humor Jeff when he would talk about neural networks and making the brain-- making a machine that's wired like the brain. And then 25 years later, it takes off, and it is also unstoppable.

So what's interesting is, of course, it's still a metaphor. And the neuroscientists are benefiting from deep learning and so on just as all other fields are benefiting. It's basically image analysis, and Manuela can tell you about her latest and greatest there, as well. But in terms of medical science, neuroscience, and so on, a lot of it is image processing. So think EEGs and fMRIs, think about combining the two-- all the stuff that you can do with just images, they're going full out with that.

But what's interesting is the reverse now. So the neuroscientists are saying to us-- computer scientists-- well, your model of the brain-- and look, it was just a metaphor. It wasn't really supposed to be a real model of the brain. But they're saying, your model of the brain is too simplistic. The brain is really much more complicated. First of all, it's not just a bunch of homogeneous nodes that do sigmoids. The wirings, it's not just hierarchical.

We've got connections across the layers. We've got all kinds of connections. And all those nodes, they could be doing different functions. And I think there are both combinations of neuroscientists and computer science working now together to say, OK, well, let's make these neural networks a little more complicated, make them look more like a real brain. Even that was meant to be a metaphor. So there's where I think some exciting science is going to happen, in both directions. There was a question back there.


AUDIENCE: So is there any currently, or there was, any project related to financial inclusion? Like getting data from people and [INAUDIBLE] any credit to low-income people and get data from there. Because I saw that the institute is related to social good and social work, so I wanted to know if there is any project related to that, or which projects it's currently-- are you currently working on with that area?

JEANNETTE WING: Projects related to--

AUDIENCE: [INAUDIBLE] I'm referring to using data from people-- for example, where do they live, which are their incomes.

JEANNETTE WING: What our School of Social Work does routinely, as many other-- probably the School of Public Health and so on-- is they are still collecting data about people through the normal instrument, which is a survey. So there are lots of population surveys, and they're looking into issues of inequity usually. They're usually looking into issues of why is it-- or causality now-- why is it that children growing up in this poor neighborhood with these kinds of circumstances end up in one life path over another life?

Is there something we can intervene and change about that? There's a very interesting work I thought I would share with you on that particular topic-- I didn't include in my talk-- on using natural language processing-- NLP-- pay on Twitter data, but specific to social justice. This is a social work faculty member working with Cathy McEwen-- Manuela knows her-- on looking at the tweets that Chicago gang members write to each other.

It turns out that the Chicago gang has its own language. And this is not surprising. Little subpopulations have their own tweet language. So first, the idea was, if we can understand what they're tweeting with each other, then maybe we can intervene before a violent act happened. So the idea is you're a Chicago gang, one of your gang members is killed. You start tweeting a feeling of sadness or depression versus aggression and some violence in nature. If I'm a social worker or a law enforcement person and I can see your tweets, and I detect that you're expressing a feeling of aggression, then I can intervene.

So that was a long story, and they were able to do this. Interestingly enough, from the computer science point of view, it's a brand new language. So it needed parts of speech analysis, a tagging and annotation. But now it's out there. It's a public data set. And not that many people will speak in this language, but it's out there for you to look at. So there is very interesting work going on at the School of Social Work. But most of the data still is collected through surveys. He has that question.

AUDIENCE: I had a question which is sort of a corollary to the neuroscience and human behavior question. We are more and more interacting with somewhat intelligent or semi-intelligent systems. How do you see this impacting us as humans, and where we can use this for good, or for evil, potentially?

JEANNETTE WING: It's a great question. I actually would go even further back and say, before we even got to the fancy shmancy AI machine learning, how has the digital transformation affected our way of life, but also our way of learning, and our way of teaching? And I just witnessed this in the classroom. I'm sure Manuela did, as well, I saw generations of students grow up with technology to the point where it's a different kind of-- and I talk to my neuroscience friends about this because the mean would be, oh, they're wired differently because they learned-- they didn't learn to work in blocks of time to solve a particular problem, and continue-- but they kind of learn in this scattered way because all the information's on the web, and you can just go around, and maybe their brains are wired-- I think that we still don't know.

As recipients of such minds, I have hypotheses, but I don't really know. And I think it is an interesting question. How has digital technology changed the way in which we think and our brains are [INAUDIBLE]? But now, if you throw in the AI machine learning, when we get used to using and AI system as our accompaniment, do we give up something, or does it make room for more creativity on our part, that will always, I hope, elude the machine?

MANUELA VELOSO: [INAUDIBLE] I'm sorry, we've ran out of time, but Professor Wing [INAUDIBLE]. Let's just thank her kindly.



Thank you, thank you.

[00:00:00.99] SAMIK CHANDARANA: So I recognize a lot of people in this room. But for those of you that don't know me, I'm Samik Chandarana. I'm part of the team that looks after the applied ML and AI. And I'm obviously very lucky to have Manuela Veloso, who I'm joined at the hip with on the AI research side.

[00:00:15.99] So today, we have the fourth lecture in the series that Manuela has been organizing. When Manuela joined the firm, we were very excited because we wanted to make sure that on top of using her own mind, we actually delved into her network. And part of her network today, we're very lucky to welcome Professor Wooldridge from Oxford. Manuela will tell you a little bit about him, and then we get on and hear from the man himself.

[00:00:43.28] MANUELA VELOSO: Perfect. Thanks all for coming. And we are trying to put some more chairs. But also, welcome to everyone who is online. So it's a pleasure to be here and, actually, to have our first distinguished lecture hosted in London. So the other three were hosted in New York. We are planning on coming every other time, actually, to London and host it here. So thanks all for coming.

[00:01:10.21] It's a great pleasure for me to introduce without further ado the speaker today, who is Professor Michael Wooldridge. And I have known-- where's Mike?

[00:01:21.02] Here.

[00:01:21.73] Oh, right here. I was expecting you to be there. I'm like, wait, he's up here. I've known Mike for, I believe, more than 20 years. We have been faculty in an area of AI which we call multi-agent systems, multi-agent learning. And Professor Wooldridge will tell you more about this.

[00:01:43.41] But we have been partners in crime in this particular kind of area of research. In fact we co-edited a book together, where we have our own book on AI. And Professor Wooldridge has a distinguished career of researching AI. He's the current head of the Department of Computer Science at Oxford University.

[00:02:07.53] And he was also President of the European Association for Artificial Intelligence, as well as the International Joint Association, what we call ECCAI Conference on AI has its own board. And Professor Wooldridge was also the president there.

[00:02:24.45] And then, just so you know, in academia, there are these honors which are associations for which we can be fellows. And the main associations of our areas are the ACM, which is this Association for Computing Machinery, AAAI, which is the Association for the Advancement of AI and the European, also, Association for the Advancement of AI. And Professor Wooldridge is a fellow of these three institutions. In addition, he has published more than 500 articles.

[00:02:59.85] And it's a great pleasure and, in fact, a great honor to welcome him to JP Morgan. And so please, we'll let that Professor Wooldridge speak for as much as he wants-- hopefully, within time-- and then, we'll have a session of questions. So I would ask you if you could hold the questions to the end of the lecture would be great, unless you have a real clarification question, and then you can ask. But otherwise, let's please welcome Professor Wooldridge and we are going to listen on Siri. Let me introduce you, Siri.

[00:03:33.42] [APPLAUSE]

[00:03:40.47] MICHAEL WOOLDRIDGE: Thank you, Manuela. Thank you, colleagues of Manuela for doing me the honor of inviting me. I'm used to hanging out in Oxford, and when you first get to Oxford, there's all these grand buildings, these ancient medieval buildings. It's very intimidating. After a while, you get used to it. And you forget when people come in how intimidating it is. And then I got off the train this morning at Canary Wharf, and it's like, oh, my god.

[00:04:03.06] So I'm trying to cope with that. I'm very grateful to have the opportunity to speak. It's very exciting to see an organization like JP Morgan start to take AI so seriously. I think it's going to be incredibly exciting to see what Manuela achieves. Everybody in the field knows that Manuela has boundless energy, boundless determination, boundless enthusiasm, which she communicates to everybody around her. So I think you've made a great hire. And there are exciting times for AI at JP Morgan ahead. And I look forward to seeing what's happening there.

[00:04:35.34] So before I begin, a small confession. This morning, I left Oxford at about 7:15. And as I was leaving, my wife said, you're not wearing a tie. I don't normally wear a tie. Actually, this is me looking smart, I have to say. And she said, you're going to JP Morgan. Everybody is going to be wearing a tie. And I look around the audience, and I see maybe two-- two-- two people wearing. You must be the bosses.

[00:04:58.14] [LAUGHTER]

[00:05:00.69] AUDIENCE: [INAUDIBLE]

[00:05:02.78] MICHAEL WOOLDRIDGE: OK. So let me get started. So this is a talk about artificial intelligence. But I have to begin with a confession. There is no deep learning in this talk whatsoever, OK? So sorry if that comes as a shock to some of you.

[00:05:18.34] But actually, the truth is, AI is a very broad church. And deep learning is kind of the poster boy or poster girl for the successes of AI at the moment. But it is one thread in a very rich tapestry of AI research, which is now delivering really impressive results. It isn't all just about deep learning.

[00:05:38.49] And in fact, if you just remember one message from what I'm talking about today, just take that one away, all right? It's not all about deep learning. There is an awful lot more going on than that.

[00:05:48.21] What I want to talk about particularly, and it is an area, my own field, which Manuela has also worked in, a field called multi-agent systems. I've never really done anything else. This is what I've been doing since around about 1989 when it was an idea, right, that we thought would come to fruition. And it's an idea, I believe, which now has come to fruition.

[00:06:09.34] And I give you a little bit of a flavor about how that happened. So OK. So I'm going to start off by motivating the idea of multi-agent systems, what are multi-agent systems.

[00:06:31.35] [LAUGHTER]

[00:06:33.15] It worked a minute ago.

[00:06:34.54] AUDIENCE: [INAUDIBLE] AV. Oh. Sound?

[00:06:38.91] MICHAEL WOOLDRIDGE: OK, brilliant. Thank you. So I'm going to start off by motivating the idea of multi-agent systems and give you an idea of where this idea came from. And I'll show you a little video, which Manuela will have seen a million times before. Many of you will also have seen it but many of you won't.

[00:06:52.22] And what's remarkable about this video, which was made in the 1980s, is how clearly it anticipated a bunch of things which we're now seeing happening. So I'll show you that video. And that will lead me in to discuss the idea of multi-agent systems.

[00:07:07.49] And then, the bulk of my talk-- so this is the high-level motivational stuff about why we're doing what we're doing and why this is a natural thing to be doing and what we might want to be doing. In the middle part of the talk, there's some more technical material. And the more technical material is really to do with an issue which arises in multi-agent systems, which is that they are inherently unstable things.

[00:07:28.77] We build things which are inherently unstable. We need to find ways of understanding and managing their dynamics. And so the technical part of the talk is about understanding the equilibria of multi-agent systems and the tools that we've developed that enable you to do that.

[00:07:45.25] So I'm going to talk about two different approaches there. The first is an approach based on ideas in game theory. And game theory is an area of economics which has to do with strategic interaction. And I'll show you how you can apply game theoretic ideas to understand the dynamics of multi-agent systems, what behaviors those multi-agent systems might exhibit.

[00:08:05.98] And that's one approach, which has some advantages and disadvantages. I'll show an alternative approach very, very briefly, an approach called based on agent-based simulation, where instead of trying to formally analyze the equilibria of the system at hand, what we try and do is just simulate the system in its entirety. And these are both ideas, complementary ideas-- they're not in competition because they deal with quite different types of systems-- but both ideas whose time has come.

[00:08:36.00] And then I'll wrap up with some conclusions. So let's start by motivating multi-agent systems. So I wrote a textbook on multi-agent systems. It's appeared in a couple of editions now. And the first edition I wrote in around about 2001.

[00:08:50.88] And the very first page of that first edition essentially had this slide on it. It said "what is the future of computing?" And the future of computing, I reckoned in the year 2000, was the following. The future of computing was to do with ubiquity, interconnection, delegation, human orientation, and intelligence.

[00:09:09.22] So what do I mean by those things? Sorry. I should have added, by the way, I think, looking back, I think I was absolutely right. I don't know very much in life. I don't know very much in life, but what I know is that, actually, this is how things turned out, and this is how things will turn out for the foreseeable future in computing.

[00:09:28.56] So let me explain what's going on. Ubiquity is just to do with Moore's law, right? Moore's law, the number of transistors that you can fit on a chip or whatever, basically, what it means is computer processing power just gets exponentially cheaper year on year. The devices, the computer processes that are used to drive computers get smaller, have lower power requirements, are more powerful year on year.

[00:09:52.53] And all that means is that you can put computers into places and in devices that you couldn't have imagined. So ubiquity just means putting computer processing power everywhere. And that's made possible by Moore's law.

[00:10:05.71] So a neat example of this-- a magazine, a home computer magazine, which costs 5 pounds to buy in the UK recently handed out on the front cover a Raspberry Pi computer-- free. You got it free with the magazine, a computer which, 10 years ago, would have had the power of a typical desktop computer of the day. So that's ubiquity. We can put computer processing power everywhere. Any device that we might care to build, we can augment it with computer processing power.

[00:10:32.79] And the second aspect of that is interconnection. These devices are connected, right? When I was studying computer science as an undergraduate in the 1980s, I did a course on networks. And the guy that was teaching this course told us, look, these networked systems, these distributed systems, they're really hard to build. But don't worry, because you'll probably never make one, right?

[00:10:53.61] I mean, no lecturer has ever been more wrong than that lecturer was on that day, right? Because we now realize-- and this was a change in the way that people thought about computing-- we now realize that, actually, networked systems, distributed systems and interconnection, communicating systems, these are the norm. These aren't the exception. And that's resulted in a fundamental shift in the way that people think about computing.

[00:11:20.05] OK. Then the third trend is a bit more subtle. It's towards delegation. So what do I mean by delegation? Well, some extreme examples of delegation are, again, if I go back to the mid 1980s, Airbus were talking about fly-by-wire aircraft getting rid of the human in the loop. And the aircraft onboard computers would actually have control and, in some circumstances, could overrule the instructions of human pilots.

[00:11:46.08] And a lot of people were outraged at this. They thought this was absolutely the end of civilization, that machines were taking over that. Well, some things went wrong. But actually, the truth is, it ended up being a really good thing.

[00:11:57.60] What we're now seeing, the shift towards driverless cars, whether or not full, level five autonomy that you jump in the car and just state your destination, that's some time away. But nevertheless, there are all sorts of features-- smart cruise control features on cars-- which we are delegating the task of driving the car to.

[00:12:18.13] But those are extreme examples of a much wider phenomenon. We hand over control of ever more of our lives and, very relevant for JP Morgan, ever more of our businesses to computers. We are happy for them to make decisions on our behalf, OK? And that's a trend towards delegation.

[00:12:38.64] Human orientation-- what do I mean by that? When Alan Turing arrived in Manchester in the end of the 1940s to program the first computer-- and the first programmable computer, stored-memory computer, I think was their chief claim to fame, the Manchester Baby-- he actually had a bunch of switches on the front of it.

[00:12:56.37] And you had to flip a switch this way to put a one in this particular memory location and then flip it the other way to put a zero there. To program that computer, you had to understand it right down at the level of-- and it wasn't even transistors then-- at the level of the valves that were in this machine. Nothing was hidden from you.

[00:13:13.95] Well, this was an incredibly unproductive way of programming. People were not very good at programming computers in that way. And if you learn to program on Manchester's machine, you couldn't have gone to Oxford and programmed Oxford's machine because they had a completely different architecture.

[00:13:27.60] The trend since then, firstly, with the arrival of high-level programming languages-- Fortran and COBOL in the 1950s, then on towards languages like ALGOL, Pascal and C-- meant that you could learn to program on one machine and transfer your skills to another. But the key point is what those languages present you with is ever more human-oriented ways of thinking about programming.

[00:13:52.72] Object-oriented programming, which is a state of the art-- I'm sure there are some programmers in the room, and you probably do object-oriented programming-- takes its name from the idea that it was inspired by the way that we interact with objects like this clicker in the real world. And it was supposed to reflect that. It's a human-oriented way of thinking about computing. And interfaces-- human-computer interfaces-- will get ever more human-oriented, and we'll see that in a moment.

[00:14:19.33] And then, the final one was intelligence. Now, intelligence, here, I mean two things. All I mean is, really, the scope of the things that we get computers to do for us continually expands. Year on year, computers can do a wider range of tasks than they could do previously.

[00:14:38.37] Now, there's a very weak sense of intelligence, which is just that, actually, it's just decision-making capability. But actually, what we've witnessed over the last decade is an explosion of intelligence in the AI sense, that now, we're seeing computers that are capable of a much richer range of tasks than we could have imagined when I wrote the first edition of this book.

[00:15:01.41] So I say the future of computing, with absolute certainty, I think, lies in that space. It is towards ever more ubiquity, ever more interconnection, ever more delegation, ever more human orientation, and ever more intelligence. Now, that's still a wide space, and that gives us a huge range of possibilities for where we might end up. But there are a number of other trends in computing, each of which picks up on different aspects of these trends.

[00:15:32.05] So if you look at the trend towards interconnection and intelligence, that takes you towards a thing called the Semantic Web. So after the World Wide Web was developed, the idea of the Semantic Web was putting intelligence on the web, so having smart web browsers so that, for example, if you did a search for weather in Canary Wharf, right, that your browser would be smart enough to realize that if it couldn't find a website which referred to weather in Canary Wharf, that weather in East London, or just weather in London, would be a reasonable proxy for that, involving some reasoning, the kind of common sense reasoning that you will do, but your web browser doesn't. And that's the Semantic Web. And that's been, historically, over the last 20 years, a big tradition in AI-- adding AI to the web.

[00:16:15.93] Peer-to-peer-- don't hear too much about it these days, but 15 years ago, it was all the thing. Peer-to-peer is just one aspects of this ubiquity and interconnection. Similarly, cloud computing, the Internet of Things.

[00:16:28.62] The Internet of Things is just the idea that all our devices-- our toaster, our fridge, our television-- are all connected to the web. It's all one, big interconnected mass. Right now, what you might want to do with that, I don't know. But the point is it's a really exciting potential.

[00:16:44.57] But where I want to go is just pick up on this last manifestation of these trends. And this is the trend towards what we'll call agents. And at this point, I'm going to show this video that I referred to. So this is an old video, OK? It's a video that came from Apple computers. And so you have to set the scene.

[00:17:04.93] This is the late 1980s. John Sculley was then CEO of Apple. He was the guy that evicted Steve Jobs, one of the famous business decisions of all time. They'd just released the Mac. The Mac was being a smash hit. But John Sculley was already worrying about what would come after the Mac.

[00:17:23.36] And the innovation on the Mac was the user interface. It was suddenly a human-oriented interface. It was an interface that people could use without specialist training about interfaces. And so to think about what would come next, they came up with this video, which is called Knowledge Navigator.

[00:17:44.86] [VIDEO PLAYBACK]

[00:17:45.86] [MUSIC PLAYING]

[00:18:10.76] [BEEP]

[00:18:11.76] MICHAEL WOOLDRIDGE: iPad.

[00:18:14.25] - You have three messages. Your graduate research team in Guatemala, just checking in, Robert Jordan, a second-semester junior requesting a second extension on his term paper, and your mother reminding you about your father's surprise birthday party next Sunday.

[00:18:31.55] Today, you have a faculty lunch at 12 o'clock. You need to take Kathy to the airport by 2:00. You have a lecture at 4:15 on deforestation in the Amazon rainforest.

[00:18:43.48] - Right. Let me see the lecture notes from last semester. No, that's not enough. I need to review more recent literature. Pull up all the new articles I haven't read yet.

[00:18:59.90] - Journal articles only?

[00:19:01.82] - Mm-hmm, fine.

[00:19:03.02] - Your friend, Jill Gilbert, has published an article about deforestation in the Amazon and its effects on rainfall in the sub-Sahara. It also covers drought's effect on food production in Africa and increasing imports of food.

[00:19:18.19] - Contact Jill.

[00:19:20.20] - I'm sorry. She's not available right now. I left a message that you had called.

[00:19:25.16] - OK. Let's see. There's an article about five years ago-- Dr. Flemson or something. He really disagreed with the direction of Jill's research.

[00:19:36.76] - John Fleming of Uppsala University. He published in the Journal of Earth Science of July 20 of 2006.

[00:19:44.58] - Yes, that's it. He was challenging Jill's projection in the amount of carbon dioxide being released to the atmosphere through deforestation. I'd like to recheck his figures.

[00:19:54.91] - Here is the rate of deforestation he predicted.

[00:19:57.58] - Mm-hmm. And what happened? Mm. He was really off. Get me the university research network. Show only universities with geography nodes. Show Brazil. Copy the last 30 years at this location at one-month intervals.

[00:20:31.71] - Excuse me. Jill Gilbert is calling back.

[00:20:34.89] - Great. Put her through.

[00:20:36.84] - Hi, Mike. What's up?

[00:20:38.19] - Jill, thanks for getting back to me. Well, I guess that new grant of yours hasn't dampened your literary abilities. Rumor has it that you've just put out the definitive article on deforestation.

[00:20:48.69] - Aha. Is this one of your typical last-minute panics for lecture material?

[00:20:54.18] - No, no, no, no, no. That's not until, um--

[00:20:58.41] - 4:15.

[00:21:01.50] - Well, it's about the effects that reducing the size of the Amazon rainforest can have outside of Brazil. I was wondering, um-- it's not really necessary, but--

[00:21:11.01] - Mm, yes?

[00:21:13.77] - It would be great if you were available to make a few comments-- nothing formal. After my talk, you would come up on the big screen, discuss your article, and then answer some questions from the class.

[00:21:24.24] - And bail you out again? Well, I think I could squeeze that in. You know, I have a simulation that shows the spread of the Sahara over the last 20 years. Here, let me show you.

[00:21:40.32] - Nice. Very nice. I've got some maps of the Amazon area during the same time. Let's put these together.

[00:21:58.62] - Great. I'd like to have a copy of that for myself.

[00:22:02.67] - What happens if we bring down the logging rate to 100,000 acres per year? Hmm. Interesting. I can definitely use this. Thanks for your time, Jill. I really appreciate it.

[00:22:23.90] - No problem. But next time I'm in Berkeley, you're buying the dinner.

[00:22:28.63] - Dinner, right.

[00:22:29.64] - See you at 4:15.

[00:22:31.20] - Bye-bye.

[00:22:33.31] [MUSIC PLAYING]

[00:22:36.66] - While you were busy, your mother called again to remind you to pick up the birthday cake.

[00:22:41.16] - Fine, fine, fine. Print this article before I go.

[00:22:44.97] - Now printing.

[00:22:46.70] - OK. I'm going to lunch now. If Kathy calls, tell her I'll be there at 2 o'clock. Also, find out if I can set up a meeting tomorrow morning with, um, Tom Lee.

[00:22:56.99] [END PLAYBACK]

[00:22:57.91] MICHAEL WOOLDRIDGE: OK. So he's a professor at Berkeley, apparently. They have a somewhat more relaxed life than I would imagine.

[00:23:05.03] [LAUGHTER]

[00:23:07.48] So what's remarkable about our video is quite a number of things that are anticipated. So number one you saw, right? There was an iPad, a 1980s iPad, but clearly, it was a tablet computer. There were no tablet computers, and they were not on the horizon at the time. But that was clearly the way they thought it was going. It had a little selfie camera. I don't know if you-- well, actually, quite a big selfie camera. So they anticipated that.

[00:23:29.65] Other stuff that's interesting is-- and this is before the internet was a big thing-- they anticipated a web search or something like it, that they were already thinking about the devices that people would have in their homes being connected in that way. They picked up on the idea of visualization. Visualization is an area that's grown the way that he's visualizing that data and putting together those different data sources to be able to visualize it in neat ways. That's been a huge growth area over the last 20 years.

[00:24:01.66] But the thing that we picked up on, the thing that drove my community, is the idea of an agent. The thing that he was interacting with on the tablet screen was not a person. It was an animated piece of software. Actually, there's an interesting story there that this animated piece of software, clearly, the idea was that they wanted this to look as lifelike as possible.

[00:24:24.31] And actually, the received wisdom these days is if you're doing something like that, you really don't want it to look as lifelike as possible. Because you don't want to mislead people into thinking that they're talking to another human being. You've got to be explicit, you've got to explicitly show them that they're talking to a piece of software.

[00:24:42.05] So what my community picks up on is that notion of an agent. And what we saw is the idea that, instead of interacting like the 1984 Mac screen, where you went to Microsoft Word, and you went on a menu, and you dragged down that menu to select some item where everything happened because you made it happen, right, where the software was a passive servant, right-- it was only doing the things explicitly that you told it to-- that there was a shift, that instead, the software would become a cooperative assistant, something that was actively working with you, taking the initiative to work with you on the problems that you were working on, OK, so the fundamentally different shift away from this idea of software being something that you do stuff to to something that works with you in the same way that a good human personal assistant would work with you, OK? And that's exactly the metaphor that they had there for their agent.

[00:25:47.32] So that was the idea. That's the kind of vision that launched the agents community at the beginning of the 1990s was that vision. So just remember that video was made at a time when Ronald Reagan was president in the United States. Margaret Thatcher was prime minister here. Nigel Lawson was Chancellor of the Exchequer in the UK. That's how old it was. But actually, a lot of what they predicted was pretty much bang on the nail. It's an impressive vision.

[00:26:12.71] OK. So the first research on agents started in the late '80s and early '90s. And a lot of the thrust of the work at the time was about building specific applications like software that would help you read your email-- so something which would help you prioritize your email-- software that would help you browse the web-- anticipate which link, for example, you were going to follow next and proactively help you with the tasks that you're working on.

[00:26:40.30] But it took 20 years from that video before we actually saw the first commercial agents really start to take off. And the one that grabbed my attention at the time, because I knew some of the people involved and so did Manuela, I think. The people involved in Siri were working at Stanford Research Institute in the US. And where they came from is exactly this work on agents-- software agents-- that we were doing in the 1990s.

[00:27:07.69] And then, we've seen, of course, a flurry of others-- so Alexa on Amazon, Cortana on Microsoft, Bixby on Samsung. I've never actually seen that one. And there are a whole host of other software agents, OK? And those software agents are embodying exactly those ideas, that idea of human orientation, moving away from machine-oriented views towards human-oriented views, presenting you with an interface which you can understand and relate to through your experience of the everyday world.

[00:27:37.90] And the most important aspect of that in that video is communicating with just very natural language, right, communicating in English, which isn't nuanced, which isn't in some kind of strict subset of English. It's not some special artificial language. You're just talking as you would to a human assistant.

[00:27:58.24] OK. So why did this idea of agents actually take off? Well, it's no accident that Siri was released when the iPhone got sufficiently powerful that it could actually cope with the software. There's an awful lot of very smart AI under the hood on Siri. And understanding spoken language requires a lot of processor time, right? So it could only happen when we had sufficiently powerful computers. In other words, it was the ubiquity. It was Moore's law took us there and made that feasible.

[00:28:35.22] So advances in AI also made competent voice understanding. And speech understanding made that possible, OK? It couldn't have happened in the 1980s because we just didn't have the compute resources available. We didn't have the data sets that we now have available to train speech understanding programs and so on.

[00:28:56.49] But probably the most important is that the supercomputer that you all carry around with you in your pocket, right, the smartphone that you have in your pocket-- I mean, we call it a phone. That's the dumbest thing it does, right? That's the most trivial of things that it actually does.

[00:29:10.71] It's a supercomputer. It's equipped with incredibly powerful processors, massive amounts of memory. It knows where you are. It can hear its environment. It can sense movements. And it's connected to the internet. And it's the fact that it has all that stuff which has made these agents in the way that they envisaged back in the 1980s, has made them now possible.

[00:29:33.24] So the agent-based interface, whether or not it's realized in the way that Apple envisaged in that video, right, the agent-based interface, the idea that you interact with some software through that kind of human-oriented interaction, is just inevitable. It is the future of the computing because there is no alternative, right? If you think about your smart home and, in the future, all homes will be smart homes, there isn't really any other feasible way to manage it other than through a kind of agent-based interface. It's got to happen, right?

[00:30:09.86] If you think about a sector like banking, right, where you download an app which you interact to manage your accounts, where that's going, inevitably, is that that app is going to be more and more like an agent, somebody that's working with you to help you manage your accounts, to help you manage money, not just something which is doing something dumbly when you tell it to, but actually something which is actively helping you to manage your finances.

[00:30:39.00] Now, rich agent interfaces-- really rich agent interfaces-- are still some time away, OK? And by rich agent interfaces, I mean that they're still very limited in the kinds of language that they can understand. You don't have to dig very deep to understand the limitations of Siri, in particular, actually. But even the better ones, you don't have to dig very deep to understand their limitations.

[00:31:02.63] But what I want to now dig into is one aspect of this which has really been ignored. So if I say to Siri, Siri, book an appointment to see Manuela next week, why would Siri phone up Manuela herself? Why wouldn't my Siri just talk to her Siri to arrange this?

[00:31:21.89] That's what a PA would do, right? They wouldn't go straight to the boss. They would go to the other assistant. In other words, my field is concerned with what happens when these agents can talk to each other directly, OK? If I want to book a restaurant, why would my Siri phone up the restaurant?

[00:31:41.78] There was this famous example-- I forget whose software it was-- that did exactly this-- it might well have been Apple-- about phoning up a restaurant to book a table you may have seen in the news last October or so. But why would they do that? Why wouldn't they just go direct to the agent at the other site? It just makes perfect sense, all right?

[00:32:00.23] We were discussing over lunch that one of the frustrations in my life, and I'm sure many of yours, as well, is diary management. I spend crazy amounts of time juggling meetings and trying to find suitable things. Why don't we have agents that can do that?

[00:32:11.96] This is not, actually, AI rocket science. It ought to be feasible to have such things now. And there, why wouldn't my Siri just talk to your Siri to arrange this? That is the vision of multi-agent systems, right? Multi-agent systems are just systems with more than one agent.

[00:32:29.45] So if we're going to build multi-agent systems, what do they need to do? Well, your agents need to be able to talk to me, right? My Siri needs to be able to talk to me, but it also needs to be able to interact with other agents. And I don't just mean the plumbing, the pipes down which data is sent. I mean the richer social skills that we have.

[00:32:48.84] So for example, my Siri and Manuela's Siri need to be able to share knowledge. My Siri and my wife's Siri need to share skills and abilities and expertise. If I've acquired some expertise in something, I want my Siri to be able to share it with my wife.

[00:33:04.79] Actually, in neural nets, this is called transfer learning. It's very difficult to extract expertise out of one neural network and put it into another. It's a big research area at the moment.

[00:33:15.08] How can agents work together to solve problems, coordinate with other agents, or, really excitingly, negotiate? Just something as simple as booking a meeting is a process of negotiation. I have my preferences. I don't like meetings before 9:00 in the morning. I like to keep my lunchtimes free. But maybe you have different preferences. How are our agents going to reach agreement? They need to be able to negotiate with each other.

[00:33:41.91] All of these things have been big research areas over the last 20 years in multi-agent systems. And we're beginning to see the fruits of that research make it out into the real world. So just one example. If you've booked an Uber recently-- well, firstly, shame on you because they're not nice people-- but secondly, what happens when you book a ride, that process of allocating somebody to pick you up and do your transport, right, that process, that basic protocol, is a protocol called the contract net protocol.

[00:34:18.08] The process through which that happens is a protocol that was designed within the multi-agent systems community. And it has a ton of other applications out there right now, as well. It's probably the most implemented cooperative problem-solving process, allowing you to allocate tasks to people in a way that everybody is happy with.

[00:34:36.66] So all of these things are active areas of research. If I want my Siri to talk to your Siri, my Siri and your Siri need social skills, the same kind of social skills that we all have-- cooperation, coordination, negotiation, the ability to solve problems cooperatively, OK?

[00:34:57.32] So the debate about general AI, the grand dream about AI, has sort of kicked off again recently because of these advances. I'm not a big believer that that's going to happen. Well, I'm not a believer at all that that's going to happen any time soon.

[00:35:10.85] And nor do I envisage that an agent will have these skills in a very, very general sense. But for specific applications, like meeting booking, right, negotiation skills for meeting booking, protocols that will allow agents to book meetings-- taking into account everybody's preferences so that, for example, I can prove in a mathematical sense that my agent is not going to get ripped off, right, it's not going to end up with a bad deal, that we end up with an outcome which is fair to all of us-- these are all big areas in the multi-agent systems community that we've made a lot of progress with.

[00:35:47.32] So I want to emphasize multi-agent systems are used today. There was an article by a colleague of ours called Jim Hendler-- I don't know if you remember it, Manuela-- about 15 years ago. And his article was called "Where Are All the Agents?" And he said, well, we've been working on these agents for 10 years, but, actually, I don't see them.

[00:36:04.73] Well, I'd love to have Jim here today because, of course, firstly, we've all got an agent with us, right? You've all got a smartphone in your pocket. There are hundreds of millions of software agents out there in the real world and not just agents that interact with people. There are multi-agent systems.

[00:36:20.48] So high-frequency trading algorithms, in particular, are exactly that. These are algorithms to which we have delegated the task of doing trading. People are out of the loop-- completely out of the loop. And they couldn't be in the loop because the timescales on which high-frequency trading algorithms operate are way, way, way too small for people to be able to deal with in any kind of sense at all.

[00:36:44.76] But here's the thing-- so this is going to introduce me to the next part of the talk-- is that when you start to build systems like high-frequency trading algorithms, they start to get unpredictable. They start to have unpredictable dynamics. So here are a couple of examples of this.

[00:37:02.36] So the October 1987 market crash-- so the guys with ties on will remember that. So was it Black Monday or Wednesday? I forget which. Does anybody remember? It was Black something. And what happened? What led us to this October 1987 market crash?

[00:37:20.77] As with all of these things, there was no one cause. But actually, one of the big contributing factors was that the London Stock Exchange and International Stock Exchange had computerized just a couple of years before. I think they called it the Big Bang in London, right? It was when all the stock markets went computerized, and you went from handing people pieces of paper to actually doing trades electronically.

[00:37:43.24] And people built agents to do automated trading. And they gave agents rules like if share price goes down, sell. And you don't have to be an AI genius to see that if every agent has that kind of behavior, then a sudden event like a sudden sharp stock price fall for some reason creates a cascading feedback effect. And that's exactly what happened. It wasn't the only cause but generally accepted that that was one of the key causes of the October '87 stock market crash.

[00:38:14.95] More dramatically, recently-- 6 May 2010-- we were having a general election here. But over in the US, in the middle of the afternoon over a 30-minute period, the markets collapsed. And briefly, more than a trillion dollars was wiped off the Dow Jones Industrial Average. It was the largest one-day drop in the Dow Jones history.

[00:38:39.90] But it only lasted 30 minutes. The markets bounced back. They didn't quite regain, and it wobbled a bit. But actually, they regained their position. And the joke was, of course, if you were having a cup of coffee at the time, you would have missed the whole thing, right?

[00:38:53.64] This was happening on timescales that people simply couldn't comprehend. By the time they understood that something weird was going on, it was already starting to bounce back. So some very strange things happened. So Accenture were trading at a penny a share for a while-- a very, very brief period of time. And Apple shares, bizarrely, were trading at $100,000 each for a very brief period of time.

[00:39:14.71] The scary thing about this is that, of course, now, all these markets are connected, right? We're not operating in isolation. We're operating in a global marketplace. And a nightmare scenario was that you would have a flash crash at the end of the trading day.

[00:39:30.69] And if you hit the trough-- the bottom of the trough-- at the end of the trading day, OK-- you hit the bottom of the trough at the end of the trading day-- nobody knows whether or not this is a real collapse or just a freak phenomenon which is just going to rebound. And then, you've got contagion. It starts to spread over Asia and the rest of the world. And these are very real and very, very scary phenomenon.

[00:39:52.64] So the next point in my talk is if we're going to build multi-agent systems-- and the problem is that people are frantically running ahead to build high-frequency trading algorithms, right, and trying to build them faster and using things like AI sentiment analysis on Twitter to drive the decisions that are being made-- then they are going to be prone to these unpredictable dynamics. We need to be able to understand them. We need to be able to manage them.

[00:40:19.03] And at the moment, management is just hitting a kill switch. It's unplugging the computers, right? That's how these things are managed at the moment. I mean, it's not all it is.

[00:40:27.16] So let me just briefly give you a feel for one of the two approaches that we look at. And the first approach that we look at is what's called formal equilibrium analysis. So this is relevant for systems where there are small numbers of agents. It doesn't work for big systems for various technical reasons. So the alternative technique that I'll talk about in a moment works for big systems.

[00:40:50.07] But for small systems where there are just a handful of interacting agents, what we can do is we can view this as an economic system and start to understand what its equilibria are and what kinds of behaviors the system would show in equilibrium. So to put it another way, what we do is we view a flash crash as a bug in the system, right?

[00:41:12.55] If our system that we have is exhibiting a flash crash or some other undesirable behavior, what we do is we treat it as a bug, and we say, how did this bug come about, and how can we fix that bug? And so the technology that we apply is exactly the technology that's been developed in computer science to deal with bugs. And the most important of these technologies is a technique called model checking.

[00:41:36.92] And so here, I've got a simple illustration of model checking. So the idea is what this thing is here is just a description of a system-- a little bit more technical. I said it was going to get a bit more technical but not too much.

[00:41:47.33] These are the possible states of the system. And then these arrows correspond to the actions that could be performed by the agents in the system. So if the system is currently in this state and some agent does this action, then the system transforms to this state.

[00:42:01.33] And what that gives us is this structure here, which we just call a graph structure. And this is just a model of the possible behaviors of my system. So it could be, for example, that some state down at the bottom here is a bad state, right? This is a flash crash state. And what we want to understand is, how does that flash crash state arise? How can we get to that?

[00:42:22.16] OK. So in model checking, what we do is we use a special language, a language called temporal logic, to express properties of the systems that we want to investigate. So here is a property written in a standard temporal logic. It just says if ever I receive a request, then eventually I send a response. That's what that says. You don't need to worry about the details.

[00:42:42.32] And what the model checker does is it will check whether or not that property holds on some or all of these possible trajectories. So each path that you can take through that graph, right, corresponds to one possible trajectory of our system, right? And imagine that there is some flash crash trajectory where bad things are happening. So a classic example of a query would be something like, eventually, I reach a flash crash. And so what we're asking is, is there some computation in my system that will lead to that flash crash?

[00:43:15.80] So this is, again, a very big body of work. And colleagues of Manuela's at CMU won the Turing Award for their work in developing model checking technology because it's industrial strength. It really works, with all sorts of caveats. You can really use this to analyze systems.

[00:43:34.16] And many model checkers are now available. And really, the reason is for that that, actually, these model checkers are relatively easy to implement, OK? So the algorithmics of these things are really, really quite simple. And actually, you can end up with tools that really, really work if you want to do this.

[00:43:54.17] So the two basic model checking questions, then, are, is there some computation of a system on which my property-- like there is, eventually, a flash crash-- holds, or does that property hold on all computations of the system? Is it inevitable that I'm going to have a flash crash on all the possible trajectories of the system? So those are the two basic questions which are introduced in model checking.

[00:44:15.95] OK. So now, if we turn, instead, to multi-agent systems, the idea is that our agents now, each of them is trying to do the best it can for itself. My Siri is acting for me. Your Siri is acting for you.

[00:44:30.26] And what we now want to ask is, OK, under the assumption that our agents are acting rationally, doing the best that they can for us, then what properties, what trajectories, will our system take, OK? Assuming that your agent is doing the smartest thing it can in pursuit of what you want to do-- like meeting booking-- and mine is doing the smartest thing it can for me, what will happen?

[00:44:56.00] And to cut a long story short, that's the question that we ask in this work, OK? And that's the approach. The approach is that we call is equilibrium checking. It's understanding the equilibria that a system can have.

[00:45:08.76] And the basic analytical concept that we use for this-- what is a rational choice-- is an idea from game theory called Nash equilibrium, named after John Forbes Nash Jr., who just died a couple of years ago. They made a film about him, A Beautiful Mind. The film is terrible, but the book on which it's based is much better.

[00:45:27.65] And he formulated this notion of rational choice in these strategic settings. And the notion of Nash equilibrium is extremely simple. It's the idea that we use in our work for analysis. Suppose all of us are busy. We all have to make a choice, right? You have to make a choice. You have to make a choice. You have to make a choice. We, all of us in this room, make a choice.

[00:45:46.24] It's a Nash equilibrium if, when we look around the room and see what everybody else has done, none of us regrets our choice. We don't wish we'd done something else, yeah? Given that you lot did all your bits, I'm OK with what I did. But similarly, given that we all did our bits, you're OK with what you did. That's a Nash equilibrium.

[00:46:04.81] And what we look at in our system is, suppose our agents make Nash equilibrium choices, then what trajectories will result? So the picture looks very similar. We've got our model of our systems we did in model checking. We've got our claim, like there is eventually a flash crash. But now, we know what the preferences are of the agents in the system. We know what each of them is trying to accomplish. And the question is, can I get a flash crash under the assumption that we all make rational choices? Or is a flash crash inevitable under the assumption that we all make rational choices, yeah?

[00:46:45.84] So that's the work that we look at. And we have a tool that does this. It's available online. So the tool is called EVE for Equilibrium Verification Environment. It's available online at eve.cs.ox.ac.uk.

[00:47:04.69] And what you can do is you can describe a system using a high-level language called reactive modules. It's a programming language, so you should expect to see a programming language. And then, you specify the goals of each of the players. And those goals are specified as temporal logic properties, like the example that I talked about earlier.

[00:47:25.21] And what it will do is it will tell you what properties hold of that system under the assumption that all the component agents make Nash equilibrium choices. So that's what we mean by formal verification. So it's game theoretic verification.

[00:47:41.93] Because what it's doing is it's looking at the system from a game theoretic point of view. It's saying, you're going to do the best for yourself, I'm going to do the best for myself, then what will happen? We're going to make Nash equilibrium choices. What will happen in my system, OK?

[00:47:58.48] OK. So that's a formal approach to understanding equilibrium properties in a precise mathematical sense, right, in the precise mathematical sense of game theory. And it will tell us what properties will hold inevitably under the assumption of rational choice or could possibly happen.

[00:48:18.12] So the idea is, in this setting, the fact that something is possible in principle might not be relevant if it doesn't correspond to rational choices. All you're concerned about is what would happen under the assumption that our agents chose rationally, OK? So that's equilibrium analysis.

[00:48:39.32] OK. So this really only works for small systems. If you've got a handful of agents, it works. For technical reasons to do with game theory, if you've got large numbers of agents, which, of course, you do on the global markets, it doesn't really work. So what do we do instead with large systems?

[00:48:54.56] So the idea, instead, is we've got an alternative approach, which is called agent-based modeling. And to cut a long story short, what agent-based modeling does is it says you simulate the whole system. You build, literally, a model of the economy with all of the decision-makers in that economy modeling, and you model the interactions-- the buying and selling and lending behaviors, all the other stuff that you might want to do-- you model them directly, OK? And then, you run a simulation.

[00:49:25.63] And this is an old idea. But it's possible now, for familiar reasons. Why is it possible? Because we have large data sets that we can get, right? For example, in the finance sector, regulators require that banks and other groups make their data publicly available, or parts of their data, publicly available. And we can scrape that and use that in our simulations. And that's what we do.

[00:49:52.84] And we've got the compute resources to be able to simulate this at scale. So the kind of simulations that we do involve seven million decision-makers, right? And those decision-makers correspond to banks and individuals and so on, and we simulate that at scale.

[00:50:08.84] There are some challenges with this. So when you start doing agent-based simulation, it looks like a beautiful thing to do. Literally, you're modeling each of the decision-makers in the economy as an individual program that's making decisions about what to do. But actually, just getting to meaningful simulations, where it doesn't just wobble up and down crazily, never settling down, that's actually a challenge in itself.

[00:50:31.82] Once you've got simulations, you then discover that what you've done is you've plugged in what are called magic numbers. So to get anything sensible, I had to set this parameter to 13.3, but why, right? 13.3 is a magic number in the simulation. And this is a real problem. Because it just feels arbitrary. We don't want to have to do that.

[00:50:51.26] Calibration is a huge problem. So calibration means if your model tells you, this is going to happen, how do you know that $1 in the model actually corresponds to $1 in the real world? At the moment, that's the cutting edge of agent-based modeling, doing meaningful calibration. And predictably, the way that you do calibration at the moment, the state of the art technique, is to do lots of heavyweight machine learning on your model to try to understand what it's actually doing, and finally, whether you interpret the data that it's providing as quantitative or qualitative.

[00:51:26.00] So my colleague, Doyne Farmer in Oxford, uses the analogy of weather forecasting. If you go back to the 1940s, how did they do weather forecasting? They would look at the pressure over the United Kingdom, the weather patterns, and they would just go back through their records to find something similar and then look what happened the next day.

[00:51:43.85] And simulation of weather systems was widely regarded as something which was impossible for a long time. It eventually became possible when you could get the grain size of what you were modeling down to a sufficiently small area and you had the compute power available to be able to do these simulations at scale. And now, it works. And I think the claim is that we will be able to do similar things with agent-based modeling.

[00:52:09.44] But this is, it's simulation. It's Monte Carlo simulation, which means it involves random numbers, basically, right? You have to do lots of simulations and see the results that you get. And whether you interpret the results literally to give you quantitative data or qualitatively to say this trajectory could happen, you could get a flash crash under these circumstances, that's also at the cutting edge of the debate on agent-based modeling.

[00:52:35.63] OK. So I was, actually, originally planning this talk-- I was going to make this the center point of the talk. But then, I panicked because I'm not a finance person at all. And this work is only possible because we have somebody who works in the finance industry doing this. But this is a quote from his [INAUDIBLE].

[00:52:49.46] So for example, he's looking at the conditions that can give rise to flash crashes. And one of the things that he looks at, for example, is crowding. And this is where everybody is buying into a particular asset. And if that asset becomes distressed, then the concern is that that, then, propagates.

[00:53:05.18] And so the conventional wisdom is if everybody is buying into the same asset, this can be a bad thing because it leads to contagion and propagation of distress. But actually, he's discovered, for example, under some circumstances, this can be a good thing. So these are qualitative things that he's doing here. The next stage is to try to calibrate this.

[00:53:27.46] OK. So to wrap up. So the agent paradigm, it seems to me, it's a 30-year dream for AI, but it's now a reality. We all have agents with us, right? It's not necessarily the case that your Siri is talking to somebody else's Siri.

[00:53:40.14] I think that's an obvious thing, actually, to happen, right? So I genuinely believe that that will happen. And it won't happen in the sense of your Siri being generally intelligent. It will be in niche applications.

[00:53:53.11] The next step for the agent paradigm is to put agents in touch with other agents, for Siri to talk to Siri. But multi-agent systems have unpredictable dynamics. So we need to be careful about the systems that we build.

[00:54:04.71] And I've described in fairly high-level way two different approaches to doing that. One is, for small systems, you can view these things as an economic system, a game in the sense of game theory, model it as a game, and then understand what its game theoretic equilibria-- its Nash equilibrium-- behaviors are. Do I get something bad happening in the Nash equilibrium?

[00:54:23.07] An alternative is agent-based simulation, OK? So the alternative is to directly model the system. And we can do that because we have compute resources at scale, and we have data at scale. OK.

[00:54:35.52] So I'm going to wrap up. Thank you for your attention. I've giving you a tour of where agents came from, why it's a natural idea-- it took 20 years for it to become a reality, but it is a reality, and we've all got agents in our pockets now-- and where that might go.

[00:54:48.98] And that vision of the future of computing-- ubiquity, interconnection, intelligence, delegation, human orientation-- just seems to me to be inevitable. And the agents paradigm, it seems to me, is bang in the middle of that. OK. Thank you for your attention.

[00:55:02.99] [APPLAUSE]

[00:55:11.62] MANUELA VELOSO: A couple of questions?

[00:55:20.18] AUDIENCE: Test. Thank you for your time, and thank you for your talk. I have two questions. But for the sake of time, I'm going to let you choose which one to answer, if that makes sense.

[00:55:30.56] MANUELA VELOSO: No. For the sake of time, you choose one, and just say one.

[00:55:32.92] AUDIENCE: OK.

[00:55:33.38] MICHAEL WOOLDRIDGE: Choose the easy one, please.

[00:55:35.19] AUDIENCE: All right, cool. One of the hot topics in artificial intelligence is the impact of automation in society. And I am curious as to what you think the impact of the evolution of multi-agent systems will have in automation and, subsequently, how we can think about educating our children, educating ourselves, and, ultimately, educating society.

[00:56:00.24] MICHAEL WOOLDRIDGE: So I think there's two slightly conflated-- I think the issue of automation is, and how multi-agent systems will impact that-- let me just take that one first. So a lot of my job, and I daresay your jobs, is full of fairly routine management of tasks-- processing paperwork, passing it on.

[00:56:17.54] There will be any number of workflows within an organization like this, as there is at the University of Oxford, to deal with processes which involve multiple people. And they're extremely tedious and time-consuming. The first thing that I could really see agents doing is simply automating the management of an awful lot of that so that you have agents managing, in our case, for example, when a student applies to us.

[00:56:40.52] You have an agent that manages that process, can remind me when I need to do things, can make sure that the paperwork gets to the next people, can flag up to the right people when things aren't processed in time, and so on. And that seems to me to be crying out for agent-based solutions. So I think there will be big applications there in that kind of scenario. And I think that will be an area where multi-agent systems has an awful lot to offer.

[00:57:07.91] I think the second part was to do with education, was is? It that right?

[00:57:10.37] AUDIENCE: Yeah, how to fine-tune education to overcome automation.

[00:57:17.18] MICHAEL WOOLDRIDGE: Well, I think that's a bigger question about AI itself rather than multi-agent systems. And I think the answer to that one is that the skills that won't be easily automated are human skills. Doctors, for example, they're not going to be automated by X-ray reading machines. So we have software that can read X-rays and diagnose heart disease and so on very, very effectively.

[00:57:39.10] But that's a tiny sliver of what a doctor does. An awful lot of what a doctor does-- most of what a doctor does-- is to do with human skills that requires huge amounts of training which are not going to go away any time soon, although I'm put in mind of this news article that you may have seen over the last 24 hours of this patient that was told he was going to die by a telepresence robot. That wasn't AI, by the way. It was just a crass application of telepresence technology.

[00:58:04.98] MANUELA VELOSO: OK. So there's one more question.

[00:58:07.41] AUDIENCE: Just a quick question.

[00:58:08.38] MANUELA VELOSO: Just a second. Just a second.

[00:58:14.45] AUDIENCE: How would you suggest we should deal with biases, applying databases full of old bias and inefficiencies? So how, by delegating to agents, how do you deal with biases in the data?

[00:58:25.62] MICHAEL WOOLDRIDGE: OK. So wow. That's a huge question. So I think what's interesting about bias is that the algorithmic treatment about bias-- and it's not just AI, anything to do with an algorithm which has to make decisions, it's the same thing-- but the algorithmic treatment of bias is something we didn't really anticipate there's going to be an issue. And it's just exploded over the last couple of years.

[00:58:47.85] So there, I think, we're just developing the science to understand what it means, for example, to be able to say in a precise sense, when is a data set biased? When is an algorithm unbiased or biased? We're just getting there. And people are frantically running ahead to try to understand those issues.

[00:59:04.46] And I'm pretty confident over the next 10 years, we will have a much richer understanding of that. What will be interesting is to see that experience fed back into undergraduate programs, for example, so that when we teach people about programming, we don't just teach them about programming. We teach them about those issues of bias.

[00:59:22.01] At the moment, it's a huge, great, difficult area. And there are no magic fixes for it. We're just at the beginning of a process to understand what those issues really are.

[00:59:32.79] MANUELA VELOSO: OK. We'll have two more questions. I see many hands up. OK. You have the mic. So then we select him for that time.

[00:59:41.59] AUDIENCE: Thank you very much. And thanks for the talk. The question you spoke [INAUDIBLE] about defining what's meant to be each agent's interest because originally, once we define what each agent is doing in its best interest, then you can define the Nash equilibrium. But

[00:59:55.48] In real-world applications, agents may have different interests, right? So what would be the two, three topics, or methods, that you think are the state of the art or inferring what is the reward function or the actual interest of each agent so we can model each agent and then go to multi-agent level?

[01:00:13.55] MICHAEL WOOLDRIDGE: OK. So the slightly technical answer, so I apologize for that, so the problem of I'm interacting with people, and I don't quite know what their preferences are-- they could be this sort of person, they could be this sort of person-- is a standard one in game theory. And there are standard tools.

[01:00:29.34] There's a variation of Nash equilibrium called Bayes Nash equilibrium, which deals with exactly that. We haven't done that in our work because it's an order of magnitude more complex. But nevertheless, it's a standard technique. And in principle, you can use that technique to understand this.

[01:00:43.51] You could then argue against the models that they use in-- what they do in Bayes Nash equilibrium is you've got a space of possible types. You could be this type or this type or this type of person-- in other words, have these preferences or these preferences or these preferences.

[01:00:56.44] And what you know is the probability that they're of this type and this type and this type. And you could immediately argue, well, actually, even that, actually, is asking quite a lot. On large-scale systems, that might not be an unreasonable thing to do. But for the reasons I've said, we don't look at large-scale systems using game theory.

[01:01:12.31] MANUELA VELOSO: We have [INAUDIBLE]. So one final question there.

[01:01:18.70] AUDIENCE: I have a question about agent-based modeling. So take the example of the flash crash. Statistics say this is something that will happen once every billion years or something. Do you think that there is some issues with the way we do agent-based modeling or the reliance on simulations to model whether things are likely to happen or not?

[01:01:40.03] MICHAEL WOOLDRIDGE: So is that an actual quote, the once every billion years? Because it seems a very silly quote, given that it's actually happened.

[01:01:46.39] AUDIENCE: Yeah, something like that. Usually, these events are super rare [INAUDIBLE] risks.


[01:01:51.97] MICHAEL WOOLDRIDGE: Well, there have been smaller-scale flash crashes since then, right? There've been a number of them. So the scary thing about flash crashes, if they happen in the circumstances where, for example, like I say, at the end of the trading day, when you hit the trough when the markets closed, that's what's potentially very scary. And there could be others, right-- there could be other circumstances.

[01:02:13.00] So in our simulations, what we aim to do is, at the moment, we're just getting qualitative indications. Look, these are the kinds of conditions-- because these are hugely complicated events, it's not just one factor. Certainly, high-frequency trading, the flash crash couldn't have happened without high-frequency trading.

[01:02:32.02] So that's certainly a contributing factor, but by no means is it the only one. And actually, I'm not sure whether they got to the bottom of whether or not somebody had actually done something fraudulent in the flash crash. I'm sure some of you know the answer to that. But yes, so there's a huge range of things.

[01:02:44.71] But what we aim to do is be able to give you the kind of characteristics. Look, if your system has these properties, this is the kind of trajectory that it might exhibit under these circumstances-- so qualitative indicators, which is still useful for us. As I say, we're right on the frontier of going from that to be able to make-- if your leverage is this much and the crowding is this much, then the probability is this much. We're on the edge of being able to do that. We can't do that with confidence yet. That's a way off, I think.

[01:03:16.63] MANUELA VELOSO: OK. We'll have one final question. There was-- there in the back. Sorry. We also have in the back row [INAUDIBLE]. OK. Thank you.

[01:03:30.31] AUDIENCE: Hi, professor. When I hear you talk about the agents, and I see you show the video from Apple, I'm curious whether you see agents as a solution to the IO problem when dealing with computers. Do you see it as the next step towards, or the next trend towards, people interface with computers?

[01:04:00.12] Obviously, now, we see companies in America like CTRL-labs. And in their work, they're looking at gestures, reading the nerve signals to interact with computers-- one of the big revolutions in the industries.

[01:04:15.08] MANUELA VELOSO: [INAUDIBLE] a question.

[01:04:17.18] AUDIENCE: So yeah. The question is, do you see it as a solution to the IO problem, or is it a bigger application than that?

[01:04:23.11] MICHAEL WOOLDRIDGE: So yes, it's a problem. It's a solution to the human-computer interface problem, which, I think, is what you're saying. You're talk about input/output problem, is that right? So I think it's a solution to the human-computer interface problem.

[01:04:36.09] The reason that so many people are working on it is because, at the moment, they don't see any alternative. So gesture-based interface is certainly going to have a role to play. I think they're not on the stage at the moment, even remotely, where they could be rolled out.

[01:04:48.99] And I think that it's hard to, with a gesture-based interface, to think about booking a meeting with Manuela. It just seems easier to say that than it does to try and do something with gestures. Maybe somebody will come up with something innovative there. I don't know. But at the moment, I think gestures-based interfaces are very, very niche area.

[01:05:09.63] Brain reading, I think, is-- again, we're nowhere near being able to do anything like book me a meeting with Manuela through brain reading. I think the state of the art there is one or zero, and possibly something a little bit more sophisticated but not much.

[01:05:26.68] So at the moment, I say the reason that people are chasing that up is because they just don't see any alternative, right, for an awful lot of these systems. If you're driving a car, how are you going to interface? You can't take your hands off the wheel and start typing. And you certainly can't do gestures, right? So it's the only alternative that's there.

[01:05:43.79] MANUELA VELOSO: Very good. So let's thank Professor Wooldridge again.

[01:05:46.68] [APPLAUSE]

[00:00:00.30] MANUELA VELOSO: OK. Thank you very much for coming, and welcome to the JP Morgan distinguished lecture on AI. It's my great pleasure, my very great pleasure, to just say a couple of words to introduce you to our speaker today who is Professor Yann LeCun. So Yann LeCun currently has appointments both at NYU and Facebook.

[00:00:29.76] Yahn is the silver professor of computer science and neuroscience at NYU, which he joined in 2003 after actually having been at Bell Labs having a group on image processing at Bell Labs, and also having done a postdoc at University of Toronto, and after his PhD, which was in Paris at Pierre and Marie Curie University. So Yann has this academic kind of life, and in about, I think in 2013, he took another hat which was to head and create the Facebook AI research at Facebook.

[00:01:15.69] He's currently a VP and chief AI scientist at Facebook, and he co-founded and directed, actually the FAIR, the Facebook AI Research. And in terms of research, we all know that his passion, his contributions have been at the deep learning level, machine vision, a lot of knowledge on mobile robotics and AI in general. So finally, you know we should also know that our speaker Yann LeCun is a member of the National Academy of Engineering, which is one of the highest honors for researchers in the United States. So please, let's welcome Professor Yann LeCun, and welcome him to JP Morgan.

[00:02:03.26] [APPLAUSE]

[00:02:08.50] YANN LECUN: Thank you, Manuela. A real pleasure to be here. I rarely go above 23rd Street, really, in New York but--

[00:02:14.62] [LAUGHTER]

[00:02:15.61] That gives me an excuse. I guest I didn't have to fly, which is good. I spent about half my career in industry and half in academia. Now, I have one foot in each. I guess it's one half of a foot in academia now because I'm spending most of my time at Facebook, but things changed. I basically started the first research lab at Facebook, FAIR. There's a bunch of research labs now at Facebook, but this was the first. And this was a bit of a cultural shock for Facebook, which was a very sort of engineering short-term oriented company that had to invent itself a new culture for research. And that's what I liked about it, the fact that we could start from scratch and basically establish this culture.

[00:03:04.83] So what we're doing at Facebook is open research. Facebook FAIR research is really outward focused, and all the research we do is published, and almost all the code that we write is open sourced. With that we hope to foster interesting problems that we think are interesting, and see the creativity towards working on provenance that we think are important. The thing here is not whether Facebook technology is ahead of Google's or Microsoft's or whatever, but more the fact that the products that we want to build are not possible today. We don't have the science, even the basic principles, to build the stuff we want to build. And so since we don't have a monopoly on good ideas, the best we can do is basically accelerate progress for the entire community, and that's one of our roles.

[00:03:53.96] Of course, we have a big impact on the company. In fact, a much bigger impact than Mark Zuckerberg thought we would have five years ago when FAIR was created, and today Facebook is entirely built around deep learning. If you take deep learning out of Facebook today, you get dust essentially. Not entirely, but you know--

[00:04:14.19] [LAUGHTER]

[00:04:14.97] You know what I mean. OK. So most of machine learning today-- Machine learning has had a huge impact in various areas of business and society, and science, certainly, but most of the applications of machine learning today, basically, use supervised learning, one of the three main paradigms of learning. So there is supervised learning, there is reinforcement learning, which people have been talking about a lot in the last few years, and then there is another thing that's not very well defined called unsupervised learning or self supervised learning, which I'll talk about later.

[00:04:51.88] So supervised learning is this idea by which if you want to train a machine, for example, to classify images of cars from airplanes you show an image of a car, you run this through the machine, and the machine has adjustable knobs on it. And if it says car, you don't do anything to the knobs. If it doesn't say car, you adjust the knob so that the answer the machine produces gets closer to the answer you want. There is a desired answer that you give to the machine, and you can measure the discrepancy between the answer you want and the answer the machine produces.

[00:05:21.51] And then you show an image of an airplane, and you do the same. And by tweaking the parameters with thousands of examples, eventually, perhaps the knobs will converge to a configuration where all the cars are correctly classified, and all the airplanes are currently classified. And the magic of this is that it may even work for airplanes and cars it's never seen before. That's the whole purpose of learning, which is that you learn the concept without having to memorize every example.

[00:05:50.61] So this type of learning, supervised learning works really well if you want to do speech recognition, that being speech to words, images to categories, face recognition, generating captions for photos, figuring out the topping of a text, translating from one language to another, that's all supervised learning.

[00:06:09.38] And that's the basic idea of this, or the tradition, starts with models from the late '50s, early '60s, the Perceptron and the Adaline, which interestingly at the time, were really sort of hardware devices. They were not programs on the computer. They were actually analog computers that were built. At the bottom here is the Perceptron. What you see here is Bernie Widrow reviving one of his old Adeline systems at Stanford.

[00:06:38.49] So that created the standard model of pattern recognition that really was prevalent until fairly recently, by which you take a rough signal and you feed it to what's called a Feature Extractor, which is hand engineered. It's designed by people to basically extract relevant information from the raw signal, and then you see the result, the feature vector, to the classifier, something like an [INAUDIBLE] classifier or [INAUDIBLE] or the tree or whatever. There's a lot of techniques that people have come up with over the last 50 years, if not 60. More like 60 years, actually to do this. And that's kind of the standard way.

[00:07:20.33] And what deep learning changed is the idea that you can learn this feature extractor. So instead of having to engineer-- spend a lot of time, and expertise, and money on building those things for every new problem, you can basically train the feature extractor as part of the entire process. So basically you build a machine as a cascade of trainable modules, which you can call layers, and essentially all those modules are [INAUDIBLE] using supervised learning. And you hope that the machine will learn, not just to classify, but also to figure out what the relevant features are that need to be extracted at every layer for the system to do a good job.

[00:08:06.00] So the next question you can ask is, what do we put in those boxes? And the answer to this is not recent. It's the idea of artificial neural networks. So in a artificial neural net, essentially the layers are of two types. One type is just a linear operator. So imagine that the signal is represented as a vector, in our recent numbers, the pixel values or the signal, whether it's audio or financial tensor or whatever.

[00:08:36.90] Represent this as a vector, multiply this by a matrix. So what a matrix does is that when you computed the product with this matrix by this vector is you computing the dot product of the vector by every row in the matrix, and that's like computing a weighted sum of input features. It's actually represented here. So you have a bunch of-- Here's a vector. With components of this vector, you're computing a weighted sum where the weights are the coefficients in the matrix, and that gives you an output.

[00:09:05.30] And then there is another type of function here, which is a point-wise nonlinearity. So you take this vector and then apply the nonlinear function to every component of this vector independently. In this case, what's called a value, which is really just a halfway rectifier, OK?

[00:09:17.76] So function is identity for positive arguments and equal to 0 for negative arguments. Very simple nonlinearity. And there are theorems that show that with only two layers of this-- so linear, nonlinear, linear-- you can approximate any function you want as close as you want, as long as the dimension of this vector is sufficiently large, possibly infinite. But-- and there is no real theoretical results on this. But what we know empirically and intuitively is that by stacking lots of layers of those, you can represent many functions very efficiently. So that's the whole motivation to deep learning, which is that by stacking multiple layers of alternating linear nonlinear operators, you can approximate a lot of useful functions very efficiently.

[00:10:03.57] AUDIENCE: [INAUDIBLE] in theory, prove that [INAUDIBLE]?

[00:10:16.20] YANN LECUN: What do you mean by simple?

[00:10:17.07] AUDIENCE: Simple function is [INAUDIBLE].

[00:10:26.31] YANN LECUN: Yeah, yeah. So yeah, the theorems from the late '80s that show that with just two layers-- with just one layer or nonlinearity, you can approximate any function you want, OK? But there is no limit on the dimension of the middle layer. So for the longest time, because of the limitations of computer power and because the data sets were small, those approaches-- the number of applications that we could apply these two were very limited. I mean, the basic techniques for this are from the late '80s, but the amount of problems for which we could have enough data to train the system was very small. We could use it for maybe handwriting recognition and speech recognition and maybe a few other applications, but that was kind of limited.

[00:11:10.38] People at the time, actually, in the '80s, were really interested in hardware implementations and this is coming back. So there is a whole industry now that has been restarted over the last three to five years on building special-purpose chips to run those neural nets efficiently, particularly for embedded devices. So probably within the next two years, every smartphone will have a neural net accelerator embedded in it. And within five years, it'll be in every car and shortly after that, basically every electronic device that you buy will have a neural net accelerator in it. And so your vacuum cleaner will have smart computer vision in it because it will have a $3 chip that does neural net acceleration.

[00:11:56.10] So how do we train those things? So we train them by-- it's basically large scale optimization. So the supervised learning. You measure the discrepancy between the answer the machine produces and the answer you want through some sort of objective function that basically just measures the distance or something like that of some kind. Then you can average this over a training set of samples of pairs of input and output, and the process by which you tune the parameters of the system is just gradient descent. So figure out in which direction to change all the knobs so that the objective function goes down, and then take a step in that direction. And then keep doing this until you reach some sort of minimum.

[00:12:39.04] So what people use is something called Stochastic Gradient Descent, where you estimate the gradient on the basis of a single sample or maybe a small batch of samples. So you show a simple example, figure out the error between those two things. Then compute the gradient of that function with respect to all the parameters, tweak the parameters, then go to the next sample. Stochastic because you get a noisy estimate of the gradient on the basis of a single sample or small batch.

[00:13:05.14] So the next question you might ask is how do we compute this gradient? And that's where back-propagation comes in. And I'm sure many of you are familiar with this. Don't attempt to understand the formula. You don't need to.

[00:13:16.29] But it's basically the idea that to compute the gradient, which is really the sensitivity parameters of the cost function with respect to all the coefficients in the system-- all the weights and all the matrices in the weighted sums-- you can compute all of those by doing a backward pass, which is basically just a practical application of chain rule, OK? So you know that by tweaking a parameter in this block here, it will affect the output in a particular way. And you know how tweaking this output affects the overall cost. And so it's very easy, by using this back-propagation method, to compute all the terms in this gradient.

[00:13:57.40] So now for every parameter in the system, you have a quantity that indicates by how much the cost will increase or decrease if you tweak the parameter by some given delta. So that gives you the gradient. Take a step in the negative gradient direction. There's various tweaks to make this fast.

[00:14:19.65] So what made deep learning possible and what makes it easy to use is you don't have to figure this out at all. The modern deep learning frameworks, you basically build a network either by writing a program in Python or your favorite language, or by assembling blocks that have been pre-defined into a graph. And the system automatically knows how to compute the gradient. So you tell it how to compute the output and it keeps track of all the operations that are done during this computation, and then can trace back those operations so that, automatically, the gradient of whatever it is you're computing, with respect to all the parameters you have, will be computed. So that's a very simple concept, automatic differentiation. But that's really what makes deep learning so easy to deploy and use.

[00:15:07.95] OK, now here is a problem. If you want to apply deep learning or neural nets into the way I describe them to images, it's not really practical to view an image as a vector where the components of the vectors are the pixels. Because if you have an image, let's say, 200 by 200 pixels, that's 40,000 pixels and if you multiply this by the matrix, that matrix would be 40,000 by something. It's going to be large, OK? Too large.

[00:15:40.68] So you have to figure out how to specialize the connection between the neurons or basically, how to build sparse matrices in such a way that the competition becomes practical. That's the idea of convolutional networks, which is something that my name is associated with. The inspiration for this goes back to classic work in neuroscience from the '60s. Actually, it's Nobel Prize winning work by Hubel and Wiesel on the architecture of the visual cortex. There were various people who tried to make computer models of this, but didn't have things like back-prop.

[00:16:11.88] So what's the idea behind convolutional net? You take an image and the operation you're going to do, the linear operation, is not going to be a full matrix, but it's going to be what's called a discrete convolution, which consists in taking a little patch of the image-- 5 by 5, in this case-- and then computing the weighted sum of those pixels with a set of 25 weights. And then putting the result in a corresponding pixel on the output, OK?

[00:16:38.15] And then you take that window, you shift it over a little bit by one pixel, and do the same. Compute the dot product or the weighted sum of the pixels by those coefficients and record the result next to it, OK? So by swiping this window over, you get an image of the output, which is a result of convolving this image by this so-called convolution kernel.

[00:17:00.38] So the number of free parameters here in your matrix is very small. It's only 25, in this case, OK? And the amount of competition is relatively small. And the advantage of using this kind of competition is in situations where the signal comes to you in the form of an array-- single or multidimensional array-- in such a way that the statistics are more or less stationary. And in situations also that neighboring values tend to be highly correlated, whereas far away values are not, or less. It's the case for Financial Times series, for example, that belongs to this, right? And people have been using convolutional nets on Financial Times series. Yes.

[00:17:39.01] AUDIENCE: [INAUDIBLE]

[00:17:40.17] YANN LECUN: Say it again?

[00:17:40.72] AUDIENCE: [INAUDIBLE] like charter spanning tree and things like that?

[00:17:44.67] YANN LECUN: So there certain configurations of those coefficients that will produce edge detection, yes. But we're not going to hardwire those coefficients. They're going to be the result of learning. We're not going to build it by hand. We're just going to initialize them randomly and then train the entire thing end to end supervised to produce a right answer at the end on millions of examples, or thousands of examples. And then look at the result.

[00:18:10.97] OK, so that's the first layer. We're going to have multiple filters of this type. In this case here, four. So each of those-- you had four filters here, producing four so-called feature maps. And then there's a second type of operation here called pooling, which consists in taking the results of those filtering small neighborhood and pooling the results, which means computing an average or a max of the values. And then sub-sampling the image. So sub-sampling means that this image is half the resolution of that image. The pixels are twice as big if you want.

[00:18:46.97] The reason for this is to eliminate a little bit of position information about the location of features in the image. That's important if you want to have a system that is robust to small deformations of the input.

[00:18:59.52] So this is a convolutional net in action. It's been trained to recognize handwritten digits. And it's showing the output here. And this is the input first layer. After pooling, third layer. Another layer of pooling, then yet another layer. By the time you get here, the representation is very distributed and abstract.

[00:19:18.08] But every unit here is essentially influenced by the entire input, OK? So the entire input and the representation of the input is this list of those values. You can get those things to recognize not just single characters, but multiple characters. And do simultaneous segmentation. This is really important because eventually, you'll want to use those systems with natural images. So this is sort of vintage, early '90s convolutional net, which was built when I was at Bell Labs.

[00:19:49.32] Eventually, at Bell Labs, we built a check reading system based on those convolutional nets and various other tricks. And by the-- it was deployed in the mid-'90s, and by the end of the '90s, it was reading somewhere between 10% and 20% of all checks in the US. So a big success. But by that time, the machine learning community had lost interest in neural nets. Nobody was working on neural nets, essentially, in the late '90s until the mid-2000s, roughly.

[00:20:17.19] I left industry in 2003 and joined NYU, as Manuela was mentioning. And I wanted to reignite the interest of the community for those methods because I knew they were working. And they had a reputation of being very finicky, right? We had our own framework for deep learning, but nobody was interested in this, so nobody was using our code. And neural nets had the reputation of being so hard to train that only I and a few people working with me were able to train them, which of course, was not true. It's just that people are lazy.

[00:20:53.97] [LAUGHTER]

[00:20:56.71] I'm being facetious here. So around 2003, 2004, just when I joined NYU, I got together with my friends Yoshua Bengio at University of Montreal and Jeff Hinton at the University of Toronto, where I had done my post-doc many years before. And we decided to basically start a conspiracy to renew the interest of the community in neural nets. And we started with various algorithms that we thought would enable back-prop, perhaps, to train very, very deep networks. So not networks with just three or four or five layers, but networks, perhaps, with 20 layers or something like that.

[00:21:37.23] And we started working with unsupervised learning algorithms, which were only partially successful, but they were successful enough to get enough interest from people that the community started building itself. Around 2007, there was enough of a community that our paper started to get actually accepted at [INAUDIBLE]. Before that, we could never publish a paper in any conference on neural nets, essentially.

[00:22:00.18] And then we started getting really good results on standard benchmarks, but they were still kind of dismissed to some extent. That changed around 2009, 2010 in speech recognition, where the results were so much better that people started really switching to using neural nets. And then, around 2013 in computer vision, and the history of this is well-known.

[00:22:21.38] But in the meantime, starting in the mid-2000s-- I'm hearing myself now. I started working on robotics. It was something that, Manuela, I was very familiar with. A project that, actually, that Tucker Bosch was involved in as well. He's now at JP Morgan. He was at Georgia Tech at the time. Still at Georgia Tech. And this was a project to use machine learning to get robots to drive themselves in nature. [INAUDIBLE] roughly between 2004, 2005, 2009.

[00:22:55.86] So the idea here was to use a neural net basically to do what's called semantic segmentation, which means to label every pixel in an image with a category of the object it belongs to. So it's using a convolutional net, which sees a band around the horizon of the image. And it's trained to produce another image, which is this image that has essentially three categories.

[00:23:18.48] Here is something I can drive over. I'm going to label it green. Or is it something that is an obstacle? My video is not working for some reason. That's interesting. OK.

[00:23:32.36] Oh, here we go. Oh, that's fun. All right. Oh, wow. OK. All right, this one is working. So this is an example of semantic segmentation that took place a couple years later, around 2010 or 2009, where there were data sets with a few thousand images where people had painfully labeled every pixel with the category of the object it belongs to. So things like road and sidewalk and cars and pedestrians, trees, et cetera.

[00:24:10.41] So we trained this convolutional net to be applied to the entire image and it basically labels every pixel with a category. It makes mistakes. It labels this as desert. This is the middle of Washington Square Park.

[00:24:25.62] [LAUGHTER]

[00:24:27.21] There is no beach I'm aware of. But at the time, it was state of the art. It was, in fact, quite a bit better than the state of the art at the time. This was 2010. It was also 50 times faster than the best runner-up competing technique. So we submitted a paper to CDPR, the big computer vision conference, at the end of 2010. Pretty sure that the paper was going to be accepted because it was faster and better than everything else people had done before. And it was rejected by all three reviewers, who said, basically, what the hell is a convolutional net? And we don't believe that the method we never heard of could do so well, so this could be wrong. Essentially, that's what the reviewers said. And it's funny because now you can't actually have a paper accepted at CDPR unless you use convolutional nets.

[00:25:23.88] Oops. That's not what I wanted you to do. Sorry about that. Bear with me for just a second. OK. So con nets are really useful a lot of things today, for self-driving cars. Every self-driving car project has a convolutional net in it. And for all kinds of other things. I gave a talk in 2013 that gave some ideas to people at MobilEye, which now belongs to Intel. Also to NVIDIA. They're using convolutional nets for all their self-driving car projects.

[00:26:04.28] In fact, there is a self-driving car project taking place in the Holmdel building, which is the building where I used to work at Bell Labs by a group from NVIDIA. And the guy running this project at NVIDIA is actually a former colleague from Bell Labs who worked with us on this robotics project that Tucker was involved in.

[00:26:22.06] OK, so deep learning today. It was a revolution in 2013 in [INAUDIBLE] division because our friends at University of Toronto, in Jeff Hinton's group, figured out how to implement convolutional nets on the GPU in a very efficient manner. They're one of the first ones to do this. It was done at Microsoft in the mid-2000s. But they applied this to image net and managed to get results that were so much better than what people were doing before that. That really created a bit of a revolution.

[00:26:50.79] So this was the error rate on ImageNet that people were getting in 2011. And in 2012, with the so-called AlexNet project from Toronto, the error rate went down by a huge amount. And then, over the last few years, it went down to levels that are so low that now this benchmark is actually not interesting anymore. It's better than human performance on this particular data set.

[00:27:12.26] What we've seen simultaneously is an inflation in the number of layers in those networks. So the [INAUDIBLE] I showed you from the '90s earlier had seven layers. The one from 2013, one of the ones that worked best, had 20 layers. And now that the best ones have anywhere between 50 and 150 layers.

[00:27:33.13] And Facebook uses those convolutional nets very widely for a lot of different things. And one of the most popular ones that's used in production is something called ResNet50. So ResNet is this particular architecture that is here, where there are layers of convolutions and pooling and nonlinearities, but there's also skipping connections that allow the system to fail gracefully. If some layers don't learn appropriately, then they become transparent essentially. And so that is what enables us to train very, very deep network. This is an idea that came from [INAUDIBLE], who was, at the time, at Microsoft Research Asia, who is now at Facebook.

[00:28:15.16] And so that's a graph that was put by Alfredo Canziani, who's a postdoc with me at NYU. But he did this before he came. On the y-axis, you have the accuracy. On the x-axis, the number of operations, which is billions of operations that are necessary to compute one output. And what people have been trying to do in industry is bring everything down to this corner essentially, where for the minimum amount of computation, you have the best accuracy on ImageNet or similar things.

[00:28:44.90] And so ResNet50 is right here. There are better results now. And then the size of the bubble, if you want, is the memory footprint, the number of parameters that are necessary. There's a lot of work on optimizing running those networks on regular processors or specialized processors to save power. And the reason this is important is-- to give you an idea, Facebook users upload somewhere between 2 and 3 billion photos on Facebook every day. And this is just on the blue site. I'm not talking about Instagram or anything like that, just Facebook.

[00:29:22.37] Every single one of those three billion photos goes through six convolutional nets, roughly-- half a dozen-- within two seconds of being uploaded. And those do things like essentially represent the image into a feature vector that can be used for all kinds of things-- retrieval search, indexing, feature vector for other purpose-- generic feature vectors, if you want. And one that does face recognition and face detection. Another one that generates captions, that describes the images for the visually impaired. And there is a couple that basically detect objectionable content-- nudity, violence, things like that.

[00:30:08.10] So the advantage of deep learning is that the system basically spontaneously learns to represent images in a hierarchical way from low-level features like edges, to parts of objects and motifs and things like that. One trend over the last few years is the use of weakly-supervised learning, or semi-supervised learning. This is weakly-supervised learning.

[00:30:32.03] So this is an experiment that was done at Facebook with one of the applied computer vision groups which consisted in taking 3 and 1/2 billion images from Instagram and then training a convolutional net to predict the hashtags that people tag images with. So they decided on about 17,000 different hashtags that would correspond to physical concepts, if you want. And then run 3.5 billion images through the convolutional net, asking you to predict which of those 17,000 hashtags is present.

[00:31:09.09] Then you take this network, chop up the last layer that predicts the hashtags, and just use the second last layer as a feature vector, which is an input to a classifier that you train on other tasks, like say ImageNet. And you can actually beat the record on ImageNet this way, OK? So until fairly recently-- it was actually beaten by another team at Facebook-- but until fairly recently, the record on ImageNet was held by this system, which is trained on a different task than the one you actually finally trained it on.

[00:31:37.76] So that shows points towards something that is going to become more and more important in the future, which is this idea that you pre-train with lots of data in a relatively task-independent way. And then you use this relatively small amount of data to actually train your system to solve the task you want to solve. I'll come back to this afterwards.

[00:31:59.04] So a lot of progress over the last few years in computer vision using convolutional nets with-- I'm not going to go into the details of how this is built, but you can get results like this, where every object in an image is outlined and identified. This is called instance segmentation and it can detect wine glasses and backpacks and count sheep. And it's optimized-- my videos aren't running for some reason.

[00:32:32.52] It's optimized to the point that you can run those things in real time on smartphones. So this is unfortunately a video that you can't see. And I'm not sure why you can't see it. Wow. It disappears. Which is a person essentially being tracked-- [INAUDIBLE] is being tracked on a smartphone in real time at something like 10 frames per second. So a lot of work has gone into those optimizations to run on small platforms and on iPhone, also, you have accelerations libraries.

[00:33:08.90] This is all open source, so if you want to play with computer vision, the latest systems, you can just download this. This is using the PyTorch framework, which also was developed at Facebook. And there is similar things for tracking body poses densely.

[00:33:24.27] ConvNets are used for all kinds of stuff-- in medical imaging. It's actually one of the hottest topics now in radiology, which is how you use deep learning for analyzing medical images. This is a project which I'm not involved in, but it's colleagues at NYU at the medical school and in the computer science department who have been developing those architectures for analyzing MRI images of hips and getting really good results with this. So this is a really hot topic now and it's probably going to have a big effect on radiology in the future.

[00:33:58.54] OK, so I don't want to do a laundry list of [INAUDIBLE] convolutional nets. This is one that was also developed at Facebook for translation. It's a little complicated to explain here with so-called gated convolutional net. But basically, the input is a sequence of words and the output is also a sequence of words in a different language [INAUDIBLE] translation. And that goes through convolutions that include something called attention circuits. And there is some sort of module in the middle that tries to match the-- warp the sequence to the-- so that words appear in the right place in the output sequence.

[00:34:39.74] This had the record on some data set for a short time. It's since been overrun. And you can use it for sound generation or for sequence generation here. So this is generating synthetic sounds by specifying what type of sound you want. This is a project that was done at Facebook in Paris.

[00:35:04.20] And interesting projects in unsupervised learning for translation. So this is a project also that was done in Paris, partly in New York, where you feed a so-called unsupervised embedding system. So you can learn vector representations for words in a language by figuring out in which context they appear. This is a very classic technical [INAUDIBLE]. This doesn't use [INAUDIBLE]. This uses something different, but it's very similar. So with this technique, completely unsupervised manner, you give a big corpus of text in one language to a system and it figures out the vector representation for each word in such a way that similar vectors correspond to similar words essentially, depending which context they appear in.

[00:35:42.20] You do this for several languages, and then you ask the question, is there a simple mapping that will take the cloud of points corresponding to all the vectors in one language and transform it into the cloud of points of another language? And if you can find such a mapping, there is some chance that you'll find a mapping between the two languages, and this actually works. And so what this allows you to do is basically build a translation system from one language to another without having any parallel text for those two languages, which is dumbfounding to me, but it works, OK?

[00:36:15.33] It doesn't give you record-breaking results if you had data, but it's amazing. It's very important for Facebook because people use thousands of different languages on Facebook. In fact, we just open sourced something which is actually not this particular project. This project is open source too, but where we provide embeddings for words and sentences in various languages-- in 92 different languages, actually. That's all open source.

[00:36:44.84] Oh, that's nice. OK. All right. OK. Question answering. I'm going to skip this. OK, so a lot of applications of deep learning and convolutional nets. A whole new set of potential applications that are trying to pop up, which are enabled by a new type of neural net, which instead of being applied to basically multidimensional array data-- things like images or things like that-- you can now apply neural nets to graph data. So data that comes to you in the form of a graph with values on it. Function on a graph, if you want. And the graph doesn't need to be static.

[00:37:33.92] I'm going to point you to a review paper that I'm a distant co-author on. Geometric Deep Learning, Going Beyond Euclidean Data. So this is the idea of how you can define things like convolutional nets and things like this on data that is not an array, but basically a function on a graph. And the cool thing about this is that you can apply this to social networks, regulating networks, networks of financial instruments, let's say, 3D shapes, functional networks in biology, things like that.

[00:38:02.03] There's essentially three types. The classical ConvNets where the input is known. It's a function on a grid, if you want. Like an image, for example, you could think of as a function on a grid. Things where the graph is fixed. For example, the graph of interactions between different areas of the brain. But the function on the graph is not fixed. And so you'd like to apply convolutional nets to domains of this type. How do you define a convolutional on such a funny graph?

[00:38:32.57] And then there are applications where the graph changes for every new data, right? So for example, the data point could be a molecule, and molecule is best represented as a graph. Can we run a neural net on the graph? And the answer is yes. And I think this whole area opens an entire Pandora's box of new applications of neural nets that are [INAUDIBLE] unforeseen, and so I think it's really cool.

[00:38:57.80] Last year, I co-organized a workshop at IPAN-- the Institute for Pure and Applied Mathematics-- at UCLA on new techniques in deep learning and there was a lot of talks about this. So if you want to learn about this, that's a good way to get started.

[00:39:12.99] OK, now, there's been a lot of excitement about reinforcement learning, particularly deep reinforcement training, in the last few years. Everybody has Alpha Go and things like that. And reinforcement learning works really well for things like games. So if you want to train a machine to play Doom or play Go or play chess-- StarCraft not so much yet-- reinforcement learning works really well.

[00:39:37.84] So reinforcement learning is the idea that you don't tell the machine the correct answer. You only tell the machine whether it did good or bad, right? So you let the machine produce an answer, in this case, an action or an action sequence. And then you tell it you won or you lost, or you did good, you gain points, or you didn't.

[00:39:55.34] So it works amazingly well except that it requires many, many, many interactions with the environment. So a few people were thinking that-- so it works really well for Go, for example. So this is a Go system that is actually produced at Facebook, which is similar to AlphaGo and Alpha0, which works at superhuman level and everything. We're working also on a similar project for StarCraft where we train a StarCraft agent to win the battle.

[00:40:27.36] The big problem that I was just mentioning is that reinforcement learning is very inefficient, in terms of samples. So this is a figure from a recent paper from DeepMind where they measure as a function of the number of millions of frames that the system sees for playing an Atari game. So this is a classic Atari game from the 1980s. Using the best algorithms, it takes roughly 7 million frames to reach a performance that humans will reach in a few minutes. And that corresponds to something like 100 hours of play if you translate this into real time. So these systems are much, much slower than humans-- or animals for, that matter-- at learning new skills.

[00:41:15.74] And that's why they are not really practical for real world application for which there is no gigantic amounts of interactions that are accessible. So if you want to use reinforcement learning to train a car to drive itself, it's basically not going to work in its current form. Machine will have to drive off a cliff several thousand times before it figures out how not to do it, OK?

[00:41:40.32] [LAUGHTER]

[00:41:42.65] Now, how is it that humans are able to learn to drive a car in about 20 hours of training without crashing? It's kind of amazing. This would require hundreds of thousands, if not millions, of hours of training to get a car to drive itself. You could do this in simulation, but simulations are not accurate and people are working on how you can transfer from simulation environment to the real world.

[00:42:07.79] Yeah, this is just in passing, a list of major open source project that Facebook research has put out. So PyTorch is the environment we use for deep learning. FAISS is a very fast similarity search library for nearest neighbor. It's very useful. This is used everywhere within Facebook. This stands for dialogue. There's this reinforcement learning framework for Go. OpenGo is the system I just mentioned. FastText for natural language understanding. FairSeq for sequence processing, things like translation and things like that. There's a whole bunch of projects coming up. You can get them all from this GitHub here, github.com/facebookresearch.

[00:42:48.47] OK, so obviously, you know, we can't get our machines to run as fast as humans, so we're missing something really essential here to get to real AI. And in my opinion, we're missing three things. One thing we're missing is the ability of learning machines to reason. So right now, all the applications I've shown you is for perception. And for perception, deep learning works amazingly well. It can learn to represent the perceptual world very well.

[00:43:10.48] But learning to reason, that's more difficult. There are a lot of ideas. Some work on it, but I don't think we have the answer to that. The second problem is learning models of the world. So the reason, perhaps, that we are able to drive a car with 20 hours of training without crashing is that we can predict the effect of our actions. We can predict what's going to happen in the world before it happens. The whole front part of our brain basically is a prediction engine.

[00:43:43.09] And our machines don't really have the ability of basically predicting. Not that they don't have it, we can train them to predict in certain ways, but there are technical difficulties which I'll come to in a minute. And the last thing, which I'm not going to talk about, is the ability to learn not just hierarchical representations of the perceptual world, but hierarchical representations of the action world. When we decide to go from here to Atlanta, we have to decompose that task into sub-tasks, all the way down to millisecond-by-millisecond control of our muscles. And so it has this sort of hierarchical representation of action sequences. And we don't really know how to do this automatically with machine learning today. But I'm not going to talk about this.

[00:44:31.68] So it's a big problem because we can have all those cool things that we can build with deep learning, but we can't have those things, which is what we really want. We'd like to have machines with common sense that know a dialogue system that we can talk to. And doesn't have a very narrow set of things it can do for us, right, like playing music and giving us the weather and the traffic. We'd like machines to help us in our daily lives the way a human assistant would.

[00:45:00.60] So we want to build things like intelligent personal assistants. We won't have that until we have machines that have some level of common sense. We'd like to have household robots that are agile and dexterous. We don't have that. We don't have robots that are nearly as agile and have nearly as much common sense as a house cat, with all their superhuman performance in Go and everything. So that's what we need to think of. What's the next step?

[00:45:31.04] OK, so with that reasoning, there is an avenue which is interesting because it might lead to a new way of doing computer science, essentially, which is the idea of differentiable programming. And it's the idea that when you build a deep learning system, you don't actually build a graph of modules anymore in frameworks like PyTorch. You just write a program. The purpose of this program is just to compute the output of your neural net. And every call of functions in this program is like a module that you can differentiate.

[00:46:01.72] And so essentially, it's a new way of writing software where when you write the program, the function of each instruction is not entirely specified until you train the program to actually do the right thing from examples, OK? So it's like a weakly-specified program, essentially. So it's called differentiable programming because it's the idea that you write programs. So essentially a neural net architecture is really a program. It's like an algorithm whose function is not completely finalized until you train it.

[00:46:32.13] And there's lots of really interesting stuff that are viewed in this context. For example, the idea of memory augmented neural nets. So the idea that you have a neural net and you attach to it something that works like a an associated memory that this then can use as a working memory to do things like reasoning, long chains of reasoning. Or maybe it can use the memory to store factual knowledge like relationships between net knowledge bases between objects, objects and relations, things like that. There's quite a bit of work on this. Again, I don't think we have the complete answer, but it's interesting.

[00:47:17.62] Here's another example. This is an interesting project where you'd like a system to be able to answer questions like that here. So you show it an image of this type and you tell it there is a shiny object that is right of the gray metallic cylinder. Does it have the same size as the large rubber sphere? And for us to answer that question, we have to configure a visual system to basically detect the shiny objects and the gray metallic cylinder. We have the strategy. We detect the gray metallic cylinder, and then we look for objects nearby that are shiny, and then we compare sizes, right?

[00:47:53.94] And so the idea behind this project, which is at Facebook in Menlo Park, is you have a neural net that reads the sentence. And what it does is that it generates another neural net whose only purpose is to answer that particular question from an image. So the modules here are dynamically wired, if you want, depending on the input. So it's one of those examples of dynamic neural net whose structure is data dependent-- that's the essence of differentiable programming. Software 2.0, some people have called it this way.

[00:48:25.04] So that's the-- so PyTorch was really designed from the start with this idea that you could have dynamic neural nets. Not quite the case with TensorFlow, which is the Google framework. But TensorFlow is kind of catching up. They're trying to do the same thing.

[00:48:42.30] OK, so how do humans and animals learn? Babies, in the first few days and weeks of life, months of life, they learn an amazing amount of background knowledge about the world just by observation. Babies are kind of helpless. Their actions are very limited. But they can observe and they learn a lot by observing.

[00:49:00.98] So if you play a trick on a baby before the age of six months or so, you show a baby-- put a toy on a platform and push the toy off, there's a trick that makes it such that the toy doesn't fall. Before six months, the baby doesn't pay attention to this. They are sure that's how the world works. No problem.

[00:49:22.09] After eight or nine months, you show this scenario to a baby and she goes like this. Because in the meantime, she's learned that an object is not supposed to float in the air. It's supposed to fall if it's not supported. So she's learned the concept of gravity in between, intuitive physics and things like that, inertia.

[00:49:45.82] In fact, there was this chart that was put together by Emmanuel Dupoux, who is a cognitive neuroscience in Paris who spends part of his time at Facebook, which is when babies learn basic concepts of this type. You know, gravity, inertia happens around seven months or so-- seven or eight months. And object permanence is an important one that pops up very early. The difference between animate and inanimate objects also appears quite early.

[00:50:10.48] So we learn those things just by observation. It's not in a task-dependent way. And this is what allows us to predict what's going to happen in the world. We have a very good model of the world that we learn since we're born just by observation. And we're not the only ones. Animals also have good models of the world. Here is a baby orangutan here. He was being played a magic trick. There was an object in this cup. The object was removed, but he didn't see it. Now the cup is empty. And is rolling on the floor laughing.

[00:50:36.97] [LAUGHTER]

[00:50:39.73] His model of the world was violated, and so it causes you to do one of two or three things when your model of the world is violated. You laugh, you get scared, because maybe something dangerous is going to happen that you didn't predict. In any case, you pay attention, OK?

[00:50:57.70] All right, so I think the way to approach that problem is through what I call self-supervised learning. And it's basically the idea that for a system to be able to learn from raw data just by observation, what you're going to do is you're going to feed a piece of data to the system-- let's say a video clip, OK? And you're going to tell the system pretend you know a piece of this input and pretend you don't know this, and try to predict this piece that you pretend you don't know. And then I'm going to show you this piece and you can correct your internal parameters to make the prediction that actually occurred.

[00:51:32.57] OK, so for example, I show you a piece of a video clip and I ask you to predict how the clip is going to continue, the next few frames in the video clip. And then I show you the frames and [INAUDIBLE] predict. But it's not just predicting the future. It could be predicting the past. It could be predicting the top from the bottom, whatever, the piece of the input.

[00:51:53.65] So there's really those three types of learning. The reinforcement learning, where the feedback to the machine is very weak informationally. It's just one scalar value that tells submission whether it did good or bad once in a while. Supervised learning, you give more information to the machine. You tell it what the correct answer is. But still not very strong because all that data has to be curated by humans and so it's limited in the amount.

[00:52:14.49] And then there is this self-supervised predictive learning idea where the amount of data that machine is asked to predict and the amount of data that's given to train is absolutely enormous. With just an hour of video, it's a ridiculously large amount of data that you're asking the machine to predict. Every future frame from every past frame, for example.

[00:52:36.14] So Jeff Hinton made this argument many years ago that if you have a very large learning system, like, say, a brain, that has 10 to the 14 parameters, three parameters, the synaptic connections, you need a lot of data to constrain the system to learn anything useful. And that's pretty much the only way, predicting everything from everything else, essentially. We're not going to do this with supervised learning or reinforcement learning.

[00:53:00.87] That lead me to this certainly obnoxious analogy here that if the stuff we can learn, or intelligence, is a cake, the bulk of the cake is not supervised learning. Almost everything we learn, we learn just in self-supervised fashion. We learn a little bit with supervised learning. We learn a tiny amount through reinforcement learning, so that would be the cherry on the cake. People working on reinforcement learning get a little upset when I show this, but it's become a bit of a meme now in the machine learning community.

[00:53:29.40] OK, this is not-- this doesn't mean reinforcement learning is not interesting. It's necessary. This is a Black Forest cake and Black Forest cake has to have a cherry. It actually has cherries inside even. But it's really not where we learn most of our knowledge. So with things like-- image in painting, for example, is an example of self-supervised learning, and people are working on this in computer vision. So the next revolution in AI is not going to be supervised, that's for sure.

[00:54:03.98] OK, so let's say we want to build predictive models of the world. So it's a very classical thing in optimal control, and I'm sure some of you may have a background in this kind of stuff. There's a system you want to control. Optimal control people call it Plant. And you have an objective you want to minimize, or maximize, in your case.

[00:54:27.01] And you can run your simulator forward and then figure out an optimal sequence of commands that will optimize your objective, given your predictive model, OK? That's the classical thing in optimal control. In fact, that should be a classical thing in the architecture of an intelligent system. The intelligent system should have a way of kind of predicting what's going to happen before it happens to avoid doing stupid things like running off a cliff, right? And we don't run off cliffs even if we don't know how to drive because-- mostly-- because we have this ability to predict the consequence of our actions. So we need this for our simulator in an intelligent agent, as well as other components I'm not going to talk about.

[00:55:11.34] So how do we learn predictive models of the world? We can observe the state of the world, at least partially, through observation, and we can train a function to predict where the next state is going to be. And then we're going to observe where the next state is going to be and we just train our system in a supervised manner to do this.

[00:55:24.50] So this is something that some of my colleagues at Facebook have tried to do a few years ago, where you have those kind of scenarios where you put a stack of cubes and you let the physics operate. And the cubes fall. And the predictions you get-- this is predictions produced by a convolutional net. The predictions you get are blurry because the system cannot exactly predict what's going to happen. There is an uncertainty about what's going to happen to this tower.

[00:55:49.10] And so you get those blurry predictions. If I take a pen and I put it on the table and I let it go, you can predict that it's going to fall, but you probably can't predict in which direction it's going to fall. And so that's a problem because we have to be able to get machines to learn in the presence of large uncertainties.

[00:56:12.22] So there is the pen example. And the only way we can do this is through models that have latent variables. So basically, we observe the past, the [INAUDIBLE] where I put the pen on the table. And we're going to make a prediction.

[00:56:25.03] What we'd like is to make multiple predictions depending on the circumstances. And so what we're going to need is a set of extra variables-- latent variables-- that we can observe. And when we vary those variables of this vector, it makes the prediction vary among all the possible predictions that may occur, OK? Let's call it intervariable model.

[00:56:51.88] And a good example of how to do this is adversarial training. So adversarial training says I'm going to sample this latent variable randomly. And now what I need to train this predictor is something that tells me whether my prediction is on this set of plausible futures or whether it's outside, OK? But of course, I don't have any characterization of the set of possible futures, so I'm going to train a second neural net to tell me whether I'm on this manifold or outside, OK? That's called a discriminator in the context of adversarial training.

[00:57:26.53] And you can think of it as a trainable loss function. A trainable objective function, basically. The objective function tells you how far you are from this manifold, and there's a gradient of it that points you towards the manifold. So that's the idea with adversarial training.

[00:57:40.96] Let's say you want to do video prediction. You show the system a piece of video. And of course, in your data set, you know what the video is going to do. That's the real data. But then you run this through your generator, which from a source of random vectors, tries to predict what's going to happen. And initially, it's not trained properly. It's going to make a bad, blurry prediction or something like that.

[00:58:00.79] So now you train your discriminator-- your function that tells you whether you are [INAUDIBLE] or not of data. You train it to produce low values for this and high values for that. So that's kind of a representation of what this discriminator is doing. And what it's going to try to do is the green points that come from here that are not on the manifold of data is going to try to push up the output here. And for the real ones, they're going to try to push them down. Those are the blue spheres. And so this function is going to take that shape.

[00:58:32.53] And then what you're going to do is use the gradient of that function with respect to its input to train this generator to produce images that this guy can't tell are fake, OK? So now what you have is an objective function in the discriminator that can tell the generator, you are on the manifold or outside. You can use the gradient of that by propagating through the generator to train it to do the right thing. And eventually, it makes-- gives some predictions.

[00:58:55.34] People have used this-- I mean, this kind of technique now have taken over the field, basically. A lot of people are working on this for all kinds of stuff, generating synthetic images. This is work from a few years ago. These are fake faces. This is work from NVIDIA in Finland, and they've trained a system to transform a bunch of random numbers into a face image. They trained it on a database of photos of celebrities, and by the end, you feed a bunch of random numbers and out comes the image of a face. And these are synthetic faces. At high resolution, you can't tell they're fake. But none of those people exist.

[00:59:41.46] At Facebook, we've been working on similar techniques to do things like generating fashion elements. So we-- it's in France, so you know--

[00:59:52.92] [LAUGHTER]

[00:59:54.82] So we got a big data set from a very famous designer house, and trained one of the generating networks on this. And these are examples of generations of-- and this is not textures that humans would come up with, essentially.

[01:00:09.92] OK, I'm going to talk a little bit about video prediction. So video prediction is interesting because, in the context of self-driving cars, for example, you'd like a self-driving car to be able to predict what the cars around it are doing, right? I realize I'm out of time, so let me--

[01:00:31.18] MANUELA VELOSO: Yes.

[01:00:35.90] YANN LECUN: And this is a project that we've done with people at NVIDIA who are trying to predict what cars around us are going to do, and then use this predictive model to learn a good driving policy. So basically, we feed the system with a few frames of what the environment of cars looks like around us. And we train the system to predict what the cars around us are going to do. And we use data that comes from an overhead camera for this.

[01:01:02.04] And so these are examples of predictions. So this is if you have a completely deterministic system that doesn't have any latent variable and basically, it makes those blurry predictions. And these are predictions that occur if you have a system with latent variables in it, and I don't have time to go into the details of how it's built.

[01:01:19.34] And then you can build a system to basically train the system to run a driving policy. So you start from a real state. You run your predictive model forward. You can compute the costs, which is how close you are or far you are from other cars, whether you are in the lane or not. And by back-propagation, you can learn a policy network that learns to produce an action that will minimize the probability of collisions over the thing. And if you do this, it doesn't work. But if you add a term in the cost that indicates how certain the system is of its prediction, then it works.

[01:01:49.87] And so I'm just going to end with the cube video here. So these are-- the blue car is driving itself, basically, and the white point indicates whether the car is accelerating, decelerating, turning. The other cars are real cars around it that are just recorded. And so our own car is invisible to them. So it's like we're driving on a highway, but nobody sees us, right? So we can get squeezed in between two cars, basically, and there's nothing we can do. But this thing learns how to merge on a highway and things like that.

[01:02:23.71] OK, so I'm going to end here. Just remind you, there's interesting areas of research in deep learning in things like graph or search of data, reasoning, self-supervised learning, learning hierarchical representation control space. We need more theory. And maybe there is a new type of computer science emerging through different [INAUDIBLE] programming. Thank you very much.

[01:02:47.48] [APPLAUSE]

[01:03:01.15] MANUELA VELOSO: We have time for a couple of questions. Yes.

[01:03:05.46] AUDIENCE: Thank you. Really cool talk.

[01:03:08.11] YANN LECUN: Thank you.

[01:03:09.06] AUDIENCE: When you spoke of deep learning, you mentioned how it's important to alternate the linear and the nonlinear layers, and [INAUDIBLE] stack up the nonlinear layers in the middle, for example.

[01:03:25.42] YANN LECUN: Well, there's no point stacking two linear layers because the product of two matrices is a matrix, right? So it's equivalent to a single one. And the role of the linear layer is to mix up-- to detect configurations of their inputs that matter, basically. The nonlinear functions are point-wise, and so they don't combine things, right? They're just point-wise functions usually. I mean, you can imagine other types of nonlinear functions, but more often, they are-- so there's no point stacking them either because they don't actually combine stuff.

[01:03:59.22] MANUELA VELOSO: Very good.

[01:03:59.50] AUDIENCE: Thank you.

[01:04:00.70] MANUELA VELOSO: Yes.

[01:04:02.16] AUDIENCE: I had a question. I had a chance to listen to your talk last year and [INAUDIBLE]. Hi, thank you for the talk. I had the chance to listen to your talk last year as well at a different venue. And in the promising years for research and your cake slide, you had unsupervised learning instead of self-supervised learning.

[01:04:17.07] YANN LECUN: Right.

[01:04:17.47] AUDIENCE: I was wondering if you see self-- because I'm seeing the term for the first time now, when [INAUDIBLE] obviously look up after this lecture. I was wondering if you're defining this as something under unsupervised learning or something separate, or semi-supervised? Just want to hear your thoughts on that.

[01:04:31.38] YANN LECUN: So unsupervised learning is a loaded term. Whenever you say this, people think you're doing exploratory data analysis and clustering and PCA and things like this. And so I kind of moved away from that. Self-supervised learning, you could think of it as kind of a special case, where it's this idea that you have a piece of data and you're trying to predict part of the data that you pretend you don't know from a part that you know, OK? And it's a particular way of doing unsupervised learning, if you think-- if you want to think of it this way. But I think it's the most promising one.

[01:04:54:15] MANUELA VELOSO: And thank you very much for a wonderful talk.

[01:05:00.94] [APPLAUSE]

[00:00:00.00] MANUELA VELOSO: Welcome to the distinguished lecture series on AI. And it's a great pleasure to introduce Andrew Moore. And Andrew may not need any introduction, but I really want to tell you specifically to point of Andrew. Andrew was born in Bournemouth.

[00:00:24.07] Do you guys know where Bournemouth is? It is a little town-- or a big town-- by the south east by the ocean in the UK, and actually also from our point of view from JP Morgan, it is a place where there is a big group of technologists-- in particular, also [INAUDIBLE] scientists and machine learning. So I'm going to be visiting in December. I couldn't believe that Andrew was from that place that I did not know existed before I actually learned about these data scientists in JP Morgan.

[00:01:01.62] But Andrew actually has a distinguished career in AI and machine learning. He did his PhD-- well he's undergrad in math and computer science at Cambridge. And then he worked at HP for some time. And then he actually has a PhD from Cambridge. And he then moved to the United States. And he was for three years a postdoc at MIT working on robotics and machine learning well before any reinforcement learning was fashionable or even like, known.

[00:01:37.26] Andrew pioneered this field of reinforcement learning. And in fact, a survey on reinforcement learning co-authored by Andrew is the most cited reference in reinforcement learning ever. It's a perfect reading to learn about reinforcement learning. And if you did not read it yet, I strongly encourage you to take that as your bed night reading tonight. That's homework number one. And that's called, really a survey in reinforcement learning, I believe. And I have it actually in my office in case you want a hard copy.

[00:02:16.74] And then Andrew, actually, after these three years as postdoc at MIT joined the faculty at Carnegie Mellon University in 1993. And that's when I was fortunate enough to meet Andrew. And in fact, Andrew and I were part of the AI faculty at Carnegie Mellon since always. And we co-taught courses on AI really since '93-- many years.

[00:02:47.01] And then Andrew in 2006 took a leave to go to Google and create the Google office in Pittsburgh where if I am correct-- I forget if this is correct or not-- Andrew grew it to a gigantic kind of like, machinery of engineers in data science from which maybe he always had 99% recruiting ability at Google. And that was tremendous.

[00:03:17.82] In 2014, we were lucky enough to have Andrew join Carnegie Mellon as the Dean of the School of Computer Science. And again, that gave me a chance to interact with Andrew as a faculty and also as head of the machine learning department, I was fortunate to have my dean be Andrew Moore. And it is a great pleasure.

[00:03:39.93] And I think you are going to learn a lot about AI and how CMU sees AI, actually through Andrew's eyes. It's not very common to have a dean of such a big school like School of Computer Science at Carnegie Mellon who cares about research and the teaching and education and connections with the outside in the area of artificial intelligence. So with great pleasure, let's hear Andrew Moore tell us about how we see it-- this AI in 2020-- at Carnegie Mellon University. Thank you, Andrew.

[00:04:14.54] ANDREW MOORE: Thank you. I'm really excited to be here today. Thank you so much for the invitation to Manuela and the whole team. First, let me encourage questions or comments throughout the talk. I love it when folks want to jump in.

[00:04:34.69] I'm also going to learn from this talk as well, which is Manuela and I have worked in many industries, many verticals together. But I, at least, am not expert at all about the important aspects of AI in the whole world of financial technology. And so I would love folks to jump in and tell me, you know what? That might all be very well for if you're using AI for patrolling jungles but it is useless for the financial world. And so that's a little aggressive. But if you want to jump in with any comments or questions like that, that would be totally welcome.

[00:05:15.46] All right. So what I want to remind folks of is the distinction between artificial intelligence and machine learning. And right now, I and-- I would say many of the faculty and students at Carnegie Mellon-- are passionate that it's an important distinction. It's actually one that Manuela brought to Carnegie Mellon's department of machine learning. And it is very much about the future.

[00:05:48.99] So to tell you about the distinction between machine learning and artificial intelligence, I'm going to give you a quick recap of what happened at the start. So in the 1960s when AI was getting started, the heroes of early AI such as Allen Newell, Herb Simon, John McCarthy were really looking at the question of what is human level intelligence? What does it comprise of?

[00:06:18.13] And they broke it down into three pieces-- perception, deciding what you want to do, and then acting. And there was this big discussion about the loop that an autonomous or intelligent system goes through-- we go through in our lives-- where we're constantly seeing stuff, making a decision what we want to do, acting to change the world, observe the results of that. And we keep going around this control loop.

[00:06:47.43] And this was the version of artificial intelligence that I and, I think Manuela, were both sort of born into in the late 80s, early 90s when we were so excited about a future where we can have autonomous decision makers. Probably the peak of these early days of AI was in '97 when computers had superhuman performance of the game of chess. What was interesting about that is it was kind of the peak. And after that, there was a little bit of a sort of plateauing or decline in the use of artificial intelligence.

[00:07:35.97] And the reason for that decline is this thing about decide is really difficult, except in a few special cases such as the game of chess. The reason is, as you all know-- and I know many of you here are very advanced technologists so this is not news to you-- all that happens in an implementation of decide is you iterate among possible actions-- or if you're a slightly more advanced and you're doing planning, possible sequences of actions-- and for every single one of them, you look at the outcome. And then you do an arg max. And whichever is the best one, you choose. And you keep going around like that.

[00:08:19.30] But in order to figure out what the best one is, for each of your possible actions you've got to know what you expect or what the probability distribution is over what will happen next. It's easy in chess because you can write down the rules of chess in a very short program. In general though, that was the really difficult part for artificial intelligence.

[00:08:43.50] You heard people talk about rule based systems where folks were coming up with sets of methodologies for allowing robots and computers and other scenarios other than games to make decisions. And you could do it for circuit board layout. To some extent, you could do it for scheduling. Not in a very meaningful way were you able to do it for trading.

[00:09:07.62] And in general, the problem was we couldn't find a way to have software engineers write down the predictive rules. And there's no point in arg maxing over a thing which is not going to be accurate. So that led us to the sudden growth of an area of artificial intelligence, which had been a little bit quiet in the 1990s. Which is, can we get rid of the idea of humans writing the predictions of what happens next and instead have computers observing historical data predict what happens next?

[00:09:53.80] There were a few huge successes there. One of them was at Google where it turned out that all the companies which had tried to do search engine results by coming up with a set of rules for how to turn a query from a user into a useful thing to show were being unsuccessful because it was so hard. But if you've got enough clicks and enough data, you could build a predictive model of what the likelihood is that someone wants to find something useful in an answer to a question. And that's what led to the massive growth of Google.

[00:10:29.86] Subsequently, similar things were used for all kinds of recommendation systems and then starts to jump into other verticals such as weather prediction, some aspects of physics, many medical decision systems. And so during the mid 2000s to about now, there's been an explosion of people successfully using machine learning to predict what happens next. And we're all excited about that. That means that this big bug which stopped us in 1997 seems to have been overcome to some extent.

[00:11:11.89] What we at Carnegie Mellon-- and in fact, it's through a lineage of geniuses of AI starting with Herb Simon and Alen Newell, followed up by subsequent folks such as Hymie Carbonell who was one of the forefathers of natural language processing and Tom Mitchell who is one of the people who introduced the idea of computational machine learning, and then frankly going up to folks like Manuela Veloso-- Carnegie Mellon, has always held on to the idea that we have got to be good at perception, decision, and action to build autonomous systems. And now that machine learning is able to help us with decision, we really want to go and work once again on the full loop.

[00:12:06.57] So this picture-- which you're going to see a lot of them this talk-- is a way to think about the field of artificial intelligence. And it also helps me personally structure my thinking when it comes to what is the right curriculum for creating a person who is an expert in AI, not just a person who can use AI but people who can invent new parts of AI.

[00:12:35.42] It's also, over and over again in my career, it's been the diagram that helps me set up a large AI project. Because this is a deep composition of the engineering that you need to do for an AI system, which kind of helps you actually see I'm going to need a lead for machine learning. I'll need a lead for perception. Probably, I'll need a lead for decision making. And then at the very top, I'm going to need people who can work out exactly what the consequences are of acting, either discussing stuff with humans or acting autonomously.

[00:13:18.69] So I want to now walk over this diagram, giving a bit of flavor of the key technologies involved. And we'll begin with sensors. The interesting thing, those of you who own a Tesla-- actually, as everyone here lives in New York, you don't own cars-- but for those remote folks, especially our Bournemouthians if there's any Bournemouth people in the audience who are able to drive, the interesting thing right now is that because computer vision which is itself really grown through the use of machine learning is now a commodity we're doing things like detecting and counting vehicles. It's basically something that you can download open source or buy from a software vendor.

[00:14:14.38] We can do things like this is a very nice example of a system in about a mile radius around Carnegie Mellon where all the traffic lights can see how many cars are waiting at which intersection in which directions. And the traffic lights can be a little smarter than about how they timed themselves on the basis of this. And they can talk to their neighbors so that between them, they can actually start to strike deals. Like OK, I'm going to give you these six cars as long as you can promise that by the time they get to you, you'll be ready for them to take them further. And so that's the nice sort of thing we can do now that perception is a commodity.

[00:14:58.01] Another example of where perception is really changing things is it's getting so much better every year at the moment in almost a frightening way. So Marios Savvides in the ECE department at Carnegie Mellon is an example of someone who has solved a problem. It's actually an interesting problem of the control of the pan and tilt of a camera so that if you walk into a room-- if you walked into the back of this room-- and Mario's camera was here, the camera in a fifth of a second would pan and be able to zoom in on your eye to get this quality of image and then do identification and in fact, actually some other diagnostics based on eyes to do with emotional state and other information within in total a quarter of a second.

[00:15:56.97] So that's this year. Next year, we may perhaps with some additional technology I'm discussing later on be able to look at all the eyes of someone in an audience or track all the faces of someone in the audience. And so don't for a moment think that we're kind of plateaued in perception. We can go a long way forward.

[00:16:22.74] And as many of you are probably realizing, this gives us the opportunity to create an awful Orwellian world. It can be really useful. Certainly the idea of supermarkets where you need to walk into the supermarket and take out a piece of plastic to show your credentials for purchasing something is immediately old fashioned. And as we've seen with Amazon Go stores, it is now perfectly reasonable to create a retail experience where you just walk in, pick stuff up, walk out, and the payment without you having to produce anything can happen behind the scenes-- lots of conveniences, lots of consequences and potential dangerous things.

[00:17:12.64] I want to mention another very important thing that's going on. And I think this is extremely relevant for folks in your organization who are looking at edge computing of various kinds or computing that's forced to be extremely close to the sources of data. And that is how do we get to do extremely low latency massive processing of information?

[00:17:45.05] One of my dreams, which we may have time to talk about later, is a world where after a disaster you have-- such as a flood or a terrorist event or a fire-- you have autonomous fixed wing aircraft in the air at about 3,000 feet able to watch an entire city to immediately see who's in danger, who's doing what, where people might be trapped, where's a dangerous location.

[00:18:14.41] If you want to do that, you would need to put tens of thousands of cameras on a single plane. The moment you can't do that, first even if the cameras are just an ounce each, the plane will plummet from the weight. That's solvable. Second, you could not afford to have a network link getting real time data from 10,000 cameras to ground anyway. So you end up wanting to put the compute on an aircraft. And it needs to be very, very low power.

[00:18:52.03] So a big push in the world of artificial intelligence right now is something that I don't think you would ever have normally being discussed in AI lecture or an AI textbook. It is how we can make these neural networks and arg maxes-- the big search algorithms-- run at about 1,000th of the power that they run at right now on a lot to very tiny piece of equipment.

[00:19:19.49] What's interesting to me and what might be a topic of follow up discussion is those of us who are working with aerospace, massive scale surveillance and satellites, we're doing this work for that use case. What's interesting to me and what I'm sure is interesting here is what would we do in the financial industry if we've got down to 1,000th of the power requirements for doing things like neural network inference? And would that actually have a big impact on the business, especially as that technology we're going to develop anyway because we need it so much in aerospace?

[00:19:58.78] I'm just going to--

[00:20:00.00] [MUSIC - BRUNO MARS, "UPTOWN FUNK"]

[00:20:02.18] All right. I apologize for the music, but I want to give an example of the use space. This is A Couple of grad students in the Robotics Institute who were able to come up with a deep learning system for detecting all the elbows in an image, all the knees in an image, all the pelvises in an image, and be able to run that in real time on GPUs inside regular cameras on regular cell phones.

[00:20:32.90] And what they're able to do now is those of you who have used Microsoft Connect which can track one human body, they can now track whole crowds of people. It's an example of the kind of direction that we're thinking of moving. Unfortunately at the moment, this technology is not quite real time for crowds of thousands of people. And so that's the next thing.

[00:20:57.98] Those students and their advisor, they're now looking primarily at these hardware questions so that they can run this on massive groups of people, again, to help out with situations of public emergency or to do various studies. What do we know at the moment about whether as a infectious disease is spreading in a city, we can see about the actual gaits and walking movements of the population coming into work on any given day to sort of notice small changes in behavior.

[00:21:35.60] The other place the power is really important-- this one was power and algorithms-- is in the three dimensional understanding of the world. When you are building rescue robots or things like this, you need to be able to go into environments which have never been mapped before and have the robot figure out how to map them. These days, it seems like such a 2015 thing to imagine taking the data off your robot, going to a data center to compute that stuff, and then send it back to the robot. Now with both hardware technology and algorithms, we can do this sort of thing in real time.

[00:22:21.62] I'm going to keep moving now as we cover other areas of machine learning. Because I think I've given a sense of this explosion in uses of machine learning. I want to mention one thing which I need everyone to be aware of because it is going to be a big change in the early 2020s for all of us. This is the first piece of work I know of for doing neural network inference at about a ten thousandth of the power that it uses now.

[00:22:59.64] This will be critical for self-driving cars. Many of you probably know this but at the moment, the computing that you need for a self-driving car raises the temperature of the car by about 10 degrees because it's so compute intensive. So you actually have to put in extra air conditioning in the vehicle to make it reasonable for humans. That's one of the many reasons why we want this.

[00:23:26.73] These guys have now built something which is a camera you can wear on your clothes. It is powered entirely by your body movement. And it can do face recognition, currently very slowly. It takes about 10 minutes to do recognition from a single frame. But that power requirement is amazing. As that starts to get commoditized, we can have cameras doing compute, which is just selling Walmarts on sticky labels strips to place them all around your house if you want to for some reason.

[00:24:03.94] This, again, might be relevant to you guys. For the purposes of preventing anyone from disrupting the entire global communications system, it is now possible to imagine putting up billions of very small computers into orbit instead of having all your compute sitting in a few cans floating around the world that are extremely vulnerable.

[00:24:33.95] And here is another thing which has been really interesting for us is voice.

[00:24:50.24] [SOUND]

[00:24:52.69] From that piece of sound, one of the faculty at Carnegie Mellon, Rita Singh, was able to deduce this much about the person who was uttering the sound. And why was she able to do this? It turns out that a lot of the frequencies in your voice are very much characterized by the size and shape and flexibility of your trachea. From your trachea, you can actually deduce a whole bunch of things about you including height, age, and weight as well as ethnicity.

[00:25:32.35] And one of the things I really liked was she was able to figure out that he's sitting on a metal chair on a concrete floor. Again, it's not actually science fiction. It turns out that mayday is about 100 megabytes of data. And the frequency analysis and the careful looking at various forms of echoes means that this is quite solvable. In this particular case, it was used to successfully apprehend a serial false alarm caller who had been sending out lifeboats in serious storms for unnecessary reasons.

[00:26:18.23] Now here's where I want to be really clear about something. We at CMU are very excited about machine learning. We formed the world's first machine learning department. But I don't know if I'm speaking for Manuela as well as myself-- it is not enough to just do machine learning. The real art comes in the decision making systems you put on top of machine learning. The things exactly how you do this arg max of over possible actions.

[00:26:56.01] And it's a message that I cannot be too clear about. Do not get obsessed with neural networks or deep networks as the sole component of an AI solution. The number of things where you actually just need to do a one pass prediction of something, a minor compared with the number of use cases where this is wrapped up in a bigger control system, where a decision you make on this time step is going to have repercussions for the next few seconds-- in some cases the next few minutes and in some cases the next few years. So you still have to be doing pretty sensitive optimization over what you're predicting.

[00:27:40.98] Here's one part of this is safety of machine learning based systems is still a black art. And it has serious consequences. This is an image of a system developed by the Army called TARDEC. In 2010, it first came online. It's something where a series of trucks going in a dangerous area-- Afghanistan, Iraq, and so forth-- can have one human driver in the front. And then the rest of the convoy just follows each other.

[00:28:18.39] It is still not being deployed. Why is it not being deployed? Because absolutely correctly, this is a safety critical system. You cannot risk it causing an inadvertent death of a civilian or a member of the armed forces. So you have to be able to prove its safety. Eight years later, we still haven't managed to prove its safety.

[00:28:44.97] Because it is using cameras and LIDAR and other sensors, that's going into machine learning systems which create these models of what the world is doing, which goes into control systems. And unlike advertising which I was doing it Google or many other aspects of the economy, you cannot make mistakes. So a big growth area for us are systems which can make proofs about the behavior of an overall machine learning system.

[00:29:17.95] There are two warring factions in the technology world as to how to do this. And like a good dean, I am encouraging both sides of the war. These folks-- Mike Wagner and Philip Koopman-- are using statistical machine learning methods. They are actually adversarial AIs who watch an autonomous system practicing. And the AI figures out from all the data what is the most difficult thing they could ask the system to do. So it's constantly searching for ways to break the system. It's a technology used in security at the moment, but is now far more automated. And with their help, we've now moved TARDEC several years forward in being able to rapidly discover weaknesses in the overall machine learning system.

[00:30:19.75] The other side of this battle-- I'm going to stand on this side-- are the theorem provers. This, I love because I always used to love mathematical logic. These are the folks who say forget the statistical testing or having an automated agent trying to break it. You need a mathematical proof of correctness of an automated system. The surprising thing is that in some very real situations, we can now get proofs of correctness.

[00:30:53.76] A good example has been the new FAA anti-collision system for aircraft over the United States. That is a new autonomous system powered by-- it doesn't learn online, but it has a model which it learned from data. And then actually, very intelligent aerospace control surface stuff to takeover if two aircrafts are actually about to collide in some way.

[00:31:27.32] When André Platzer applied his theorem proving to prove the safety of this system, he discovered hundreds of thousands of edge cases where the system would actually not follow its specifications and would unnecessarily fail to manage a potential collision. And so in his case, the use of theorem proving has been able to let us iterate to the point where we are not seeing the edge cases where there's a disaster in the system. And so this was a perfect example of theorem proving.

[00:31:59.07] The problem with theorem proving, you end up being a little conservative. You have to-- for instance, if you are putting bounds on the shape of a jet engine, you cannot computationally afford to do anything other than call it a big sphere. And so you're conservative. But obviously in safety engineering, that's the way you want to go.

[00:32:24.70] Another important area which we're betting on is game theoretic reasoning. Here, over and over again-- and I personally had a big disaster in one of my previous jobs based on not accounting for game theory-- when you are using a machine learning system to predict what's going to happen next as the effective action and then you use that to make your choices subsequently, if someone else knows you're using that methodology there are a whole bunch of vulnerabilities for how they can exploit this.

[00:33:01.30] I had an anti-spam system. This was something for detecting if an ad being served on Google's search engine was likely to be a scam of the form of a what we called an unbelievable claim. So unbelievable claim ads, which you've probably all met, usually have a caption-- something like, what the government doesn't want you to know or doctors hate this trick. And not surprisingly, we built good detectors which tried to figure out if either the landing page or the copy on the ad were in this form.

[00:33:50.76] And it worked really well. We had a several factors reduction in the amount of spam ads that got through as a result. And it got better and better. And then its sort of remained at a current sort of stable state. And so we went on to other things to improve. Then about nine months later, it suddenly started to fail miserably. And a whole slew of different spam ads-- not just one, but many, many different spam ads-- were getting through the system.

[00:34:23.21] And as we did a post-mortem, we figured out that someone on the other side had figured out the form of statistical model we were using for doing the spam prediction. And they had successfully sent us a whole bunch of training examples, which meant that our model became uninterested in a certain critical piece of information.

[00:34:49.95] And so they were adversarially gaming the fact that we were using machine learning to make our model continue to perform really well, but actually become unaware that it was creating a vulnerability. And so they were able to attack. It was annoyingly on Thanksgiving a few years ago, so lots of us had to scramble on our Thanksgiving because we had not put a game theoretic assumption into the machine learning system.

[00:35:20.94] So this is another area where you will see places like CMU aggressively hiring at the moment because so many parts of the economy cannot get away with this thing called a certainty equivalence model where you continually learn from data assuming that the data is independently distributed. It's not just that we're looking for outliers. We're looking for adversaries.

[00:35:48.63] Another really nice example of this kind of AI is one of our faculty members who in the spring has come up with a system for an AI to do redistricting, so the kind of problems where we're all worried about gerrymandering. And the interesting thing about this AI is the problem that this faculty member gave himself was how can we have a third party, if you like, do the redistricting which both the Democrats and Republicans will trust?

[00:36:29.23] And thinking about it, he came up with a game where both players can play the game actually as selfishly as they like to come up with the set of districts being drawn in a region and you have a guarantee that the results of the game will be something which an optimal algorithm would have also decided to use based on no concerns about fairness or bias. It's an amazing discovery. It follows a protocol which a few of you may have heard of called the pie cutting problem-- question of how a group of seven people can cut up a pie into seven slices in a way that none of them feels that they've been hard done by.

[00:37:23.72] If you want to look at something even more interesting than the reinforcement learning survey that Manuela mentioned, look up the pie cutting problem on Wikipedia. It's really, really cute, the algorithm that it comes up with. The fun thing is now it's being used really usefully in important areas.

[00:37:43.49] Oh yes, and another really good example of this-- I mentioned earlier-- patrolling jungles. One of our faculty members is very interested in reducing poaching in gang parks, primarily African game parks. She used machine learning-- very similar story to the one I told you about my own use. She used machine learning to predict which areas of the jungle were most likely to experience poaching on any given day according to the migration patterns, the weather, and other information so that rather than deploy the understaffed forest rangers group over to the whole jungle, they could really be focused on the places with high likelihood of poaching.

[00:38:28.34] This was successful for a short while. For a while, they were able to really cause a dip in the amount of poaching that went on. But before long, things went back up. And you won't be surprised to know what was happening was that the poachers were basically choosing the second most likely spot for poaching instead of the first most likely spot and in general, were just think trying to think one step ahead of the algorithm.

[00:38:54.77] With game theoretic reasoning, Fei Fang instead solved this problem using the same kind of technologies that have been used in game theory throughout the 20th century, including the kind of game theory used in building up Cold War protocols, to form a randomized strategy where patrols now are actually are non-deterministic. But they're much more efficient than if they randomly patrolled the whole forest, which almost never hits anyway. They can be maximally inconvenient to the poachers. And they have found what is essentially a Nash equilibrium for what to do.

[00:39:37.31] Again, emphasizing the big theme of you've got to use classic-- or maybe to be discovered-- decision methods on top of your machine learning model. Don't just use the machine learning model naively. That will work fine for little games that us academics play with in our labs. It will not work fine with the real world.

[00:40:05.57] I'm going to move to the top of the stack. There another area where we're hiring and another area where-- again, I don't want to put words in Manuela's mouth-- but we feel as a CMU faculty it would be irresponsible not to train AI practitioners. When you're about to take one of these systems and put it into the world, you have to care about what it actually means to the people who are operating with it. What are the not just immediate effects, but what are the second order effects of having AI assistants or AI advice in your life?

[00:40:44.21] The place that we all think of initially are the big AI chat bots. Whoever created these slides, why the heck didn't they put Google Home on this slide? It was actually me, but-- at the time I created the slides, these were the top three chat bots. So this is an area where when you ask a question like, where can I get a good cappuccino near here-- underneath, that machine learning system is all in place both for your speech and for predicting how satisfied you will be with particular actions. There are search algorithms. Certainly in New York if you ask that question, you're going to be looking at 10,000 possible answers, evaluating each of them before you give your response to the user.

[00:41:40.34] But there's more to it than that. There is also the question of, how am I going to make it so that the user trusts me in future, that I don't risk giving a 1 in 20 of my answers being crazy, wrong answers? Because if I do that, then the human is not going to be able to take advantage of me in the future because I'll lose their trust. So you end up making decisions based on what is known as risk averse reasoning when you're giving advice from a chat bot.

[00:42:10.52] Again in the interest of time, I will move forward on the discussion of a concept called the open knowledge network.

[00:42:19.07] [INTERPOSING VOICES]

[00:42:22.61] Sorry about the noise. I don't know if we can mute the noise.

[00:42:27.35] Here are some other really nice areas of human AI in action. There are really interesting things we can do now with figuring out how people with what some might call impairments such as missing a limb can operate as effectively or more effectively than other folks. This is an example of a game which trains you to do rock climbing by projecting test paths for you to climb with. And it cleverly adapts its level of difficulty so that it can train you from being a beginner up to being a high quality player.

[00:43:13.23] This is a system which allows you to talk to a physician in a way where your emotional state is being recorded during the discussion. And what this is being able to do for this researcher, LP Morrissey, is he has been able to effectively help physicians ask questions which gets to the root of a problem when humans are nervous about talking about a particular subject or going into detail. This has been effective. It has now shown in a bunch of blind tests to come up with better diagnostic prediction.

[00:43:56.75] This question overall of assessing human emotional state is another area which I don't quite know where it's going. But it is a huge growth area in the same way that computer vision was a huge growth area five years ago or natural language processing was a huge growth area 10 years ago. I don't know how it applies in the world of fintech. But the question of being able to get some notion of stress and these things called involuntary microexpressions from users is both Orwellian and really exciting.

[00:44:34.94] Another piece of LP's work has actually been in predicting whether a patient who is under treatment for depression-- predicting whether that specific treatment is going to be successful or not. And he has been able to show that he can predict this before the physician who's treating the patient or the patient themselves, based on analysis of eye movements and microexpressions. So very interesting new world that we're entering.

[00:45:13.20] Many other examples-- I want to keep moving though-- autonomy. So this is an example of an autonomous system produced by some faculty somewhere. I was trying to remember. I don't know who they were. Actually, it was Manuela Veloso for those of you who don't know. To me, it's the perfect example of putting the whole of perceive, decide, act into place. In fact, I'm going to play it again. You're seeing here a group of robots which are cooperatively and in real time planning how to get that ball into the goal. And you see here how the plan is actually executing where these things successfully-- even given uncertainty-- and it was hard to see that that actually went into the goal.

[00:46:03.23] That is an autonomous system. It is the least understood of all the parts of the AI stack. In fact despite the pioneering work here, I am unsatisfied with the work of the academic community or of the commercial community here that at the moment if you want to build an autonomous system, you're going to do artisanally. There is no strong theory to support it. Let me show you some other examples of autonomous systems.

[00:46:35.09] Built with Leidos-- a big defense contractor-- we have a semi-autonomous battleship. It usually has 300 crew. This one has 12 crew. It is currently patrolling the Pacific Ocean. It's a submarine hunter. It does not have weapons on board. That's important to mention. But it does autonomously obey the rules of navigation when other ships or coastal activities are in sight. This has not yet been validated. You can't validate it with a single experiment like this, but there is a real concern that navigation errors are actually kind of serious for the Navy right now. There's so much information that human decision makers are not able to process at all. And so you are seeing ridiculous things like ship collisions.

[00:47:25.49] So that's another highly important autonomous system. Again, it would be against the rules of warfare. And most of us in the world of artificial intelligence would not agree to do it if we were applying this to armed platforms. But for unarmed platforms, this is another place where you want to build in autonomy.

[00:47:49.82] Another really good example is in cyber defense. And here is an example of a rare use of cyber technology. This was a challenge created by DARPA where you're actually using both offensive as well as defensive network security. And this was a challenge with about 16 racks of computers. This was one physical rack. And there were 16 of them arranged in a circle networked together where after the start button was pressed, the systems had to try to break into each other and take over each other. And it really was the last machine standing.

[00:48:38.09] The system from Carnegie Mellon won. It was created by Professor David Brumley. It was the only one which was actually using game theory as well as machine learning and such. What it was doing at a very high level was it had a known set of exploits that it could use. But it was also able to watch for incoming exploits used against it, which it could then redeploy against third parties within the game. And it's a very interesting hidden state game where you hoard the exploits to use at the right time trying not to get them out into the wild until you know you really need to use them. And this won.

[00:49:24.03] The very interesting thing about it is that in Def Con 2017-- which is a human controlled cyber warfare simulation-- this machine placed 17th. So there were still 16 human teams which could beat it. But there were over 200 teams entering. And it was really interesting that we have an autonomous system placing in the top 20. So another area where autonomy is important.

[00:49:58.46] And the thing which I'm not satisfied about is if I want to build an autonomous system like one of these, I can't really train up new engineers with the skills to do it. I have to search around for one of the 100 or maybe 200 people in the world I know who have experience building autonomous systems to put this thing together. So academia or commercial world needs to come up with an engineering discipline around autonomy instead of it being an artisanal thing. Where with autonomy, we're back where we were with data management before the relational database was invented. And something similar to that I believe needs to happen.

[00:50:47.21] Here's a faculty member who really inspires me and really scares me. Her name is Claire [INAUDIBLE]. She has had an important insight-- she's not the only one with this insight-- and now, there's a rapidly growing academic community around it. Those of you who are in software development or manage software development systems, you know you use big change management systems-- probably not get in the commercial world. Maybe you're using perforce or something like that.

[00:51:18.73] As your organization grows, you get lots and lots of data about all the changes to the software which have happened. And it's captured by your source control system. And you get patches where someone has changed the line of code saying, preventing a fence post era or patching security problem 37. What Claire and a few others are doing is using machine learning over these big repositories to predict in advance which pieces of code will eventually get patched and for what reason.

[00:51:58.07] The thing that Claire is also doing, which makes a really stand out, is where it gets a little thrilling, shall we say. She's predicting the fix. So she's trying to predict how a really experienced programmer would actually improve that system. And I want to be really clear. What I'm about to say is science fiction. We have no idea if this is actually going to happen. But if things go well, my hope is that by the middle of the 2020's, we just have autonomous demons running over code, constantly improving it autonomously instead of relying on messy wetwear of human brains to improve the software. So that is a big, grand challenge. I think it could make the world a much better place-- certainly deal with security patches and efficiency mistakes much more effectively.

[00:52:57.07] All right, I want to mention one more point before I wrap up. I'm going to go back to this example. And I've already shown it to you. But you know, you can see it in full scale. This is an AI system which watches you. It gives you tasks of where your hands are meant to get placed. And then it watches you. And it gives you increasingly difficult or less difficult problems. A perfectly fine AI instantiation, not much different from some of the others you've seen here.

[00:53:36.38] The reason I think this is so important is in 2013, this would have been a PhD thesis. 2017, this was built by people who-- most of them-- there was a team of six students who built. Only one of them could program before the class where this was their end project. And by using open source libraries and Cloud for the big data management and so forth, they were able to create this prototype system within the time of their course.

[00:54:13.90] That's really exciting to me because before I went to Carnegie Mellon, I thought that the big problem for the world is that we have one or two orders of magnitude-- too few people who are qualified to build AI systems. Now I believe-- although it has yet to be shown-- that we can educate them relatively rapidly. The reason we're able to move much faster than I had expected is the components you have seen on the stack-- with the exception of autonomy-- we can encapsulate the performance and behavior of those components in a way where you do not have to be an expert in how every single one of those components is implemented. You can glue them together if you know what they're for and how the glue works.

[00:55:02.98] So that is my overall summary of where we see AI at the moment-- what we're scared about what we're excited about. And again, the big call to action here is I've been in the situation where I began to think that machine learning was the only important thing in AI. And over and over again, I've learned that that is wrong. And so places like Carnegie Mellon-- although we continue to invest in machine learning-- to be really relevant in the world of artificial intelligence, we're also investing in the parts below machine learning and above machine learning on the stack. And that's it. Thank you.

[00:55:44.23] [APPLAUSE]


[00:56:00.54] AUDIENCE: Hello. First of all, thank you for the wonderful talk.

[00:56:04.71] ANDREW MOORE: Thank you.

[00:56:05.62] AUDIENCE: I have a question. When you mention proof-- systems like military caravans-- I read somewhere that Google, which is of course advertising search-- believes in testing their systems so that they are 99% correct because it's hard to get to that 100%. But for the kind of the high risk systems you mentioned, do you believe there are no black swans and it can be proven 100%?

[00:56:34.94] ANDREW MOORE: So I would never trust a 100% proof, even for a safety critical system-- but not for the machine learning reason that you're describing. It is absolutely true that I would never allow any product or system to be released which was safety critical and based solely on machine learning. The standard design pattern is-- we are just talking about this at breakfast with some of your team here-- the design pattern is the machine learning system can make recommendations. And then there's a hard wired hand built control system around it, which looks at those recommendations. And using a much, much simpler system-- usually based on linear control rules or hard constraints-- can actually decide what to do. And so the proof work is to show matter what the machine learning system predicts, the overall system is safe.

[00:57:37.82] Now for many robots-- robots, you can say, worst case scenario, we'll just stop. Of course, you can't for things which are in flight or which have momentum, nor for things like big production lines where it may take an hour or two to shut down the lines. So in those cases, this proof method is really difficult. But yeah, I want to just disentangle these. Because really, it would be unethical in my opinion to put the direct output of a neural network into a safety critical system. Great question.

[00:58:17.66] AUDIENCE: Hi, I actually have a couple of questions. Let's see if we have time for that. So one is this last story that you told about just a few novices making this system in 12 weeks or whatever, combining it out of essentially built boxes of AI-- would the next logical step be actually having AI system combine those boxes? So that's one question.

[00:58:43.58] And the next question was about earlier in the talk, you were talking about power management and how important it this. What's a comparison between power used by one of those decision making systems and say, human brain? And are you looking at utilizing some biological components to augment that?

[00:59:06.40] ANDREW MOORE: Very good, those are good questions because they've both got pretty short answers. On the brain power issue, human brains are still the least-- I can say with confidence-- at least three orders of magnitude more power efficient. I think it may be more like six. So there's something going on in our brains which is completely beyond any technology that we know about.

[00:59:30.52] The first question of using AI is to strap together these modules. I actually have not thought about it on that level. It's actually well worth trying. And I bet Manuela is going to strangle me for what I'm about to say. I think it's too difficult. The reason is that that high level of system design-- we're using aspects of intelligence, such as analogical reasoning and sort of higher first order logical reasoning, which we humans can do well, but we AI researchers have not managed to come up with a satisfactory answer for. Manuela, do you agree 100% or only 98%?

[01:00:14.88] MANUELA VELOSO: I agree 10%.

[01:00:16.36] ANDREW MOORE: OK. So you've got two answers there.

[01:00:19.64] AUDIENCE: In the case of [INAUDIBLE]?

[01:00:22.76] ANDREW MOORE: Yes, exactly-- something which I'm extremely pessimistic about.


[01:00:29.30] ANDREW MOORE: That's right.

[01:00:32.18] AUDIENCE: Hello. I have a question regarding the hardware. Do you think that we will need some kind of significant hardware breakthrough before we can truly move forward with the software of AI? Or do, you think that the current silicon transistors are enough, even though Moore's Law is reaching its limits?

[01:00:53.71] ANDREW MOORE: I do think that we're actually in a place of technology where if there were no further innovations on hardware, we could spend the next 15 years coming up with really powerful algorithms based on the current set of hardware. But as I described earlier, things which are-- for example-- managing a whole city during an emergency, we do not have the bandwidth or the compute at the moment under reasonable power assumptions to be able to do it. So that one seems really important. Also, the complexity of what we humans do-- I don't know how we're going to implement that. But my gut tells me that we'll be using millions of times greater computational power than we're currently using.

[01:01:39.08] MANUELA VELOSO: Last question.

[01:01:41.44] [INAUDIBLE]

[01:01:46.67] AUDIENCE: I guess my question is almost a little game theory, in a way. Because you mentioned about reducing the battery output needed for these systems. Isn't battery food to a computer? And if we get to a point where they can be almost infinitely powered, isn't that a risk in and of itself? I mean, won't we also-- as the battery becomes more efficient-- also have to restrain the battery to make sure that we have a way to reign in AI? Because that's the only thing we have to keep them from doing anything, that they need power from us, right?

[01:02:19.59] ANDREW MOORE: That's a good point. So I do think there's an issue there. Before I'm worried about the algorithms turning against us, I've got to worry about nation state adversaries building evil AIs. That is a very real issue. And if-- just in case there's a public aspects to this talk, I'm not going to go into specifics. But you could probably all imagine or try to imagine what you would do if you were evil and actually said, I'm going to use AI to disable an economy of a country, maybe with certain kinds of robots.

[01:03:00.21] It is absolutely the case that-- in my opinion-- after the first one of these things has happened that there will be much more regulation on the use of GPUs, TPUs, or other bits of hardware involved. I'm not sure if it's power that's the issue. But misuse of AI by humans is going to happen. It's going to be disastrous. And it will happen probably several decades before even beginning to worry about the AIs doing it.

[01:03:26.63] MANUELA VELOSO: Thank you very much. [INAUDIBLE]

[01:03:28.92] ANDREW MOORE: Thank you.

[01:03:29.73] [APPLAUSE]

[00:00:00.84] MANUELA VELOSOS: I've known Daniela for many years. She did her PhD at Cornell University. And then she went to Dartmouth, where she pursued her research on distributed robotics, created a great lab. I visited a couple of times in Dartmouth. And then she moved to MIT where she was a professor and then became the head of the CSAIL, that famously CSAIL. And Daniela's been the head for seven years. And she has made a lot of contributions as both research and also as the head of CSAIL.

[00:00:35.64] It is a great pleasure to have her here with us. And she's going to tell us about the impact of AI and robotics advancements and aspirations. And I welcome also all the people that are watching on the webcast. And let's enjoy this talk. Thanks a lot, Daniela.

[00:00:52.53] [APPLAUSE]

[00:00:56.79] DANIELA RUS: Thank you so much, Manuela for this very generous introduction. I am so delighted and honored to be the first speaker in your distinguished lecture series. So today, I want to tell you about some of the extraordinary things we are experiencing these days in both academia, business, and industry. So think about it. Today, doctors can connect with patients, and teachers can connect with students that are thousands of miles away.

[00:01:26.19] We have robots that help with packing and factories. We have 3D printing that creates customized goods. And we have network sensors that monitor facilities. We are surrounded by a world of opportunities. And these opportunities will only get bigger as we imagine the impact of the latest and greatest AI and robotics technologies.

[00:01:46.53] So picture this. Picture a world where routine tasks are taken off your plates. Garbage bins take themselves out. And automated infrastructure ensures that they disappear. And food gets delivered to your doorsteps. Fresh fruit and produce gets delivered by drones. And intelligent assistants, whether embodied or not, help you optimize all aspects of your life to ensure that you live well and you work effectively.

[00:02:17.61] How are we going to live in this future? And what are the supports for taking us to this future? Now AI provides support for cognitive tasks by providing autonomy at rest and support for physical tasks by providing autonomy in motion. And I would like to talk about these two topics for the rest of the talk.

[00:02:42.72] Together in the future, these technologies have the potential to eliminate traffic accidents, to better monitor, diagnose, and treat disease, to keep your information private and safe, to ensure that people connect and communicate instantaneously, no matter what language they speak, in general, to take care of the routine tasks and leave people to focus on critical thinking, strategic planning, the kind of thing all of you guys like to do.

[00:03:13.29] Now, progress in these areas is enabled by three interconnected fields. Robotics puts computation in motion and gives machines the ability to move. Artificial intelligence gives machines intelligence that enables machines to see, to hear, and even to communicate like humans. And machine learning cuts across AI and robotics and aims to learn from data and make predictions.

[00:03:40.22] Now, progress in the AI is enabled by a convergence of three things, algorithmic advances, very important. People often talk about the explosion of data and the increase in the power of computing. But without the algorithms, we would not have anything. So these three pillars, algorithmic advances, data, and computing, are enabling this huge progress that we're seeing today in practically any field that has data. So all companies, all industries that have data can benefit.

[00:04:15.34] And the benefits due to machine learning, which refers to a process that starts with a body of data and then aims to learn a rule or make a prediction about future use of the data. And medicine is a great example of a field that can benefit. Today, machines can read more radiology scans in a day than a doctor will see in a lifetime. Think about that.

[00:04:39.42] Now, in a recent study, doctors and machines were given scans of lymph node cells, and they were asked to label them, cancer or not cancer. The humans made 3.5% error as compared to the machine 7.5% error. But working together, the doctors and the machines achieved 0.5% error, which if you think about it, it's a significant reduction, a significant percentage reduction in the air. And it's extraordinary.

[00:05:07.84] Now, these techniques are currently employed by the most advanced treatment centers in the world. But imagine a day when all doctors have access to these techniques, doctors in rural areas, or doctors that are overwhelmed with work and don't have time to stay on top of the latest and greatest clinical trials can offer their patients the most advanced results by taking advantage of AI solutions that will, in principle, bring the most relevant information to the patient for the doctor to make the decision.

[00:05:42.91] Now, machines and people working together can do so much more in finance. I like to think about a machine as kind of an intern running around and doing errands for you. And when the intern finds a good pattern or something interesting, that pattern is brought up to the human to act on, to make a decision about. And so thinking about using these techniques to improve the conversation between people, to improve how you organize a portfolio, or to even make a prediction of what might happen to the markets are extraordinary possibilities. And if you bring blockchain into the whole system, then you ensure that all the processes done by the machine can be checked, can be trusted.

[00:06:36.70] One more example, machines and people working together form better lawyers. So word processing, internet, and email have already revolutionized how we draft documents, how we look up information, how we exchange information. And the next wave of technologies, which is getting really, really good, is natural language understanding. And with natural language understanding, we are able to get machines to read and remember and interpret entire libraries of documents. And so think about how useful that is for lawyers who need to know stuff about thousands and thousands of documents that are all very large.

[00:07:15.49] And so again, the idea is that the machine could bring the right information at the right time. And yet machines are not able to be lawyers, because they cannot write compelling briefs. They cannot counsel clients. And they cannot persuade judges. But they might be able to support in predicting the decision that a lawyer might take.

[00:07:39.83] So my last example is about traffic. And this is actually from my research group. We built a new algorithm that matches supply and demand. And this can be applied to any field where this is a problem. We applied it to the traffic in New York City. And we have shown that, with our algorithm, we can reduce the number of taxis required to meet the 400,000-plus taxi rides requests a day with 3,000 vehicles. So think about getting rid of 11,000 taxes that are constantly roaming.

[00:08:16.83] So the taxes are not like my car. My car drives to my parking garage, stays there for 10, 11 hours, and drives back. But taxis provide constant movement in the city.

[00:08:26.45] And the catch is that these 3,000 cars, if you're taking a ride-- let's say I'm going from here to the airport, to Laguardia, if somebody is at the street corner and is also going to Laguardia, I have to make room for that person in the car. But I only get to share with up to three people, four people total.

[00:08:50.87] So think about the benefits to the city in terms of traffic, in terms of lower pollution, and in terms of noise. And all this introduces only less than three minutes delay in arriving at your destination. And this does not take into account the improved traffic you get by removing 11,000 cars from the roads.

[00:09:13.91] So all of these examples are extraordinary. And many of the advances you hear about in machine learning today are due to a technique called deep neural networks. And in deep neural networks, you have very large computer networks, usually with millions of nodes. And millions of manually labeled data items are presented to the network to figure out the weights of the nodes inside the network.

[00:09:42.34] And so for instance, in the case of a network that processes images, a person might label this picture as beach, palm tree, and sea. And this is so that when another similar image gets presented to the network, the network says, ah, this is a picture of a beach.

[00:10:00.38] So this process with images works in two steps. Given the photograph, the network has to find out which pixels go with what objects. This is called image segmentation. And we have very good algorithms to do that automatically. Now, once you have segmented the image, you to add labels to the objects recognized in the image. And this is actually very challenging.

[00:10:27.69] So for instance, in this case, you might want to say you have a building, sky, and car. And you can do the same thing with more images and even more images. And when it comes to labeling images, this is how it's done. So when it comes to labeling images, lots and lots of people sit across the world and manually say this is a car, this is a building, this is a road.

[00:10:54.15] So when we think about machine learning, we have to consider what it means for the machine to learn. So for instance, when we say that the network has learned that this is a picture of a beach, what this means is that the pixels that form this picture look the same as the pixels in other images that a human being said this is a beach. The system has no idea what the beach represents. It doesn't know what we do with it. Do we eat it? Do we drink it? Do we play on it? What do we do with a beach? How does the beach feel? What is the purpose of the beach? How much does it weigh? None of this is part of what the system learns.

[00:11:37.19] And it's important to remember this, because we tend to anthropomorphize machines. And when we say the machine has learned, we tend to imagine what a human might have learned at the same time. So it's good to keep in mind these issues.

[00:11:52.22] Furthermore, neural networks are not perfect in how they produce their results. So have a look at these two pictures. They're two pictures of two dogs. Do these dogs look the same to you? Yeah, they look the same to me too. But in fact, they're slightly different. The second picture was obtained by injecting a little bit of error. And you can see the noise that was added to the picture. We can't detect that with the naked eye. And yet, this little error is enough to trick the network from saying this is a dog to saying that the second picture is an ostrich. And if I play around with the shape of the error, I could get the second picture to be a car or a chair or anything I want. So while machine learning is making great progress, it's really important to keep perspective on what it can and cannot do for us.

[00:12:49.17] Furthermore, have a look at this video. So this is a video of a child, an 18-month-old child that's watching this scene for the very first time. It has never seen this scene. And look at what this child does. The child has figured out that the adult needed help and has figured out an action to support the adult with help. Our machines are not able to do anything close to this. But this is an important and interesting grand challenge for the future of artificial intelligence and computation. Can we understand what is going on inside the brain of this 18 months old and create machines that are engineered for similar levels of cognition, for similar cooperative behavior?

[00:13:44.97] So machine learning is changing the world. And it's enabling so many extraordinary applications. But there are significant challenges. Fields that have data, that have lots and lots of data benefit much more than feels that don't. So obtaining massive data sets is a challenge.

[00:14:02.09] The image data sets contain tens of millions of images. In fact, image recognition took off about five years ago when ImageNet reached about 10 million images. Today it's much higher.

[00:14:21.27] Data labeling is also a challenge, because these tens of millions of data points have to be manually labeled. And then it's important to know that the results that come out of machine learning systems are not easily explainable nor generalizable.

[00:14:36.39] Finally, your answer is only as good as the data you have to train your network. And so if your data has bias in it, then your results, the results of the network will have bias. And we have to keep in mind what it means to learn.

[00:14:52.02] A few more challenges. Most of today's machine learning solutions are one-off solutions. And machine learning is usually done by experts only. So the program that plays Alpha Go with you is not going to be able to play chess or poker with you, whereas a human would very naturally switch from one to the other.

[00:15:10.05] Also just crunching the data does not mean you have knowledge. And making complex calculations does not mean you have autonomy. And so we have to keep in mind where are the benefits. And we have to think, what problems should we work on in order to expand the capabilities of these techniques.

[00:15:33.12] Still, we have a lot of opportunities with the technology we have today, as we have seen in the area of medicine, of transportation, of finance, of law. And these opportunities are around personalization and customization, around using natural language to interpret knowledge, to interpret what is in our libraries, and to bring that information at our fingertips. And altogether, with machine learning, we can really increase the quality and the efficiency of the time we spend doing various tasks, mostly low-level tasks, routine tasks. But this is all about machines and people working together.

[00:16:19.21] So let me switch gear and say a few words about autonomy in motion. And as I think about this, I observe that our world has been so changed by computation. Just try to imagine a day in your life without your smartphone, without the web, and everything they enable, no social media, no online shopping, no email and texting, no digital media. It's incredible to think about what this might be like. But I will tell you that 20 years ago, which I remember, we didn't have any of this.

[00:16:58.97] So in a world so changed by computation that's helping us with all these different tasks, what might it look with robots helping us with physical work? So I believe that autonomous driving is absolutely going to ensure that there will be no road fatalities. And it will give our parents and grandparents much higher quality of life in their retirements. And it will give all of us the ability to go anywhere, anytime. It is not a matter of if. It is a matter of when, in my opinion.

[00:17:31.74] So how much work should we be able to offload to machines? So imagine driving home from work knowing that your car has the intelligence to keep you safe and the smarts to make that ride fun. Let's say you have to pick up supplies for dinner. Your car pulls over at the nearest grocery store where you hand the dinner manual to a robot at the door. This robot connects with your home. And the home figures out what items you're missing. And a few minutes later, a box gets presented to you by another robot. And when you get home, you hand the box to your kitchen robot. And you might even let your children help with cooking, because even though they make a mess, your home cleaning robot will clean up the mess.

[00:18:17.06] Now, I know what you might be thinking. You might be thinking that this sounds like one of those cartoons about the future that never comes to pass. But that future, in my opinion, is not that far off. Today, robots have become our partners in domestic and industrial settings. They work side by side with doctors in hospitals. They work side by side with workers on the factory floors. They mow our lawns. They vacuum our pools. They even milk our cows. And in the future, they will do so much more for us.

[00:18:50.78] But in order to get to this future, we really need to build the science of autonomy. We need to improve our robots. And that means making robots that are much more capable of figuring things out in the world, making the whole process of creating robots faster, and making the process of interaction between robots and people much more intuitive.

[00:19:14.69] Now, the first two are very important, because each machine is made of a body and a brain. And for any task, the machine has to have the body that is capable of that task and the brain that is capable of controlling the body to do the task. A robot that rolls on wheels is not going to be able to fly, nor is it going to be able to climb stairs. So you see, the body and the brain together are very important in thinking about machines.

[00:19:43.91] So let me spend the rest of the next segment talking about some examples of making better brains, better bodies, and better user interactions. Let me start with brains. And I will say that today, robots have a limited ability to figure things out. Most of their interactions are fairly carefully specified. And in some cases, you have limited adaptation. But robots do not have a general ability of figuring out what is happening around them. And I would like to use the example of self-driving cars to illustrate this point.

[00:20:23.78] This the recipe for making a self-driving car. It's the recipe that's used by all the companies and most research groups around the world. So autonomous driving usually works in a closed environment. In the first phase, the robot drives on every road in the environment, makes a map. And that map is then used to plan paths and execute the parts when ride requests happen.

[00:20:51.06] So here's how you might take your own car and turn it into a robot, if you like. You start with your car. You add sensors, usually laser scanners and cameras. And then you'll write some code. You'll write some code to make maps, to identify obstacles in the maps, and to label what they are. And you also write code to figure out where the new obstacles that were not there when the map was made are located.

[00:21:20.36] And then these maps can also be used in real time to help the robot figure out where it is, because using sensors, the robot makes a profile of the road at the current moment and compares that with a map. This allows the robot to figure out where it is. And then planning and control enables the robot to figure out how to go from one location to another and how to execute that path.

[00:21:47.18] So this is the recipe. And by the way, this is one of our most recent cars. At MIT, we have a whole suite of vehicles for autonomous driving, ranging from wheelchairs, golf carts, and cars. And they were all done with the recipe I showed you.

[00:22:04.54] Now, here is the problem. So generally, the sensor pipeline for autonomous driving has a bunch of steps. And the steps in the middle are usually very carefully and manually fine-tuned. For every instance, we will usually put code, or we specify the parameters that are needed in order to cope with that instance. And this necessarily limits what the robot is able to do. And in particular, it is very hard to deal with nighttime driving. It is very hard to deal with situations where we do not have maps for the road. And it is very difficult to deal with rainy or snowy weather. Rain and snow is actually a deeper problem, because the sensors we have today do not work well in rain and snow.

[00:22:53.84] So what can we do? Well, here comes machine learning where, instead of manually configuring everything that goes in the planning and reasoning system of a car and trying to anticipate all possible road situations that would come to the car, what we can do is we can try to learn how to drive by watching a human how to drive. And in our research, we are asking whether it is possible to learn how to steer the car by looking at single images of the road and watching what a human driver has done in those instances.

[00:23:32.02] So the answer is yes, it is possible to make driving more flexible. And here is our car that's taking a first ride on a country road after being trained by human drivers driving in Cambridge, which is very urban as compared to this road. And so in red and blue, you see what the car should be doing and what the car thinks it should be doing. And you can see that the red and blue arrows are very close together for the shorter distance. But they get further apart as you go further out. But eventually they converge.

[00:24:10.66] And to be honest, I'm really delighted by this first drive of our car. If I think about my own first drive, it was not this smooth. So I am I'm very encouraged by the possibility of using machine learning to move away from having to manually code all the parameters that go in the reasoning engine of a robot.

[00:24:36.49] Now, we can pose similar questions with respect to other robot tasks. So a similar approach was done to teach this robot how to do this task. And I might ask you how many robots does it take to screw in a light bulb. And in this case, the answer is one, but it needs machine learning, and it needs soft hands. And I will tell you in the second what those soft hands are. If you look at these robot hands, they're very compliant. They look a lot like the human hands, unlike today's industrial manipulators, which are very rigid and hard.

[00:25:16.13] So the other thing to notice is that the robot grasps the bottom using one type of grasp and the bulb using a different one. So how should the robot learn how to do that? This is a very important aspect of grasping and manipulation. And it turns out that a very similar approach to the one we have developed for the cars can be used to learn how to grasp objects. And this can be used to teach a robot the approach direction. So should I go this way or this way or this way? And also the pose of the gripper when the robot grasps.

[00:25:55.70] And what's kind of exciting about grasping, especially with soft fingers, is that soft fingers wrap around the object they are grasping. So the soft fingers do not need to know accurate models of what it is that they're going for. And also with soft fingers, we don't need to know exactly the location of the object. So we can do much more compliant, much more error-tolerant behaviors.

[00:26:22.60] Furthermore, what matters mostly is the aspect ratio of the object we are trying to do. So it turns out that for this particular approach, grasping, we do not need millions of examples to train the robot. We just need the examples of the critical aspect ratios that result in different approach directions and in different poses. And if you specify that any object that fits in the enclosing box of that aspect ratio is in some sense considered trained for that particular task.

[00:26:56.98] So here it is. You can see the robot apply using different approach directions and using different grasp poses. And this was all trained using not 10 million examples but using a very small data set. But again, the data sets represented classes of different possibilities for grasping. So you see, it's very exciting to think about bringing machine learning into robot control systems, because through machine learning, we are really advancing and moving away from manually coding all the parameters that go in the brain.

[00:27:43.42] And I should tell you that in fact, Manuela has been quite the pioneer in this. She figured this out way before the rest of the world. So Manuela has been telling us to do this for many, many years. So I'm very excited and optimistic about the possibilities of making the robots much more capable of figuring things out in their surrounding environments.

[00:28:12.58] Now, let me say a few things about the robot bodies, because as I said earlier, a robot really needs to have a body capable of the task that the robot has to do. And right now, I would say that designing new robots is kind of the way things were with programming before we invented the compiler. So every robot is designed from scratch. Every robot is designed in a very bottom-up way. And we have mostly spent the last 60 years of industrial robotics thinking about robot bodies that are either inspired by the human form, so humanoids or robot arms, or robots on wheels.

[00:28:56.14] And occasionally, we had some inspiration from nature. But nature is so much broader. There are so many more things we can keep in mind in terms of robot bodies and shapes.

[00:29:08.32] And so a question is, can we speed up how we design and we fabricate new robots. Can we imagine a future where anybody can make a robot? So let's take Alice, for example. And let's say we want to give others the ability to automate tasks in her home. And Alice works. So let's say Alice wants a robot to play with her cat while she's at work.

[00:29:31.03] Well, to do so in this near future, Alice could head to a new type of store called 24 hour robot manufacturing where, equipped with an intuitive design, Alice could figure out the shape of a machine she wants for the robot. And once she settles on the design, the store could make the robot overnight for a very low cost. And now the cat has a playmate.

[00:29:57.01] So how crazy is this idea? Can you imagine a future where we could specify the function of the robot? Let's say I want a robot to play chess with me. And from this natural language specification, can we imagine using natural language understanding to parse a specification, to identify what behaviors are needed for playing chess? We have to be able to pick up a piece, to move it from here to there, to not to knock off other pieces. And then, using databases of available mechanisms, synthesize a device that is able to do exactly the behaviors that are needed for the task.

[00:30:37.09] And then we'd like to do something very simple like printing to create the robot. And so here is the robot that plays chess with you. Now, this robot was not developed starting with a natural language specification. But many parts of making this robot were in fact automated. And if you're interested in how it was done, I can tell you later in great detail.

[00:31:03.49] But here's the general idea. If you have a body shape, and you want to create that body shape, you can use off the shelf technologies to turn a photo into a mesh and then to take that mesh and unfold it, either by slicing it or by an origami unfolding process to the point where you end up with a flat representation of the face that can be folded into the 3D object. And with that, you can then create a compilation system that starts with a picture and creates an actual robot that turns that picture to life, like this bunny.

[00:31:47.62] And I believe that with this approach, we can stretch our minds to think about all sorts of objects becoming roboticized. Imagine if we can awaken many of the objects in our surrounding world and turn them into a type of robot, into a machine that can exert work for you.

[00:32:10.33] So for instance, we could ask ourselves what might the Sydney Opera House look like if it were a robot. Can we have sound? Can we increase the sound? And this is what the Sydney Opera House would be if it were a robot.

[00:32:25.72] [OPERA MUSIC]

[00:32:28.22] [SINGING IN ITALIAN]

[00:33:06.17] So we can get the Sydney Opera House to make itself. Well, we haven't quite gotten to that point. But here is a robots that starts as a piece of plastic that was designed like in the case of the robot bunny. And when exposed to heat, this robot can grow into a fully fledged three-dimensional object that can move around and can do all sorts of interesting things.

[00:33:34.83] Now, we can make these robots at any scale. But it turns out that even at the small scale that you see in this picture, which is centimeter scale, these robots can do interesting things for us. We're looking to use these kinds of robots, which we call origami robots, to enable a better future for medicine, to enable the creation of mini-surgeons that will provide surgeries without incisions, without pain, without physical infections.

[00:34:12.44] And the particular surgical task we looked at is whether we can use such robots to remove button batteries that people accidentally ingest. And the reason button batteries are dangerous because the acid in the batteries pierces a hole in the stomach very quickly. Like in half an hour, you'll get the battery to be fully submerged. So you want those out today. You have to have surgery for that.

[00:34:42.26] But now, imagine taking that little origami robot, compressing it, and surrounding it in an ice, so putting it inside a pill-shaped ice. The patient could swallow the pill. When the robot gets in the stomach, the ice could melt. The robot could deploy. And then with a little magnet embedded in its body, the robot could go and remove the foreign object using feedback from an fMRI-like machine.

[00:35:15.56] So here is the robot. And here's how it goes. And it pulls the battery. And now the whole thing can be eliminated through the digestive system. And later on, the robot could be sent back in the stomach. And now the robot could serve as a patch, or it could have medicine in its body. And it could be directed to the location of the wound to deliver the medicine in a very precise way. Think about doing that with cancers where, right now today, you expose your whole body to the cancer medicine.

[00:35:51.08] So it's kind of exciting to also think about the fact that these robots are actually made out of food. So we think of robots as being made out of plastic or hard plastics or metals. This robot is made out of sausage casing. And it's digestible. Nevertheless, it's still a robot. So there are so many possibilities for using these technologies to enable a future for medicine that does not require incisions for every procedure that does not lead to the risk of infections and does not give physical pain.

[00:36:30.28] Now, I want to spend the last part about robots looking at machine interactions. And when we talk about machine interactions, we usually talk about interactions between machines. And in fact, Manuela is quite an expert on this. She has been the reigning champion of RoboCup for many, many years, where her machines worked together to win against other machines. I'm going to let her present next time on this topic. Today I will focus on intuitive interactions between machines and people.

[00:37:06.01] And I want to go back to my autonomous driving example. And during Q&A, we can talk about where we are, really, with autonomous driving, because we are not ready for level five autonomy. But we are ready for some applications in level four autonomy.

[00:37:21.76] But another interesting application of all of that know-how, all of those algorithms is in aiding visually impaired people and blind people to experience the world in ways that is unprecedented. There are tens of millions, hundreds of millions of visually impaired people around the world. And today, the most they have is the white walking stick that gives them one point of information about the world.

[00:37:47.41] Close your eyes, and try to imagine what it would be like to enter into this auditorium and find a seat and determine whether it is empty or not using a white walking stick. Well, in today's age of technology where we count our steps and we send machines to other planets, we should be able to do better than the walking sticks.

[00:38:10.42] And here's my idea. So you take the technology of a self-driving car, and you map it onto wearables. So you take the laser scanners, and you make them into laser belts that are backed by vibrating motors. Then you take the cameras and you make them into cool necklaces. And then you sew all the computers and everything else inside the clothes. And with this, you have essentially the hardware of a system that can look at the world, map the obstacles in the world, and give you a vibration when you're close to an obstacle.

[00:38:38.95] So let's say I'm close to this wall. This side of my belt will vibrate. And also, through a Braille belt buckle, the system could also talk to me. It could tell me words. So with this technology, you could then walk down Fifth Avenue and describe the fabulous window displays. Or you could alert the person to obstacles. Or you could say, hey, your friend Alice is walking by, or there's a cat next to you.

[00:39:09.04] And so this kind of solution is really close to being used, to being deployed. And here's an example of an implementation on our user at MIT, where the system aids the user to walk down hallways, to go down steps without bumping into any obstacles. The system is able to guide the user to a bench where he could sit there and wait for the ducks and feed the ducks. This is an incredible increase of quality of life. And so you see, with AI and robotics, we really have the opportunity to make the world better for many people.

[00:39:57.07] With AI and robots, we can also begin to dream about the future of manufacturing, where robots are no longer separated from people on the factory floor. And here we show an example where our human is collaborating with a robot. And using sensors that monitor muscle activity, the robot essentially adapts to what a human is doing. And so by stiffening and relaxing the muscles, the human can communicate to the robot to stiffen or relax its own grip. So in this case, the muscles are stiff. The robot is stiff. In the previous case, the robot was quite limp.

[00:40:37.37] So we can begin to think about interesting sensors that we could put on our bodies. And I suppose the ultimate question is can we go directly from our head to the machine. And the answer is no. We cannot do that in general today. The sensors we have available for that are EEG caps that are quite sparse, 48 sensors distributed on a cap. And they measure electric activity in your brain. And most of the times, what you get from these sensors is really very complex and cannot be decoded.

[00:41:15.74] But it turns out that there is one signal that, with machine learning, we can detect quite accurately. And this is a signal we all make. It's not trained. It doesn't matter what language we think in. You know what the signal is? OK, this is the you are wrong signal. It is called the error related potential. It turns out the you are wrong signal is something very strong and profound that we all feel. And it's a localized sensor. And it has a unique profile. And it can be detected.

[00:41:48.15] So with the you are wrong signal, we can watch machines. And we can tell whether they're performing well or not. And so to demonstrate that this is a possibility, we have--

[00:42:00.73] [RHYTHMIC DRUMMING]

[00:42:01.48] Could you put the sound down?

[00:42:04.43] So we can watch robots and correct their mistakes in real time. These signals can be detected in 100 milliseconds. And here's a user that's watching a robot. The task of the robot is to sort paint cans in a bin labeled paint and wire spooling in a bin labeled wire. The robot is presented these objects in a random order and then randomly goes one way or another. And so here, the robot went to wire and then moved right back, because the human said that is wrong. You see? So it went, and the correction was done instantaneously here it went directly to paint in that was correct

[00:42:42.86] now here's the wire. It goes mistakenly to the paint box. And the robot gets directed by the person to shift over.

[00:42:54.11] So it is really exciting to think about a future where machines adapt to us rather than the other way around. Today's machines do not adapt to us. We actually have to code them. But imagine advancing our know-how, our technologies to the point where we reach that place. It's not quite there yet.

[00:43:18.25] OK, so we talked about AI and its possibilities. We talked about robots and its possibilities. I want to address an elephant that's always in the room when I talk about these technologies, which is jobs. And usually when I tell people what I do, I get one of two reactions, either people start cracking jokes about Skynet and ask me when the robots will take over their jobs, or people ask when their cars will become self-driving. So I believe that, of course, our cars will be self-driving too. I'm very excited about the technology.

[00:43:54.68] But we have to understand the fears of the first group. We have to understand how to provide alternatives for how to see things differently. And this starts with understanding that AI and robotics are tools. They are tools that are not inherently good or bad. They are tools. And they do what we choose to do with them.

[00:44:15.17] And today they can mostly do routine low-level things. So if you think about four classes of jobs in terms of how much cognition and how much manual unskilled labor there is, I will tell you that the jobs at the top and at the bottom are too hard for machines today. It is much easier to send a robot to Mars than it is to get that robot to clear a tabletop. And likewise, robots are not going to make the decisions in finance or in medicine or in any other field. But even the jobs in the middle, like accountancy, where there's a lot routine activity, even those jobs have critical aspects that cannot be done by machines. So for instance, the accountant has to meet with you and discuss with you. And that the machine cannot do.

[00:45:05.24] So it is better to think about what we can automate in terms of the tasks that we do than in terms of professions that disappear. And here you see an excellent study from McKinsey that shows a variety of different jobs and the amount of time people spend managing others, applying expertise, doing stakeholder interactions, unpredictable physical work, data collection, processing of data, unpredictable physical work.

[00:45:32.09] It turns out that the tasks that can be automated today with our level of technology are the data tasks and the predictable physical work tasks. And I'm actually quite excited when I think about the future, because we can spend a lot of time analyzing these graphs and thinking about what might go away. But it's very difficult for us to imagine what will come back.

[00:45:55.62] So for instance, in the 20th century, agricultural employment dropped from 40% to 2%. But nobody predicted the growth in the service jobs. And similarly, when the airline industry took off, the jobs in the airline industry increased, and the jobs in the train industry decreased.

[00:46:21.75] So I would like to say that as we advance technology, we create new jobs. And we have no idea what those jobs might be. In fact, I'm sure most of you can remember 10 years ago. I remember very well 10 years ago. There were no smartphones. There was no social media. There was no cloud. So all of these are sectors that are employing a lot of people today. These jobs did not exist 10 years ago.

[00:46:50.43] So getting there, though, requires that we think about whether we train our kids with the right skills. And we have an education problem, both short-term and long-term. Long-term, if we start teaching computing-- in fact, I like computational thinking and computational making. If we do this from early on, then we will end up with a graduating class in 10 years time where everyone can participate in the IT economy.

[00:47:19.92] In the meantime, we can take lessons from companies like BitSource, which is a startup in Kentucky, a very successful startup. It's been training coal miners to become data miners. And it's one of the great successes in Kentucky.

[00:47:36.72] So I'd like to end by a few reflections on our future with robots and machines. This is the first industrial robot. It is called a Unimate. It was introduced in 1961. By 2020, we will get to 31 million industrial robots, so from one to 31 million. These industrial robots are masterpieces of engineering. They can do so much more than people do. And yet they remain isolated from people on the assembly line, because they are large and dangerous to be around.

[00:48:09.08] But in fact in nature, organisms in nature are soft and compliant and much more dexterous. And so look at this example of an octopus bending and twisting to escape through a narrow hole. Or look at this gentle elephant that is able to pick up the banana from the child. And yet, the elephant can use the trunk to fend off a competitor.

[00:48:37.81] We can begin to think about new materials and new approaches for making robots in a way that enables them to be safer to be around, in a way that is much more inspired by how nature forms its organisms. And I wanted to show this example of one of one of my beloved robots called Sophie. And here's Sophie slimming side by side was with robots in the natural world. And if I didn't tell you that this was a robot, maybe you could indulge me and say, oh, yeah, it looks like a real fish.

[00:49:13.30] So with the development of soft materials, machines and materials are getting closer together with machines getting softer, more like materials, and materials getting much more intelligent, more like machines. And so this raises an interesting question. What is a robot? What kind of materials we use in order to make robots?

[00:49:36.58] And so I would like to propose that we should expand our view of what a robot is so that the next 60 years will usher adaptive soft machines that could work side by side with humans and could give us pervasive support of machines and form diversity. And here's how this future might look like. So imagine waking up enabled by your personal assistant that figures out the optimal time when you wake up and helps you organize the outfit you want to wear and also what you might need for work.

[00:50:15.22] On your way to work, you walk about by beautiful stores that display your own image was the latest and greatest fashions. And when you walk inside the store, your body gets scanned, and you get bespoke shoes and bespoke clothing done right away. And materials that have sensors and computation embedded in them and even the ability to reprogram what they look like might even allow you to match your outfit to your friend's outfit you might come across in the store. So here they're matching the outfits.

[00:50:55.04] And at work, intelligent rooms might notice that you're stressing out over a meeting and adjust the temperature accordingly. And intuitive design interfaces might allow you to connect with your colleagues present virtually far away to design, let's say, the next flying car. And this flying car could be connected to the rest of the infrastructure and to your home to become them the ultimate assistant in managing your transportation and your chores.

[00:51:35.65] So for instance, in this case, Alice's mother gets the task to pick up some plants. And these plants are for Alice's grandmother who's planting a garden. Your packages could arrive delivered by robots. The garbage bins could take themselves out. The bikes could have adaptive wheels that give you just the right level of support when you want to exercise. And while the robot digs in the garden, you can stop to have a nice conversation with your grandmother.

[00:52:12.28] And when the end of the day is near, after a good day, is it time for a bedtime story, which allows you to enter the story and interact with a dragon. So these advances in the science and engineering of autonomy can create an extraordinary future for all of us, with machines taking on increasingly more difficult tasks, machines allowing us to focus on what we find exciting and interesting and also, frankly, each other. So I'm happy to take questions thank you very much.

[00:52:52.57] [APPLAUSE]

[00:53:13.75] SUBJECT 1: Oh, thanks. Yeah, so the opera house was amazing, obviously. I was wondering if you could talk about some of the ethical considerations around the future of robotics and AI.

[00:53:25.03] DANIELA RUS: Yeah, it's very important. And it's something that a lot of people in the field have began to think about. It's kind of interesting to think about where we are with computer science. Physicists had to face the consequences of their science. Chemists had to face the consequences of their science. This is the time for computer science to look at all the good but also to think about consequences. And this is in privacy. This is in security. This is in ensuring that all these tools are used for the greater good.

[00:53:56.66] So I would tell you that I don't have a crisp answer. But I can tell you that Harvard has started an amazing, inspiring ethics program in their classes. Every computer science course at Harvard has an ethics module associated with it.

[00:54:12.34] At MIT, we recently launched the College of Computing, where ethics and applying AI for the benefit of society is very important. Stanford just did the same on Friday.

[00:54:26.65] So there is a lot of desire in the community to get that question right. Now, getting that question right requires academics, requires industry leaders, business leaders, and policymakers to come together. And I will tell you that from my point of view, when I think about deployment of AI technologies in the world, I really believe that it is important to put in place what it takes to ensure the consumer confidence in the use of those systems.

[00:54:58.60] And so what does that mean? Well, you want to ensure fairness. So you can think of fairness as an attribute that cuts across all industries. But maybe each industry will have a different way of measuring fairness with respect to manufacturing or finance or transportation. Or additionally, maybe we want the systems to be able to explain themselves.

[00:55:21.01] So in finance, if I did not get a loan, why not? Or in medicine, if I got a diagnosis, why? Or in transportation, if there was an accident, what caused that accident? So I think that we have to identify a certain set of attributes that are generic, that cut across many industries. And then each industry could take those attributes and work out the checks and balances with respect to those attributes.

[00:55:46.33] And some of the ideas are fairness, explanation, accountability, data provenance, eliminating bias, et cetera. This is all ongoing research. And I'm sure that with so many minds focused on these problems, we will get some good solutions.

[00:56:08.09] SUBJECT 1: Thank you.

[00:56:18.04] SUBJECT 2: I know a barrier to entry, a lot of times when you talk about robotics, is the programming element of it. How far away do you think we are from maybe a vocal programming language so that people can just talk their instructions or train through vocal elements instead of actual hardcore programming?

[00:56:32.68] DANIELA RUS: Yeah, so natural language understanding has advanced significantly. And there are a number of researchers who are advancing the use of natural language to guide robots or to task robots. And some things are easy. So if I want to give my robot very simple tasks, like go to the door, go out the door, we can do this already.

[00:56:56.29] But there are other things that are much more challenging. So for instance, in our autonomous driving project, we really want to get the robots to get better situational awareness, the robot cars that is. And you can imagine the human talking with a car and describing the scene. Oh, we're going past the church, and we're down Madison Avenue. So you can imagine that there is a lot of information associated with speech, with the description of a scene. But right now, we do not have a good connection between the visual stream of the robot and the speech part.

[00:57:35.08] And so thinking about more complex interactions where you actually do want to connect the context into the robots' action is much more difficult. So you can talk with a robot. You're going to have context in your own mind. If the robot needs to look at the world with its sensors to figure out that context, then it gets challenging.

[00:58:01.53] SUBJECT 3: Your work is tremendously integrative. You're cutting across materials science, mechanical engineering, electrical engineering, computer science, all sorts of stuff. Is that unique to your laboratory? Is there a broader shift in how knowledge is being organized at MIT and other universities? And if so, are there any implications for industry and our kind of organization?

[00:58:26.54] DANIELA RUS: The discipline of robotics and of artificial intelligence is in itself interdisciplinary. And in fact at MIT, when we announced the new College of Computing, we made a special effort to say that this new structure will advance core computing. And also, we'll look at how computing-- and by computing, we also mean AI and robotics-- how this connects to other disciplines so that we can support scientific advancement in other disciplines but also so that we can enable new applications.

[00:59:02.19] So at MIT, when we think about AI, robotics, computing, we think usually about two broad categories. There's the core of the field, and there's the bridge to other fields. And I personally believe that, at this point in time, it is so extraordinarily to think about bringing different fields together and bringing techniques from different fields that will enable the creation of completely new disciplines of study.

[00:59:30.48] So think about journalism and computation. Or we just launched at MIT a new degree in computational urban science. This is the idea of training students simultaneously in the issues that are related to urban development but also computing. So I think that there is a lot of opportunity for the future to bring knowledge from different things together, invent new things that would not be possible to do by one subgroup or the other.

[01:00:05.81] INTERVIEWER 2: We have one question on the webcast as well.

[01:00:09.55] SUBJECT 4: Thanks a lot for the presentation. My question is more in terms of how we can do use the existing solutions that you have built for autonomous driving or the way you have mentioned about how you used the autonomous driving to use and lift or grab object. How can these existing solutions be used in different disciplines, as you've mentioned? What is the right kind of thought process that one should have in order to understand that is the solution over here, probably we can think about it and put this in a new problem at a different place?

[01:00:37.31] DANIELA RUS: Well, ideally, we would have a compiler that will say, for every problem, this is what the domain specialists do, and this is what the computing specialists do. Right now, we don't have that.

[01:00:46.64] And so we start with connecting and collaborating. And so that's at the level of people. But it's really exciting to think about machines and people as a heterogeneous group where they become, in some sense, super powerful, because people help the machines, and machines help the people.

[01:01:06.73] Now, there is a simpler question under your question, which is can robots share experiences and learn from each other. And through cloud computing, there are a number of research projects around the globe where people are looking exactly at that problem. People are looking at how they can aggregate what different robots in different parts of the world have learned about different tasks and creating, somehow, super activities that aggregate the union of everything that was learned. Again, a very cutting edge research topic. Awesome.

[01:01:44.50] [APPLAUSE]

Unless otherwise specifically stated, any views or opinions expressed herein are solely those of the individuals listed, and may differ from the views and opinions expressed by J.P. Morgan and its affiliates.

In no event shall J.P. Morgan be liable for any use by any party of, for any decision made or action taken by any party in reliance upon, or for any inaccuracies or errors in, or omissions from, the information contained herein and such information may not be relied upon by you in evaluating the merits of participating in any transaction or activities. This material is not intended as an offer or solicitation for the purchase or sale of any financial instrument.