SDS 655: AI ROI: How to get a profitable return on an AI-project investment

This is episode number 655 with Keith McCormick, prolific data science educator and executive data scientist in residence at Pandata. Today’s episode is brought to you by Glean.io, the platform for data insights, fast. 

00:00:14

Welcome to the SuperDataScience Podcast, the most listened-to podcast in the data science industry. Each week, we bring you inspiring people and ideas to help you build a successful career in data science. I’m your host, Jon Krohn. Thanks for joining me today. And now let’s make the complex simple. 

00:00:50

Welcome back to the SuperDataScience Podcast. Today I’m delighted to be joined by the oh so, very wise Keith McCormick on the show. Keith is executive data scientist in residence at Pandata, a consulting firm focused on transparent, human-centered AI. He’s also a predictive analytics instructor at the University of California Irvine. He’s the creator of 20 LinkedIn learning courses on machine learning and AI with an aggregate hundreds of thousands of students. And on top of all that, he’s the author of four statistics books with a recurring focus on doing stats with a software application called SPSS Modeler. Today’s episode should appeal to anyone who’s eager to get a return on an investment in an AI project, no matter whether you have a technical or non-technical background. In today’s episode, Keith details his straightforward approach to ensuring AI projects are successful, how AI projects need to be set up and managed in order to get a profitable return on the project, the corporate roles that need to be in place in order for a data science team to complete projects that drive value what AI transparency is and how it relates to the field of explainable AI and how data scientists who have advanced software writing skills could benefit from the use of low-code, no-code tools. All right, you ready for this practical information-rich episode? Let’s go. 

00:02:13

Keith, welcome back to the SuperDataScience Podcast. Where in the world you calling in from? 

Keith McCormick: 00:02:19

Thanks. I’ve been looking forward to this. I’m home, been home more lately, which is lovely. I’m in the Raleigh Durham area of North Carolina, so not too far from Research Triangle Park and all that cool stuff. 

Jon Krohn: 00:02:34

Nice. That’s a pretty good region weather-wise, year-round, I guess, eh? 

Keith McCormick: 00:02:38

It is. Yeah. No, definitely a temperate zone. So spring and fall are fabulous here. 

Jon Krohn: 00:02:44

And for our listeners listening to this as opposed to watching this, which is most of you I do actually, if you wanna see perhaps the most magnificent background of any guest that we’ve ever had on the show, I think it could be Keith’s. It is definitely one of the top ones. It is magnificent. It is so magnificent, in fact, that I spent a while when we were before recording, trying to figure out if this was a Zoom background or not. And it is not it, but yeah, it’s beautiful. Lots of appreciate rich, rich woods and leather bound books and beautiful artwork. Looks like a beautiful home there in North Carolina. 

Keith McCormick: 00:03:25

Yeah, I, I appreciate it’s, it’s well, because, because this is where I do webcam stuff, it’s probably the room that I keep the nicest. But it really started before shut down. It was because this is, this was kind of a dark room, so I kept nice stuff in here to keep it out of the sunlight. That’s really, that’s really how it started. And then all of a sudden shutdown happened. 

Jon Krohn: 00:03:47

Yeah. Yeah. The Covid Lockdowns. Well, those are long gone. And actually thank goodness for us because we met with Covid lockdowns over. You and I met in San Francisco at ODSC West 2022, which was in the Northern Hemisphere autumn in 2022. And that was my first time back at ODSC West. and my first time back in California, since the pandemic had happened, it was really nice to be there. I was able to catch up with tons of people who had been guests on the show. Actually, some of the most popular episodes of all time. Those guests were all there together. So people like Ben Taylor, Sadie St. Lawrence, Matt Harrison, I’m missing others, great ones. Serg Masís, of course, whom I had met before in person. And then I also had the opportunity to meet people like you that I hadn’t met before.

00:04:43

It was really a treat. And we did something special. ODSC provisioned for us, a little quiet room for us to record a short episode. So episode number 628, you did a, you know, a shorter episode of Friday episode on the critical human element in AI. And so that’s a cool one to check out if you just want to hear quickly from Keith about how, if you don’t take into account the human that’s gonna be in the loop with the AI system, your AI system is sure to fail. And that is something actually that we’re gonna talk about in this episode. We’re gonna dig deeper into this idea of how to make AI projects successful. But first, let’s talk about your day-to-day role. So you are executive data scientist in residence at Pandata. And so Pandata is a data science consultancy. And before I met you, I had never heard of an executive data scientist in Residence. So can you elaborate, Keith on what that is? 

Keith McCormick: 00:05:44

Before Cal and I chose that title, I don’t think I’d heard of of this particular combination of of titles before. And a fun fact we did the Five-Minute Friday at ODSC, which was near the very beginning of my time at Pandata. So to put Pandata into context, I’ll just kind of tell you kind of the phases of my career. So, started out as a software trainer for SPSS, and then IBM acquired SPSS. So for quite a few years, if I was sent out to do a consult, it’s because I was the guy that SPSS, or IBM sent, you know and, you know, for data scientists that are trying to do, you know, get established and do a little bit of a personal branding, that can be a limitation, right? 

00:06:33

Cuz eventually you want people to find out what, what you can bring to the table. So for years I was a solopreneur and Cal and I met not at the ODSC that we met at, but ODSC East some months earlier and at this stage in my career, my favorite things are mentoring other data scientists, designing solutions, and I really enjoy kind of the, the business development side too. You know, going out to conferences and so on. So a difference between how my career has progressed and Cal, who’s the CEO of Pandata is that he’s very good at building a team, but building a team is hard. You’ve got accounting and logistics and staff and, and all those kinds of things. It’s not something that I aspire to. So with Pandata, what it’s allowed me to do is contribute the parts of the machine learning project that I think I’m best able to do and that, frankly I best enjoy.

00:07:37

And it dovetails beautifully with what Pandata does because they have some wonderfully talented data scientists. But, you know, something magical happens about 10 years into your career or after you’ve done about a dozen projects or so, you start to be able to look around corners and those kinds of things. So that’s, that’s the whole idea of being a data scientist and, and in residence, it’s, it’s it’s a part-time role for one, cuz I’ve got other irons in the fire, like LinkedIn Learning, but it’s a combination of mentoring solution design and going to conferences and getting the Pandata brand out there. All stuff that I really enjoy. But then when the project is more complicated and you need two or three data scientists, I couldn’t take on those projects as a solopreneur. I used to just kind of hand them over to colleagues that I trusted. But at Pandata I can take on projects like that because we have a whole team. 

Jon Krohn: 00:08:32

Nice. That was a great explanation. Yes. A solopreneur, the person that I’ve heard used that the most is Noah Gift, who was one of my first guests when I started hosting the SuperDataScience Podcast. So he was in episode number 467, and yeah, he has a kind of a similar profile to you where he does a lot of university instruction and curriculum development, which you do. And he does do some consulting. It’s cool to see that you’ve not developed beyond that. But to see this kind of evolution where now you’re able to kind of get the best of both worlds it sounds like, you get the independence and the leadership that you enjoyed as a solopreneur, the thought leadership that you enjoyed as a solopreneur. But now you also have a dedicated group of talented data scientists and other kinds of technical people and probably non-technical support people who altogether they allow you to broaden your impact into kinds of projects that you previously wouldn’t have been able to tackle.

Keith McCormick: 00:09:39

Yeah, no, that’s, I think you’ve framed it exactly right. I mean, take what somebody at a startup has to do, and we both have lots of startup founders in our circle of friends, right? You’ve got fundraising, recruitment, accounting, all kinds of stuff, which I just never aspired to. You know, it wasn’t the part of it that I wanted to do. But Noah gift is an interesting example of this, actually. I know him by reputation only. I’ve never met him. I believe his specialty is ML Ops. Is it not? Am I remembering that correctly? 

Jon Krohn: 00:10:18

That is, he does a lot on like AWS, GCP, that kind of stuff for sure. Yeah. But yeah, he’s got a broad range of, he does machine learning stuff. He teaches machine learning courses at graduate courses at universities. 

Keith McCormick: 00:10:32

And, again, I haven’t met him, but I imagine his motivations were very similar to mine in that if there is a pretty intense project coming along, you have to like stop the world, which is hard when you’ve got the book authoring and the teaching and the conference speaking and all these other things going on. But the other limitation of being a solopreneur, and this is why I, I feel so fortunate that I met Cal and and joined Pandata, is that the limitation is that I can come in as kind of an architect type role when I’m a solo external resource to support a team. And I’ve had lots of very interesting gigs over the years where that worked. But it assumes that the client side has quite a bit of bandwidth and quite a bit of talent to work with me so that I’m doing design and mentoring, but that they’re doing much of the execution.

00:11:33

That’s not always the case. Sometimes there is no data science team and the client really needs you to do that first project while they’re trying to figure out even what their team composition will ultimately be. And again, those gigs I had to just walk away from. But Pandata can support those, Pandata, frankly, can just support a wider variety of projects than than I could. So what could be better? You know, I think I’ve got another 15 plus years doing this in me, but I’m definitely in the last third of my career and what could be better than contributing where I think I’m most valuable but also the parts of the process that I think are the most fun, you know, because being the main execution lead is all but impossible for me. Cuz that means that for weeks or months, that’s all I’m doing and I can’t stop the world that long at this point. 

Jon Krohn: 00:12:37

Instead of actually finding useful insights from your data, are you spending more time setting up your BI tool and answering repetitive questions from end users? Well, it’s time for you to adopt a modern data visualization tool. As detailed in episode number 653 with founder Carlos Aguilar, Glean.io is the lightweight BI solution that empowers teams to find insights within 10 minutes. With Glean.io, you can standardize metrics from your data warehouse and provide your team with explorable data visualizations right out of the gate. Start building a collaborative data culture today, visit Glean.io to request access. 

00:13:19

Cool. And so then, you know, now we have a good sense of what you do at Pandata, what kinds of projects does Pandata specialize in? So I know that there’s kind of this responsibility, AI responsibility and transparency element to Pandata work. So like, maybe you could fill us in a bit more on that. Maybe even if it doesn’t step on any kind of intellectual property or, or privacy issues. If you could go into like a couple case studies of, of projects that you’ve taken on at Pandata. 

Keith McCormick: 00:13:49

Well you know, I’ll probably talk more in, you know, in general terms, but in the early days of Pandata, and I’ve been with Pandata just, well, I’ve been working with Cal when we announced the role was a little bit later in the relationship, but I’ve been working with Cal just about a year now and some of the earliest key clients for Pandata where healthcare analytics, and of course, those are areas where transparency are required. But one of the reasons that I knew that Pandata was gonna be a good fit for me was that I believe philosophically is kind of an ethical thing for me, that model transparency is something that’s important even when regulation doesn’t force you to have it. You know, it’s interesting, his name will probably come to me, but the developer of LIME you know, one of the XAI techniques, he developed LIME because he was, I think it happened to be a computer vision thing, but it was performing well in his train and test partitions, but with totally not working when he went to deploy it. 

00:14:53

And, you know, sometimes I think we have a little bit too much faith in holdout sampling because there are times when it can fail. And what was happening is the model was picking up an artifact of his data prep, which was polluting both the train and the test samples. But anyway, long story short is no one was forcing him to do explanatory AI, but he needed it to debug because if the model was a black box, he couldn’t figure out what the problem was and get rid of it. Okay, so Pandata may have started with client situations where regulation or other reasons force, you know, their hand and they had to have a transparent solution.

00:16:00

But I believe, and I think certainly Cal and, and really everybody at Pandata believes that it’s always desirable to have transparency even when you’re not required to. But now here we are in the current timeframe, and right on the horizon, the EU is working on an AI law that may require transparency. And all the details of the law aren’t clear yet because it hasn’t been passed yet. But it’s, but this kind of regulation is probably coming. So what’s happened, just like a lot of things in one’s career in life, something that may have started with a couple of client situations starts to become a whole approach and a specialty of, you know, really trying to have this clarity and transparency, not just for regulatory reasons, but also ethics and because it’s just good practice.

Jon Krohn: 00:16:54

Yeah. And so, explainable AI, fascinating area. One of the most popular episodes of 2022 was with Serg Masís, who is our researcher on this show. And I mean, hugely grateful to him for regular listeners, you will hear his name all the time, but he does research on our guests. So, like, for today’s episode, a a huge chunk, all of the best questions, yeah, that you hear me ask Keith, they probably were suggested by Serg. So he did an amazing episode on explainable AI. He’s an expert in that space. He wrote a book on it. And yeah, SuperDataScience episode number 539. So you can check that out. But for you, Keith, I would love to know, when you talk about being able to develop transparent models, what does that mean? I mean, so like, you know, one thing, one obvious thing to me is I’m like, okay, well we could limit ourselves to regression models where we can see the beta weights and we can say, you know, this independent variable is contributing exactly this much to the result. 

00:17:58

Cuz we know exactly what the beta weight is and it doesn’t interact with the other input variables. So that’s like one way of doing it. But then we’re pretty limited in the approaches we can have. So you mentioned LIME there. So LIME is a tool alongside SHAP. They’re probably the two most popular tools for doing explainable AI. Is that enough? Is using these kinds of tools post hoc after the model’s been developed to have some sense of how inputs relate to outputs even if we can’t get down to an individual weight level? Yeah. What is, what do we need, I guess, in the healthcare space specifically, what do you, what do you need to show to be able to say we’re transparent and, and yeah, what are you happy with at Pandata in any industry? 

Keith McCormick: 00:18:41

Well, it’s a, it’s a tricky issue. And I’m gonna, what I’m gonna speculate about here isn’t so much a Pandata answer as much as what I’ve come up with myself exploring the same kinds of things that I’m sure that Serg was talking about in an e in that episode because I prepared a, an XAI course. In fact, I’m gonna, I’m losing track of my calendar here, but I think I’d given a half day XAI workshop in less than two weeks or just about two weeks or something like that. I’m speaking at TDWI in Las Vegas in February. So the way that I usually describe it to folks is that that first scenario that you were saying, where you have something that’s inherently interpretable is what some folks, I’d like to use this phrase call interpretable machine learning. 

Jon Krohn: 00:19:43

Right? So for me, the regression model, the regression model to you is interpretable machine learning. Yeah? 

Keith McCormick: 00:19:49

Yeah, I would agree. And, and then I would include in that at, for instance, a single decision tree. So I think part of it is not being shy about using these simpler techniques in two senses. One is, I think you should always, always use a simpler technique when you’re prototyping. It drives me crazy when a modeler will go right to a complex technique, maybe because they did a tournament of algorithms in AutoML or something like that. They probably had some kind of justification for doing it, but they’ll go right to the more complicated technique. And then if I ask them, how much additional accuracy did you gain by going more complicated, I don’t get an answer. That frustrates me. I just think that’s bad practice.

00:20:43

So, so I treat the IML solution as a pace car, so to speak, right? And then I go to see if I go more complicated, what’s the gain? One of the reasons I think this is important, even for analytics leadership, you know, so somebody like an analytics director or something like that, and, and they may have gotten that role because they get promoted out of BI or IT, you know, so they might, they might possibly be managing data scientists but not be a data scientist themselves. This kind of thing happens all the time. I always coach them to do that because that additional gain and accuracy, it’s not just that you’re gonna lose transparency, but you’re probably gonna be forced to do XAI on top of it. So you’ve just lengthened the project so there’s actual, you know, dollars involved now where the project has just become more expensive. So so I reserve explainable AI not for the interpretable piece, but when you have to add that explainability layer on top of a complex model. 

Jon Krohn: 00:21:50

Yeah, yeah, yeah. And to your point of expense, those more complex models are probably going to cost more to train and they’re going to cost more to run at inference time as well. So and other reasons why we might wanna have a simpler model, like a regression model or a decision tree. And if you get an, if you get a negligible, you know, there might be, in terms of winning a Kaggle competition or whatever, you know, there might be this 10th of a percent improvement in accuracy or AUC and okay, cool, that wins you the tournament. But in practice that isn’t, that doesn’t change the human’s experience of using this model. Nobody’s going to notice that difference. But they will notice if it takes several seconds for the model to output something to the screen as a, you know, somebody in a production and web interface or something you know, your end user of this model, if they have to wait, you could have been using a simpler model that was tiny, little bit less accurate, negligently less accurate, but gives you results instantly, and it is cheaper to run.

Keith McCormick: 00:23:02

Yeah. Well, well said. First, thank you for making the distinction between training time and scoring time, because I find that folks forget about that because maybe you can tolerate a long-ish training time if it’s gonna be done infrequently. But that scoring time might be critical. And what, you’re gonna wait until you get to the ML Ops phase and you’re gonna have your machine warning engineers or whoever’s in charge of scalability issue, you’re gonna dump that problem on them. You know, you, you have to be thoughtful about that early on. I remember being on a gig it happened to be an IBM gig, and scalability was an issue, and I don’t remember the details, but it was something like you know, Keith, your model’s taking like seven or eight, you know, milliseconds per record at score time, we’d love to get it down to four, how much accuracy will we lose if we go from 40 variables to 30 or 40 variables to 25 and so on. So in other words, I was having a constant dialogue with the person who was gonna put my model into production. But what was the other thing that I was gonna say about that? Yeah, so about score time, but then also, oh, yeah, the trade off. So what I think is so important for us to remember on the, you know, on the technical side, on the data science side, although, you know, people, people are technical in different ways. 

00:24:48

Like I don’t think of myself as being very savvy on the data engineering side, for instance, you know, that it’s all kinds of different technical, but for those of us that are technical on the data science side, I don’t think it’s our decision whether or not we go for that extra one-tenth of a point of accuracy. I think it’s a business decision. I mean, I really, I, I really do. I, I hope that most people agree with me, but I think what happens is the fear is that the senior leadership or even analytics leadership won’t fully understand all the details. Everybody’s always talking about how they don’t trust their boss to, to go into the weeds with them. But that is really something that they have to decide. We don’t get to decide that. So as an external resource, or as a consultant, you know, at Pandata, that’s a, that’s a client decision, and that’s a different kind of transparency. That’s not necessarily model transparency, but it’s a kind of process transparency that we would take very seriously to on a consult. There are some consultants that just want to take the data kind of disappear and then come back with the model. And if that’s the case, they’re not giving the client the opportunity to decide between the more expensive model that’s a 10th of a percent more accurate. 

Jon Krohn: 00:26:12

Yep. It’s, yeah. I, you know, we’ve, we’ve had a lot of guests on the show talk about this problem of data scientists being too siloed and not kind of working with the business iteratively to see what’s best. So yeah. So thank you for highlighting that. There’s an interesting kind of nomenclature point of difference between you and Serg. So XAI, this explainable AI area is in its infancy in a lot of ways. There aren’t a huge number of tools or a huge number of books out there today. I know that there will be in five years and even more in 10 years. And it’s interesting to hear, so you define interpretable machine learning as a as, as models, like a regression model or a single decision tree where it’s very straightforwardly interpretable. And it’s interesting. So Serg’s book is called Interpretable Machine Learning, and it’s all about XAI techniques like using LIME and SHAP. So it’s interesting. So as, as the field develops we’ll probably start to have more standardized ways of describing these things, I, I totally get the way both of you are doing it, even though it’s different. Both ways make sense to me. 

Keith McCormick: 00:27:28

Yeah. And I, I’m glad you raised that issue because I’m when I give talks on the subject, I’m, I try to be cautious about that. I adopted the language that I did. I was influenced by two sources. The Association of Computing Machinery did a, a cover story on this. And I said, well, you know, if I try to use the terms the way the ACM is doing it, then why not? I’m just gonna stick. I’m gonna kind of stick with that. Right? But, but there really isn’t a, you know, a consensus. And somebody else that’s been a real influence by the way, is Cynthia Rudin, who’s at Duke. I’ve never met her, but I’m a huge fan. And she thinks that generally speaking, XAI is a mistake in that it’s, it’s never more than an approximation of the black box model anyway, right? Now, it might be a darn good approximation, but approximation nonetheless. Right? So she thinks that it’s not really truthful to say this XAI is your model. You gotta be clear to the, to the client, to the decision maker. Right? Right. You know, that it’s not also, she thinks that we exaggerate this notion that you have to be more complex to be accurate. So she’s tried to develop algorithms where there’s a very, very thorough search of the problem space, but it results in one tree, not an ensemble. 

Jon Krohn: 00:29:07

Right. 

Keith McCormick: 00:29:08

And the reason we do reinforce and other things is that trees are greedy, so they tend to optimize locally, and we don’t get a good overall optimization. So she says, well, that’s true. You don’t have to do cart like it’s 1984. Go ahead and leverage your gpu, leverage your fast contemporary computers, but try to have one tree pop out at the other end, not a thousand. 

Jon Krohn: 00:29:33

Right. 

Keith McCormick: 00:29:34

I’m not an expert certainly on her approach, but she’s been a real big influence. So whenever I present XAI, I don’t take as strong as a position as she does. I think we have to have this skillset, we have to do this, but it’s this constant kind of cautionary reminder that maybe there are times that we go complicated when we don’t have to. 

Jon Krohn: 00:29:59

For sure. As we often discuss on air with guests, deep learning is the specific technique behind nearly all of the latest AI and machine learning capabilities. If you’ve been eager to learn exactly how deep learning works, my book, Deep Learning Illustrated, is the perfect place to start. Physical copies of Deep Learning Illustrated are available in seven languages, but you can also access it digitally via the O’Reilly learning platform. Within O’Reilly, you’ll find not only my book, but also more than 18 hours of corresponding video tutorials, if video is your preferred mode of learning. If you don’t already have access to O’Reilly via your employer or school, you can use our code SDSPOD23 to get a free 30-day trial. That’s SDSPOD23. We’ve got a link in the show notes. 

00:30:53

Now, what you’re saying there about not having to go complicated makes perfect sense to me, particularly when we’re working with structured data. I can believe that there’s a simpler model, but you know, you’re nodding your head, so you gotta get where I’m going with this. When there’s situations like raw images or raw video or natural language, then it gets trickier. And it seems like you know, having like the deep learning models that are very difficult to explain, they are able to handle all of the nuance and the pattern recognition in those big unstructured media files.

Keith McCormick: 00:31:24

Yeah. And I, I, you came up with exactly the same list that I would video, image, audio, natural language processing, even if it’s even if it’s text, but trying to do that, finish the sentence or, you know, text to image and all this kind of stuff that’s where not only deep learning shines, but where five or 10 years ago we had made so little progress.

Jon Krohn: 00:31:50

Right, yeah. 

Keith McCormick: 00:31:51

But I’m a skeptic when it comes to deep learning on structured data. Maybe it’s because it hasn’t happened yet, or I haven’t been convinced yet. I’m not sure, but I, I haven’t seen anything that convinces me yet that I should be doing loan default with deep learning.

Jon Krohn: 00:32:13

Yep. And myself personally, as well as dozens of deep learning students that I’ve had over the years, so I have taught deep learning courses online or in-person for many years now, and you get these people that come from finance and are like, I’m gonna build the best stock prediction model ever, using structured data, or like you’re saying like loan default prediction or any of these kinds of things where you just have discreet independent variables as opposed to unstructured media files as inputs. And yeah. They don’t outperform the, you know, best practice kind of linear regression model, so yeah. Anyway, I just wanna kind of highlight the difference. So did we, have we answered the question? I kind of, we’ve gone down on a long tangent here, but my kind of, my initial question was, what does transparency mean anyway? So, you know, so we know now that, you know, of course, yeah, you and I are on the same page. If we have what you call an interpretable model, like a regression model or a single decision tree, obviously that’s transparent. But then are there situations that you run into Pandata where you are working with unstructured data and deep learning models and you need to be applying XAI, and there’s a point that you get to where you’re like, okay, it is transparent? Or is that just kind of, that’s not really in scope for the kinds of work you guys do? 

Keith McCormick: 00:33:33

No, no, no. I think again, I, well, for instance medical imaging is something that Pandata’s done work in. Now, so the XAI is on the table again, and that’s where, although I’m a huge fan, I’m not prepared to go as far as Cynthia Rudin because I’m just not sure how you do that. Right. I’m not sure how you always restrict yourself to, yeah. That’s inherently interpretable if you’re dealing with something like medical images. But I think, you know, as we’ve been talking about it, and I bet my colleagues at Pandata would agree with this, that such an important part of transparency is not just the transparency into the math, is the transparency into the process so that the client can be a full educated partner in the development of the solution. You know, what, what I always remind folks of is that the way that you judge, you know, a model isn’t that it’s better than some set of business rules that existed before the model or something like that. I mean, this gets into our Five-Minute Friday topic, really, where what you’re doing is you’re, you’re entering this new world. There’s the new world that has the model in it. So a proper model evaluation has to compare the entire human computer collaboration that existed before the model was built. And after the model was built, the humans are still part of the equation. So transparency, I think, is transparency into the process, right. So that the client really understands this because they have to embrace it. 

Jon Krohn: 00:35:20

Cool. That makes a lot of sense. I like that. I like that definition. Yeah. Transparency is having insight into the process. You have some sense of why a result is coming out the way that it does. Cool. All right. So that was kind of the first, you know, the kind of thing there and understanding what you’re doing at Pandata was the first topic that I wanted to cover, and we ended up having a really rich conversation there. I loved it. Now I’d like to transition to a topic that I alluded to that we’d be covering in this episode, which is getting a return on investment on AI projects. So for two decades now, you’ve been a practitioner, a trainer, and speaker in what used to be known as data mining. You don’t hear that as much anymore. Now it’s just you know, that’s, I guess that’s just part of machine learning.

00:36:10

And one of the topics that you’ve been addressing lately is ensuring the return on investment in AI projects. So, Keith, how do companies fail to make a profit from an AI project? There’s lots of excitement about AI. Everyone’s like, oh my goodness, ChatGPT. Yeah, this is gonna, this is gonna change everything, and it will absolutely change everything. But I think, you know, the vast majority of ideas that people have for AI projects don’t end up coming to fruition in a way that users end up being able to make use of it in the way that was originally conceived. So, yeah. How do you succeed at an AI project in particular? How does a company make a profit from an AI project? 

Keith McCormick: 00:36:54

Well, to, to try to tackle the first question, and I’ll, I’ll run the risk of being a little bit blunt. I think the reason that sometimes companies don’t get a return on investment is that they, they don’t even try. They, they don’t even, they don’t even kind of recognize that as the goal. Yeah. So sometimes I get a little bit of gentle pushback when I’m hanging out with data scientists on this, because I think they want to preserve their ability to explore the data, not entirely knowing where that’s gonna take them. And I get that. So, you know, sometimes people refer to the 20% time that I think Google popularized. I totally get that. But when I talk about how projects should always have ROI, I mean, you’re officially starting the project. There’s a kickoff. There is somebody that’s designing the solution. 

00:37:58

There’s probably somebody in a data steward role, there’s an internal customer, there’s a sponsor the whole nine yards, right? If you’re gonna, if you’re gonna jump in and you’re talking hundreds of thousands of dollars of salary hours and external expense, or software expense or what have you, it is crazy to embark on that. Right? So, I don’t wanna step on anybody’s toes that they shouldn’t have a few hours a week where they’re just looking for opportunity. I get it. But if you’re gonna do a real project that’s gonna take months and cost hundreds of thousands of dollars, if not, you know, break that million dollar barrier, you’ve gotta have some kind of, you’ve gotta some kind of a plan. So I think what happens is that if this exploratory thing gets almost, you know, put into just the long-term thing, that’s all the data science team does. 

00:38:54

Or if the data science team turns into a help desk, that’s the other thing that I think is a real killer. Oh, yeah. Management’s just curious about something, right? So I call those insight projects, and sometimes I’ll get a funny look when I give a talk on this and people look at me like, what’s, what’s wrong with insight? I thought insight was like the goal. Well, you know, insight doesn’t, doesn’t pay the bills because the whole idea of an insight project is the data scientist explores the data. Also, when do you know when exploring is done? You know, does exploring take two weeks? Does exploring take 10 weeks? Does exploring take four months? You know what I mean? So, I, I struggle with that. So, again, I’m cool with this a little bit of time each week where the data scientist is given some freedom to explore a new opportunity. 

00:39:37

I get, I like that idea actually. But you get, you get my idea, right? Is that yeah, just poke around in the data and try to find something valuable. When, when did that process really begin and end? How do you even measure how much money you spent on that, right? But then the notion is, is that the deliverable is some kind of a slide deck that gets presented to management. Management goes, wow, that’s like so cool. They bring that into a C-level meeting or what have you. But it’s just this exchange of ideas. It’s very hard to measure what’s going on, right? So for me, all data mining was, is what we now call predictive analytics. And even predictive analytics is a little out of fashion, but, and I know some of the authors of CRIPS DM which is the cross industry standard process for data mining, and I’ve always felt like CHRISP DM was a better fit for supervised.

00:40:30

And the folks that wrote it said, no, we’re, we’re trying to cover unsupervised too. I don’t know if I’ve ever been completely sold on that. But the point is, that’s all it is. It’s making some kind of a prediction. So what I’m sitting down with a client, and this is literally the first hour, I want to know what are we predicting? And sometimes it’s either not clear or there is no prediction, and we can maybe talk about that. But I want to know what we’re predicting, what benefit they get, tangible, measurable benefit do they get by knowing that prediction in advance? And then what the heck are they gonna do with it once they know, right? So I remember having a long conversation with it was actually a, a government gig. So it was about federal funding and it was about project management and so on, right? 

00:41:25

So they wanted to know could we predict in advance what projects would go over time you know, over schedule, over cost. So I said, well, great, if you knew that about a project, it’s an 18 month project, big government expensive project, and you knew at the six month mark that it was likely to go beyond 18 months, what action were you gonna take? And Jon, this, it took 10 minutes to get a straight answer to this question because, I was like, well, “What do you mean?” I said, you know, you’ve now learned that this is gonna happen a year from now. How is that measurably valuable to you? Say, well, you know, if they go over, they just get extended, they’re important projects. Once they’re approved, we have to finish them. But it would be nice to know that it’s gonna be late. You’re not gonna accelerate it so it ends on time. You’re not gonna, there was no action. And you’d be surprised how many times conversations like that fizzle out.

00:42:35

So what I would, what I urge clients to do, and this is the first step I think, in ensuring that this happens, is to have more than one project that’s in a analytics portfolio so that you’re constantly betting them and deciding not only which, what, what projects are worth doing, but what projects should be done in the current calendar year. Cuz what is a data science team supposed to do, if this is an extremely top-down process and they’re not allowed to kick the tires and question the project, then the head of the data science team, if they say no to a project, then what they have then the then have no current work. What should really happen is there should be three or six or a dozen projects that are important to management, and the data science team should get a chance to kick the tires, to estimate ROI and say, you gave us this half dozen, we recommend as the data science team that four out of these six are a good fit for our team. And these two out of the six we should do immediately because they really have the potential for good, good ROI. 

Jon Krohn: 00:43:46

Yeah. As the chief data scientist and the co-founder of a machine learning company called Nebula, I am involved directly in this kind of process for, so we prioritize projects. So there’s dozens of projects that we, you know, if we had an infinite number of resources, an infinite number of data scientists, if we had dozens of data scientists, then they could each be tackling one of these projects. But we don’t have a dozens of data scientists. And so I liaise between the executive team and between my data scientists to it’s, it’s like a, it’s a bit of a chicken and egg, but the end state is clear. So you work with the data scientists to define, to estimate how much work a project might take. And some projects are relatively discreet where you can say, you know, there’s very few unknowns in being able to deliver that to you. And so, you know, we can estimate that it’s about two weeks of work full-time for one data scientist to be working on or a month or whatever. 

00:44:56

Like many things in computer science, we do our estimates in powers of two. So it’s like, it’s a one-week project. It’s a two-week project, four weeks, eight weeks. Yes. But then there’s other kinds of projects that are more like the exploratory ones that you’re describing where we’re like, okay, we just got access to dozens of new API endpoints from this new vendor that we’re working with. We are pretty sure that some of them are gonna be really useful to us, but there’s also lots of other ones there that we should look into. And, that’s kind of, that falls into the bucket of the thing that you’re saying, like, you could explore this forever. Like, what other things could we be doing with our data or blending with our data? So for those kinds of projects or projects where we wanna make some, you know, we might have some production machine learning model where we’re like, oh, you know, transformer architectures, maybe that’s something we should be considering for this natural language model that we already have in production. 

00:46:01

And then it’s like this other open-ended task of how, like, you know, we, we don’t have a transformer model already in production, so it’s gonna be, there’s gonna be some time learning which kinds of architectures we might wanna work with. Getting used to having to use a lot more hardware than we ever did before, and running into roadblocks time again there. So there’s, there end up being these projects where it’s difficult to say when you succeeded, where it’s difficult to say you know, it’s difficult to, to know how much time you should invest in advance. Whereas on the other hand, there are projects that are, you’re like, okay, there are few unknowns and we can make these concise. And so what I like to do, and I’d actually love your feedback on this, getting some free data science consulting here, on air.

00:46:50

But what I end up doing is I like to blend projects for my team. So have, have, and I’m lucky to have data scientists that all are capable of working independently and executing efficiently. So I can say with those tasks that are relatively circumscribed, we’re getting those kinds of things done all the time. But then I’m like, you know, spend, spend your morning working on that. Let’s make sure we’re getting somewhere on that, but then spend the afternoon trying to get this big open-ended project off the ground. Let’s understand that better and try to get it to a point where we can define a clear next step and express that clearly to management and say, you know, we have this big idea, you know, it’s gonna take two weeks or a month of someone full-time to explore this fully in order to know whether there’s something here. And then we’ll be able to give you a better estimate after that, on next steps. Anyway, so I’ve been talking for a really long time. It’s your episode. 

Keith McCormick: 00:47:48

Yeah. No, I, I love the way you’ve kind of positioned that question, because as I’ve been listening to you, what it makes me think of is that there are probably a lot of data science teams that should picture almost like they’re an internal consultancy. You know imagine that you’re a consultancy like Pandata, for instance, right? You’re gonna have some, you’re inevitably gonna have some non-billable time. It’s just part of getting the job done. So it would be internal meetings, professional development. You’re always just gonna have some non-billable time. But if a consultancy was 90% non-billable, you’ve got problems. So I would think that if you are an internal data science team at a startup software company, all these different scenarios, I don’t wanna rob, I mean, data scientists are creative people. That’s why they went into data science. They like to explore and try new ideas, and they, and they want the freedom to take a shot at something that maybe has a low probability of success and they don’t want management looking over their shoulder like every second. Right? In fact, that’s probably a good way to push a data scientist out the door if it was like way too structured. But if those data scientists think of this kind of like billable and non-billable time, the projects that have non-ROI are analogous to billable time. And the projects that they go, that they really don’t know if they’re gonna pan out, they’re kind of exploratory, they’re riskier if they think that is non-billable time.

00:49:32

And it’s just like, “Hey, let’s not let our non-billable time become the majority of our time.”, right? I mean, there’s a reason why, there’s a reason why you know, Google came up with this 20% thing, right? Then I think that alone kind of solves the problem, right? So I always like to say that predictive analytics teams should be self-funding. They should be profit centers, not cost centers. But I’m totally okay with 20% of their time not being profitable because we need that time for employee satisfaction and to know what are we gonna be doing two or three years from now. I really think it’s just, it’s keeping into balance. 

Jon Krohn: 00:50:12

Yeah. And that’s also where the breakthroughs come from. Like the, that 20% time, while it’s short term, is a cost center. It is where you could have a multiplicative effect in terms of profitability company-wide. If, if in that 20% time, the data science team can come up with something that automates a big part of your platform or delivers a lot of value for your users.

Keith McCormick: 00:50:39

Yeah. I’m gonna, if I’m not careful, I’m gonna misquote him, but I’m sure he will understand. So a quick shout out to Ken Jee who I’m sure everybody will, will be familiar with. I saw him give a keynote just about a year ago, and he was talking about you know, keys to like data science success and so on, and one of 

Jon Krohn: 00:51:01

ODSC East? 

Keith McCormick: 00:51:02

Oh, no, no, I don’t, yeah, 

Jon Krohn: 00:51:04

He did a keynote on ODSC East. Yeah. That was a… 

Keith McCormick: 00:51:07

Even been in London maybe? No, no, it was in Boston. Yeah. ODSC East. Yeah, it probably was. Yep. Yep. No, I remember it was it was in Boston. But one of the things that he was saying was, don’t, don’t pressure the data scientist to always produce, you know, ROI. I think I’m quite sure for the reasons that we just said, right? If you know, if you limit what they can do too much, you’re not gonna get one of the things that is most valuable about what they do. And we had speculated at some point we would have a conversation probably recorded specifically on this one issue. But, but that, when I said the gentle pushback that I sometimes hear, that’s really it, right? We, we have to come up with a way of doing this where the data scientists feel free, yet nonetheless, they have to spend the majority of the time, maybe not a hundred percent, but they have to spend the majority of the time on things that are known to produce value. Otherwise, you can’t keep the lights on.

00:52:10

So in terms of how to know that a particular project is gonna meet that criteria, for me, it’s pretty simple. You know, if you are gonna make a prediction, this is gonna sound for some people, that’s gonna sound terribly old school. But I’m gonna take this risk because I feel strongly about this. I think that generally speaking, we should be trying to predict yes-no questions. Not because we’re always presented with yes-no questions by the business. I, I know that that’s not the case. I’ll, I’ll give you an interesting healthcare example in a second. But because when we deploy it and we’re trying to decide whether or not to intervene, that almost always comes in a yes-no form. So, for instance, take a, a classic use case like preventative maintenance almost invariably you have to decide whether or not you’re gonna shut this thing down, inspect it, and verify that it needs maintenance.

00:53:10

But this could be a wind turbine. This could be you’re driving out to the middle of nowhere and climbing up those ladders, or, or you’re sending a drone up maybe to verify it or something like that. Every time you have unscheduled maintenance, it’s gonna be expensive. So it’s crazy to me not to do an estimate. So back to the healthcare example, and then I’ll, I’ll comment a little bit on how to make this estimate. So a healthcare example, a classic use case in healthcare analytics is 30-day readmit. Will a patient be readmitted on the same diagnostic code within 30 days? That’s a simple yes-no question. And then if the answer is yes, they are likely to retain, return with the same diagnosis, then you talk to the subject matter experts, medical doctors among others and say, how can we prevent this from happening? 

00:54:01

So it’s a simple yes-no, I remember being asked by a data scientist that worked for a healthcare provider once, let’s come up with a model that predicts the most likely next medical code that this patient will suffer from. So in that sense, it’s like a next best offer or like Netflix and Amazon and so on, right? But let’s just compare those two scenarios. What’s your chance of being correct out of the gate with the 30-day readmit, assuming the data’s in balance, or you force it to be in balance, you know, something like 50-50, you get, you get the idea. What about guessing which of 15,000 or 60,000? It depends on how granular you let it be about the diagnostic code. You’re saying also, what if you do know that diagnostic code and you’ve built such an amazing model that there’s a 20% chance that you’re right, which really would be remarkable if you’re trying to predict which of 15,000 it is and there’s a 20% chance that you’ve nailed it, right? 

00:55:05

How does that help? What, what’s the hospital do with that? It’s not clear, right? I want clarity about what you’re gonna do with it. I, if I have clear instructions, I pretty much assume, and I know that not all data scientists would agree with me, but I pretty much assume if I have a clear mission statement, I can build the model. A lot of people say, oh, you might not have the data, or the data might not be good enough, or simply might not be capable of being predicted. Over the years, I haven’t encountered those kinds of situations that often, if I have a clear mission statement, you, you know, why I have faith that the model can be built If it’s core to the business, if it really is critical to the business, they’re tracking it. And if they’re tracking it, there’s historical data. 

00:55:49

That’s why. So something new or too exploratory or too obscure might not have the data to support it, but something that’s core to the business almost always does, right? So I, I don’t want to get into a lot of speculation about whether or not we can build the model. If we’re choosing good projects, we can build the model. What I want to know is what they’re gonna use it for. I don’t need to know the interface. I’m not trying to step on anybody’s toes if they’re agile fans, for instance. We’re not trying to overthink the whole thing. We’re just trying to figure out what yes-no question are we trying to predict and why that’s useful. And that brings us to a confusion matrix. It’s really that simple. So I say that somebody is gonna come back within 30 days with the same disease that’s either a true positive or a false positive.

00:56:40

And in 30 days we will know which, and a subject matter expert at the hospital should be able to identify for me, for each quadrant of the confusion matrix, what the cost is. What if we do the intervention and it wasn’t needed that has an exact dollar estimate associated with it, or a narrow range, what if we fail to prevent it and the 30-day readmit happens, just look at the last 12 months of data and you have a darn good estimate of what that cost costs. So it’s really as simple as trying to solve the problem, at least initially with something that fits a confusion matrix, and then populate the confusion matrix with a financial estimate for each of the four outcomes. It’s not rocket science. 

Jon Krohn: 00:57:27

This was awesome. That was so articulately explained so clearly, and I had never thought of this before. This kind of your litmus test for a successful AI project is one that has a binary outcome that’s key to the business. So therefore historical data are inevitable. And yeah, when anytime we have a binary outcome with a model, we can create that confusion matrix and yeah, we can associate a financial cost with each of those four possible outcomes. And the confusion matrix. I love that. That is such a clear system. You know, it’s so easy when, when we came into this topic and I knew that I was gonna be asking you about ROI. It’s the kind of thing where I feel like people often give hand waving answers, and you just gave a crystal clear one. 

Keith McCormick: 00:58:31

Well, and you know, some people might say that that’s been oversimplified to the point of being naive. So let’s just, let’s let that devil’s argument devil’s advocate argument play for a second. Let’s say they come back and say, well, that’s lovely, Keith, but we really want to know why they’re gonna be a 30-day readmit. This, this almost gets us back to explainable AI, right? Maybe we want a reason code or something like that. I would say, well, fine, you’ve built your 30-day readmit. You can do a good value on that. If you already have that in place and your risk scores are accurate, then why not do something simple like K-nearest neighbors and find out what patient was most like patient Smith and not just come up with one diagnostic code, but maybe the whole list of diagnostic codes that, that closest patient had.

00:59:30

So you can, the, the point is you’re not going to derive value from the diagnostic code part, but it certainly can be a supplementary part of the solution. And I think you can see part of the problem is if you just go for the diagnostic code, it’s never gonna be accurate. It’s not as clear how you can deploy it, right? But now what you could do is you could take a model that maybe has only a middling level of performance, but combined with an accurate risk score, Hey, here’s our best guess of the cluster of issues that is gonna be presented this way. I remember I had a similar idea on a insurance fraud project where we built a really good model that gave a risk score on whether or not a particular claim was fraudulent. So that’s where the value was gonna come, because that’s what had the intervention associated with it. Do we send the investigators out or not? But I said, well, you know, let’s just play around with a little bit of a, you know, it is projects ended and it’s not always easy to renew them that the team’s not available and so on. But we had this idea at the end, and I always wish we had followed through on this, where it would’ve been cool as an add-on to the project, say what file that’s already been investigated looks like this particular accident.

01:00:52

It could even turn out to be the same criminal, right? This was this, these were staged accidents, so it was a organized crime thing that we were specifically looking for, right? But you could, and you can do that as a supplementary thing, or you could even do social network analysis. Go ahead and start with the binary, get your money back. Which by the way, for that project, I think was 30 mil or something, but get, get your money back. Well, I mean, like annually. Yeah. Right? So you’ve got your money back, you’ve got the goodwill. Now go ahead and supplement it with things like social network analysis, and maybe you can help the, maybe you can give the investigator a clue as to who might be behind this because phone numbers are matching or something, you know? But you get the idea. You, you, you have to have the core to the project has to be something that’s deployment friendly, and that’s gonna produce value. It, it’s not to imply that we’re gonna stop doing unsupervised and association roles and other things. It’s just that if you lead with the binary, that’s something that senior management’s gonna be able to understand, and it helps you come up with a budget and it helps you come up with a schedule. 

Jon Krohn: 01:02:02

I love it Keith, I have learned a lot from you today, and I’m going to be using this in my career. This has been invaluable for me. So earlier on, we were talking about in the context of ROI, we were talking about how we want data scientists to be a profit center as opposed to a cost center. So that the, the exploratory cost center stuff, the, the less well-defined stuff you know, is taking up a fraction of the time, say whatever the Google 20%, which I don’t think they do anymore. I don’t, but you know, they did for a long time. So then in your view, is there an ideal org structure for the data science team within an organization in order to ensure that they can be profitable or that they can be perceived favorably by the executives in the company? 

Keith McCormick: 01:02:57

Yeah, so this is a tough one because we were talking about a lack of consensus and XAI terminology. Talk about a lack of consensus, who data scientists report to and what to make the organizational chart look like zero consensus, right? In my experience, probably about a third of the time, data scientists report up through it and, and we’ve been candid about a whole bunch of other issues. So I might as well be candid about this too. I, I, I don’t think that’s ideal, right? So that’s probably the mode. It’s not the majority, but it’s the most common. And I think you really get that cost center, profit center confusion there, because in the, and then you get into this whole, oh, if we spend more on software, then we can spend less on people. And I don’t think people say that explicitly, but that’s why people get excited about these tools that citizen data scientists can use and so on.

01:03:57

That all that conversation seems to happen over on the IT side about how, oh, and then the other thing that ends up happening, I think sometimes is a bit of an obsession. I’m being a little bit strong here, but a bit an obsession with keeping the number of tools to a minimum. Everybody has to use the same tools, and that’s to minimize tool maintenance and tool licenses. I, I get it. But all those conversations, I hear more on the IT side than I do from data science teams when they don’t report up through the data scientists. So I’m a fan of having somebody like a CAO, which historically, this is my understanding at least, that it’s mostly in healthcare and quant on Wall Street, where you get, you know, a CAO, but I really think you need somebody with C-suite influence, even if they don’t literally have a C in their title, right? And that’s because only somebody with that amount of political capital is gonna be able to ensure that when you’re talking about value, you’re talking about maximum value across the whole enterprise. I, I just think there’s somebody, there has to be somebody with some influence there. And I don’t think that a CAO is the same as a CDO. And you know, now we run the risk of kind of getting in the weeds here, right? But I don’t think that data science and data governance are the same thing. I mean 

Jon Krohn: 01:05:30

No, no, no, no. I certainly, you know, they’re very different. Yeah. That’s not even to me, that’s super, obviously a chief data officer is more concerned with data privacy and yeah, good governance on data structures, whereas yes, a chief analytics officer, which could be more similar to a chief data scientist those two roles CAO and CDS are more concerned with building new models and automating things and providing predictive analytics to the business. 

Keith McCormick: 01:06:02

Well, here’s another distinction too. And, and the thing is, it’s, we agree that it’s almost like a no-brainer that they’re different. Yet I think there are a lot of organizations that try to kind of combine it because maybe they can’t justify having both roles or whatever, but the chief data officer and even the CIO for that matter, clearly those are cost centers, right? It’s, it’s a cost of doing business. We need a certain data infrastructure, we need to have data governance, we need to have privacy. But you’re, you’re not, you’re not ensuring data privacy to make, make money. It’s because it’s a cost of doing business, right? So I really want the CAO to be laser-focused, solely focused on identifying which projects have value and making sure that those are the projects that their team is assigned to do.

01:06:54

So another way of putting it is the number one job of the CAO is to maintain the analytics portfolio. So now we’re getting into almost portfolio management PMO type stuff, right? So what does that mean for an organizational structure? One of these days I wanna have a long conversation, preferably over some line, would somebody that’s a true Worldclass PMO expert. There actually are a couple that also teach for LinkedIn Learning that I’ve had short conversations with, but someday I wanna have a lengthy conversation with them is how, how to really make that work. How should a CAO be collaborating with a project management office and so on, right? But again, the CAO should be all about making sure that their team has the resources that they need to complete the projects that have greatest value. That’s it. But I don’t think most organizations have anybody like that. So, and a lot of organizations, the highest rank if to put it that way of the, the most important data scientist in the organization is maybe more like a senior manager or a director level. 

Jon Krohn: 01:08:10

Yeah, I think that’s right. Yeah. I mean, it’s there might be companies out there that wanna find somebody that can be like a chief data scientist or a CAO, and it’s hard to find people with that level of experience, or when you do find them, they can be very pricey. 

Keith McCormick: 01:08:26

Yeah. But you know, that’s absolutely true. But think about the number of folks, I’m sure you have them among your acquaintances on LinkedIn. Think about the number of folks that are in that league that are really incredibly talented, that if you look at their LinkedIn, they’ve been at, you know, eight organizations in 10 years, or maybe more than 10 organizations in 10 years, right? I don’t think it’s just Cost Organizations don’t know how to keep them. And I think part of the problem is they think that somebody like that is their first hire. So now that person arrives and they’ve got no team. Well, how are they supposed to do anything? So they’re going crazy, spending probably a fourth or a third of their day working with HR to try hire team. And hiring a good data scientist can take three to six months or more. So next thing you know, it’s almost the end of the year, and they have that painful, what-have-you-done-for-me-lately conversation with senior management? And they say, well, for a, for almost a year, I’ve been trying to put an infrastructure into place. I, I’m pretty sure that’s the story at a lot of organizations. 

Jon Krohn: 01:09:47

Yeah, that’s insightful. You could be, right. So you’ve mentioned LinkedIn a number of times in recent sentences. 

Keith McCormick: 01:09:55

Oh, yeah. 

Jon Krohn: 01:09:56

And you know, you are an instructor of data science and machine learning courses as has come up already in this episode through LinkedIn Learning, as well as through the University of California Irvine. And so, from your vantage point there we’ve talked a lot already in this episode about what’s missing from companies in order to have success with AI, but what do you think is missing from data science education, both formal and informal, to help aspiring practitioners prepare for industry? 

Keith McCormick: 01:10:31

Well, again, I’m apologizing for being old school, I guess, almost 30 years doing this, depending on when you start counting. I, I guess I shouldn’t apologize for that, right? But, you know, it’s, it’s it’s, people are always influenced by how they learned. But also, I can take the long view, right? So here it is, here’s the number one thing that’s missing, is that CRISP DM or something like it really understanding not just the machine learning lifecycle. Like, you know, people put a little diagram on a slide, and you start with problem definition, then you do data prep. I, I don’t mean lip service to it. I mean, really understanding the machine learning lifecycle and the implications for project management and working with clients, in other words, points in the process where the client should be making the decision and not me, you know? So as much as I think that Kaggle has been a positive influence for our community, and I really do think that that’s true, it has distorted our view of what a machine learning project is, because it’s a modeling competition. It’s not a machine learning project competition, it’s a modeling competition.

01:11:53

So I think that we have, I mean, again, I’m gonna make myself sound like the old, old guy, okay? I think we have a whole generation of data scientists that are, well, now I, I, I think I’m oversimplifying because it was true when I started out too, but we’ve always had an obsession with modeling and modeling algorithms, and haven’t understood enough how you get a project from beginning to end, including the, the cultural and organizational stuff that comes with that. And I don’t know why no time is spent on that, you know, in something like a machine learning bootcamp, where the primary reason somebody’s there is to really sharpen their skills, usually with a specific set of tools like Python libraries and so on, right? I kind of get it why they don’t get it fitted in there, but how data science master’s programs, for instance, don’t spend enough time on this. They have no excuse. It’s crazy. I mean, they should have entire courses dedicated to it, not half dozen of them, but for goodness sake, at least one. 

Jon Krohn: 01:13:11

Right. I think you’re absolutely right. And I haven’t thought the thought that you just conveyed so clearly, but you’re absolutely right. And it kind of reminds me of a, a thought that I have had clearly, it reminds me of how it, it’s surprising to me how much of elementary and so primary school and secondary school education is focused on abstract skills like calculus and chemistry, that the vast majority of the people that take those courses and memorize everything about them and really know how to do partial differential calculus and understand how what compounds need to be combined together in chemistry to create this other compound. And more than 90%, maybe more than 99% of people who do those courses never use those skills in their work or in their life. And it’s, I don’t know, like I’ve, I’ve heard of, you know, I’ve seen on TV shows that there are things like home economics courses or whatever, but it, I, I don’t know. 

01:14:26

We never, growing up in Ontario in Canada, we didn’t have anything. There was nothing like that. There was, so you can kind of see the analogy here. It’s interesting how there are topics that I guess in curriculum developers think, wow, this is a really fascinating thing. And it really, it’s something about the universe that is extraordinary. So this could be data science models, like, you know that this, the ChatGPT works. Wow, that’s crazy. You know, how does it work? Or that organic chemistry works, how does that work? Partial derivative calculus. Wow. But it’s rarely practical information that somebody needs to know. And now, some small portion of those people that do those courses, yes, they, they go on to study it in university, they do a PhD in it, they, they become an expert in it. And you don’t have to get a PhD to be an expert. You can also develop it in the field. But it, it is just interesting to me. It seems unbalanced and yeah. 

Keith McCormick: 01:15:36

Well, and, and you mentioned, you know, I’ve seen a lot of these lists of the 10 skills you need to become a data scientist or whatever. So let’s just take one thing that often comes up. I went to an engineering school, so I took quite a bit of calculus, you know, up through differential equations and, you know, and everything. And it wasn’t really my jam, but I survived it. You know, linear algebra for whatever reason was a mental block. Maybe I didn’t like the professor. I don’t know. But that’s always on a list of things you absolutely positively need, you know, to do data science. And that’s not, that’s not in my bag of tricks, but why does this even come up in conversation? Well, because if you’re gonna work for one of the, one of the fang companies, then you’re doing, you’re doing machine learning at such an unbelievable scale.

01:16:31

I mean, take Netflix for instance. I mean, is it billions of transactions? Like how, how many, how many seconds or minutes do you have to wait? I mean, it, it’s just an unbelievable scale. I mean, what, something like a fifth of the world’s population probably is on Netflix, and I think it really is something like that, right? Hundreds of millions, certainly. LinkedIn is almost a billion, isn’t it? Seven or 800 million, something like that. Unbelievable scale. So if you’re working for companies like this, you’re not using off-the-shelf software. You have to write these algorithms from scratch. But why on earth is that our model for what a data science team should look like at a regional bank that’s trying to prevent loan defaults, or an insurance company that’s trying to prevent fraud, or a small health group that has a dozen hospitals? It’s insane. 

01:17:28

To me, it just doesn’t make any sense. Now, I’m, I’m not saying across the board that you shouldn’t solve those problems, you know, with code or that knowing, really knowing the behind the scenes, but for me, knowing the history of the algorithms and how they work is sufficient to manipulate the hyper parameters and so on, right? I don’t necessarily have to write the algorithms from memory from scratch. I get why some people find that skill valuable, but I’ve been doing this for enough decades that I’m quite comfortable saying I don’t need to know that to do what I do. And I think I’m bringing plenty of value to these clients, right? So there just seems to be some real confusion about what you really need. So to send someone out with an extra helping of that kind of stuff, but leave them completely and utterly unprepared to scope a project and write a client contract doesn’t make any sense to me, because then the solopreneurs have to learn that on their own, which can be very painful. Or hopefully if they’re gonna go the consulting route, then they do a round of consulting at more on the business consulting side before they go more technical or something, right? I mean, somehow that has to be addressed, but I don’t know why that’s not addressed in a data science masters. 

Jon Krohn: 01:18:59

Yeah. I think Serg noted for me in his research that you have discussed previously something like a doctor’s residency for a data science. 

Keith McCormick: 01:19:08

Oh, well, that’s that’s actually a gentleman, Usama Fayyad says that, and I borrow that from him. So he’s, he’s been in the business for decades. He was one of the co-chairs of the first KDD conference, and he is the inaugural director, I think is his title at Northeastern’s program for experiential AI. And that’s the metaphor he uses. And I think it’s really powerful that, okay, you’ve got your medical degree, but you have to do residency before you go out into the field. But he says that because it’s completely absent from everybody’s program, except for the program that he chairs, right? And, and I agree with him that more people have to do that. So at Northeastern, they work with postdocs, they work with doctoral students, they work with the whole, the whole gamut. And they could speak to the details better than I, but they work on real world, world projects together, almost like university environment and think tank and consultancy all rolled up into one. 

Jon Krohn: 01:20:19

Yeah, it’s a great idea. Although I didn’t know, that it came from that that vantage point of, of having a program that does it. Yeah.

Keith McCormick: 01:20:30

To my knowledge, the only one. 

Jon Krohn: 01:20:32

Yeah. Good. All right. So we’ve spent a lot of time in this episode on what we need to have successful data science projects within a company. We’ve now spent time also talking about what’s missing in data science education. Are there any particular tools that you think an aspiring data scientist must learn? Like, is there some, is there some tool or tools out there that you think are critical that some people are missing? 

Keith McCormick: 01:21:01

Well it’s not so much. I mean, there are some tools obviously, that I use on a regular basis, and I’ll, I’ll share those in a moment. But I think we have to find a way to deal with this fact that for a long time, at least, the cool kids had to do everything a hundred percent coding, right? Because the idea was that if if you didn’t do that, then somehow you were revealing that you didn’t have the skillset that the cool kids had, or, you know, what have you, right? Because I’ve been on calls with clients and they’re working in, I’ve actually had tons of these kinds of calls where I’m having mentoring calls where for whatever reason we have to do a calculation or rerun a model on the fly, and we just say, you know, the most efficient way is gonna be able to do it like right this minute. So I’ve had that experience where on the other side of the Zoom call, they’re using a tool that’s like like IBM SPSS modeler, which I used forever, or now more often it’s KNIME, right? But I’ve also been on calls where they’re in a Jupyter Notebook and they’re, you know, doing Python or whatever, okay? I think that low-code, no-code is just playing faster.

01:22:35

And again, I, I’ve run the risk of not being invited to have lunch with the cool kids, you know, if I say that. But the, again, it’s, it’s partly, I, I don’t wanna be overly blunt, but sometimes it feels like a show off kind of a thing. But the other reason, and this is, this is the moral legitimate reason. If we don’t do everything in code, then we’re gonna run into problems when we go into production, you know, because we’re gonna have to rewrite it or something like that. Usually the argument that’s made, or if everybody in the organization is a Python coder, then that is inherently gonna make things more efficient. It makes a lot of logical sense, doesn’t it? But I’ve never observed it to be true. 

01:23:22

I’ve never observed a seamless deployment from a prototype model ever in almost 30 years. And I’ve never actually experienced being inside a building where everybody uses the same tool, right? So I I, I get that it’s a logical argument, but if it doesn’t exist in the real world, I, I don’t think it carries a lot of weight with me. So I think that low-code, no-code shouldn’t be just for the so-called citizen data scientist, because I’ve never really understood why everybody in the building should be building models. Everybody in the building should be exploring data, but that’s BI not machine learning, right? So I don’t, I, I don’t think that’s the argument for low-code, no-code. I think the argument for low-code, no-code is, it’s faster. 

Jon Krohn: 01:24:21

Yeah. I think especially because there are lots of gotchas that you need to have some expertise in, like selection bias when you’re building models feature drift, there’s tons of examples. So those could be dangerous if anybody could be doing it. But in terms of exploring data and being able to draw your own conclusions from data, you know, plotting some trends, and yeah, everybody should be able to do that. And yeah, it’s I, I definitely get your point that it, it does not seem like the coolest thing to be discussing low-code, no-code for data scientists, but you could see how it would make things more efficient and it could reduce the number of issues on interoperability between different people’s processes, moving things to production. And now, so you have a lot of experience with low-code, no-code tools. So you’ve created a lot of courses, you’ve written a lot of books on these kinds of tools. So you started as an SPSS which is the statistical package for these social sciences, if I’m remembering correctly. 

Keith McCormick: 01:25:29

Yeah.

Jon Krohn: 01:25:30

And, that was my first exposure to statistics. And I thought it was great. I mean, it was, I felt like I had a good understanding of what was going on under the hood, you know, whether I am using a click-and-point UI in SPSS or writing it as a line of code in some Python package you know, there’s, it’s the same level of abstraction, really. 

Keith McCormick: 01:25:57

Well, I mean, there’s so much to say about this. A quick, a quick note that supposedly they retired the acronym a long time ago, so it became you know, kind of like MCI, the, actually does that company still even exist? But there’s certain

Jon Krohn: 01:26:13

Acronyms, it’s like BP British Petroleum. 

Keith McCormick: 01:26:15

Yeah, yeah. Eventually it just becomes the acronym because they didn’t wanna limit themselves to just the social sciences. But the metaphor that I think is appropriate here, and it really fits SPSS too, is it’s like a single lens reflex camera or, you know, even those really cool muralist cameras now. And I, I think you probably have more a audio-video talent, you know, than I do. But, you know, these cameras, it could be $5,000 camera or something. It’s got an auto setting, you know, but you can turn that thing off. And that’s what I think is powerful about a low-code world, probably accurately to call it a something like KNIME for instance, the KNIME analytics platform. Yeah.

Jon Krohn: 01:27:03

And that’s, let’s spell that out for our listeners. Yeah. KNIME. And I’ll be sure to have a link to that in the show notes. It is a tool that I actually, I first learned about it at ODSC West 2019 at a lunch or something. I was sitting next to somebody who was giving a talk of KNIME, and I hadn’t heard of it before, but now I, it pops up all the time. And so yeah. Let our listeners know what is. 

Keith McCormick: 01:27:28

Yeah. So in the K in KNIME, by the way, stands for Konstanz Germany, cuz that’s where the, the company university was founded. But it’s it’s what they call visual programming, which is also SPSS modeler, SPSS Statistics is what you were referring to. You know, it, it physically looks like Excel, but it’s got very different features and it does statistics. SPSS modeler looks like a flow chart. And that came out, believe it or not, in 94, 94, it was called Clementine way back then. KNIME has been around for probably a little bit less than half that time, but many years now, 10, 12, 15 years plus. And KNIME also looks like a flow chart. So you can rapidly draw the flow chart, and if you are gonna run everything on defaults, you don’t have to go in to the symbols and make changes, but you can indeed make those changes.

01:28:28

So in that sense, again, I invite people to kind of imagine that they’re working with a really nice camera and they could get a simple everyday lens that worked under most, you know, a, a simple zoom lens or something, and then keep everything on auto and they’d never have to go to photography school, you know, learn all the, all the fancy settings, but the fancy settings are there. So you can rapidly prototype, but then you can go deeper when you need to. And I think that’s the, the power of tools like this, so particularly young data science teams, where I really think the coding thing starts to become a problem is when it completely drives not just what the team aspires to do, but it starts to affect things like hiring. “Oh, we can’t hire that our person because we’re a Python shop.” Right? Or we can’t hire that person with 25 years of insurance fraud experience because they’re a SAAS person.

01:29:44

Then, then I think the tail is wagging the dog, and I think it starts to get a little crazy, right? So I just encourage folks to say that these, these tools should be on the table. That doesn’t mean that everybody in the world should adopt one of these tools, but we should make it clear that it’s okay to learn these tools. And the other thing I would add is that I think introductory data science classes, generally speaking, should use these tools because then you can focus on the concepts and, you know, somebody might take two semesters of a data science class using a tool like this, and at the master’s level, they might decide to do an MBA, what a talented analytics manager of that person’s gonna be because they really understand some of it, right? Whereas if we put Python 101 as a barrier of entry, we’re reducing the top of the funnel.

01:30:42

And from an education standpoint, that doesn’t make a bit of sense. So the way I’ve solved this problem at UC Irvine, because I’m not faculty, I, I, well adjunct, whatever you want to call it, I’m, I’m an instructor in the department of continuing ed, and I nearly walked away from it a couple of years ago because they were, they were gonna change the, they were gonna change the program and it wasn’t quite clear what my courses were gonna fit. Because invariably there’s issue about whether or not my courses should come after a Python intro or what have you, or if everybody’s gonna do Python or if there’s ever gonna be a program that involves an option other than Python, right? It’s a debate that universities are all having. But what they did to entice me, and it was kind of an offer I couldn’t refuse, was I’m working on a three course sequence, it’s probably gonna start next fall, but where I’m the only instructor, and the reason that this solved this problem was since it’s only three courses, there’s no pushback that it’s gonna be KNIME because there’s no room in three courses. 

01:31:52

You can’t have the first course be Python grammar, a third of your time is over, right? You’re not gonna be able to learn any in a short certificate program like this. I don’t know where they’re headed. I don’t know if they’re doing a career change or if they’re someone that is maybe more IT or data engineering, but they wanna learn more about machine learning than they already know, right? But with three courses that are 10 weeks each, I can weave in the project management and the value estimation that we were talking about earlier, because if I had a course that was called Machine Learning Project Management, nobody would want to take it, because they would think that wasn’t sexy enough for their, for their CV or for LinkedIn, right? So I have to kind of sneak it in. Reminds me of when I was a little kid of TV dinners, I’m dating myself big time, because that was like, I grew up when Tang and TV dinners and all that kind of stuff, they were the, the tin foil thing that you put in the oven, but they would put this little, this is like what I was a little kid. 

01:33:03

They would put underneath the vegetables, they would put like a little picture of a pirate or whatever, They had to, they had to trick you into, they had to like trick you into having the vegetables. I think it’s crazy that we have to trick people into worrying about analytics project management, but universities will tell you that if that’s a separate course, it doesn’t get the enrollment that a parade of algorithms gets. 

Jon Krohn: 01:33:34

Yeah. Fascinating perspective there. And I agree with everything you said, and it sounds like you have a course coming up that is tailored to, so you have a LinkedIn Learning course that you’re currently developing that is tailored to a business audience. 

Keith McCormick: 01:33:50

Oh yeah, very quickly on that, yeah, because what I was describing just now was a course was like the UC Irvine, but I do have, yeah, I do have a course on LinkedIn Learning that addresses exactly that. It actually already exists, but it’s Predictive Analytics Essentials: Data Mining. Right. And the reason that I still have data mining in the title, even though that phrase is a little out of fashion, is because there’s a substantial amount of material on it, on CRISP DM, not just a statement of CRISP DM, but how I use CRISP DM in working with clients and writing client contracts and managing projects on the whole nine yards. And I’ve got actually two versions of that. One for data scientists and one for executives. So there’s predictive analytics essentials for executives, and then there’s predictive some, it has some other name, but if, if they search for me on LinkedIn, they’ll find it. 

01:34:52

But I, I know what you’re referring to. I’m gonna do in just a couple of weeks, I’m gonna fly out to Santa Barbara, twist my arm. It’s a beautiful place to be. And the LinkedIn studios are there, and it’s actually in a studio environment that you do this recording, either audio only, as will be the case with this one. Well, actually it’s a, there’s a camera inside of the soundproof booth. It’s a really cool setup. They have, they also have full-blown studio with like two cameras and director and stuff. But this particular course won’t be in that format, but it’s an executive guide to AutoML, so it’s not a practitioner’s AutoML course. It’s rather walking senior executives and analytics management through what are the implications of this technology for my hiring, for my team composition for how long projects take and so on. And no one will be surprised at this stage what my general opinion on this is, which is that these are tools that can facilitate the data scientists not replace them. That’s the one-sentence version. But I go into all the phases and what phases the technology does really remarkably well and what phases are really human phases and how that all works together.

Jon Krohn: 01:36:08

Cool. I haven’t heard of a course like that before, but it sounds like it’ll be valuable. It sounds like you’re tapping into a great market there. Big opportunity. Nice. So what an episode, Keith, I’ve learned a ton from you today. Thank you so much for sharing your wisdom with us. As we’re reaching the end of the episode, this is the time that I ask for a book recommendation. And given all of your wisdom, I expect this is gonna be an interesting one.

Keith McCormick: 01:36:34

Sure. So a little bit of a random choice of a bit of a book geek, but this is called the Ghost Map. So can you see the author? Oh, it’s up at the top, Steven Johnson, and he actually has, I think it was a Google Talk or something like that. If, if you Google this, you’ll be able to see him giving a lecture on it. But what’s the deal with the Ghost map? So there was this epidemic really is the way to say it in London many, many, many years ago. And everybody thought that it was a cholera and everybody thought that it was airborne, but it’s actually waterborne. But the reason I’m recommending it, one is it’s a cool story. I’ve actually been to the neighborhood in London where this happened, and there’s a famous pump there. And what’s cool about it is that proving to others that your hypothesis about the data is right, is not the same as figuring it out yourself. 

01:37:36

Right? It in other words, it’s a really, a whole story around correlation is not causation, but the way, the way you usually talk about that is therefore give up. We can’t talk about causation, but if you’re trying to save lives, you do have to get to causation because you have to either decide, is this air, is this an air problem or a water problem? And in this case, it was the water problem and nobody thought that it was right. So by the way, the hero of the story is John Snow. Not the Game of Thrones, John Snow, but he’s the one that figured out that a particular pump was the source of this problem for a neighborhood. And you can actually go there and there’s a John Snow’s pub. So for Stats Geeks, it’s, reads like a novel, but it’s really cool. 

Jon Krohn: 01:38:22

Yeah, and you’ve got the right audience here because I can’t remember who it was with, but a recent episode of the show, we were discussing this same this same story, not the same perspective as you had there. And so this, this book, Ghost Map sounds great. We didn’t discuss the specific book, but something epidemiological. Oh, I know who it was with. It was with Charlotte Deane, she’s an Oxford professor, and she was involved in the UK government effort to fight Covid. And so we got talking about kinda epidemic… 

Keith McCormick: 01:38:52

Oh, wow, wow. 

Jon Krohn: 01:38:53

Yeah, John Snow came up. Anyway Keith, it has been so awesome having you on the show. We’ll have to do it again sometime soon because it feels like we really only just got started, you know, we’ve been recording for hours, but there’s still plenty, I feel like I’ve only scratched the surface of all the the wealth of knowledge that you have there on data science. So we’ll have to have you on again soon. In the meantime how can our listeners keep up with the latest from you?

Keith McCormick: 01:39:21

Three quick ways. I have, I do have a keithmccormick.com, but better than that, sure, find me there if you like, but better than that is follow me on LinkedIn because I’m very active on LinkedIn almost every day. Certainly check out LinkedIn Learning. If you go to LinkedIn Learning and you search for a name like my name as opposed to one of the courses, you, there’s actually a homepage of, of sorts that lists them all. One of which is on causality by the way. So I have a little video and one of the courses on this, which is kind of a fun tie-in. And then finally, once a month I do an office hours. So it’s usually around the middle of the month. And what I do is I either talk just on my own about stuff that’s in the courses or I invite other instructors or other guests to talk about that stuff. So that’s a lot of fun and that’s a great way to stay in touch. 

Jon Krohn: 01:40:16

Nice. Thanks for those. We will be sure to include links to those in the show notes. Keith, thanks so much, man. It’s been awesome connecting with you again. And yeah, catch you again soon. 

Keith McCormick: 01:40:28

Thanks so much. I enjoyed it. 

Jon Krohn: 01:40:34

Keith really knows his stuff. Hopefully, we can get him on again soon so we can continue to benefit from his wisdom. In today’s episode, Keith filled us in on how AI is transparent if the client it’s being provided to has clear insight into how the model works. How in order for an AI project to be profitable, it can’t be open-ended. You’ve got to structure it to have a specific revenue goal from the onset. How AI models with a binary yes-no output are very likely to result in a successful AI project, because all four quadrants of the resulting confusion matrix can be associated with a specific income or cost. There is a YouTube tutorial by yours truly in the show notes if you aren’t sure what a confusion matrix is or would like a refresher. Keith also talked about how data science teams need to be led by someone with C-Suite level influence in order for the team to get the resources they need to complete projects that drive value enabling data science to be a profit center instead of a cost center within the organization. 

01:41:27

And he talked about how CRISP DM the cross-industry standard process for data mining is a process model that helps ensure AI projects are successful. As always, you can get all the show notes including the transcript for this episode, the video recording, any materials mentioned on the show, the URLs for Keith’s social media profiles, as well as my own social media profiles at www.superdatascience.com/655. That’s www.superdatascience.com/655. Beyond social media, another way we could interact is coming up on March 1st when I’ll be hosting a virtual conference on natural language processing with large language models like BERT and the GPT series architectures. It’ll be interactive, practical, and it’ll feature some of the most influential scientists and instructors in the large natural language model space as speakers. It’ll be live in the O’Reilly platform, which many employers and universities provide access to. Otherwise, you can grab a free 30-day trial of O’Reilly using our special code SDSPOD23. We’ve got a link to that code ready for you in the show notes. 

01:42:26

Thanks to my colleagues at Nebula for supporting me while I create content like this SuperDataScience episode for you. And thanks of course to Ivana, Mario, Natalie, Serg, Sylvia, Zara, and Kirill on the SuperDataScience team for producing another information-rich episode for us today. For enabling that super team to create this free podcast for you, we are deeply grateful to our sponsors whom I’ve hand selected as partners because I expect their products to be genuinely of interest to you. Please consider supporting this free show by checking out our sponsors links, which you can find in the show notes. And if you yourself are interested in sponsoring an episode, you can get all the details on how by making your way to jonkrohn.com/podcast. 

01:43:03

Last but not least, thanks to you for listening. We wouldn’t be here at all without you. So until next time, my friend, keep on rocking it out there and I’m looking forward to enjoying another round of the SuperDataScience Podcast with you very soon.