SDS 730: How GitHub Operationalizes AI for Teamwide Collaboration and Productivity, with GitHub COO Kyle Daigle

This is episode number 730 with Kyle Daigle, COO of GitHub. 

00:20

Welcome back to the Super Data Science Podcast. Today we’ve got a special episode for you that was recorded in front of a live audience at the ScaleUp: AI Conference in New York a couple of weeks ago. The slick conference was put on by Insight Partners, which with $100 billion in assets under management, is one of the world’s largest venture capital firms. My onstage guest at ScaleUp AI was Kyle Daigle, the Chief Operating Officer of GitHub. He’s exceptionally passionate and well-spoken about the way generative AI tools like GitHub’s super popular Copilot tool improve not only the way individual software developers and data scientists work, but also dramatically transform the way people work across entire firms, how they collaborate. All right. So let’s jump right into this exciting conversation.

01:07

Kyle, you work at GitHub. GitHub is home for all software developers around the world, and so it’s natural that software is used across your operations there, which is really relevant to you. You’ve been the COO of GitHub since May, but you’ve been at GitHub for a long time. You started as a developer there 10 years ago. So this morning, you published a blog post and original research that developers are the first group truly adopting AI at work. Why should these findings matter to developers as well as non-developers? 

Kyle Daigle: 01:38

Yeah. Thanks for having me, Jon, and Insight, it’s so great. I think it’s interesting, we’ve been hearing all the stories this morning about how AI a year ago, or AI two years ago, and it was only really a year ago that we brought Copilot to market, GitHub Copilot to be specific nowadays. And since we’ve done that, we’ve seen such this incredible in-the-flow productivity unlock that Copilot has allowed, makes developers 55% faster. We’re seeing code acceptance rates between 40 to 60%. So when you’re writing code, Copilot is writing 40 or 60%. And I think the real secret to that is that developers are doing their normal job. They’re in the editor, they’re writing code, and AI is helping them in that moment. And it’s making them incredibly powerful by augmenting what they already do well. So when I have a question, it makes perfect sense that I might turn to Copilot Chat and say, “Hey, what does this function do? Because that’s a normal flow for developers. 

02:39

So I think for businesses, the thing that we’re seeing is developers believe AI is actually making them more collaborative. I think when we put it out in the world, we thought, “Well, it’s going to make them more productive, obviously more efficient.” But 81% of the developers that we talked to in this study said it makes them more collaborative with their teams. And we have stats that show once they write their code in Copilot, when it gets to the pull request, it’s 15% faster to actually review that code when it’s written by Copilot and you have access to Copilot Chat. So it’s still early days, but I think what businesses should be thinking about and what I’m thinking about at GitHub, because my goal is to help every employee at GitHub harness the power of AI, not just our awesome developers. It’s how can you bring AI into the existing flows that your employees are doing? Not to a new app, not to a separate box because they don’t think that that’s giving the same power and efficiency that Copilot does.

Jon Krohn: 03:38

Yeah. Let’s talk through the logistics of what that feels like to be using GitHub Copilot in flow. So as an individual user, you have code completion, so a line of code, maybe even a paragraph of code. And then on the side, you have a chat bar that you can be throwing questions in natural language, and that is context-aware. So it’s aware of the code that you already have going, and so you don’t need to provide detail. It’ll interpret that context and answer for you. You can add if there’s anything I’m missing on the individual experience. But on top of that, what is it like to collaborate with Copilot, what’s that logistically? 

Kyle Daigle: 04:13

Yeah. I mean, I think what it’s really unlocking is it’s unlocking the creative part of software development where there’s design documents, there’s planning, there’s issues depending on your organization. Your developers are figuring out what to build before they build it with a product manager and designer. So when we talk about collaboration, we’re talking from beginning in that collaboration flow all the way to running in production or on a mobile app or on the edge or in store, whatever that is. So Copilot is helping go from that idea conversation and having more time focused on that conversation. It’s what we hear from customers is the design process is kicking up a little bit more than it was, not that waterfalls back, at least it shouldn’t necessarily be back, but that beginning conversation is more crucial.

04:56

And then when you’re building out your application, it allows the senior devs, the principal devs, the distinguished engineers to focus on giving the architectural system feedback, not, you shouldn’t use Gsub, you should use Split to split this sentence up or whatever, which is what so much time and code review ends up being. It’s all these small moments of teaching and helping the team. Now you can focus on a much higher level. 

Jon Krohn: 05:20

Perfect. Yeah. An example that you gave when we were in the Green room as well was this idea of having been a developer 10 years ago at GitHub, there’s code that could be still being used in production today, but nobody is aware of how we’re, and you’re not aware of how it works because it is 10 years old. 

Kyle Daigle: 05:35

100%.

Jon Krohn: 05:36

But your time’s really valuable as a COO. Somebody doesn’t Slack you and say, “What is this line of code?” Instead, they asset in Copilot. 

Kyle Daigle: 05:42

Yeah. So I mean, we all have projects of merit that are running in production that have a very small team running them. So it’s quite funny, just this morning I got an email, we have a bunch of automated systems that are saying, “Hey, you have to update this, or something happened.” And there was one system this morning that emailed me, “K Daigle,” my handle on GitHub. “Can you please log in and update this repository because it has to have a code change in it?” And I’m like, “Well, this isn’t my thing anymore. I don’t even remember what it was like.” So what we’re finding when you actually talk to the devs with our customers or in the community is a big secret superpower of Copilot is how it’s helping developers learn and upskill on the job. It’s teaching them how to do the next step or teaching them the new method that came out with the new library that you’re using, et cetera.

06:33

So just the other day I was like, “Okay. I want to try this out.” My team were about to launch our Octoverse report in two weeks. That’s the state of open source software and state of software on GitHub. We needed to make a website change. So sure enough, I said, “Well, hey, marketing team, why don’t you help me? We’ll open it up. We use Copilot Chat and make those updates.” And they were also able to do a little bit of that process as well. So it’s enabling us to not just write Greenfield code. That’s the easy pitch. The real pitch is let’s send a team into a 10-year-old repository that they don’t have experience with and let them ask the questions they need to be effective. 

Jon Krohn: 07:10

Yeah. And to be clear, this isn’t something that is unique to GitHub. Any client can be using GitHub Copilot as an individual contributor or for collaboration.

Kyle Daigle: 07:18

Yeah. We offer it to teams, we offer it to individuals. The chat is in beta and we’re continuing improve it. To your point, adding more and more context around the code. It’s not just the codes you see, it’s the issues, it’s the pull requests. Eventually, it’s the production systems that are showing you the statistics of what’s working, what’s not working. It’s your error tracker. We believe that all of that context is incredibly valuable to the developer while they’re writing code. And I think that’s the future of AI at GitHub is how do we bring AI to every step of the flow, in flow not a separate task that you have to do. 

Jon Krohn: 07:54

Yeah. So let’s talk more about that. So the developer experience, so the DevX, if you will, your research, which you publish today in your blog post specifically looked at the developer experience at this DevX lending data that a happier and more productive developer leads to better business outcomes. So the findings in there, like people, you’re talking about L&D, the thing that developers love most is learning something new. But at the same time, one of their biggest pain points, the thing that slows them down the most is mandatory L&D. So it seems like this is the solution. 

Kyle Daigle: 08:29

Yeah. I think it’s a solution. GitHub is not the only one struggling with this. We do our annual or biannual Harbor Pulse surveys that talk to our employees and say, “Hey, what do you need? How’s it working?” And of course, just like every organization, there’s some answer that says, “I need more time for learning and development.” So GitHub added learning days, and GitHub will pay for a large array of learning and development activities. But I think what the studies have shown is that developers are learning on the job with Copilot, particularly the junior developers or earlier-in-career developers or early-to-your-company developers. But also the senior developers, the principals that are going into different projects they haven’t quite worked with before and new technologies. 

09:09

There’s a customer that is making the change from Scala to Java. They’re going that direction and they realize they don’t have those engineers because it’s different enough. So they’re using Copilot to both do the work as well as to teach the difference via chat at a much higher speed than before. So whether it’s building code or reviewing code like Duolingo has nine… Sorry. Duolingo is a large customer of ours that has demonstrated that their code review process, a very collaborative effort has been 67% faster since Copilot came in. Now I want to be clear, every customer’s results are quite different. It very much means what’s your code base and what’s your experience. But I will say that there’s a floor here that is higher than zero. It’s usually around 50, 20%. Everything ends up being a bit faster, which creates new and novel problems for the rest of the workflow because now you’re generating so much new code that you previously weren’t. 

Jon Krohn: 10:14

Yeah. I’m sure that 67% number isn’t surprising to anyone here, whether they write code or not, because of our experiences, especially since March with things like the GPT-4 release that OpenAI had, you get this amazing sense of the context awareness over very large context windows and the expertise over so many domains that can be blended together into bespoke responses to you. So you were getting that same experience with code with GitHub Copilot. So the 67% doesn’t surprise me at all, and it’s amazing to think where we could be a year from now, given the amazing things that have just happened in the last six months in this space. In terms of enterprise readiness around AI, which I’m sure a lot of people here are interested in. You’ve worked with thousands of developers on implementing AI into their work and into their team operations. So what’s your advice for businesses looking to adopt AI into their developer teams? 

Kyle Daigle: 11:08

Yeah. I think the thing that we’ve realized as we’ve rolled out Copilot is two things. Whether it’s Copilot or another tool, a side tool, a tool in your office suite or G-suite or whatever, first look for tools that don’t require net new behavior. Because even with Copilot, which I think for the software developers, they would believe there’s really no net new behavior. I’m just typing and it’s filling things out. A lot of organizations that are larger multinational, have code bases more than 10 years old, need a surprising, in my opinion, amount of enablement to use Copilot. Because it changes the way you need to work. If you’re used to working in a particular way in an old code base that isn’t very clear, the variables don’t make any sense, et cetera, you have to show, okay, well, you have to open a few more files. You have to comment here to help Copilot understand what you’re doing. 

12:00

So I think I’m going to keep harping in the flow things. I really think that’s the secret to successful AI rollouts. But don’t underestimate enablement with whatever tool you’re rolling out with AI. It is magical and quite powerful, but we see customers that just roll it out to their developers and they don’t get that immediate win. The win takes a much longer period of time, even though it feels like an automatic increase in productivity or efficiency. 

Jon Krohn: 12:25

Cool. And a term that I learned in discussion and preparation for our talk today in front of all these folks is innersource. So tell us about that term.

Kyle Daigle: 12:34

Yeah. I’m curious, does that term ring a bell for anyone or not really? 

Jon Krohn: 12:38

Innersource?

Kyle Daigle: 12:38

A couple folks. So when we talk to customers a lot, GitHub clearly is the home of open source. All open source software lives on GitHub, and the interaction model in open source is very different than probably the interaction model in your company, which is Kyle works on project A, Jon works on project B, I can’t see project B’s code, project B can’t see my code. We’re very separated. In open source, that’s not how it works. I can see all the open source on GitHub. I can contribute. If I see a problem in a library I’m using, I can send a pull request and update it. So the idea of innersource is how do we bring that to your company and how do we help you open up, maybe not all the repositories, but more of the repositories. 

13:20

So when two teams have something that they need to work on together, they’re not putting a ticket in someone else’s backlog, and then that’s going through prioritization, and then that gets delayed for three months. When you have a developer over here that can make the three-line change over to this library, that team can review it, accept it, and ship it. So all innersource is, is how do you use the entirety of your developer team in order to bring that impact to your business versus focusing in verticals or silos in private industry, which is not how open source works. 

Jon Krohn: 13:49

Yeah. It’s an obvious and intuitive concept. It’s nice to have a name to be able to put- 

Kyle Daigle: 13:54

Well, it’s intuitive until you go to your company and you say, “Hey, I’ve heard of innersource,” and now let’s open up every repository to the entire employee base. There’s going to be 14 different departments that are like, “Hold the hell on. We can’t do this.” So that’s usually where we spend most of the conversation is with compliance and security and your cert teams and so on in order to make it work. But American Airlines went through this transformation with us for a while to an incredible amount of success because it speeds up just like Copilot speeds up the developer to make them more productive. And usually just like Copilot rollouts, I say just start with a small number of repos. You don’t need to convince them that it’s every repo.

14:33

When we go to Copilot, we can show you all the stats. You can do all the bakeoffs, you can do all the next steps. Let’s just start by getting 10 developers, 50 developers, 100 developers, depending on the size of your organization starting to use it. We can measure that impact and then learn from there versus this feeling of we have to have all the information to start. I truly believe this gen AI moment is, it’s compound interest and the interest rate changes. It’s not always going to be high interest, hopefully. It’s not always going to be low interest, but if you add it into your organization, you’re going to get benefit of varying size as the technology changes, as you enable your team, as you roll it out to more people. The only people who are not getting benefit are the people that are stuck in a room whiteboarding for two years deciding how to roll this out. They can at least get 1%, 2% improvement, and then take it from there. 

Jon Krohn: 15:25

Nice. Well said. So yes, at a ScaleUp: AI Conference like this, obviously developers are really important because they are bringing AI to life for your users, for your customers. But beyond just developers, what insights do you have for non-developer teams on how they can be prioritizing where to apply AI? 

Kyle Daigle: 15:44

Yeah. So I mean, as COO, this is my bread and butter now. I’m a developer by heart, so I get passionate about all the technology topics, but ultimately, my goal is to bring AI to the hover population, all of our employees at GitHub. And what I’ve asked the teams that report to me is to look at the vendors we’re currently using, because I think it’s easier to start there, to be blunt, and then also understand how can we add this without a change in behavior. I keep saying this, but I think it is crucial. We’ve attempted to roll out various AI products that require a change in behavior, and they pop and they drop and they don’t stick because you have to do so much enablement to change someone’s behavior. It’s no different than changing benefits providers going from a cloud to Azure or something like that. 

16:29

So at GitHub, the one that has worked really, really well for us is what we call Octavo internally. It’s by a company called Moveworks. What they do is they help us solve our IT cases. I’m in charge of IT. Hubs at GitHub they do everything over Slack. There’s no email at all. Everything is in a Slack message. So whether that be teams for you or Slack, for us they’re going into a channel and they’re saying, “Hey, I need a new laptop. I map my two year refresh.” Instead of a human getting involved at all, the bot knows what that means, asks what laptop they want, and it’ll send it out automatically. So we’ve been able to drastically reduce the amount of IT support that we need while still having better customer satisfaction from our employees than when the humans were fully in the loop.

17:13

So that was a great and easy example from our side of where it’s really made a huge difference. And now we’re going down the line of, well, what about email? What about calendar? What about prep? What about calendar management for people that don’t have assistance to help them out? In looking at where do we not need to change behavior, trying it out, measuring it. And as long as we can prove that it’s going to have roughly equal ROI or best case scenario actually going to save us money, then we roll it out to the company.

Jon Krohn: 17:41

Fantastic. Well, yeah, so GitHub, the company that all of your developers already know and love making solutions for you, like Copilot, allowing them to be more efficient, and lots of great insights for us today. Thank you so much, Kyle, around how these kinds of, how AI can be leveraged to boost productivity, boost collaboration for developers and non-developers alike. Thank you. 

Kyle Daigle: 18:01

Awesome. Thank you. 

Jon Krohn: 18:04

Nice. Hope you enjoyed that dynamic conversation recorded live on stage at Insight Partners ScaleUp: AI Conference in New York. In today’s episode, Kyle covered how generative AI tools like GitHub Copilot are most useful and efficient when they’re part of your software development flow. How these kinds of generative AI tools can be used for collaboration. So not just on an individual basis by doing things such as speeding up code review. And he talked about how innersourcing takes open source principles, but applies them within an organization on their proprietary assets. 

18:34

All right. That’s it for today’s episode. I hope you found it both insightful and useful. If you enjoyed it, consider supporting the show by sharing, by reviewing or by subscribing, but most importantly, just keep listening. And until next time, keep on rocking it out there. I’m looking forward to enjoying another round of the Super Data Science Podcast with you very soon.