The future of software engineering, tokenmaxxing and AI in higher education Transcript, AI Summary & Key Points

00:01 Tokenmaxxing. is one way of bragging rights. It's like, how many pounds do you lift? And so on and so forth. But really, you know, are you getting hernia, or are you getting strong? We don't know. All that and more on this week's Mixture of Experts. Hi, I'm Aili McConnon, and welcome to Mixture of Experts. Each week, MoE brings together the sharpest minds working at the frontier of artificial intelligence to take you through the week's news.

00:26 And this week, it's New York Tech Week. So we're here at IBM's One Madison office and studio to bring you some of the on the ground conversations. And with me today, I have Kaoutar El Maghraoui, Principal Research Scientist in AI platforms, Neel Sundaresan, General Manager, Automation and AI, and Thiru Venkatachalam, Senior Partner and Enterprise AI Transformation Leader.

00:49 After the panel, we'll have a special section on AI and higher education with Justina Nixon-Sантil, VP of CSR and Chief Impact Officer. And today, there's three big stories we're going to cover. Tokenmaxxing is back in the news. We're also going to talk about Nvidia has a new super chip. But first let's get into the big conversation from New York Tech Week.

01:09 And this is the future of software engineering. And certainly on the various panels that I've sat in on this week around the city, there's been talk, of course, of AI replacing entry level coders. On the one hand, we have data on that front. And then on the flip side, you know, people are arguing that data engineers, you know, people who can supervise these systems, who can test them, you know, kind of deploy them.

01:35 And basically you don't have the human in the loop or, you know, even more important than ever. And so it seems like a great topic to kick off with. And, Neel, I know this is a topic that you're covering this week in Tech Week, so I'd love to start with you on the future of software engineering. I mean, we have seen this in software over years. Better tooling, better infrastructure, better ecosystem makes software developers productive.

01:55 Always, right? So even before AI, we did that. We had metrics on how to measure developer productivity. Years ago it used to be lines of code. The number of PRs. Number of commits, etc. and, you know, when your metric becomes the output, then, you know, it gets. Tends to get gamed, right? So we are seeing that with AI as well, right? AI is changing the way we develop software.

02:25 I mean, increasingly software is AI agentic. So in some way when I'm starting to design any software system, I'm going to say what are all the different components of it? Again, this is not. This is not new to software component based software development or service based software development or architecture was always known. But now we know that there is intelligence in each of these components are intelligence in each of these agents.

02:46 And how do I orchestrate? How do I bring them together? Is it the design phase? And then below each of these systems is this AI, which is translate to LLMs. How do I bring this LLMs into play? Now, one big thing that has happened that has changed, over the last, I would say 7 or 8 years in the way software is looked at. So for example, if you can, I think 2016 was it when Andrej Karpathy said software 2.0 where data is code?

03:13 Who needs code when you have gradient descent? It's kind of like, yeah. So I could replace chunks of my software with data and then put a machine learning model on top of it and change it. So if you take, for example, the Tesla self-driving car, right, story goes that it was hundreds of thousands of rules which said how to, you know, take the next decision when the car is a self-driving car, if it's at a stop sign or if there's a bicyclist passing, etc..

03:42 Now, if I have all the lidar data or the camera data, all the video data, can I build a model on top of it and make automatic decision? That was a big thing, right? But what changes with that changes? That is we go from deterministic rules systems or a deterministic logic that built in the code to probabilistic system that intelligent agents tell you.

04:00 So these agents are only as good, or these models are only as good as the data that's fed to them and what they learn from them. Right? So that is software 2.0. Software 3.0. We got into foundation models or frontier models where there is general intelligence, I mean quote unquote intelligence. I'm not going to say human intelligence. It's like there is general knowledge and intelligence that's built into the system.

04:20 And I can extract from that those frontier models to bring it to my ecosystem. So it's like when I have the GPT-5s or the Sonnets, etc. of the world. How do I bring that? I mean, these are the large language models. Small language models are a bit different. Play now when you go to software 4.0, which is kind of what I think of as the next generation of software.

04:40 These software systems are going to be a bunch of agent systems that are going to orchestrate, you know, either they're going to work with humans or are they going to work with each other? So human language is the language of communication. And these agents will either communicate with humans or communicate with each other. Today they communicate through human languages, but tomorrow it could be something else.

05:00 So that's where we are going into the future of software development. That's fantastic. And I love the others to jump in. You know that even as we get more and more complex with these agentic systems, you know, it seems like the role of the software engineer changes, you know, in terms of kind of orchestrating all of these. Certainly, I have a different angle to the same perspective that, Neel offered because I'm on the consulting side where we do custom application development for our clients.

05:29 And in addition to the evolution of this capability on commercial software development, like Neel talked about, the custom software development on the consulting side has dramatically changed. And what we have is capabilities this quarter, in Q2 of 2025, we did not even have it in 2024. Last quarter, 20 Q4. And it is, it is having a profound effect.

05:58 So just to give you an example, many of the coding engines that we talked about, whether it is IBM's amazing Bob and I've used it, it's fantastic. Claude Code, which is very popular, are Codex, and other coding engines, we in IBM has created a harness on top of it. Because if you think about software development, and I am a developer myself, start up my career in mainframe.

06:26 You do a number of things outside of coding, right? You do requirements gathering at the front end. You interact with other people to collect that requirements. You create design documents, high level, low level design, and then comes the coding and then post coding. There is testing deployment. What we are able to do in commercial software development for our clients, we are able to automate this full end to end lifecycle.

06:53 What would take maybe a 6 to 8 months period now into a very constrained period, right. And let's say a month, and it's getting better by the day to a level. I'm one of the ones that are very skeptical about its possibility because being a developer, you're always questioning, is it going to be as accurate as what I would code? But it has opened up a lot of possibilities.

07:19 So what we have now that is game changing is on top of these coding engines, which can actually code, we're able to automate the full lifecycle of the SDLC with context engineering and memory management at the front end. We provide as much input to the computer and AI so that it is able to write the requirements document, allowing us to iterate. It's writing the high level, low level design, allowing us to inspect, and then it writes the code, like Neel said, end to end, and then it is doing the testing and even

07:49 deployment. I am able to and differentiate whether it should be deployed in Azure versus AWS or IBM cloud. So it has come a long way is what I see. That's fascinating. And yeah, the strong consensus is that software engineering is more important than ever as it evolves. And how do you want to address I. I think these are all great points, and I totally agree with what was said.

08:14 But there is, I think, a different angle here, which is how is the the role of the software engineer or the developer changing over time? You know, right now it's not about, you know, writing code or syntax and so on. I think it's shifting to verification system to system architecture. It's it's shifting also to like overall integration, the efficiency, the security, those are really important skills.

08:36 But also this is also bringing challenges to like the new, uh, you know, like the students, you know, doing software engineering and studying that. What should we focus on, you know, that onboarding right now is also kind of we're facing challenges like how do we onboard right now these new fresh graduates rate. And to get them to that because right now those entry level jobs that you mentioned, they're disappearing.

08:58 They're being replaced because it's all automated now with all these code generators and so on. So right now the skills we're really needing, it's at a higher level. It's the verification, the optimization, the security, the hardware software co-design. Those are things that these LLMs still struggle to do well, especially having that holistic view, integration, getting the efficiency, performance, energy, all of these things are becoming super critical.

09:25 And as you know, now we're having more code generators at this rapid rate. We're also hitting the point that, okay, we're generating more code, more systems, more, you know, holistic end to end pipelines and so on that are fully automated. That raises complexity, more code, more technical debt that we're getting. How do we manage all of this? How do we to make sure that, you know, where are we generating all this code?

09:50 Is it needed? Can we maybe shrink that code base? And that requires somebody who's really understanding the system deeply. So that's onboarding. How do we you know, I think get that hurdle off because initially we had the sandbox where new software engineers, they will build experiments and so on. That is sandbox that we had initially for new beginners is disappearing.

10:13 And I think that also should revolutionize the way we educate and train software engineers. So it has to change so we can keep up with, you know, this new world of AI generated code. Yeah. Just to add to that, I think it's an excellent point about the evolution of our roles. The developers are becoming agent builders that. Orchestrators. Orchestrators, because it's not about coding, it's about how do you define the agent to do the coding?

10:42 Testers are becoming agent auditors. So when when you when AI develops the code, you can't go and edit the code. That that is gone. Right. So what you could do is validate the output and the rules that it has invoked to produce the code is what you could do. So it's definitely evolving. And we have evolves that we are creating that are auditing right.

11:04 So. Excellent. Even think that I mean we used to have these human computer interactions. I think now we're shifting toward agent computer interactions. And the role of the human in the loop is also becoming kind of a verifier orchestrator. And deeper like system thinking skills are definitely going to become very important. So if we look at IBM, right, to be doubling the number of entry level engineers.

11:28 And you can start by saying, hey, software engineer jobs are disappearing. Actually I have a very different view of that. You can go back even two years. Teams that hired new software engineers. What do they do? In the beginning they had given some document to read or there are some testing to do. Most of them play ping pong for the first three months because the managers don't have time for them, right?

11:49 And that is changing. I mean, we are running an experiment now. We want boarded thousands of software developers into our division and we give them tools like Bob. Now Bob can automate and augment in the entire SDLC lifecycle. So suddenly they feel more empowered, right? So in some sense, you know, we have internal studies have shown, right? IBM builds enterprise software.

12:10 Right. So if I were to say, hey, take the software and then get it FedRAMP ready. Now entry level engineers do not know what that means or how to do it. So they need a starting point. And tools like Bob become their starting point. So in some sense we often say, I mean, this is again feedback coming from developers within the company, which is, you know, Bob is a distinguished engineer or the fellow or the senior engineer, principal engineer for, for a beginner developer and for a principal developer or a fellow, it's

12:36 a beginner engineer who would take those ideas and put them into into code. Right. So I think it without time will tell. I mean, we can talk in six months and we'll tell you exactly what the journey has been. But we see more and more people, software developers, when they start fresh of college, right, are able to do things that they were not ready to do or they were not empowered to do or they were not trusted to do.

13:01 Right? So actually, that's a positive side of the story. Right. I like this idea of Bob having apprentices. And, you know, we can unleash Bob on the all the summer interns that have flooded into exactly. What we do. So in their welcome back, they get the Bob's track and above API key. Great. And then they go from there. And that can be involved in the upskilling as well.

13:21 Yeah. And you'll hear in the afternoon in Jessica we are working with the university to create curriculum to to get them ready for this AI style software development. I mean, the idea is not new, right? You can go back 40 years. Donald Knuth said. Well, you know, you know, the idea for literate programming was there before there was AI. And it was like, well, if I have an assistant and if I could speak to the assistant in English or human language and say, hey, go call this up for me, I just need to be a domain expert,

13:43 right? I don't need to be that engineer. So while people say it's taking away software developers job, I think everybody is becoming a software developer. And we see that. Yeah. So internally, yeah, internally we are seeing Bob being in not just by software. It's used by infrastructure, consulting, operations, finance, MISL, CIO's office yesterday communications.

14:07 You know everybody is using Bob in their own way. Marketing. Everybody is using Bob's I mean, so people are more empowered are and feel more capable, right? This is a rich topic and we'll move to the next. We could talk and I'd love to follow up actually in six months and see how it's all evolved. Our next topic that we're tackling is token maxxing.

14:28 And this is back in the news. Uber's CTO made some comments that went viral about, you know, whether, it's going to be hard to justify the AI spend, you know, when in the case of Uber, I think they blew through their token budget for the year in four months. You know, and kind of leaders are rethinking, you know, how to incentivize employees to use AI.

14:52 Um, and I know there's a new term floating in the ether valuemaxxing. And in terms of how companies need to be a bit smarter in terms of the AI spend. And I think this would be a rich topic. And I know, Neel, you're interviewed on it recently, so why don't you kick us off. This is a that is something called Goodhart's law. Everybody talks about it these days.

15:11 I mean we are learning all kinds of new economic terms. Jevons paradox are good. But the point is, if you create metrics and tell people to optimize on those metrics, people are going to eventually game it. I mean, I've got a great story from my economics friend who is to the he has three daughters, and he was encouraging his older daughter to potty train his youngest daughter.

15:33 And he told her, every time you take her to the bathroom, you'll get a jelly bean. And suddenly you are seeing that the older daughter was eating a lot of jelly beans, and the younger daughter was going to bathroom all the time. Then he was seeing that she was feeding her water all day long. So you can create incentives by, you know, creating a poor reward system, right?

15:53 So it's kind of like that. We used to have lines of code as a metric. So what would people do? They'll write a metric. So what would people do? They'll write a ton of lines of code. In fact, back then when you program a program in C, C++, people will argue about where to put the brace. Because we put the brace below, you get an extra line of code, right?

16:08 Uh, counting PR, counting PR without counting PR reversals, number of files that were changed. I mean, all of these metrics have been used, even with AI. And every time this metric was used, it was, it was gamed or it was I mean, it turns out to be incomplete metric. We measure these things because we can measure them because there is no real measure of productivity.

16:29 Right. So tokenmaxxing is exactly the same thing. It's like, hey, if I use a lot of tokens, that means I'm very productive. But, you know, it comes in various shapes. Number one, I can write elaborate prompts for some simple questions. I can make it create elaborate outputs for some simple, simple answers I should give. The models can give me. Two and we see that.

16:50 So for example, the reason people talk about tokens is because the model providers measure, you know, they charge you by tokens. So they say, hey milli per per million tokens, etc. but they can be verbose as well. So you can compare an older and newer model. And you can just say for the same task, it is actually using a lot more tokens than before. So it doesn't really matter how you price your million tokens, right?

17:14 If you say, oh, my million tokens are $10 and your million tokens are $20, I could. All I need to do is generate twice as many tokens and I'm good, right? I can game the system. So model providers are gaming system. The tools are gaming system. And when we talk about tools like Bob, it is not just the models and not just the tokens. It's also the, the context engineering that the developers have to do.

17:34 So it is kind of the harnesses of the developer produce, the skills of the developers as well. So with the same model, we can produce very different kind of results in terms of token consumption, in terms of quality of the products, in terms of the, you know, performance of the product, etc.. So we always think about like, you know, in Bob, we have we have multiple models that orchestrate and we measure in terms of cost, in terms of quality and in terms of performance.

18:00 And this spirit of frontier is what we try to optimize on. So it is not just the model, just not the number of tokens. Of course, tokenmaxxing is one way of bragging rights, right. It's like how many pounds do you lift? And so on and so forth. But really, you know, are you getting hernia or are you getting strong? We don't know. Right. And a lot of people and I think the mistake was made where they created a leaderboard and there is a leaderboard.

18:22 People want to be on top of the board. Right. And we see that even with AI it's like, oh, we beat that model by 5% because you're optimizing. I mean, we had a plan for, for example, we optimize to be on top of the leaderboard, not to solve the problem. So the focus has shifted because of that. And I think it's one of the many metrics we should count tokens.

18:44 But we have to count everything else as well. I like the the jelly bean maxing. I was one of five children, and there's a lot of jelly bean maxing when when people were being potty trained. But also the, you know, the deeper themes of the kind of the challenge of measuring productivity, but also, you know, kind of, you know, measuring many more outcomes or, you know, outcome focus.

19:04 And, um, yeah, I'd love to bring Kaoutar and Thiru, you into the conversation on this. Definitely. Tokenmaxxing by itself is not the right metric. And I think Neel said it rightfully here. Because that also has implications on the infrastructure. Token Maxing means like huge GPU, you know, spins and massive, you know, infrastructure costs and energy and all of that, you know, had environmental effects.

19:26 And so it's not just, you know, just looking at the model, but looking at also the entire environment and the hardware behind it. And all of that is very important. But again, I think the productivity measures, it takes multiple facets here. It's, you know, what's the ROI? Uh, you know, while all of these tokens, what's the efficiency also that you're driving from this.

19:47 So I think it is a complex metrics. Like you said, it has to take multiple things into account. And it's mostly are the tokens are generated leading to useful solutions. And you know things products that ship and bring, you know, money and solve real problems and so on. Those are the real metrics that we should be focusing on. Yeah. Just to add to that, I think the outcomes are probably the ones that are going to stick.

20:11 From what I have seen. Um, the cost of tokens is kind of keep going down, uh, continuously over the last 18 months, the cost of tokens have gone down, but in the same period, the consumption of tokens have gone up dramatically. Right. So that's the two parts of it. And I draw parallels to, you know, memory management and storage costs. I come from the mainframe world, right.

20:40 So we used to do programming just to move chunks of code in and out of memory, just to save memory. And it has become, you know, a non, non uh back then it was super important. So I think the tokens are going to go through that evolution. Their cost of tokens will become very very commodity, be very minuscule. At the same time, I also feel that the consumption of tokens would go dramatically up where I see outcomes play a role.

21:11 As one of the clients, conversation I had was very telling. The client said to me, we've invested in these tokens, invested in these LLMs and AI programs. They are working on it, but I am not really sure that leveraging AI to do a particular work is more cheaper than the human right. More importantly, I want IBM to tell me how can I measure the productivity and efficiency of a AI enabled workflow?

21:42 Is better, faster, cheaper than my human led work? Exactly. Let's say take a call center, right. And when we are in a position now to transform a human led legacy workflow into an AI led, we are still not in the place where evolving there to measure the ROI of a business workflow at the workflow level, right? We are working on it. We are kind of developing solutions, but that's what the client was asking.

22:07 And tell me, you're asking me to modernize this workflow, remove humans, make it automated. But can you tell me it was costing me X dollars before and Y dollars now and it's cheaper, right? The conversation also went to let's say I cut $2 billion in labor on the old legacy workflow, because I eliminated these roles and automated them. But I spent 2 billion in tokens, right.

22:35 Where does it take? And we concluded that there is one advantage there. With the 2 billion of labor, you would take about eight months to develop that function, whereas even if it cost 2 billion on AI, you could develop it in a month, right? So there is a speed to value. You could develop it faster, but it costs you the same thing. Element is sticking, right.

22:58 But we're still not there at a point where we can measure the outcomes. We can measure the, the, the cost of a redefined workflow. So it's a it's an area that is evolving. But certainly tokens are a very narrow way of looking at productivity. And I think also it's very important to also have very smart and intelligent orchestration of these models. I think we do this very well at IBM.

23:22 Like we don't you don't want to always use the frontier model for all the tasks. You have to choose carefully. There are certain tasks that you just need a small LLM. Right. And then these intelligent like careful routing, I think it's going to be important to capitalize on that because that is a big cost saving. Maybe I can just run things locally.

23:41 I don't need to pay like the cost of a big frontier model API for all the tasks. So certain tasks I just need a small model. Certain tasks I can just maybe use some simple regex expression software. You know, that could do these things for me. Instead of calling a big AI model, I think that's going to be also very important because the key thing is getting into the MVP with the tokens, that gives me, you know, the right products at the right cost.

24:08 I think that's why with Bob, we don't actually reveal the model. We don't reveal because we don't let you choose the model. But under the cover we orchestrate between the frontier models, even the multiple frontier models, open source models and our own LLMs, etc. I mean, like that's why we repeatedly say don't take your Ferrari back. Yes, I was going to say that it fits perfectly with Kaoutar's fit for purpose.

24:27 It's really important, because it's also you're not going to get the value even if you like. For example, if I ask you what's your name? You really don't. Only thinking for five minutes, right? You want an answer right away. I'm not asking for family history. So and a lot of the conversations about tokenmaxxing comes from, you know, I go to a tool and I pick the latest model that's available because, you know, I've got this principal agent problem.

24:50 I'm not paying for it. Somebody else is paying for it. Why not? Why not choose the Ferrari? Right. And then I just let it run, RAG. And you see in the cars. And that's why you see the Uber stories or the Amazon stories, or who's the latest story about spending $500 million in a month? I don't know if that's true or not, but you know, stories like that because it's it's I think it's irresponsible.

25:11 There is there is a thing to be said to ROI, right? Many times you have to invest in technology because it makes sense. And the ROI may not be there right away. So we should measure too soon. Then we'll have the wrong metrics, right? So we should continue to invest while we create multiple measures. I mean, a statistician a while ago told me, we measure this because we can measure this.

25:31 That doesn't mean that it's useful. That's the only thing we can measure. But as we go further and we get mature, we can kind of develop a bunch of metrics when we can say, now, is it adding value or not? If on day one, if you tell me A or B, show me the value, it may not be that because these technologies are developing. I mean, remember GPT-3? We said AGI is here.

25:50 No not really. Right. And we are here today much more capable models. And still there is work to be done. So I think that's something that should be remembered in the back of our minds. So for our last topic today, we're talking about Nvidia's new super chip, the Nvidia RTX Spark, which I think is interesting because of the fact that when combined with Microsoft Windows means that people can run agents, accurately, securely on their personal PCs.

26:19 And so I'd love to start with you, Kaoutar, because I feel like we were having a conversation, or I was perhaps listening to a Mixture of Experts episode a little while ago, you know, where you were predicting, you know, this this was in the wake of the OpenClaw, you know, kind of kind of. I'd love to have you put this, um, development from Nvidia in a bit of context of what's happened this year and whether it's, you know, kind of important going forward.

26:40 Yeah, I think it's a big development that actually these announcements are kind of shifting a lot of the focus to edge. And AI, you know, in personal computers. And I think also the partnership between Microsoft and Nvidia, it's a big kind of revolutionizing the PC. So one thing, you know, if you look at what's happening in the data centers right now and the cloud.

27:02 So Nvidia is also kind of trying to find other markets because many of its big clients, like Apple and they're, they're, you know, building their own silicon, they're building their own, you know, AI accelerators and challenging also Nvidia's dominance with that. So now, you know, it's, I think a strategic move for Nvidia to go into the personal computing and kind of changing the way, you know, we interact with our computers.

27:26 Like having agents run in on your personal computers with no, you know, cloud access. And that's also like I think security here is a big, big aspect because what this implies is, you know, you can have an agents go like watch every stroke that you have read, you know, have deeper access to all your files, personal records, etc.. And then, you know, the OS right now is moving from to traditional kind of just maybe like a police traffic officer, like managing resources and so on, to also have an intent, understanding

27:58 what you want to do. Because traditionally operating systems, you know, you just have it's just an application launcher that manages all these resources and tries to, you know, make things run. But it had no idea what you were doing. So the end user has to orchestrate. Open the app, click here, move things from here to here. Now we're changing that.

28:21 Now we're asking, you know with the RTX announcements having these powerful, you know, GPUs. So they even you know if you look at the hardware story it's pretty impressive to be able to run 120 billion parameter model locally. Right. Having 128 GB of memory, right, having even, you know, the NVLink between the, the, the, the CPU and the GPUs locally, all of these things were not possible before in a personal PC.

28:49 So I think this is going to bring a lot of the agentic workflows to the OS. But also it's changing, you know, the nature of operating systems and the role that the OS is playing. So I think it's going to be interesting to watch how this, you know, falls or unfolds. And I wonder if this is, you know, revolutionary technology or normal technology. I mean, I was talking to one of IBM's earliest Fortran compiler developers and he said, well, we have to fit a Fortran compiler in 4K of memory back in 1960.

29:19 And that's how we how we invented the half world. We come a very long, long, long way from there to 128 gigs of memory on your computer. We still it will run slow. It will still not be sufficient. Right. So we've come a long way. And the IBM was there back then? Is there now. You can run a trillion parameter model on your machine and that's great. But you know it's a part of the evolution.

29:41 We'll be interesting to see where it goes. In theory, to run, two on closes out with some thoughts on the topic. I definitely feel there is a this is a big development in the world of AI, and to some extent, a rebirth of the PC and its relevance in the computing world. Because if we remember back in the days, the servers and the mainframe to distributed computing, to mobile computing, there was a forward momentum towards, you know, continuously going towards that.

30:07 But this actually sort of takes us back a little bit to making the PCs more relevant. And I think that's going to then evolve into, again, edge being more mobile. Right. The same thing is going to happen in mobile apps. It is super interesting for someone like me to look at it, because in the edge scenarios, the arrival of Claude Art, we were all experimenting.

30:30 It's doing some things that are extremely, extremely helpful. For example, for years I've struggled with finding that one email from my email inbox that I could not find. It's able to find it, right? And it's able to, you know, I had like the storage maximum and it was able to find the two videos that were hogging the maximum space. And intelligently, I actually copied that video one more time.

30:59 For whatever reason, it was able to delete those two videos very quickly, freeing up space. So in terms of personal productivity, this development is going to be a game changer in the role of PC in what it used to be to where it's going. I'm very optimistic that there's going to be a lot of future development in this space as people who are innovators come up with new use cases of how to use it, because now with this chip PC is now enabled to, and equipped to do it, people will come up with ideas that will be very game

31:34 changing. Well, Neel, thank you so much for joining the panel. I'm now going to move this on to our next segment, which is looking at AI and higher education. I'm joined today by Justina Nixon-Sантil, vice president of Corporate Social Responsibility and our Chief Impact Officer. Justina, thanks so much for joining us on Mixture of Experts. And I'd love to start on the heels of New York Tech Week.

32:01 You were just on a panel about AI and higher education. What are some of the top challenges that, you know, administrators, students, professors are facing with AI adoption of this kind of moment in time? Yeah. I mean, this is so topical. You know, AI is a huge challenge for universities these days. Um, number one, students are graduating into a workforce that's completely different from when they entered university four years ago.

32:29 Professors are very focused on how do I retain students critical thinking, right. And retain the information and knowledge they've been gaining for years when they now have this AI tool to their disposal? Um, you know, from a governance perspective, right? Administrators and others are thinking, how do I protect student data? How do I make sure that students are being assessed the right way as well, which, you know, assessments have to, to change?

32:59 Um, and then how do I look at all the different tools that are being brought into the college system and making some real decisions about safety and, you know, AI ethics and trust? So those are the things at the top of everyone's mind. But I would say students are also thinking about this very carefully because again, they entered university in, in a market that's very different than what it is today.

33:23 And they're really thinking about how am I going to get a job, you know? So all of those things are coming together. And, you know, we're having a lot of discussions with higher institutions about some of the best practices and some of the ways that we can help them and partner with them as well. Yeah. And I'd love to follow up on the partnership a little bit later in the conversation, but I'm fascinated of those, you know, the various groups that are wrestling with it, administrators, professors, teachers, students.

33:49 Do you find that any of those groups are the most optimistic about the potential for AI? You know, it can be a tool to empower, of course, as well as, you know, needing to be governed and regulated. But yeah. How do you find that balance? I mean, I think students are optimistic, right? This is technology that they've grown up with overall. Right. These are like digital natives, and they've used technology ever since they were very young.

34:13 And now this is a new tool at their disposal. So on one hand, I do think that they're very optimistic about AI. I do not know a university student that has not used AI. And, you know, so they have access to the tools. They're using them. I think on the other side of that, it's a bit of apprehension, right? So although they're excited about having this tool, they're using it in different ways.

34:37 They're also thinking about what are their future prospects from a workforce perspective. And what does this mean? Because the workplace is changing so quickly. So I do see both sides in many of the students that I meet with. But just generally, you know, as you meet with professors and higher ed leaders, those are the things that they're thinking about as well.

34:57 And you mentioned earlier like assessment. And, you know, many facets of education having to change. But I'd love to dive in on, you know, kind of some of the ways that people or, you know, leaders in the field are thinking about how assessment needs to evolve. Um, you know, given the prevalence of AI tools. Yeah, I think the assessment is going to completely change.

35:15 Um, you know, if you give an assessment, a test or a paper, uh, for for students to complete, they can use AI, right, to acquire the information they need to, to, um, you know, create the answers or to write the paper. So you really have to think about assessment in a different way. A lot of schools are moving to oral assessments, right. Making sure the student can actually defend, right, the homework, the assignment, without any technology or any tools in front of them.

35:47 Um, and others are actually accepting it and saying, okay, great, use AI. And I think the best schools are moving in that direction. It's okay, you could use AI, but then you have to demonstrate to us, right? What have you learned from this? When what when did you use human judgment? Right. What did you think about the output that you got from AI? And does it make sense?

36:08 So I think the professors who are probing a bit more on these assignments, it actually mirrors the workplace. And I think that's where you're going to prepare more students to be successful when they graduate and they get jobs. That's so interesting. And this sort of plays into the theme of the sort of humans in the loop, you know, and kind of tying, you know, teaching students, you know, universities being places to hone critical thinking.

36:33 Of course, critical thinking applies to, you know, before, it used to be the kind of how do we use the internet? How do you trust what's going on? And it sort of feels like the, you know, evolving the kind of pedagogy around, you know, critical thinking to factor in AI. Yeah, absolutely. And I think a lot of schools are also bringing in more practical, hands on learning opportunities for students because it goes beyond just understanding the information or, you know, the content.

37:01 It really is around. How do you then demonstrate teamwork and collaboration and the human judgment and, you know, your presentation skills that all needs to be built into a university structure. And, you know, many universities have been doing this, you know, for quite some time. But I think it's even more critical now that this is just integrated in every classroom, in every situation.

37:23 And this hands on learning is really how students can demonstrate competence right across whether it's an AI tool, a topic, etc., especially when they are moving into the workplace. I like that in this sort of idea of extending evaluation to even, like you're saying, some of this feels like teamwork, that, you know, things that AI can't do where, you know, perhaps those had a role in the past and higher education, but they weren't as much, you know, as part of formal a part of the, you know, what students were being

37:52 evaluated on. I heard one university as well, or I think there's maybe a movement of universities, you know, in the realm of assessment as well for, you know, actual exams, having people kind of go back to sort of paper and pen. So you use all the tools, you know, to prepare, but now, you know, make sure that you can kind of also demonstrate. So it's an interesting kind of the combination of like, um, you know, the new and old.

38:14 Yeah. And I agree, I think all universities have to decide what makes sense for them and the governance structures right around AI. What we're encouraging universities to not do is to ban AI. I mean, this ship has left, right, like it has sailed and the workforce is using AI. Every major company is investing in it. So you want to make sure that you give access to those tools in a responsible way to students and make sure they are using it.

38:45 And then you have to bring in the layer of the assessments. And how do you make sure they're actually acquiring the knowledge. And they're, they're demonstrating competence, but you don't want to ban those tools. Students are using it anyway. And it feels like you are doing a disservice, as you say, for when they go to the workforce, if they, you know, because I feel like one of the amazing things about a, you know, center for Higher Education is that they can get access to these tools that, you know, if they're large

39:07 frontier models, they might not otherwise, you know, be able to. Um, and I feel like the, you know, many roles, of course, are evolving. But the I'm fascinated to chat a little bit about how, you know, the role of a professor, you know, and the person who imparts the knowledge. Um, you know, you see that evolving, you know, in these discussions about AI and kind of higher education.

39:28 Yeah. I mean, let's be honest, you know, higher education has historically been slow to move. And you always have the early adopters, you have the professors. And a lot of them we work with because they're the first ones that are willing to say, I want to bring in this new tool or this new practice. You know, I want to try something different. I want my students to learn in a different way.

39:49 So you always have these early adopters. I think what's different now is the speed of change. So although you're continuing to work with these university professors who are willing to step in, who are willing to try different things, willing to bring these tools, you have to actually move very quickly to all professors, right? So usually use them as champions, you usually use them as part of your overall change management.

40:13 But I think what's going to be critical is this can't just be the computer science professor, right? Or the engineering professor. You have to look at the humanities. You have to look across all of the different disciplines and schools at a university to really think about how AI could be implemented everywhere. It can't just stay in one area. So I think professors are looking at how do we bring this in, whether I'm in the school of business, right.

40:39 Whether I'm in the School of Arts, how do I bring in AI in a very disciplined approach with trust and governance around it? And then how do I change my curriculum? Right. So to incorporate the use of AI and then how do I change assessments for a professor to think about all of this? It's a lot. It's a lot. So I think this is where partnerships with IBM come in.

41:03 I think a lot of schools are creating like rubrics and principles around governance, and also showcasing the best examples that professors, especially the early adopters, are using to quickly be able to bring this across an entire school system. And then you have universities like Purdue University. And I, we just met with them. They've actually incorporated AI as a mandatory class across all of their freshmen.

41:31 So to me, that's where you get real change is no longer an option. It is something everyone needs to do. And we have a number of universities that we've worked with to incorporate our SkillsBuild program and our content in a similar way. I love that in this sort of idea like AI literacy, like, you know, you have kind of writing workshops, AI literacy being, you know, kind of a mandatory for all freshmen.

41:53 And I'm glad you mentioned, you know, IBM's work in this area because I just wanted to chat about, you know, the role of the private sector in helping higher education sort of step up because, as you were describing it, I was imagining various, you know, poetry professors who trained, you know, they're not, you know, as used to, perhaps a computer science professor in, you know, thinking of technological, technological tools to, you know, convey their information.

42:14 But, you know, what sort of role do you think the private sector, and particularly tech companies like IBM, have as this space evolves? I think they have a huge role. Number one, industry partnerships for higher ed. They are so critical, right. Higher ed institutions really need to understand what's happening in the workplace, and they need to understand the pace of change and how companies are integrating AI, what their expectations are for an entry level hire.

42:43 And then what does that mean for higher ed institutions and how they adjust their curriculum and prepare their students? So I think these partnerships with industry so critical for higher and higher ed institutions. Um, you know, when I look at the work that we are doing, right, and the partnerships we have with universities, we are bringing our expertise and our tools and our SkillsBuild program to universities globally, because providing that access to content, right is so critical for students, but also making sure

43:16 IBMers are part of this, right? We have so many academic advisors who are IBMers who are actually mentoring students, teaching classes as guest lecturers, sharing with them how work is changing within IBM so they could be prepared. So I think it's not just the responsibility of companies to provide that access to the training and content like we're doing, but how do you make sure they also have access to your employee base to really demonstrate those types of changes?

43:45 Um, and then also how do you partner that with talent acquisition? How do you make sure that you're also looking at the pool of students who have acquired some level of AI competence, right, and fluency? And how do you look at that as your pipeline, not just for your company, but for your clients? So those are the types of things that we are doing in our partnerships with higher education.

44:05 That's wonderful. And yeah, and I feel like for all of those students, we, you know, spoke earlier about the fact that they started university, the skills that they thought they needed, the sort of contract of like, you know, if I work hard in this, in this area, I have a job, you know, has changed. And, you know, it's combined with the pace at which it's changing.

44:24 It's good to hear that, you know, IBM, you know, I'm sure some others as well are, you know, helping inform and educate students because it's like these roles are, you know, all of our roles are changing kind of almost by the day. So to. Exactly, you know, for students to know what the how they're going to redirecting, um, you know, seems to be a critical.

44:39 I think one of the things that we are working with a lot of students around is how do you really focus on those other skills that you've gained? So if you entered school and you were focused on data science, right, or computer science. And some of those, you know, disciplines have changed right over the last four years. How do you take what you've learned, especially the hands on, practical, experiential learning that you've done and parlay those skill sets into what the new job looks like.

45:11 So don't focus too much on what you know, the actual degree that you've gotten. Focus more on the skill sets that will enable you to be successful in any of the new roles that are now available to you. Well, especially with each of our roles we'll kind of continue to change and evolve, you know, to translate. Justina, thank you so much for taking the time to chat with us.

45:31 And thank you to all our listeners. If you like what you heard, you can get us on Apple Podcasts, Spotify and podcast platforms everywhere, and we'll see you next week on Mixture of Experts.