Assessment and Community in the AI Age: A Conversation with Dr. Bill McCumber

Welcome to another episode of AI Goes to College, the podcast that helps higher ed professionals figure out just what in the world is going on with generative AI. I'm joined once again by my friend, colleague, and co host, Dr. Robert E. Crossler of Washington State University. And today we have another guest.Dr. Bill McCumber is the Associate Dean of Graduate Programs and Research at Louisiana Tech University's College of Business and the JPJ Investments Endowed professor of Finance. Bill came to academia by way of a career in wealth management. He has a PhD from the University of Arkansas.His research ranges across corporate governance, banking risk, hedge funds, microfinance, and quite a bit of it is related to how connections among the people running firms shape what those firms actually do. He also built our student managed investment fund, where undergraduates get to manage real money. Yeah, we get enough, I assume, for the foundation.So, yeah, we get none of it. That's right.

Well, thank you for inviting me.

All right, well, let's just get to it. So one of the reasons that we invited Bill on is that he and I were sitting next to each other at our fall faculty meeting.And in addition to making snarky comments, we chatted a little bit about AI. And during that meeting, Bill announced the formation of an AI Task Force committee or something like that. I don't know because I'm not on it.

That's right.

That's right. I was specifically excluded. I think the way you put it is Van Slyke is too obvious. And this is for the College of Business at Louisiana Tech.So, Bill, tell us, why are you putting together yet another task force?

The task force is really meant to be an effort to bring lots of different people with different skill sets in a room and do essentially what you're trying to do here. Let's figure out what people are doing. Business people and colleges of business, at least in my experience, are pretty entrepreneurial folks.And faculty generally are entrepreneurial folks. They're going to figure out what they're doing. They're going to do their classes the way they do it.They're going to have their materials the way they do it. So instead of having everyone in their own silos kind of doing their own thing, which they're still going to do, to be clear.But instead of having this just be a silent thing that people are doing and devolving into entrepreneurial chaos, let's at least compare notes and figure out what people are doing, what are they using? How and why are they using it? Where is it helping? Where is it becoming a problem?Maybe it's group therapy more than anything else in the beginning, but it's an attempt to get our arms around the elephant in the room. AI is everywhere. And frankly, in five years, if we're still saying, AI, AI, AI, I'll be shocked. It's.It's kind of like back in the day, we talked about the Internet. Oh, my gosh, the Internet. What are we doing with the Internet? The kids are using the Internet. It's all cheating.Well, obviously, kids are required to have laptops in classrooms for a reason because it can be a learning tool, and we just need to approach this in a proactive manner. This, to me, is culture work. It's not a tech rollout.

So, Bill, one of the things you said there is something that I've been pretty big proponent of. If we all stay in our offices and do this, we're all trying to reinvent the same wheel.And what it sounds like you're doing is creating a space to where we can reinvent the wheels together and figure this out together rather than alone. In higher ed, we really like to have meetings. We really like to do those sorts of things.What's the pace and the cadence that you're looking at for these conversations and doing things with this task force in a way that'll be productive but won't carry that burden of, oh, my goodness, another meeting.

Yeah, another thing we have to do. Right. Well, we're doing it anyway. And the. The problem is that, as you say, we.We could all be siloed and all this kind of thing and reinventing this wheel. The idea is to get as many people who want. So this is voluntary. I didn't appoint anybody right now.I tried to make it 10 people, and then I stopped saying no to people who wanted to join it. So Now I have 17. I have faculty from every department. I have administrators, I have administrative assistants. I have students.And we're all trying to get in the room and figure out what's what. So the survey went out to everybody in the college of business a couple weeks ago on my to do list today is to pull the results from that survey.The response to the survey was fantastic. This is on everyone's mind, obviously, so the participation was solid. I'm going to pull that data and just see what patterns emerge from that.The initial conversation is going to be looking at the survey data and asking some questions. And then my expectation, we'll see what happens, because it's always herding cats.But my expectation is that we're going to naturally split into essentially Subcommittees, how are people using it for admin assistant work? What about advising? What about scheduling class schedules? What about room deployment? What about research? What about teaching?How are students using it to help them understand material better? And we're just going to see, I don't want anything to be prescriptive.I want it to be community building and helping as opposed to we're going to use this tool in this way and you need to put your policy on the syllabus this way because that's not going to work.

Was really worried your first response was going to be leaving Van Slyke off of a.

Well, you know, again, you are the AI person around here.So I didn't want it to be Craig's committee only because I wanted to encourage people who are very much I don't know how to use AI or I don't like AI and I don't want to be told what to do. We need those voices in those meetings. So I mean my participation was role driven more than anything else.I don't want to be the loudest voice in that room.

The first workshop is going to have a pretty well known AI skeptic, which is an interesting first choice.

Why the speaker is and I don't think I'm speaking out of class here. The director of the School of accountancy, she is an incredible researcher. Her student evals are off the charts. Students absolutely love her.But her whole position at this point is no AI in my classroom, which is really interesting because it allows her to speak to what she's seeing with AI in the classroom and student use. But she's also going to share how she uses it, which is really cool.And I'm hoping that since Dr. Hunt is going to be the kickoff to the brown bag, that the people who may benefit but have this resistance given the state of AI, that they will come to the brown bag. And this is not missionary or conversion work.We're not going to convert the heathens, but we're going to open up some minds, hopefully to what is possible.

That's an interesting tension. God, that sounded like AI. That's an interesting tension. You know, she doesn't want it in her classroom, but she uses it.Have you talked to her much about that in her reasoning?

Well, her throwaway was, well, maybe I'll just talk about how I use it for therapy. Raises some other interesting question. She is all in. I think that's going to be great and I'm going to learn something from guaranteed.So I don't know what she's going to talk about other than it's not going to be, don't use AI. It's going to be a principal discussion like when is it appropriate to urge the tools?When is it appropriate to have students do this useful struggle? And the MIT report is a perfect example of that language coming to the surface of these broader conversations.

For those of you who don't know, MIT released a big report earlier this week, late last week. I'll put a link to it in the show notes, and we may, if we get time, we may talk about that a little bit.

No, I think what Bill mentions there is that I've heard a lot in conversations amongst faculty is at some level, when we start talking about AI, it's a push of, oh, you want us to teach AI and everything and do everything in the classroom.And what Bill's describing is a faculty member who's being very intentional about why they're not using AI and leaning to some of those skills and some of those aspects of humanity, if you will, that students need to have in order to be successful, even acknowledging that they will eventually be using AI to do many of these things.

Absolutely.

The good Lord save us from yet another across the curriculum initiative that might just push me into retirement to come up with another one.But that being said, AI is being used across the curriculum, not by faculty choice, maybe not by faculty encouragement, and maybe in spite of faculty prohibitions. But I think that's something we've got to deal with.Let's go ahead and go off script a little bit, if you don't mind, because you brought up the MIT report. I scanned through that, and we don't need to get into it in detail right now, but I think this is relevant to what we're talking about.As I scanned through it, I thought, good night. We're almost, what, four years into this.

Yep.

And we're still talking about the same stuff we were talking about three and a half years ago. I mean, I'm looking at that report. This is one of the finest educational institutions in the world.And I'm not really being critical of the report at all. What have we? Well,.

You look at an institution who arguably is a well known and well respected institution, probably one of the institutions that we would think of at the forefront of AI everything, and yet we're all having the same hallway conversations and classroom conversations three and a half years later, like this is some tsunami or Covid event. And to an extent, I guess it is, but it's pervasive and we need to figure out Educators. And this is across K through post secondary certainly.If this is the new world, how do we help students best? What are we trying to accomplish? What does it mean to be an educated person when information is no longer a barrier?

Yeah. And I think what makes this a really interesting conversation as well and attention that we as higher education have to wrestle with.As long as I've been in higher education research was really the coin of the realm of. And what we're asking people to do over the last three or four years is push into how to effectively make sure we're educating students well.But yet our systems aren't necessarily designed to reward as strongly as a in the world of Udallis level journal publication.And so I think it's really a redeploying of resources at some level to say this is valued as much as some of the things that we leaned into a whole lot for the last 20, 30 years.

I want to point out that's very institution dependent. There are some schools that really don't care about a top journal article if you're a crappy teacher.We're not at those institutions, but they do exist. But I think my broader point is it's time to actually do something.We've been talking about it, talking about it, talking about it and in fairness it's a very dynamic environment and it's really unlike anything we've seen before. Bill, you brought up the comparison to the Internet. You brought up Covid.Rob and I have talked about E commerce and distance learning and we've seen these kinds of things that have affected education before, but not to this extent and for this duration. Covid was massive, but it had an endpoint. Yeah, this is not going to have an endpoint, not anytime in the that I can foresee. Ever.So are we going to be having these same conversations three years from now? I'm kind of worried that we might.

I think I'm a finance guy, so I'm going to default to markets. Markets don't care about history.They don't care about anything other than there is a need and there's going to be solutions created to meet that need. All right. It's not a matter of should preference or anything like this. It's.And I guess my point is what we're starting to see is some real efforts around how we wrestle with these things.You mentioned that we've known for 20, 30 years basically that for example, reciting slides at the front of the room may be correlated with knowledge acquisition, but it's not the best means to do that. Right. So now you have these tools that can provide information ad nauseam. And we're going to have to figure out, again, back to basics.What does it mean to be an educated person when information itself is a commodity? Right. I think back to. I think. Was it Socrates?I think it was Socrates who bemoaned the spread of the written word as not representative of true knowledge. Because if it's written down and you don't have to orally defend it under uncertainty, then it's just a. It's a poor proxy for actual knowledge.

And he wasn't necessarily wrong.

No, I think we're right back there.

Yes.

Thousands of years later, we're wrestling with.

The same question, but there's an interesting nuance here that I think sometimes gets lost.I don't disagree with anything you just said, Bill, but I think AI starts to edge into knowledge production and putting knowledge production at everyone's fingertips, which can be very useful, but it's also very dangerous because there's also got to be knowledge verification somewhere along the line. And I think that's what we're seeing.

That's exactly right.

And so it's just more, you know, we can look to the past, but I think it's fundamentally different. So I don't know. I'm encouraged to hear what we're doing. I know institutionally, I'm doing a workshop later this morning for the university. Why?I have a shirt with a collar on it, which, you know.

That's good. You look good tonight, Craig.

That's great. That's great.So the committees that you're putting together, I know our Office for Faculty Excellence is trying to build some community around all of this. Rob, you've been doing a lot at Washington State University, so it's encouraging, but I think it's time to do more.And then I'll get off of that soapbox because I had another question. And, Rob, I want to get your take on this as well.So Louisiana Tech is not alone in trying to figure out how to help faculty and staff not just cope with AI, but learn how to leverage it in ways that improve their lives. So any listeners that are researchers, you know what a pain it is to fill out the IRB forms.That's the Institutional Review Board, sometimes called the Human Subjects Committee. AI is awesome at that. So, I mean, literally cuts, you know, it's not cutting a day of labor, but it's cutting an hour of labor out of my life.And getting an hour back is a big deal to me.And so what do you see as the main barriers in getting higher ed professionals the knowledge and the capabilities they need to learn how to use AI in ways that improve their working lives. And I want both of you to weigh in on that.

I could, I could try first. Right.So this goes to, I think a broader point, and not just for faculty, for anybody using it, is that AI can produce all of this information and knowledge and fill in forms and all that kind of stuff. What it lacks is context. Right. You need to earn the right to automate. So you just said AI could be dangerous and it can.I like to think of AI as an ever sharper scalpel. You get a sharper and sharper and sharper and better tool. But if you don't know how to use a scalpel, things get real messy real fast.I just don't think people know this. But like any other tool, you're going to need to spend some time working with it so you know how to leverage it properly.It could create a presentation for you. It might be 80% wrong, but it'll create the presentation.So you need enough domain knowledge to be able to use the tool appropriately and know where your domain knowledge is stronger or weaker because it's going to change how you use the tool.

I'll add to that.I think part of that struggle with using the tool and giving yourself the space to use this, and I see this in faculty, is that they feel like they still have to do all the things that they've always done. And oh yeah, I need to take some extra time and add into my life the time to play.And some people embrace that and love that, but other people are like, I feel so overwhelmed by all the other things. To have that space becomes an overload that is just difficult to overcome.So it's finding ways to give people space to play and experiment in ways that will make them more productive. I also think university bureaucracy gets in the way.There's a lot of tools that I think can be useful, but I know at our institution we have one that is FERPA approved.And so if you're dealing with your classroom related stuff, if it's plugging into things legally, you have to be really careful of what you're doing and how you're doing it. And heaven forbid if you break the law.So universities are really good at saying no to creative experimentation, finding ways to get around that, to make things available for people. That's what I spent a lot of time doing, is how can we get more available, at least approved.So that if Copilot is not the solution, that Claude is a safe place to experiment or whatnot.

Well, and so I think this is going to evolve over time. But you're absolutely right. FERPA exists for a reason.The university structure says no for a reason, because it has to go through the legal everything for some of these things to be used in the most effective manner.But as soon as you say, well, you're going to, not to get too far in the weeds, but you're going to take student data from Blackboard or Canvas, you're going to de identify it, run answers through a local model, then re identify it and upload the scores, faculty's heads will explode.So when what needs to happen over the next few years is kind of some plug and play solution where it's like, okay, here's how you go about doing this. Here's the tool set that you run it through and look in five minutes you have the results that are FERPA compliant.And we're a ways away from that, I think in higher education, but that's where we're going to have to go. Same thing with medical data.Other realms where you have sensitive data that are potentially and probably being exposed to the ability for someone else to sift it or find it.

Well, and I think part of that struggle too is having the expertise to help people to be able to do that. Right now, if Louisiana Tech's anything like we are here, if people say, hey, Craig, how do I do this?And at some level Craig's job becomes tech support technician around how to use AI, which at some level is great and good, but at what point do we have people whose jobs it is to help, you know, know how to do these sorts of things and train people up that way and, and hopefully your task force gets to that space where it creates more and more flow down of expertise to where there's lots of people in the hallways that know how to do these things.

The good news I think is that on the other side of that, however we get to the other side of that, for those of us who have had the opportunity or have had to figure out how to do these things in a compliant and effective manner, once you get to the other side of that, it really does free up a lot of creative things that only we could do. Silly example, I am going to the Graduate Business Council roundtable meetings in New York and I am leading a session on these kinds of things.And they said, well, here you need to use our PowerPoint templates. I said, great, here's what I'm covering. Here are the notes, AI, here's the template we need to use. Let's do a first draft.And what would have taken me probably days in terms of back and forth and iteration with last generation's tools, I guess was done in a couple hours to a place where I was satisfied with the output. And it's very much an iterative back and forth process, but again, freed up days to do things like this.

I want to talk more about this idea of assessment of learning that you mentioned and have a message for administrators.So I think right now administrators need to focus on what's preventing people from using AI tools in a way that's ethical and effective and in ways that free them up for frankly, more important work. There are lazy people out there that if you give them an hour of time back, they're going to take a nap, which I will do on occasion.But I think a lot of the colleagues I've had, if you remove an hour of trivial work, they're going to spend that hour on something that's more fulfilling and probably more rewarding for the institution. Yep.But I don't think there's been enough work on understanding exactly what's preventing people who want to do that, use AI in those ways from using it, whether it's tool access, knowledge. Rob, you mentioned space for experimentation and this is going to vary by institution.That's why I was so glad to see this task force and this survey go out, because I think that's what administrators need to be doing. So of course I don't have to actually do any of that because I'm not an administrator anymore. I don't know what you all think about it though.

Yeah. I'll share a conversation we had just last Friday. We were talking about oral exams and what would it take to give oral exams.And there's a few people who are experimenting with that and right now giving oral exams. One person did metrics.I think it's a finance professor had metrics that said giving oral exams was about 4 to 6x the amount of time he spent in grading, delivering and doing those sorts of things.

Things.

And so the scalability became a huge challenge.And as we were talking about that, the, the idea came up of faculty felt very important that they were the ones who were assessing and giving the grades, especially for these high stakes exams.And so I see that as people are excited about this idea of oral exams, but the scalability of faculty still wanting to be engaged with that is something that AI, I'M sure can help with. But there's also the hesitancy to say I'm going to devalue my job as an instructor by not personally assessing each student's score.

You know, we've been turning grading over to teaching assistants for decades. And so I think it's not a binary thing, and it's not even binary within an assignment.As you were talking, I thought, I don't want to call it trivial, but it would not be a difficult technical challenge to have an AI tool that would look at those oral presentations for their content or those oral exams and go through and at least do first pass grading on it. I've done this in the past and do it when I teach undergrads. I've got a TA that looks at the Excel projects, and probably 2/3 of them or more.It's pretty obvious what's right and what's wrong. And so they grade those and then they pass and give me a report, and then they pass off the other third to me.So now instead of grading 100, I've got to grade five of them roughly. So I think we need to move away from binary thinking.And I know that's not what you were implying, Rob, but I wanted to make sure we got that point out. All right.I wanted to go back to assessment because, Bill, I know you've been doing a lot of work and a lot of thinking around AI and assessment of learning. And to be clear, we're not, at least I'm not talking about like AACSB's assurance of learning or SAC's assessment.We're talking about how do individual faculty figure out whether or not their students are learning what they're supposed to learn. So what have you been thinking?

Oh, dear. Talk about time to play with AI and think through these things.Took a step back some time ago, and whether it's chegg or the Internet or now AI, whatever, we've lost this arms race. Okay, so we've, we've known for some time that attendance and test taking.At one point, maybe 30 years ago, those were, those were halfway decent proxies for actual knowledge acquisition. But AI has just made this very, very acute. It's just an important part of this. But that, that's almost the end of this whole thing. To me.Assessment should be very fair, guided by faculty and scalable, and those things are really hard to put together all at once. But I am running a pilot this fall and I'll describe the thing and you guys poke holes in it or ask questions as necessary.But instead of introducing the same material over and over and over again every term, same slides, same bad jokes, same faith remarks, same bullet points.I've actually taken the time to move a lot of that foundational material online in a way that whether it's an in person or an online class where those things, those vocabulary lessons, essentially the basics are put online in an accessible manner where students can watch and re watch and all this kind of stuff.What happens in the classroom then is under the assumption, and it's actually, so far it's been really encouraging that the students are watching those videos because when we get into class, this is where the faculty really can have fun teaching again. It turns into coaching, group projects, discussion under uncertainty.Whatever happens in the classroom is this social cohesion, group experience that really gets students into whatever it is we're talking about. Now, the assessment piece. We've all done oral exams, either taking or receiving, right? And they could be brutal.But the reason I think we're going back to these oral exams is to be able to prompt a question under uncertainty and have the student defend a judgment or explain an outcome and demonstrate some knowledge. Well, how do we do that at scale? I mean, MIT's report was talking about oral exams.And broadly in higher education, we've been returning to this idea. That's great. How do you do that with 200 students in the large classroom? How do you do that with online instruction?So I've spent a good amount of time building a system that essentially allows students to use AI to develop their stuff, but that opens up a system by which they upload their work. Then they are asked under uncertainty to explain this, that or the other thing.Then it goes through a pipeline that I've developed where the oral responses are de identified and go through models that then flag those that need follow up but are graded using tools. And the outcomes are then re aggregated and result in grades.

So I want to. That all sounds really good. Bring up a point that just occurred to me around oral exams. What are we really assessing?So we could be assessing the domain knowledge. That's certainly what we want to assess. But we're also assessing the ability to communicate verbally.We're assessing responses to stress, we're assessing a host of other things. And if that's intentional, it's fine.But I think just throwing out oral exams as a correct response to AI ignores an entire layer of uncertainty and of potential noise that we need to consider. I don't have any answers for that right now because it literally just as you were talking.It's like, wait a second, I know people that are really smart but may not express themselves well or they may have some sort of anxiety that gets in the way. And so I think we really need to be careful about that.

Agreed.So I want to be clear that the oral exam is one part of the overall assessment, the way it would be with any type of project or in a group meeting in the workplace or whatever it is. So it's overall, it allows us to inject this little bit of uncertainty. And you're right, some students are going to really struggle with that.I would like to think those are the minority. And communication skills and comfort with their own material can be taught well.

And to be fair, we've had other kinds of noise in the way we've been doing assessment in the past. So it's not like this is. We're just relocating it a little bit.And I could envision where there's some sort of a second level of assessment where I know I've been involved in this with doctoral students where they just bomb their oral exam and you know, they know the material, they're just nervous. We did away with oral exams when I was at St. Louis University because I thought they were just pointless.You know, they didn't really tell us anything. And I know there are different views on that.But what we would do is somebody would sit down with that student one on one in their office and just have a chat with them about the material. And every time it was clear that the student knew the material. And so what were we doing there?So I think maybe you have, okay, whether it's an AI kicking it out for further review, we'll do the booth review of this one. Some sort of a secondary thing might be a way around that. And Bill, you said hopefully it's a minority and it probably is.We're talking 10% or 15% or something like that. So maybe that's a solution.

Well, and I think iteration is also going to be a part of this. For example, with this pilot, I'm not just blasting on the screen and saying, ha ha, gotcha.There's a series of exercises where they go in and work with the plumbing and do the assessments. Very, very low stake stuff where they know.Okay, in step two, after you click submit, this thing is going to pop up and it's going to ask you some of those questions you could prepare for. You're going to be asked one of these things.But there is always going to be one question that is going to let me Think about that and I'm going to respond to it. The first one is very, very important things like is a hot dog a sandwich? Would you prefer the scenic route or the direct route on a trip?The responses to these have been hilarious so far. My favorite on the trip one was it depends on the company and whether.

Or not there are hot dogs on both routes.

That's right.

I know I was playing devil's advocate there a little bit, but I really like this idea, especially since it could scale and we should want our students to stretch and do things that are hard for them as long as we're fair about what we're assessing.

I'll also put out there. I used to have students do presentations.So this is a different kind of presentation, obviously, but students would present their slides and make their case at the front of the room. And by the 10th one, first of all, everyone knew every question that was going to be asked. Right.So there was an advantage to the people going later on. And frankly, I'm tired. I want to be sure that I'm fair in how I'm grading these and not also defaulting to. Yeah, you all did great.You all get a minuses or whatever.

Bill, I wanted to follow up on the piece of it of what you're doing in the classroom because to me that almost sounds like your classroom has become a lab where the theory, the things that they've learned outside of class are put to fire, put to the test where they can have a safe place perhaps to struggle. That friction of what do I do with what I've learned?What does your assessment look like inside of that classroom environment where they're doing that does. Is that baked into to this experiment that you're doing?

Yeah. So the students don't know this, but they are. Well, they've agreed that we're going to do things later.

They will once I listen to this episodes.

Well, yeah, let me make sure I answer your question, Robin. If I don't kick back. But I've created a company that is meant to deliver this type of experience.The pilot is to make sure the plumbing works and to get a pulse on the students reactions to a different way of going about an educational experience. This is for a graduate finance class. So this is not Econ 201 with 200 students.This is a mixed bag of in person and online students who at least purportedly since this is graduate education, have had some experiences and a lot of them are around the globe and have families and jobs and all this kind of stuff. Again, I want to make clear that this is not an 18 year old experience. Okay.On the other hand, the responses, we're in week three, I guess, of a ten week program here at Tech. The response has been really encouraging. They like the idea that I'm not doing tests.This is not a read the textbook cram for your exam, multiple choice with high detectors and wave your ID around and all that kind of stuff. We're not doing any of those things anymore because frankly, they're not going to do that in the workplace.You're going to be asked to, to be capable to do a thing. You're going to do the thing and someone at some point is going to assess whether you knew what you were doing.And that's what I'm trying to replicate in an educational environment. So yes, they will get grades, but there are no tests. They're doing three assessments where they build a thing with real coaching in the classroom.They get their own version of the thing, whether it's capital budgeting or it's hedging with futures or whatever the actual assessment is. But we set the expectation. They know the material is online.The in class coaching sessions have been very positive and they're frankly kind of jazzed. So I'm hoping this translates. Well, outside of my little lab, students.

Want to do something.

Yeah.

Unless you're an extraordinarily talented lecturer, they would rather do stuff than listen to us.

Right. At the same time that they've spent time and money to have access to us, who are purportedly the keepers of the knowledge.

That's right.

So they want to be coached by those.

And what I meant is they want to do something and then you evaluate it, you coach them, you do those sorts of things. So I wasn't implying that they didn't want anything to do with us, although that may be true for some of us.But when they're active, I mean, our whole book, the book that Rob and I and our colleague Franz Belanger have, is all about actually being active in the classroom.

Craig, what I like about this analogy is the word coaching. And I've been thinking a lot about this. One of the things I do on the side is I referee soccer matches.And so in sport, that soccer match is really the example of what all of these kids have been doing at practice, which is duplicating some of those same skills. But then they get put into an environment where they can see how well those skills play off.And you have the coach on the sideline who's encouraging the kids say that was exactly the right idea. If you do it just a little bit this way, you're gonna see better results.And so it is really learning from some of these other spaces that help us to what are those drills? What are those things that we can be doing?And then where is that opportunity to actually compete, if you will, or to demonstrate that the drills have actually resulted in better execution?

I'm so glad you mentioned the refereeing, because I've been wondering why, as I'm talking, you're holding up a yellow card. I didn't quite understand that. All right, well, I want to kind of move our conversation to focus more on students in particular.And, Bill, this is something that Rob and I are just curious about as much as anything else. So what are you telling your students, especially the undergraduates, about how AI is going to change the world that they work in and live in?

I don't know that I need to tell them anything. They already know it.

Do they really?

Well, they think they know it.Okay, we talked about this years ago when big data was the thing, big data was the verbiage, and I forget who, but someone in a presentation said, big data is kind of like teenage sex. Everyone's talking about it, everyone's claiming they're doing it. No one knows what they're doing, so they're all claiming.

And nobody knows exactly what it is.

Right. So I think students have been using it longer and more than we have across the board.So they're experimenting, and you can get yourself in trouble if you don't use it correctly. There's a good amount of pessimism amongst undergraduates that I don't know what my job is going to be.Five years ago, the hottest job you can get was daatalysis. Well, that's gone. You don't need to learn coding anymore because the new coding language is English with an AI tool.So what I'm spending time doing, or at least finding myself spending time doing, is working through AI projects with them.Not because we're going to use AI to use AI, but what I'm trying to do is show them cool stuff that we could do now that we couldn't do a few years ago to present something they were wrestling with or do research for. This, that, or the other thing. I'll give you a stupid example. It is again the beginning of week three in the student managed investment fund.Every fall, I'm turning a bunch of early 20somethings who have learned about finance in a classroom to actually be fiduciary, responsible for millions of Dollars ready, go. They don't know. They've never traded anything outside of their Robinhood accounts. They certainly didn't know what a fiduciary was.They don't know what's in the portfolio or what real diversification means. What does being responsible really mean? And what they're figuring out is that, oh, it's not about this or that asset or stock or bond. It's why do.Why would we. That's like the end result, right?So one of the things we did in the first couple days is we just went to the custodian, we downloaded all of the positions and said, let's together figure out how to use AI to slice and dice and visualize and figure out what it is that you are now responsible for. What does a global asset allocation portfolio mean?And in short order, what would have taken us weeks with coding and webs and all this kind of stuff.We now have a dashboard on a website that shows every position, every Sharpe ratio, correlation matrices, performance gain, risk allocation, the whole thing. And we could hover over things and drill down.So like any good data visualization, it allows us to see things literally and figuratively, that would not have been available just on a spreadsheet or a CSV file and to ask questions and ask better questions.

And that's the big point, is the AI gives you a better starting point rather than, this is the answer. It's like, okay, this is a better vehicle for getting at the answer. That's awesome. That's awesome.Well, let's shift gears and talk at least for a couple of minutes about the MIT report. And we won't really have time to get into a lot of this.I came across this in a New York Times article that made the headline about cognitive surrender. And Rob subsequently and independently sent me a message about this report.And so I eagerly look at the report to find out what they're talking about with cognitive surrender. And they actually weren't. It had one mention, not anything that we didn't already know. Here's the direct quote.Getting the right answer from a chatbot can create the illusion of learning, but it can also trigger cognitive surrender, where students fall back on AI at the first hint of struggle. We already knew that it's an important point.But the report is not about cognitive struggle, by the way, I'm pretty sure somebody at the New York Times used AI there, because right in the middle of all of that is a big EM dash where it doesn't really need to be.

I like my EM dashes. You leave Those alone.

I do too. But what was really more interesting about the the report is the effect that AI is having on the social fabric at mit.

Yeah, right.

That was like. That should have been the headline because we have not been talking about that enough.I know Rob and Tom Stafford and I talked about that a little bit on the last episode.And Rob, you in particular have talked about this multiple times, kind of at the individual level, but this is at the campus culture level, which I've not seen a lot about.

So someone needs to write that and send it to the Wall Street Journal of the Times or someone say, this is to me a huge problem, but it's also an opportunity for us to really help our students along. Right.It makes perfect sense, especially when you think about a lot of these kids were on their own more than other kids, perhaps because of COVID and these types of things where it's very isolating. If you've used. Used AI or think of social media the same way, it could be very siloing and echo chambery.And I think if we're just reiterating information at the front of the room and then telling students to go and take a test and use AI tools or not, we do lose that social cohesion and we lose what makes us human and communities of curiosity.

Yeah, I think this is going to be where things get really interesting in this next year or two.And I'm going to share anecdotally, one of the things we've been talking about in the hallway is students are much more likely to want to pick up a phone and make a phone call now than they were a year ago. And it's this old technology becoming new again at some level.And it's almost, as I think about it, we're creating a world where people want to be able to communicate in the way they want to communicate when they want to communicate that way. So I don't think text messaging, social media goes away as part of that.But that human connectedness, what AI has done is it questions electronic communications in a way of am I really talking to that person? And so I think people want that human connectedness.And so I think there's going to be some really cool creative ways that we're going to be able to lean back into that to meet the market demand of what students are expecting. Maybe they'll want to show up for office hours a little more often because they know they're interacting with us.

I was just going to say that. And I'll tell you anecdotally of Course that, that's already happening. I teach an online course on coding.Even though I said no one needed a code anymore, you still need the domain knowledge to understand what you're doing. So everything is online and AI could do all the coding for you.And the number of students who've just walked in my office and plumped down their laptop and said, could you see what I'm not seeing has been. Well, on the one hand, it's been a time command that I didn't budget for this fall, but it's been really, really interesting.And you're right, if everyone's AI agents are talking to anybody else's AI agents, the human connection is at the very least confirming and welcome. Right. I do think we're going to see a return to good old fashioned handshakes and real office hours.

Is that really a thing with them doing phone calls? I have not experienced that yet. Huh.

So I had a faculty member called out because they said, yeah, you guys don't pick up the phone and call each other anymore. And had like five students immediately raise your hand and say, you don't know what you're talking about.

So I guess I need to check my voicemail more often than I'm not.

Sure if they're calling us, but they're calling each other. Even if that's what the world is, that's great.I actually think on the business side of the house, this is where we're going to see differentiation is I've heard a lot of frustration of people who call or who start texting or doing something and they know it's a bot interacting with them and they just get fed up and want to talk somebody.

Yeah, but it's not like that wasn't going on before. It's just better now in a lot of cases.

Well, maybe that's where it's a problem is it's so good that you don't know, but you are skeptical of the who or what am I really talking to? To. And it's no, that could be.

And it's that uncanny valley where it's, it's almost perfect, but it's just not perfect enough to where something feels off but you don't know exactly what it is. Sorry, Bill, I interrupted you.

No, but this is something that again, it's just more acute now. How many times do you call a helpline and just start yelling, agent, Agent. I don't Want to press 3. Give me an agent.I need to talk to a human about this thing. And I think it's just Going to be more than that. No one uses the phone for the phone anymore. I don't even know why we call it a phone.Any number that is not already in my contacts automatically gets silenced and sent to ether hell. Right. But I think there's enormous value or there's going to be enormous value in being in the contact list.And in order to be in the contact list, you and I need to have had a conversation and an understanding that now you're in the club and we can have human conversations.

So you're disrespecting Heather from the tax resolution department?

No, no diss on Heather. I'm sure her job is not what we would want it to be, but.

So. All right, so to move off of this and kind of wrap, I have a quick question for you. Are you all using the latest models like Fable or Astra? Yeah.

Yes. I'm looking at my little bot right now and it says Fable 5 High is whatever I used last.So I'm tending to default to the highest models unless I'm doing something that I know specifically. And you probably are way ahead of me on this, Craig, but I have a task that I could send to a lesser model or a more focused model.I will do that, but I'm kind of letting the tools figure out which model gets what task.

Rob, how about Copilot?

I use Copilot. We've got the premium, and I primarily use Copilot for work purposes. A, for FERPA reasons.But B, I'm also the person a lot of people ask, how do I get it to do this thing? How do I do this thing? And it is the sole approved technology at WSU right now for AI.So I'm trying to be careful not to say, oh, you can do all these things, but only if you go outside of the guardrails in place.

That would be bad. Yeah.

So I try to figure out how best to push the envelope with what I have access to in our sandbox.

So my reason for asking is I got an email from one of my podcasting friends that said she had run out of ChatGPT credits very quickly, and I suspected I knew what was going on. And when I met with her, I was right. She was using Astray, which is their newest model.But I'd want to caution people against that for a couple of reasons. The main one is you can blow through your usage quota really quickly and unnecessarily.Remember, yesterday's Frontier model is today's second tier model, so you don't always need the absolute best model.And so I would really encourage people to test out the capabilities of the models and maybe default to one that's one step below whatever the best is because the numbers aren't this precise, but it's like a 5x or a 10x capacity use. In other words, you're going to blow through your quota many times faster using the cutting edge model and it's often not necessary.So anyway, just a little pro tip. Don't just default to the best model. It's kind of like using your most expensive employee for every task. You don't want to do that.All right, we are just over an hour with the raw audio. So any last thoughts before we wrap?

This has been great. Craig, thank you.

Thanks for coming on. We'll have you back on. Maybe we'll talk about that MIT report in a little bit more detail and you can give us some updates on your project.All right, for all things AI goes to college, go to aigostocollege.com you can email us. Rob is at ro.

Rob Crossler.

I goes to college.com and I'm craig@aigostocollege.com we'd love to hear from you. All right, that's it for this time. Talk to you next time. Bye.