When You Bring AI to the Party Matters More Than Whether You Bring It

Welcome to another episode of AI Goes to College, the podcast that helps higher ed professionals figure out just what in the world is going on with generative AI.

I am joined once again by my friend, colleague and co host, Dr. Robert E. Crossler of Washington State University.

Rob, how are things out west?

Craig, the semester is wrapping up and it will be a beautiful summer before we know it.

I am heading to Flagstaff in the morning to give a keynote at the Northern Arizona University Nonprofit Leadership Conference, which will be fun.

All right, well, let's get to it.

I want to start out by talking about a number of updates with AI tools, at least updates in how I've been using them.

And so I'm going to go through a lot.

Love to hear your reactions.

The first one was Microsoft Office Agent, which as nearly as we can tell is doing something with Claude, maybe Cowork on the back end.

It's a little mysterious at the moment.

Yeah.

What I like about that, Craig, is as Microsoft starts bringing some of these other tools in house, right when they started with Copilot, it was very much a wrapper that you got access to ChatGPT and their different models.

And now whether it's through Researcher where you can get to Claude's Deep Reasoning or this frontier model that gives you access to Cowork, it is the problem of governance and the perspective of which tool does an organization or a university authorize people to use.

Microsoft is trying to position themselves so these things become useful and not a series of one off arrangements with a whole bunch of different vendors.

So it hasn't been perfect.

I hear there's some friction that people aren't loving, but it's moving in a good direction as we rapidly see these AI tools change and become available to do different things.

Yeah, and there's a lot of benefit to having that kind of aggregator wrapper, whatever you want to call it, especially when it comes to things like FERPA and HIPAA, the EU's various regulations.

But what agent is is it's a sort of autonomous agent that will just go out and do stuff for you based on a prompt.

And it, it's limited.

Right now I'm using it on my personal Office 65 subscription.

It's listed as a frontier tool and right now I think I get 15 tasks a month and 50 conversations.

So I'm not even sure what the difference is there maybe in fine tuning the output.

But what I did is I wanted it to create slides our textbook Information Systems for Business and Experiential Approach Prospect Press.

It's coming out with Edition 5.1, which is basically adding a chapter on AI and we need to do slides.

So I thought, well, how are these tools going to do with the slides?

And it was a good opportunity to try Agent.

It went poorly to, okay, so the first problem, it created nice slides.

Then it went outside of our chapter.

I gave it our chapter and gave it our chapter's artwork, screenshots, that kind of thing, and then gave it just some instructions and it went outside of the material I gave it, which sometimes is good.

In this case, it was bad.

I intentionally tried a really simple prompt.

So I didn't tell it, only use the chapter.

I wanted to see what it would do and that was okay.

So I went in and I said, now just use the chapter that I gave you.

And this is where it got kind of funny.

I don't know whether Agent got salty on me because I pushed back, but it refused to create slides.

Right at the end.

It would go through the whole thing, get ready to produce the slides.

I could see it going through and producing individual slides.

And then it would say, I can't do this because I can't create harmful material.

And there is nothing, there is zero, that could be seen as harmful in that chapter.

Unless it's a meta response where AI is saying AI itself is harmful.

Yeah.

Maybe it's thought it was doing something it shouldn't do because we had this proprietary information.

I don't know what was going on, but I pushed back a couple of times and finally it created the slides.

And you saw them.

I thought they were fine.

Nothing spectacular about them, but more than serviceable.

Yeah.

And I think, Craig, that's an important thing that I'm seeing with a lot of AI tools that I use, at least in the work that I do, creating documents, creating slides and whatnot is I have yet to find where I get perfect output, where I just say, boom, go.

Even if it's writing them an email or simple tasks of making paragraphs sound better.

I always have to review, I always have to update, but I find it increases the speed to completion.

It's not a replacement for my ability to engage, but it can make me way more productive and get more done in a shorter amount of time.

Yeah.

It's what we've been saying all along, that a lot of times with AI, 50% is better than 100% because you can get the 50% really quickly and it can take way more time to get to 100% than it would have to.

Just do it by yourself.

So one more interesting thing about the Microsoft Office Agent is even after the explicit instructions to only draw from the chapter, it completely made up a quote that wasn't in the chapter and I don't think it existed.

So it's not bad.

It was not as good as coworkers Claude Cowork, but it's not bad.

So I would keep an eye on it.

If you have a personal subscription to Office 365, go in, activate the Frontier Experimental or Labs or whatever they call it in the Microsoft world.

Try it out on something that's low risk, but I'd give it a try.

So speaking of slides, we ran kind of a big test.

As long as we had to do these textbook slides.

I'm trying to remember.

I ran it in Microsoft Office Agent Claude Cowork and I feel like I'm leaving one.

Oh, chatgpt.

And they were very different.

The one that was really good, I thought was the new Claude design that put out the best looking slides.

Claude went nuts, created 66 slides.

I told it specifically, if you're not sure if you should include a slide included, I'd rather have more slides than I need and cut back than the other direction.

The others were kind of.

I liked the content of Claude the best, but they were all kind of okay.

They all would have saved a bunch of time.

The big thing is, Rob, back to your point, none of these are perfect, but all I did was put the prompt in and go do other things and come back and I had a slide deck that was more than serviceable.

Yeah.

And Craig, I think this is a great point to talk to any students maybe that are listening to this podcast because I've been seeing this a lot with student slide decks that they create really nice looking slide decks.

You can tell that they're better than they were pre AI in how they're being designed.

But there are oftentimes things about the slides that if they would have gone in and took a critical eye to them, that would have made them better.

So just because something's created that looks really nice, do take the time to say, do I really need that many words on that slide or is this font the right size?

Is the audience going to be able to read what's on there?

Because I even saw that in some of the slides we created that there was just some stuff that the slide was basically there, but it needed some of that human touch to make it a better communication tool to do what we needed to do.

Yeah, it's bad at making the text too Small and at trying to jam too much text on a slide.

I mean, you can control a little bit of that with prompting.

But even with the Claude design, which I actually used for my keynote that I'm giving on Thursday, there were some things that just didn't make any sense.

I've got one slide that's about the slow drip of messaging.

You know that it builds up over time.

You've got to give it time.

You can all understand the metaphor.

And it gave me some weird bar chart.

And it's okay.

I kind of get what you're going with here, but it really makes no sense.

And if your visualization makes people think too hard in ways you don't want them to think, then it's a bad visualization.

So I had to replace that, which was no big deal.

But what would have taken me probably a couple of hours took me maybe 30, 45 minutes and could have been done more quickly than that.

I really like to fine tune slides, but they would have been okay in probably half an hour's worth of work at the most.

So if you're not using AI for that kind of thing, start using AI for that kind of thing.

We're going to be going into summer prep for fall mode before too long.

And things like particularly Cowork.

I'm a huge fan of Cowork can be a big help with this.

In fact, we're working on the instructor's manual for this new chapter.

And that involves giving the answers to their questions at the end of the chapter and that sort of thing.

So I thought, well, I'm going to try Cowork on this.

And just a quick glance.

I think it's going to take hours of work and cut it down to minutes of work because all we have to do is scan to make sure everything's right.

And since we know the material so well, that's not a big ask.

So AI did the first 50%, 75%, whatever number, and saved a huge amount of time.

Yeah, I do that for a lot of things.

Right.

Creating long, monotonous documents that just are well defined.

AI is a great tool when you've got pretty absolute guidance on where you're going with it.

I'm about ready to give a speech on Thursday.

Five minute little intro to an event we're doing in our college.

And I've been using Copilot to help me craft it and to get it into my own words.

And it saves so much time.

I don't know why it takes so long to write a five minute speech.

It gets stuck in my head of do I want to go this direction or that direction?

And it's just really helpful in figuring those things out.

So use the tools.

I think they're good for any of these sorts of things.

Yep.

Could be harder to do a five minute than a 30 minute talk.

So two more things on Claude.

And I do not get paid by anthropic.

I don't get a discount on anything.

But I'm spending most of my time in Claude now.

So I mentioned the design.

There's also something that's been around for a while, but I don't think many people are using called skills.

So I've created a number of skills.

Like I've got one for reviewing journal articles, not for doing reviews of other papers, but doing pre reviews of my own work.

I want to be clear about that.

I've got several that mimic my writing style.

I have at least three different writing styles depending upon what I'm working on, but doesn't get it all the way there.

But there aren't EM dashes, there aren't those triplets, There aren't.

I'll be honest here because I've been lying up to now.

All that stuff is out and it really.

Unless somebody really, really knew my writing style, it would pass muster.

Nobody would think a thing about it.

Look, get a Claude subscription, the 20, 25 bucks a month, whatever it is, and start playing around with the skills.

Claude will build the skills for you.

It literally has a skill builder skill which is meta and I love that.

But there are all kinds of uses for these things.

I built a copy editor skill that does a really good job of going through and doing a copy edit pass.

Well, the thing about Claude 2 Craig and I think the marketplace is telling us this is it is becoming the number one choice of people paying for subscriptions and using generative AI tools.

So they are doing something right right now.

Is that going to be the forever solution?

I don't know.

It seems like every three to six months we're talking about somebody else who's taking the lead of doing something.

But right now it feels like Claude is leading the way.

Yeah, I don't know if you had a chance to look at the little chart I put into our notes.

It comes from the Neuron, which is one of the largest AI newsletters out there.

It's really good.

So I read it every day.

They did a poll of their users.

Now they've got tons of users and they had about 2,300 people that responded to this survey Claude had twice the use of ChatGPT, so Claude was just shy of 1500.

ChatGPT looks like about 7.

Oh, they 790.

Gemini is a little below 500.

And then everything else is small.

But that's pretty surprising to me.

If you're not using Claude, you need to check it out, pay the money.

Look, we're privileged, we have decent incomes.

But 20 bucks, it's not that much money if it can save you a ton of time.

One more Claude shill moment and then we'll move on.

So Rob, I've been sending you my daily AI briefings that were created by Claude routines which they just released.

They're pretty good.

They're very good.

Yeah.

I mean it's basically I've got this task, it goes out at 9 o' clock every morning if my computer's awake and does a scan of the Internet and writes kind of a paragraph about things that we ought to be taking a look at.

And it's impressive, it's really impressive for something that's largely hands off.

And what I love about at least we created, and I think this is where real opportunity is, is in the past we relied on some other aggregator to say these are the five articles that you should read this week.

And it was based on somebody else that maybe you're aligned with in your thinking to put those things in place, but with the ability to do it yourself, you can customize basically what's important to your workflows and to your thinking and what you're dealing with in your day to day based on just very unique things about you.

And it does it automatically and puts those right in your inbox.

Let's move on entry level jobs.

Rob, there was a pretty interesting article out of IEEE Spectrum about entry level jobs.

And this is one of those big topics that a lot of people wrestle with right now is what does entry level work look like in the world of AI.

We've seen a lot of decreases in hiring of students coming straight out of very technical programs over the last few years.

And so a lot of angst about what degrees people should be pursuing and what those degree programs look like.

And so IEEE put out a paper where they're seeing that they expect these entry level jobs to increase, but not exactly the same students with the same skill sets that they were hiring in 2023.

So the roles demand higher order thinking where there is more of that human nature, that human critical thinking components baked into what they're doing in using those technical skills.

So it's Pushing the boundary of what's possible or expected from students beyond just the ability to code and to do what AI is beginning to do a really good job of.

Yeah, exactly.

And for those of you who don't know what IEEE is, it's the world's largest technical professional organization.

It's the Institute of Electrical and Electronics Engineers.

So it's engineering and Computer Science.

This is a legit organization.

When I read through this, we were trying to make sense out of all of this.

A couple of interesting things here.

One is that employment for programmers fell a lot.

Software developers, which has more of a design piece to it than just writing the code basically was flat.

And then information security analysts, AI engineers, they have double digit growth in entry level hiring.

So if we look at all of that, one of the things it says to me is that in kind of a weird, not paradox, but something like a paradox is the solution to helping our students get jobs is to show them how to lean into their humanity.

And so it's not that AI tools can't critically think, it's that they critically think in a certain way.

And we'll see that here in a few minutes.

I think it doesn't have knowledge of all of the context, no matter how big the context window that humans have.

And so I think we need to help our students, particularly those who are in more technical fields, learn how to leverage being human in the best way of being human, not the irritating way.

Yeah.

And I think one of the things that AI does very well is it comes up with answers and answers that generally work and that it's confident in, and especially for early stage people in learning, it's the good enough solution.

And I think where the human aspect comes in is knowing the problem set well enough and critically evaluating that well enough that you can push that further.

And whether that's through programming skills or continued prompting conversations, it's deciding that you're going to push the envelope in what's being created in ways where in prior times you might work pretty hard to get to that good enough solution.

Now that's not adding much value to the equation.

That's right.

Anybody can get the good enough solution with a moderate amount of skill.

Well, as educators, it used to be we'd be excited when students gave you that good enough solution because it demonstrated a struggle that they had to go through in order to get to that point to cognitively develop those skills and those abilities to do that thing.

And so it really does put the onus on instructors for how do you approach education In a way that success is by getting creative and going beyond what's created in two minutes by a machine.

Yeah.

And if you can't do that, then they don't really need you.

But I think that's going to be a subtle shift in kind of where the floor is in higher ed.

That floor has got to go up.

And so we've got to help move our students up that stack to things that are more human.

Yeah.

More ambiguous.

Yeah.

I had a conversation the other day with the peer who was saying that the floor is going up, but their fear is that our ceiling is coming down because we haven't found those ways to push our students through the ceiling.

And I wrestle with that thought a little bit.

How can we raise everybody by raising that floor but push many beyond where they currently were?

Are we going to find those creative ways of designing class projects and class engagement that really does say, wow, that college degree is moving people way beyond where we ever thought possible.

I get what you're saying, and I do not disagree.

But that's always been what we should be doing, is setting the conditions that allow students.

It's such a trite thing, but to live up to their potential.

It really is.

I think we need to move a little bit away from what activities do we do to help them push through that ceiling to how do we create the conditions that encourage them and allow them to push through that ceiling?

But I don't know what that is at the moment.

But I think that there's a mindset shift that needs to occur where we.

We don't try to fight the battle of being very detailed and particular in how we do some of these things, because as AI gets better and better, what those details need to be is going to change.

And if we start thinking about how to set the conditions, we might be better off.

But again, I don't know what that looks like at this point.

I'm not sure anybody else does either.

Well, it sounds like what we're wrestling with as instructors is a world of ambiguity.

The same way we want our students to struggle through ambiguous problem solving, we're all struggling through that ambiguity and trusting that with the right guardrails, the right direction, that the outcome is going to be positive.

And that's hard sometimes because you don't know the absolute ways that this all should go.

Yeah, it's not going to be easy, but I think there's a lot of potential to really make things better in the long run, because a lot of the criticisms of higher ed and our students and the way we prepare them are not new.

They've just been magnified by AI.

So one more quick thing on this and then we'll move on to the next article, which is very interesting.

The.

The ability to use AI and apply AI to practical problems is now almost table stakes where that's just an expectation.

And I think employers are going to be surprised.

That's not really going to happen for them.

And it's a real big signal for higher ed that we need to come up to speed on helping students be able to meet those table stakes.

And God, that was very ineloquent.

That's what I expect from you, Craig.

That's right.

At least I'm consistent.

So employers are not going to find masses of students graduating that know how to use these tools effectively yet.

We need to start working on that as higher ed.

I think where this gets interesting, though, Craig, is as students are in our classes, as we're trying to build them up, we're starting to see research around how cognition is developed and what impact AI is having on the development of that cognition.

And I think, you know, again, it comes back to the humanity, the humanity, the human, the human nature of students.

Oh, who's in eloquent?

I know I learned from the master.

But is, you know, to lean into that critical thinking and perseverance when AI can make things easy?

Because big problems that you're going to face in organizations aren't going to have some of the slick and easy solutions that our current approaches pre AI, may have had from textbooks and different things like that.

So how do we scaffold, if you will, in such a way to where the students might be interacting with AI and how they're doing things, but at the same time they're developing that cognition, that confidence that they know how to do things even if we took AI away, that their thinking wouldn't be hampered by that, that we weren't all of a sudden hamstringing their ability to perform, which I think is going to require some experimental approaches to how we do things in the classroom.

I don't think there's a one size fits all recipe for how we ensure that that cognition is developed.

We're going to have to be willing to have some failed experiments, without a doubt.

But the article you sent Rob used the analogy of the boiling frog, which I have no idea if it's really true, but that's the one where if you try to throw a frog in boiling water, it'll jump out.

If you start with warm water and crank up the heat, it'll just let itself get cooked.

Which I don't want to perform an experiment that validates that empirically.

But I think there's a real danger there that goes beyond over reliance.

I think it starts to rewire how we think about things.

And that's especially dangerous for the obvious reasons, but not only the obvious reasons.

It's because AI has some really weird ways of thinking.

If I put thinking in quotes.

For example, we had a dog that we thought was dying over the weekend.

And so I started plugging things into Gemini and it turned out the dog was fine.

It was just.

It had some storm medication.

It's getting older and it just hit it a lot harder than it normally did.

And going back and looking at it, Gemini was feeding into what I was saying.

So it wasn't quite confirming it would push back, but it felt like it was picking up on my assumptions about what was going on.

Because this was classic.

This dog is at the end of life.

She's 13.

Suddenly her hind end wasn't working.

You know, she was just laying down that slow breathing.

But it turned out she was just stoned.

What it amounted to.

But Gemini was really feeding into what I was thinking and my anxiety over it.

So this is going to be tough.

But we've been saying this for a while.

It's time to get after it.

We can't put this off much longer.

All right, so let's move on to our third, third topic.

This is an article from Computer Human Interaction Conference, which is a huge association for Computing machinery conference.

I mean, it's huge, huge, huge, huge.

This just came out I think a week ago today about AI and critical thinking.

But this is not just another AI and critical thinking article.

The authors did something really clever here.

It's three authors from University of Chicago and University of Toronto.

So it's no surprise that well written paper.

They did an experiment with almost 400 participants where the participants had a realistic civic decision making scenario where they acted like they were city council members deciding whether or not to accept a proposal on water contamination cleanup.

And so they chose that thinking that almost nobody that's in the experiment would know anything about that world.

And then they ran them through, they gave them some documents that varied in quality and asked them some questions about this proposal.

And what was really interesting is under different time pressure conditions, using AI at different stages in this process led to different outcomes.

So I know, Rob, I sent this to you really late.

I don't know if you had a chance to scan the article, but what did you think?

I thought it was interesting because it isn't the typical AI better than human or is human better than AI Sort of things that you see in the press that's out there, that it put some conditions in place that allowed us to see when is AI helpful and when is it not as helpful.

I really like seeing how they attempted to help set up those boundary conditions for knowing when to use these tools and how they can be the most.

Helpful, not just what tasks to use them for, but when in terms of time during the performance of that task should you use it.

So they gave students either 10 minutes or 30 minutes to come up with their evaluation when there was only 10 minutes.

If they used AI from the start, those groups outperformed the groups that started their AI use later did not have access to.

But when participants had a half an hour, that pattern completely reversed.

If they didn't have access or if they had late access, they wrote stronger reports than those who had the AI early on or throughout the process.

And there's a lot of inference that goes on when you're reading a paper like this.

But one thing I'm thinking is that if you had AI from the start, you got to a good enough solution.

It's Herbert Simon satisficing idea good enough.

It's not literally where you stopped, but it's kind of where your engagement stopped.

I see that a lot, even when I'm using the AI tools.

Right.

You kind of anchor to where this feels like it's the right answer.

And why do I need to put a whole lot more effort into it?

So to see some level of evidence of that in this study, I think confirmed maybe what I hadn't put an exact name on, but I've definitely experienced that myself.

Something to think about.

If you've got a really important task, like I'm working on a paper and it's got an interesting story, but we have to be really careful that we stick to that story.

It's one of these where we have a lot of results that kind of are what we expected.

But there's one thing that kind of came out from a post hoc analysis that it's like, oh, this is the paper.

And I don't want to go back and rewrite the front end because that's not really entirely honest.

So I want to kind of show the story as it unfolded.

But the conclusion to that is going to be critical to the paper.

A lot of times I write a paper, I just want the conclusion.

The reviewers probably aren't Even going to read it a lot of times.

But this one, this is a critical piece of the paper.

So I'm trying to work on it first.

Think about how I want that conclusion to go, what I want it to communicate.

Even if I don't sit down and write it, that goes into my prompt or I write it and then tell Claude, you know, where are the holes in this?

What am I missing?

Am I communicating this story?

So I think that's a huge message.

If something's really important, you've got time, think about it.

At least give it a lot of thought first before you turn to AI.

Yeah.

And I think what you're suggesting is that late access to the AI is a huge approach, actually, especially with something that's important and critical.

And when I translate that sort of thinking into what's going on in the classroom, and I've seen some faculty in my hallways that are doing this, is they'll have students attack a problem and very purposely say, for right now, we are not going to use AI at all.

And then they bring AI in later into the game and are able to demonstrate through some activities the role that AI can play after you spent that time wrestling with a problem set or a solution.

So you're actually bringing in that unique human perspective to solving that problem and not just saying the machine's going to have it.

Because we may actually pick up on something that is important that might be ignored, but that might be where the competitive advantage is or otherwise in solving that problem.

Right, Right.

And you don't get that lock in if you see a solution, whether it comes from AI or somebody else, we might tend to lock in on that, which would be a really interesting study, is to replicate this, but instead of just AI access, you give them access to another human.

Because I think you would see some of the same patterns.

But I don't know that, of course.

It's just what I'm thinking.

There was another interesting result buried.

They had 30 minutes sufficient time.

Participants with late AI access produced the same number of arguments as those with no AI access at all.

But they gave a more balanced view of the pro and con reasoning.

So they didn't lock into their own view.

So AI kind of provided counterbalance or another perspective, which is a huge benefit.

So this is absolutely not a don't use AI article.

It's use it at the right point in time.

It's a really good article.

I haven't said it yet, but there'll be links in the show notes to all of this stuff.

So Anything else on that article, I.

Would encourage people to read these things and start putting the filter on that says what does this mean to me?

And where I interact with AI and educating others.

How does this change my thinking?

I would caution people from picking one article as their end all, be all approach for what they anchor on, for how they approach these things.

Because I think there are a lot of different directions and a lot of different ways that we can be purposeful and engaging.

And the more that you process and read on that, I think the better prepared you're going to be.

Right.

Right.

This is just one more piece of evidence and what I thought was a reasonably well done article.

It's really pretty good.

But it is not, as you said, it is not the end of the story.

It's just one more piece of the puzzle.

All right, Rob, anything else?

I'm going to end today with an encouraging word to people as they approach the end of the semester, assuming you're on a similar timeframe that we are here as we're recording this podcast.

And that is as you step into reflecting on what you're doing heading into next fall.

If nothing else, pick one thing, one thing that you're going to engage with over the summer months to make your classroom more equipped to meet students where they're at and preparing them for what the next is, whether the next is a job in the workplace or graduate school.

But just pick one thing.

Don't feel overwhelmed where it has to be.

Throwing out everything and starting from scratch.

Find one thing and that one thing will make your class better.

And regardless of what that one thing is, one thing that will help you with that is subscribing, liking, following whatever your podcast app uses.

Following AI Goes to College.

Everything at AI Goes to College is@aigostocollege.com the Substack Newsletter is linked there, the podcast is linked there.

And we'd love to know what you're planning to do and maybe what some of your questions are about what your one thing should be over the summer.

So you can reach us at Rob Crossler, C R O S S l e r aigostocollege.com or craig@aigi goes to college.com all right, anything else, Rob?

No, I'm good.

All right, thank you very much.

That's it for this time.

See you next time.

Thank you.