Why AI doesn't follow length instructions, the best $40 you can spend, and more

0:14 Welcome to AI Goes to College, the podcast that helps higher education professionals navigate the changes brought on by generative AI. I'm your host, doctor Craig Van Slyke. The podcast is a companion to the AI Goes to College newsletter. You can sign up for the newsletter at ai goes to college dot com slash newsletter. Welcome to episode 4 of AI Goes to College. In this episode, I talk about generative AI's paywall problem, Anthropic's release of some excellent new Claude models that actually beat GPT. Talk about Google's bad week. And why generative AI doesn't follow linked versus few shot prompting and the best $40 you can spend on generative AI. It is not what you expect.

1:13 I also wanted to give a shout out to Rob Crossler. If you haven't checked out his interview on AI Goes to College, you ought to. It's very interesting. Rob is a smart guy. Before we get into the main content though, I wanna thank Grammar Nut for catching a small typo on the AI Goes to College website. It's really great that some of the readers and listeners are so sharp eyed. I very much appreciate it. So here's my rant of the week. Large language models, as some of you probably know, are trained on data, and data that aren't included in the training data aren't reflected in the output. This means that biased data on the input side is reflected in biased output on the output side. So none of this is groundbreaking.

2:00 We've talked about this before. You probably talked about this before with others. Demographic based bias is well known and it's a serious ethical issue related to generative AI. But when I was playing around over the last couple of weeks, it occurred to me that biased training data results in a different problem for academic researchers. I'm speculating here because I really don't know exactly what data the various large language models are trained on, but it seems to me that the training data may underrepresent articles from top academic journals, which is a huge problem. And here's why. A lot of top journals are behind paywalls of various sorts.

2:42 For example, if I want to access a recent article from MIS Quarterly, I either have to get it through Association For Information Systems or maybe through my library. And a lot of top journals are like that. I'm not sure in other fields, but that's certainly the case in my field and I suspect it's the case in most fields. You know, sometimes we can get lucky lucky and an article's available through Google Scholar or some other non paywalled repository like ResearchGate or something like that. And eventually many of these articles make their way around the paywalls to become more freely available. But if those articles weren't available as part of the training data, then their findings may not be reflected when you interact with that large language model for your research.

3:34 Now the training data may include abstracts or citations from articles that are included, but those are limited. The abstract only contains so much information and so you're really not going to get the full representation of the article. And since many of our top journals are behind paywalls, you may miss out on articles that are in top journals. That's kind of been my experience so far with perplexity. I think this problem is even bigger for disciplines that rely more on books than journal articles. Now, I'm not sure about the extent of the problem, but it's something to be aware of. For now, I advise caution. Look, generative AI is great. I'm a big believer in it, but it really isn't a shortcut for the hard work of scholarship.

4:27 You still have to put in human thought and effort. So the big news item over the last couple of weeks is Anthropic releasing some new Claude models. This is a little bit complicated, so I'll refer you to the newsletter which is available at ai goes to college.com. But here's the low down. Anthropic, who's a competitor to OpenAI and produces competitors to JetGPT, released 3 new models: Claude 3 Haiku, Claude 3 Sonet, and Claude 3 Opus. I don't think Haiku is available yet, but Sonnet, Sonnet, and Opus are.

5:09 I kind of like the names, since they're loosely the one that's the most capable. Both SONNET and Opus have 200,000 token context windows. So the context window is how much data the model can consider when it's creating its output. You can think of it as the model's memory capacity. Roughly, this is very roughly, a 200 k context window should be able to handle 3 to 400 pages of text. Now there are a lot of variables there. Opus can even go further for some uses. According to Anthropic, it can have a context window of up to 1,000,000 tokens, which is huge, but right now that's reserved for special cases. You can get the 200,000 context window for Claude 3 directly from Anthropic, or you can get it through Poe, and you know I'm a big fan of poe.com. But in Poe at least, Claude 3 OPUS and Claude 3 OPUS 200 k are different models. Claude 3 OPUS is a smaller model than Claude 3 Opus 200 ks. Yeah. It gets confusing, but if you're trying to deal with large documents, either just default to the 200 k version or try the non 200 k version and see how things go. If they don't go well, then try the 200 k version. There's some

6:42 limits through Poe of how many interactions you can have with Claude in a 24 hour period, but they're pretty liberal, so I don't think it's going to be a big big deal. I don't think you're going to run into the usage limits for most use cases. What is a big deal is this ever enlarging context window. That just opens up a lot of interesting possibilities. For example, club 3 should be able to summarize and synthesize across multiple journal articles in a single session. I haven't tested this out yet, but I'm going to soon and I will certainly let you know how it goes. The other big thing about Claude 3 is that according to Anthropic, Claude 3 outperforms GPT 4 across the board. Now if you go to the newsletter there's a nice little table, it's from Anthropic though, that shows a bunch of popular benchmarks and how well Claude 3 and the various models and GPT 4 and Gemini Ultra and Pro did. And in every single instance, Claude 3 was better. Now who knows? You know, is Anthropic cherry picking here? Maybe, but even if they are, the performance is quite intriguing. And I think it bodes well for the future of these models.

8:10 Competition is gonna push everybody. Google and Anthropic and OpenAI, they're all going to push each other and whenever Apple gets into the game and Meta and on and on and on. So I think the competition is good. It's gonna help us push the boundaries of what's possible with these models. Okay. Let's switch to another little bit of kind of amusing but insightful news. Google had a bad week. This is a couple of weeks ago by the time you listen to this. So the basic problem was that Gemini was creating some, shall we say, interesting images. America's founding fathers as African Americans, the pope is a woman, and there were some others. Of course, the world being what the world is, there was a bunch of outrage over this. Although I kind of chuckled and I think a lot of other people did. So according to Google, the problem came when they were fine tuning Gemini's image generation model, which is called Imagen 2. So the fine tuning was intended to prevent the tool from, and I'm quoting here, creating violent or sexually explicit images or depictions of real people. Engineers were also trying to ensure gender and ethnic diversity, but Google seems to have overcorrected, which resulted in some of these curious and historically inaccurate images. I think we need to get used to this. Widespread use of generative AI is still very new and we're still trying to figure out how to

9:42 implement appropriate guardrails. In fact, there's not even widespread agreement on what those guardrails ought to be. So we're gonna continue to see these sorts of problems. There'll be a problem, There'll be an over correction. These are going to go back and forth swinging like a pendulum until we eventually find the equilibrium and the right balance between freedom of use and freedom from harm. I'm pretty confident that we'll get there eventually, but it's gonna take a while. So when you see these kinds of problems, it's good to be aware of them, but don't get unduly upset that there's some right wing or left wing conspiracy going on. I think most of it is just honest engineers trying to find the right balance between freedom of use and freedom from harm.

10:27 So in the meantime, I think one of the big messages that I wanna take away and I want you to take away from this is be careful of relying on generative AI for anything important or anything or anything that might be seen by the public unless there's human review. The human in the loop is critical especially at this stage of generative AI. So make a human check part of the process whenever you use generative AI for anything important. Maybe it wasn't practical for Google, but for most of us, it will be. If you want some more details, there are a couple of links to articles about this whole brouhaha in the newsletter. Again, the newsletter's available at aighostocolllege.com. You really should subscribe.

11:14 Okay. So here's my first tip of the week. And this comes from listener Ralph Estep, who's also a friend of mine, who sent me an email asking me why generative AI is so bad at following instructions about length. By the way, Ralph has a really good daily podcast that focuses on financial health. You ought to check it out. It's, available at askralphpodcast.com. It really is very good, and we all need to keep an eye out on our financial health, especially given inflation and some of the uncertainties in the world. So you have probably experienced the length problem of generative AI. You tell Chad GPT or Gemini or Claude or whomever to produce an output of 500 words, and there's no telling how long the output will be, but I'll bet it's not 500 words.

12:02 When I try this, I get 200 words. I might get 750 words. And this can be kind of frustrating. So I wanted to understand what this is all about. So I asked Gemini, why is this a persistent problem with generative AI tools? I actually kind of liked Gemini's response, so I put it verbatim in the newsletter. I even put a link to the conversation in the newsletter, so you ought to check it out. But here's the bottom line. What you need to do is give it a range, not a target. So don't say 500 words, say between 3 56 100 words, or something like that. You can provide examples of writing that fits the length that you want. These can be templates that AI can follow. Another good approach is to start small and then build up. Ask for a short summary first and then ask for more detail on specific points. This gives you more control. And so, how you phrase your request might also make a difference. And this is according to Gemini. If you say summarize this topic in about 400 words, it might work better than write 400 words on this topic. It's gonna take practice, so I just wouldn't rely on it ever to give me a specific number of words. But as you practice, you can find a way to get it closer and closer to the link that you want. This is kind of a good thing for those of us who are instructors because it's gonna make students actually work a little bit instead of just spitting

13:34 out 500 word papers. Okay. Here's a very useful tool. There's a little bit of a pun there. More useful things, which comes from the creator of one useful thing, which is a newsletter I really like, Ethan Moloch and Lillek. I think it's Lillek, Moloch. They've got a new website called More Useful Things. You can go to more useful things dot com and find it there. It includes an AI resources page that, no surprise, includes some pretty useful AI resources. There are 3 sections. 1 is a pre order section for Ethan's book, Co Intelligence Living and Working with AI. I pre ordered it, and I think it's probably gonna be pretty good. There's other resources that has some stuff like an AI video, not an AI video, a video on AI and links to some of their research on AI. But what I really want to talk to you about is their prompt library. Their prompt library includes instructor aids, student exercises, and some other stuff. The instructor aid prompts are really pretty good, but they are long and they they are complex. For example, they've got one that will create a simulation and that prompt is over 600 words long. Look. There's nothing wrong with this. In fact, complicated prompts are often very effective, especially for

15:07 more complicated tasks. But I want to be careful here. I don't want you to look at the complexity of these prompts and go, oh good Lord. I'm never going to be able to learn this. Don't have to know how to write those prompts, especially not at first. You can accomplish a lot with pretty simple prompts. So both simple and complex prompts have their places. You can start off simple, everything will be fine. But you really ought to check out more useful things. Even if it just gives you some ideas about how generative AI can be used, it's worthwhile checking out if for no other reason than that. So check it out. Okay. On to the next topic.

15:48 Recently I was listening to an episode of Dan Shipper's How You Use Chat GPT. It's really good. I think it's on YouTube. I listen to it as a podcast. Dan was interviewing Nathan Lebens on how he uses chat gpt as a copilot for learning. Episode was very interesting. Check it out. But what caught my attention was a discussion of something called chain of thought versus few shot prompting. This is a little advanced, so I want you to stay with me here. But if it gets to be too much, just move on to the next segment. Few shot prompting is pretty easy to understand. You just follow your task description with a few examples.

16:30 So let's say that you wanna create some open ended exam questions. Do you give chat GPT or Gemini or whomever, whomever, whatever? Is it whomever or whatever? Oh, that's scary. You give the tool of choice your prompt, say, create some open ended exam questions on this topic and give it some parameters, and then you give it 2 or 3 examples. Now what the AI tool will do I'm gonna say chat gpt here to just make it easy. What chat gpt will do is it will look at and analyze your examples and try to create questions that are similar to your examples. Sometimes just giving a single example is really useful. They call that one shot prompting.

17:17 Chain of thought prompts are a lot more complicated. The main idea is that you ask chat gpt to think aloud. So I'm gonna give you an example of a chain of thought prompt, and this one was produced by Chatt GPT. Explain the concept of chain of thought prompting using the chain of thought approach. I thought that was clever. Begin by defining what chain of thought prompting is. Next, break down the process into its key components explaining each one step by step. Then illustrate how these components work together to guide an AI in processing and responding to complex tasks. Finally, conclude by summarizing the advantages of using chain of thought prompting in AI interactions, especially in educational contexts. And then the result was pretty long. I'm gonna have to send you the newsletter send you to the newsletter to check that out, but this can be a good way to really get chat gpt to do more complicated things. I don't use chain of thought prompting very much. I think few shot prompting works really well, but few shot prompts require knowing what good output will look like.

18:37 If you're not sure what you want, you might consider chain of thought prompting. But if you're a beginner, stick with few shot prompting. Even one shot prompts really are quite useful. I use that quite a bit actually. Okay. So there are a couple of messages here. First, keep things simple when you can. Simple is often very effective. 2nd, don't be afraid to experiment with different approaches and even to blend approaches. So you can decide. Your prompts can be simple or they can be complicated. All right. Here's my favorite little part of this episode. This is, I think, the best $40 you can spend, and it's $40 a year, on your generative AI use and productivity.

19:24 Look. I know monthly software subscriptions are totally out of hand. I don't even wanna know how much I'm spending at 5, 10, 15, $20 a pop every month. But despite this, I recently added a $40 per year subscription, and it's already proven to be one of my best software investments ever. Alright. So what is this great investment? Well, it's for a text expander, and I wanna give a shout out here to Dave Jackson of the school of podcasting who talked about this on one of his episodes. And so a text expander just replaces a short bit of text with a longer bit. For example, if I want to type the web address for AI Goes TO College, I type semicolon uaig. U is short for URL. And the text expander, it just gives the full address. The semicolon here is used to indicate that what follows is gonna be or could be an abbreviation for a text snippet. The semicolon works well because it's usually followed by a space rather than characters, but you could really use anything you wanted. Now this doesn't save me a lot of time, but it saves me 5 or 6 seconds every time I wanna type the AI goes to college website.

20:39 And it's long. You know, it's got the HTTPS, colon, etcetera, etcetera, etcetera. It just takes a while. I have to give people my biography periodically. Matter of fact, once a month or so somebody asks for it. So normally I go find the Word file and attach it to an email or copy and paste it into the message. Now I just type semi colon bio and my bio pops up. And I use this for student advising. I use this for grading. If you're a professor, you have to grade a lot of kind of projects, that sort of thing. A text expander will change your life when you have to grade. My spring class has 85 students in it, and I'll grade 85 projects for twice. What is that, A 170 projects? And a lot of the feedback will be exactly the same. You know, they forget to use the 1,000 separator, the comma, or their their spreadsheets aren't formatted well, that sort of thing. Well, now I can just in a few characters pop in the feedback and the number of points I'm taking off for that.

21:48 So what does this have to do with generative AI? Well, as you start I Now I just type semicolon w d y t. I have one for please critique this text. I have one for this is really lazy. I have one for thank you. And you'll find more and more and more uses for a text expander once you get into it. So I use generative AI for a lot of tasks, so this helps a lot. But it also helps when I need to provide some context. So for example, I try to tell generative AI whether I'm working on something related to my teaching or one of my podcasts or this newsletter. When I'm working on the newsletter, I have a whole blurb. It's, I don't know, 5 or 6 sentences long. Now, I just type in an abbreviation. That's it. So it's semicolondaig, which just means description of AI goes to college and this whole bit of context pops up. So if you're not using a text expander, you really ought to consider it. I use one called you're gonna love this text expander. I use it. That's the one that costs about $40 a year. I use it because it's cross platform.

23:12 It'll work on a Mac. It'll work on a PC. I think it may even work on smartphones, although I haven't tried that yet. So I really would encourage you to consider making that investment, not just for generative AI, but for your use in general. Okay. All right. Last thing I wanna talk about is the interview I had AI Goes to College podcast, which is surprisingly available at ai goes to college.com. We talked a lot about a lot of different things. Rob is a really smart guy. He's got a lot of experience. He's a department chair. He's a fantastic researcher. So he's used generative AI in a lot of different contexts. We talked about how he's using AI to create more creative assignments and to generate questions, how he's helping students learn to use AI tools tools to explore and understand important concepts. We talked about the importance of being willing to experiment and fail with generative AI. We discussed why it's important to help students become confident but critical users of these fantastic new tools, and we talked about a lot of other things. So go to ai goes to college.com/rob, r o b, and you can check out the entire interview. I'd love to hear what you think about it. You can email me at

24:43 craig@aighostocollege.com. Let me know if you've got any ideas for future episodes or if there's something you wanna see in the newsletter, and you might get featured. Alright. That's it for this time. Thank you. Thanks for listening to AI Goes to College. If you found this episode useful, you'll love the AI Goes to College newsletter. Each edition brings you useful tips, news, and insights that you can use to help you figure out what in the world is going on with generative AI and how it's affecting higher ed. Just go to ai goes to college.com to sign up. I won't try to sell you anything, and I won't spam you or share your information with anybody else. As an incentive for subscribing, I'll send you the getting started with generative AI guide. Even if you're an expert with AI, you'll find the guide useful for helping your less knowledgeable colleagues.