podcast friday

[sabotabby]

Oh shit did I almost forget to do this again?

Okay, so I have Opinions about AI and I know they are not 100% shared by all of my friends and AI controversies caused drama in at least one Discord I'm in, so I'm going to put this under a cut and you can read/engage with it if this is a thing that's interesting to you, or not. It also deals a little bit with bad things happening to kids in a weird, hard-to-define way that makes me uncomfortable and may make you uncomfortable as well.



"AI Is Coming For Your Children" (Part 1 | Part 2) is Behind the Bastards two-parter about a particularly strange grift in the world of publishing. Basically, people generate children's stories using ChatGPT and illustrate them using Midjourney and then put them up on Amazon and let the money roll in.

This grift relies on a bunch of things to work. For one, there would have to be money in publishing. I can assure you that there is no money in publishing. The only money to be made in publishing is to convince other people that if you give them money, they will teach you how to make money in publishing. But grifting off people who are stupid enough to think that publishing children's books will get them rich isn't generally enough to warrant a Bastards ep, because these are silly and greedy people being grifted and taking their money is a moral good.

Unfortunately, there is genuine harm being done here. The reason being goes to some of the deep-rooted issues with AI. For one thing, ChatGPT and Midjourney are not AI. They are just tools for combining other people's words and images in an order that makes sense some of the time. In a lot of cases, it's intelligent but not artificial—there's a ton of work being done by underpaid people in the Global South that's being labelled as AI. This is why writers aren't actually going to be replaced by ChatGPT. What happens instead is that ChatGPT will "write" a script and a writer will be hired to "edit" it for less money than the writer would get for actually writing it. It's grifts all the way down. Also no one wants to pay actual money to watch or read this kind of content when you can just get it for free. The tech itself isn't actually the problem; it's just capitalists in the Global North who are upset because they don't get to personally own slaves anymore. So AI will not replace authors and screenwriters any time soon.

But children's books are a different story, because children have developing brains. Robert and Ben talk a little about Elsagate, one of my bizarre special interests. This was a situation where a tired parent would plunk a little toddler in front of Peppa Pig on YouTube and stop watching because who wants to watch that?? YouTube would autoplay the next video, and eventually you would get some weird shit where Elsa from Frozen is pregnant and being injected with needles and no one knew where this content was coming from. Was it AI generated? Pervert generated? About 10,000 viewers, many of them little kids, saw this content, and it was hella disturbing and not the kind of thing that kids should watch.

When you have a ChatGPT-generated story, it's not a narrative as such. And Midjourney images have well-documented issues, such as hands, or character consistency across poses. The "books" being produced here are some uncanny valley shit, and they will be purchased mainly by parents and organizations that are low on cash. Robert suggests a world where wealthy kids have real books written and illustrated by real people, and poor kids learn that dinosaurs have creepy little people-hands. And since the illustrations don't quite match up with the story, they lose a vital step in reading development called "pre-reading," where the images help kids learn what language is and how it works. Not to mention what's happening when kids think that ChatGPT stories are how narratives work, and then you have a generation that grows up to expect and even produce contentless narratives.

There are some cool uses for this tech, like creating your own non-commercial art and memes, or making it write epic rap battles between Zizek and Peterson, but like any new tech, it's being seized upon by the worst actors who don't think about the consequences. And there are quite likely to be consequences here.

If you'd prefer to read it, you can do so here.

Comments

[ioplokon]

Yes & is also the acceleration of things happening elsewhere. When I was a TA, I was instructed not to correct students' factual mistakes & to just grade based on the structure of their argument...

Also, people tend to think quality does not matter in children's entertainment, when of course it actually matters quite a lot. There's a pretty limited window for language acquisition, after all :(

Also also, I think it will effect the next generation of professionals bc it kind of eliminates the entry/apprentice level. This is what you see in translation. You still need highly skilled people to correct the output & create the dictionary for it to use, but how do you get to that level if all the simpler tasks are fed to the machine translation?

Same w/ writing. Skilled writers probably only spend like 5-10% of their project time actually writing (vs research, planning, revising, thinking). But of course you need to be a proficient writer to get to that level.

[sabotabby]

Another area of concern for me is journalism. There was already a lot of chum content out there and this accelerates it. I'm increasingly seeing kids believing that ChatGPT is a search engine, which is fucking terrifying. I also don't love the idea of AI-powered search engines. I used to work for a non-Google search engine and there were significant numbers of humans involved in making sure that results were reliable and what the user was likely looking for. And of course Google has monopolized the search engine game and has become the portal to how most people access information.

[ioplokon]

Yeah, I was talking w/ my coworker about it & she was like, so everyone will just get stupider & not know anything?

And it's like, not quite. It's more that information asymmetry will increase. If you already know where to go for reliable info, you can find it. If you don't & especially if you don't know what you don't know, you're screwed.

[sabotabby]

Hell, I don't know what is a reputable publication now.

[ioplokon]

no one's good on all topics but the nice thing w/ specific publications & bylines is you can remember who is full of shit. I generally mostly judge based on what holds up months/years later (bc i am a geek who sets myself reminders to check up on stuff). You can't do that w/ an llm that produces some different nonsense each time you ask it.

I do also think it's worthwhile to make the effort to engage w/ actual investigative stuff vs just commentary (or stenography)

[sabotabby]

Yes, absolutely. It's more like, I know who to follow for particular stories and themes but I am left horribly betrayed by, say, the misinformation that CBC put out about covid, given that they're supposed to be the national publicly funded news here.

[dewline]

Someone somewhere already decided that if there aren't bad consequences befalling good people because of these choices, then that someone didn't make the right choices.

I'm paranoically certain of this.

[sabotabby]

Yup. There are all kinds of good ways to use this, I'm sure, but people picked the worst ones instead.

[greylock]

Elsagate was so freaking weird. IIRC it predated AI, but I might be misremembering.

I worry about my nephew on Roblox and YouTube.

[sabotabby]

It was 2016-17, so the technology was starting to exist. I think people don't realize that LLMs aren't AI and aren't new.

Check in with your nephew. Those are hotbeds of radicalization even without accounting for AI.

[greylock]

It was 2016-17, so the technology was starting to exist

It was pretty bad... but then I realised: YT didn't need to be that good.
And kids, being kids, don't register the uncanny valley like we do.

Check in with your nephew. Those are hotbeds of radicalization even without accounting for AI.

I cannot even had a discussion with him. (I think he's on the spectrum). He's obsessed with being a game dev/YouTuber. And his references are not my world.

While I am not the most WOKE person in the world I've not detected anything for what that's worth. But I am not entirely sure what I would be looking for. If he gets X I'll introduce him to my lady friend and her lady friend, I guess.

I am hoping my BIL has it under control.

[sabotabby]

That's the thing—people assume that kids aren't discerning, but they can tell when something's wrong. There's a certain amount of uncanny valley that I think is good for kids (I grew up in the 80s) but they should be exposed to things that are. you know. narratives. Not just word salad nonsense.

It's a good idea to familiarize yourself and/or BIL with what the kids are using as racist slang. For example, when I was dealing with a radicalized kid, my coworkers didn't know what (((triple parenthesis))), Pepe, or remove kebab meant, but I did. The jargon has changed since then.

The gateway for most young boys is misogyny, not racism. Tate is still wildly popular amongst the junior high set.

[greylock]

(I grew up in the 80s)
Same.

but they should be exposed to things that are. you know. narratives
Oh, jesus. This just triggered me.

my coworkers didn't know what (((triple parenthesis))), Pepe, or remove kebab meant, but I did.

"remove kebab" was new for me. OUT of nowhere.
Jesus fuck.
I hate the world.

I'll have a chat with BIL (we don't get along, but we don't not) I suspect a lot is voice. But my SF is friends w on Roblox, and he's ALWAYS LISTENING.

I hope N is too stupid/ASD to be a Tate fan, but I'm going to be paying attention to a lot more now. If he uses trigger words...

[dissectionist]

I strongly advise watching this video, and getting BIL to watch it if possible. I say this as an autistic parent with two autistic kids who are also big Minecraft/Terraria/Roblox nerds (and one of whom wanted to be a Youtuber). I guarantee kids ARE being exposed to this and will pick some of it up, and some of those will go on to pick up a whole lot more.

I ended up watching the video with my kids (who were about 10 and 14 at the time) so they could recognize if they saw this happening to their friends and be more aware of when it’s done to them. I don’t know your nephew’s age, but it could be useful for him to see if it’s age-appropriate.

https://youtu.be/pnmRYRRDbuw

[sabotabby]

Co-endorse this recommendation.

[greylock]

Never going to watch that myself, and my BIL: nope.

I will flag this, but my sister is useless

*

[minoanmiss]

There are some cool uses for this tech, like creating your own non-commercial art and memes, or making it write epic rap battles between Zizek and Peterson, but like any new tech, it's being seized upon by the worst actors who don't think about the consequences. And there are quite likely to be consequences here.

Word.

*saves this to come back to*

Re: *

[sabotabby]

I have had fun with it! But I'd never use it for commercial, or even broadly public, purposes. I know teachers who use ChatGPT for report card comments, which is both hilarious and upsetting.

[blogcutter]

This is all hugely interesting to me. It reminds me of the pre-Chatbot era of the 1980s and 90s when personal computers were just starting to become prevalent. A parent or relative or friend could commission a potted story with generic-looking illustrations "customized" to a particular child by listing, for example, the names of the child, the child's siblings, the family pet, 2 or 3 of the child's friends, the city or town where the child lived and other details they wanted included. For example:

Sabotabby's Birthday Party: Sabotabby was so excited! It was her birthday in just 3 more sleeps and she was going to have a party! She hoped that her good friends Alpha, Beta and Gamma would be able to come. Mommy was going to make her favourite foods: carrot cake with cream cheese icing, to be served with rocky road ice cream!

(Well, you get the idea)

One practical use I could see for AI-generated children's book would be in media literacy courses, whether for kids, teachers, librarians, parents or basically anyone who was interested. You could do practical exercises like offering a selection of real human-written books (the text of some Little Golden Books, for example) mixed with examples of AI-generated text and getting the participants to distinguish which were which.

Just some preliminary thoughts. I'll be mulling it over further.

[sabotabby]

Ah I remember those. I was violently against them as a child because I hated people saying my walletname (I still kind of do). So even when a character randomly had the same name as me—which is a quite common name—it would make me cringe and I wouldn't like it.

There are a lot of interesting pedagogical exercises you could do with ChatGPT. Like get it to answer a question and fact-check that question. Or play around with voice. I did an exercise in a Discord server (the same one mentioned in this post that I left/got booted off for drama) where other people were trying to show how great ChatGPT was and I was showing why it sucked. We were getting it to do Jack and Jill in the style of different authors or genres. They mostly have shit taste in books so it was working pretty well for them. Like I have no doubt you could get AI to replace Brandon Sanderson. But as soon as I tossed Cormac McCarthy in there and then Iain M. Banks, the thing fucking choked. It knew that McCarthy wrote a lot about cowboys and had landscape porn, and it knew Banks wrote sci-fi, but after one or two paragraphs, it fell back into the word-by-word generation, and both stories ended the same way—Jack and Jill knowing that they could face any obstacle as long as they were together. This was actually how I picked up on how ChatGPT's authorial voice sounded and how to distinguish it even if the auto-detectors can't.