Bespoke Communication Devices for Kids with Autism... by Muriel Green

Yeah, I think that's a good thing. Um

Speaker 1: thank you for coming to see my talk and thank you for being so welcoming to me since I'm basically a hobbyist and this is a passion project. So I feel certain that most of the people in this room know more about Django than me and I'm going to take advantage of that during the question and answer session. to pick your brains to try to solve some of the problems that I'm running into with this project. To begin , Uh the the what I put on my name badge for the conference and what I feel like are my qualifications for this. presentation are that I'm I call myself a mom and inventor, but I don't think that those are requirements to do a project like this.

Speaker 1: What I mean when I say that is really just that the only qualifications that I have are that I was passionate about this problem and curious enough to keep trying to find ways to solve it every time I would run into an obstacle. Um one of my children, Malcolm, has autism with language delay. And He has been in speech therapy since he was two, and he's seven now. He still has a really hard time forming words. And he uses a tablet type device at school to communicate. And so really that's where my passion for the project came from

Speaker 1: was trying to find a way to help him and to also help other parents that were facing the same sort of situation because I don't know which slide it's on here, but the the devices that are available on the market are really expensive. The one that we were recommended to get is just over six thousand US dollars and when I was um trying to figure out how we were gonna buy this I was Googling thinking, well surely somebody's got like a a knockoff version or like a generic version or an open source version and there wasn't that I could find. And so that's how this all started

Speaker 1: was I just want there to be a version that people can do themselves. There has to be an alternative. There has to be a version that people can make themselves so that they're not pressured into spending this money. So AAC um stands for augmentative and alternative communication device. And I had never heard of them before except for in r uh with regards to Stephen Hawking. who uh used uh a speech generating device um to to speak towards the end of his life and um There's this clip from The Simpsons where he appears that I think is really great. And this so I'm gonna play it.

Speaker 2: Ladies and gentlemen, I must apologize. This man does not speak for the Council of Alphas. We hold you subomeguloids in the highest rigor. Where are we gonna get to my broccoli juice program?

Speaker 3: Quit butting in, please. Your IQ is a mere 155, while mine is a muscular 170. I am smart, much smarter than you. Hibbert! You should all do what I say. My IQ is 199 for crying out flame. 198, 197.

Speaker 4: Big deal. My IQ is 280.

Speaker 5: Stephen Hawking!

Speaker 2: The world's smartest man!

Speaker 5: What are you doing here?

Speaker 4: I wanted to see your utopia. But now I see it as more of a fruit

Speaker 2: I'm sure what Dr. Hawking means is

Speaker 4: silence. I don't need anyone to talk for me, except this voice box. You have clearly been corrupted by power, for shame.

Speaker 6: Larry Flint is right! You guys stink!

Speaker 4: I don't know which is a bigger disappointment My failure to formulate a unified field theory, or you

Speaker 2: I don't like your tone.

Speaker 4: If you are looking for trouble, you found it.

Speaker 2: Yeah, just try me you

Speaker 6: Oh! Come on, you idiots! We're taking back this town!

Speaker 3: Yeah, let's make literatities literati!

Speaker 6: That's too clever, you wanna!

Speaker 5: Please stay calm, everyone

Speaker 4: Time for his hog to fly. Wrong button Lisa, thank God you're okay.

Speaker 6: Did you have fun with your robot, buddy? Dad.

Speaker 5: Oh, Dr. Hawking, we had such a Beautiful dream. What went wrong?

Speaker 4: Don't feel bad, Lisa. Sometimes the smartest of us can be the most childish.

Speaker 5: Even you?

Speaker 4: No. Not me. Never. I guess everyone has a different vision for the pur Perfect world.

Speaker 5: Wow mom, that's very profound.

Speaker 4: Hey, you read that off my screen.

Speaker 6: Who's up for some beers? I am.

Speaker 4: That's the smartest thing I've heard all day Your theory of a donut-shaped universe. It's intriguing, Homer. I may have to steal it.

Speaker 6: Wow, I can't believe someone I never heard of is hanging out with a guy like me.

Speaker 3: Alright, it's closing time. Who's paying the

Speaker 6: damn? I am

Speaker 4: I didn't say that.

Speaker 6: Yes, I did.

Speaker 1: All right, so when my son uh was diagnosed with autism with language delay and his speech therapist said we need to get a talking machine, I thought, okay, cool, it'll be like a he'll be like Stephen Hawking. And um so w uh it turns out there are a lot of different kinds of speech generating devices, and some of them are specifically geared toward preschool age kids, and they use uh icons or cartoon-like symbols instead of uh text-to-speech or um typing. And the reason that these machines are cool is because it assumes a high level of competence from the children.

Speaker 1: It's possible to just have like some laminated index cards that you keep on a keychain in your pocket to do rudimentary communication. with with people who who don't know you, but um how many cards can you really keep in your pocket? If you've got a digital device then you could have tens of thousands of words at your disposal and um That his speech therapist was really keen on that because she wanted to give him as many opportunities to express himself as possible instead of starting out with a limited vocabulary. And also these machines in general are customizable. You can mask out different icons if you only use certain ones, or you can put in pictures that are specific to your family. But they are really expensive.

Speaker 1: So when I was thinking about this problem, how uh there are these really expensive medical devices that are people need, and I was looking around my house, I noticed that we have all these toys that um that do generate speech. And so I I did a little demo here of a few of them I love you so much

Speaker 7: B O G that smells dog. B E A R that spells bear

Speaker 5: An octopus turns white when it's scared.

Speaker 1: And this this leap pad toy in particular, um

Speaker 5: let's play. That 's the animal.

Speaker 1: Malcolm taught himself the whole alphabet and all the animals, all the names of the animals and how to spell the names of the animals with that thing. So that's that was a good toy. Um but It was like less than $30. And so

Speaker 5: matching letter buttons. Keep trying. Well done.

Speaker 1: That's what I just keep coming back to is this idea, um What's the difference between a toy and a medical device? And um the main difference that I can see is that the the speech therapy devices uh primarily the the words are the words are mostly nouns. No mostly verbs. Mostly like a a toy, the the words are mostly ver. I'm so nervous. On toys The words are mostly nouns like the names of animals or the names of colors. And then in the speech therapy devices, it's mostly verbs because that's what that's what people use to say speak in in natural language.

Speaker 1: And um so this idea That was the basis of this project is just imagine if every child could have an AAC as affordable and rugged as a toy. So I said about building one and I used um Django and I wrote uh I wrote some source code which is available on GitHub and then for the hardware, I was I was basically doing everything I could. To avoid just making an app. I'm trying to make this an embedded system, a standalone device. So I used a Raspberry Pi with touchscreen and speakers and battery. And in the Django Girls tutorial, there is some really helpful troubleshooting for installing Django on Debian.

Speaker 1: So I wanted to highlight that because that is an error that occurs when you're trying to load Django on a Raspberry Pi. And then what I wind up doing is just running the development server as the production environment. So here's the here's the part where I start asking you for help. This is currently what I have and it Seems to work pretty well on Chrome, but it doesn't work in other browsers. And so I think there might be some sort of JavaScript thing that I'm doing wrong. So if anybody wants to make a pull request about that, I would love it. Uh it's just basically looping through a dictionary of of uh JPEGs and then

Speaker 1: um W on mouse down, whenever you touch them on the touch screen, it plays a an associated MP3 file. So my GitHub username is amendment 19 if anybody wants to go look at the repo. And um here I'm going to talk talk through a little uh Demo uh handling of the oh well that's loud um So you can see the the micro USB is where the Raspberry Pi gets power and then there are um there are regular USB slots. The SD card is where the operating system

Speaker 1: resides. And in order to do all this setup, you have to plug in a keyboard and mouse. There is a way to add uh your s reference your The script that you want to execute on startup in the RC local file and then specify it to run in kiosk mode so that it'll just automatically start running your app whenever you power it on And then that white thing is the battery that I found on Adafruit. And the 3. 5mm jack is the default. sound output for the Raspberry Pi, but I've uh I can't get the sound to be very loud and when I've been researching this it sounds like uh it's a generally accepted

Speaker 1: flaw in the system that that port is kind of underpowered so you can't get the sound very loud of it. I've been trying to To create an amplifier that would let me use stereo-enclosed speakers, but it adds a lot of bulk to the project. Also, I've looked at shields. that would go on like Amp hat or or or other um Raspberry Pi shields that that have an amplifier in them but uh It's interfering with the pins that are used for the touch screen. So if anybody is into hardware tinkering and you have some ideas on how I could improve that, I would I'm definitely all ears. This just shows what's in the cloned project folder, the same things that are usually in a Django project. And then in this part of the demo, I'm just spinning up the development server.

Speaker 1: And it seems to be pretty stable. Um I just run the development server on the Raspberry Pi and It works fine for as long as I'm using it. So um since it's a self-contained system, I'm not really very worried about a threat model. It's mostly just will it will it work without failing and it it seems to be working just fine for what I'm using it for

Speaker 1: And this is a Raspberry Pi 3, so it comes with Wi-Fi built in, so I was able to um

Speaker 8: hi

Speaker 1: Um, so I was able to pip install Django easily The first time I tried to build it, I tried to build it on Raspberry Pi 2, which does not come with Wi-Fi. And I thought, oh, I can just copy over every single one of those little tiny files, those little tiny egg files from um From Django and that didn't that didn't ever work. So um if anybody knows a way to do that, I'm still I'm still curious as to why I couldn't get it to work. But um So I made, I don't know if we have time to show the contest video. Um I was looking for a a deadline, like a an enforced deadline, and so I entered the um the Hackaday Prize competition last year,

Speaker 1: which had a whole segment of assistive technology projects. And um It was really fun and I didn't make it into the semifinals, but I did meet a lot of people who were also working on assistive technology projects and the community community there is um is really neat. There are lots of people who are willing to help. And uh so if you want to find me on Hackaday. My username is Muriel G. And if you're interested in building your own hardware, software projects, um they have that contest every year. And there are cash prizes and there are also there's also like a fellowship for the grand prize. So it's I mean, like there's a a prizes

Speaker 1: that are are really exciting and worth winning. Okay, so I want to demo the interface really quick. Like I said, it it seems to work better in Chrome. So if you want to look at it on your phone, this is live on Python Anywhere, murielg. pythonanyware. com.

Speaker 8: And

Speaker 1: And so these um these icons are I drew I drew them but I copied them from a s a speech therapy icon set called Unity. So Like I was saying, they're mostly verbs because they're designed to help children learn how to put together syntactically correct language. So the the icons aren't They're really abstract, but uh the the kids get used to them like this bucket of water is Just means is. And then this the bucket of water that's in the process of spilling over. Were. And then the bucket of water that already spilled. Was.

Speaker 1: And then more everyday concepts

Speaker 8: work. Have feel

Speaker 1: So um I I have a I have a prototype as you've seen in the video but it's a works-like prototype, so it's pretty ugly and I'm trying to find ways to make it smaller and um I've tried a lot of different ways to build enclosures for it. I I I'm not very good at 3D printing or designing things on Tinkercad, so I didn't want to spend a whole lot of time learning a whole nother skill set just to build an enclosure. So um I actually built enclosures out of Legos and just super glued them together and that worked really well because it was way faster. I could iterate way faster. And uh also sewing like stuffed animals, like just kind of stuffing it into a stuffed animal, that

Speaker 1: that worked well too. So one of my main goals with this talk was to encourage people who already have developer skills. to choose passion projects or portfolio pieces that are assistive in some way. According to the 2010 census, nearly 19% of the civilian non-institutionalized population of America lives with some sort of disability. So um here the numbers 56 million people um have a disability um according to the census and um 12% of the population are living with a severe disability that impacts their ability to do day-to-day activities that a lot of people take for granted.

Speaker 1: And so I feel fairly certain that everybody in this room already knows someone with a disability. And so I would encourage you to think about how you can use your skills to help just talk to them and find out what they need and then just make it. Yeah, I think my only um reference was the the census citation there, if you want to look that up. And if you want to get in touch with me, here is my email and GitHub and Hackaday profile. And I'd be happy to take questions for the rest of the time.

Speaker 6: Hey, uh first of all, I thought that was an incredible talk.

Speaker 1: Thank you. Um Uh

Speaker 8: and this may already be a suggestion which is going against your request of it just staying within Python. So apologies if this is unhelpful. Um there is a JavaScript library called P5js. which is the most recent version of something that was originally kind of on top of Java called literally processing.

Speaker 1: Okay.

Speaker 8: Which is a terrible name because if you just Google processing, you'll see a million other things But it was uh it's been used for uh interactions, um visual audio, um and the P5 new version is trying to be much much more mobile. I don't know how well that would apply

Speaker 7: uh

Speaker 8: to um a Raspberry Pi. I would have thought interface-wise that would be applicable. Um but it's also if the possibility of having a I guess a website that could be applicable uh outside just uh a handheld thing that's a library that might be worth having a peek at.

Speaker 1: Okay.

Speaker 8: And uh I would be more than happy to show you some stuff. that I've done with that.

Speaker 1: Yeah, excellent. You should give me your email address. And I'm I'm not totally averse to making this into an app or a mobile site. Um I I really liked the Kivi presentation yesterday. I don't know if that guy is in here but I'm gonna look into that too. So P5 also. Thank you. Sure.

Speaker 7: I'm just wondering if we're gonna go about like, you know, addressing some of these issues, what are some places that we can find like resources or just like uh maybe places where people have like voiced uh needs in this um community and like where things aren't being addressed necessarily. Like I know you said um in your talk there was something like s there wasn't like Uh I totally but anyway, sorry, totally forgot what I was gonna say. But yeah, is there any like places where we could go to find out like needs that aren't being addressed or there just like is only one problem out there.

Speaker 1: I'm trying to think of the people who were the organizations that were encouraging to me. Support groups like parent support groups. Also, there's a speech therapy organization called ASHA. And my both of my younger sons went to a speech therapy preschool. So I know there are that those exist uh preschools that are specific Actually, I think a lot of public schools have special ed preschools in in the public school too because of um they start providing services at age three because of the um idea act.

Speaker 8: What sort of uh feedback have you gotten from your son uh uh in in comparison to like the six thousand dollar uh devices.

Speaker 1: Yeah, he's he's my he's my tester. He and um really The the most exciting thing about this project is um it's he's more interested in the hardware now, so it's almost turning into like I would love for it to be the sort of thing that he can see what I'm doing and then do something better himself someday. Like I th that's why I want to stick with the Raspberry Pi instead of like getting smaller and doing like a an RTOS-based bored with a a dedicated microcontroller because um I would love for it to be the sort of thing that autistic people can do themselves

Speaker 1: and just have the instructions out there on the internet that are really easy to follow.

Speaker 5: All right, and on behalf of a very thankful conference, thank you so much. We'll send you home with some treats.