Design for one challenge with ai assistive technology

Gia is a design for our stakeholder Tom who has dyslexia. 

Implemented for both mobile app and smart watch, Tom navigates between the two in order to better manage his life

Recordings are stored on a calendar and/or a memos log.

memos are distinguished by date time, and category of recording (what the recording pertains too)

How it works

Tom selects which category the recording needs to be

Screen Shot 2020-05-10 at 5.01.03 PM.png

Gia is triggered by using hand motion gestures or by voice command

Tom finds his recordings based on categories with the time and date 

Screen Shot 2020-01-05 at 10.34.57 PM.pn

When Tom needs to quickly find information he can simply ask GIA

He can also ask GIA to add events that have been recorded to his calendar

Screen Shot 2020-01-05 at 10.35.09 PM.pn

GIA is able to suggest to Tom which tasks to focus on based on his time availability and recorded dates

Design Process

This project was done in 6 weeks and was my first full UX project done at the end of my sophomore year in human computer interaction 

 

Assignment: Working in teams of three or four, students explore design challenges of interactive systems on AI enabled Assistive Technology. 

Students choose a technology domain with a chosen user/stakeholder group, conduct field research, synthesize research findings and drive the design process based on insights from the field research

Chosen Stakeholder

Screen Shot 2020-01-04 at 3.13.24 PM.png

Tom (not real identity)

Tom is dyslexic. Dyslexia has made simple tasks difficult for all of toms life.

Dyslexia is a brain based condition and a common learning issue. Kids with dyslexia have trouble recognizing letters and knowing which sounds the letters make. They may struggle with rhyming and sounding out new words. They may forget words they’ve seen before.

Secondary research

Dyslexia affects hearing and speech, writing and motor skills, math and time management, memory and cognition just to name a few. There are a few solutions to dealing with Dyslexia, but there is no cure. 

Solutions include; unique educational approaches and assistive devices. Current devices that help people with dyslexia are the Scanmarker Air, Livescribe Smartpen, Dragon Naturally Speaking, and Reading Pen 2.

Experts have attempted to break down dyslexia into “types.” They suspect that each of these types is linked to a different area of the brain. 

The types are :

                           Phonological Dyslexia         Double Deficit Dyslexia      

                           Surface Dyslexia                  Visual Dyslexia

                           Rapid Naming Dyslexia

Dyslexics think differently. Stress, confusion and heavy concentration only increase perceptual and attentional problems. The harder the person affected struggles, the more difficult reading becomes. 

 

Once finishing school, many with learning differences thrive due to their ability to mentally combine imaginary with real world images in a creative or intuitive way. This talent can play havoc with reading and writing, but it is highly useful for the arts, engineering, invention, etc.

 

All Dyslexics’ challenges aren’t the same. Some may not be able to match sounds with letters while others find it difficult to recognize words with sight. Most Dyslexics have difficulty in more than one area.

Screen Shot 2020-01-04 at 7.20.59 PM.png

Over 50% of NASA employees are dyslexic

 

Dyslexia runs in families; parents with dyslexia are very likely to have children with dyslexia

 

Dyslexics may struggle with organizational skills, planning and prioritizing, keeping time, concentrating with background noise

In the process of learning, reading is an important step. Done through coding the letters (orthography) into their sounds (phonology).

Orthography

spelling, hyphenation,

capitalization, word breaks,

emphasis, and punctuation

Phonography

different sounds and the way they come together to form 

speech and words

Screen Shot 2020-01-04 at 7.30.51 PM.png

Children with Dyslexia have a harder time re-coding.

Their brains don’t make the same concrete 

associations when reading.

Screen Shot 2020-01-05 at 12.29.12 PM.pn

Primary Research

Meeting 1

When did you first realize you had dyslexia?

I guess I never realized until my parents told me. My teacher noticed it took me a long time to grasp basic concepts  and recommended I get tested. After I tested positive they moved me to a school that was for kids with learning disabilities.

What was your experience going to that school, what techniques did they introduce to you?

I had to ride a bus an hour and a half each way and I missed being with my other friends. They focused a lot on small words that have not visual representation (example: the, and, here). Let’s say there a sentence “the cat went in the door”. The words cat and door would stand out and register in my mind. But the other words get jumbled and specifics of what happened get lost. What we’d do is form the letters out of clay so they would be more physical in our minds. I’ve also tried colored glasses which did make a difference. What was nice about the school was the class sizes, there was no place to hide. 

Do you find yourself hiding a lot?

Yes. In school I avoid raising my hand. When I’m texting someone and I can’t spell a word I’ll totally rewrite the message and use another word or not bother texting them at all. 

How does it affect you day to day? 

It makes it difficult to communicate with people. They’ll have a hard time understanding my texts or I’ll get dates confused. I get very scrambled when it comes to dates or estimating numbers. It’s also annoying  giving presentations. There are always a ton of typos that people point out. It discredits my work because people automatically think I didn’t put time into it.

What do you find most frustrating when people try to help you?

I really hate when people scribe for me but change the words. It’s hard to write exactly what someone says but people will change it so much it’s not my writing anymore, When I dictate an answer I want the person to write what I say not what they think I mean. Basically I really get annoyed when people keep trying to correct me. I know they’re trying to help but it’s pointless, I spell everything phonetically it’s not like I’ve never been told the proper spelling. It’s funny because with dyslexia people think it’s my problem. In reality it’s kinda other people’s problem, they’re so focused on my spelling errors they don’t pay attention to what I have to say. 

What helps you currently?

Lingo Phone or any speech to text converter. The problem is it doesn't understand my accent. So I have speak differently which makes me lose track of what I’m saying. 

Grammarly is really good as well. It also tends to create a lot or run on sentences that don’t flow when I read it back. 

Toms Workspace

What does dyslexia affect other than reading/writing? 

I also always forget people’s names. I have a lot memory problems. I remember everything but it’s like I can’t recall it all without some sort of trigger and then all the information floods back. I think it definitely affects my organization. When I need to find something on my computer or in my notes I can’t find it. I don’t misspell things the same way every time, so when I try to find a file I have to guess how I spelled it every time. 

How do you learn best, how do learn things if you can’t rely on reading? 

I like total immersion. When I get into a topic, I need to hang pictures on my  walls, be listening to podcast/ lectures as well as read about it. It’s much easier to learn that way rather than a 2D page of text. I need the redundancy, to see it, feel it, hear it. That way something is bound to stick, and I can make a lot more connections. I think the stronger the connections I have to a concept the more it stick with me. When I read I make almost no connections, do I don’t understand the concepts. I think having a virtual reality for dyslexia would be amazing. I could be totally immersed in what I’m learning but not make a mess in the process. 

Meeting 2

Define your traits?

Interested, Active

Would you call yourself an organized person?

No. I do organize parts of my life, but overall disorganized. (In your workspace?) No, there are certain times but not always. Rock climbing organization is mandatory. (In schoolwork/final projects?) Intricate work that I get lost in. I am interested, but hard to get organized. A template 

would not be helpful.

Are you more organized personally than academically or vice versa?

More personally. Comes naturally, more easily organized.

Are you more organized personally than academically or vice versa?

More personally. Comes naturally, more easily organized.

Do you enjoy physical activities?

Yes. Rock climbing, kickboxing. Skateboarding.

Do you think you choose physical activities to avoid reading and writing?

Double physical activity. Per written. 

Do you think that that has affected your major?

Instant feedback from physical creation. 

Do you think this stems from your dyslexia?

The amount of time it takes is frustrating. I take double the time anyone else does and I 

waste people’s time. Lingo-phone it for texting. I don’t text, I just call people. 

Why do you think your memory is bad?

I don’t know if my memory is bad or overlook things. Sometimes I think of a lot at once. Not overwhelming. I stay busy.

Have you been tested for what kind of learner you are? Visual, auditory, etc?

Tactile and visual. 

Difficulties: dates, times, important readings.

Observation Analysis

Toms notebook

Screen Shot 2020-01-04 at 8.22.25 PM.png
Screen Shot 2020-01-04 at 8.22.33 PM.png
  • large handwriting

  • disorganized

  • alternates from cursive to freehand

  • various simple misspellings

Screen Shot 2020-01-04 at 8.33.07 PM.png
  • organized desk

  • intricate object sketches

  • remnants of previous projects evident

  • significantly more organized

Screen Shot 2020-01-04 at 8.33.19 PM.png
  • fairly unorganized, aside from some attempt with baskets

  • stickers for personality/reflection of self

  • rolling cart implies movement/ hands on

Screen Shot 2020-01-04 at 8.33.28 PM.png
  • lot of outdoor gear

  • slightly organized

  • gear taken good care of​

Screen Shot 2020-01-04 at 8.34.21 PM.png
Screen Shot 2020-01-05 at 12.19.01 PM.pn
Screen Shot 2020-01-04 at 8.33.43 PM.png

Toms main challenges

  • His accent is not translated well through speech-to-text applications.

  • Gets frustrated when technology tries to correct/ “help” his spelling mistakes.

  • Finds it difficult to write notes during lectures

Screen Shot 2020-01-04 at 7.31.18 PM.png

“When I dictate an answer I want the person to write what I say, not what they think I mean.”

Opportunity Space

Provide Tom with a solution that enables him to capture,store and access information he needs to accomplish his daily tasks

Concept

Working with Tom, the team decided to develop a system that best suits his needs and wants, we came up with a digital voice recording system that utilizes AI to schedule events, reminders,and information that can be accessed when he needs it.

Wire flow 

Full voice interaction

Wireframe NEW 2.png

Flow Chart

watch

I watch flow.png
Iphone flowchart (GIA).png

Phone

StoryBoard

Screen Shot 2020-01-05 at 10.19.43 PM.pn
Screen Shot 2020-01-05 at 10.20.05 PM.pn
Screen Shot 2020-01-05 at 10.20.14 PM.pn
Screen Shot 2020-01-05 at 10.20.26 PM.pn
Screen Shot 2020-01-05 at 10.20.36 PM.pn
Screen Shot 2020-01-05 at 10.20.45 PM.pn
Screen Shot 2020-01-05 at 10.21.06 PM.pn
Screen Shot 2020-01-05 at 10.21.14 PM.pn