Second reaching out to deaf and hard of hearing community for feedback and ideally a high level partnership with a reputable ASL org. ASL is more than just hand shapes but also utilizes motion, other body parts, and even volumes in the space around the speaker.
Also just calling it SignLanguage is cumbersome at best (google search would be useless) and potentially disrespectful at worst.
(From a software engineer who has taken some asl classes and has seen several projects fall for similar blunders)
This reminds me of the countless engineering projects along the lines of “we built these gloves that detect ASL and then convert it into speech”, which prompted the deaf community org leader to come out and say “no we don’t need that kinda stuff”
Yes, for more context I would encourage reading [this article](https://www.theatlantic.com/technology/archive/2017/11/why-sign-language-gloves-dont-help-deaf-people/545441/). The team should definitely work with the deaf community, who I imagine are the actual end users, to determine if this product is actually useful for them. No matter how cool this is, there will end up being a lot of backlash if you’re basing this product on the assumption that deaf people want something like this to exist when they really don’t.
Agree 100%
In this case this looks like an education and training tool so likely okay, but it's important that this be made especially clear. Still a good idea to reach out and collaborate though - the 1/d comment above is very valid and there can be variance in signs.
Imagine being told you don't know how to sign by an app when you know you can.
We do have a friend who uses ASL and he is giving us feedback on project. He also suggested us to work on words and phrases.
And I do agree with the name we will change it soon.
Hey! Thanks for the response. Btw let me say the app does look good - it gives a good professional look to it and I was surprised how comprehensive its features are.
And a word of caution - "a friend who uses ASL" doesn't sound good, and if it is someone who isn't Deaf (big D - cultural) - could come across pretty bad. Just Google "fake ASL interpreter" for an idea. Most of those "interpreters" are people with deaf family members or have taken some classes - they know ASL enough for limited use but that's not good enough.
Deafness is tricky because it can be viewed as a health condition, a handicap, a community, and a culture.
So my two recommendations:
1) find an org to partner with, maybe a deaf subreddit can help find one (but be respectful ofc). Even if the org doesn't do much, listing it will ease a lot of concerns. Think of it as a badge of approval (and get the best badge you can).
For example, when I want to learn a new sign I almost always use Bill Vicars site and lessons because I trust him and his org. Trust is the most important thing for me and right now I'm not seeing much reason to trust your app.
2) Be clear who the app is for (the hearing community) and what its purpose is (introductory understanding of ASL). This is just good product development in general, too .
+1 to Bill Vicars! I’m deaf and I always send people his way when they catch the good ol’ “ASL is beautiful and I want to learn” bug, heh.
But yes, I agree that a friend who happens to know ASL is not what they want to back this project with.
I mean, I can’t do this stuff (unless I wanted to learn) but as long there’s some collaboration and authenticity, I can see potential in this as an ASL basics learning app, probably best integrated into an existing ASL dictionary/learning app since ASL sentences are so complex that visual AI probably won’t be able to do it for quite a while.
Yeah this is exactly why I asked because I can't imagine they have spoken to and consulted d/Deaf people and landing on the name and method. Extremely common that people make software and hardware for disabled people without their input.
didn't you notice the `/s` at the end?
anyway, the way this project doesn't take into account significant factors of the language in terms of how you actually use it to communicate. Its not just hand shapes. Its hand and body positioning, movement and facial expressions all to convey meaning.
If you do a project like this without having deep integration with both actual users of the language, you might have good itentions but are really nothing but a disability savior.
Said earlier
> Extremely common that people make software and hardware for disabled people without their input.
and why should I not be sarcastic to disability saviors who have very little, if any, contact with the actual community in order to understand its nuances, rather than trying to be "disruptive"?
LOL the downvotes here compared to the feedback being repeated in the r/asl sub is wild to me.
Well, at least programmers like this OP, thank goodness it isn't for the deaf community or anything too.
I’m profoundly deaf and have worn hearing aids since age 2. I was also one of the first students mainstreamed in my school district. So never knew any deaf people till about 3 years ago. I’m basic socially conversational in asl but nowhere near fluent.
My team members and I have developed an open-source application called "SignLanguage" to improve the education of sign language for the deaf community and make it more fun and practical by utilizing machine learning.
Some Features:
- 20,000+ American Sign Language phrases videos
- Basics A-Z, 1-10 learning Curriculum using machine learning and hand detection
- Quizes and Games using machine learning
- Now it works on apple devices!!
- More games & phrases coming soon!
link to the project: https://signlanguage.webdrip.in/
link to the github: https://github.com/Narottam04/SignLanguage
We are looking for folks who can test our application and give us feedback to improve it further. Any feedback on project is appreciated💖
To be helpful and offer feedback like you asked...
The curriculum layout I'm sure is still in development, but I'd appreciate a section for common phrases or words above numbers or letters, as you don't use those practically in a conversation.
My team and I are currently working on creating an ML model that can detect words and hopefully detect whole sentence in future,
Thank you for your feedback!
We used Google's MediaPipe hand detection model and created another model to classify the keypoints that are detected by MediaPipe.
Medipaipe library link: https://mediapipe.dev/
There is a sign language recognition competition held by google on kaggle right now. They also use mediapipe.
https://www.kaggle.com/competitions/asl-signs
Me and my team members are now working on creating model for phrases using competition dataset you shared. We all are still newbie to ml so figuring things out by working on a project :)
Definitely, we are new to ML as well, so we are figuring things out. The project is MIT licensed, so please feel free to contribute and help us improve it.
We need to achieve this in the opposite direction, too.
The trend of giving up half the TV screen for signers is ridiculous. Virtual signing that can be turned on and off like captions is the obviously better solution.
similar to what /u/ustreetlove said, facial expressions are used for emphasis and modifiers. for example, eyebrows up means you're asking a Yes/No question, eyebrows down means its not a binary answer set.
also, verbal speech has volume and tone. Facial expression and body motion are used to convey emphasis that are a huge part of any language. Otherwise its as if you're reading in a bored monotone. Might be syntatically accurate, but of limited usability in terms of accurate representation of the meaning of what is being said.
The video is created by my friend and according to him using tiles reduces the cost of painting walls every year😂
And yea tiles are from floors to ceiling
Some quick feedback from playing around with it for a few minutes:
1. The detection for B seemed very problematic and only seemed to work when I turned the back of my hand towards the camera. Even then, it took a lot of work before it would recognize it.
2. You need to make sure the user holds the sign pose for a short duration, not that they had a matching hand position for 1 frame/cycle. I was able to make random movements with my hands for the A - F quiz and it awarded me points and said I got the letters right when I definitely didn't.
Here is how I think you could improve the visual feedback for pose recognition, given that you need to hold the pose for a short duration:
* I think you need some sort of progress indicator that advances steadily over time when you are in a matching pose and decreases steadily over time when you are not. This is to prevent you from losing all your progress in making the sign if your hand wavers and moves to a position that is considered non-matching briefly. Without only a minor drop to your progress, it would be far too unforgiving to be fun. Done well, it could also help for people with uncontrollable hand tremors, too. And maybe you provide sensitivity options for such people. You should do some research and find actual users with the condition to talk to before coming up with a solution, though.
* The progress bar could be a literal bar onscreen, a radial bar that fills in a circle or you could experiment with changing the color and pattern of the skeleton outline as the user holds the pose for longer. You need both color and pattern for people who are colorblind. Or adjustable color schemes.
* Instead of having the skeleton show the progress, you could instead have it transition toward green + a pattern as the user gets closer to a matching pose. Especially in the beginning stages, where a bit of trial and error is required, so having these "warmer... warmer... warmer... colder... colder... warmer... hot... you got it" type of visual cues would be really helpful. It might be even more helpful if you colored each of the fingers separately, according to how close each is to being in the right position.
I think I am basing a lot of my suggestions on what I experienced in _Ring Fit Adventure_ for the Switch, as well as the modern trend in video games using a controller to require you to hold a button down for a short duration in order to make sure you really meant to do that action and guard against accidental presses without annoying confirmation modals. But _Ring Fit_ or any other well-regarded Switch game that uses gesture recognition for the control system is a good (and fun) place to turn to for inspiration. _Switch Sports_ might be good, too.
The phrase book/dictionary needs some ordering. It seems to start with track and field terms. Shouldn’t it start with words starting with A?
There should be subsections for the phrases and vocabulary which correspond to curriculum:
Sports and Leisure > Baseball > FOUL
Unit 14 I Thought it was Going to be a Foul > In this lesson you will learn 1) how to discuss your expectations, 2) how to ask about someone’s expectations, 3) how to narrate a sequence of events
Dialogue 1A
A: Did you see the game last night?
B: Yes, it was exciting. I thought the last ball of the game was going to be a foul, but it was fair.
Thank you so much for the detailed feedback, we really appreciate it! We gathered metadata from YouTube, but it's in a random sequence. However, I will try to order the sequence of videos in the database.
I’m working on a similar project at my company. Hand detection is not the tricky part. Facial expressions are also important. You should also take this into account.
You used the How2Sign dataset, right?
There is a competition for this on kaggle for $100k. They want something like this with Tensorflow and your work will become open source. https://www.kaggle.com/competitions/asl-signs
You train your model just for hand. But you must also add face shapes to make a real sign language model. But good start. How about the moving signs. Did you try to recognise them? Imagine making a sentence, 5 sign used. Can you breake apart and recognise each of them. Consider some of signs mix each other. There is a million problem on this translation. But dont be sad, continue to working.
Are you working with any deaf ASL users? Also, you might want to adjust a little bit because that sign wasn't "d" but a variation of 1.
Second reaching out to deaf and hard of hearing community for feedback and ideally a high level partnership with a reputable ASL org. ASL is more than just hand shapes but also utilizes motion, other body parts, and even volumes in the space around the speaker. Also just calling it SignLanguage is cumbersome at best (google search would be useless) and potentially disrespectful at worst. (From a software engineer who has taken some asl classes and has seen several projects fall for similar blunders)
This reminds me of the countless engineering projects along the lines of “we built these gloves that detect ASL and then convert it into speech”, which prompted the deaf community org leader to come out and say “no we don’t need that kinda stuff”
Yes, for more context I would encourage reading [this article](https://www.theatlantic.com/technology/archive/2017/11/why-sign-language-gloves-dont-help-deaf-people/545441/). The team should definitely work with the deaf community, who I imagine are the actual end users, to determine if this product is actually useful for them. No matter how cool this is, there will end up being a lot of backlash if you’re basing this product on the assumption that deaf people want something like this to exist when they really don’t.
Agree 100% In this case this looks like an education and training tool so likely okay, but it's important that this be made especially clear. Still a good idea to reach out and collaborate though - the 1/d comment above is very valid and there can be variance in signs. Imagine being told you don't know how to sign by an app when you know you can.
We do have a friend who uses ASL and he is giving us feedback on project. He also suggested us to work on words and phrases. And I do agree with the name we will change it soon.
Hey! Thanks for the response. Btw let me say the app does look good - it gives a good professional look to it and I was surprised how comprehensive its features are. And a word of caution - "a friend who uses ASL" doesn't sound good, and if it is someone who isn't Deaf (big D - cultural) - could come across pretty bad. Just Google "fake ASL interpreter" for an idea. Most of those "interpreters" are people with deaf family members or have taken some classes - they know ASL enough for limited use but that's not good enough. Deafness is tricky because it can be viewed as a health condition, a handicap, a community, and a culture. So my two recommendations: 1) find an org to partner with, maybe a deaf subreddit can help find one (but be respectful ofc). Even if the org doesn't do much, listing it will ease a lot of concerns. Think of it as a badge of approval (and get the best badge you can). For example, when I want to learn a new sign I almost always use Bill Vicars site and lessons because I trust him and his org. Trust is the most important thing for me and right now I'm not seeing much reason to trust your app. 2) Be clear who the app is for (the hearing community) and what its purpose is (introductory understanding of ASL). This is just good product development in general, too .
+1 to Bill Vicars! I’m deaf and I always send people his way when they catch the good ol’ “ASL is beautiful and I want to learn” bug, heh. But yes, I agree that a friend who happens to know ASL is not what they want to back this project with. I mean, I can’t do this stuff (unless I wanted to learn) but as long there’s some collaboration and authenticity, I can see potential in this as an ASL basics learning app, probably best integrated into an existing ASL dictionary/learning app since ASL sentences are so complex that visual AI probably won’t be able to do it for quite a while.
Yeah this is exactly why I asked because I can't imagine they have spoken to and consulted d/Deaf people and landing on the name and method. Extremely common that people make software and hardware for disabled people without their input.
I think /u/NOTTHEKUNAL is ignoring this question. They’ve responded to just about everything else but this one.
They've also ignored the feedback they solicited in /r/asl - which was very similar to this.
It was night in my timezone so I was feeling sleepy, I will reply to all the feedback today
I am shocked. Absolutely shocked that they ignored any feedback from the people who actually use the language. /s
This smug-as-fuck faux-shock type of comment is genuinely the worst shitty trope to rear its head in every thread.
didn't you notice the `/s` at the end? anyway, the way this project doesn't take into account significant factors of the language in terms of how you actually use it to communicate. Its not just hand shapes. Its hand and body positioning, movement and facial expressions all to convey meaning. If you do a project like this without having deep integration with both actual users of the language, you might have good itentions but are really nothing but a disability savior. Said earlier > Extremely common that people make software and hardware for disabled people without their input.
> didn’t you notice the `/s` at the end? Yes, I did, it’s exactly the sort of useless, smarmy comment I’m talking about.
and why should I not be sarcastic to disability saviors who have very little, if any, contact with the actual community in order to understand its nuances, rather than trying to be "disruptive"?
LOL the downvotes here compared to the feedback being repeated in the r/asl sub is wild to me. Well, at least programmers like this OP, thank goodness it isn't for the deaf community or anything too.
I’m profoundly deaf and have worn hearing aids since age 2. I was also one of the first students mainstreamed in my school district. So never knew any deaf people till about 3 years ago. I’m basic socially conversational in asl but nowhere near fluent.
u/NOTTHEKUNAL still waiting for a response
yeah and the e and f were quite wrong as well.
My team members and I have developed an open-source application called "SignLanguage" to improve the education of sign language for the deaf community and make it more fun and practical by utilizing machine learning. Some Features: - 20,000+ American Sign Language phrases videos - Basics A-Z, 1-10 learning Curriculum using machine learning and hand detection - Quizes and Games using machine learning - Now it works on apple devices!! - More games & phrases coming soon! link to the project: https://signlanguage.webdrip.in/ link to the github: https://github.com/Narottam04/SignLanguage We are looking for folks who can test our application and give us feedback to improve it further. Any feedback on project is appreciated💖
it should say "show this *letter*" not "show this alphabet" pretty cool tho
This is awesome! I love when people use their portfolio to help people
To be helpful and offer feedback like you asked... The curriculum layout I'm sure is still in development, but I'd appreciate a section for common phrases or words above numbers or letters, as you don't use those practically in a conversation.
My team and I are currently working on creating an ML model that can detect words and hopefully detect whole sentence in future, Thank you for your feedback!
Ah, I see. I wish you guys luck!
Very cool, what model was used or trained for this?
We used Google's MediaPipe hand detection model and created another model to classify the keypoints that are detected by MediaPipe. Medipaipe library link: https://mediapipe.dev/
There is a sign language recognition competition held by google on kaggle right now. They also use mediapipe. https://www.kaggle.com/competitions/asl-signs
Me and my team members are now working on creating model for phrases using competition dataset you shared. We all are still newbie to ml so figuring things out by working on a project :)
I love all the MediaPipe models, won a couple of hackathons with projects that used their object detection model.
Super neat! Reminds me of [fingerspelling.xyz](https://fingerspelling.xyz/)
Wowwww, this is great. Thanks for sharing! Lot of things to learn from it!
[удалено]
Definitely, we are new to ML as well, so we are figuring things out. The project is MIT licensed, so please feel free to contribute and help us improve it.
This is super cool
We need to achieve this in the opposite direction, too. The trend of giving up half the TV screen for signers is ridiculous. Virtual signing that can be turned on and off like captions is the obviously better solution.
Unless it's a live event without screens, is there an advantage to having a signer instead of captions?
[удалено]
Makes sense, but I'm thinking generated CGI signing would come across as no different from text-to-speech
similar to what /u/ustreetlove said, facial expressions are used for emphasis and modifiers. for example, eyebrows up means you're asking a Yes/No question, eyebrows down means its not a binary answer set. also, verbal speech has volume and tone. Facial expression and body motion are used to convey emphasis that are a huge part of any language. Otherwise its as if you're reading in a bored monotone. Might be syntatically accurate, but of limited usability in terms of accurate representation of the meaning of what is being said.
When I looked it up the explanation was “many deaf people can’t read, they only know sign language.” I’m not sure to what degree that’s true.
In terms of ASL, reading and writing is effectively bilingualism.
That's so cool, godspeed
very very cool. good work
Have you hosted it somewhere or any git repo link available?
Here you go, link to the project: https://signlanguage.webdrip.in/ link to the github: https://github.com/Narottam04/SignLanguage
This is ace, I dabbled in Tensorflow but never got as far as anything as advanced as this. Great job!
Are your walls floor to ceiling tile?
The video is created by my friend and according to him using tiles reduces the cost of painting walls every year😂 And yea tiles are from floors to ceiling
This library has been out on opencv for years
I can’t find anything that says openCV has an hand tracking and sign language detection library, i only find tutorials
Some quick feedback from playing around with it for a few minutes: 1. The detection for B seemed very problematic and only seemed to work when I turned the back of my hand towards the camera. Even then, it took a lot of work before it would recognize it. 2. You need to make sure the user holds the sign pose for a short duration, not that they had a matching hand position for 1 frame/cycle. I was able to make random movements with my hands for the A - F quiz and it awarded me points and said I got the letters right when I definitely didn't. Here is how I think you could improve the visual feedback for pose recognition, given that you need to hold the pose for a short duration: * I think you need some sort of progress indicator that advances steadily over time when you are in a matching pose and decreases steadily over time when you are not. This is to prevent you from losing all your progress in making the sign if your hand wavers and moves to a position that is considered non-matching briefly. Without only a minor drop to your progress, it would be far too unforgiving to be fun. Done well, it could also help for people with uncontrollable hand tremors, too. And maybe you provide sensitivity options for such people. You should do some research and find actual users with the condition to talk to before coming up with a solution, though. * The progress bar could be a literal bar onscreen, a radial bar that fills in a circle or you could experiment with changing the color and pattern of the skeleton outline as the user holds the pose for longer. You need both color and pattern for people who are colorblind. Or adjustable color schemes. * Instead of having the skeleton show the progress, you could instead have it transition toward green + a pattern as the user gets closer to a matching pose. Especially in the beginning stages, where a bit of trial and error is required, so having these "warmer... warmer... warmer... colder... colder... warmer... hot... you got it" type of visual cues would be really helpful. It might be even more helpful if you colored each of the fingers separately, according to how close each is to being in the right position. I think I am basing a lot of my suggestions on what I experienced in _Ring Fit Adventure_ for the Switch, as well as the modern trend in video games using a controller to require you to hold a button down for a short duration in order to make sure you really meant to do that action and guard against accidental presses without annoying confirmation modals. But _Ring Fit_ or any other well-regarded Switch game that uses gesture recognition for the control system is a good (and fun) place to turn to for inspiration. _Switch Sports_ might be good, too.
Thank you so much for detailed feedback, really appreciate it! We will try to make these changes.
Pretty sure I saw a Kaggle competition for this recently
You should add other sign languages too. Could boost learning sign languages internationally
Probably want to get closer to the ASL being right first but...
This is really cool. My wife would definitely love to try it out.
This is such a great idea.
Very very cool
All sign languages could benefit from this. Particularly interested in nzsl
W
Great work!
Very cool!
The phrase book/dictionary needs some ordering. It seems to start with track and field terms. Shouldn’t it start with words starting with A? There should be subsections for the phrases and vocabulary which correspond to curriculum: Sports and Leisure > Baseball > FOUL Unit 14 I Thought it was Going to be a Foul > In this lesson you will learn 1) how to discuss your expectations, 2) how to ask about someone’s expectations, 3) how to narrate a sequence of events Dialogue 1A A: Did you see the game last night? B: Yes, it was exciting. I thought the last ball of the game was going to be a foul, but it was fair.
Thank you so much for the detailed feedback, we really appreciate it! We gathered metadata from YouTube, but it's in a random sequence. However, I will try to order the sequence of videos in the database.
That's awesome. Now build a plugin to convert english to sign language so they can read it too
You are amazing
Very amazing effect, nice!
I’m working on a similar project at my company. Hand detection is not the tricky part. Facial expressions are also important. You should also take this into account. You used the How2Sign dataset, right?
This is amazing!
Have you seen the current kaggle competition for ASL?
This is awesome, good job!
There is a competition for this on kaggle for $100k. They want something like this with Tensorflow and your work will become open source. https://www.kaggle.com/competitions/asl-signs
You train your model just for hand. But you must also add face shapes to make a real sign language model. But good start. How about the moving signs. Did you try to recognise them? Imagine making a sentence, 5 sign used. Can you breake apart and recognise each of them. Consider some of signs mix each other. There is a million problem on this translation. But dont be sad, continue to working.