Much as I love the excitement it's a lot early to be talking about what board to use for prototyping. Initially we will want to see if we can develop an effective classifier for either sounds or a preset list of words for a set of EMG data. This classifier should be developed on a full-fledged computer for ease of use, and can be ported to a microcontroller once it works. All EMG data I record I'll try to put in a CSV file with signal amplitude plotted against time. I'll make explicit the file structure whatever it is.
I'm a C++/.NET programmer so that would be my development preference, but I wouldn't be opposed to using C# or vanilla C/C++ for non-UI components depending on what others are most proficient with.
I'm still studying for my bachelor's in Software Engineering, but I'm most familiar with Java, with C++ coming in a close second. Next question, though: Do we have anyone with experience programming for this kind of data interpretation?
@TheGreyKnight storing raw EMG data isn't required for the exocortex design, except possibly in the training phase - but even then I wouldn't want to store it internally, I imagine it might make more sense to train it with a desktop computer and store the resulting HMM or Neural Net on its own internally once it gets an acceptably low error rate.
On the subject of programming languages, my general favourite approach is to write a library in C++ for the CPU-intense work and then producing a python binding using SWIG and using Python as the "glue" or the main UI. The UI itself should be mostly audio driven at first simply because it's easier to implement in hardware than an eyetap (plus pure audio is more discrete than an eyetap).
As for hardware, i've started another thread talking about inductive chargers and my idea of putting together a generic computing device for implantation using bluetooth to talk to other implants - that's something worthwhile for other misc projects so deserves separate attention.
I have a lot of experience in Lisp and Javascript/Node with handling million point datasets. I can probably set up a script to examine whatever you're looking for.
What we need is probably a hidden markov model, though it's possible a neural net using something like libfann might do the trick too.
All the literature on the subject of subvocal recognition used hidden markov models in some way (sometimes with a bit of feature extraction beforehand).
Basic thing we need is a machine learning algorithm which does the following:
Take as input the EMG data and the intended phonemes, train until it can be recognized every time.
Then take as input the EMG data alone and spit out the most likely phoneme.
It's a classic machine learning problem, the tricky parts might be segmentation of the signal (deciding when to start classifying) and making it run in realtime on a lower-powered device. Training should definitely make use of a beefier CPU on a desktop, as should initial development in general as @bciuser keeps pointing out.
<3 luvu too anyway today learned how to use the biopac EEG recording system we have, it's actually pretty sexy. Data will be recorded at 4000Hz and I will test out various gains to see what the ideal one is for recording this type of data. Here is the plan for data recording. All recording will be taken multiple (let's say 5?) times as spoken speech, silently mouthed speech, and imagined speech. Each set of five repeated sounds of the same type (silent, spoken, imagined) will be saved as a single CSV file with a descriptive name. If anyone has ideas for changes or additional things they want, please post below.
In the interest of getting this prototyped quickly, it may be worth looking at AWS' ML stuff. It's a "throw it data, tell it what you expect, it gives results" type deal. I haven't worked with it personally but the APIs and things look simple.
Also, Coursera had https://www.coursera.org/course/neuraldata running, which may not be directly related, but will probably provide a solid grounding of the type of processing you're going to need to do for this project.
So, does anyone have any advice on learning how to construct neural nets? I've been studying those on my own for about 3 months now, and they're pretty mystifying...
ElectricFeel- Basically, most any of the "windows optimized"/intel SBCs are pains in the ass to work with if that's not your forte. I feel that there are a lot of reasons for this, both on the developer and tinkerer side of the table; the thing you have to keep in mind is that they tend to also be the best hardware. Much more development resources are put into these things in comparison.
Windows optimized SBCs? What a horrendous idea, especially for this (i'm a card carrying free software foundation member - not going to put nonfree code into my own body).
@ss23 Amazon's machine learning stuff is interesting, but the trained net can't be extracted for offline use, not to mention that a neural net is not the most appropriate approach anyway - a hidden markov model is far more appropriate.
Simple backprop feed-forward neural nets are a good place to start for anyone interested in the subject.
@garethnelsonuk Can you post direct links to the abstracts of the papers discussing the Hidden Markov model for use in subvocal recognition and a brief explanation as to why it is the way to go moving forward?
As for why HMM is the way to go - a neural net might work, but HMMs are generally more suitable for streaming realtime data and are used in speech recognition for vocalised sound.
Generally speaking, a neural net or a bayesian classifier is good for lumps of data (like images or spam emails) while HMMs are better for realtime signal analysis.
I have a very silly theory that CMU Sphinx could be used if a new vocabulary and model was built treating EMG data like it was sound data. If that worked, it would be a "plug and play" solution, but somehow I doubt it will, there's probably feature extraction specialised to vocal speech in that kind of engine.
Ok, do you have any experience programming things similar to this? I have very little experience with classifiers, but can definitely learn what I need or help write any specifics you would like. I know @AlexSmith has machine learning experience, maybe he is willing to help as well. I'm going to shave *tear* this evening and plan on doing the recording between tomorrow and Tuesday.
Ironically, HMMs are something I lack experience in and i've never implemented one - I have experience with neural nets and bayes classifiers and have used both for various applications (including EEG analysis for a gaming BCI).
For that reason I plan to use an existing library but it'll still be a learning experience for me.
I might try the data out with some neural nets too and see if they outperform HMM (unlikely, but i'd be more comfortable working with neural nets).
Tell your beard that its sacrifice will not be in vain, for science!
I was doing some hunting, and found a C++ library for working with hidden markov models, so I'm going to fiddle around with it, and see if I can get it to recognize pulses and tones generated with audacity. For the purposes of testing, what waveform does EMG data most closely resemble? (Sine, square, saw), and what frequency range are we going to be dealing with?
Also, thinking ahead for the main program, would a GUI, or is a console/cmd window going to be sufficient for our purposes?
The human nervous system in general uses spikes that look like sine waves with complex harmonics.
Google EMG data and click images to see some waveforms.
As for the main program, to be honest I find it surprising you need to ask - a GUI is absolutely not needed, wanted or wise for the intended purpose, how would it even display in the implanted device?
I asked about the GUI for use on an intermediary program, between the end product and now, which likely will be running on a computer. You're absolutely right about the display though.
Recording some initial data now. I'll go for all of the numbers and words tonight, and then the consonants later if the data is usable. It's actually kind of a pain to position EMG pads on one's own neck, so I'm getting a friend to help.
Just from initial recording it looks like the both the spoken and mouthed speech create usable signals, but even with maximum gain on the amplifier I'm not sure imagined speech is going to be able to be recognizable. Going to reposition EMG pads and update with a photo. Tonight the numbers and words will be recorded, if the data is usable I'll record the phonetic sounds. If not, we'll figure out what needs to be changed.
So here is a screenshot of some EMG data collected while I was imagining the word "zero" repeatedly. The regular repeating spikes you see are my heartbeat. I don't think with these electrodes and locations that we will be able to get any usable data for imagined speech. Ideas?
I doubt you'll get imagined speech unless you stick an electrode right in the speech center of the brain. Although the brain does send out the imagined signals there's so much other noise by the time you leave the brain the bit you're hoping to find will probbaly be lost. At least I presume so.
Try a bit higher, and try if you can to put more effort (if that makes sense) into imagining the speech - it might make sense to observe the EMG data in realtime while you try.
The research i've read claimed it needs more effort than "just" thinking.
If you have access to them, active electrodes would be wise too.
Also, record a totally silent few minutes to get a baseline to compare to.
@chironex This has been done before, so we're not trying to do the impossible, what we're aiming for is a recreation of this commercial device: http://www.theaudeo.com/
Something else that comes to mind is the size of those electrode pads - i'd give it a try with smaller electrode pads and check an anatomical reference to be sure they're over the vocal chords.
Comments
On the subject of programming languages, my general favourite approach is to write a library in C++ for the CPU-intense work and then producing a python binding using SWIG and using Python as the "glue" or the main UI. The UI itself should be mostly audio driven at first simply because it's easier to implement in hardware than an eyetap (plus pure audio is more discrete than an eyetap).
As for hardware, i've started another thread talking about inductive chargers and my idea of putting together a generic computing device for implantation using bluetooth to talk to other implants - that's something worthwhile for other misc projects so deserves separate attention.
All the literature on the subject of subvocal recognition used hidden markov models in some way (sometimes with a bit of feature extraction beforehand).
Basic thing we need is a machine learning algorithm which does the following:
Take as input the EMG data and the intended phonemes, train until it can be recognized every time.
Then take as input the EMG data alone and spit out the most likely phoneme.
It's a classic machine learning problem, the tricky parts might be segmentation of the signal (deciding when to start classifying) and making it run in realtime on a lower-powered device. Training should definitely make use of a beefier CPU on a desktop, as should initial development in general as @bciuser keeps pointing out.
1. The numbers 1-10,
2. The words "up, down, forward, back, left, right
3. All phonetic sounds that can be found here: http://www.antimoon.com/how/pronunc-soundsipa.htm
5 for training, 5 for testing
@ss23 Amazon's machine learning stuff is interesting, but the trained net can't be extracted for offline use, not to mention that a neural net is not the most appropriate approach anyway - a hidden markov model is far more appropriate.
Simple backprop feed-forward neural nets are a good place to start for anyone interested in the subject.
http://www.bu.edu/nmrc/files/2011/01/c_010.pdf
As for why HMM is the way to go - a neural net might work, but HMMs are generally more suitable for streaming realtime data and are used in speech recognition for vocalised sound.
Generally speaking, a neural net or a bayesian classifier is good for lumps of data (like images or spam emails) while HMMs are better for realtime signal analysis.
I have a very silly theory that CMU Sphinx could be used if a new vocabulary and model was built treating EMG data like it was sound data. If that worked, it would be a "plug and play" solution, but somehow I doubt it will, there's probably feature extraction specialised to vocal speech in that kind of engine.
For that reason I plan to use an existing library but it'll still be a learning experience for me.
I might try the data out with some neural nets too and see if they outperform HMM (unlikely, but i'd be more comfortable working with neural nets).
Tell your beard that its sacrifice will not be in vain, for science!
Also, thinking ahead for the main program, would a GUI, or is a console/cmd window going to be sufficient for our purposes?
The research i've read claimed it needs more effort than "just" thinking.
If you have access to them, active electrodes would be wise too.
Also, record a totally silent few minutes to get a baseline to compare to.
@chironex This has been done before, so we're not trying to do the impossible, what we're aiming for is a recreation of this commercial device:
http://www.theaudeo.com/
Can we automatically filter out ECG artifacts with a couple of electrodes over the heart?