-
Notifications
You must be signed in to change notification settings - Fork 6
Expand file tree
/
Copy pathindex.html
More file actions
252 lines (230 loc) · 22.4 KB
/
index.html
File metadata and controls
252 lines (230 loc) · 22.4 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
<!DOCTYPE html>
<html>
<head>
<title>Electrooculogram with an Arduino and a Computer.</title>
<meta charset='utf-8'>
<link rel="stylesheet" href="index.css">
<link rel="shortcut icon" type="image/png" href="images/favicon.ico">
<script src="http://code.jquery.com/jquery-1.10.1.min.js"></script>
<script>
$(document).ready(function () {
$('.clicktoshow').click(function () {
var text = $(this).text();
if ($(this).next().is(':hidden')) {
$(this).next().slideDown('fast');
$(this).html(text + " ^ ");
} else {
$(this).next().slideUp('fast');
$(this).html(text.slice(0, text.length - 2));
}
});
});
</script>
</head>
<body>
<a href="https://github.com/pranavrc/hairyplotter"><img style="position: absolute; top: 0; right: 0; border: 0;" src="images/fork-us-on-github.png" alt="Fork us on GitHub"></a>
<h1>Tracking Eye movement with an Arduino and a Computer.</h1>
<div id="exp">A Do-It-Yourself Electrooculogram.</div><hr /><br />
<div id="parent">
<a href="http://en.wikipedia.org/wiki/Electrooculography">Electrooculography</a> is a beautiful concept that we stumbled upon during our Junior year of study, which was, in fact, so intuitively simple that we decided to whip up a little model by ourselves. So here's a Do-It-Yourself Electrooculogram we came up with. As a bonus, we wrote a little extensible quasi-framework over which you could write quasi-plugins (Cursor movement and Toy Car Navigation being ours).<br />
<hr />
<div id="list">
<ul>
<li><a href="#premise">Premise.</a></li>
<li><a href="#how-it-works">How it works.</a></li>
<li><a href="#electrodes">Electrodes on a glass frame.</a></li>
<li><a href="#circuit">Amplifying and filtering circuit.</a></li>
<li><a href="#arduino">Arduino and some pseudo-Frontend code.</a></li>
<li><a href="#code">Computer and some pseudo-Backend code.</a></li>
<li><a href="#plugins">There is no spoon - Moving cursors and toy cars.</a></li>
<li><a href="#improvements">Usability and improvements.</a></li>
</ul>
</div>
<hr />
<div id="premise">
<h3>Premise.</h3>
Straight out of <a href="http://en.wikipedia.org/wiki/Electrooculography">Wikipedia</a>: <br /><i>To measure eye movement, pairs of electrodes are typically placed either above and below the eye or to the left and right of the eye. If the eye moves from center position toward one of the two electrodes, this electrode "sees" the positive side of the retina and the opposite electrode "sees" the negative side of the retina. Consequently, a potential difference occurs between the electrodes. Assuming that the resting potential is constant, the recorded potential is a measure of the eye's position.</i><br /><br />
In ELI5 (Explain Like I'm Five) terms, we're going to stick a bunch of electrodes across our eyes, and since they generate different potentials for different positions, we should be able to get an idea of the current position of the eyes based on the potential observed across them.
</div>
<hr />
<div id="how-it-works">
<h3>How it works.</h3>
Here's a DLI5 (Drawn Like I'm Five) of the setup: <br /><br />
<a href="images/roughdiag.png"><img src="images/roughdiag.png" alt="Rough diagram"></a><br />
<ol>
<li><strong>Electrodes on a glass frame.</strong> When the eyes move, the corresponding change in potential across them is transmitted from electrodes placed across them, to a noise-reduction circuit. <i>Biological signals are goddamn noisy.</i></li>
<li><strong>Amplifying and filtering circuit.</strong> The raw signals are, yeah, noisy. It's not possible to completely dust off the signal and ready it for data extraction, but <i>something</i> can be done.</li>
<li><strong>Arduino - The Microcontroller.</strong> Arduino here acts as an interface between the EOG circuit and the Computing device. We use it to take the cleaned up signal and send the values to a Computer over USB ports.</li>
<li><strong>Computer and some code.</strong> We continuously read the USB port for the arduino-transmitted potential values and classify them based on observed values for each position of the eyes.</li>
</ol>
</div>
<hr />
<div id="electrodes">
<h3>Electrodes on a glass frame.</h3>
<a href="images/electrodes.jpg"><img src="images/electrodes.jpg" alt="Electrodes in a glass frame"></a><br />
This step is pretty straight-forward. We use a pair of glasses as a frame to wind the electrodes such that when worn, the electrodes make contact with our <a href="http://en.wikipedia.org/wiki/Temple_%28anatomy%29">temples</a>. There are two electrodes (Right/Left or Up/Down), along with a Ground Electrode placed on the forehead to isolate noise and for electrical reference. <a href="https://www.google.co.in/search?q=medical+electrodes">Medical electrodes</a> are a good choice.<br /><br />
<a href="https://www.google.co.in/search?q=electrooculography&tbm=isch">Here</a> are some snazzy images of how it looks like when worn.
</div>
<hr />
<div id="circuit">
<h3>Amplifying and filtering circuit.</h3>
The signals received from the electrodes are <a href="http://en.wikipedia.org/wiki/Noise_%28electronics%29">noisy</a>, so we need an amplification and filtering circuit to get rid of the noise. We're only interested in the frequency range of the usable signal that lies in the narrow bandwidth of around 1 Hz to 35 Hz. <i>Everything else be damned.</i><br /><br />
As for the Amplification, that's required because we're dealing with skin surface difference potentials in the range of microvolts. Our Arduino, meanwhile, has an input reading range of 0-5 V and it needs to comprehend what's going on.<br /><br />
Here's a condensed form of the circuit:<br />
<a href="images/initial_ckt.png"><img src="images/initial_ckt.png" alt="Initial Circuit"></a><br /><br />
Here are the components used:<br />
<span class="clicktoshow">Instrumentation Amplifier, INA118.</span>
<div class="hide">
Using the <a href="http://www.ti.com/product/ina118">INA118</a> Instrumentation Amplifier, we'll amp up the Gain all the way to 500. For this chip, we have the Gain formula <i>Gain = 1 + (50kΩ/Gain Resistance Rg)</i>.<br /><br />
That should give us the value of Rg as 100Ω.<br /><br />
<a href="images/instrumentation_amplifier.png"><img src="images/instrumentation_amplifier.png" alt="Instrumentation Amplifier"></a>
</div><br />
<span class="clicktoshow">Notch Filter, LM358.</span>
<div class="hide">
We use an <a href="http://www.ti.com/product/lm358">LM358</a> OP-AMP with a cut-off frequency of 50 Hz and a Gain of 1.<br /><br />
Fo = 50 Hz, -> C1 = C2 = 0.1μF, R1 = R2 = 27kΩ, C3 = 0.2μF and R3 = 62kΩ.<br /><br />
<a href="images/notch_filter.png"><img src="images/notch_filter.png" alt="Notch Filter"></a>
</div><br />
<span class="clicktoshow">Low-Pass Filter.</span>
<div class="hide">
Two cascaded first order filters with a passive circuit, making a second order filter.<br /><br />
Fc = 16 Hz, C = 0.1μF -> R = 100kΩ.<br /><br />
<a href="images/low_pass_filter.png"><img src="images/low_pass_filter.png" alt="Low Pass Filter"></a>
</div><br />
<span class="clicktoshow">High-Pass Filter.</span>
<div class="hide">
A simple differentiator circuit can be used to remove DC <i>drifts</i> or act as a high-pass filter. In conjunction with the low-pass filter, this essentially acts as a band-pass filter.<br /><br />
dv/dt = 0 -> I = 0 ∴ V = 0 ∵ V = IR -> V = 0 * R -> V = 0.<br /><br />
<a href="images/dc_drift_removal.png"><img src="images/dc_drift_removal.png" alt="DC Drift Removal"></a>
</div><br />
<span class="clicktoshow">Non-Inverting Amplifier, LM358.</span>
<div class="hide">
We use another LM358 OP-AMP as a non-inverting amplifier for secondary amplification. We'll amp up the gain to 20.<br /><br />
<a href="images/non_inverting_amplifier.png"><img src="images/non_inverting_amplifier.png" alt="Non Inverting Amplifier"></a>
</div><br />
<span class="clicktoshow">DC Offset.</span>
<div class="hide">
We'll inject DC offset using a potentiometer, in order to ensure that the input to the ADC of the Arduino doesn't stray outside of 0 V to 5 V.<br /><br />
<a href="images/dc_offset.png"><img src="images/dc_offset.png" alt="DC Offset"></a>
</div><br />
<span class="clicktoshow">Voltage Follower Circuit, LM741.</span>
<div class="hide">
Run-of-the-mill OP-AMP <a href="http://www.ti.com/product/lm741">LM741</a> to get the output voltage to <i>follow</i> the input voltage.<br /><br />
<a href="images/voltage_follower.png"><img src="images/voltage_follower.png" alt="Voltage Follower"></a>
</div><br />
<span class="clicktoshow">Final Circuit.</span>
<div class="hide">
Here's a final version of all the pandemonium that's going on:<br /><br />
<a href="images/final_eog_ckt.png"><img src="images/final_eog_ckt.png" alt="Final EOG Circuit"></a><br /><br />
Note that RE/LE and UE/DE are the Right, Left, Up and Down Electrodes respectively. The whole circuit can be condensed down to a series of stages:<br />
<ol>
<li><strong>Instrumentation Amplifier</strong> with a Gain of 101 (Rg = 500Ω).</li>
<li>
<strong>Passive high-pass Filter.</strong><br />
C = 10μF and R=100kΩ, Cut-off frequency fc = 1/(2Π(100kΩ)(10μF)) ≈ 0.155 Hz.
</li>
<li><strong>Secondary Amplification with adjustable gain</strong> (minimum gain is 51 (Rg = 1kΩ)); this stage also provides a DC Offset of 2.5V to center the output between the 0 to +5 V range required by the Arduino.</li>
<li>
<strong>Passive low-pass Filter.</strong><br />
C = 10μF and R=560Ω, Cut-off frequency fc = 1/(2Π(560Ω)(10μF)) ≈ 28.42 Hz.
</li>
</ol>
The combination of the HPF and the LPF effectively creates a band-pass filter with a pass range of 0.155 Hz to 28.42 Hz.<br />
<i>Voila!</i><br />
That's pretty close to the 1-35 Hz pass range that we were aiming for!
</div><br />
<span class="clicktoshow">Physical Realization.</span>
<div class="hide">
We initially prototyped the circuit with a <a href="http://en.wikipedia.org/wiki/Breadboard">Breadboard</a>.<br />
<a href="images/breadboard.jpg"><img src="images/breadboard.jpg" alt="Breadboard"></a><br /><br />
Then we got a <a href="http://en.wikipedia.org/wiki/Printed_circuit_board">PCB</a> made. Here's the front along with the schematic.<br />
<a href="images/schematic_front.jpg"><img src="images/schematic_front.jpg" alt="EOG Schematic Front"></a><br />
<a href="images/pcb_front.jpg"><img src="images/pcb_front.jpg" alt="PCB Front"></a><br /><br />
And the back.<br />
<a href="images/schematic_back.jpg"><img src="images/schematic_back.jpg" alt="EOG Schematic Back"></a><br />
<a href="images/pcb_back.jpg"><img src="images/pcb_back.jpg" alt="PCB Back"></a><br />
</div><br />
The output from this circuit is given to the Arduino, which has an inbuilt 10-bit <a href="http://en.wikipedia.org/wiki/Analog-to-digital_converter">ADC</a> to convert the analog signals to crunchable digital values.
</div>
<hr />
<div id="arduino">
<h3>Arduino and some pseudo-Frontend code.</h3>
Our Arduino here initializes serial communication and provides the native serial interface between the apparatus and the Computer through an USB Port. It maps circuit behavior to serial outputs of a preset pattern which can then be parsed or 'depatterned' with a general-purpose programming language, say, like Python.<br /><br />
<a href="images/arduino_uno.jpg"><img src="images/arduino_uno.jpg" alt="Arduino Uno"></a><br /><br />
We use an <a href="http://arduino.cc/en/Main/arduinoBoardUno">Arduino Uno</a> which comes with an ATmega328 Chip. 3 connectors connect our PCB with the Arduino board - one of them is connected to the <strong>GND</strong> pin, while the other two transmit the PCB output to analog pins <strong>ANALOG IN A0</strong> and <strong>ANALOG IN A1</strong>. And of course, there's a USB connection to our Computer.<br /><br />
Here's the code running on our Arduino to continuously transmit values over the connection:<br /><br />
<script src="http://gist-it.appspot.com/github/pranavrc/hairyplotter/blob/master/readandprint.ino"></script>
As you can see, we're using commas to delimit the values from the left and right electrodes so that we can parse the values back out on the backend with Python.
</div>
<hr />
<div id="code">
<h3>Computer and some pseudo-Backend code.</h3>
So now we're receiving data over a USB Serial Port from the Arduino. All that's left is to crunch the data and find out what those eyes are up to. Here's what we're gonna do:<br />
<ol>
<li><i>"Hey, look down, will ya? I need to know what kind of signals I get from your eyes when you look down."</i><br />
<strong>Calibration</strong>. Ask the user to wear the eyepiece and calibrate the backend for each eye position.
</li>
<li><i>"Hey, the signals I'm getting from your eyes now seem very similar to the signals that I got earlier when you were looking down. So you must be looking down now!"</i><br />
<strong>Classification</strong>. Use the stored dataset from Calibration as reference to do some live classification.
</li>
</ol>
We chose to use <a href="http://python.org">Python</a> for the task.<br /><br />
The <strong>Calibration</strong> part is a step-by-step process where the user is prompted to move her/his eyes to different positions to record the signals received at each position. This data is then stored as a reference for live classification. Here's the code (comments and explanation inline):<br />
<span class="clicktoshow">Show.</span>
<div class="hide">
<script src="http://gist-it.appspot.com/github/pranavrc/hairyplotter/blob/master/calib.py"></script>
</div><br />
Great, now we have <a href="http://github.com/pranavrc/hairyplotter/blob/master/datasets.p">the serialized dataset</a> for reference! On to live classification.<br /><br />
The <strong>Classification</strong> module consists of two implementations that classify live data with respect to the reference dataset we just serialized. One is a barebones approach where we calculate different distance metrics between the dataset and the live data, and take the mean of these metrics since they scale up in linear fashion. Another approach is to use <a href="http://en.wikipedia.org/wiki/Support_vector_machine" target="_blank">Support Vector Machines</a> and <a href="http://en.wikipedia.org/wiki/Linear_discriminant_analysis" target="_blank">Linear Discriminant Analysis</a> using the <a href="http://scikit-learn.org/stable/" target="_blank">scikit-learn</a> library. Here's the code (comments and explanation inline):<br />
<span class="clicktoshow">Show.</span>
<div class="hide">
<script src="http://gist-it.appspot.com/github/pranavrc/hairyplotter/blob/master/factory.py"></script>
<script src="http://gist-it.appspot.com/github/pranavrc/hairyplotter/blob/master/svm_lda.py"></script>
</div><br />
<i>Houston, we have activity!</i><br /><br />
We now have the <i>infrastructure</i> to capture ocular activity! Alright, so now we've setup the whole workflow of the Electrooculogram, enabling us to track eye movements live. On.
</div>
<hr />
<div id="plugins">
<h3>There is no spoon - Moving cursors and toy cars.</h3>
Our classifier now spouts out RIGHT! when the eyes move right. That's a bit of a stretch, but yeah, the general idea is that our computer now knows where the eyes are positioned. With that kind of power, we can move the world, but we'll be nice and move cursors and toy cars instead.<br /><br />
<strong>Cursor movement:</strong> For moving the cursor, we use <a href="https://github.com/pepijndevos/PyMouse">PyMouse</a>. We initially place the cursor at the center of the screen, and for each update on the eye position, we accelerate the cursor in that general direction. Blinking does a mouse-click.<br /><br />
We hired a test subject for validation:<br /><br />
<a href="images/mousepointer.gif"><img src="images/mousepointer.gif" alt="Cat testing cursor movement"></a><br /><br />
On a more serious note, it doesn't work as seamlessly, but some movement in the appropriate direction is observed with some (read: <i>some bad</i>) lag. Here's the code (comments inline):<br />
<span class="clicktoshow">Show.</span>
<div class="hide">
<script src="http://gist-it.appspot.com/github/pranavrc/hairyplotter/blob/master/cursor.py"></script>
</div><br /><br />
<strong>Toy car navigation:</strong> This requires an additional microcontroller that provides input to a certain wheel of the toy car based on the position of the eye.<br /><br />
<a href="images/toy_car.jpg"><img src="images/toy_car.jpg" alt="Arduino-controlled toy car"></a><br /><br />
A different Arduino is connected to a different serial port, that has some fairly straight-forward code which receives data from the computer and writes output to the respective motors, all of which are connected to different output pins. Here's the code (comments inline):<br />
<span class="clicktoshow">Show.</span>
<div class="hide">
<script src="http://gist-it.appspot.com/github/pranavrc/hairyplotter/blob/master/wheel.py"></script>
And we have the following Arduino code running on our toy car, listening to values transmitted over the serial port by our Python code, and correspondingly rotating wheels:<br /><br />
<script src="http://gist-it.appspot.com/github/pranavrc/hairyplotter/blob/master/wheel.ino"></script>
</div><br />
We also messed around with live sockets (<a href="http://www.gevent.org/">gevent</a>) to enable webpage scrolling with the eyes, but it's a mostly failed/defunct experiment that you can find <a href="https://github.com/pranavrc/hairyplotter/blob/master/extras/plug.py">here</a>.
</div>
<hr />
<div id="improvements">
<h3>Usability and improvements.</h3>
<i>"Your product is bad, and you should feel bad." - Groucho Marx</i><br />
<i>"Bullshit." - Argus Filch</i><br />
<i>"Come on baby, light my tyre." - Toy car</i><br /><br />
After all that commotion, we now have a proof-of-proof-of-concept, but it's far from usable. But that's fine, we weren't really starting up here. A lot of improvements could be made, and here are some limited ideas:<br />
<ol>
<li>To avoid clutter, we could go <a href="http://arduino.cc/en/Main/ArduinoWirelessShield">Wireless</a>.</li>
<li>The serialization takes time, so we could use an in-memory store like <a href="http://redis.io/">Redis</a> or <a href="http://memcached.org/">memcached</a> to speed up the process.</li>
<li>We could look into ultra-small, higher-res and faster ADC alternatives to the Arduino, such that everything could be built into a small compartment attached to the glass frame itself. <i>Google Glass, anyone?</i></li>
<li>We could use <a href="http://en.wikipedia.org/wiki/Assembly_language">Assembly</a> instead of Python. Sadly, we weren't feeling suicidal, so that never happened.</li>
</ol>
Making this workflow faster than Image Processing alternatives would be quite the feat, though.<br /><br />
Feel free to <a href="http://github.com/pranavrc/hairyplotter">fork us</a> on Github or mail us for queries!
</div>
</div><br />
<hr /><br />
<div id="footer">© <a href="mailto:jona.mailbox@gmail.com">Jona Frank</a> and <a href="mailto:me@onloop.net">Pranav Ravichandran</a>.</div><br />
</body>
</html>