A few months ago I discovered a smart phone app, called PhotoMath, which uses your phone’s camera to evaluate typed mathematical expressions (such as those found on a homework assignment). Although, this app is undoubtedly very awesome, I was surprised to see that it doesn’t support handwritten expression evaluation. With enough users, it seems possible for such an app to harvest "training data" from each user and learn to read a large variety of handwritten mathematical operators/expressions via artificial intelligence algorithms.
To demonstrate this, I’ve built an algorithm which contains an artificial neural network (ANN) that has learned to read my handwriting from example images.
I provided the algorithm ten images containing hand written digits (0-9), a variety of letters (x, y, and m), and a variety of operators (so far, +, -, /, %, *, and =). Each character was written in my handwriting, on the chalkboard pictured above. I then provided the algorithm with some "segmentation rules" — that is, rules on how to locate/isolate individual characters for analysis — and an "answer sheet" which mapped each written character to it’s correct value.
Through an iterative learning process called backpropagation, the ANN "studied" each example character, incrementally strengthening the appropriate neural pathways necessary for recognition. Given this small dataset, it took the ANN about 120 seconds to learn each character.
Below is the first test footage taken after a couple weekends of development, in which you can see the algorithm has learned to solve basic arithmetic.