Augmented Reality Captioning

So this December, for the third time in a row, I captioned the New York Public Library's Holiday Songbook via closed device captioning. I wrote a post about it from 2010, talking about its pros and cons, but here's an actual candid shot from the audience, taken by a caption viewer using an iPad.

It's definitely better than nothing, but as I mentioned in the original post, it has significant drawbacks, such as having to constantly look away from the stage and adjust your eyes between far vision and near vision. In performance situations like this one, open captioning is usually far preferable, since it's on the same plane as the people on stage. For the same reason, many people who use captions in movie theaters prefer open captions to the Captiview devices that sit in a viewer's cupholder. But there's a new accommodation that's starting to be used in both live theater and cinema: caption glasses.

I've heard mixed reviews from Deaf and hard of hearing patrons who've used these glasses. On the one hand, it's nice to be able to have the captions superimposed over the picture. On the other hand, they're apparently quite heavy and bulky (as this cartoon from That Deaf Guy gets across so well), which can cause neck, nose, and ear pain, since you apparently have to hold stock still or else the captions jump around all over the screen.

I've been interested in augmented reality for a long time, both in its application in CART and captioning specifically and in the possibility of being able to compose text (blog posts, novels, emails, etc.) while walking without bumping into things. It's been on a mostly theoretical level up 'til now; when I wrote my What Is Steno Good For: Mobile and Wearable Computing article, I said "It's a problem that still hasn't been solved to anyone's satisfaction, even after several decades of trying. They're too heavy, too fragile, too stupid-looking, too headache-inducing. But let's posit that someday soon the problem will be solved, and we'll be able to go out and buy lightweight, stylish augmented reality overlay monitors that look just like ordinary pairs of eyeglasses."

Well, the problem might not have been solved completely; there are a few AR glasses such as:

The Vuzix M100



and

Google Glass



Which are currently releasing prototypes to developers, and which are expected to have widespread commercial release in 2014. As you can see, they don't look just like ordinary eyeglasses, and it's unclear to what extent the eyestrain issues that seem to have been endemic in all previous AR solutions have been fixed. It will probably take a few iterations to iron out all the kinks.

Thing is, it's all gotten a lot less theoretical to me recently. I'm currently working with a first-year dental student. When he becomes a second-year dental student, he will begin working in the clinic with actual patients. Having a tablet display mounted somewhere near the chair is not a very good solution; he'll have to look back and forth between the patient and the tablet, and it will probably be both awkward and inconvenient when trying to do his job. A much better solution would be a pair of AR glasses, and I'm actively looking into my options there. I've joined the AR Glasses LinkedIn group and I've been doing a bit of research on my own, with an eye to purchasing a pair (probably as a developer, since consumer models are still at least a year away) sometime this summer. Of course, like any technology, there's that paradox where the longer you wait, the better the technology you're going to get, but the more you put it off, the less time you have to make adjustments and adaptations so that it works the way it's supposed to. I don't know whether I'll be able to make it work out of the box -- just bring up a browser window, adjust it so that it fills the lower 1/8th or so of the visual field, and have the captions displaying from StreamText without any other modifications -- or if I'll have to commission bespoke software to get my captions onto the glasses without completely obscuring my client's vision.

I'm also concerned about WiFi strength, battery life, physical comfort, and the all-important eyestrain. I wish I could just go try out both the Vuzix and the Glass head-to-head, asking questions from their manufacturers, but I suspect it'll be a bit trickier to get the information I need, and if I wind up buying one of them, it'll be at least $1,500 or so, plus any fees involved in getting customized caption-display software written. Also, dental clinics can be messy places. If the glasses get spattered with water or other less savory substances, can they be cleaned and sterilized without damaging them? Will my client's rapport with the patients be compromised by wearing this strange-looking headgear, or will it blend in with the rest of the dental equipment? There are a lot of questions still to resolve. I'll keep you all updated as I go along.