Technical Accuracy and Semantic Accuracy

It's been a few days since I posted the video of my CART demo. If you've seen it, its subject material has probably gotten fairly hazy in your mind by this point. If you haven't seen it, don't watch it yet. For the record, the audio used in this excerpt was of very good quality, a single speaker with a standard American accent and clear diction, speaking at a rate of approximately 160 words per minute. First, read this transcript, created by YouTube's Autocaptioning software.

"for implants support and you know as I haven't said anything about biology those folks didn't really need to be educated and genetics biochemistry more about it so about the to solve those problems and that's because biology as it used to be was not a science that engineers could addressed very well because in order for engineers really analyze study quantitatively develop models at the bill technologies all for the parts there's a lot of requirements on the science that really biology that that's why %um %um the actual mechanisms a function work understood yes you could see that moving your arm requires certain force in of would where certain load we really didn't know what was going on down in the proteins and cells and tissues of the muscles on the books okay but still you could decide maybe an artificial %uh to do this %um %uh in the plan you really know the molecular compliments so how the world he actually manipulate the system he didn't even know what the molecules work they're really underlying yes you couldn't really do the chemistry on the biological all %uh it's very hard to quantify you can even though the parts of the mechanisms how could you get quantitative measurements for them develop models so there's good reason why they're never really was a biological engineering until very recently while he wasn't the science that was released soon right here in dallas so there for the world biomedical engineering bailey although the deposition prompted it just talked about that that necessarily require miles per se but that's changed the good news for you folks is biology this change it's now a science that engineers had it been that to very well"

Okay. Now, based on the above, I'm sure you were able to get the very general gist of the subject the professor was talking about, but if you were a student, sitting in class at an expensive, prestigious private institute of technology, and the paragraph above was the only access you were given to what the professor was saying, how would you feel about it? What if I told you that "at the bill technologies" was actually "and to build technologies", "muscles on the books" was actually "muscles and the bones", "the world biomedical engineering bailey although the deposition prompted" was actually "the world of biomedical engineering mainly involved all these application problems", and "released soon right here in dallas" was actually "really suited for engineering analysis and engineering synthesis"?

Here's the actual transcript of the excerpt:

"Or implants and so forth. And you notice I haven't said anything about biology. Those folks didn't really need to be educated in genetics, biochemistry, molecular biology, cell biology, to solve those problems. And that's because biology, as it used to be, was not a science that engineers could address very well. Because in order for engineers to really analyze, study, quantitatively develop models, and to build technologies, alter the parts, there's a lot of requirements on the science that really biology didn't satisfy. The actual mechanisms of function weren't understood. Yes, you could see that moving your arm required a certain force, and would bear a certain load, but you really didn't know what was going on down in the proteins and cells and tissues of the muscles and the bones. Okay? But still you could design maybe an artificial bone to do this. An implant. You didn't really know the molecular components, so how in the world could you actually manipulate the system, if you didn't even know what the molecules were, that are really underlying this? Okay? You couldn't really do the chemistry on the biological molecules. It's very hard to quantify, since if you didn't even know the parts and the mechanisms, how could you get quantitative measurements for them, develop models? So there's good reason why there never really was a biological engineering until very recently, because biology wasn't a science that was really suited for engineering analysis and engineering synthesis, and so therefore the world of biomedical engineering mainly involved all these application problems that I just talked about, that didn't necessarily require biology per se. But that's changed. Okay? The good news for you folks is biology has changed. It's now a science that engineers can in fact connect to very well."

My unedited realtime CART output had one error (I wrote "that didn't really require biology" instead of "that didn't necessarily require biology"), giving me an accuracy rating of 99.67%. I would argue that it gave me a semantic accuracy rating of 100%, since "necessarily" and "really" are more or less synonyms. YouTube's autocaptions, graded only on words it got right (not penalizing it for extra wrong words added), got 213 out of 299 words correct, for an accuracy rating of 71.24%. The big question is: What's its semantic accuracy rating?

If all you had to go on was the autocaptions, how useful would you find them, and how much meaning could you extract from the 71% that was correct? Keep in mind that you wouldn't have any external guidance as to which parts were correct and which parts were erroneous. Would you rate this transcript as "worthless", "better than nothing", "pretty good", or "quite useful"? Do you feel that a 71% technical accuracy rating translates to 71% of meaning transmitted and understood? Or do you feel that the scattered and jumbled effect of the machine translation interferes with understanding more severely than the 29% error rate would suggest?

A few caveats: I'm not addressing voice writing here. Independent machine translation and operation of voice recognition software by purposefully dictating or respeaking humans are two very different things. I also recognize that YouTube's autocaptions are not as advanced as those produced by other speaker-independent VR software out there. The main point of this post is to look at the difference between technical accuracy and semantic accuracy. I'd like to do it again sometime with automated software that boasted a technical rating of 90% or more; then I think the difference between technical and semantic accuracy would come out into even starker relief. Remember, 90% accuracy means one word out of every ten is incorrect. But this is the technology I've got available at the moment, so I welcome the input of everyone reading this blog. How useful do you find transcripts like this? Can you put a percentage on it? I'm really looking forward to reading the comments, so I hope a lot of people weigh in.

Exciting new graphic!

Check out the new sidebar graphic! It's a piece I commissioned from brilliant Deaf cartoonist Adrean Clark of TerpToons, whose skill, professionalism, and dizzying speed of creation completely blew me away. Sadly, I don't actually own a set of plate armor, and I have the sneaking suspicion it'd be sort of tricky to provide CART in, but now I have a picture to show how awesome it would look if I did. Click the picture for the full detailed view, and if you ever need a design, cartoon, or logo for your business or publication, definitely drop Ms. Clark a line. She's just fantastic.

Demo page

The StenoKnight demo page is live! I go through the various display options I offer my clients -- laptop, tablet, ambulatory, remote, and projected -- with pictures and even this brand new video I recorded yesterday:



Sadly I wasn't able to show off my awesome steno editing tricks (I've got a series of macros, including an entire mixed case editing alphabet, that lets me navigate to errors, delete them, and then write in the correct word, all from my steno machine. I can also select and define entries, pluralize words, and insert punctuation, all without lifting my fingers from the steno keyboard. It's pretty cool, if I do say so myself, and incredibly useful. This is what a summer of boring transcription work will inspire a person to get good at!) because the screen capture software I use, Microsoft Expression Encoder, slows down my computer too much to let me move the cursor around in time to make my split-second edits. One of these days I'll look around for a less resource-hungry screen capture program, and then maybe I'll make a steno editing tutorial screencast.

All that aside, though, I think the video is a good demonstration of the service my CART clients receive in the classroom. Special props to the amazing MIT Open Course Ware Project, which is where I got the audio. It's an inspiring thing, and if my schedule ever gets slow, I'm planning to take some of their free online courses. I'm always thrilled whenever one of my clients takes a science class, because it's a subject I love but which I've never gotten to study as much as I'd like to. I took three years of science in college, but it was of a very specific and peculiar sort, which focused on the history of scientific thought rather than actual modern science.

I've got a few extra photos that I wasn't able to use on the demo page, since it was intended to be just a basic overview for clients. Pretty soon I'll put together a blog post that gets into the geekier details of what I keep in my bag, the big sticker on the back of my laptop and why I put it there, and possibly some more pictures from other public events I've CARTed. This week's going to be busy, though -- on top of my regularly scheduled 28 hours of weekly CART work, I've got a three-hour per diem job on Tuesday, an extra two-hour CART job on Thursday, and I'm previewing a play Thursday night (whose script I have to prep some time before then) to caption on Sunday. Phew! So I might wait until the next week to get started on the equipment geek-out post.