Technical Accuracy and Semantic Accuracy

It's been a few days since I posted the video of my CART demo. If you've seen it, its subject material has probably gotten fairly hazy in your mind by this point. If you haven't seen it, don't watch it yet. For the record, the audio used in this excerpt was of very good quality, a single speaker with a standard American accent and clear diction, speaking at a rate of approximately 160 words per minute. First, read this transcript, created by YouTube's Autocaptioning software.

"for implants support and you know as I haven't said anything about biology those folks didn't really need to be educated and genetics biochemistry more about it so about the to solve those problems and that's because biology as it used to be was not a science that engineers could addressed very well because in order for engineers really analyze study quantitatively develop models at the bill technologies all for the parts there's a lot of requirements on the science that really biology that that's why %um %um the actual mechanisms a function work understood yes you could see that moving your arm requires certain force in of would where certain load we really didn't know what was going on down in the proteins and cells and tissues of the muscles on the books okay but still you could decide maybe an artificial %uh to do this %um %uh in the plan you really know the molecular compliments so how the world he actually manipulate the system he didn't even know what the molecules work they're really underlying yes you couldn't really do the chemistry on the biological all %uh it's very hard to quantify you can even though the parts of the mechanisms how could you get quantitative measurements for them develop models so there's good reason why they're never really was a biological engineering until very recently while he wasn't the science that was released soon right here in dallas so there for the world biomedical engineering bailey although the deposition prompted it just talked about that that necessarily require miles per se but that's changed the good news for you folks is biology this change it's now a science that engineers had it been that to very well"

Okay. Now, based on the above, I'm sure you were able to get the very general gist of the subject the professor was talking about, but if you were a student, sitting in class at an expensive, prestigious private institute of technology, and the paragraph above was the only access you were given to what the professor was saying, how would you feel about it? What if I told you that "at the bill technologies" was actually "and to build technologies", "muscles on the books" was actually "muscles and the bones", "the world biomedical engineering bailey although the deposition prompted" was actually "the world of biomedical engineering mainly involved all these application problems", and "released soon right here in dallas" was actually "really suited for engineering analysis and engineering synthesis"?

Here's the actual transcript of the excerpt:

"Or implants and so forth. And you notice I haven't said anything about biology. Those folks didn't really need to be educated in genetics, biochemistry, molecular biology, cell biology, to solve those problems. And that's because biology, as it used to be, was not a science that engineers could address very well. Because in order for engineers to really analyze, study, quantitatively develop models, and to build technologies, alter the parts, there's a lot of requirements on the science that really biology didn't satisfy. The actual mechanisms of function weren't understood. Yes, you could see that moving your arm required a certain force, and would bear a certain load, but you really didn't know what was going on down in the proteins and cells and tissues of the muscles and the bones. Okay? But still you could design maybe an artificial bone to do this. An implant. You didn't really know the molecular components, so how in the world could you actually manipulate the system, if you didn't even know what the molecules were, that are really underlying this? Okay? You couldn't really do the chemistry on the biological molecules. It's very hard to quantify, since if you didn't even know the parts and the mechanisms, how could you get quantitative measurements for them, develop models? So there's good reason why there never really was a biological engineering until very recently, because biology wasn't a science that was really suited for engineering analysis and engineering synthesis, and so therefore the world of biomedical engineering mainly involved all these application problems that I just talked about, that didn't necessarily require biology per se. But that's changed. Okay? The good news for you folks is biology has changed. It's now a science that engineers can in fact connect to very well."

My unedited realtime CART output had one error (I wrote "that didn't really require biology" instead of "that didn't necessarily require biology"), giving me an accuracy rating of 99.67%. I would argue that it gave me a semantic accuracy rating of 100%, since "necessarily" and "really" are more or less synonyms. YouTube's autocaptions, graded only on words it got right (not penalizing it for extra wrong words added), got 213 out of 299 words correct, for an accuracy rating of 71.24%. The big question is: What's its semantic accuracy rating?

If all you had to go on was the autocaptions, how useful would you find them, and how much meaning could you extract from the 71% that was correct? Keep in mind that you wouldn't have any external guidance as to which parts were correct and which parts were erroneous. Would you rate this transcript as "worthless", "better than nothing", "pretty good", or "quite useful"? Do you feel that a 71% technical accuracy rating translates to 71% of meaning transmitted and understood? Or do you feel that the scattered and jumbled effect of the machine translation interferes with understanding more severely than the 29% error rate would suggest?

A few caveats: I'm not addressing voice writing here. Independent machine translation and operation of voice recognition software by purposefully dictating or respeaking humans are two very different things. I also recognize that YouTube's autocaptions are not as advanced as those produced by other speaker-independent VR software out there. The main point of this post is to look at the difference between technical accuracy and semantic accuracy. I'd like to do it again sometime with automated software that boasted a technical rating of 90% or more; then I think the difference between technical and semantic accuracy would come out into even starker relief. Remember, 90% accuracy means one word out of every ten is incorrect. But this is the technology I've got available at the moment, so I welcome the input of everyone reading this blog. How useful do you find transcripts like this? Can you put a percentage on it? I'm really looking forward to reading the comments, so I hope a lot of people weigh in.

Exciting new graphic!

Check out the new sidebar graphic! It's a piece I commissioned from brilliant Deaf cartoonist Adrean Clark of TerpToons, whose skill, professionalism, and dizzying speed of creation completely blew me away. Sadly, I don't actually own a set of plate armor, and I have the sneaking suspicion it'd be sort of tricky to provide CART in, but now I have a picture to show how awesome it would look if I did. Click the picture for the full detailed view, and if you ever need a design, cartoon, or logo for your business or publication, definitely drop Ms. Clark a line. She's just fantastic.

Demo page

The StenoKnight demo page is live! I go through the various display options I offer my clients -- laptop, tablet, ambulatory, remote, and projected -- with pictures and even this brand new video I recorded yesterday:



Sadly I wasn't able to show off my awesome steno editing tricks (I've got a series of macros, including an entire mixed case editing alphabet, that lets me navigate to errors, delete them, and then write in the correct word, all from my steno machine. I can also select and define entries, pluralize words, and insert punctuation, all without lifting my fingers from the steno keyboard. It's pretty cool, if I do say so myself, and incredibly useful. This is what a summer of boring transcription work will inspire a person to get good at!) because the screen capture software I use, Microsoft Expression Encoder, slows down my computer too much to let me move the cursor around in time to make my split-second edits. One of these days I'll look around for a less resource-hungry screen capture program, and then maybe I'll make a steno editing tutorial screencast.

All that aside, though, I think the video is a good demonstration of the service my CART clients receive in the classroom. Special props to the amazing MIT Open Course Ware Project, which is where I got the audio. It's an inspiring thing, and if my schedule ever gets slow, I'm planning to take some of their free online courses. I'm always thrilled whenever one of my clients takes a science class, because it's a subject I love but which I've never gotten to study as much as I'd like to. I took three years of science in college, but it was of a very specific and peculiar sort, which focused on the history of scientific thought rather than actual modern science.

I've got a few extra photos that I wasn't able to use on the demo page, since it was intended to be just a basic overview for clients. Pretty soon I'll put together a blog post that gets into the geekier details of what I keep in my bag, the big sticker on the back of my laptop and why I put it there, and possibly some more pictures from other public events I've CARTed. This week's going to be busy, though -- on top of my regularly scheduled 28 hours of weekly CART work, I've got a three-hour per diem job on Tuesday, an extra two-hour CART job on Thursday, and I'm previewing a play Thursday night (whose script I have to prep some time before then) to caption on Sunday. Phew! So I might wait until the next week to get started on the equipment geek-out post.

Plover Blog Crosspost: New Steno 101 Lesson

If you're interested in steno but you don't already follow The Plover Blog, now might be a good time to start. I just posted Steno 101, Lesson Two, which wraps up the anatomy of the steno keyboard, preparing the way to learn some more advanced concepts for writing every English word in as few easily memorized strokes as possible. Lesson Three should be coming along soon, so subscribe to the Plover Blog if this sort of thing spins your beanie.

Photo Session Outtake

On Friday, I had a photo session for a new page I want to put on my business website. Quite often, people ask me what I do, and when I tell them "I transcribe college classes and other events in realtime for Deaf and hard of hearing students and professionals" (my standard elevator-pitch boilerplate), they have a hard time picturing what that actually looks like. So I'm going to be putting together a demo page, complete with screencasts of my CART output, pictures of my setup both in my home office and onsite at various events (drawn from some of the same photos as the ones in the banner collage at the top of the blog), and eventually I even hope to have a full video signed in ASL with voice description and captions demonstrating the CART process from start to finish.

It'll be a while before I can get that last one together, but I'm hoping to have the preliminary demo page up and running in the next week or so. It would have been sooner, but right after the photo session, I saw that my laptop's power cord had spontaneously stopped working. Yes, three days before the semester. Always the way, isn't it? Fortunately there was time to get a new power cord shipped out with Monday morning delivery (in time for my Monday evening CART gig), but it put a crimp in my weekend website improvement plans. So until I get my laptop working again (I'm writing this on the tablet PC you see me holding above, with an external USB keyboard plugged into it), I'm only posting my favorite outtake from the session. See if you can spot the reason why it's not going up on the site. Hint: Look in the lower right hand corner.

Change of Seasons

I'm sitting outside a sandwich place on the Upper West Side, waiting for it to open so I can grab some food for the trip to Boston. It was originally supposed to be a four-day gig, and I was originally supposed to leave at 2:30 am yesterday so I'd arrive in time this morning, but the Saturday job was canceled, so instead my partner and I are able to make our way down in a leisurely fashion this afternoon. We'll spend the night in the hotel, maybe have a swim in the pool, and then I'll start the gig tomorrow fresh and rested. Even though the loss of a day's work means a bit less cash in my pocket, the logistics are much more workable this way, so on balance I'm pretty pleased.

This is always an odd time of year for CART providers. School starts the week after next, and we're usually counting our pennies at this point, because summers tend to be much slower than during the school year, and because we have to save up for our next quarterly tax payment on September 15th, which is usually after we've started our academic work, but before the checks have started to come in.

This summer's actually been much better than the previous two; I had the good fortune to CART for a student who was taking a summer class, and I was able to fill in the gaps with a good amount of onsite event work for the Library and the Disabilities Network (part of the ADA anniversary celebrations), plus some remote CART here and there, and my old summer standby, transcription.

Transcription isn't very lucrative compared to CART, but it's easy, I can do it anywhere whenever I've got some free time, and I've got a great client that's sent me one interview per day nearly every weekday this summer. It's a nice supplement to my savings and the piecework I've been able to dig up. Oh, I've also finished my fourth summer of theater captioning, which I do once or twice a month. This summer's highlight was definitely A Little Night Music with the incomparable Bernadette Peters. I previewed it twice to get all the tricky Sondheimy bits just right, so counting the show I captioned, I got to see it three times in all.

Now I'm in the midst of the pre-semester rush. My own fall schedule has been fixed for a while now (and will stay that way, with any luck), but I've been getting phone calls and emails and text messages asking if I can cover this class or that class, and sadly I've got very little space left to fill. I've been able to pass on a lot of the extra work to colleagues, including two days at the United Nations, which I had a blast covering last summer; I guess they decided to hold the conference a week later this year, which meant that it overlapped the first week of the semester. It's always a good feeling to be able to help out a friend and fellow CART provider with some extra work, but it also makes me a little wistful that there aren't more hours in the day. I'm sure I only feel that way after the comparatively mellow months of summer. Check back with me after I've settled into my fall schedule and I might not be so eager to take on additional hours. This fall I'm CARTing for four students at three different schools, in subjects ranging from Architecture to Latin, plus the weekly radio show, the daily transcription work, and the theater captioning. It's a busy life, but I love it.

Captioning That Keith Wann Show Tonight

Ever wanted to see live streaming CART from the convenience of your own computer? Every Wednesday night at 8pm EST, it's as simple as going to http://toginet.com/togicaption.html, where I'll be live captioning That Keith Wann Show: Cultural Bridges. You can also log into the chat room and kibbitz/heckle/comment on the proceedings with other caption viewers, who run the gamut from Deaf to late deafened to hard of hearing to Hearing to CODA to interpreter to you name it! Tonight's show features interviews with Peter Cook and Arlene Malinowski: Two performers, one Deaf and one CODA. It should be great, so tune in with captioning, audio, or both!

Oh, and just in case you've never heard of the legendary Keith Wann, here's a sample of his comedy act: