Tip of the tongue phenomenon

Feature idea - what if Calico used speech recognition to map words that were said while something was being drawn? It wouldn't have to be accurate in the least. If only a few words can be captured, it would key people in to what was being discussed. For example, after a discussion is done, you could use a lasso that would circle a set of figures, and a list of words that were mentioned while those were being drawn appeared in a transient floating box. It wouldn't necessarily be sentences, just key words. The speech recognition software could have a high error rate, even 50 percent miss, but it would still be extremely useful. It would need to be calibrated, test if the gap would need to be only while it's being drawn, or if it needs everything said 1 minute before and after. It could also be calibrated to pick out "important" words, or all words. This could be done retroactively by assigning a time stamp to all drawn objects, and a time stamp to all words recorded, then map them to each other afterwards.

What makes this useful and possible is the tip of the tongue phenomenon. People are much better at recognition than they are at recall. While reviewing a session, the words would be similar enough that they would instigate a recognition of events. A psych study was done on tip of the tongue events, and people recited items which were either close phonetically or semantically. So, even if the voice recognition software completely butchered the word, it would be close enough that it clue the reviewer to an idea that was at the tip of their tongue.

I would like to perform a study eventually using Calico and this concept. The study would involve two sessions, one session using the board, and one session a few days later that reviewed the first session to allow an incubation process for ideas. The latter test would involve questions reviewing the first drawing session, having them look at key words, and measure how many new ideas are spawned by looking at the old. Also, a search feature would send the user to elements drawn that included that key word.

I don't want to develop this feature myself, but if another person joins the team, I'd like have them work on this.

Reference 1
Reference 2
Reference 3

Multitouch up and Running!

After the hardware had a few misadventures around the world, I finally got the multitouch setup up and running. I'm using the multitouch technologies based off the guide put out by the NUI Group, and had my hardware mostly made by Harry van der Veen (I owe them a lot of thanks!)

My setup is pretty crude, but it worked for a good test run. The acrylic screen is only 15" inches, but I was pretty surprised at how close I could get the projector. My home projector only needed to be about 3 1/2 feet away.

Here's the setup I was working with.

I had the screen setup vertically, but the fingers didn't give it enough pressure for my FTIR setup. I had to press my fingers a little harder than it was comfortable, but I think gravity would make this much easier once the screen is horizontal.

The hardware responded a bit slow once I got it running. You can see a few points were missed in the picture below.
I'm using a camera that advertises 25 fps, so maybe a faster camera will give better results. Maybe if I tried some software that used point interpolation it would react better. As it was though, the recognition was pretty slow. (Btw, the hanging projection paper gives the image the ripple effect, another reason to go to a horizontal setup =D ).

Overall though, the project is looking good. The touchlib framework acts as a web server, so the multitouch is compatible with any programming language. You just need to connect to the proper ports (3000/3333), and any C++, Java, or Flash application can work with it. Technically you can interact with other computers across the network, but if the reaction time is less than a fraction of a second it'll feel wrong.

Next steps are to build a cage and look into getting a faster IR camera.

java this statement

There's a peculiar situation that I got myself into, and wanted to keep a note for myself here. When you've got an inner class, and you want to refer to the outter class using the this statement, you reach it by using ParentClass.this

Example:

public class MyClass {

public void someFunction() {
...
}

private class InnerClass {
public void function() {
MyClass.this.someFunction(); // <-- important line here!
}
}
}

Feature: Autoexpand for scraps

When I am writing something, and the mark finishes just outside the scrap I'm writing in, the scrap will gain a pimple to encompass what I just wrote.

Implementation: Collect all of the points in the last stroke, find the maximum bounds in all four points, and draw an ellipse that encompasses that region. Must be minimal, must look natural.

Electronic paper surfaces

Something odd I've just noticed, the physical consistency of my 19" AMW LCD screen is the same as paper, while my tablet PC's screen is as slick as glass. A complaint that I've heard (and share) is that tablets don't feel like paper, so it's awkward to write on them. If only there was a very thin sheet of clear paper that I could apply over the tablet PC to make it rough, I'd be much more willing to use the tablet PC for every day note use.

I've also read about the problem with consistency in several papers. I wonder what if there are any rough plastic surfaces I could attach to my tablet screen...

Question to self

Does more notes equal more creativity?
Does encouraging more sketching equate to more creativity?

Enough Free Space to Feel Creative Freedom

Just a quick thought that came to mind while reading a DENIM publication. A feeling I get when am working with a small monitor is feeling cramped. Does this space vary from person to person? Is there an optimal monitor size and resolution (x/y) that feels good?

I shove an object until it's at my peripheral, and then my main work is centered on my center of vision (fovea). Exactly how much space do I need to not feel cramped. Does the noise that exist in my peripheral vision affect my ability to give attention to the center of vision?

Note to self, update later when I have time with research on the above questions.

Update: I was reading Francois's (from Stanford) thesis, and he wrote that one group preferred not to use LiveBoard (a digital tool) because it couldn't fit all of their ideas successfully. They said a proper tool should include at least 100 different ideas.

Undo Tree, Transparent Overlay

Somethings I wanted to make a quick note on.

Undo Tree
After André's talk, two fellows asked for a demonstration and gave some really good criticism on how to better improve the use of the undo list (a guy and gal whose names I can't remember for the life of me, but Alex knows them well). They mentioned that they wished their progress was represented in a tree rather than a linear list. To them it looked like it was too much information. Alex also admitted that he didn't find himself using the undo list very much either in his own usability trials, the amount of states that he could select from was overwhelming. I've toyed with the idea of representing it in a tree before to show branching, but it wasn't until the guy with the beard made the point that people don't think linearly that it really ran with me.

Here's a concept of what I'm imagining a very *crude* implementation of the tree diagram would look like real time:
This would act like a widget, however it will eventually be represented. The idea in representation is really simple, a progression in the development will follow down a path from top to bottom. Whenever the user decides to back up and try something else, a "fork" is created in the representation. The tree contains several dots, each acts like a way point that the user can click to return to a state. The way points would be recorded automatically (I'll get to how these save points are decided later in this post). Ooo, quick thought, maybe when you click the dot, a bubble appears next to it that shows a snapshot of that state, and if you click the snapshot, it restores that state.

This is a large contrast to the linear representation of the undo history right now. In personal experience, the step by step undo list is good for regressing a few specific steps, but after that (aka 10 steps), I never looked at the undo history again. Alex suggested a more trippy approach to making an undo tree: have it so when you zoom out enough, you begin to see all of the other branches and undo history. It would be exactly like the tree I drew above, except the dots would be replaced by actual snapshots of what the state looked like. This could work, but maybe if it acted like an exposé type feature. This feature would be neat, but I'm having a hard time imagining a simple way of building this.

Salvaging the Linear Undo
Now I've also gotten suggestions about the linear undo. Adding something like landmarks to different save points. You can see these in the picture to the side, the parts that are marked "Anchor" would be the landmark or anchor points. Maybe the designer can mark these points as significant and go back to them.

Another idea is having the buckets refer to a point in the undo progression. Although, this is a really bad idea on second thought because then the link would be broken when the user begins to fork an idea. A quick idea is to allow dragging from the undo list to the buckets. Hrmm...

Another problem with the linear undo list is that it has a lot of data on it that the user doesn't care about. It needs a trimming of sorts. The most recent 10 actions will be saved precisely, but everything before the 10 actions will be pruned with the following suggestions. Here's a few ideas to trim down the amount of states necessary:

• Group consecutive marks to the same object.
• Remove all entries for translating objects, unless they "snap" objects together.
• Group together consecutive thickening and thinning of relationships.

I want to mention something about transparencies, but I'm out of steam for that right now. There's quite a bit of research about transparencies and their uses, especially in the field of architecture. I think there's a few lessons we can pull from these guys, but not in the phase we're in right now.

Past Works
Enhancing data exploration with a branching history of user operations

Meeting Notes

Had a meeting today, so I'm posting my notes here so I can freely access them.

** Meeting Notes **

Capability of encouraging exploration. Discussed topics ready at hand. (The trigger for a desired action that distracts the user is not ready at hand. For example bolding a phrase is ready at hand if you press Ctrl+B. It is not* ready at hand when you trigger it by: Tools --> Fonts --> Bold --> OK.)

Professor mentioned someone else said the greatest tool for exploration is Undo. (You go Alex! You weren't alone in your intuition for Undo)

When I take a rough sketch and formalize it for distribution, what do I lose? Should I be able to return to the crude sketch from the formal presentation?

More on my side: When I sketch ideas, I have an affinity for what I draw. The affinity becomes more important when I am extending a *formal* document. For example, when I'm appending to a formal design (formal as in pretty and easy to read), I feel more inclined to explore my own ideas.

~~ Works to Reference ~~

Mark Gross
Ellen Do
--- Drawing on the Back of an Envelope


VKB Shipman
--- Visual hypertext


Terry Winograd
--- Computer and Cognition (check out chapter on ready at hand)

Paul Dourish
--- Where the Action Is: The Foundations of Embodied Interaction


Haowei Hsieh
--- Nothing

Michael Terry
--- work on evaluating alternatives (in ACM Chi conference)

1. Schön, D. The Reflective Practitioner: How Professionals Think in Action, Basic Books, 1983, Chapter 2, pp. 21-69.

Brenda Laurels’ latest stuff on poetics? Maybe for the future.

Cognitive walkthrough for evaluation?
--- Maybe this??

Shoving objects to the side of the board

This new feature extends the metaphor of a desk. Imagine now, you're sitting at your desk. Now imagine you have several desks around you, and at any time you can swivel between them with your fancy leather padded chair (I program in high class ;). That has been the inspiration of the buckets at the bottom of the screen as they are currently implemented.

Now imagine the metaphor that as you're piecing together your design, you suddenly have a piece of it that you don't want to see right now, so you shove it off to the side of your desk. At any time, you can reach over to it, but literally it's off to the side. This is the inspiration behind the new feature.

The "off to the side" buckets will exist on the right hand side of the screen. At any time, you can highlight a group of objects, then drag them to a "bucket" on the right hand side of the screen. The selected objects will be cut from your current diagram, and you'll see a tiny little preview of it in the "off the side" bucket on the right hand side. You can drag these back on at any time to your current work space.

Here's a snippet of Alex's interpretation: *one* direction, of many, you could go with this is that you are at a desk, and in the top left corner are objects, like a pen holder, and a little ticker tape (either as icons, or as part of an actual good-looking background image) that when you mouse over, expand into concepts and undos respectively, and on the right are little drawers that are closed, like into the side of the desk which is offscreeen, so all you see are little handles and drawer-fronts, and when you mouse over them, they open and have design parts inside, and the bottom of the screen, metaphorically enough, represents pushing away from the desk and going to another one (though presumably all your tools would come with you, undos and such) so the metaphor breaks a little there.

There are a few problems, like links drawn between objects that are taken out. I can just assume all links to the object that was cut out are removed. That would be simple, I'll have to do some practical tests to see if it works.

New Component Hiding Feature

The time for me to appear mature is quickly coming. Not that I will become mature, but it's one more mask I have need to forge to succeed in my upcoming endeavors. Specifically, grad student!

More to the point, it's time to distill the ideas floating around my mind, and materialize them in to tangible arguments. To the lovely friends who have subscribed because of my crazy ramblings, you might find yourself a bit bored with my developing theories, and for that I apologize ahead of time!

Note: I use designer and user interchangeably since my intended users are software designers :)

My current thought is with regard to the project I'm currently working on: VBoard. A good friend pointed out that, as the project is evolving, it's losing the simplistic charm it originally had. The cluster of components is creeping on the user, making it seem like any other bloated tool. He makes good sense, and there is psychology behind the explanation too. The solution? Hide the components, but make them easily accessible.

Psychologically, presenting too much information at one time can limit the designer's imagination. The average human brain can hold, on average 7 or smaller thought chunks (where each chunk is a concept like a car, reasonable number, etc). [Note to self: Look in to papers that analyze the effectiveness in clustering design ideas in to chunks, possible quantitative analysis or other studies] This means, if the designer consciously occupies his mind with the design environment at any point, a chunk of memory devoted to his current task is bumped out of his mind. The environment can invade the working memory through 2 different scenarios: 1) The user must "hunt" through the program to find the trigger for their desired action, and 2) Information not immediately relevant to their current task occupies the workspace and unintentionally attracts the designer's attention.

In order to combat the first scenario, action triggers must have a logical access path, and the amount of effort required must be small. For example, if I want to make a string of text bold, I can simple highlight it and press Ctrl+B. Likewise I can high light it and click the bold button on a tool bar immediately visible in my environment. The interaction for bolding a set of text works wonderfully, I can bold a set of words without much interruption in expressing my thought. A contrasting example would be if I wanted to insert a bit of clip art: I would have to pull the menu down, search for the clip art, and finally insert it. Whatever was in my working memory at that point is obliterated. Talk about working against yourself! Of course, any action done the first time will occupy conscious memory, but as I become accustomed to pressing Ctrl+B for bolding my phrases, the component of the brain responsible for this action shifts and we can now perform the same action with more working memory. It's like a free RAM upgrade for brain since you don't need to page file a thought to complete that action! (I appologize for that crummy joke ). In the case of my project, I'm going to attempt to make all actions 1 or 2 strokes of the pen away at all times. This may not be possible once it reaches a certain amount of features, but it's a thought I'll keep in mind as the project progresses.

If the information presented to the user is not relevant for their current task, it should not capture their attention. In an extreme example, a user is attempting to think up as many words that rhyme with purple as possible (impossible I know ), but at the same time the task manager is visible on screen reporting the CPU usage. While that's a nice thing to know, the user involuntarily has one of the chunks in his working memory occupied by this information. [Note to self: email professors about references for involuntary semantic processing, where a person involuntarily interprets a written word. Specifically D'Zmura.] In my case, the designer may be working on a specific portion of an architecture, but an accidental glance at the concern list may remind him of another idea they had in their mind. This may or may not be good for design exploration, I don't have authority to say one way or another, but my intuition says this information is better hidden. On the other hand, they may be actively working with a concern, so seeing it open may be useful as well. Either way, I want this data to be immediately accessible, but not imposing. In order to accomplish this task, the concerns will be hidden, but the colored circles for those concerns will be visible. Clicking the colored circle will reveal the concern until they choose to hide it. This way the user can decide which is most comfortable.

Ooo boy, I can't wait to get to implementing features that address these concerns. Too bad I have about 7 bugs I must muck through before I can get to this :(.

Test

Blog test!