Exploring Assistive Technologies

Moving from a device that's purely "fun" to work with, to something that has real value and applicability, I've been considering applying the sound-activated cursor mechanism for use in existing assistive technology.

Severe sufferers of cerebral palsy, without voluntary limb, head or tongue movement, have a tough time communicating. Often, they are only able to control their speech using a series of grunt-like noises. The Sing-a-Sketch mechanism could feasibly be adapted for use by a sufferer of cerebral palsy, who only has the ability to control the duration of vocal noises.

By attaching a bluetooth-enabled headset to their ear, and synchronizing their noises with a cursor moving on screen, a user can navigate through a "switching" communication device driven by software, which is then able to "speak" the sentence to carers.

While the existing software for assistive technology is very comprehensive, users are still required to have significant control over particular parts of their body, with input devices designed for finger movement only, tongue and head movement being particularly common.

Even eye-movement readers are inappropriate for particular cases of cerebral palsy, as the sufferer often has limited control over the position of their head.

The portability of the Bluetooth protocol would allow for light-weight software / hardware pairing with the headset, allowing something as small as a PDA to be able to interpret vocal input into spoken or written messages by both the user and the carer. A carer could conceivably be sent SMS-like messages from the user who transmits textual translations to a web-enabled PDA.

The ever-moving cursor in Sing-A-Sketch then, would have to be adapted to continually be at the centre of the screen, with only the translation icons and commonly-used buttons scrolling underneath in the direction dictated by the user.

Once certain 'dead zones' (blue in the diagram) in the screen are overcome (allowing for both a sound and interaction threshold), the screen is able to "switch" to the next-lowest branch in the heirarchical tree of iconography, while keeping the cursor at the centre of the screen.


Once the sentence or phrase is complete, the user can navigate to the ever present "speak" or "cancel" buttons to complete the interaction. "Yes" and "No" buttons are omnipresent, and navigating up and down the tree uses a consistent interface.

The screen would "switch" between a series of four options, while also keeping the most-recently selected option in view, allowing a user to navigate both up and down the heirarchy.
Selecting 1, for example, switches to the screen surrounded by green, where the next available options are 5, 6, 7 and, to navigate back, 1.

Textual Input via Microphone

Not voice recognition (far too complex), but MORSE recognition *gasp*!

Not unlike these attempts:

• MRP Morse Software Decoder
• CWGET Morse Decoder

Perhaps this could be a means of having users input textual information to provide their signature, etc., particularly if there was an on-screen morse alphabet as a legend.

Vidoe Assessment - Evolution of an Idea

Wanting to differentiate ourselves from Rokeby's work, we observed a [possibly unintended] stencilling effect in his work Seen - where the static objects in the foreground of a scene quickly stand out once substantial movement occurs behind those objects.

Exploring this concept further, we realised that the patterns created by movement in a scene could also be used to reveal a "hidden" image in the installations projection - by making this hidden image 2-bit greyscale negative, the classic "afterimage" optical illusion could be created and gradually revealed, then hidden again, as the day (and its associated motion) progressed.

Hence, we drew further inspiration from the stencil graffiti movement, as well as the optical illusion crowd. Each motion though a given scene is noted and quickly 'fades in' behind the stencil, then changes size and/or opacity slowly over time to fade away. The secret stencil, then, may only be revealed during periods of heavy pedestrian traffic, whilst simultaneously being susceptible to the recent phenomenon known as "flash mobbing".

Video Assessment - Initial Concept

Next week feels a lot like a "placeholder" assessment piece, IMO. The challenge is in proposing a video interaction piece that DOESN'T feel like a cheap EyeToy knock-off or lame-o real-time filter effects.

My discussions with Ben (our tutor) suggest that, even in 15 years of R&D, very little new applications have been designed or developed, merely that the technology has become more affordable for the consumer (cheap web-cams, etc.).

I'm proposing to attempt something reasonably simple that doesn't just feel like an "art for art's sake" project:

[drum roll...]

>>> Temporal Pattern Analyser <<<

Much like a long-exposure photograph, this web-cam enabled installation tracks clustered movement in a public space, with each movement leaving an ever-so-slight outline (ghosting) for each second that they are in motion. These movements can be highly stylised in the camera's interpretation and representation on-screen, but should ultimately allow the viewer to analyse where in a particular scene users tend to congregate (or skip right through). Commercial application would be most viable in the retail industry, where shop-owners could use the tool to gauge where potential customers are drawn to in their shop layouts, without the fear of privacy violations.

Not entirely dissimilar to this installation:

http://homepage.mac.com/davidrokeby/seen.html

So, the theory and application of this could be reasonably easy generated, as would the context of the installation (large, open public space). I imagine that the use of the WebCamXtra for director would be able to continually write on a per-pixel basis to the screen based on movement detected by the camera.

Just a thought - I'll have to wait until I get home to have a crack at the software side of things, but I'm very open to suggestions and/or assistance.

Presentation 2

Harry does the text. Gordon does the piccies. Shane does the demo. Sweet.

Download PPT

It Works (The Way We Want)!

Awesome, awesome job. Got to borrow a Bluetooth headset off a mate, and, after the pairing process with my own generic Bluetooth dongle plugged into the PC, we were on our way.

The only thing required to allow BT-enabled audio was to set the BT-hardware as the default input device - with further tweaking, we could produce some audio feedback (transmitted direct to the headset), but for now things are very sweet - users bring their own headset, pair it to the installation, and sing along! Nice.

Feasiblility? Check!

Technical Feasibility: Lose the Mobile Phone

After discussing possibilities with several electronics, computing, communication, hardware and audio specialist dudes, I came to the conclusion that the only way to use the mic on a mobile to communicate with our Bluetooth dongle is to write a custom app that offers our desired functionality.

Instead, I believe that the use of Bluetooth-capable headset devices (which are becoming more and more prevalent) would be far more feasible. This particular device from Blueant (http://www.blueant.com.au/X3micro_Bluetooth_Headset_voice_over_ip_voip.htm) would do the trick nicely:

When connected to a computer, the X3 micro becomes the default Sound playback device so that any program which utilises the Soundcard can send and receive audio signals from to the headset.

Perfect! That sounds like us! Now to find one to experiment with...

PPT Presentation Work-In-Progress

Now available for your perusal:

http://student.ci.qut.edu.au/~n4661117/kib311/Presentation1.ppt

Feedback and comments welcome. This is a dot-point summary of most of my prior notes, with fancy additions for other users (i.e. you guys).

Presentation Images - First 3 Slides

Welcome to "Sing-a-Sketch"!

Sing-a-Sketch is a unique conglomeration of a variety of familiar toys and technologies, fused in an unconventional way and projected on an unexpected location. Sing-A-Sketch relies on everyday users' sense of play, allowing simultaneous or iterative collaborations of community- created artwork, and, by way of vocal interaction, spontaneous performance art.




Users familiar with the SingStar series of games for the PS2 may recognize the voice-activated game mechanic used here, but that's largely where the similarity ends. Children who grew up with the classic "Etch-A-Sketch" toy may also feel at home, but the privacy of their creations is no longer limited to a friend nearby to whom they can pass the sketchpad.

Graffiti artists have long used public space to make a statement, or to add to existing statements created by earlier artists. The final result never fully solidifies: the space is consistently added to, refined, roughened, or simply painted over. The space itself is as fragile and as engaging as an Etch-a-Sketch.

Mobile phone users have long had gaming facilities on their devices - some of which even have the facility to play against other users. Mobile users are also very familiar with the voice-driven nature of the phone's primary use, and will no doubt be comfortable speaking into the device.

Using their voice with their phone to control their contribution to collaborative community artwork is the key mechanism of play in this piece. Users are invited to pair their Bluetooth-enabled devices with [possibly one of several] Bluetooth audio receivers. These receivers then take the users' sound input as a single button device to control the direction and speed of the users' pens, represented and projected onto a suitably flat public surface.

Without further instruction, users could potentially play with the interface: some may feel they need to sing to the device a particular song - others may sense that they have to move around the device's broadcast radius. Still others may invite their friends to help them control their pen. Users may even unknowingly influence each others' pens if their voices carry far enough.

Similar Projects - Worth Citing for Reference

We Make Money, Not Art:
http://www.we-make-money-not-art.com/archives/cat_locative.php

Phillip Pocock (pioneer in the internet / installation artwork form):
http://en.wikipedia.org/wiki/Philip_Pocock


Node London (funky interactive artworks, using a variety of input medium)
http://nodel.org/projects.php?type=67

Public-Access Bluetooth Work:
http://www.orbit.zkm.de/?q=node/412