Concept Present Day TalkingDesktop developers have grown up with SciFi films such as Star Wars , Star Trek , Hal, X-Files and Wall-E Our program is on the leading edge of listening and talking software. It combines traditional dictation features, windows desktop voice control and verbal responses from animated characters. Our emerging basic user awareness which hints at possible emotions is truly unique and a sign of things to come. Targeting new users and reluctant existing users. Why do the majority households not use their home computer on a regular basis ? One of the the main reasons is because they are still just too hard to use. Software complexity and having to learn difficult menu procedures with a keyboard and mouse has to be eliminated in order to grab the first-time buyer and re-ignite interest by existing users. The learning process must be fun and intuitive enough to satisfy the majority of the computer buying population that did not grow up learning a rigid command structure. Verbal conversation must replace the existing interface that so frustrates many existing owners and newcomers. The easier it is, the more people will jump on board. Talking and listening to Jessica is fun !! Does your computer talk ? Computers must become useful enough and comfortable enough for people to bring them into the kitchen or the family room. In order to do this, they must have the ability to talk, ask questions and anticipate the needs of people based on prior experience. Room awareness Your computer has no room awareness and just ignores you when you approach. A computer capable of room awareness will sense (hear) your approach and greet you Jessica does this. She will also provide relevant information obtained during your absence such as new email and stock alerts. Intelligent assistants has long been the hope and promise of technology dreamers. Freedom of movement Computer software will free a person from having to sit in front of a metal box and require them to do mouse clicks, hunt for tiny icons and navigate menu structures in order to obtain information, perform tasks or run programs. In the future, we will request information just by asking for it, no need to sit somewhere or to learn a computer program. Information will be easy and comfortable to obtain. The computer will come to us, rather than the other way around. It will be present in each room and adapt to you and its environment. You will use your voice to access the computer and the world wide web. User frustration Understanding how the human mind works is the key to designing software that is easy to use. There is alot of frustration with how difficult and annoying computers are to use. Part of the problem is that programmers who designed software, designed it for themselves. The rest of us who are not as analytical or logical were basicly ignored. The other problem is that the user interface had to accomodate the restrictive architecture of the original computers and that legacy approach has continued to this day. A visual system like Windows has improved the situation but is a far cry from being able to verbally ask the computer a question and get an answer. Talking Desktop Main Menu Concept Computers can be fun which is why people like them. They provide a sense of magic. Designing the screen with easy to read graphics and what you see is what you get text. Emphasizing the perceptual rather than the cognitive, means that people are much more comfortable and understand more easily when shapes, color and position are used to represent tasks and information. In the next generation of personal computing, everyone and everything will be connected to the Internet. Personal computers will recognize users and provide customized infomation and displays. For instance, the display for a child will be different from that of an adult. Daily routine Get ready, before long, your morning routine is going to change. Instead of turning on your computer and launching your email program, you'll simply ask your desktop computer a question: Any new email In response, your computer will display your latest messages and read them out-loud if desired. Jessica does this Conversational speech recognition has come a long way ! Future possiblies The everywhere computer Houses will be built or refitted with all the light switches, thermostat controls, telephone jacks, alarm switches on the windows and doors and maybe even the plumbing all tied together in a system monitored by cameras and sensors until the building becomes, in a sense, a living thing. Almost everyone doing any kind of work could benefit from easy access to computers, your plumber could get a lot more work done if he could see the problem at your house before he gets in his truck, and order the parts he needed as he was driving to the job. Workers already are able to telecommute, working at home and communicating with the office over the Internet. As technology improves, many of us will be freed from our cars and offices. New York, July 2009. The deadline for the reporter's next article is looming. As he hammers away at the keys, the synthetic voice of his computer interrupts. "I can tell from your pulse rate Dave, that you're upset. Why don't you take a break ?". "I'M NOT UPSET and I don't need a break," Dave says, lifting his hand off his mouse and rubbing his thumb. "I'm in a hurry, that's all. Why don't you check your other sensors?" Dave takes a tissue and wipes down his mouse. The electronic rodent is packed with sensors that, with every click, monitor his pulse, temperature and the electrical conductivity of his skin. "Your other signs, Dave, support my conclusion," chirps the voice. "But I cannot get a clear image of your face or a voice imprint, so I cannot refine my prediction.""I'm sorry that I stare down at the keyboard when I type. And I'm not going to give you a running commentary on what I'm doing just so you can monitor my voice. I think we're going to need another calibration session with the mouse once this is finished." Does the computer know how I feel Today, many people spend more time interacting with computers than with other humans. Not that computers notice: they are indifferent to our attention, oblivious to whether we love or hate them--and completely blind to our personal moods. But how much better if they knew how we felt? Take the way your computer is set up. You spend long hours tailoring the "one size fits all" interface to your needs. You get everything just as you like it when, guess what, it's time to upgrade. While it's good that people can choose how their machine is set up, the burden should not fall on them to make all the changes. Responsive self-altering computer screen The machine should customise itself to their liking the same way that your dog customises his behaviour to your whacking him with the newspaper." The computer should recognise whether or not the user likes what it is doing, and adapt its behaviour accordingly. A responsive, self-altering interface is just the beginning. A machine that is able to detect, respond sensitively to, and even transmit an emotional state offers endless new opportunities. Our emotions If you think you don't get emotional in front of a keyboard, just remember how you reacted the last time your computer crashed or a Web page wouldn't download. Researchers have monitored people using everyday applications such as spreadsheets and e-mail, and found emotional responses on average once every two minutes. By observing facial expressions and eye and hand movements, they identified responses such as frustration and boredom, happiness and when people were interested in what they were doing. "Emotions like these play a critical role in perception, decision-making, social behaviour, learning and memory. Gathering emotional clues The first essential for an affective computer is some way to read our emotional signs. Video cameras and microphones can capture expressions, gestures and intonations. But these give only part of the picture. As every poker player knows, it's possible to hide many outward signs of emotion. Another approach, then, is to search for physiological signs, such as clammy palms or a racing pulse, which are more difficult to disguise. These are the changes that the "lie detector", or polygraph, is designed to pick up. Evaluating emotional responses Researchers have designed the equivalent of a polygraph for computer users, which measures heart and breathing rates, skin conductance and muscle tension. In one set of experiments, readings were taken as an actress expressed eight emotions ranging from anger to romantic love and reverence. The machine easily picked out changes in the strength of the physiological signals--the level of arousal--but it had difficulty telling the "valence" of the actress's emotions--whether they were positive or negative. This presented a problem because emotions as different as joy and grief both pushed up the body's signals. So without an idea of valence, they looked similar. Likewise, hate and platonic love looked similar, this time with low levels of arousal. "Sorry to interrupt, Dave, but I think we have a problem." "What is it now?" "I know that your present activity causes you stress, but your signals suggest that you have exceeded your normal stress levels and are now experiencing anger." "I am not angry. My stress level is probably high, but this piece has got me quite excited." While overall levels of arousal proved less than ideal for their needs, the latest findings show, for example, that anger has a distinctive pattern of high muscle tension, increased heart rate and deep breathing, while grief leads to low skin conductivity and less rapid, shallow breathing. By homing in on such patterns, researchers have found that they can distinguish between all eight emotions with around 80 per cent accuracy. All supposing that this physiological approach can be made to work for more people and for more, "real" emotions, there is another problem that needs to be solved if it is ever to become widely used--the intrusive nature of the sensors. Researchers use one belt, strapped around their subjects' chests, to measure breathing rate, and another fitted with electromyogram sensors across the back or jaw to measure muscle tension. They then tape sensors to the fingers to monitor pulse and skin conductance. People won't be willing to don this type of gadgetry at home or work. Or will they? Wearable computers Looking ahead to the era of "wearable computers" in which smart devices will be fitted inside belts, caps, watch straps, shoes and so on. These will touch the body as a matter of course and with the correct sensors will be able to pick up our emotional signals. Work has begun on designing "affective jewellery". Prototypes include earrings that monitor pulse rate, and rings and bracelets that measure skin conductance. These baubles inform the computer of their readings via infrared transmitters. Another approach of embedding sensors in a mouse to measure heart rate, temperature, skin conductance and the pressure that people exert when pushing and clicking. Facial clues While physiological signs show promise for telling how somebody feels, researchers have not forgotten the behavioural cues that we humans rely on so much. A system that recognises facial expressions. It tracks the movement of individual pixels in a video image of a face and transfers them onto a virtual face that has a full set of functioning muscles. When a subject smiles, the model copies, and the software calculates which muscles need to expand and contract to produce that movement. From these patterns, it recognises that the subject is smiling. With expressions such as smiling or looks of surprise, anger, disgust and sadness, a recognition rate of up to 98 per cent has been achieved. Such rates make facial-imaging systems unbeatable for identifying the valence of an emotion. But these results are for a limited set of faces and expressions, filmed under ideal conditions. Testing is underway to see how well the system performs with more people, more expressions and in more natural surroundings. HAL in 2001 A Space Odyssey In Arthur C. Clarke's classic 2001: A Space Odyssey, the psychotic computer, HAL, tells the emotional state of its human charges by analysing their voice harmonics. In reality, extracting emotional information from speech is proving to be a tough task. A variety of vocal features change with emotional state. Speech rate tends to increase slightly when somebody is angry, for example, while intensity and pitch rise. By contrast, when a person is sad they tend to talk slower and at a lower pitch and intensity. The difficulty comes in designing systems that can identify such patterns and so spot emotional states. One novel approach was to train a neural network with many voices expressing different emotions. The network "learned" to identify eight emotions ranging from joy to disappointment Emotion from vocal feature To date, the best systems can identify emotion from vocal features in about 60 per cent of cases, which is about what humans can manage. Machines are good at recognising arousal but not so good on valence. Recognition rates for humans increase as soon as they can hear the content and context of the speech, and as computers learn to "understand" speech, they too should improve. A simple expletive detector, for example, should raise recognition rates. Still, the signs of some emotions, such as jealousy, are subtle and depend greatly on the individual and their culture. People also express emotions differently according to whether they're talking to children or the boss. This applies both to systems that analyse vocal and facial expressions and it makes reading their emotional content very difficult. "The context makes it more difficult to interpret than some other forms of signal processing," Given the limitations of all the systems, both Cosier and Picard believe that a combination of methods will be needed to recognise emotions reliably. That, after all, is how we do it. " Dave...I know you have a lot to get done but I am still worried." "What is it now?" "I know you said you were excited, but my model shows you are likely to enter a state of rage." "You may be right about that if you keep interrupting! But what's this model you're going on about?" "Well, Dave, from the signs I have collected, I recognise your emotional state, which looks like anger. I can then predict the chances that you will move to another emotional state. According to my model, given your present readings, your most likely next state is rage." "You know what--I could be moved to violence if I hear any more of this. But maybe you're right, I have been pushing it a bit hard. I think I'll take a break." "Good idea, Dave." Once a computer can single out emotions, it will need a model of what they are and how they relate to human behaviour. "Computers will have a tremendous amount of data about the user's state at any given time,". "Somehow, those data need to be reduced to a manageable representation of the person." Markov model In essence, an affective computer would need a multidimensional map onto which the emotions are plotted against all the sensors' readings. There are a number of ways of constructing such a model. One basic constituent might be a hidden Markov model, a mathematical model of the likelihood that one event will lead to another. Here, it would contain the probabilities that a person will move from one emotional state to another. This type of program would probably have general skills for recognising a person's emotional state from its sensor inputs, and locating their position on its map. It would also be "trainable" so that users could tailor its performance to their own temperament--in much the same way that people train speech recognition systems today. As it monitored emotional signs, the program would also be able to predict how the user is likely to feel in the near future. If such machines are ever to help people to modulate their emotions, they will need an idea of which emotions are good and which bad, and then know how to nudge somebody from, say, a state of frustration to one of calm and creativity. To do this, an affective computer would need to be able to find out if its own activity caused a mood change, perhaps by simply asking questions. ("Are you angry because I didn't back up your file?" The answer "yes" would trigger the machine to back up files in future.) Real life But, inevitably, even while we're in front of the screen, events elsewhere in our lives will intrude to affect our mood. So the model also needs a more cognitive element, which could reason about emotions, making allowances, say, when a person working to a tight deadline appeared to be more than usually stressed. And, it would know not to intervene if a person got very excited when playing a competitive computer game. "Right, I'm back. Let's get going." "OK, Dave. I'm opening your file. But wait. What's happened? Dave, you were supposed to calm down during your break but your signals are showing even higher spikes than before." "Hey, relax. I just called Monica and she agreed to go to the movies with me tonight. I've been trying to talk her into a date for weeks so it's no wonder my pulse is jumping a little. Come on, open up and let's get going." "I'm reluctant to do that while your signals are so volatile, Dave." "Look, just open the file...Oh, all right, show me my mood ball and let's get this thing sorted." 3D graphical view of cued emotions When a three-dimensional graphical ball is used to represent someone's physiological signals. The speed at which the ball spins represents heart rate, its colour represents skin conductivity, while other dimensions can represent breathing rate, muscle tension and so on. It's a quick, visual way for people to get a fix on how the computer is reading their signals--and of calibrating how those signals relate to mood. Bored user pick me up A simple form of personal interaction with an affective computer might include telling it how we prefer to deal with particular moods, so it would play a Chopin CD when we're feeling sad or load a game if we appear to be bored for more than 15 minutes. Researchs shows that even a simplistic affective system can help people to recover from negative moods. Artificial intelligence Researchers are only too well aware of how artificial intelligence has been dogged by the grandiose claims made by some researchers, and they are at pains to point out the enormous challenges still facing them. But when and if affective computers become commonplace, they could provide some spectacular services. Advertising companies could use the technology to test the impact of their campaigns, while games-makers could use it to enliven their products by, say, upping the pace if players showed signs of boredom. This could give computer-based learning programs some of the sensitivity of human teachers and be used to provide biofeedback, helping people to be aware of their emotions and to control them. To make it to market, affective computers will not have to be perfect at recognising emotions. This is particularly true of jobs where the cost of being wrong is low, such as choosing which Web pages you might want to see, based on past reactions. But if machines are ever to oppose or control human actions, they'll need to be very accurate. Can a sympathetic agent reduce user frustration ? CAN A COMPUTER turn frustration into a more positive mood, even when the computer is the cause of the frustration? To test this idea, a computer game was created in which a character hunted for treasure. Then the trappings of a Web browser to make people think they were playing the game over the Internet. In reality, the game was connected to a local computer that could mimic the Web's notorious response delays. Volunteers were invited to help test what was claimed as a prototype for a new game, and offered $100 for the person who scored the highest in a five-minutes session. After this initial bout, volunteers had to complete a questionnaire on the screen and then play the game again for at least three more minutes. Subjects were divided into three groups. The first played the game once without experiencing any bugs, completed a bland questionnaire, and then played again. The second group experienced a number of apparently random, but actually controlled, delays in the Web's response that frustrated their attempts to achieve high scores. This group completed a different questionnaire that allowed them to moan about the delays and to score how anxious, tense or angry the game made them feel. The third group also suffered delays, but as they started answering their questionnaire, a software agent intervened. Instead of letting the subjects simply vent their frustration, it tried to show that the computer understood what they were saying by paraphrasing their statements. If it didn't get it right first time, it tried again until the subjects were happy that the computer reflected how they felt, or it apologised that it was unable to capture their feelings. The agents' feedback also included statements showing empathy such as: "It sounds like you had a pretty mediocre experience. That's not much fun." The interaction with the sympathetic agent had a marked effect on subjects in the third group. They continued playing for significantly longer than subjects in either of the other two groups. The interaction appeared to make them feel more positive and interested in the game. "Computer interfaces can be designed to actively help users recover from strong, negative emotional states, especially those related to frustration." SciFi, Aeon Flux, Fantastic 4 The Talking Desktop software concept emerged from a vision of computers that was molded and excited by watching SciFi TV and movies such as Aeon Flux, The Clone Wars and Star Trek. Many ideas and concepts from television have tended to foresee upcoming technologies and trends. The Starship Enterprise featured a futuristic computer that was available everywhere such as the flat panels in corridors and monitors in every room that responded to voice requests. The computer was always on and ready to serve the crew whenever they said Computer or in our case Mary, Jessica or Companion. This computer was aware of the crew, provided information on a wide range of questions, performed tasks and suggested basic solutions.