There is rapidly growing interest in the development of intelligent interfaces in which operator state information derived from psychophysiological and/or video-based measures of the human operator is used directly to inform, interact with, or control computer-based systems. Adequate signal processing power is now available at reasonable cost to implement in near-real time a wide range of spectral, neural network, and dynamic systems algorithms for extracting information about psychological state or intent from multidimensional EEG signals, video images of the eyes and face, and other psychophysiological and/or behavioral data. Neural human-systems interface (NHSI) technology can use operator state information to give operator feedback, to control adaptive automation or to perform brain-actuated control. Aspects of cognitive state that might be monitored include alertness, attention, perception, intention, and emotion.

Following is a list of participants in the December, 1995 NIPS*95 workshop on NHSI developments. Abstracts of their presentations follow:

Scott Makeig - (UCSD/NHRC) (NHSI Overview)
Sandy Pentland - Media Lab, MIT, Cambridge MA (Video-based interaction)
Alan Gevins - EEG Systems Lab, San Francisco, CA (EEG-based cognitive monitoring)

(EEG-based alertness monitoring)
(Eye-closures and alertness)
Babak A. Taheri - Integrated Biosensing Technologies, Redwood City CA (Active-electrode technology)

Georg Dorffner - University of Vienna (Brain-actuated control)
Grant McMillan - Wright-Patterson Air Force Base OH (Brain-actuated control)
Andrew Junker (Cyberlink) - (Brain-actuated control)

Charles W. Anderson - Colorado State University, Ft Collins CO (Abstract)
Jose Principe - University of Florida, Gainesville FL (Neural-human communication)
Curtiss Padgett - UC San Diego (Video-based emotion monitoring)

Organizer:

Scott Makeig
Computational Neurobiology Laboratory
The Salk Institute
La Jolla CA 92037
email: smakeig@ucsd.edu
(858) 453-4100 x1455

Summary:

I will present the microfabrication, packaging, and testing of an active, dry, scalp EEG (electroencephalogram) electrode. The electrode consists of a silicon sensor substrate and a custom circuit substrate (2-5m CMOS technolog y) bonded together, back-to-back. A via hole technology was developed using reactive ion etching with a SF6/O2 gas mixture to make electrical contacts between the sensor and circuit substrates. The substrates and batteries (power source) are assembled and connected in a custom package for testing on bench and human subjects. The signals recorded with the active electrode (from eight subjects) in both spontaneous EEG and evoked potential tests were comparable to those obtained with standard wet electrodes.

INTRODUCTION

Electrodes that employ impedance transformation at the sensing site via active electronic devices or circuits are referred to as active electrodes. Published research on active biopotential recording electrodes began in the late 1960s and temporarily ended in the early 1970s, until the recent reports by Padmadinata [1] in 1990 and Taheri, Smith, and Knight in 1993 [2]. Ko and Hynecek [3], and Richardson [4] demonstrated that both dry and insulated active electrodes can be used to pick up electrocardiogram (ECG) signals with good signal characteristics, compared to those of wet electrodes. But dry electrodes were found to have the following disadvantages:

(1) bulky size due to additional electronics and power sources, (2) noise due to the limitations of 1970's microelectronics (3) motion artifacts due to poor skin-to-electrode contact, (4) higher cost, and (5) corrosion of the electrode material due to contact with skin [4,5].

The literature on active electrodes has focused mainly on ECG recording, with one report on electromygraphy (EMG) recording. No reports of EEG studies with active electrodes has been found. In addition, no previous work on active electrodes appears to have been done for low-level signal recording (below 100-uV), a region critical to EEG recording. The dry electrode array presented here requires no electrolyte and no skin preparation. It has fast setup and cleanup times, and has redundant sensing sites that are significantly smaller than those of conventional wet electrodes. These features make it highly attractive for long-term EEG recording, where problems with traditional wet electrodes are encountered because of drying of electrolyte paste, and for high-resolution EEG recording, where shorting of neighboring electrodes through the electrolyte paste precludes close placement of electrodes. The dry electrode is fully compatible with commercial EEG monitoring systems. The electrode array contains four capacitive sensors with local circuits. It is designed to detect EEG signals in the 10-200 uVolt range, over the frequency range of 0.5 Hz to 5 kHz.

ELECTRODE SYSTEM

The active electrode consists of an interconnected circuit substrate and sensor substrate, which are bonded together back to back, with a silver paste. The electrical contacts between the substrates are made by ultrasonic wire bonding. A side view of the bonded substrates is shown in Figure 1. The sensor substrate contains a planar array of four thin-film electrode s on one side and bonding pads on the other. The electrodes are electrically is olated from one another and the substrate by a 1.5- 5m film of silicon dioxide. Each electrode is attached to a bonding pad on the opposite side of the sensor substrate by a connecting thin film of aluminum running through oxide-coated vi a holes. The surface of each electrode is coated with silicon nitride. When the nitride surface is brought into contact with the skin, biopotentials are capacitively coupled to the electrodes. Each of the four sensing electrodes are 500 5m X 500 5m in size ( Figure 2).

The following headings will be discussed.

a) Electronics Architecture
b) SENSOR MICROFABRICATION AND PACKAGING
c) Bench Testing and Human subject testing
d) CONCLUSIONS and FUTURE WORK

Back to list of presenters.

Prof. Alex (Sandy) Pentland
Head, Perceptual Computing Section
The Media Laboratory, Massachusetts Institute of Technology
20 Ames St., Cambridge MA 02139
Email:sandy@media.mit.edu

Interactive Video Environments (IVE) are experimental testbeds that allow research aimed at making computers that understand people, and to work with them in the manner of an attentive, human-like assistant. IVE's are instrumented with cameras and microphones, and allow people to control programs, browse multimedia information, and experience shared virtual environments in real-time without wires or special goggles. While the IVE technology began within the Media Lab, it has now grown into a collaborative experiment with laboratories around the world. There are currently five IVEs, three at different labs in Boston, and one each at ATR in Japan, and BT labs in the UK. They are networked together to allow collaborative experiments. Additional IVEs are planned at research labs in Paris, New York, and Dallas.

Technical accomplishments of particular interest include:
* real-time tracking of head, hands, body, and feet, including head pose,
* real-time reading of a forty-word subset of American Sign Language,
* high-accuracy face recognition with databases of up to 3,000 people,
* free-space speech recognition by combining visual and auditory cues,
* high-accuracy recognition of facial expression,

Technical reports on the MIT portion of the IVE experiments are available from whitechapel.media.mit.edu in ~ftp/pub/tech-reports. Other information is available from http://www-white.media.mit.edu/vismod

Back to list of participants.

Eye Blink Rate as a Practical Predictor for Vigilance

Steven R. Quartz, Magnus Stensmo, Scott Makeig and Terrence J. Sejnowski
Society for Neuroscience Abstracts, Vol. 21, 1995 (in press).

ftp://www.cnl.salk.edu/pub/magnus/neuro95.ps.Z

Performance on signal detection tasks depends on a subject's state of alertness. We are developing methods to automatically assess the state of alertness of human subjects using electrooculogram (EOG) signals. Eye movements were monitored on human subjects in 28-minute sessions during which they responded to randomly-occurring targets (rate 10/min) consisting of barely audible noise-bursts. The responses were used to compute a local error rate (percent of recognized targets during the past minute) Ten subjects performed two or more experiments on different days. The vertical EOG (VEOG) signal was analyzed to extract eye closure and opening events, which were also translated into per minute rates. In an initial study only eye closure events were used. This simple measure of alertness nonetheless showed good correlation over the entire sessions to local error rate (max. -0.85) Further analysis of the VEOG showed that error rate was also correlated with amplitude and blink rate (where a blink is defined as an eye closure followed by an opening within 1 second) We trained a nonlinear neural network predictor on the first experiment session, and tested it by prediction of the local error rate from blink events on the second session, for six subjects with meaningful blink rate/performance correlations in both sessions. We were able to predict the local error rate of the subjects with an average accuracy of 0.176 +/- 0.03 s.d. root-mean-square error. This was not significantly different from the accuracy of estimates based on spectral data from two channels of the electroencephalogram (EEG) 0.156 +/- 0.05 s.d. Psychophysical experiments from other laboratories support our findings. However, our study is, so far as we know, the first in which eye blink data were used to predict as well as analyze error rates. A measure based on eye blinks, and also perhaps eye movements, could be used in a practical device to monitor the alertness of, e.g., sonar operators, truck drivers or air-traffic controllers.

Back to list of participants.

EEG-BASED CONTROL UTILIZING SELF-REGULATION OF THE STEADY-STATE VISUAL EVOKED RESPONSE (SSVER)

Grant R. McMillan
Human Engineering Division
Armstrong Laboratory
Wright-Patterson Air Force Base, OH

Current approaches to EEG-based control may be divided into two general categories: (1) recognition of the EEG patterns normally associated with specific sensory or cognitive events, movements, or utterances and the use of these patterns as control signals, and (2) the operant conditioning of specific EEG patterns and the use of these as control signals. Our approach is an example of the second category. It incorporates a visual stimulus, sinusoidally modulated in intensity, in the user's task display. The signal used for control is the magnitude of the bipolar SSVER recorded over occipital sites O1 and O2. SSVER magnitude estimation is performed by a lock-in amplifier system which is synchronized with the evoking stimulus. The task display also includes biofeedback elements which permit the user to observe their instantaneous SSVER magnitude. With training, users can develop voluntary control of this response. By employing threshold and duration criteria, reliable control of a variety of tasks can be performed.

The presentation will briefly describe four applications that have been demonstrated: (1) control of the roll-axis motion of a simple flight simulator, (2) control of knee extension using a neuromuscular stimulator, (3) control of a computer-based color matching task, and (4) control of a computer-based switch selection task. Video clips, time history plots and performance summaries will be used to illustrate the control achieved with these applications.

Although the SSVER self-regulation mechanisms are by no means identified, scalp-wide recordings show that consistent topographic patterns are associated with voluntary SSVER enhancement and suppression. In addition, monopolar analysis of the O1 and O2 signals suggests that the generation of inter-hemispheric differences in signal timing (phase) may play a role in SSVER enhancement. The presentation will briefly illustrate these findings.

Key scientific and technical challenges that remain include: (1) increasing the reliability, precision and flexibility of EEG-based control, (2) identifying optimum training strategies, (3) understanding the mechanisms of neural self-regulation, (4) investigating the feasibility of EEG-based control in multi-task, high-workload environments, and (5) evaluating the relative promise of the two general approaches to EEG-based control described above.

Back to list of participants.

Prof. Charles Anderson
Assistant Professor
Department of Computer Science
Colorado State University
Fort Collins CO 80523
Email:anderson@cs.colostate.edu

The purpose of this project is to explore the practicality of EEG recognition as a new mode of communication for severely disabled persons. A system that can identify, on-line, which of several mental tasks a person is performing would provide an alphabet with which the person can compose commands to devices like a wheel chair. Some success has been achieved in distinguishing mental tasks using EEG waves, but the limitations of current recognition schemes preclude their use in on-line systems. Most research has focussed on recognition accuracy, not on fast, real-time responses. A practical on-line system requires both high recognition accuracy and fast response.

Critical to the success of on-line EEG recognition is the selection of a representation consisting of features that are low in number and in computationally complexity, yet capture information related to the mental state of a person in a way that is invariant to time and subject. The goal of this project is to evaluate a variety of EEG signal representations of data recorded from subjects performing several mental tasks. Representations are evaluated according to how well the EEG samples can be classified as examples of the correct mental task. Representations considered to-date include the raw signals, Karhunen-Lo\`eve and Fourier Transforms, and scalar and multivariate AR (autoregressive) models.

Classification is performed with a neural network trained via standard error backpropagation. Cross-validation is used to control for over-fitting. Experiments show that the AR and Fourier representations produce the most reliable discrimination between a pair of tasks. Our latest results are the following. For three of the subjects, 80-85% of the data are correctly classified, while for a fourth subject only 70% is correctly classified. Currently we are investigating this difference and considering other representations and additional tasks and subjects.

Publications can be found on-line at http://www.cs.colostate.edu/~anderson

Back to list of participants.

Recognizing mental processes in EEG using neural networks- Brain-Computer Interface research and the ANNDEE project

Gert Pfurtscheller, Martin Pregenzer
Ludwig Boltzmann Institute of Medical Informatics and Neuroinformatics,
Graz, Austria

Arthur Flexer, Georg Dorffner
Austrian Research Institute for Artificial Intelligence
Vienna, Austria

In this talk, work from the European Biomed-1 project ANNDEE on recognizing mental processes in the EEG is reported. Examples are the identification of different spatial imagination tasks in coherence maps, and the identification of motor commands and plans to be used in immediate brain-computer interfaces. The talk will mainly focus on the latter.

The general idea of an EEG-based Brain Computer Interface (BCI) system is to provide a direct link between the human brain and a machine without the detour through motor output. Especially for handicapped people, whose motor abilities are restricted or sometimes totally lost, such a system could be of great help. Various different groups around the world have recently begun to explore the possibilities to develop such BCI systems. Within the ANNDEE project two partners are involved in BCI research: a group around Steven Roberts at Imperial College, London, UK and the group around Prof. Pfurtscheller in Graz, Austria, which is closely working together with a third group in Albany, USA. For about 5 years, the Graz group has been exploring the possibility to separate different EEG patterns observed during preparation of movement. It has been shown that planning of movement generates characteristic signals over motor- and related areas and that these patterns are similar, independent of whether the movement is then actually performed or not. Through planning of movement a patient could, therefore, generate control signals which could then be used to control, for example, a wheel chair or an artificial limb.

The Graz group investigates ways to realize such a system. In a very simple experiment a subject has to move a cursor on a computer screen. A cursor and a target are presented on a the screen and different types of movement (hands, foot, tongue) control the cursor. During the first sessions the subject performs the movement, in later sessions the subject just imagines the movement. EEGs are recorded at up to 56 electrodes with a sampling rate of 128Hz. The signals are digitally filtered and compressed before they are classified by a neural network. The number of possible features for classification is very high (56 electrode positions, different time windows before movement onset, different frequency bands). One major problem is to find the optimal features and feature combinations. Another difficulty are inter-subject and inter-session variations. At the moment the classification rate is between 70% and 90% for a two class problem (only left and right hand) and between 50% and 80% when three patterns have to be distinguished. Even though these results are not yet sufficient for a practical application to help the handicapped, they are very interesting especially since it is known that the signal quality can be dramatically increased with implanted electrodes. It can be speculated that the single trial classification rate and the number of patterns which can be distinguished could be dramatically increased with sub-cortically recorded signals. The Graz group will attempt to investigate such prospects as well as possibilities to improve the signal processing, feature selection and classification methods for signals on available and new data sets from scalp electrodes.

ANNDEE is sponsored by the EU-commission, DG XII, and the Austrian Federal Ministry of Science, Research, and The Arts

Back to list of participants.

Estimating Alertness from the EEG Power Spectrum

Tzyy-Ping Jung, Ph.D.
Computational Neurobiology Lab
The Salk Institute

Drs. Makeig, Sejnowski and I have been working on an alertness monitoring research project for ONR during the past few years. The goal of project is to develop a practical and portable prototype of an EEG- and eye-movement based alertness monitoring/management system which estimates, continuously and in near real time, shifts in haman's level of alertness.

Real-time monitoring of alertness is highly desirable in a variety of operational environments where operators must sustain readiness during periods of low or repetetive task demands. Many studies of vigilance during the past half century have shown that maintaining a constant level of alertness is difficult or impossible. Alertness deficits can have serious consequences in occupations ranging from air traffic control to monitoring of nuclear power plants.

In the past few years, we have demonstrated a scientific basis for using EEG signals to non-invasively monitor human alertness (Makeig and Inlow, 1993; Makeig and Jung, 1995). We demonstrate here that changes in the electroencephalographic (EEG) power spectrum accompany these fluctuations in the level of alertness, as assessed by measuring simultaneous changes in EEG and performance on an auditory monitoring task. By combining power spectrum estimation, principal component analysis and artificial neural networks, we show that continuous, accurate, and near real-time estimation of an operator's global level of alertness is feasible using EEG measures from as few as two central scalp sites. This demonstration could lead to a practical system for monitoring of human operators in attention-critical settings.

References

Back to list of participants.

PROBLEMS AND PROSPECTS FOR DERIVING A METRIC OF MENTAL WORKLOAD FROM EEG MEASURES

Alan Gevins, Harrison Leong and Michael Smith

SAM Technology & EEG Systems Laboratory
One Rincon Center, 101 Spear St. #203
San Francisco, CA 94105.
email: alan@eeg.com

In operational environments that involve multiple tasks competing for limited attentional resources, continuous monitoring of the mental workload of a user could provide critical information for adaptively modulating a human-machine interface. There is widespread discussion and appreciation of the potential utility of employing neurophysiological measurements to derive accurate and unobtrusive assessments of mental state, and rapid progress is being made towards overcoming the problems and technical barriers that have prevented transition of such measures to practical applications. For example, good progress is being made in the engineering of recording systems that are small, rugged, portable, and easy-to-use, and thus suitable for deployment in operational environments. Progress is also being made in the development of signal processing algorithms for eliminating recording artifacts, and for increasing the amount of useful information that can be derived from brain signals. Further, results from basic research studies suggest that accurate and reliable inferences about the mental load of an individual can be derived from neurophysiological measures in a practical fashion using either ongoing EEG, probe transient evoked potentials or steady-state evoked potentials. For example, in this presentation we will describe the results of a study in which workload-sensitive features of the ongoing EEG were identified and extracted and in which effective automated means for classifying the features according to task difficulty were developed and tested.

An experiment was designed to systematically vary mental workload (in terms of increasing working memory load) while holding stimulus- and response-related factors constant. Performance and electrophysiological data were collected from eight subjects while they performed tasks that taxed spatial and verbal working memory, functions which are critical for performance of most complex tasks. When neural-network pattern recognition classification functions were applied to automatically discriminate high and low workload levels, over 95% accuracy was obtained on independent testing data. Further, networks trained on data from one day were found to reliably classify data collected on another day, networks trained on data from one task were found to be general enough to accurately classify data collected in another task, and networks formed from data collapsed over a group of subjects were found to be general enough to classify data from subjects who were not part of the training group. Together, these results strongly support the feasibility of deriving nonintrusive measurements of mental workload in real-time in the context of adaptive human-machine systems.

Research supported by the US Air Force

Back to list of participants.