Blake Hannaford Dept. of Electrical Engineering University of Washington Seattle, WA 98195-2500 http://rcs.ee.washington.edu/BRL
The chapter will focus on the issues of time delay, control, and stability, with illustrations from the history of telerobotics and teleoperation. It is impossible to do justice to all of the important technologies and the innovative engineers who developed teleoperation in a short chapter, so I will present only a sample of the key ideas. Telerobotics remains an active research area pursued by engineers internationally.
Physical actions mediated by teleoperation change the state of some remote system that we will call the environment. During these physical actions, energy is exchanged between the environment and the manipulator in either direction. In some cases, teleoperation systems include force feedback so this exchange of mechanical energy can be perceived directly by the human operator. A remote control system is not a telerobot if it only permits the on/off selection of state. For example, a doorbell is remotely operated but would not be consider a telerobot. Although remote driving of vehicles can also be considered teleoperation, I will focus exclusively on remote manipulation. Distance, broadly defined, is any barrier to direct manipulation including physical distance, differences in size scale between the operator and environment, or presence of danger in the environment. A recent example of an industrial teleoperation system is shown in Figure 0.
|
Fig. 0. "Hot-Line Telerobot System" from Kyushu Electric Power
Co. This system allows a human operator to safely repair high-voltage
electrical power lines. Thirteen of these systems were deployed in the
field as of 1996. [click images to enlarge]. |
 |
 |
 |
Many teleoperator and telerobotic systems use dedicated private communcation links between human operator (master) and the remote mechanism (slave). Recently the Internet has supplied the communication link for some systems. On a technical level, except for the randomly variable time delay it imposes, the Internet functions essentially the same as any other communication link. An important social difference is that the Internet may allow public participation. If the system is so configured, anyone may participate, either as operator of the robot, or as a "lurker" who merely watches the proceedings. The emphasis of this chapter is on the technologies which support a single known user with trusted remote site.
For example consider the case of pressing on a spring. As we depress the spring, we do work on it and it stores energy. As we release it, the spring does work on us and we dissipate the resulting energy in our muscles. This type of interaction is essential to what psychologists call "haptic" or "kinesthetic" perception (Lederman & Klatzky, 1999).
Information interactions take place at a "terminal" such as the human retina, or the input to an electronic communication link and are "directed" which means that the flow of information at a single terminal is one-way. Energetic interactions take place at a "port" such as a human hand grasping a handle, a mechanical link between two machines, or a heat exchanger, which is a point of contact between two systems capable of storing, converting, or dissipating energy. An energy port is bi-directional, in principle at least, energy can flow in either direction. The energy flow is measured by the signed product of the two "conjugate variables", "effort" and "flow". The effort and flow variables most relevant to telerobotics are force / velocity, and voltage / current. The physical units of their product in all of these cases, Watts, measure the rate of energy flow - the power transmitted through the port.
In technological systems, a link capable of bi-directional energetic interaction can be simulated with two information links pointed in opposite directions. The information links send the conjugate variables back and forth to transducers which simulate the energetic interactions at each end of the link. The distal end of this link may be a telerobot or teleoperator, or may be a computer simluation. Thus, although energetic interaction gives a compelling sense of reality, it can be simulated just as an information only interaction can.
"Telerobotics" technology implies communication on a higher level of abstraction in which the human communicates goals and the slave robot synthesizes a trajectory or plan to meet that goal. Telerobotics primarily supports information interaction because of the higher level of abstraction.
In both cases, the operator accepts sensor information transmitted from the remote site to explore the remote environment, plan tasks, verify that tasks are completed, and create plans to resolve problems.
The communication link between master and slave sites has, until recently, been exclusively private. With the introduction of Internet Telerobots, as Goldberg points out in the introduction, the issue of tele-epistomology becomes more accute since we may not know or trust the creator of a telerobot. Similarly, the telerobot may not be able to trust the operator. If a telerobot is created for a specific task and is intended for operation only by a pre-defined set of trusted, authorized users, then the issue of authenticity can be addressed by standard internet security measures. If the telerobot is intended to be open to the public, the issue is more significant - especially from the operator's point of view. How does one know if the site is an authentic telerobot and not a simulation? This question however, applies to many forms of internet information, or for that matter printed information (such as printed spoofs of commercial magazines which can be hard to distinguish from the original).
Finally, a teleoperator or telerobot can sometimes return an actual artifact to the user. For example, a submersible robot can bring back a distinctive part from a sunken ship. Once the robot has returned with the part, the operator has increased confidence that all of the images and sensations he/she experienced in the preceding period of remote control were valid. Of course such an experience can still be faked, but the cost of such forgery increases when the physical artifact is produced.
The original teleoperators created by the nuclear weapons complex (Goertz, 1954, 61) used joint coordinates. The Manhattan project created the need to manipulate highly toxic materials in precise ways. The conflicting goals of protection of workers from radiation and the ability to manipulate such materials into precise shapes were not being met by what were essentially tongs which grew longer and longer or automated fixed purpose special machines. Telemanipulation therefore became a critical need.
The response was the development of precision engineered mechanical systems which allowed dexterous manipulation behind a 1 meter thick quartz window (FIG 1). Two identical arm mechanisms are positioned in front of the operator and task respectively. Corresponding joints of the two devices are connected by flexible stainless steel ribbons running over pulleys.
 |
Fig 1. Mechanical Teleoperator
The original telemanipulators were mechanical devices engineered to handle toxic materials in the nuclear weapons complex. (Source: Novelty postcard book published by Klutz Press, 2170 Staunton Ct., Palo Alto CA 94306 -- originally from Central Research Laboratories, Red Wing Minn. ) |
From today's point of view these devices seem primitive because they lack electronics or computer control. Nevertheless, they were quite effective and are still in wide use. Besides careful mechanical design, the one-to-one connection between the two sides creates a compelling sensation reproducing the actual sensations of manipulation. These mechanical teleoperators were the first highly dexterous mechanisms. They were the immediate precursor of today's industrial robot manipulators.
Fundamentally, the purely mechanical devices were limited to about 5 meters separation between the two sides. Furthermore, this separation had to be fixed at the time of installation, neither side could be moved relative to the other. Newer applications demanded that the remote side be able to move (for example along the length of a particle accelerator). The response was to develop an electronic version of the mechanical remote manipulator (FIG 2) (Goertz, 1952).
 |
Fig 2. Ray Goertz of Argonne National Laboratories.
Shown is a small scale mechanical link teleoperator. Goertz later invented the first electronic remotely operated manipulators. (Source: Argonne National Labs public info office, Ms. Pat Canneday, (630)252-5562). |
This involved two arms similar to those of the mechanical system, but the steel tapes running between the two were cut, and connected to motors and sensors. At first, motors were installed only on the remote (now called the "slave") side. The control system applied torques to the slave side in such a way that it's position followed that of the operator (now called the "master") side. Later the master side was motorized in response to the operator's complaints about lack of force feedback.
 |
 |
Fig 3. Prof. Tachi's "tele-existance" system
"Tele-existance" system developed by Prof. Sumumu Tachi, University of Tokyo. Camera on slave robot is slaved to user's head motion. Operator's view of robot arm accurately aligned with kinesthetic sensations of his/her own arm. (Source: Prof. Tachi via email) |
Bejczy and Salisbury designed the first 6-axis(1) [footnote 1: at least six mechanical degrees of freedom (or "motion axes") are required to make an aribitrary manipulation of a rigid object in space. ] mechanism specifically for human bi-directional telemanipulation [FIG 4]. Unlike previous devices, this "hand controller" was designed specifically for the human operator without regard for the slave device. In this system the slave was a PUMA 560 industrial robot manipulator fitted with a computer controlled robot hand. The slave robot hand included a six-axis force/torque sensor (Fiorini 1988). Motors on the hand controller allowed a force and torque to be applied to the operator's hand based on the sensed force on the slave. This established a virtual bi-directional energetic interaction between the operator's handgrip and the robot hand.
 |
Fig. 4. Salisbury 6-axis "hand controller"
First master unit with six degrees of freedom which was designed for the human - independent of the slave robot. Device can measure displacements of operator and apply feedback forces and torques in all directions necessary for positioning and constraining an object in space. (Source: JPL negative number 9647 Ac) |
Coordinate transformations between the joint spaces of the master and slave devices (in both directions) were carried out at 1000 Hz by microcomputers. The system was evaluated in laboratory experiments in which operators performed simulated tasks (Hannaford et al., 1991). In this study, the simulated bi-directional interaction generally improved performance as measured by the time it took to complete tasks.
Alternatively, teleoperated and autonomous functions can coexist at the same time, for example, by controlling different degrees of freedom at the same time. For example, in a system developed by Bejczy & Kim (1990), a telerobot could perform assembly of precise rigid parts in spite of significant time delay if the slave computer locally controlled orientations while the remote operator controlled displacement in x,y,z translation. Shared control can also be performed on the same motion axis (degree of freedom). In Kim et al,'s system, the operator can specify a "reference" position and orientation, while the slave's computer controls deviations from that position in response to measured forces. This type of sharing can significantly improve the ability to handle precisely mating parts in the presence of time delay, yet can also introduce a discrepancy between the operator's commands and the robot's actions, perhaps reducing the operator's perception of actually manipulating a real object.
A graphical user interface was developed by Hannaford et al. (1990) to allow an operator to select among these shared control options (FIG 5). Each of the six Cartesian degrees of freedom could be in one of 10 different operating modes, including those described above. Since each axis could be controlled independently, 10^6 possible modes are available to the operator. For a dual arm system, 10^12 possible modes are available! This creates the problem of how to select the best mode from this set for a given task. From the viewpoint of Telepistomology, we have the additional problem of how to calibrate our own senses for each of these many possible modes because a given environment will "feel" differently to the operator in each mode.
 |
Fig. 5. Graphical User Interface used to select control modes.
The JPL system had as many as 10 possible modes of operation which could be independently selected for each motion axis. This graphical user interface was developed to help the operator choose among these one million possible operating modes. (Source: JPL negative number 11525Ac) |
To illustrate supervisory control, a recent application to an internet telerobot will be described. In the UW/UAB/Boeing Telerobotic Protein Crystal Mounting Cell (Hannaford et al., 1997ab) a small robot manipulator was integrated into a scientific glove box prototyped for the International Space Station [FIG 6]. In order to deduce the structure of biologically significant proteins, the most commonly employed method is to grow crystals of the protein in aqueous solution and to analyze the crystals with X-ray diffraction. This 3-5 day process can create more regular crystals in micro-gravity hence the desireability of performing this procedure in outer space. These crystals are about 0.5mm in size and have the consistency of gelatin. They must be aspirated from the water solution into a capillary with 1mm inside diameter. The system consisted of a 5-axis high precision mini robot with DSP controller, linear motion system to position the robot base, microscope, video cameras, fluid pump, slave side "server" and power control system.
Supervisory control was employed for this experimental system. The user interface provided two distinct methods for controlling equipment at the remote site. The primary control method used pre-determined “macros” which encoded sequences of low-level commands such as “move robot along a trajectory to position ‘X’, move linear rail to position ‘Y’, and dispense ‘Z’ microliters of fluid from the fluid pump" (Hannaford et al., 1997). These macros were assigned to individual buttons on the graphical user interface, allowing the operator to quickly accomplish tasks by pressing a sequence of macro buttons. The macro definitions were stored and executed on the "server" computer at the remote site.
 |
Fig, 6. UW Protein Crystal Growth (PCG) Cell Telerobot
Internet controlled mini-telerobot for handling protein crystal samples in a simlulation of the International Space Station. Operators in Huntsville Alabama (password protected access from trusted site)successfully captured simulated 0.5mm protein crystals by clicking virtual control panel icons and reviewing progress through compressed video signals. Developed at University of Washington, in conjunction with Boeing Defense and Space Company, and the University of Alabama Center for Macromolecular Crystalography. (Source: UW Biorobotics Lab) |
The user interface also allowed low-level control of robot joints, linear rail position, fluid pump parameters, etc. via sliders and buttons in pop-up dialog boxes. This low-level control capability was intended only as a secondary or backup control method and to be used for performing unplanned or experimental procedures, and generating new macro definitions. Video signals from the workcell were sent back to the operator via CuSeeMe teleconferencing software (Fig. 7).
 |
Fig. 7. Video screen for PCG cell operator.
TV images available to the operator of the telerobot in Figure 6 transmitted via CuSeeMe teleconferencing software. (Source: UW Biorobotics Lab) |
Besides information capacity, a second key property of the communication channel is time delay, the time required for a message to arrive at the destination. Since a telerobotic system requires communication in two directions, we must consider delays introduced by both links. These delays introduce a dissociation between the operator's commands and the action of the slave. More significantly, delay of the returning sensor information is also introduced between the operator's action and the resulting sensory feedback displays of the robot response.
There are two sources of this time delay. Most fundamentally, there is the delay due to the speed of light. While significant in contemplated space applications, this delay is rarely dominant except for interplanetary applications such as the Sojourner Mars Rover. The time delay due to lightspeed between the earth surface and a communication satellite in geosynchronus orbit can be signicicant however; about 250 milliseconds round trip.
The second major source of delay is that introduced by switches in computer networks. These delays are caused by processing of information in computers at nodes in the network. Pietro Buttolo measured this delay in 1996 (as well as loss rate of UDP packets) for different distances on the Internet (Table 1).
TABLE 1 Distance Delay (ms) Scale loss rate(%) min avg max ------------------------------------------------------- room 0.0 2.0 3.0 18.0 dept 0.0 2.5 3.5 23.0 campus 0.0 3.0 3.0 5.0 continent 8.0 62.3 94.5 205.3 planet 11.5 278.3 421.0 746.8 ------------------------------------------------------- Internet transmission properties for UDP packets on 19-Feb-1996 (P. Buttolo, used with permission).
For applications such as interplanetary exploration in which delays exceed a few seconds, the "move-and-wait" strategy becomes extremely cumbersome. Supervisory control has been used as a means to get around this problem. With "move-and-wait," the time taken to complete a task is
T = N(d + a)where d is the round trip time delay, a is the time that the slave can operate on its own without human intervention (alternatively the average duration of a teleoperated movement which can be performed without feedback), and N is the number of such moves required to complete the task. If Na is considered the time required for the task with no delay, the performance penalty due to delay can be expressed as Nd. If a can be increased (for example due to smarter supervisory control) then N can be reduced and the performance penalty due to time delay is less.
Another approach is to provide better planning aids to the operator so that he/she can create a longer series of commands with high confidence. One part of a telerobotic work cell which can be predicted with a high degree of confidence is the response of the slave robot to commands. In a "predictive display," (FIG 8, Kim and Bejczy, 1993), the operator could manipulate a computer graphics simulation of the slave robot. This simulated robot could be superimposed over the video returning from the remote site (by video "keying") which proved to be a significant help in planning the commands.
 |
 |
Fig. 8. Kim & Bejczy's Predictive Display
Computer graphics images of remote manipulator may be superimposed ("keyed") over the returning video image. Computer graphics image responds instantly while video image of robot follows after communication time delay. When there is no motion, actual video image of robot is hidden behind computer graphics image. Excellent registration between computer graphics and robot image is assured by accurate geometric modelling of robot, and human assisted calibration procedure. (Source: JPL negative numbers: 11718Bc & 11719Bc via Dr. Wonsoo Kim) |
A significant issue becomes calibration of the computer graphics camera model with the perspective display parameters representative of the actual camera. If the two are not calibrated, the simulated slave robot does not appear in the proper perspective relative to the video image of the real slave and therefore does not seem to the operator to be actually present in the remote site.
Once the operator is provided with a realistic simulation of the remote site (for planning purposes) and with "predictive" control techniques, a level of abstraction is introduced between the operator and the remote site and it is not as straightforward to keep track of the state of the system. Conway and Volz (1990) introduced some compelling metaphors, the "time clutch" and the "space clutch" to clarify the profound but sometimes obscure difference between controlling a telerobot and a simulated robot. The "Time Clutch" is a foot pedal which, when pressed and held like an automotive clutch, allows the predictive simulated robot to move ahead of the positions attainable by the slave robot(which might have a relatively low maximum speed). The operator's inputs were held in memory as necessary until the slave could catch up. A "position clutch" can also be used which disengages the operator's commands entirely from the slave robot so that the operator can interactively experiment with the simulation. Finally, a "time brake" is also provided. Pressing this pedal, disengages the two clutches, and also reverses the sense of time for the predictive robot simulation, progressively emptying out the command memory until the predictor "comes back" to the currently visible robot state.
Digital transmission is rapidly emerging for video signals. Of the two major current standards for video source coding, MPEG-2 and H.263, H.263 is currently most relevant because of the requirement for real-time, low latency compression. However, preserving NTSC picture quality requires 6-9 megabits per second depending on amount of picture motion. We thus need 18-27 Mbits per second to send back our three video signals. Do to the rapid development of digital television, this area is in rapid flux. However little is being done on digital compression in the context of teleoperation.
"Can we make a valid simulation of the total up and down link delay by delaying ONLY the up-link information by the total of the up-link and down-link times?"
This question has not been formally studied, but the consensus seems to be that operator behavior is unaffected by such an expedient. Although slave responses are shifted in time by a fixed amount (equal to the downlink delay), the operator will not perceive a difference unless operations must be synchronized with an event which occurs at an absolute time (unlikely in a laboratory setting).
When a virtual energetic link is unstable, oscillations emerge and may grow in magnitude. For small oscillations of bouned amplitude, the user experiences a "noise" or distraction which is destructive to the illusion of remote presence or manipulation. When the oscillations are stronger, the system can be totally unusable or even dangerous. Stability is determined by a complex interaction between time delay around the loop and gain. This interaction has been studied by analysis and experimentation (Hannaford and Anderson, 1988, Hannaford 1989, Anderson and Spong, 1989, Neimeyer and Slotine, 1991,98). The mathematical difficulty is compounded by the problematic task of making useful mathematical models of the human operator and the environment. By applying the notion of "passivity", a clever class of control laws were synthesized for which stability could be guaranteed regardless of the intervening time delay (Anderson and Spong, 1989, Neimeyer and Slotine, 1991). Passivity in this context means that the net energy absorbed by the teleoperator system (through its two interaction ports) over all time exceeds the energy that it supplies. If the teleoperator can be made passive, then stability can be guaranteed assuming only that the operator and environment are passive - an assumption which works well in practise (Colgate and Hogan, 1988). Unfortunately, passivity appears to be an overly conservative criterion. For while such systems are indeed stable, they are characterized by very slow response and "sluggish" feel (Lawn and Hannaford, 1993).
Scaled teleoperation systems create new challenges for the notion of knowlege and belief. This issue was studied upon the invention of the microscope, but new issues arise in teleoperation. One of the complexities is illustrated by consideration of scaling physical characteristics. Assume that a microteleoperation system is intended to scale up the environment by a factor of K. In the visual side, this corresponds to a microscope with magnification K. On the kinesthetic side, it is more complex. We can set appropriate force and position scales to keep kinesthetic perception unchanged. However as an object is scaled down, its Mass drops with K^3 while its apparent visual size scales only as K. Its surface area (source of many signficant physical effects for small objects) scales with K^2. So the resulting virtual object manipulated by the operator will have a mix of "natural" and "unnatural" properties.
Consider a scaled system in which position commands are multiplied by a factor Lambda-P between master and slave and in which force feedback is multiplied by Lambda-F from slave to master. We can construct a diagram (FIG 9) illustrating qualitative features of the resulting teleoperation system over the plane formed by these two parameters:
 |
Fig.9. Manipulation scales and their effects on haptic perception.
When scale factors are interposed between human displacement / force and robot displacement / force, qualitative changes are induced in human mechanical perception as well as human power output. The remote environment can be made to feel stiffer/heavier or softer/lighter. Because different physical attributes scale with different power of size, it is inherently impossible to "map" physical reality accross large differences of scale. (Source: The author) |
The diagram is plotted for the arbitrary range of scale factors, 0.01 to 100 but it applies to all magnitudes. The diagonal, Lambda-f = Lambda-p, is the locus of systems which have unity power gain between master and slave. All passive mechanical micro or macro manipulators including the original nuclear materials handling systems, the ones Feynman anticipated, and typical tools such as tweezers, pliers, and pry bars, fall on this line. All of these systems distort the operators perception of the kinesthetic properties of the environment in a predictable way. From the physical principle of "virtual work," we can show that mechanical properties such as stiffness and mass of the environment are scaled by a factor of Lambda-p^2.
However, when bi-directional interaction is realized by a teleoperator system, other combinations of gains are possible. For example, another important relation is
Lambda-f = (Lambda-p)^-1
It can be shown (Hannaford, 1991) that operation on this line does not
alter the mechanical properties (more precisely the mechanical
impedance) of the environment which are felt by the operator. But this
relation implies that the systme must be active and therefore more
difficult to make stable.
To place this problem in more concrete terms, consider handling a tuning fork and feeling it vibrate. Now consider a miniature tuning fork 100 times smaller with which we interact through a scaled teleoperation system. The teleoperation system will have
Lambda-f = (Lambda-p)^-1 = 0.01so that the operator will feel the correct mechanical properties of the environment, and a microscope will be provided with 100:1 magnification.
If the miniature tuning fork is made with the same materials, its frequency will be 10 times higher. If we view the miniature tuning fork through 100:1 magnification, it will look identical to the original tuning fork, it will be composed of the same materials, and yet it will "feel" different as exemplified by the ten times higher natural frequency.
Historical parallels to this problem exist in the area of naval architecture. There was a critical need to study the performance of ships by towing scale models of their hulls in a tank. But, to what extent did this model system represent the "reality" of an actual ship? The dynamics of water depend critically on scale. Research aimed at this question lead to fundamental advances in hydrodynamics by Froud and Reynolds who developed dimensionless ratios (Reynolds and Froud numbers) which predicted qualitative behavior of fluids. In other words, if the model and the towing speed were scaled by a simple constant, the towing behavior was not realistic, but if the scale factor and fluid properties were scaled in such a way that the Reynolds number was constant, then certain aspects of the towing-tank model was accurate.
Teleoperator researchers have addressed this problem with elegant applications of scaling theory (Colgate 1991, Kobyashi 1992, Goldfarb, 1998), but no consensus has emerged on how to solve it. The dissociation between visual and kinesthetic percepts imposed by scaling of manipulation processes poses fresh difficulties for the nature of knowledge.