The value hypothesis and acquisition of preference in concurrent chains

Experiment 1 compared the acquisition of initial- and terminal-link responding in concurrent chains. The terminal-link schedules were fixed interval (FI) 10 sec and FI 20 sec, but some presentations were analogous to no-food trials in the peak procedure, lasting 60 sec with no reinforcement delivery. Pigeons completed a series of reversals in which the schedules signaled by the terminal-link stimuli (red and green on the center key) were changed. Acquisition of temporal control of terminal-link responding (as measured by peak location on no-food trials) was more rapid than acquisition of preference in the initial links. Experiment 2 compared acquisition in concurrent chains, using the typical procedure in which the terminal-link schedules are changed with a novel arrangement in which the initial-link key assignments were changed while the terminal-link schedules remained the same. Acquisition of preference was faster in the latter condition, in which the terminal-link stimulus-reinforcer relations were preserved. These experiments provide the first acquisition data that support the view that initial-link preference is determined by the values of the terminal-link stimuli.     

Determinants of pigeons’ choice between certain and probabilistic outcomes

Pigeons chose between 50% and 100% reinforcement on a discrete-trials concurrent-chains procedure with fixed-ratio 1 initial links and fixed-time terminal links. The 100% alternative always provided food after a terminal-link delay, whereas the 50% alternative provided food or blackout equally often after a delay. Additionally, the terminal-link stimuli on the 50% alternative were correlated with the outcomes in signaled, but not in unsignaled, conditions. The effects of intertrial-interval duration and length of the terminal-link delays on choice of the 50% alternative were investigated in four experiments. Preference for the 50% alternative varied with signal condition and duration of the terminal link leading to food, but not with duration of either intertrial interval or the terminal link leading to a blackout. The results are discussed in terms of conditioned-reinforcement effects, Mazur’s hyperbolic-decay model, and delay reduction. This research was supported by a Natural Sciences and Engineering Research Council of Canada research grant awarded to the first author.     

Preference between fixed-interval and variable-interval schedules of reinforcement: Separate roles of temporal scaling and predictability

Pigeons’ preference between fixed-interval and variable-interval schedules was examined using a concurrent-chains procedure. Responses to two concurrently available keys in the initial links of the concurrent chains occasionally produced terminal links where further responses were reinforced under either a fixed- or variable-interval schedule. In previous studies, preferences for the variable schedule with such a procedure have been interpreted as reflecting atemporal scaling process that heavily weights the shorter intervals in the variable schedule. The present experiment examined whetherpredictability, i.e., the presence of external stimuli correlated with the reinforcement interval, might also influence preference in such situations. When the two intervals in a variable schedule were made predictable by being associated with different key colors, preference for that schedule increased. This increase was reliable but small in magnitude and transient when initial-link responses only occasionally produced terminal links; it was large in magnitude when only one response in the initial link was required to produce the appropriate terminal-link schedule. The results suggest that preference between fixed and variable schedules may be influenced both by temporal scaling and to a lesser extent by predictability of the reinforcement intervals.     

Discriminative stimuli that follow the absence of reinforcement are preferred by pigeons over those that follow reinforcement

Clement, Feltus, Kaiser, and Zentall (2000) found that when pigeons have to work to obtain a discriminative stimulus that is followed by reinforcement, they prefer a discriminative stimulus that requires greater effort over one that requires less effort. The authors suggested that such a preference results from the greater change in hedonic value that occurs between the more aversive event and the onset of the stimulus that signals reinforcement, a contrast effect. It was hypothesized that any stimulus that follows a relatively more aversive event would be preferred over a stimulus that follows a relatively less aversive event. In the present experiment, the authors tested the counterintuitive prediction of that theory, that pigeons should prefer a discriminative stimulus that follows the absence of reinforcement over a discriminative stimulus that follows reinforcement. Results supported the theory.     

Initial-link duration and acquisition of preference in concurrent chains

Initial-link response allocation in concurrent chains becomes less extreme as the absolute duration of the initial links increases (Fantino, 1969). The present study asked whether initial-link duration affected how quickly response allocation reached asymptote (i.e., acquisition of preference). Six pigeons were trained on a concurrent-chains procedure in which the terminal links were fixed-interval (FI) 8 sec FI 16 sec or FI 16 sec FI 8 sec and were reversed every 20 sessions. Across conditions, all possible combinations of transitions between variable-interval (VI) 8-sec (short) and VI 24-sec (long) initial-link schedules were studied. Overall, the rate of acquisition was faster when the durations of the initial links preceding the reversal were short rather than long, and when the durations of the initial links following the reversal were long rather than short. By contrast, initial-link duration had no effect on acquisition or asymptotic measures of temporal control of terminal-link responding. These results support the core principle of delay-reduction theory (Fantino, 1969) that the impact of a conditioned reinforcer varies directly with initial-link duration, but also suggest that temporal learning during the terminal links proceeds independently of initial-link duration. nt]mis|These data were presented at the annual meeting of the Association for Behavior Analysis, Boston, May 2004.     

Do pigeons prefer information in the absence of differential reinforcement?

Prior research has indicated that pigeons do not prefer an alternative that provides a sample (for matching to sample) over an alternative that does not provide a sample (i.e., there is no indication of which comparison stimulus is correct). However, Zentall and Stagner (Journal of Experimental Psychology. Animal Behavior Processes 36:506?C509, 2010) showed that when delay of reinforcement was controlled, pigeons had a strong preference for matching over pseudomatching (i.e., there was a sample, but it did not indicate which comparison stimulus was correct). Experiment 1 of the present study replicated and extended the results of the Zentall and Stagner (Journal of Experimental Psychology. Animal Behavior Processes 36:506?C509, 2010) study by including an identity relation between the sample and one of the comparison stimuli in both the matching and pseudomatching tasks. In Experiment 2, in which we asked whether the pigeons would still prefer matching if we equated the two tasks for probability of reinforcement, we found no systematic preference for matching over pseudomatching. Thus, it appears that in the absence of differential reinforcement, the information provided by a sample that signals which of the two comparison stimuli is correct is insufficient to produce a preference for that alternative.     

Matching-to-sample in pigeons: In the absence of sample memory,sample frequency is a better predictor of comparison choice than the probability of reinforcement for comparison choice

In matching-to-sample, comparison choice should be controlled by the identity of the sample and, when the sample is not available, by the overall probability of reinforcement associated with each of the comparisons. In the present research, pigeons were trained to match a frequent sample (appearing on 80% of the trials) to one comparison (C _fr) and an infrequent sample (appearing on 20% of the trials) to the other (C _inf), with the number of reinforcements associated with each sample equated. In Experiment 1, the task was identity matching; in Experiments 2 and 3, it was symbolic matching. We asked whether, when control of comparison choice by the sample was reduced (by inserting a delay between the sample and the comparisons), pigeons would choose comparisons on the basis of (1) the number of reinforcements per comparison (and thus show no comparison bias), (2) the comparison associated with the more frequent sample during training (and show a preference forC _fr), or (3) the probability of reinforcement given a correct response (and show a preference forC _inf), or (4) inhibition produced by nonreinforced choice of the more frequently correct comparison (and show a preference forC _inf). Pigeons showed a significant tendency to chooseC _fr. In Experiment 3, we showed that this bias did not result from the effects of intertrial facilitation or interference. Thus, it appears that when control of comparison choice by the sample is reduced, pigeons’ choice is controlled not merely by the probability of reinforcement but also by overall sample frequency.     

Choice and percentage reinforcement in pigeons

Pigeons responded on a two-key concurrent chains choice procedure with the same level of percentage reinforcement on each key. During the initial links, a choice response on either key occasionally produced a conditioned reinforcer—which on one key was associated with a 15-sec, and on the other key with a 30-sec, interreinforcement interval—or an extinction stimulus. In Part 1, the initial links were equal. With successive decreases in the probability of a reinforcer, choice shifted from preference for the 15-sec terminal link toward indifference. In Part 2, the initial links were unequal and were arranged so that the shorter initial link preceded the 30-sec terminal link. At a high probability of a reinforcer, the pigeons again preferred the 15-sec terminal link. However, at a low probability, the pigeons reversed and preferred the alternate key. It was concluded that the conditioned reinforcers tended to become functionally equivalent at a low probability of a reinforcer, despite the nominally different interreinforcement intervals, with the result that choice was then modulated by the relative size of the initial links. The data are inconsistent with the view that choice and the strength of conditioned reinforcers are isomorphic with the reduction in delay to reward correlated with terminal link stimuli.     

Signal properties of reinforcement and reinforcement omission on a multiple fixed-ratio schedule

A series of experiments used food-deprived pigeons to examine several parameters of reinforcement omission in an attempt to control changes of keypeck response measures on a subsequent schedule. In Experiments 1 and 2, the pigeons were tested with a multiple fixed-ratio schedule on which reinforcement was occasionally omitted at the completion of the first component. The duration of the delay occurring in lieu of reinforcement was systematically varied. In Experiment 3, the stimulus that signaled the second component of the schedule was altered to appear either more or less similar to the stimulus that signaled the first component. Two principal results are reported: (1) Response latency decreased and, to a much lesser extent, terminal response rate increased as the delay occurring in lieu of reinforcement decreased; and (2) both latency decrease and response-rate increase were enhanced by a second component stimulus which was similar to the first. The results are evaluated in terms of Amsel’s frustration theory and an analysis by Staddon which suggests that reinforcement inhibits responding. The data appear to support Staddon’s argument that rate increases and latency decreases following reinforcement omission are largely a function of an attenuation of the inhibitory influence of reinforcement, an effect that is enhanced by stimulus generalization. Accordingly, it is proposed that an animal’s response to reinforcement omission is determined by a stimulus complex that minimally includes the omission event and component cues.     

Response variability in pigeons in a pavlovian task

Instrumental response variation is inversely related to reward probability. Gharib, Derby, and Roberts (2001) theorized that individuals behave more variably when their expectation of reward is low. They postulate that this behavioral rule assists the discovery of alternative actions when a target response is unlikely to be reinforced. This suggests that response variability may be unaffected in a situation in which an animal’s behavior is inconsequential to outcome delivery. We trained 6 pigeons in a within-subjects Pavlovian autoshaping procedure. On any given trial, the pigeons were presented with one of six colored discs on a touchscreen; each stimulus was associated with a particular probability of food, ranging from 100% to 0.6%. Pecking was more variable with low probabilities of food delivery, thus extending the rule relating variability and expectation to a Pavlovian situation.     

Behavioral contrast and reinforcement value

Behavioral contrast was produced in two target components of a four-component multiple schedule by having two target stimuli followed either by a higher rate of reinforcement or by extinction. Response rate was higher in the target followed by extinction. Periodic probe trials were then presented in which the two target stimuli were presented together. Choice on these probe trials was in favor of the stimulus followed by the higher rate of reinforcement during regular training. Experiment 2 replicated this finding but with probe trials presented throughout training. Here, preference for the stimulus followed by the higher rate of reinforcement was evident early in training, substantially before the contrast effects developed. The results challenge interpretations of contrast based on the concept of relative value.     

Deprivation level and choice in pigeons: a test of within-trial contrast

Within-trial contrast has been proposed as a mechanism underlying preferences for stimuli that follow relatively more aversive events over stimuli that follow less aversive events. In this study, we manipulated deprivation level to test within-trial contrast predictions. In Experiment 1, pigeons encountered two discriminative stimuli, one presented when they were deprived and the other when they were prefed. When later given a choice between the two stimuli, pigeons strongly preferred the stimulus encountered when deprived, independently of their deprivation level at test. In Experiment 2, pigeons learned two simultaneous discriminations, one when deprived and the other when prefed. Here, subsequent tests between the two S⁺ or the two S⁻ stimuli revealed no consistent preferences. These contrasting findings suggest that differential aversiveness is necessary but not sufficient to induce preferences via within-trial contrast.     

The influence of brief stimuli uncorrelated with reinforcement on choice between variable-ratio schedules

Experiment 1 investigated the behavior of rats trained to leverpress on a concurrent variable ratio (VR) 30 VR-30 schedule with a brief, 500-msec, light occurring at the midpoint of the ratio on one of the levers. Higher response rates were recorded on the lever associated with this stimulus, a finding that paralleled the effect produced by inserting primary reinforcement at the midpoint (i.e., by training on a concurrent VR-30 VR-15 schedule). Similar results were found in Experiment 2 using a concurrent VR-20 VR-20 schedule with a 2-sec visual stimulus presented midway through one of the components. In addition, a brief stimulus inserted midway through the VR-20 component of a concurrent VR-20 VR-10 schedule retarded the development of a difference in response rates between the components relative to a VR-20 VR-10 group lacking the signal. In Experiment 3, multiple VR VR schedules were used. Again, the response rate was higher in the component that had the added stimulus or, for a second group of subjects, on the component with the smaller response requirement. Probe-choice trials revealed a preference for the component that generated the higher rate in both groups. Presenting a stimulus partway through a ratio appears to reduce the effect on response rate and choice of a large ratio value.     

Training reinforcement rates,resistance to extinction,and the role of context in reinstatement

Behavior reduced as a consequence of extinction or intervention can relapse. According to behavioral momentum theory, the extent to which behavior persists and relapses once it has been eliminated depends on the relative training reinforcement rate among discriminative stimuli. In addition, studies of context renewal reveal that relapse depends on the similarity between the training stimulus context and the test stimulus context following disruption by extinction. In the present experiments with pigeons, we arranged different reinforcement rates in the presence of distinct discriminative stimuli across components of a multiple schedule. Following extinction, we attempted to reinstate responding in the presence of those target components with response-independent food presentations. Importantly, we arranged the reinstating food presentations either within the target components or in separate components, either paired with extinction (Experiment 1) or reinforcement (Experiment 2) during baseline. Reinstatement increased with greater training reinforcement rates when the reinstating food presentations were arranged in the target components and the separate components paired with reinforcement during training. Reinstatement was smaller and was not systematically related to training reinforcement rates in the target components when reinstating food presentation occurred in separate components paired with extinction. These findings suggest that relapse depends on the history of reinforcement associated with the discriminative stimuli in which the relapse-inducing event occurs.     

Behavioral contrast as a function of component duration and baseline rate of reinforcement

Three experiments examined changes in size of multiple-schedule behavioral contrast with changes in an independent variable. Experiment 1 found that positive contrast generally increased with increases in component duration when pigeons pressed treadles. Experiments 2 and 3 found that positive and negative contrast generally increased with increases in the baseline rates of reinforcement when pigeons pecked keys. The experiments show that positive and negative contrast vary as similar functions of the same variables. Experiment 1 also suggests that these functions are different for different responses.     

Decision making by humans in a behavioral task: Do humans, like pigeons, show suboptimal choice?

Consistent with human gambling behavior but contrary to optimal foraging theory, pigeons show a strong preference for an alternative with low probability and high payoff (a gambling-like alternative) over an alternative with a greater net payoff (Zentall & Stagner, Proceedings of the Royal Society B, 278, 1203-1208, 2011). In the present research, we asked whether humans would show suboptimal choice on a task involving choices with probabilities similar to those for pigeons. In Experiment 1, when we selected participants on the basis of their self-reported gambling activities, we found a significantly greater choice of the alternative involving low probability and high payoff (gambling-like alternative) than for a group that reported an absence of gambling activity. In Experiment 2, we found that when the inhibiting abilities of typical humans were impaired by a self-regulatory depletion manipulation, they were more likely to choose the gambling-like alternative. Taken together, the results suggest that this task is suitable for the comparative study of suboptimal decision-making behavior and the mechanisms that underlie it.     

Failure to obtain value enhancement by within-trial contrast in simultaneous and successive discriminations

The present research tested the generality of the “work ethic“ effect described by Clement, Feltus, Kaiser, and Zentall (2000). In Experiment 1, we trained 10 pigeons on a pair of either simultaneous or successive discriminations. One discrimination followed a high-effort requirement (20 pecks to the center key) and the other followed a low-effort requirement (1 peck). Contrary to Clement et al.’s results, we found that preferences between the S⁺ and S⁻ stimuli in transfer tests depended on the event that initiated the trial: Pigeons preferred the stimulus from the baseline discrimination whose initiating event was most dissimilar from that preceding the test trial. Preferences were similar but less extreme in the successive condition. In Experiment 2, we investigated whether test preferences depended on the amount of training. A total of 12 pigeons were trained on a pair of simultaneous discriminations, except that test sessions were scheduled after every three baseline sessions. Preferences increased across test sessions but were similar to those in Experiment 1. Together with Vasconcelos, Urcuioli, and Lionello-DeNolf (2007a), our study represents a second failure to replicate Clement et al.’s work ethic effect. The finding that preference depends on the event that initiates the test trial suggests that choice probes may not provide unambiguous assessments of stimulus value.     

The effect of rate of reinforcement and time in session on preference for variability

Pigeons pecked keys on concurrent-chains schedules that provided a variable interval 30-sec schedule in the initial link. One terminal link provided reinforcers in a fixed manner; the other provided reinforcers in a variable manner with the same arithmetic mean as the fixed alternative. In Experiment 1, the terminal links provided fixed and variable interval schedules. In Experiment 2, the terminal links provided reinforcers after a fixed or a variable delay following the response that produced them. In Experiment 3, the terminal links provided reinforcers that were fixed or variable in size. Rate of reinforcement was varied by changing the scheduled interreinforcer interval in the terminal link from 5 to 225 sec. The subjects usually preferred the variable option in Experiments 1 and 2 but differed in preference in Experiment 3. The preference for variability was usually stronger for lower (longer terminal links) than for higher (shorter terminal links) rates of reinforcement. Preference did not change systematically with time in the session. Some aspects of these results are inconsistent with explanations for the preference for variability in terms of scaling factors, scalar expectancy theory, risk-sensitive models of optimal foraging theory, and habituation to the reinforcer. Initial-link response rates also changed within sessions when the schedules provided high, but not low, rates of reinforcement. Within-session changes in responding were similar for the two initial links. These similarities imply that habituation to the reinforcer is represented differently in theories of choice than are other variables related to reinforcement.     

Stimulus preference and the transitivity of preference

Four pigeons were exposed to multiple schedules with concurrent variable interval (VI) components and then tested for preference transfer. Half of the pigeons were trained on a multiple concurrent VI 20-sec, VI 40-sec/cuncurrent VI 4G-sec5 VI 80-sec schedule. The remaining pigeons were trained on a multiple concurrent VI 80-sec, VI 40-sec/concurrent VI 40-sec, VI 20-sec schedule-After stability criteria for time and response proportions were simultaneously met, four preference transfer tests were conducted with the stimuli associated with the VI 40-sec schedules. During the transfer tests, each pigeon allocated a greater proportion of responses (M=0,79) and time (M=0.82) to the stimulus associated with the VI 40-sec schedule that was paired with the VI 80-sec schedule than lo the VI 40-sec schedule stimulus paired with the VI 20-sec schedule. Absolute reinforcement rates on the two VI 40 sec schedules were approximately equal and unlikely to account for the observed preference. Nor was the preference consistent with the differences in local reinforcement rates associated with the two stimuli. Instead, the results were interpreted in terms of the differential value that stimuli acquire as a function of previous pairings with alternative schedules of reinforcement.     

Feedback functions for reinforcement: A paradigmatic experiment

A reinforcement schedule states a rule for obtaining reinforcement as a function of behavior actually emitted and perhaps as a function of additional variables. These functions are here called “feedback functions.” Behavior actually emitted is, in turn, a function of obtained reinforcement. This reciprocal interdependency was quantified for an experiment in which pigeons chose among either three or two alternatives. Shifting from three to two alternatives, and vice versa, produced changes in the distribution of responding which were approximately accounted for by equations that combined the feedback functions with the matching law for reinforced responding. These equations predicted, among other things, a violation of the constant-ratio rule of formal choice theory and an absence of simultaneous contrast effects between response alternatives reinforced on variable-interval schedules. Both predictions were approximately confirmed.