The Rat Days are Over

Overall, I really enjoyed the training process. Although it was difficult at times to schedule other things around my training schedule, I was actually surprised by how much progress I saw in the relatively small amount of time that Mitzy and I spent training. I was so impressed with Mitzy's rate of learning, especially on her first few days of training. She seemed to very quickly form an association between the response and consequence (reward), and I felt like a very proud psychology student/ trainer/ rat mommy. I was also surprised when I was somewhat disappointed that it was over!

I really liked being able to work with a live animal, which I was initially nervous about--not because I disliked rats, but rather, because I was worried about what kind of conditions we might have to subject the rats to. In general, I think that I had some very misguided notions about what using animal subjects is like. Spending time in the rat room, however, and seeing how they were all treated by Dr. Trench, Devon, and all of the rest of the students, greatly improved my attitude toward using animal subjects. As far as I know, everyone really bonded with their rats, and I think that Dr. Trench and Devon are pretty protective of them :P

Over the course of this project, I continued to look forward to hanging out with Mitzy for a few minutes after each training session, even though she tended to be kind of antzy at this point. I never had any problems with her biting me, and she seemed perfectly comfortable with being picked up and handled. Mitzy was a great subject, and she made my job pretty easy most of the time. I really enjoyed carefully observing Mitzy during shaping to figure out exactly when or what to reinforce, and I appreciated being able to watch her progress on her own for the rest of training. After shaping, I just sat back with a notepad and a pencil and let the magic happen! :)

Love,

Lindsey and Mitzy

PS: In case bar pressing gets a little bit old, here's some inspiration for the next training project:

Running the Rat Race: Mitzy vs. Sniffy

Overall, magazine training and shaping Mitzy were both faster and more enjoyable processes than magazine training and shaping her virtual counterpart, Sniffy. For Sniffy, magazine training took approximately 50 minutes, and shaping required a little more than an hour. For Mitzy, magazine training took very little time (approximately one 30-minute session), and shaping was completed in two full training sessions (so, after the 3rd day). 

However, one difference that I must note in comparing magazine and shaping durations between the two rats is that in Mitzy’s case, I more or less combined the two processes in our first training session. That is to say, by the end of Day 1, I had already begun to shape Mitzy to touch the bar, and I stopped reinforcing behaviors having to do with the hopper. I am glad that Devon recommended this approach, because I believe that it accelerated Mitzy’s rate of learning, and it also allowed both associations (i.e. sound/food and bar press/food) to become strengthened simultaneously.  

Shaping the two rats was very similar in terms of the successive approximations that I reinforced. That is to say, after “practicing” with Sniffy, I used the same procedure to shape Mitzy: reward for sniffing the back corner, reward for rearing above the bar, reward for touching the bar with hand or nose, reward for touching the bar with both hands, etc. Both rats emitted similar behaviors of the response class (i.e. group of movements similar to bar pressing), even though they varied in the time required to form a strong association.

One thing that I could not learn from shaping Sniffy was how to deal with unwanted or “extra” behaviors. Although Sniffy would rear up or sniff the bar every now and then in his VR training, he would only receive a reward for a normal bar press. On the other hand, Mitzy demonstrated a large amount of such behaviors, and they would be reinforced as long as the bar was depressed when she was doing them.

Additionally, Sniffy very rarely became distracted for more than ~20 seconds at a time, so training him did not shed any light on what I should do when Mitzy decided to wander off for 3 minutes at a time. Mitzy’s cumulative records, therefore, display somewhat less steady rates of response than do Sniffy’s. (However, we are also comparing FR and VR schedules, so Sniffy’s should appear more steady).

Related to this is the fact that Sniffy could be trained for 2 hours in a row, whereas Mitzy often lost motivation after ~20 minutes. This, however, did not necessarily impair what I observed in training the rats or what I learned from the process, but it is an important difference in the two processes.

Lastly, a major difference between the two extinction processes was that Sniffy does not feel frustrated as a live rat likely feels after she is no longer rewarded for her instrumental responses. Although I saw somewhat of an extinction burst from Sniffy, his overall extinction results were not caused by any element of frustration as were Mitzy’s.

Overall, I appreciated having the opportunity to practice on Sniffy before training my live rat. The comments in the book and in the program were very helpful, and I learned a great deal from shaping in particular. Seeing Sniffy’s rate of learning in the shaping process allowed me to modify my own technique—how “picky” I should be in reinforcing behaviors and at what rate, how quickly I should administer a food pellet, and so on.

In conclusion, what it really comes down to is this: Mitzy is way cuter than Sniffy.

The End.

Sniffy (AKA Mitzy 2.0)

Meet Sniffy

Magazine Training + Shaping

Goal 1: Magazine train Sniffy. Use a classical conditioning procedure to form an association between a consequence and a secondary reinforcer (some stimulus that it typically neutral). In this case, I reinforced Sniffy with food pellets in such a way that allowed him to learn the association between the food sound and the reward of a food pellet in the magazine.

Procedure: I completed magazine training by delivering a food pellet whenever Sniffy approached the food hopper, allowing him to wander for a bit before giving him the next pellet.

Discussion: At first, Sniffy would not regularly go to the magazine when I administered a pellet. He might continue walking around or exploring whichever part of the box he was in. After some time, I would administer a pellet only when he was very close to the magazine or was heading directly to it, and he began to more quickly check the magazine after hearing the food sound.

Goal 2: Shape Sniffy to produce the desired instrumental response (bar pressing) by reinforcing successive approximations of this behavior.

Procedure: I completed shaping by gradually reinforcing behaviors that resembled or led up to bar pressing. At first, I would administer a pellet when Sniffy approached the back corner (magazine or bar), and as he began to form an association with general behaviors in the back corner, I narrowed the range of behaviors such that he would then have to rear above the bar, and then sniff, touch, press (etc) the bar in order to receive a pellet.

Discussion: Overall, this process took a lot longer than I thought it would, and it required very careful observation on my part. I noticed that if I allowed too much of a delay to occur between the target behavior and the food reward, Sniffy would often reproduce another behavior he had done since then (that wasn’t desired), and the Bar-Sound and Action-Strength measurements would decrease.  

This figure contains the cumulative records for Sniffy's magazine training (first bar, small portion of second bar), and shaping for bar pressing (starting from yellow highlighted "CRF Press Bar" section).


Variable Ratio + Extinction

Goals 3 & 4: Place Sniffy on a Variable Ratio schedule, and ultimately extinguish the behavior through extinction.

Procedure: After Sniffy had formed very strong associations between the bar and the food sound (“Bar Sound” meter) and between bar pressing and the food reward (“Action Strength” meter), I placed him on a variable ratio schedule. I began with VR5 and worked up to a VR50. Then, I simply set up the program to run extinction, and ran it so that Sniffy received no rewards for his bar presses.

Discussion: Sniffy’s rate of learning was very quick using VR schedules. Compared to his behavior in shaping, he took far fewer and shorter breaks (grooming, exploring), and his focus was more or less fixated on the bar throughout the course of training. After his first VR schedule (VR5), he was pressing the bar very continuously and quickly for the rest of training. 

This figure contains the cumulative record for an early part of Sniffy's variable ratio training. This VR5 record demonstrates an increase in the rate of response (graph becomes more steeply sloped).

In extinction, Sniffy initially would press the bar repeatedly, and take his hands off the bar to sniff inside and around the food hopper. Additionally, Sniffy’s rate of bar pressing seemed to increase in the first few minutes of extinction (extinction burst). However, as the association was further extinguished, he would typically press the bar a few times (to no avail), and come off of it to groom himself or sniff another region of the op box. I stopped the session after a 5-minute interval had passed in which Sniffy had pressed the bar only once. 

Extinction: Not Just for Dinos

When Mitzy started the session, she immediately began to press the bar rapidly and persistently (as was reinforced in previous training). Throughout the session, she also demonstrated a great deal of spontaneous recovery of previously-enforced behaviors. As I mentioned before, bar biting was a problem throughout the two weeks of training, but in extinction, she displayed a dramatically greater rate of such behaviors. Mitzy was biting the bar, holding it down, pressing up on it with her nose, and even biting the magazine as she was initially shaped to do in the first days of training.

In addition, she appeared quite agitated at the beginning of both extinction session, jumping and rearing around different areas of the box. However, after some time (~10 minutes), she would begin to more calmly explore the box, stopping to sniff or groom. This suggested to me that the instrumental response was likely on its way to extinction (or at least, was getting closer).

The following video was taken in the 10-15 minute interval on Day 1 of extinction:

Although I did not see full extinction of the behavior, I believe that it would have been (mostly) reached if we extinguished for more than 2 days.

As you can see from the graph, Mitzy showed a dramatic decrease in response for the first 20 minutes of each extinction session, and then experienced a bit of an extinction burst in the 20-25 minute interval. In addition, she began her second day of training with a fairly large number of bar presses, but demonstrated an overall decrease from the previous session.

Extinction Day 1 Cumulative Record

Extinction Day 2 Cumulative Record

Mitzy Results: Weight & Bar Press Graphs

Average Number of Responses per 30 minutes
Shaping, Fixed Ratio, and Variable Ratio Schedules

Weight Chart
Target Weight: 206 grams

Date	Weight (g)	Food (g)
12-Sep	237.0	3.3
13-Sep	225.6	4.0
14-Sep	219.9	5.3
15-Sep	219.0	4.5
16-Sep	214.4	6.8
17-Sep	214.3	4.8
18-Sep	207.9	6.1
19-Sep	205.7	7.2
20-Sep	210.0	6.0
21-Sep	212.7	3.8
22-Sep	209.6	5.0
23-Sep	211.0	3.3
24-Sep	206.6	4.6
25-Sep	206.3	7.0
26-Sep	206.5	6.5
27-Sep	206.9	5.7
28-Sep	201.2	7.9
29-Sep	204.3	7.0
30-Sep	203.6	6.9
1-Oct	201.6	8.4
2-Oct	203.5	6.5
3-Oct	200.5	8.7
4-Oct	202.9	8.8
5-Oct	204.3	Free feed

Trouble in Paradise

Rats Gone Wild!

As noted throughout my posts, one of the problems I had in training Mitzy was "extra" behavior and effort that she put into her bar presses. In early FR training, Mitzy would put increasing effort into each of her presses. For example, in FR3, she would (1) press the bar normally, receiving no food, (2) press and hold the bar for a few seconds, still receiving no food, and (3) press. hold, and bite the bar, finally receiving her food. I was worried that, as this behavior continue to earn her rewards, she would form an association between her "added effort" and the consequence (receiving a pellet). 

After talking to other students, I came to see that variations of this behavior are a common problem in training, and we briefly discussed this in class. Overall, due to Mitzy's gradual increase in response rate and her large number of bar presses per session, I concluded that this was not a major interference in her learning process. However, I think that if I had seen this pattern of behavior sooner (e.g. in a much lower FR schedule), I likely would have switched to manual reinforcement on whatever our current schedule was. This way, I could have reinforced every 3, 5, 7, etc bar presses only, rather than letting the operant box automatically reinforce any behavior that resulted in a bar press. 

Day 13: VR10

10/3/12 
(notes taken on day of training)

-------

Goal:
Attempt a variable ratio schedule for the first time. I hope to see a steadier pattern of responding than demonstrated in FR, as well as a decrease in predictable pauses.

Discussion:

I was somewhat surprised by the results of this schedule. Although Mitzy's response rate was often quite steady, I did not perceive it to be very different from her higher ratio FR schedules. She took several long breaks (e.g. a few minutes at a time) to explore the box, groom, etc. Although she stayed fairly motivated throughout the session and had a very high number of total responses, her cumulative record displays [what seem to be] post-reinforcement pauses.

In addition, I was not sure how to start Mitzy on her first day of a variable ratio schedule. After leaving off on FR12, I was unsure as to whether I should reduce the # of responses required (e.g. move from FR12 to VR5), or simply continue from there (e.g. from FR12 to VR12). After consulting my class notes, I decided that it was not necessary to reduce the ratio dramatically, as she had already learned to press the bar repeatedly for a treat, and had been training for almost two weeks at this point. However, as the results were not what I expected, if I were to try this again, I would probably reduce the ratio when starting a VR schedule.

Overall, Mitzy appeared to do well on the VR schedule, and the number of bar presses was a record high (yay!). I would have liked to spend a few more days on different VR schedules in order to more clearly see a difference in the cumulative records between FRs and VRs, but alas, time for extinction!

Bar Presses: 618
Reinforcements: 61
Run Time: 30 minutes

Fixed Ratio Adventures

9/23: Day 5, FR3
9:10- 9:40 AM
Noted behaviors: pressing and holding bar, biting bar, checking magazine between presses; beginning to press more quickly; often presses in quick succession without going elsewhere or checking magazine; overall, high number of responses
Run time: 30 minutes
Bar presses: 458
Reinforcements: 149

9/24: Day 6, FR3 cont...
8:11- 8:31 AM
Noted behaviors: continues to put increasing amounts of effort into each press when she is not rewarded; some long pauses within her 3 presses; starts pressing 3 in a row fairly consistently at ~10 min.; appears somewhat nervous today (jumping at any sounds, frequent grooming, moving around box)
Run time: 20 minutes (seemed to lose motivation)
Bar presses: 206
Reinforcements: 68

9/25: Day 7, FR5
8:06- 8:27 AM
Noted behaviors: started training well, pressing fairly quickly and consistently; some bar biting throughout; beginning to see longer post-reinforcement pauses (grooming/ walking away after receiving her reward); lost focus after ~18 minutes
Run time: 21 minutes
Bar presses: 336
Reinforcements: 67

9/26: Day 8, FR7
8:05- 8:28 AM
Noted behaviors: pressing in fairly quick succession, very rarely checking the hopper between presses; biting the bar for some of her presses; apparent high motivation, held focus for most of the session
Run time: 23.5 minutes
Bar presses: 438
Reinforcements: 62

9/27: Day 9, FR10
8:09- 8:29 AM
Noted behaviors: some instances of biting; pressed bar with nose quite often (which was still reinforced automatically); long post-reinforcement pauses; on a few occasions, she demonstrated behavior that I haven't seen since shaping, such as biting the magazine, trying to squeeze into it, etc
Run time: 20 minutes
Bar presses: 456
Reinforcements: 45

9/28: Day 10, FR10 cont...
8:05- 8:26 AM
Noted behaviors: Checked magazine in between presses far less frequently as in lower ratio FRs (learning to press consecutively until she hears the sound); still biting the bar, trying to press it upward from below, holding bar for several seconds; large post-reinforcement pauses, but appeared fairly motivated; stopped early because of high number of bar presses, and a long grooming session around 20 minutes
Run time: 21 minutes
Bar presses: 513
Reinforcements: 51

10/1: Day 11, FR10 cont...
8:10- 8:33 AM
Run time: 23 minutes
Bar presses: 465
Reinforcements: 46

10/2: Day 12, FR12
8:04- 8:26 AM
Noted behaviors: Demonstrated similar behaviors (including distracted behaviors, bar biting, etc) and the same high rate of response seen in FR10 schedule.
Run time: 22 minutes
Bar presses: 501
Reinforcements: 42

------------------------

Overall, I have been very pleased with my rat's rate of learning. In general, she seems to be very motivated (at least for the first 20 minutes of training). She typically demonstrates the instrumental response fairly quickly (i.e., many presses in a row without walking away or checking the hopper). As predicted, I am observing longer post-reinforcement pauses as I continue to stretch the ratio. Lastly, many of her "presses" result from her biting the bar, and she tends to increase such behavior the higher the ratio gets.

Day 4: FR2

9/22/12 
(notes taken at time of session)

Goal:
Start a non-continuous FR schedule after shaping + FR1.

Procedure:
Begin stretching the ratio in Mitzy's training by moving onto an FR2, observing progress made since shaping, development of a post-ratio pause, etc...

Results + Discussion:
For approximately the first half of the session, Mitzy demonstrated a chain of behaviors between bar presses--she would press the bar quickly, check the magazine, and on her second press she would hold the bar for several seconds before releasing and obtaining her reward. Dr. Trench had previously told the class that it was okay if the rats checked the magazine in between presses, so this behavior did not seem out of the ordinary or worrisome. However, I was somewhat concerned that Mitzy's response of putting more "effort" or force into her second bar press (either holding down the bar or biting it) would become reinforced and would thereby become an accidentally-shaped behavior.
Mitzy continued to check the bar between presses, but demonstrated a fairly consistent rate of response. She did not appear to be overly distracted for the majority of the training session, but would lose focus for ~30 seconds every few minutes or so (as seen in FR1 training). However, she began to jump around and rear up in various places in the box ~18 minutes in, and seemed to lose motivation at this point. I stopped the session at 25 minutes.

Mitzy pressed the bar 226 times and was reinforced 113 times. Her weight after training was 209.6 grams.

This video was taken approximately 17 minutes into the session, and demonstrates some of the distracted behavior mentioned above. Note the quick peek into the hopper between presses.

Day 3, Shaping Continued

The notes for the following were recorded yesterday

-------------------------------

Training Day 3: Shaping Continued
9/22/12

Goal: to continue and complete shaping my rat to press the bar

Procedure:
To further shape Mitzy to associate ONLY bar presses with food delivery, I became more selective in which behaviors I would reinforce over the course of the session. I began by delivering food when Mitzy sniffed the top of the bar (sniffing the bottom did not earn food), touched the top with one or both paws, or reared just above the bar. As the session continued, I began to extinguish sniffing behavior, and would only give Mitzy a treat if she touched the bar with one or both hands.

Results:

Mitzy displayed a strong association between touching the bar and receiving a food pellet. She would test out certain behaviors around the bar, and quickly check the magazine for a reward. She did very well in learning just to touch the bar with her paws, rather than explore or sniff the bar. By the end of the session, she was pressing the bar consistently, and required no manual reinforcements. She received 34 manual reinforcements (until ~20 minutes in), and pressed the bar 66 times.

Discussion:
Extinguishing "bar sniffs" required careful monitoring. There were a few instances in which I did have to back up and reinforce a bar sniff. For example, she would sniff the bar several times, check the magazine, and then lose interest for awhile (explore the front of the box, groom herself, etc). In such instances, I would give her a food pellet once she returned to sniff the bar.

Overall, though, I was very impressed by her rate of learning. She is very motivated and inquisitive, and narrowed her range of behaviors quickly in the session. By the end of the session, she did not even attempt to receive a reward from bar touches or sniffs. Instead, she would immediately press the bar, eat her treat, and repeat :). Often, she did not even move her feet at all, but would just rotate her upper body between the bar and the magazine.

Weight at the end of session = 212 grams.
Session ran from 8:35 AM to 9:05 AM (30 minutes).
Reinforcements (manual)= 34; Bar presses= 66.

Magazine Training + Shaping (Day 1)

The notes for the following post were recorded on the day of my lesson with Devon.
-------------------------

Training Day 1: 9/19/12

Goal: To magazine train and begin shaping. In magazine training, I am attempting to classically condition an association between the sound of the food magazine and the delivery of food (reward). In addition, I began shaping to form as association between being near/ touching the bar with a reward. This will prepare Mitzy for further shaping in which I will reinforce bar presses alone.

Procedure:
Mitzy was put on a food deprivation diet to maximize motivation while training. She was deprived to approximately 85% of her starting weight, and weighed 110 grams today (4g above target). For Day 1 of training, I began by delivering a food pellet when my rat was turned toward the magazine. I advanced by reinforcing approaches to the magazine/ bar, and would give a pellet each time that she ate her food, pulled her head out of the magazine, and continued to explore this back corner area.

Results:
Mitzy quickly began exploring the box, giving me an opportunity to administer a pellet when she first directed her attention to the magazine. After a few tries at this, she began to notice the food coming into the magazine. For most of the training session, she would eat her pellet and go explore a different part of the box. However, after a while she began to demonstrate an association with the food delivery and her presence in/ exploration of the back corner, and began to spend much of her time sniffing around and rearing between the magazine and the bar.

At this point I began shaping. I began pressing the button when Mitzy reared above the bar, sniffed the bar, and touched the bar, and stopped reinforcing behavior such as approaching or looking at the magazine.

Discussion:
I was somewhat surprised by how quickly Mitzy demonstrated an association between the sound of the magazine and food delivery. Although she would not go to the magazine right away for about the first half of the session, after ~15 minutes she would retrieve her food almost as soon as she heard the magazine sound. In addition, she appeared to be highly motivated by the food reward. Although she is a few grams above her target weight, I will consider keeping her at this current body weight, as it does not seem to impair her training motivation.

The shaping process seems to be off to a good start. By the end of the session, she was spending most of her time in the back corner (rearing, sniffing, etc). She still continues to sniff around the front of the box on occasion, and takes somewhat frequent grooming breaks.

I manually reinforced Mitzy 72 times, and she pressed the bar once.
The session ran from 5:40 PM to 6:10 PM.

Post 1 Questions

What was your initial impression of rats, in general, before your first Learning lab? 
I wasn't particularly fond of rats, but neither was I scared or grossed out by them. I know some people who have pet rats, so I suppose that had helped me not to see all rats as being dirty.

Did this impression change by the end of the lab? Why or why not?
Holding my rat made me feel pretty warm and fuzzy about her, so I suppose that my impression of rats improved a little bit. Also, I really like how curious they seem to be about their surroundings.

Were you able to hold your rat?
Yes, and she was very calm. She barely squirmed, and seemed to want to nestle underneath my hand for protection.

Do you have any ideas for a name?
Not yet-- I usually name everything Timmy, but my rat is deserving of a female name. TBD.

UPDATE: I have named my rat Mitzy. :) See below for a picture of her being inquisitive and cuddly.