"Learning by Success" - From Thorndike's Puzzle Box to "Houdini" Horses. A short gallop through learning theory.

Picture credits: Kim Goertz
Photo by Kaisa Rundberg

I dedicate this blog to Sandra Schneider and her embodiment of learning theory in horse training

There are many different forms of learning and all of them help living beings to deal successfully with both, their physical and their social environment. The laws of learning are something like the “law of gravity of psychology”, they have universal validity and apply to the animal world in general. Regarding training and educating our four-legged friends, the fact that humans and horses learn similarly is quite advantageous. Moreover, horses are extremely "docile" or teachable, which is confirmed by the fact that they can learn to communicate even with us humans, a species quite different from them. Of course, when investigating and understanding learning processes, we must consider the nature of a species as well as the environment in which it has developed and is adjusted to, the so-called habitat. Nonetheless, within this framework, the laws of learning are universal.


Why do living beings learn? Well, it is quite simple, it is a question of survival. The phenomenon that  precedes the ability to learn is that of "approach-avoidance". It means that living beings seek out  situations that are favorable for survival and, not only that, also for well-being. This is shown by a  simple experiment some of us may remember from our high school biology course, the "hay  infusion": In an experimental bowl or dish, the protozoa or other simple multicellular organisms derived from the hay infusion prefer to stay where the life conditions are good, where the temperature is pleasant, and food is available. At the same time, less favorable regions are avoided,  e.g. where it is colder or no food can be found. The "little animals" also behave in this way when the  differences are not that extreme, they always prefer the regions where “life is good”. In other words,  even the most simple organisms "know" where the living conditions are optimal and, moreover, they  actively migrate there! Thus, they exhibit active, goal-oriented behavior in order to optimize their  living conditions. This principle of not only surviving but surviving as good as possible is in fact a  principle of evolution and has significance for how our brain works and how we see the world.  Antonio R. Damasio, probably one of the most influential neuroscientists of recent decades,  described the relevance of this principle for human existence as early as 2003 in his book "Looking for  Spinoza: Joy, Sorrow, and the Feeling Brain”.


Photo by Agnete E. Kristoffersen


Photo by Agnete E. Kristoffersen

There are a number of different forms of learning, e.g. Habituation (getting used to repeatedly  presented, non-painful or non-toxic stimuli), Sensitization (increasing response intensity to repeatedly  presented aversive stimuli), Systematic Desensitization (method from psychotherapy, which is akin to  guided habituation), classical conditioning (Pavlov's dog or why we always get hungry at around  lunchtime), operant or instrumental conditioning (Skinner's pigeons and rats trained in a Skinner  Box), observational learning, and latent learning. In today’s blog, I will focus on "instrumental"  learning (I may come back to other forms of learning in some of the next blogs).  


Why a blog on "instrumental" learning, which also includes the so-called "operant conditioning"?  Well, there is probably no other topic in the psychology of learning that is and has been the subject of  dispute and misunderstandings as instrumental learning and operant conditioning. However, it is  exactly this form of learning that plays a major role in horse training. Therefore, it is useful and  sensible to take a closer look. Let’s start with the history of learning theory, thus with the question, “who invented it”?


The absolute pioneer of learning research was certainly Edward Lee Thorndike (1874 – 1949), who  described the "Law of Effect" in his dissertation from 1898. He placed cats in cages that could be  opened by a rather complicated mechanism and placed a food dish outside the cage. The cats tried to  open the cage by trial and error in order to reach the desired food. They generally succeeded at some  point; cats are after all very clever. But what fascinated Thorndike in particular, was the fact that the  cats were able to open the cage in less and less time through optimizing their actions. In conclusion, the cats learned from their "success": The sooner they managed to open the mechanism, the sooner  they reached the food. Thus, they learned by the effect of their actions. You can find an interesting  video describing the historical experimental set-up under this link:

    The Puzzle Box

Thorndike was thus the first to systematically investigate, document, and quantify the phenomenon  of "learning by success". In conclusion the "law of effect" is central when it comes to initiating and  maintaining actions. Therefore, an action becomes an "instrument" to achieve a desired effect. This is  why this form of learning is also referred to somewhat abstractly as "instrumental learning". Thus  goal-oriented behavior became the focus of scientific observation and investigation. This approach  will later turn into a "school" of psychology, called "behaviorism", even though I doubt that Thorndike  would have identified himself as behaviorist.


Do Thorndike's laboratory experiments with cats really have significance for us horse lovers? Oh  yesss, they do! There is nothing cats can do that horses can't. A study by Konstanze Krüger, Laureen  Esch & Richard Byrne, published in the highly respected journal PlosOne in 2019, confirms what  we've probably all know already: Horses are masters at opening doors, be it that of their own stall,  their mates' stall, the feed room, or the gate at the paddock. The authors report that they have not  found a single locking mechanism that horses could not open. The parallel to cats is of course not  coincidental, because, as already mentioned, the laws of learning are universal throughout the  animal realm! Thus, even horses wo are anatomically not necessarily perfectly suited for this task,  develop an amazing skillfulness as "Houdinis" (here are two interesting videos documenting  astounding dexterity:

    Horse Escape    Horse Escape 2

Is also a Houdini, but mostly nice and stays where he is supposed to.


Photo von Frauke Musial


Photo b< Frauke Musial

However, the father of learning theory, as we know it from behavior analysis and e.g. from the socalled "clicker training" in animal training, is Burrhus Frederic Skinner (1904 - 1990). Skinner studied  in detail how animals (and humans) learn from success (see also "Skinner Box") and defined the laws  of "operant conditioning". Skinner was also involved in social philosophy and considered free will to  be an illusion. His book "Beyond Freedom and Dignity", published in 1971, was quite controversial  and his approach to understanding human behavior was interpreted as "social engineering". Aldous  Huxley's critical and dystopian vision of such a form of society and his criticism of the emerging behaviorism found expression in his novella "Brave New World" as early as 1932. Today, the question  of free will is still under debate in the neurosciences.


Skinner considered (not only) human action to be directly dependent on the consequences of  previous actions. He summarized this approach in a theory which he called the principle of  reinforcement and punishment: If the consequences of an action are unfavorable, there is a high  probability that the action will not be repeated; if the consequences of an action are pleasant/ favorable, the probability that this action will be repeated increases.


The principle of reinforcement is central to the understanding of how learning through effect or success works. In general, it should be said that the laws of learning are still valid today, but that they are seen more in relation to other aspects of experience and behavior. These include emotions (e.g. fear vs. elation), motivation (e.g. hunger, sexuality), but also genetic predispositions. Therefore, some tasks are easier to learn by some species than others, some can be more challenging depending on the living environment for which that particular species is adapted to. So, now we finally approach the principle of “operant conditioning”!


Central to operant conditioning are the concepts of "reinforcement" and “punishment” of a  behavior. Reinforcement causes the behavior to occur more often, while punishment causes a  behavior to occur less frequently or suppresses it at all. Since we always learn by "success" in the  realm of instrumental learning, the outcome of a behavior or the consequences that follow play a  decisive role. Everything that increases the probability of a behavior occurring is called a  "reinforcer" in learning theory. Anything that decreases the probability of occurrence of a behavior is  a “punishment”.


With regard to reinforcers, a distinction is made between primary reinforcers such as e.g. food or sex, and secondary reinforcers such as money, which in turn can be used to access primary reinforcers. Primary reinforcers are generally of biological significance, at least for most animals (this maybe  slightly different for humans). In learning experiments or animal training, a distinctive signal that is  easily distinguishable from ambient noise is often used to “announce” the occurrence of a primary  reinforcer. This can e.g. be clicker, or a whistle as is used for training dolphins. Before the actual  training, the animals learn that this particular sound announces a primary reinforcer, such as a e.g. the fish for the dolphin. The whistle is therefore a signal for the Dolphin that a primary reinforcer (the  fish    ) is available.

By the way, we often use secondary reinforcers in horse training. Many of us have heard or used a  "good boy", which becomes a secondary reinforcer the moment it is followed by a pleasant scratch in  the right place. The same holds true for a "Priiiiima" or in good Norwegian "fliiiiiiink gutt" (good boy) that some may also have heard as a secondary reinforcer    .The use of a secondary reinforcer can  be quite useful when training an animal over a distance. Nonetheless, I will not go deeper into the  realm of Clicker Training in this Blog, because there are plenty of good instructions and instructors  available, so I don't need to dive deeper into this particular topic at this point.


Back to the enthusiastic early days of experimental/empirical psychology which coincided, probably  not accidentally, with a period of great discoveries in physics and chemistry between the First and  Second World Wars. The decades after the Second World War were marked by enormous technical  developments as well as social changes. The Russians "catapulted" the first man, Yuri Gargarin, into  space, the Americans sent men to the moon with the Apollo program, and the behaviorists, but also  many other researchers in experimental psychology, were dedicated natural scientists and fascinated  by formulae. And that's where the misunderstandings in learning theory began....


According to the laws of reinforcement, there are two types of behavioral consequences that lead to  a behavior being performed more often. A reward, meaning something pleasant is added, or  something not so pleasant, e.g. a loud noise, is removed or subtracted. If you love formulas, which  was obviously the case with the "old" behaviorist’s, you will express an addition as "plus +" and  subtraction as "minus -". With punishment (the probability of occurrence decreases) it is the other  way round: if something not so pleasant is the consequence of a behavior, the behavior will be shown  less often. This situation is called positive or Type I punishment, such as the pain of touching the hot  cooker top. In this case something unpleasant is added (+). Consequently, there is also a negative or  Type II punishment, when a pleasant stimulus is removed, i.e. subtracted (-). Examples for this are  when a child is deprived of television as an educational measure, or when you get a speeding ticket, a  case, where money is subtracted/removed because of driving too fast    .

And here comes the problem: If the reinforcement (increasing the probability of the occurrence of a behavior) is applied by taking away an unpleasant stimulus (it does not always have to be something terribly unpleasant, it can also simply be something "annoying"), then we do not speak of “removal reinforcement" (which would actually be logical), but of "negative reinforcement" because something is subtracted (-). Nothing has probably caused more confusion in terms of learning theory and operant conditioning as the term negative reinforcement! Negative reinforcement increases the likelihood of a behavior just as positive reinforcement does. It is as such something desirable to the individual and represents in no way something "bad" or "negative". On the contrary, when something unpleasant or even noxious is removed, then the consequences are experienced as pleasant and good, which is “positive”. It must be said that an "unpleasant stimulus" in daily life - as already mentioned - does not necessarily have to be something toxic such as pain or a very loud noise. Something "annoying" like, for example, waving a flag or even a slight, but constant tapping with a whip during horse training is quite sufficient. The stimulus only needs to be unpleasant enough that one would like to switch it off.


Thus, the main definition of "reinforcement" is anything that increases the probability of the occurrence of a behavior. In other words, any human or animal exposed to a learning situation with "reinforcement" finds this situation desirable/pleasant. No researcher today would use the terms "positive" (+) and "negative" (-) reinforcement. More intuitive and thus much better would be the term "subtraction reinforcement" or "removal reinforcement” instead of negative reinforcement as suggested by Andrew McLean and others. However, I fear it is way too late in the history of learning psychology to introduce a new terminology. Table 1 summarizes the principle of reinforcement and punishment using the example of horse training. A treat is a reward and thus an addition (positive) reinforcement, the removal of an "annoying" leg pressure is a removal (negative) reinforcement. The use of a whip in painful way is an addition (positive) punishment (Type I), a removal (negative) punishment (Type II) would be e.g. the isolation from the herd.


Photo by Agnete E. Kristoffersen

Table 1: Reinforcement and punishment using the example of horse-training.

I have "marked" the left-hand area of the table because I find it of particular importance that we  focus as much as possible on behavior whose likelihood of occurrence we want to increase in horsetraining. Thus, we want to work within the area of reinforcement, be it "removal-reinforcement" or "addition-reinforcement". Punishment should be avoided to the largest extend possible. Punishing a  horse by, for example, taking away food or isolating it ("removal punishment" / punishment Type II)  is not only unethical, but also pointless because the horse will not understand the connection  between punishment and behavior. Positive punishment, i.e. "addition punishment" in our  terminology, should only be used in extreme exceptional cases, e.g. when there is no other way to  stop a dangerous behavior.



All possible combinations of addition and removal reinforcement as well as punishment are always  associated with emotions (Table 2). It is of utter importance to bear in mind that positive emotions are only generated in the reinforcement conditions (addition as well as removal reinforcement),  while punishment always leads to negative emotions. Since we want to create a friendly atmosphere  in training, it is essential to avoid punishment to the largest extend possible. Especially as  punishment does not explain which behavior is desired, thus no alternative behavioral option is  offered to resolve the negative feelings. Admittedly, there are extreme cases where punishment is  the last option to avoid a dangerous behavior. Nevertheless, for successful training as well as for  ethical reasons, type I punishment ("addition punishment") should be avoided as far as possible and  only used in exceptional cases for safety reasons. Type II punishment (removal punishment) is  unethical and has no place in horse training.

Table 2: Reinforcement is generally associated with positive emotions, while punishment is always accompanied by negative emotions. Negative emotions are impediments to successful learning and should be avoided. Type II punishment is totally inappropriate and unethical in horse training

A technique that helps to increase positive emotions during training even more is the use of  combined reinforcement. In horses, the region around the withers is an important area relevant to  social bonding. When horses engage in friendly social interactions, they prefer to groom each other  in this area (Figure 2). A study by Thorbergson and colleagues (2016) showed that scratching the  withers or the surrounding area is associated with a relaxation response (lowering of the heart rate).  Thus, the wither-scratching is a primary (i.e. biological) reinforcer. Since this region is also easily  accessible from the saddle, we have access to a primary reinforcer, an intrinsic reward similar to a  treat even while riding. Combined reinforcement while riding could e.g. be that the correct reaction  to a leg aid (the leg pressure is taken away after the correct reaction has been made and represents "removal reinforcement"), can then be reinforced once again with a wither scratch and thus through  an "addition reinforcement". How ingenious is that   .

In horses, the region around the withers is an important area relevant for social bonding. When horses engage in friendly social interactions, they like to “groom” each other in this area. Scratching at the withers leads to a relaxation response (Thorbergson et.al. 2016) and can thus be used as a primary reinforcer for horse training, even under the saddle.

Regarding reinforcement, whether it is "addition", i.e. positive reinforcement, or "removal", i.e.  negative reinforcement, it is absolutely and fundamentally important (superlatives are really appropriate here) that it takes place IMMEDIATELY after the desired reaction is displayed. If this is  not possible due to the situation, e.g. because of distance, then a secondary reinforcer such as a  word or the clicker should be used to ensure the proper timing of the reinforcement. If the timing is  delayed, learning from “success” becomes impossible which can have dramatic consequences for the  behavior and psyche of our four-legged companions. Horses that are trained without ever  experiencing a proper and timely removal of the aids, will eventually become unresponsive and dull  to the aids. Those horses often get the reputation of being lazy… even though the reason for their  unresponsiveness is not their fault, but improper removal of the aids and consequently habituation  to the riding signals. These horses learn that the “aids” are meaningless, because whatever they do,  the “aids” don’t go away so for the horse, the best solution is to get used to the situation and ignore  them. This is also a type of learning and called “habituation”. In the worst case and especially in cases  of very strong or toxic signaling e.g. using painful stimuli, the horse can end up in a state of  helplessness. This is not only relevant with regard to animal welfare, it is also tremendously difficult  to treat and rehabilitate such a “shut down” horse. However, am also firmly convinced that the vast  majority of people reading these lines and who are interested and engaged in the topic of learning are aware of this challenge and are striving to do it right!

Learning in horse training is often based on "removal-reinforcement". Every time the term "pressurerelease" is mentioned, we are talking about "removal-reinforcement". This applies to leg and rein aids as well as to the use of a flag when lunging. In general, the goal should be to use the weakest, most subtle signals possible, which brings us back to the topic of fine aids. That's why you should always start with very weak signals and only increase the pressure slowly and systematically until the horse shows a response. The most important factor, however, is to remove the pressure immediately when the correct reaction is shown. Once the desired response is established, the signals (aids) can be more subtle until a minimum of pressure is sufficient to trigger the desired reaction. "Removal reinforcement" works very well in the absolute minimal range of signaling. Remember the protozoa from the hay infusion, which can react to the smallest differences in their "habitat". Horses are extremely sensitive animals and excellent "students". They can recognize and react to even the lightest stimuli. It is not the intensity of the stimulus that is fundamental for learning, but the clarity, unambiguity, and consistency of our reactions. At this point, it is worth noting that learned reactions to clear signals move our interaction with our four-legged friends at some point from "learning" to "communication". And is not that the most beautiful thing which can happen, that we understand each other only by the thought of it and communication becomes a dance?   


Horses in a group communicate with each other all the time. They themselves use "removalreinforcement" quite a bit. If, for example, a group member is asked to clear the way, then threatening is only lightly implied e.g. by pinning the ears backwards. If the horse who is addressed reacts accordingly, the threatening gesture is immediately withdrawn. This type of intraspecies communication leads to a reduction of intraspecies aggression and thus represents a decisive evolutionary advantage. If the individuals of a group had to constantly fight each other out of the way, the risk of injury within the species would increase substantially and this would be unfavorable for the survival of the entire species. Communication is therefore fundamentally important and takes its starting point in instrumental learning. Just for this reason alone, it is extremely important that foals grow up in a group of horses, preferably with other foals, so that they have the opportunity to learn from other horses in order to be able to communicate with horses in a species-appropriate way later in life.

"Subtraction-reinforcement", i.e. negative reinforcement, is rather the rule in nature. Like the little animals in the experiment with the hay infusion avoid an unfavorable environment, all living beings aspire to optimize their life circumstances, be it the physical or social environment. In doing so, the degree of "unpleasantness" of environmental challenges may well vary. If a group of horses is standing in a downpour and then decides to seek refuge in their shed, this is just as much "removalreinforcement" (the wetness from above is removed) and thus negative reinforcement (and probably also social learning) as the flight-reaction in panic from an anticipated (and probably imagined) sabertoothed tiger in the bushes. However, the fear of a predator in the bush is a much more dramatic aversive emotional state with high biological significance compared to the unpleasant stimulus of rain. The relief and thus the strength of the "removal-reinforcement" after the flight reaction is correspondingly all the greater. An escape reaction is a very impressive case of negative reinforcement ("removal-reinforcement") with a very aversive stimulus and a powerful avoidance reaction that is likely to constitute a "one trial learning" which can be very difficult to extinguish once it happened. This example also emphasizes that what we are good at learning is also dependent on the habitat we are adjusted to and have developed in as a species. For horses, the fear of a potential predator in the bushes is fundamentally relevant, while we humans are more afraid of the “crocodile under the bed”   . This fear may have developed due to the fact that our ancestors slept in trees at night because predators on the ground were more active at nighttime.


And let's face it, our four-legged companions are true masters of operant/instrumental conditioning, we just have to look closely. The unpleasant and rather loud uproar when we arrive with the bucket of concentrated foodd stops immediately when we give it to them. And guess what, our horses have negatively reinforced us for setting down the food bucket: The annoying noise stops and is thus removed. The aggressive rummaging through the pockets stops (at least for a moment) when we hand over the carrot. Again, our four-legged friend has negatively reinforced us. Moreover, we have at the same time also positively reinforced the behavior on the horse's side, if we were aware of it or not   .


So, this was only a small excerpt from the large topic of "learning". Learning has developed through  the interaction with the living environment, the habitat, in all species, striving not only for survival,  but also for a “good life”. Thinking it through, we may suddenly realize how strongly our lives are  determined by the laws of learning. If, e.g. I sit in the sun and find it too hot; I do as the protozoa  from the hay infusion do, I retreat into the shade enjoying a more pleasant temperature (a rare  phenomenon in the Arctic   ). Strictly speaking, I have then negatively reinforced myself. While  studying Psychology, I found the subject of learning very theoretical and a bit boring. Another 40  years later in real life, and in recent years again in the company of two wonderful rascals the size of a  mini elephant, have taught me otherwise! The laws of learning are everywhere, they are a part of  life, we just have to sharpen our attention. And once you have seen them, it is impossible to “unsee”  them again!

Recommended readings:

This blog discusses very basic facts about the psychology of learning. You can find more information  on the topic in all introductory books on psychology. That is why there are only a few direct  citations in this Blog but mainly general literature recommendations.


Andrew M. McLean 2022. Modern Horse Training: Equitation Science Principles & Practice, Volume 1.  Eds. Nicki Stuart & Sophie Wyllie, Equitation Science International, Tuerong, Australia.  Chapter: Correct use of operant conditioning pp 66-83.


Paul McGreevy, Janne Winther Kristensen, Uta König von Borstel, and Andrew McLean 2018.  Equitation Science 2nd ed. John Wiley and Sons Ltd, Hoboken, United States. Chapter 5  Associative Learning (attractive stimuli) pp 81-100; and 6 Associative Learning (aversive  stimuli) pp 101 – 120.


For German speaking readers:


Konstanze Krüger & Isabell Marr 2022. Forschung trifft Pferd. Kosmos Verlag, Stuttgart. Kapitel 6.  Lernen – Pferde lernen lebenslang. 118 – 149.


Margit H. Zeitler-Feicht 2015. Handbuch Pferdeverhalten: Ursache, Therapie und Prophylaxe von  Problemverhalten (Reiterbibliothek) Verlag Eugen Ulmer. Kapitel: Lerntheroretische  Grundlagen, Seiten 159-167


References:

Damasio, A.E. (2003) "Looking for Spinoza: Joy, Sorrow, and the Feeling Brain”. Harcourt.


Krueger K, Esch L, Byrne R. Animal behaviour in a human world: A crowdsourcing study on horses that open door and gate mechanisms. PLoS One. 2019 Jun 26;14(6):e0218954. doi: 10.1371/journal.pone.0218954. PMID: 31242266; PMCID: PMC6594629. https://journals.plos.org/plosone/article?id=10.1371%2Fjournal.pone.0218954&fbclid=IwAR 3yHRR-fNloAVIYN0lRm6Bs4Q3N98ISVngJghVKyPhf1EnDF74czs5R-Ls


Horses, who open doors:


https://www.youtube.com/watch?v=pIPcwmEszkw

https://www.youtube.com/watch?v=Aseg-w0hA_c


Thorbergson ZW, Nielsen SG, Beaulieu RJ, Doyle RE. Physiological and Behavioral Responses of Horses to Wither Scratching and Patting the Neck When Under Saddle. J Appl Anim Welf Sci. 2016 Jul-Sep;19(3):245-59. doi: 10.1080/10888705.2015.1130630. Epub 2016 Mar 9. PMID: 26958705.


Thorndike, E. L. (1898). Animal intelligence: An experimental study of the associative processes in animals. The Psychological Review: Monograph Supplements, 2(4), i– 109. https://psycnet.apa.org/doi/10.1037/h0092987


A documentation of Thorndike’s Puzzle box:

https://www.youtube.com/watch?v=fanm--WyQJo