TO conclude

Reinforcement learning(RL) is the intersection of machine learning, decisions & control, and behavioral psychology. The intersection can be approached from all the three sides, and a detailed explanation is beyond the scope.

SO, I’ll try to give a short account of all the three tpyes of Machine learning. Reinforcement Learning (RL) field, I give you here an illustrative simple example to understand the concept and also what is the relationship between Supervised Learning (SL) and Unsupervised Learning (UL).

Imagine that you are teaching your child to play super mario and you have two option <RL approach> given the child the control stick and let him try to gain as many points as he can. or <SL approach> you play one level at a time and ask him to do as you did so as to reach the end of the level like you did, that is you teach him how to play one level and ask him to play that or a similar level of the game. This is SL as you will have for every input state of the road a mapping to the proper speed and points of jump and other attributes . The main point here is that you know what is the optimal thing and you try to teach your child to follow the similar steps. by doing so you teach him by examples.

In a RL setting, you just let the child try whatever it wants and you let the game give him a reward/punishment regarding the action(s) it takes.

In the SL example the child tries to minimize the error between your recommendation and its choices. In the RL example the child tries to maximize its reward by finding on its own what is the best to do. The SL approach at its best will lead you to a child to “mimimcs” what you taught it. In the RL approach at its best, the child will have a behavior that will be optimal in terms of playing super mario and also might be better than yours. In other words it will create its own strategy.

For an analogy ,in SL you have a teacher that tells you at every single time step exactly what’s the correct response(think of math class where the teacher explains a identical problem to you on broad before giving you a problem to solve on your own). In RL you try and find it on your own and the teacher gives you a reward/punishment (think of a practical class where you are given a chemical and you have to find its element/composition of the chemical and you follow you own intuition and do some trial and error analysis to find the composition of the chemical. In UL you don’t have any external feedback. So UL falls between SL and RL(think of bio class where teacher gives you example about some classes of animals and ask you to do some classification. say, dog, cat, lion are vertebrate but snail, crab, earthworm are invertebrate so without knowing the meaning of invertebrate and vertebrate tell me which of these are invertebrate and vertebrate: frog, snake and grasshopper.

I simplified a lots just to give you a hint on the learning techniques with the example.

It is learning what to do, how to map situations to actions so as to maximize a numerical reward signal. It does not make use of any training dataset to learn the pattern, unlike other learning methods. The learner is not told which actions to take, as in most forms of machine learning, but instead, must discover which actions yield the most reward by trying them.

In reinforcement learning problems the feedback is simply a scalar value which may be delayed in time. This reinforcement signal reflects the success or failure of the entire system after it has performed some sequence of actions. Hence the reinforcement signal does not assign credit or blame to any one action (the temporal credit assignment problem), or to any particular node or system element (the structural credit assignment problem).

In contrast, in supervised learning the feedback is available after each system action, removing the temporal credit assignment problem; in addition, it indicates the error of individual nodes instead of simply telling how good the outcome was. Supervised learning methods, for instance back-propagation, off-line clustering, mathematical optimization, and ID3, rely on having error signals for the system’s output nodes, and typically train on a fixed set of examples which is known in advance. But not all learning problems fit this paradigm. Reinforcement learning methods are appropriate when the system is required to learn on-line, or a teacher is not available to furnish error signals or target outputs. Examples include:

Game playing

If there is no teacher, the player must be able to determine which actions were critical to the outcome and then alter its heuristics accordingly.

Learning in a micro-world

The agent must develop the ability to categorize its perceptions, and to correlate his awareness of its environment with the satisfaction of primitive drives such as pleasure and pain.

On-line control

Controllers of automated processes such as gas pipelines or manufacturing systems must adapt to a dynamically changing environment, where the optimal heuristics are usually not known.

Autonomous robot exploration

Autonomous robots may make feasible exploration of hazardous environments such as the ocean and outer space, using on-line learning to adapt to changing and unforeseen conditions.

The reinforcement learning model consists of:

A task is defined by a set of states, s∈S, a set of actions, a∈A, a state-action transition function,

T: S×A→S, and a reward function, R: S×A→R. At each time step, the learner (also called the agent) selects an action, and then as a result, given a reward and its new state. The goal of reinforcement learning is to learn a policy, a mapping from states to actions, Π: S →A that maximizes the sum of its reward over time.

In machine learning, the environment is formulated as a Markov decision process (MDP), as many reinforcement learning algorithms for this context utilize dynamic programming techniques.

Elements of Reinforcement Learning:

Except for the agent and the environment, we have four sub-elements of reinforcement learning system:

Rewards are in a sense primary, whereas values, as predictions of rewards, are secondary. Without rewards, there could be no values, and the only purpose of estimating values is to achieve more reward.

How does it work?

Reinforcement learning is all about trying to understand the optimal way of making decisions/actions so that we maximise reward R. This reward is a reply signal that shows how well the agent is doing at a given time step. The action Athat an agent takes at every time step is a function of both the reward and the state S, which is a description of the environment the agent is in. The mapping from environment states to actions is policy P. The policy basically defines the agent’s way of behaving at a certain time, given a certain situation. Now, we also have a value function V which is a measure of how good each position is. This is different from the reward in that the reward signal indicates what is good in the immediate sense, while the value function is more indicative of how good it is to be in this state/position in the long run. Finally, we have a model M which is the agent’s representation of the environment. This is the agent’s model of how it thinks that the environment is going to behave.

The whole Reinforcement Learning environment can be described with an MDP.

Thus, RL is learning to make good decisions from partial evaluative feedback.

Control & Decision theory
In control theory (and AI planning), perfect knowledge about the world is assumed, and the objective is to find the best way to behave.

However, for many problems knowledge about the world is not perfect. Hence, exploring the world could increase our knowledge and eventually help us make better decisions.

RL is balancing the exploration-exploitation trade-off in sequential decision-making problems.

Behavioral Psychology
The simplified goal of behavioral psychology is to explain why, when, and how humans make decisions. We consider humans as rational agents, and hence psychology is also to some extent trying to explain rational behavior.

One can study the biological principles of how opinions are formed, which have close connections to temporal difference learning and eligibility traces.

RL is the paradigm to explain how humans form opinions and learn to make good decisions with experience.

Related posts:

Alat musik Marawis adalah instrumen tradisional yang populer di kalangan masyarakat muslim. Dengan desain yang elegan dan bunyinya yang khas, alat musik ini menghadirkan keajaiban musikal bagi para…

While on a social media site, you see a men’s clothing ad in which a woman is looking adoringly at a man wearing a designer suit. The purpose of the image is for readers to associate wearing the suit…

Assuming that mascarpone is one of the most very much cherished and frequently involved things in your refrigerator, you might be searching for ways of saving it in mass for longer timeframes…