Disqus for deduction-theory

Perceptron as Seen in a Relationship Game Model


Abstract

Perceptron is fundamental model of deep learning. We have deduced the idea of Pedro Domingos[2] and created a model for interpreting perceptron into a relationship game. Then we’ve got an inference of a relativity information from a perceptron model. We further our understanding of a training process in this model and made a theoretical goal of an information processing and a neural network research.



Definition

In our previous work, we present our perspective of how we see a information processing in a world. [7] In this work, we explained our theory called “Deduction Theory” is a theory of Deducing World’s Relativity by Information Coupling and Asymmetry. This can achieved by observing and analyzing the world without assumption of axiom and deterministic point of view, but make an open information structure.
An open information structure is like a model or a rule of game but it built in a relativity between informations, not result data. In this way an open information structure is flexible and easy to apply to different field of research.

Figure1. Perceptron

Perceptron known as a functions that can decide whether an input, represented by a vector of numbers, belongs to some specific class or not.
Pedro Domingos said “the perceptron is a like a tiny parliament where the majority wins.”.[2] From this insight, we expend perceptron model to a general relationship game.
Like human relationship, perceptron distorts information while transferring. We see this as origin of cognition, memory and imagination.

In information theory, information entropy is a measure of unpredictability of the state, or equivalently, of its average information content. In a way of entropy, the more perceptron distorts which means create more informations, the more entropy increases. However in deep learning, multilayer of distortion create more predictability which mean less entropy.

We will explain how a perceptron can creates predictability and how to make a neural network which is group of perceptrons to reach a higher level of an information processing in further chapter.

A model that described the information process in perceptron as a person


Perceptron distorts information while transferring. Each connection between input and perceptron itself has a pathway that distort input data, we call this pathway, "Pathway Perspective" which is weight in perceptron model. The reason why we call it pathway perspective is it doesn’t care about what information comes in but where this information coming from. Then perceptron sumate those distortions of information to a probabilistic result information which mean it is not a result of a conditional expression or a deterministic calculation but a probabilistic distortion. Because there is no deterministic calculation to make a static and universal result data in perceptron.

We can see this relationship of pathway perspective in human society too. It’s known as a cognitive bias, a perception or a characterization. What it does is creates distorted information from an input. Like a cognitive bias or perception is not absolute reality but created by feeling and experience in each person, this pathway perspective formed by training experience and random values. (Feeling is not a random value but another probabilistic result information made in our mind.)

An incoming information could be a result data such as text, number, binary, etc. Perceptron distorted to create a probabilistic result information. We are claim this probabilistic result information is fundamentally different from result data because this view can lead us to creates open information structure.


To know how it different, see the process of deep learning. In the deep learning as the right side of [Figure], a researcher does not give an absolute rules to the program. A researcher does labeling and guiding however, researcher open the process for a deep learning agent to look for own rules.


In more detail, first researcher create a dataset which is combination of set with a result data and a labeling information. Then a result data begins to get distorted with the weights coupled by the pathway perspectives in the process of a perceptron agent. In this process, it removes the form of data by vector calculations, and replicated the relationships of the data. We call it "Similar Replication of Information" because if you do an exact duplication, you have to get the same data, but perceptron create distorted data which is a information that relatively react to an input data.
While this agent creating a similar replication of a information, the possibility occur to finding similarities and differences in the data. A probabilistic result information in perceptron is the result of finding similarities and differences; in other word, an information relativity of coupling and asymmetry.

Logically, a perceptron agent does not understands meaning of the data. Because of that, perceptron does not conditionally choice the result like in imperative programming, it only takes the difference in topology of datas.
If in case of data A is more higher place than data B and weight is slightly closer to data A, by the process it drives an output value to closer to data A. For example of perceptron in human, it is similar to someone who does not understand the concept or meaning but consequently appeals positive or negative opinion.

Likewise,  even  in  the  case  of  human,  we  experience  reality  by  using  our sensory organs and brain to make a result information. The result of information processing in our brain is ”cognition, memory, imagination”.  These are also a result information.  The difference is that cognition and memory are believed asit happened in reality in our brain, but imagination considered as not reality.This is the same activity as labeling.  Depending on how we labeling, we think of something as a memory and something as an imagination.  That’s why people deceived easily.  As same reason, a neural network could be deceived[4] and take a cognitive bias[9] too.


The process of learning

The researcher who trying to train a perceptron agent must start from making clear definition of what to train. And use fixed and quantified criteria for evaluate result data. To train this agent, researcher runs a consistent calibration system. This is current way of thinking in a deep learning field. A limitation of this way of thinking is first, the agent doesn’t understand the data it creates. Second, therefore if the definition or formation is changed, past trained weights are becoming useless. Third, the agent does not judge what it is doing, just blindly expect what it had learned is right. This is also a current limitation of a deep learning field.

,

For more detail, describe an example of the process in back-propagation method.
First, calculate a derivative of a error value to find a slope at which the error value decreases in the current state. For each perceptron in the previous layer, the connected weight is calculated inversely and the error value to be propagated to the perceptron is calculated. Then adjust the value of the perceptron according to the slope to reduce the error value.
In other words, perceptrons distort the relationship of data in a multidimensional space created by weights and produce a probabilistic result information that has a coupled relation with the data. The backpropagation method is a method of differentiating the degree of distortion of the multi-dimensional space generated by this weight to adjust.

This means the perceptron agent is mimic the characteristics of the researcher's way of thinking. The reason why we call the characteristics is that it adjusts the phase of distortion to the resulting shape that achieves the goal.

Screenshot_2017-06-24_13-38-26.png
perceptron learning simulation by hand and back-propagation [6]

We develop a simple simulational application to see how our perspective sits in the perceptron model.[6] What we understand from this is that the space distorted, the phase of the data is changed and achieving the characteristic of information processing in xor problem.


The inference of relativity information from perceptron


When there is a sentence, first empty the part corresponding to a result information. Once you have empty the result information, you can infer relativity information and information structures. Then you can use this information structure to create another information. This explains how we can create an open information structure. In a case of “Five Ws and One H”, "who, when, where and what" describes a result information while "how and why" a relativity information.


We tried same inference method to see it is applies to a perceptron model. We've emptied an input and an output from a perceptron model that contains a result information. The information remains after is a network architecture and weights. This can act as a sentence structure or a rule of game which is an open information structure.

We’ve found that perceptron not only can mimic a characteristics of human’s way of thinking, but create its own way of thinking, because it does core logical action of creating an open information structure.


Closer to scenario information processing

I have reconstructed the analogy of tarot cards presented by novelist Italo Calvino.[1] “I’ve got to join an unknown card game. I found that I can’t ask the other participant what is the rule of this game but can only watch what the others doing and do as they are doing. Because the other participant seems also doesn’t know the rule.” In this game, I must guess the rule and improve them by steps for survive and maximize rewards. Because of there is no absolute rules that governs the game, I must create my own rule by imagining a structure of it.

This is what human do while they think. The concept of human language, philosophy, logic is not an absolute rules but only imagined by each individual and improved by steps. The only difference between imagination and reality in our mind is that imagination is poorly coupled information structure compare to reality. In this way, we are saying a society is a massive information structure that established by overlapped individual’s imagination that developed through generations.

How can we make our perceptron to reach this level? We explain a perceptron can creates an open information structure which is imagining a structure of a game. For a next level, we need to train a way of thinking with method of guiding a neural network. For this purpose, we made a term “scenario information processing”. Scenario information processing compares between processes, creates a new process and selects better process.

A counterpart of this concept is an instant response information structure. It’s the way conventional learning method of neural network does. The reason why we call this an instant response information structure is because it designed to instantly respond to an input. This is quite similar to the reptilian brain as a brain's information processing. In this method, a simulation that correspond to reality is only presented by human researcher like a reinforcement learning method.

This is features of scenario information processing. First it imagines the information processing processes (scenarios), that can be practiced in reality, based on the agent’s experiences. The recent neural network research such as Pathnet [3] or Decoupled Neural Interfaces(DNI) [5] shows how to use another neural network to design architecture or/and train target neural network. About these researches, we consider a target neural network’s current architecture and weights as single scenario, see the other neural network that design and train as a scenario processor. But those researches treat this scenario processor as a managing tool for targeted neural network and limits its potential, like as fix targeted neural network’s architecture after learning process is finished, only takes current maximum reward taking architecture and weights and, limits its grow processing of scenarios.

Second, scenario information processing creates a multidimensional environment as a simulation. Then finds similarities and differences in each scenario and between scenarios. And grow this inference as an open information structure. Thereafter it can compare a probabilistic feature of each scenario and chose the right scenario at the right timing. Finally it improves its scenarios and raise a possibility of success in an information processing(predictability).


Conclusion

Perceptron distorts information while transferring. This function is key of creating an open information structure.
For a next level of neural network, we need to train a way of thinking with method of guiding a neural network. For this purpose, scenario information processing could be the answer.
Therefore, the technology that is closer to scenario information processing will be more effective and make a real life difference in this deep learning field in future.


References

[1]  Italo Calvino.The Castle of Crossed Destinies.  Random House, 1973.
[10]  Frank Rosenblatt.  The perceptron:  A probabilistic model for information storage and organization in the brain.Psychological review, 65(6):386, 1958.
[11]  David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams.  Learning representations  by  back-propagating  errors.Cognitive modeling,  5(3):1,1988.


Author

Kenneth Kijun Lee
Chase Kihwan Lee

Facebook Comments

Disqus Comments

Support Us

If you like our article and would like to support us, please mining Monero below. If you press the START MINING button, your computer will work and support us in cryptocurrency. It's about a ten cents a day. For reference, set the SPEED lower to 80%, so your computer will not bother.



Loading...
Also, you can send us existing cryptocurrency.
Bitcoin: 1Jocrm8iKUtw4h19JttGLjj9ouQnDHrmpB
Monero: 45sQXZzqtFAKgJJgtQh3MuGYwUHHTb5bEZzvfdw3QhvSKpb1KMJmgnSQCVkWGcmL1PVzqVfV4bZH5D2C5uRfeWHLUf5pMkw

Popular Posts

Visitor Map

Flag Counter