Bayesian Statistics

References

• Bayesian Statistics Wikipedia Article

Related

• Statistics
• • Statistics is the discipline that concerns the collection organization, analysis, interpretation, and presentation of data. In applying statistics to scientific, industrial, or social problem, it is conventional to begin with a statistical population or a statistical model to be studied. Populations can be diverse groups of people or objects such as all people living in a country or every atom composing a crystal. Statistics deals with every aspect of data, including the planning of data collection in terms of design of surveys and experiments.
• Event
• • In probability theory, an event is a set of outcomes of an experiment ( a subset of the sample space) to which a probability is assigned. A single outcome may be an element of many different events, and different events in an experiment are usually not equally likely, since they may include very different groups of outcomes. An event consisting of only a single outcome is called an elementary event or an atomic event; that is, it is a singleton set (a set with exactly one element). An event has more than one possible outcome is called a compound event. An event is said to occur if contains the outcome of the experiment (that is, ). The probability that an event occurs is the probability that contains the outcome of an experiment (that is, the probability that ). An event defines a complementary event, namely the complementary set (the event not occurring), and together these define a Bernoulli trial: did the event occur or not?
• Probability
• • Probability is the branch of mathematics concerning events and numerical descriptions of how likely they are to occur. The probability of an event is a number between 0 and 1; the larger the probability, the more likely an event is to occur. These concepts have been given an axiomatic mathematical formalization in probability theory, which is used widely in areas of study such as statistics, .... Probability theory is also used to describe the underlying mechanics and regularities of complex systems.
• Probability Distribution
• • IN probability theory and statistics, a probability distribution is the mathematical function that gives the probabilities of occurrence of possible outcomes for an experiment. It is a mathematical description of a random phenomenon in terms of its sample space and the probability of events.
• Statistical Model
• • A statistical model is a mathematical model that embodies a set of statistical assumptions concerning the generalization of sample data (and similar data from a larger population). A statistical model represents, often in considerably idealized form, the data-generation process. When referring specifically to probabilities, the corresponding term is probabilistic model. All statistical hypothesis tests and all statistical estimators are derived via statistical models. More generally, statistical models are part of the foundation of statistical inference. A statistical model is usually specified as a mathematical relationship between one or more random variables and other non-random variables. As such, a statistical model is a formal representation of a theory
• Law of Total Probability
• • In probability theory, the law (or formula) of total probability is a fundamental rule relating marginal probabilities to conditional probabilities. It expresses the total probability of an outcome which can be realized via several distinct events.

The law of total probability is a theorem that states, in its discrete case, if is a finite or countably infinite set of mutually exclusive and collectively exhaustive events, then for any event :

where for any , if , then these terms are simply omitted from the summation since is finite.

Notes

Bayesian statistics is a theory in the field of statistics based on the Bayesian interpretation of probability, where probability expresses a degree of belief in an event. The degree of belief may be based on prior knowledge about the event, such as the results of previous experiments, or on personal beliefs about the vent. This differs form a number of other interpretations of probability, such as the frequentist interpretation, which views probability as the limit of the relative frequency of an event after many trials. More concretely, analysis in Bayesian methods codifies prior knowledge in the form of a prior distribution.

Bayesian statistical methods use Bayes' theorem to compute an update probabilities after obtaining new data. Bayes' theorem describes the conditional probability of an event based on data as well as prior information or beliefs about the event or conditions related to the event. In Bayesian inference, Bayes' theorem can be used to estimate the parameters of a probability distribution of a statistical model.

Bayesian statistics is named after Thomas Bayes, who formulated a specific case of Bayes' theorem in a paper published in 1763.In several papers spanning from the late 18th to the early 19th centuries, Pierre-Simon Laplace developed the Bayesian implementation of probability. In several papers spanning from the 1late 18th to the early 19th centuries, Pierre-Simon Laplace developed the Bayesian interpretation of probability. Laplace used methods that would now be considered Bayesian to solve a number of statistical problems. The term "Bayesian methods" was not commonly used to describe such methods until the 1950s. With the advent of powerful computers and new algorithms like Markov chain Monte Carlo, Bayesian methods have seen increasing use within statistics in the 21st century.

Bayes' theorem is used in Bayesian methods to update probabilities, which are degrees of belief, after obtaining new data. Given two events and , the conditional probability of given that is true is expressed as follows:

where . Although Bates' theorem is a fundamental result of probability theory, it has specific interpretation in Bayesian statistics. In the above equation, usually represents a proposition (such as the statement that a coin lands on heads 50% of the time) and represents the evidence, or new data that is to be taken into account (such as the result of a series of coin flips). is the prior probability of which expresses one's beliefs about before evidence is taken into account. The prior probability may also quantify prior knowledge or information about . is the likelihood function, which can be interpreted as the probability of the evidence given that is true. The likelihood quantifies the extent to which the evidence supports the proposition . is the posterior probability, the probability of the proposition after taking the evidence into account.

Essentially, Bayes' theorem updates one's prior beliefs after considering the new evidence .

The probability of the evidence can be calculated using the law of total probability. If is a partition of the sample space, which is the set of all outcomes of an experiment, then,

When there are an infinite number of outcomes, it is necessary to integrate over all outcomes to calculate using the law of total probability. Often, is difficult to calculate as the calculation would involve sums or integrals that would be time consuming to evaluate, so often only the product of the prior and likelihood is considered, since the evidence does not change in the same analysis. The posterior is proportional to this product:

The maximum a posteriori, which is the mode of the posterior and is often computed in Bayesian statistics using mathematical optimization methods, remains the same. The posterior can be approximated even without computing the exact value of with methods such as Markov chain Monte Carlo or variational Bayesian methods.

Bayesian Methods

Bayesian Inference

Bayesian inference refers to statistical inference where uncertainty in inferences is quantified using probability. In classical frequentist inference, model parameters and hypotheses are considered to be fixed. Probabilities are not assigned to parameters or hypotheses in frequentist inference.

Statistical models specify a set of statistical assumptions and processes that represent how the sample data are generated. Statistical models have a number of parameters that can be modified. Devising a good model for the data is central in Bayesian inference. Bayesian inference uses Bayes' theorem to update probabilities after more evidence is known.

Statistical Modeling

The formulation of statistical models using Bayesian statistics has the identifying feature of requiring the specification of prior distributions for any unknown parameters. Parameters of prior distributions may themselves have prior distributions, leading to Bayesian hierarchical modeling.

Design of Experiments

The Bayesian design of experiments includes a concept called influence of prior beliefs. This approach uses sequential analysis techniques to include the outcome of early experiments in the design of the next experiment. This is achieved by updating 'beliefs' through the use of prior and posterior distribution.

Exploratory Analysis of Bayesian Models

Exploratory analysis of Bayesian models is an adaptation or extension of the exploratory data analysis approach to the needs and peculiarities of Bayesian modeling.

When working with Bayesian models there are a series of related tasks that need to be addressed besides inference itself:

• Diagnoses of the quality of the inference, this is needed when using numerical methods such as Markov chain Monte Carlo techniques
• Model criticism, including evaluations of both model assumptions and model predictions
• Comparison of models, including model selection or model averaging
• Preparation of the results for a particular audience