2014 Technical Notes
jongman@gmail.com
January 21, 2015
1
Statistical Inference Writeup
jongman@gmail.com
January 19, 2015
This is a personal writeup of Statistical Inference ”Casella and Berger, 2nd ed.). The purpose
of this note is to keep a log of my impressions during the reading process, so I cannot guarantee
the correctness of contents. :-)
Contents
1 Probability Theory
4
2 Transformations and Expectations
2.1 Transformations of random variables . . . . . . . . . . . . . . . . . . . . . . . . . .
4
4
2.1.1 Monotonic functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5
2.1.2 Piecewise monotonic functions . . . . . . . . . . . . . . . . . . . . . . . . . .
5
2.1.3 Probablity integral transformation . . . . . . . . . . . . . . . . . . . . . . . .
5
2.2 Expected Values . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5
2.3 Moments and Moment Generating Functions . . . . . . . . . . . . . . . . . . . . . .
6
3 Common Families of Distribution
3.1 Discrete Distributions
6
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.2 Continuous Distributions
6
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
7
3.3 Exponential Families of Distribution . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.4 Scaling and Location Families . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
9
9
3.5 Inequalities and Identities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
10
4 Multiple Random Variables
10
4.1 Joint and Marginal Distributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
10
4.2 Conditional Distributions and Independence . . . . . . . . . . . . . . . . . . . . . .
10
4.3 Bivariate Transformations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
11
4.4 Hierarchical Models and Mixture Distributions . . . . . . . . . . . . . . . . . . . .
11
4.5 Covariance and Correlation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
12
4.6 Multivariate Distributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
12
4.7 Inequalities and Identities. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
12
1
5 Properties of a Random Sample
13
5.1 Basic Concepts of Random Samples . . . . . . . . . . . . . . . . . . . . . . . . . . .
5.2 Sums of Random Variables from a Random Sample . . . . . . . . . . . . . . . . . .
13
13
5.2.1 Basic Statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5.2.2 Bessel’s Correction . . . . . . . . . . . . . . . . . . . . . . . . . .
5.2.3 Sampling Distributions of Sample Mean and Sample Variance
5.2.4 Using mgfs to Find Sampling Distributions . . . . . . . . . . . .
5.3 Sampling From the Normal Distribution . . . . . . . . . . . . . . . . .
.
.
.
.
.
14
14
14
15
15
5.3.1 Student’s t-distribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
15
5.3.2 Fisher’s F -distribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5.4 Order Statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5.5 Convergence Concepts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
15
16
16
5.5.1
5.5.2
5.5.3
5.5.4
Definitions . . . . . . .
Law of Large Numbers
Central Limit Theorem
Delta Method . . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
16
16
17
17
6.1 The Sufficiency Principle . . . . . .
6.1.1 Sufficient Statistic . . . . . .
6.1.2 Factorization Theorem . . .
6.1.3 Minimal Sufficient Statistics
6.1.4 Ancillary Statistics . . . . .
6.1.5 Complete Statistics . . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
18
18
18
18
19
19
6.2 The Likelihood Principle . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
19
6 Principles of Data Reduction
18
7 Point Estimation
20
7.1 Methods of Finding Estimators . . . . . . . . . . .
7.1.1 Method of Moments for Finding Estimator
7.1.2 Maximum Likelihood Estimation . . . . . .
7.1.3 Bayes Estimators . . . . . . . . . . . . . . .
7.1.4 The EM Algorithm . . . . . . . . . . . . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
20
20
21
21
21
7.2 Methods of Evaluating Estimators
7.2.1 Mean Squared Error . . . .
7.2.2 Best Unbiased Estimator . .
7.2.3 General Loss Functions . .
8 Hypothesis Testing
8.1 Terminology . . . . . . . . . .
8.2 Methods of Finding Tests . . .
8.2.1 Likelihood Ratio Tests
8.2.2 Bayesian Tests . . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
22
22
23
24
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
25
26
26
26
27
2
8.2.3 Union-Intersection and Intersection-Union Tests
. . . . . . . . . . . . . . .
27
8.3 Evaluating Tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
8.3.1 Types of Errors and Power Function . . . . . . . . . . . . . . . . . . . . . . .
27
27
8.3.2 Most Powerful Tests: Uniformly Most Powerful Tests . . . . . . . . . . . . .
29
8.3.3 Size of UIT and IUT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
31
8.3.4 p-Values . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
31
8.3.5 Loss Function Optimality . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
32
9 Interval Estimation
32
9.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
32
9.1.1 Coverage Probability Example: Uniform Scale Distribution . . . . . . . . .
9.2 Methods of Finding Interval Estimators . . . . . . . . . . . . . . . . . . . . . . . . .
33
33
9.2.1 Equivalence of Hypothesis Test and Interval Estimations: Inverting a Test
Statistic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
33
9.2.2 Using Pivotal Quantities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
35
9.2.3 Pivoting CDFs Using Probability Integral Transformation . . . . . . . . . .
35
9.2.4 Bayesian Inference . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
36
9.3 Methods of Evaluating Interval Estimators . . . . . . . . . . . . . . . . . . . . . . .
36
9.3.1 Size and Coverage Probability . . . . . . . . . . . . . . . . . . . . . . . . . . .
36
9.3.2 Test-Related Optimality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
37
9.3.3 Bayesian Optimality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
37
9.3.4 Loss Function Optimality . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
38
10 Asymptotic Evaluations
38
10.1 Point Estimation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
10.1.1 Criteria . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
38
38
10.1.2 Comparing Consistent Estimators . . . . . . . . . . . . . . . . . . . . . . . .
39
10.1.3 Asymptotic Behavior of Bootstrapping . . . . . . . . . . . . . . . . . . . . . .
10.2 Robustness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
39
40
10.2.1 Robustness of Mean and Median . . . . . . . . . . . . . . . . . . . . . . . . .
40
10.2.2 M-estimators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
40
10.3 Hypothesis Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
41
10.3.1 Asymptotic Distribution of LRT . . . . . . . . . . . . . . . . . . . . . . . . . .
41
10.3.2 Wald’s Test and Score Test . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
41
10.4 Interval Estimation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
42
11 Analysis of Variance and Regression
42
11.1 One-way ANOVA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
11.1.1 Different ANOVA Hypothesis . . . . . . . . . . . . . . . . . . . . . . . . . . .
42
42
11.1.2 Inference Regarding Linear Combination of Means . . . . . . . . . . . . . .
11.1.3 The ANOVA F Test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
42
43
11.1.4 Simultaneous Estimation of Contrasts . . . . . . . . . . . . . . . . . . . . . .
44
3
11.1.5 Partitioning Sum of Squares . . . . . . . . . . . . . . . . . . . . . . . . . . . .
44
11.2 Simple Linear Regression . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
11.2.1 General Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
45
45
11.2.2 Least Square Solution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
45
11.2.3 Best Linear Unbiased Estimators: BLUE . . . . . . . . . . . . . . . . . . . . .
46
11.2.4 Normal Assumptions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
47
12 Regression Models
12.1 Errors in Variables ”EIV) Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
50
50
12.1.1 Functional And Structural Relationship . . . . . . . . . . . . . . . . . . . . .
50
12.1.2 Mathematical Solution: Orthogonal Least Squares
. . . . . . . . . . . . . .
50
12.1.3 Maximum Likelihood Estimation . . . . . . . . . . . . . . . . . . . . . . . . .
50
12.1.4 Confidence Sets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
12.2 Logistic Regression And GLM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
51
51
12.2.1 Generalized Linear Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
51
12.2.2 Logistic Regression . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
12.3 Robust Regression . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
51
52
12.3.1 Huber Loss
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
52
1 Probability Theory
We will not discuss elementary set theory and probability theory.
• Kolmogorov’s axiom set defines a probability function for a given σ-algebra.
• Bonferroni’s inequality is useful for having a gut estimation of lower bound for events
that are hard or impossible to calculate probabilities for. It is equivalent to Boole’s inequality.
– Is a generalization of: P (A ∩ B) ≥ P (A) + P (B) − 1
* Proof: P (A ∪ B) = P (A) + P (B) − P (A ∩ B). Rearranging, we get P (A ∩ B) =
P (A) + P (B) − P (A ∪ B) ≥ P (A) + P (B) − 1.
• Random variables: function mapping from sample space to real numbers.
2 Transformations and Expectations
2.1
Transformations of random variables
This section covers functions of random variables and how to derive their cdf from the cdf
of the original RV. Say we have a random variable X with a pdf fX or a cdf FX . What is the
distribution of Y = g (X)?
4
2.1.1 Monotonic functions
We can only do this for monotonic functions, or at least piecewise monotonic functions. So
what do we do when function Y = g (X) is monotonic?
• If monotonically increasing,
FY (y) = P (Y ≤ y) = P (g (x) ≤ y) = P x ≤ g −1 (y) = FX g −1 (y)
• If decreasing
FY (y) = P (Y ≤ y) = P (g (x) ≤ y) = P x ≥ g −1 (y) = 1 − FX g −1 (y)
What do you do when you have pdf, not cdf of the original RV? You don’t have to go through
the above way; you can differentiate above using chain rule – the below formula takes care of
both increasing and decreasing functions.
fY (y) = fX g
−1
d −1
(y) g (y)
dy
In fact, this is used far more often than the cdf case.
2.1.2 Piecewise monotonic functions
You can also do this when g() is piecewise monotonic. it means, you should be able to partition
χ, the domain of the original RV, into contiguous sets so the function is monotonic in each
partition. For example, if f (X) = X 2 , we should split the real line into (−∞, 0] and (0, ∞).
Let us partition χinto multiple subsets A1 , A2 , · · · and let g1−1 , g2−1 , · · · be the inverse of g() in
each of the intervals. Then we state without proof
fY (y) =
X
fX
d
gi−1 (y) gi−1 (y)
dy
which basically sums up the previous formula for each interval.
2.1.3 Probablity integral transformation
If X has a cdf FX , and we say Y = FX (X), then Y is uniformly distributed on (0, 1). This can
be understood intuitively: let FX (x) = y. Then P (Y ≤ y) = P (X ≤ x) = FX (x) = y. This of
course assumes monotonicity on FX ’s part, which is not always true, but this can be treated
technically.
2.2
Expected Values
This section discusses the definition of expected values and their properties. The linearity
of expectation arises from integration which is how EVs are defined: E (A + B) = EA + EB
5
regardless if A and B are independent or not.
When you want to make transformation of a random variable and take its expected value,
you can calculate EVs directly from definition. Otherwise, you can transform the pdf using the
above strategy and go from there.
2.3
Moments and Moment Generating Functions
Definitions:
• The nth moment of X, µn is defined by E (X n ).
′
• The nth central moment of X, µn is defined by E ((X − µ) ).
n
• A moment generating function of X is defined as MX (t) = EetX =
´
x
etx fX (x) dx.
• Variance VarX is defined as the second central moment. Of couse, we have the following
equality as well:
VarX = EX 2 − (EX)
2
A moment generating function can be used to generate moments: we have
dn
MX (0) = EX n
dtn
Moment generating functions can be used to identify distributions; if two distributions have
same mgf and all moments exist, they are the same distribution. We discuss some theorems
regarding convergence of mgfs to another known mgf to prove the convergence of the distribution. It looks like it has more of a theoretical importance, rather than practical.
3 Common Families of Distribution
3.1
Discrete Distributions
Discrete Uniform Distribution The simplest of sorts.
• P (X = x|N ) =
• EX =
N +1
2 ,
1
N,
x ∈ 1, 2, 3, · · · , N
VarX =
(N +1)(N −1)
12
Hypergeometric Distribution Say we have a population of size N , of which M has a desired
property. We take a sample of size K ”typically K ≪ M ). What is the probablity that x of
those have this property?
• pmf is derived from counting principles, EV and Var from a similar manner to binomial distribution: rewrite the sum as a sum of hypergeometric pmf for a smaller
parameter set which equals 1.
N −M
N
• P (X = x|N, M, K) = M
/ K
x
K−x
6
• EX =
KM
N ,
VarX =
KM
N
Binomial Distribution
• P (X = x|N, p) =
N
x
(N −M )(N −K)
N (N −1)
px (1 − p)
N −x
• EX = np, VarX = np (1 − p)
Poisson Distribution
• P (X = x|λ) =
e−λ λx
x!
• EX = VarX = λ
• When a binomial distribution’s p is very small, the distribution can reasonably approximated by a Poisson distribution with λ = np. ”Proof uses mgfs)
Negative Binomial Distribution If we want to know the number of Bernoulli trials required
to get r successes? Put another way, we are interested in the number of failures before
rth success.
• P (Y = y) = (−1)
y
−r
y
y
pr (1 − p) =
r+y−1
y
pr (1 − p)
y
• EY = r 1−p
p ”a simple proof: you can find expected number of failures before each
success andsum them, because linearity. Woohoo!)
• VarY =
r(1−p)
p2
= µ + 1r µ2
• Negative binomial family includes Poisson as a limiting case ”Poisson is also related
with Binomial, so this seems natural) but doesn’t seem to have a large practical significance.
Geometric Distribution A special case of negative binomial distribution with r = 1.
• P (X = x) = p (1 − p)
• EX = p1 , VarX =
x−1
1−p
p2
• Memoryless property: history so far has no influence on what will happen from
now on. So: P (X > s|X > t) = P (X > s − t)
3.2
Continuous Distributions
Uniform
Gamma A highly generic & versatile family of distributions.
• A primer on gamma function Γ:
´∞
– Γ (α) = 0 tα−1 e−t dt which also has a closed form when α ∈ N.
– Also, the gamma function serves as a generalization of factorial, because Γ (α + 1) =
αΓ (α) if α > 0. ”This makes Gamma function easy to evaluate when we know its
values between 0 and 1)
7
– We have Γ (α) = (α − 1)! for integers.
• The full gamma family has two parameters, α”shape parameter, defines peakedness)
β the scale parameter ”determines spreadness).
• The pdf is set as
f (x|α, β) =
1
xα−1 e−x/β
Γ (α) β α
• EX = αβ, VarX = αβ 3
• Being a generic distribution, it is related to many other distributions. Specifically,
many distributions are special cases of Gamma.
– P (X ≤ x) = P (Y ≥ α) where Y ∼ Poisson (x/β)
– When α = p/2 ”p is an integer) and β = 2, it becomes a chi-squared pdf with p
degrees of freedom.
– When we set α = 1, it becomes the exponential distribution pdf with scale parameter β. ”The exponential distribution is a continuous cousin of the geometric
distribution.)
– It is also related to the Weibull distribution which is useful for analyzing failure
time data and modeling hazard functions.
• Applications: mostly relted with lifetime testing, etc.
Normal without doubt, the most important distribution.
• pdf
f X = x|µ, σ 2 =
2
1
√ e−(x−µ) /
σ 2π
2σ 2
• Proving the pdf integrates to 1 is kind of clunky.
• EX = µ, VarX = σ 2 .
• 68% - 95% - 99% rule.
• Thanks to CLT, normal distribution pops up everywhere. One example would be
using normal to approximate binomials. If p is not extreme and n is large, normal
gives a good approximation.
– Continuity correction: Say Y ∼ Binomial (n, p) and X ∼ n (np, n (1 − p) p), we can
say P (a ≤ Y ≤ b) ≈ P (a ≤ X ≤ b). However, P a − 21 ≤ X ≤ b + 12 is a much better approximation, which is clear from a graphical representation.
Beta One of rare distributions which have a domain of [0, 1]. Can be used to model proportions.
• Given that B (α, β) =
´1
0
xα−1 (1 − x)
β−1
f (X = x|α, β) =
8
dx ,
1
β−1
xa−1 (1 − x)
B (α, β)
• Since the Beta function is related to Gamma function B (α, β) =
lated to Gamma function as well.
Γ(α)Γ(β)
Γ(α+β)
, it is re-
• It can have varying shapes depending on the parameters; unimodal, monotonic, ushaped, uniform, etc.
• EX =
α
α+β ,
VarX =
αβ
(α+β)2 (α+β+1)
Cauchy A symmetric, bell-shaped curve with undefined expected value. Therefore, it is mostly
used as an extreme case against which we test conjectures. However, it pops up in unexpected circumstances. For example, the ratio of two normal variables follows the Cauchy
distribution.
1
f (x|θ) =
π 1 + (x − θ)
2
Lognormal log of which is the normal distribution. Looks like Gamma function.
• f x|µ, σ 2 =
• EX = e
2
2
√ 1
e−(lg x−µ) / 2σ
2πσx
µ+ σ 2 /2
2 µ+σ 2
, VarX = e
− e2µ+σ
2
Double Exponential formed by reflecting the exponential distribution around its mean.
3.3
Exponential Families of Distribution
A family of pdfs or pmfs is an exponential family if it can be expressed as:
f (x|θ) = h (x) c (θ) exp
X
wi (θ) ti (x)
i
!
Many common families are exponential; normal, gamma, beta, binomial, Poisson, etc. Such
form has some algebraic advantages: there are equalities which provide a shortcut for calculating the first two moments ”EV and Var) with differentiation instead of summation/integration!
Will not discuss the actual formulas because they are kind of messy.
The section also discusses natural parameter set of a exponential family: in the above definition, Also, the definition of full and curved exponential families are introduced; not sure
about practical significance.
3.4
Scaling and Location Families
If f (x) is a pdf, then the following function is a pdf as well.
g (x|µ, σ) =
1
f
σ
x−µ
σ
where µ is the location parameter, σ is the scaling parameter. Expected value will get translated accordingly, and variance will grow by σ 2 .
9
3.5
Inequalities and Identities
Chebychev Inequality if g (x) is a nonnegative function,
P (g (x) ≥ r) ≤
Eg (X)
r
which looks kind of arbitrary ”the units do not match) but is useful sometimes. It sometimes provides useful boundaries.
2
which is the number of standard deviations squared
For example, let g (x) = x−µ
σ
”square is there to make g nonnegative). Letting r = 22 yields
P
x−µ
σ
2
≥4
!
2
≤
1
1 (x − µ)
E
=
4
σ2
4
which is a lower bound of elements 2 standard deviations! This did not make any assumption on the distribution of X.
Stein’s Lemma with X ∼ n θ, σ 2 and g that is differentiable and satisfies E |g ′ (X)| < ∞,
E [g (X) (X − θ)] = σ 2 Eg ′ (X). Seems obscure, but useful for calculating higher order moments. Also, wikipedia notes that it is useful in MPT.
4 Multiple Random Variables
4.1
Joint and Marginal Distributions
Mostly trivial stuff. We extend the notion of pmf/pdfs by adding more variables, which give us
joint pdf/pmfs.
Marginal distributions ”distribution of a subset of variables without referencing the other
variables) are introduced. Intuitively, they are compressed versions of joint pdf/pmfs by
integrating them along a subset of parameters.
4.2
Conditional Distributions and Independence
Intuitively, conditional distributions are sliced versions of joint pdf/pmfs. Some of the random variables are observed; what are the distribution of the remaining variables given the
observation?
The derivation of conditional pmf is straightforward for discrete RVs, it is the ratio of joint
pmf and marginal pmf. This relationship, somewhat surprisingly, holds true for continuous
RVs as well.
Two variables X and Y are said to be independent when f (x, y) = fX (x) fY (y). Actually,
the converse is true as well; if the joint pdf can be decomposed into a product of two functions,
one on x and one on y, they are independent.
Consequences of independence:
10
• E (g (X) h (Y )) = Eg (X) Eh (Y )
• Covariance is 0
• You can get the mgf of their sum by multiplying individual mgfs. This can be used to
derive the formula for adding two normal variables.
4.3
Bivariate Transformations
This section mainly discusses strategies of taking transformations ”sum, product, division) of
two random variables. This is analogous to section 2.1 where we discussed transformations of
a single variable.
Problem: You have a random vector (X, Y ) and want to know about U = f (X, Y ).
Strategy: We have a recipe for transforming a bivariate vector into another. So we transform (X, Y ) into (U, V ) and take the marginal pdf to get the distribution of U . V is chosen so it
will be easy to back out X and Y from U and V ; which is essential in the below recipe.
Recipe: Basically, similar to transformation recipe in 2.1. However, the derivative of the
inverse function is replaced by the Jacobian of the transformation; which is defined as the determinant of the matrix of partial derivatives.
J=
∂x ∂y ∂x ∂y
−
∂u ∂v
∂v ∂u
Given this, we have
fU,V (u, v) = fX,Y (h1 (u, v) , h2 (u, v)) |J|
where g1 (x, y) = u, h1 (u, v) = x, g2 (x, y) = y, h2 (u, v) = y.
Similar to formula in 2.1, this assumes the transformation is 1-to-1, and thus the inverse
exists. When this assumption breaks, we can use the same trick as in 2.1 by breaking the
domain into sets where in each set the transformation is 1:1.
fU,V (u, v) =
k
X
i=1
which is the formula in 4.3.6.
4.4
fX,Y (h1i (u, v) , h2i (u, v) |Ji |)
Hierarchical Models and Mixture Distributions
Hierarchical models arise when we model a distribution where a parameter has its own distribution. This is sometimes useful in gaining a deeper undestanding of how things work.
As an example, say an insect lays a large number of eggs following Poisson, and individual
egg’s survival is a Bernoulli trial. Then, the expected number of surviving insect is X|Y ∼
binomial (Y, p) and Y ∼ Poisson (λ) .
11
This section also introduces a trivial, but useful equality: EX = E (E (X|Y )) This is very
intuitive if you think about it, but realizing this makes calculations very easy sometimes. A
noncentral chi-squared distribution is given as an example.
A formula for variance in hierarchical model is given:
VarX = E (Var (X|Y )) + Var (E (X|Y ))
4.5
Covariance and Correlation
This section introduces covariances and correlations. Important stuff, but it only gives rather
a basic treatment.
Definitions and identities:
• Cov (X, Y ) = E [(X − µX ) (Y − µY )] = EXY − µX µY
• ρXY =
Cov(X,Y )
σX σY
• Var (aX + bY ) = a2 VarX + b2 VarY + 2abCov (X, Y )
This section also introduces bivariate normal distributions.
4.6
Multivariate Distributions
A strategy for transforming a vector of random variable is introduced. Since the Jacobian is
well defined for larger matrices, the recipe is more or less the same with the bivariate case.
4.7
Inequalities and Identities.
Analogous to chapter 2, we have a section devoted to inequalities and identities. Apparently,
all of these inequalities have many forms, being applied in different contexts. Many of these
have popped up in CvxOpt course as well. ”Looks like most primitive forms come from field of
mathematical analysis.)
Holder’s Inequality Let p, q be real positive numbers s.t.
have
p 1/p
|EXY | ≤ E |XY | ≤ (E |X| )
1
p
+
1
q
= 1 and X, Y are RV. Then we
q 1/q
(E |Y | )
Cauchy-Schwarz Inequality A special case of Holder’s inequality when p = q = 2. Then,
|EXY | ≤ E |XY | ≤
q
E |X| E |Y |
2
2
In vector terms, it means x · y ≤ |x| |y|. This is intuitive, as taking inner product gives
chance of things canceling out each other, just like triangular inequality.
12
Also notable is that this can be used to prove the range of the correlation; just take |E (X − µX ) (Y − µY )|
and apply CS inequality. Squaring each side gives
2
2 2
(Cov (X, Y )) ≤ σX
σY
Minkowski’s Inequality This feels like an additive version of Holder’s Inequality.
p 1/p
[E |X + Y | ]
p 1/p
≤ [E |X| ]
p 1/p
+ [E |Y | ]
Jensen’s Inequality on convex function. Given a convex function g,
Eg (X) ≥ g (EX)
5 Properties of a Random Sample
This chapter deals with several things:
• Definition of random sample
• Distribution of functions of random sample ”statistics) and how they converge as we increase sample size
• And of course, the LLN and the CLT.
• Generating random samples.
5.1
Basic Concepts of Random Samples
A random sample X1 , · · · , Xn is a set of independent and identically distributed ”iid) RVs. This
means we are sampling with replacement from an infinite population. This assumption doesn’t
always hold, but is a good approximation in a lot of cases.
5.2
Sums of Random Variables from a Random Sample
Sums of random variables can be calculated, of course, using the transformation strategies
from Chapter 4. However, since each random variable is iid, the calculation can be simplified
greatly.
Also, a definition: Given a vector-or-scalar valued statistic Y = T (X1 , X2, · · · ), the distribution of Y is called the sampling distribution of Y .
13
5.2.1 Basic Statistics
Two most basic statistics are introduced.
¯ is defined as
Sample mean X
X
¯= 1
X
Xi
n i
Sample variance S 2 is defined as
S2 =
1 X
¯
Xi − X
n−1 i
The formula for sample variance raises some questions; why n − 1, not n? This was chosen
so that ES 2 = σ 2 . If we were to use n, S 2 would be a biased towards 0.
5.2.2 Bessel’s Correction
Using n − 1 instead of n as the denominator in sample variance is called Bessel’s Correction.
The legitimacy of this can be proven by taking ES 2 and seeing it equals to σ 2 , but the wikipedia
page offers some explanation.
By taking sample mean, we are minimizing the squared error of the samples. So unless
the sample mean is equal to population mean, sample variance must be smaller than variance
measured using the population mean.
Put more intuitively, the sample mean skews towards whatever sample observed, so the
variance will be smaller than real.
Another subtle, but astounding point mentioned is that by Jensen’s inequality, S is a biased
estimator of standard deviation – it underestimates! Since square root is a concave function,
according to Jensen’s inequality we have
√
√
√
E S 2 ≤ ES 2 = σ 2 = σ
Also, it is noted that there is no general formula for an unbiased estimate of the standard
deviation!
5.2.3 Sampling Distributions of Sample Mean and Sample Variance
¯ = µ: the expected value of sample mean is the population mean. LLN will state that
• EX
it will converge to the population mean almost surely as the sample size grows.
¯=
• VarX
σ2
n
: this follows from that the random variables are independent.
Var
1X
Xi
n
=
X VarX
σ2
1
1
Var
=
X
=
i
n2
n
n
So, the variance decreases linearly with the sample size.
14
• ES 2 = σ 2 . This can be shown algebraically.
5.2.4 Using mgfs to Find Sampling Distributions
We can plug the statistic’s definition into mgf, and simplify, praying the mgf would be a recog
nizable form.
For
example, this is how we show the mean of iid normal variables Xi ∼ n µ, σ 2
2
follows n µ, σn .
5.3
Sampling From the Normal Distribution
More notable facts when we sample from the normal distribution.
¯ and S 2 are independent random variables.
• X
¯ has a n µ, σ 2 /n distribution ”proven by mgfs as noted above)
• X
• (n − 1) S 2 /σ 2 has a chi-squared distribution with n − 1 degrees of freedom.
5.3.1 Student’s t-distribution
If Xi ∼ n µ, σ 2 , we know that
¯ −µ
X
√ ∼ n (0, 1)
σ/ n
However, in most cases µ and σ are unknown parameters. This makes it hard to make
inferences about one of them. We can approximate σ by S: this gives us another distribution,
however it makes it easier to make inferences about µ. So, the statistic
¯ −µ
X
√
S/ n
is known to follow Student’s t-distribution with n − 1 degrees of freedom. The concrete distri¯ and S , and their respective distribution could be found using the independence between X
butions ”normal and chi-squared).
5.3.2 Fisher’s F -distribution
Now we are comparing variances between two samples. One way to look at this is the variance
ratio. There are two variance ratios; one sample, one population. Of course, we don’t know
about population variance ratios. However, the ratio between the two ratios ”I smell recursion)
2
2
SX
/SY2
S 2 /σX
= X
2
2
2
2
σX /σY
SY /σY
is known to follow F -distribution. The distribution is found by noting that in the second
representation above, both the numerator and the denomiator follows chi-squared distributions.
15
5.4
Order Statistics
Order statistics of a random sample are the sorted sample values. This seems obscure, but
indeed is useful where point estimation depends on the minimum/maximum value observed.
The pmf/pdf of those can be derived from noticing that:
• Say we want to find P X(j) ≤ xi ”X(j) is thejth smallest number).
• Each random variable is now a Bernoulli trial, with FX (xi ) probability of being ≤ xi .
• So the cumulative functions can be derived from binomial distribution of each variable
”for discrete) or something similar ”continous).
fX(j) (x) =
5.5
n!
j−1
n−j
fX (x) [FX (x)]
[1 − FX (x)]
(j − 1)! (n − j)!
Convergence Concepts
The main section of the chapter. Deals with how sampling distributions of some statistics
change when we send the sample size to infinity.
5.5.1 Definitions
Say we have a series of random variables {Xi }: Xn being the statistic value when we have n
variables. We study the behavior of limn→∞ Xn .
There are three types of convergences:
Convergence in Probability when for any ǫ > 0,
lim P (|Xn − X| ≥ ǫ) = 0
n→∞
Almost Sure Convergence when
P
lim |Xn − X| ≥ ǫ = 0
n→∞
which is a much stronger guarantee and implies convergence in probability.
Convergence in Distribution when their mgfs converge and they have the same distribution.
This is equivalent to convergence in probability when the target distribution is a constant.
5.5.2 Law of Large Numbers
Given a random sample {Xi } where each RV has finite mean, the sample mean converges
almost surely to µ, the population mean.
There is a weak variant of LLN, which states it converges in probability.
16
5.5.3 Central Limit Theorem
The magical theorem! When we have a sequence of iid RVs {Xi } with EXi = µ and VarXi =
σ 2 > 0 then
lim
n→∞
¯n − µ
X
√
∼ n (0, 1)
σ/ n
which also means
¯ n ∼ n µ, σ 2
lim X
n→∞
Another useful, and intuitive theorem is mentioned:
Slutsky’s Theorem if Yn converges to a constant a in probability, and Xn converges to X in
distribution,
• Xn Yn = aX in distribution
• Xn + Yn = a + X in distribution
Slutsky’s theorem is used in proving that normal approximation with estimated variance goes
to standard normal as well. We know
lim
n→∞
¯n − µ
X
√
∼ n (0, 1)
σ/ n
and we can prove
lim
n→∞
σ
=1
S
in probability. Multiplying those yields
lim
n→∞
¯n − µ
X
√
∼ n (0, 1)
S/ n
Marvelous!
5.5.4 Delta Method
Delta method is a generalized version of CLT. Multiple versions of it are discussed, however
here I will only state the most obvious univariate case.
σ2
lim Yn ∼ n µ,
n→∞
n
2
g ′ (µ) σ 2
=⇒ lim g (Yn ) ∼ n g (µ) ,
n→∞
n
!
When Yn converges to µ with normal distribution when we increase n with virtual certainty,
a function of Yn converges to a normal distribution. In the limiting case, both are virtually a
constant, so this doesn’t surprise me much.
17
6 Principles of Data Reduction
This chapter seems to be highly theoretic. Without much background, it is hard to say why this
material is needed. However, it looks like this chapter is closely related with the next chapter point estimation. That makes sense, because estimating point parameters are essentially data
reduction.
6.1
The Sufficiency Principle
6.1.1 Sufficient Statistic
Colloquially, a sufficient statistic captures all information regarding a parameter θ in a sample
x; there are no remaining information to be obtained by consulting the actual sample. Formally, a statistic T (X) is a sufficient statistic if the conditional distribution Pθ (X|T (X)) does
not depend on θ.
The practical upshot of this is that this justifies only reporting means and standard deviations of a given sample; if we assume the population is normal, they are sufficient statistics
which contain all the information we can infer about the population. However, remember
that these are model dependent; the population might be coming from a different family with
different parameters – those might not be entirely captured by the statistic.
6.1.2 Factorization Theorem
The definition could be directly used to verify if a given statistic is sufficient or not, but practically the following theorem makes it easier to identify sufficient statistics. If the joint pdf
f (x|θ) could be factored as a product of two parts:
f (x|θ) = g (T (x) |θ) · h (x)
where h (x) does not depend on θ, then T (X) is a sufficient statistic. This makes intuitive
sense; if T is sufficient, the probabity of us seeing x is related with only two things: a function
of the statistic ”given θ) and a function unrelated to θ. If there were a part with another input
involved, we wouldn’t be able to make inferences about θ with only T .
6.1.3 Minimal Sufficient Statistics
There are lots of sufficient statistics, some more compact than the other. So what is a minimal
sufficient statistic? A sufficient statistic T (X) is minimal when for any other sufficient statistic
T ′ (X), T (X) is a function of T ′ (X). In other words, take any sufficient statistic, and we can
use it to derive a minimal sufficient statistic. For example, the sample mean x
¯ is a sufficient
statistic for population mean µ. On the other hand, the sample itself is a sufficient statistic but
not minimal – you can derive x
¯ from the sample, but not vice versa.
18
6.1.4 Ancillary Statistics
Ancillary statistic contains no information about θ. Paradoxically, an ancillary statistic, when
used in conjunction with other statistics, does contain valuable information about θ. The following is a good example; let X have the following discrete distribution
Pθ (X = θ) = Pθ (X = θ + 1) = Pθ (X = θ + 2) =
1
3
Now, say we observe a sample and take the range of the sample R = X(N ) − X(1) . It is obvious
that the range itself has no information regarding θ. However, say we know the mid-range
statistic as well; M = X(1) + X(N ) /2. The mid-range itself can be used to guess θ, but if you
combine it with the range, suddely you can nail the exact θ if R = 2.
Of course, in the above case, the statistic M was not sufficient. What happens when we
have a minimal sufficient statistic? The intuition says they should be independent. However,
it could not be the case. In fact, the pair X(1) , X(N ) is minimal sufficient and R is closely related
to them! So they are not independent at all.
6.1.5 Complete Statistics
For many important situations, however, the minimal sufficient statistic is indeed independent
of ancillary variables. The notion of complete statistic is introduced, but the definition is very
far from intuitive. It goes like this: let T (X) ∼ f (t|θ). If ∀θEθ g (T ) = 0 =⇒ ∀θPθ (g (T ) = 0),
T (X) is called a complete statistic. Colloquially, T (X) has to be uncorrelated with all unbiased
estimators of 0.
So WTF does this mean at all? It looks like we will get more context in the next chapter.
The most intuitive explanation I could get was from here. It goes like
Intuitively, if a nontrivial function of T has mean value not dependent on θ, that
mean value is not informative about θ and we could get rid of it to obtain a sufficient
statistic simpler .
Hopefully reading chapter 7 will give me more background and intuition so I can revisit this.
6.2
The Likelihood Principle
The likelihood function is defined to be
L (θ|x) = f (x|θ)
We can use likelihood functions to compare the plausibility of various parameter values. Say
if we don’t know the parameter and observed x. If for two parameters θ1 and θ2 , if we have
L (θ1 |x) > L (θ2 |x), we can say θ1 is more plausible than θ2 . Note we used plausible rather
than probable .
The likelihood principle is an important principle used in later chapters on inferences: if x
and y are two sample points such that L (θ|x) is proportional to L (θ|y), that is, there is a constant
19
C (x, y) such that L (θ|x) = C (x, y) L (θ|y), then the conclusions drawn from x and y should be
identical.
7 Point Estimation
This chapter deals with estimating parameters of a population by looking at random samples.
The first section discusses strategies for finding estimators; the second section deals with ways
to evaluate the estimators.
7.1
Methods of Finding Estimators
7.1.1 Method of Moments for Finding Estimator
Basically, you can equate expected values of arbitrary functions of the random sample with
realized value to estimate parameters. Method of moments use the first k sample moments
to achieve this. Suppose we have Xi ∼ n θ, σ 2 . The first moment would be θ, the second
”uncentered) moment would be θ2 + σ 2 . So we have a system of equations
"
1
n
¯
X
P
Xi2
#
=
"
θ
θ2 + σ2
#
On the left hand side there are concrete values from the random sample; so solving this is
trivial. Also note solving the above for σ gives
σ
˜2 =
X
1X 2
¯2 = 1
¯ 2
Xi − X
Xi − X
n
n
which does not have Bessel’s correction. This is an example where the method of moments
come short; it is a highly general method where you can fall back on, but in general it produces
relatively inferior estimators1 .
Interesting Example: Estimating Both Parameters Say we have Xi ∼ binomial (k, p) where
both k and p are unknown. This might be an unusual setting, but here’s an application: if we
are modeling crime rates – you don’t know the actual number of crimes committed k and the
reporting rate p. In this setting, it is hard to come up with any intuitive formulas. So we could
use the method of moments as a baseline.
Equating the first two moments, we get:
"
#
# "
¯
X
kp
=
1
2
kp (1 − p) + k 2 p2
n Xi
Solving this gives us an approximation for k:
1 However,
this is still a maximum-likelihood estimate.
20
k˜ =
which is a feat in itself!
¯ − (1/n)
X
¯2
X
P
¯
Xi − X
2
7.1.2 Maximum Likelihood Estimation
MLE is, by far, the most popular technique for deriving estimators. There are multiple ways
to find MLEs. Classic calculus differentiation is one, exploiting model properties is two, using
computer software and maximizing it numerically is three.
Couple things worth noting: using log-likelihood sometimes makes calculations much easier, so it’s a good idea to try taking logs. Another thing is that MLE can be unstable - if inputs
change slightly, the MLE can change dramatically. ”I wonder if there’s any popular regularization scheme for MLEs. Maybe you can still use ridge style regularization mixed with cross
validation.)
There’s a very useful property of maximum likelihood estimators called the invariance
prop ˆ
ˆ
erty of maximum likelihood estimators, which says that if θ is a MLE for θ, then r θ is a MLE
for r (θ). This is very useful to have to say the least.
7.1.3 Bayes Estimators
The Bayesian approach to statistics is fundamentally different from the classical frequentist
approach, because they treat parameters to have distribution themselves. In this approach,
we have a prior distribution of θ which is our belief before seeing the random sample. After
we see some samples from the population, our belief is updated to the posterior distribtion
following this formula:
π (θ|x) = ´
f (x|θ) π (θ)
f (x|θ) π (θ) dθ
which is just the Bayes formula.
Conjugate Families In general, for any sampling distribution, there is a natural family of
prior distributions, called the conjugate family. For example, binomial family’s conjugate family is the beta family. Let’s say if we have a binomial (n, p) distribution where only p is unknown.
If our prior distribution of p is beta (α, β), then after observing a sample y our posterior distribution updates as: beta (y + α, n − y + β). So classy and elegant! However it is debatable that
a conjugate family always should be used.
Also, normals are their own conjugates.
7.1.4 The EM Algorithm
The EM algorithm is useful when the likelihood function cannot be directly maximized; due
to missing observations or latent variables which are not observed. The setup goes like this;
21
given a parameter set θ, there are two random variables x and y of which only y is observed.
The marginal distribution for y, g (y|θ), is unknown, however we do know f (y, x|θ). Since x is
not observed ”or incomplete), it is hard to estimate θ from given y.
The EM algorithm solves it by an iterative approach. We start with a guess of the parameter
θ
(0)
, and the following two steps are repeated:
• Expectation step: Given the latest guess for the parameter θ(i) , and the observed y, find
the expected x.
• Maximization step: Given the x and y, find the parameter θ(i+1) that is most plausible for
this pair of observations.
It can be proven that the likelihood from successive iterations of parameters are nondecreasing, and will eventually converge.
EM Algorithm, Formally The above two steps are just colloquial descriptions; the actual
expected x is not directly calculated. In the expectation step, the following function is created:
Q θ|θ(t) = EX|Y,θ [log L (θ; X, Y)]
Let’s see what this function is trying to do; since we have a guess for the parameter θ(t) ,
we now can plug this into the joint pdf and find the distribution of the pair (x, y). For each
concrete pair of observations, we can calculate the log-likelihood of θ using this observation.
Q is simply a weighted average of these likelihoods. Q is a function of θ, and answers this
question:
Let’s say (x, y) ∼ f x, y|θ(t) . Now what is the EV of log likelihood for a given θ?
Now the second step puts this function through argmax operator.
7.2
Methods of Evaluating Estimators
This section is much more involved than the previous section, and discusses multiple strategies
of evaluating estimators – also discusses some strategies to improve existing estimators.
7.2.1 Mean Squared Error
A simple, intuitive way of measuring the quality of the estimator is taking the MSE:
Eθ (W − θ)
2
note this is a function of the parameter θ. Also, MSE has a very intuitive decomposition:
Eθ (W − θ) = Varθ W + (Eθ W − θ)
2
22
2
The first term is the variance of the parameter, while the second term is the square of the
bias. Variance represents the precision of the estimator; if variance is high, even if the estimator works in average, it cannot be trusted. If the bias is high, it will point in a wrong direction.
So this tradeoff is a topic that is discussed throughout this chapter.
P
¯ 2 / (n − 1) is an unbiased estimaXi − X
Quantifying Bassel’s Correction We know S 2 =
n
tor of σ 2 , thus the bias is 0. However, we also know n−1
S 2 is the maximum likelihood estimator.
Which of these are better in terms of MSE? Carrying out the calculations tells you the MLE will
give you smaller MSE; so by accepting some bias, we could reduce the variance and the overall
MSE. Which estimator we should ultimately use is still debatable.
No Universal Winner Since MSE are functions of parameters, it often is the case the MSE of
two estimators are incomparable, because one can outperform another on a certain parameter
set, but can underperform on another parameter set. In the text, this is shown with an example
of estimating p from binomial (n, p) where n is known. Two approaches are showcased; one is
P
just using the MLE Xi /n, another is taking a Bayesian estimate from a constant prior.
If you plot the MSE against different values of p, MSE from the MLE is a quadractic curve;
but the MSE from the Bayesian estimate is a horizontal line. Therefore, one doesn’t dominate
the other – plotting for different n will reveal the Bayesian approach gets better as we increase
n.
7.2.2 Best Unbiased Estimator
One way of solving the bias-variance tradeoff problem is to ignore some choices! We can restrict ourselves to unbiased estimators and try to minimize variance. In this setting, the best
estimator is called a uniform minimum variance unbiased estimator ”UMVUE).
Finding UMVUE is a hard problem; it would be hard to prove that the given estimator is the
global minimum.
Finding Lower Bounds The Cramer-Rao Theorem sometimes helps; it is an application of the
Cauchy-Schwartz inequality which provides a lower bound for the variance of an unbiased
estimator. So if this limit is reached, we can say for sure we have a UMVUE. This is indeed
attainable for some distributions - such as Poisson.
The big honking problem of CR theorem is that the bound is not always sharp – sometimes
the bound is not attainable. Since CR is an application of CS inequality, we can state the condition under which the equality will hold. This can either give us hints about the shape of the
UMVUE, or let us prove it is unreachable.
Sufficiency and Completeness for UMVUE Somewhat surprising and arbitrary - but sufficient or complete statitics can be used to improve unbiased estimators, by conditioning the
23
estimator on the sufficient statistic! The Rao-Blackwell theorem specifies that if W is any unbiased estimator of an arbitrary function of θ, τ (θ), the following function is a uniformly better
unbiased estimator:
φ (T ) = E (W |T )
This was a WTF moment for me. However, note that
Varθ W = Varθ [E (W |T )] + E [Var (W |T )]
≥ Varθ [E (W |T )]
for any T – you can condition on any statistic, even totally irrelevant, and it can never actually hurt. Then why the requirement for sufficiency? If not, the estimator will become a
function of θ so it isn’t exactly an estimator.
Can we go further than that? Can we create a UMVUE from sufficient statistics? We analyze
the properties of a best estimator to find out. Suppose we have an estimator W satisfying
Eθ W = τ (θ). Also say we have an unbiased estimator of 0, U . Then the following estimator
φa = W + aU
is still an unbiased estimator of τ (θ). What is the variance of this?
Varθ φa = Varθ (W + aU ) = Varθ W + a2 Varθ U + 2aCovθ (W, U )
Now, we can always concoct a U such that Covθ (W, U ) < 0 so Varθ φa < Varθ W . Note that U
is essentially a random noise for estimation - yet it can give us an improvement. This doesn’t
make sense at all. Theorem 7.3.20 actually brings order, by stating that W is the UMVUE iff it
is uncorrelated with all unbiased estimators of 0.
Wait, that definition sounds familiar. Recall the definition of complete statistics? Now we
have to guess they are somewhat related. They indeed are, by Theorem 7.3.23. This states:
Let T be a complete sufficient statistic for a parameter θ, and let φ (T ) be any estimator based only on T , then φ (T ) is the unique best unbiased estimator of its expected
value.
7.2.3 General Loss Functions
MSE is a special case of loss function, so we can generalize on that. Absolute error loss and
squared loss are the most common functions, but you can come up with different styles, of
course.
Evaluating an estimator on a given loss function L (θ, a) is done by a risk function:
R (θ, δ) = Eθ L (θ, δ (X))
24
which is still a function of the parameter. Also note when we use squared loss function, R
becomes MSE.
Stein’s Loss Function For scale parameters such as σ 2 , the values are bounded below at 0.
Therefore, a symmetric loss function such as MSE can be penalizing overestimation because
the penalty can grow infinitely only in one direction. An interesting class of loss function,
which tries to overcome this problem, is introduced:
a
a
L σ 2 , a = 2 − 1 − log 2
σ
σ
which goes to infinity as a goes to either 0 or infinity. Say we want to estimate σ 2 using an
estimator δb = bS 2 for a constant b. Then
R σ 2 , δb = E
bS 2
bS 2
− 1 − log 2
2
σ
σ
which is minimized at b = 1.
Bayes Risk
= b − log b − 1 − E log
S2
σ2
Instead of using the risk function as a parameter of θ, we can take a prior distri-
bution on θ and getting the expected value. This takes us to the Bayes risk:
ˆ
R (θ, δ) π (θ) dθ
Θ
Finding the estimator δ which minimizes the Bayes risk seems daunting but is tractable.
Note:
ˆ
L (θ, δ (x)) f (x|θ) dx π (θ) dθ
Θ
χ
ˆ ˆ
=
L (θ, δ (x)) π (θ|x) dθ m (x) dx
R (θ, δ) π (θ) dθ =
Θ
ˆ ˆ
χ
Θ
The quantity in square brackets, which is the expected loss given an observation, is called the
posterior expected loss. It is a function of x, not θ. So we can minimize the posterior expected
loss for each x and minimize Bayes risk. A general recipe for doing this is not available, but
the book contains some examples of doing this.
8 Hypothesis Testing
This chapter deals with ways of testing hypotheses, which are statements about the population
parameter.
25
8.1
Terminology
In all testing, we use two hypothesis; the null hypothesis H0 and the alternative hypothesis
H1 , one of which we will accept and the other we will reject. We can always formulate the
hypothesis as two sets: H0 is θ ∈ Θ0 and H1 is θ ∈ ΘC
0 . We decide which hypothesis to accept
on the basis of a random sample. The subset of the sample space which will make us reject H0
is called the rejection region.
Typically, a hypothesis test is performed by calculating a test statistic and rejecting/accepting H0 if the statistic falls into a specific set.
8.2
Methods of Finding Tests
We discuss three methods of constructing tests.
8.2.1 Likelihood Ratio Tests
Likelihood Ratio Tests are very widely applicable, and are also optimal in some cases. We
can prove t-test is a special case of LRT. The LRT calculates the ratio between the maximum
likelihood, and the maximum likelihood given the null hypothesis. The test statistic λ is defined
as:
λ (x) =
supΘ L(θ|x)
0
supΘ L(θ|x)
=
supΘ L(θ|x)
0
ˆ
L θ|x
(θˆ is MLE of θ)
When do we reject H0 ? We want to reject the null hypothesis when the alternative hypothesis is much more plausible; so the smaller λ is, the more likely we will reject H0 . Therefore,
the rejection region is {x|λ (x) ≤ c} where 0 ≤ c ≤ 1.
Also note that λ based on a sufficient statistic of θ is equivalent to regular λ.
Example: Normal LRT for testing θ = θ0 , known variance
Setup: we are drawing sample from n (θ, 1) population. H0 : θ = θ0 and H1 : θ 6= θ0 . Since the
MLE is x
¯, the LRT statistic is:
λ (x) =
h
i
f (x|θ0 )
2
= ”some simplification) = exp −n (¯
x − θ0 ) /2
f (x|¯
x)
Say we reject H0 when λ ≤ c; we can rewrite this condition as
|¯
x − θ0 | ≥
p
−2 (log c) /n
Therefore, we are simply testing the difference between the sample mean and the asserted
mean with a positive number.
26
Example: Normal LRT for testing µ < µ0 , unknown variance
Setup: we are drawing from n µ, σ 2 where both parameters are unknown. H0 : µ ≤ µ0 and
H1 : µ > µ0 . The unrelated parameter σ 2 is called a nuisance parameter. It can be shown ”from
exercise 8.37) that the test relying on this statistic is equivalent to Student’s t test.
8.2.2 Bayesian Tests
Bayesian tests will obtain posterior distribution from the prior distribution and the sample,
using the Bayesian estimation technique discussed in Chapter 7. And we use the posterior
distribution to run the test. We might choose to reject H0 only if P θ ∈ ΘC
0 |X is greater than
some large number, say 0.99.
8.2.3 Union-Intersection and Intersection-Union Tests
Ways of structuring the rejection region when the null hypothesis is not simple. For example,
the null hypothesis ΘC
0 is best expressed by an intersection or union of multiple sets. If we
have tests for each individual sets that constitute the union/intersection, how can we test the
entire hypothesis?
• When the null hypothesis is an intersection of sets, any test failing will result in the rejection of the null hypothesis.
• When the null hypothesis is a union of sets, any test passing will let us reject the null
hypothesis; null hypothesis is rejected only when all of the tests fail.
8.3
Evaluating Tests
How do we assess the goodness of tests?
8.3.1 Types of Errors and Power Function
Since the null hypothesis could be true or not, and we can either accept or reject it, there are
four possibilities in a hypothesis test, which can be summarized in the below table:
Decision
Truth
Accept H0
Reject H0
H0
Correct
Type I Error
H1
Type II Error
Correct
To reiterate:
• Type I Error incorrectly rejects H0 when H0 is true. Since we usually want to prove the
alternative hypothesis, this is a false positive error, when we assert H1 when it is not.
Often this is the error people want to avoid more.
27
• Type II Error incorrectly accepts H0 when H1 is true. This is a false negative error; the
alternative hypothesis is falsely declared negative.
The probabilities of these two errors happening are characterized by a power function. A
power function takes θ as an input, and calculates the probability that X will be in the rejection
region:
β (θ) = Pθ (X ∈ R)
What is an ideal power function? We want β (θ) to be 0 when θ ∈ Θ0 , and 1 when θ ∈ ΘC
0.
We can plot β with regards to all possible values of θ, and compare the different tests.
Example: Normal Power Function
Setup: drawing from n θ, σ 2 , where σ 2 is known. H0 : θ ≤ θ0 , H1 : θ > θ0 . Say we have a test
that rejects H0 when
¯ − θ0
X
√ >c
σ/ n
The power function of this test is
¯
X − θ0
√ >c
β (θ) = Pθ
σ/ n
¯
θ−θ
X −θ
√ > c+ √0
= Pθ
σ/ n
σ/ n
θ − θ0
=P Z >c+ √
σ/ n
where Z is a standard normal random variable. As we increase θ from −∞ to ∞, this probability will go from 0 to 1. Changing c will change when we reject H0 and affect the error
probabilities.
1.0
0.6
0.4
0.6
0.4
0.2
0.2
0.0
−1.0
n=10
n=20
n=100
θ0
0.8
β(θ)
β(θ)
0.8
1.0
c=0.1
c=0.5
c=1.0
θ0
−0.5
0.0
0.5
1.0
θ
1.5
2.0
2.5
0.0
−1.0
3.0
”a)
−0.5
0.0
0.5
1.0
θ
1.5
2.0
2.5
3.0
”b)
The figure ”a) gives an example of different c – since H0 is true when θ is left of θ0 , the type I
error probability is depicted by the distance between the x-axis and the power function curve.
On the right of θ0 , the type II error probability is depicted by the distance between y = 1 and
28
the curve. We can see that increasing c trades off the maximum type I error and the maximum
type II error: if we have a higher c, we have a more strict criteria for rejecting H0 . This results
in lower false positive, but in higher false negative.
Consequences of the Power Function
Figure ”b) above shows the power function for different ns. We see as we increase n, the power
function approaches the ideal step function. Typically, the power function will depend on the
sample size n. If n can be chosen by the experimenter, considering the power function will be
useful to determine the sample size before the experiment is performed.
Size and Level of Tests
For a fixed sample size, it is usually impossible to make both types of error probability arbitrarily small. As most people care more about Type I errors ”since alternative hypothesis are what
we want to prove), we classify tests by the maximum possible type I error across all possible
parameters in Θ0 : supθ∈Θ0 β (θ)
• We want these numbers to be low, since β (θ) is ideally 0 for Θ0 . Lower size/level means
more powerful tests!
• Note as we do not have a prior distribution of parameters, we use the maximum error
probability rather than the expected error probability.
• When this bound α is tight ”supθ∈Θ0 β (θ) = α), the test is called a test of size α. When this
bound is not tight ”supθ∈Θ0 β (θ) ≤ α), the test is called a test of level α.
– So levels are upper bounds of test powers; every test is a level 1 test but that is not
informative.
Unbiased Tests
A test with power function β (θ) is unbiased if
sup β (θ) ≤ inf β (θ)
θ∈ΘC
0
θ∈Θ0
Colloquially, if when the alternative hypothesis is true, it should be more likely to reject the
null hypothesis than when the null hypothesis is true.
8.3.2 Most Powerful Tests: Uniformly Most Powerful Tests
Depending on the problem, there might be multiple level α tests. What should we use among
them? Since we have only controlled for Type I errors so far, it is a good idea to look at Type II
errors this time. The Uniformly Most Powerful test is the most powerful test across all θ ∈ ΘC
0:
its power function β should be larger than any other test’s power function, for any θ ∈ ΘC
0.
29
Neymann-Pearson Lemma
UMP has a very strong argument; we can easily imagine situations where the UMP does not
exist. However, in some cases, they do. When you are only considering two hypothesis H0 :
θ = θ1 and H1 : θ = θ2 , accepting H0 when and only when f (x|θ1 ) < kf (x|θ0 ) for some k ≥ 0 is a
UMP level α test, where α = Pθ0 (X ∈ R).
Colloquially, if your test is formed like above, it is a UMP for its own class ”level α).
Extending Neyman-Pearson Lemma To A One-Sided Test
In case of a one-sided hypothesis ”for example, θ > θ0 ), we can extend the above lemma to find
a UMP.
First, we need to define the following concept: a pdf has a MLR ”monotone likelihood ratio)
property if: whenever θ2 > θ1 , the likelihood ratio f (t|θ2 ) /f (t|θ1 ) is a monotone function of
t. ”This holds for all the regular exponential families g (t|θ) = h (t) c (θ) ew(θ)t when w (θ)is a
nondecreasing function.)
Now, we state the following theorem:
Karlin-Rubin Theorem With H0 : θ ≤ θ0 and H1 : θ > θ0 . Suppose that T is a sufficient statistic
of θ and its distribution has a nondecreasing MLR. Then, for any t0 , rejecting H0 iff T > t0
is a UMP level α test, where α = Pθ0 (T > t0 ).
In laymen’s terms: when the conditions are met, a hypothesis about a parameter above or
below the threshold can be translated into a sufficient statistic above or below a threshold. So
this is actually very intuitive, maybe even trivial, stuff.
When UMP Cannot Be Found
In case of two-sided hypothesis, the above theorem is not applicable. For example; drawing
from n θ, σ 2 , σ 2 known. Say H0 : θ = θ0 and H1 : θ 6= θ0 . We look at two alternative values of
θ, θ1 < θ0 < θ2 . We are able to show that different tests are optimal for θ1 and θ2 , while having
the same level. Details:
¯ < θ0 − σzα /√n ⇐⇒ Z < −zα −
• Test 1 rejects H0 when X
¯ > θ0 + σzα /√n ⇐⇒ Z > zα +
• Test 2 rejects H0 when X
√
√
n(θ−θ0 )
σ
n(θ0 −θ)
σ
Note they are pathological devices, only controlling for Type I error for the sake of the proof;
√
2
¯ − θ · n is going to
¯
both tests have spectacular Type II errors. Anyways, as X ∼ n θ, σn , X
σ
be a standard normal variable. Therefore you can imagine both tests having level α. However,
we can easily see the power for Test 1 would be higher for θ1 < θ0 than Test 2, and vice versa
for θ2 > θ0 . So there are no UMPs for level α.
30
Unbiased UMPs
Note the above tests are pathetically biased. What we should do is rejecting H0 when
√
¯ − θ0 > σzα/2 / n
X
Obviously. The below figure shows the power function of the three tests:
1.0
0.8
0.6
0.4
biased 1
biased 2
best unbiased
0.2
0.0
θ0
Note that the best unbiased estimator has a lower probabilty of rejecting H0 for some range
of θ0 , but it is clear it is better than both tests in general.
8.3.3 Size of UIT and IUT
The size/level analysis of UIT and IUT are different. UITs are universally less powerful than
LRTs, but they might be easier to reason about.
8.3.4
p-Values
The definition of p-value is very unintuitive, actually. In the definition, a p-value p (X) is a test
statistic. Small values of p (X) give evidence that H1 is true. A p-value is valid if, for every
θ ∈ Θ0 and every 0 ≤ α ≤ 1,
Pθ (p (X) ≤ α) ≤ α
So WTF does this mean? p (X) ≤ α can be interpreted as the condition p-value having a
value of α, or something more extreme . The probability of this happening given a θ in the
null hypothesis should roughly be α. So this sort of resonates with the colloquial definition of
p-value.
Also note that a test that rejects H0 when p (X) ≤ α is a level α test from the above definition.
Defining Valid p-Values
I guess this is the more intuitive definition: let W (X) be a test statistic such that large values
of W give evidence that H1 is true. Then let
31
p (x) = sup Pθ (W (X) ≥ W (x))
θ∈Θ0
so this is the colloquial definition. We can prove that p (x) is a valid p-value according to the
above definition.
Another method for defining p-values is discussed as well that depends on sufficient statistics. If S (X) is a sufficient statistic of θ when the null hypothesis is true, we have:
p (x) = P (W (X) ≥ W (x) |S = S (x))
is a valid p-value. This makes sense as well; if we have a sufficient statistic we need no
stinking inference about θ....
Calculating p-Values
Calculating the supremum probability isn’t always easy; but it can be derived from the properties of the distribution. Our usual normal tests are good examples. In these cases, the test
statistic follows Student’s t distribution, enabling us to nail the supremum.
8.3.5 Loss Function Optimality
Since there are just two actions in a hypothesis test ”we either accept H0 or H1 ), a loss function
is defined simply by:
L (θ, a0 ) =

0
C
θ ∈ Θ0
II
θ∈
L (θ, a1 ) =
ΘC
0

C
0
I
θ ∈ Θ0
θ ∈ ΘC
0
where ai is the action of accepting Hi . CI and CII are the costs of type I and type II errors,
respectively, and they can be a function of θ as well.
9 Interval Estimation
9.1
Introduction
Interval estimation techniques give us confidence intervals of parameters; for example, we can
assert µ ∈ [L (x) , U (x)] where L (x) and U (x) are functions of the random sample that predicts
the parameter. What do we gain from going to interval estimations from point estimations?
The probability of a point estimation being correct is 0, if the parameter is continuous. However, once we have an interval estimation, we can have nonzero probability of our prediction
being correct.
Here are some definitions:
• A coverage probability of [L (X) , U (X)] is a function of θ, that represents the probability
that the interval covers the true parameter.
32
Pθ (θ ∈ [L (X) , U (X)])
Note that the interval estimates are functions of X, thus are random variables themselves.
• A confidence coefficient is the infimum of coverage probability across all possible
inf P (θ ∈ [L (X) , U (X)])
θ
I want to emphasize the consequences of this definition: don’t think interval estimation in
terms of conditional probability! A coverage probability is not related to the conditional prob¯ seemed like a very obvious way for
ability. Personally conditional probability of θ given X
interval estimation, but it requires a prior distribution of θ and is strictly not a frequentist
approach. The coverage probability is more like a likelihood function. It’s just
ConverageProb (θ) =
ˆ
f (x, θ) dx
C
where f is the pdf and C = {x|L (x) ≤ θ ≤ U (x)}. This also changes how we interpret the
results of an estimation process: the claim that 95% confidence interval of θ is [1, 10] does not
mean P (θ ∈ [1, 10]) = 0.95, but P (x|θ) ≥ 0.95 for all θ ∈ [1, 10].
9.1.1 Coverage Probability Example: Uniform Scale Distribution
In many cases, the coverage probability is constant across different parameters. However,
it may not be so in certain cases. The following example demonstrates this. Say we draw
Xi ∼ uniform (0, θ). The sufficient statistic for θ is Y = max Xi . Now, what are our estimates
for θ? We examine two candidates:
• Scale intervals:[aY, bY ] where 1 ≤ a < b
• Location intervals: [Y + c, Y + d] where 0 ≤ c < d
Now let’s send that θ to infinitely. Location interval will have a coverage probability of 0, since
the size of the interval stays constant regardless of θ. However, the scale interval manages to
have a positive coverage probability because the interval’s size roughly grows with θ. ”These
are more intuitive descriptions; formally they are proven by integrating pdfs of Y /θ. Also see
the later section on pivotal quantities.)
9.2
Methods of Finding Interval Estimators
9.2.1 Equivalence of Hypothesis Test and Interval Estimations: Inverting a Test Statistic
Hypothesis tests and interval estimations are obviously, very closely related. They both take a
random sample to make an inference about the population parameter. To draw a poor analogy: both processes throw darts to a board. Hypothesis testing tell you the target position to
33
shoot for, then you throw the dart, and we will know if we landed within a certain range from
the target. Interval estimation lets you throw the dart first; and gives us target positions you
might have shot for.
Here’s a general recipe for converting a hypothesis test into an interval estimation; just go
over all the possible values of θ, use a test of size α for H0 : θ = θ0 where θ0 is a possible value.
If this test passes, θ0 falls within the confidence interval. So:
C (T (x)) = {θ : P (T (x) |θ) ≥ α}
Of course, in real life, this is not done by enumerating all possible values but analytically.
Normal Test Example
Let Xi ∼ n µ, σ 2 . Consider testing H0 : µ = µ0 and H1 : µ 6= µ0 . Hypothesis test draws a region
on the dart, called the acceptance region A (µ0 ), upon which we will accept H0 . Let’s use a test
√ with acceptance region x : |¯
x − µ0 | ≤ zα/2 σ/ n . This test has a size of α.
Now, let’s throw the dart and get a concrete x
¯. Now, we wiggle µ0 so its acceptance
region will still contain x
¯. It’s easy to see the acceptance region will contain x
¯ when
σ
σ
x
¯ − zα/2 √ ≤ µ0 ≤ x
¯ + zα/2 √
n
n
Yay, this is a range of parameters. Now what is the confidence coefficient? From the definition of the test, we know
P (x ∈ R (µ0 ) |µ = µ0 ) = α ⇐⇒ P (x ∈ A (µ0 ) |µ = µ0 ) = 1 − α
So it’s 1 − α!
Formal Definition (Theorem 9.2.2)
For each θ0 ∈ Θ, let A (θ0 ) be the acceptance region of a level α test of H0 : θ = θ0 . For each
x ∈ X , define a set C (x) in the parameter space by
C (x) = {θ0 : x ∈ A (θ0 )}
Then the random set C (x) is a 1 − α confidence set. The converse holds true as well; if C is
a 1 − α confidence set then
A (θ0 ) = {x : θ0 ∈ C (x)}
is a valid acceptance region for a level α test.
34
Shapes of Confidence Interval
Also, note the above definition do not include the form of H1 . In general, one-based H1 produces one-sided C (x) and two-sided H1 produces two-sided C (x). The biased property of the
inverted test also carries over to the confidence set, as can be expected.
What do we do if we want C (x) = (−∞, U (x)] i.e. only upper-bounded? If a test uses
H1 : µ < µ0 , its acceptance region will look like {x : x
¯ < µ0 − t} where t is a function of various
things ”sample size, nuisance parameters, etc..). i.e. the acceptance region is left-bounded if
looked from µ0 . Now, once we have x
¯, we can move µ0 infinitely to the left, but cannot move
µ0 too far to the right. So the confidence interval is one-sided only with a upper bound.
9.2.2 Using Pivotal Quantities
A pivotal quantity is a random variable whose distribution
does not depend on the parameter.
√
¯
2
2
For example, if Xi ∼ n µ, σ with σ known, X − µ / (σ/ n) is normally distributed and is
a pivotal quantity. Then, we can trivially construct a confidence interval for µ. First note that
P
¯ −µ
X
√ ≤ a = P (−a ≤ Z ≤ a) = α
−a ≤
σ/ n
Where a is chosen to make the last equality hold. So what µ will make this inequality hold?
It’s easy to see the following will do it.
σ
σ
µ:x
¯ − a√ ≤ µ ≤ x
¯ + a√
n
n
=
σ
σ
µ:x
¯ − z1−α √ ≤ µ ≤ x
¯ + z1−α/2 √
n
n
Note that we choose a = z1−α/2 which splits the probability equally here. It’s a matter
of choice, but for symmetric distributions such as normal, it seems natural. For asymmetric
distributions such as chi-squared, a different approach will be better ”we will revisit this).
In General
If Q (x, θ) is a pivotal quantity, we can a and b so that Q (x, θ) will fall in that interval with a
probability of α ”we can plug in concrete numbers, because we know Q’s distribution). Then,
we have:
C (x) = {θ0 : a ≤ Q (x, θ0 ) ≤ b}
Now, we unravel Q’s definition and leave only the desired parameter in the middle, moving
everything to the left and right sides. Then we have a confidence interval.
9.2.3 Pivoting CDFs Using Probability Integral Transformation
The previous strategies can sometimes result in non-interval confidence sets, as shown in example 9.2.11. This strategy, when applicable, will always result in an interval. Say T (x) is
35
a sufficient statistic for θ with cdf FT (t|θ). Recall, from probability integral transform, that
FT (T |θ) ∼ uniform (0, 1). So this is a pivotal quantity!
For the confidence set to be an interval, we need to have FT (t|θ) to be monotone in θ. If it
is monotonically increasing, we set θL (t) and θU (t) to be solution of the following system:
FT (t|θL (t)) = α1
FT (t|θU (t)) = 1 − α2
where α1 + α2 = α. ”We usually set α1 = α2 = α/2 unless additional information. We will
discuss about this in section 9.3) Note the above equations do not have to be solved analytically
– we can solve them numerically as well.
9.2.4 Bayesian Inference
As noted in the beginning of the chapter, the frequentist approach does not allow to say the
parameter belongs to the confidence set with a probability of α: parameter is a fixed value and
it will belong to the set with a probability of 1 or 0. The Bayesian setup precisely allows us to
do that.
The Bayesian equivalent of a confidence set is called a credible set, to avoid the confusion
between the two approaches. The equivalent of a coverage probability is called a credible
probability.
In many cases, each approach looks poor if examined using the criteria from a different
approach. The textbook shows some examples where the credible set has a limiting converage
probability of 0, and the confidence set having a limiting credibility of 0.
9.3
Methods of Evaluating Interval Estimators
We can have multiple ways of coming up with interval estimation, with the same confidence
coefficient. Some is bound to be better than the others. But how do we evaluate them?
9.3.1 Size and Coverage Probability
When two interval estimations have the same confidence coefficient, it is obvious to prefer a
smaller interval. When we estimate the population mean of a normal distribution, we can
pick any α1 , α2 ≥ 0 where α = α1 + α2 such that
P (zα1 ≤ Z ≤ zα2 ) = 1 − α
Obviously there are multiple choices for those. Intuition tells us that splitting probabilities
evenly for the left and right hand side of the sample mean is the way to go, and it indeed is. It
is justified by the following theorem:
Theorem 9.3.2 Let f (x) be a unimodal pdf. We consider a class of intervals [a, b] where
´b
f (x) dx = 1 − α. If f (a0 ) = f (b0 ) > 0, and x∗ ∈ [a0 , b0 ] where x∗ is the mode of f , then
a
36
[a0 , b0 ] is the shortest interval which satisfies the 1 − α probability constraint.
The textbook discusses a caveat for this ”See example 9.3.4)
9.3.2 Test-Related Optimality
Optimality criteria for tests carry over to their related interval estimations. For example, we
have some guarantee for interval estimations which are inversions of UMP tests. However,
note that UMP controls type-II errors; therefore the guarantee we have for the interval estimation also looks different. UMP-inverted estimations give us optimal probability of false coverage. Colloquially, it is a function of θ and θ′ and measures the probability of θ′ being covered
when θ is the truth.
Note the probability of false coverage has to take different forms according to the form of
the original test. Say the the interval estimation is one-sided only with upper bound. We only
define the probability of false coverage for θ < θ′ , obviously – θ′ less than or equal to θ are
correct.
Intuition Behind False Coverage Probability We do offer some intuition about the linkage
between the false coverage probability and the size of the interval; if an interval contains less
false parameters, it’s more likely to be short; there are some loose links discussed in the book
”see theorem 9.3.9) which says the expected length of the interval for a given parameter value
is equivalent to an integral over false positive probability.
Uniformly Most Accurate (UMA) Intervals A conversion of a UMP test yields a UMA confidence set, which has the smallest probability of false coverage.
Also note that UMP tests are mostly one-sided – Karlin-Rubin works only for one sided intervals. So most UMA intervals are one sided.
Inverting Unbiased Tests The biasedness of tests carry over to interval estimation. In case
UMP does not exist, we can invert the unbiased test to get an unbiased interval.
9.3.3 Bayesian Optimality
When we have a posterior distribution, we can order all intervals of probability of 1 − α by
their size. A corollary to Theorem 9.3.2 mentioned above gives us that when the posterior
distribution is unimodal,
{θ : π (θ|x) ≥ k}
is the shortest credible interval for a credible probability of 1 − α =
a region is called the highest posterior density ”HPD) region.
37
´
π(θ|x)≥k
π (θ|x) dx. Such
9.3.4 Loss Function Optimality
So far we have set the minimum coverage probability we want, and then find the best tests
among them. We can do something in the middle by using a generalized loss function such as
L (θ, C) = b · Length (C) − IC (θ)
where IC (θ) = 1 if θ ∈ C, 0 otherwise. By varying b, we can very the relative importance of
the length and coverage probability.
However, the use of decision theory in interval estimation problems is not widespread; it
is hard to solve in many cases, and this can sometimes lead to unexpected types of sets.
10 Asymptotic Evaluations
This chapter looks at asymptotic behaviors of several topics ”point estimation, hypothesis testing, and interval estimation). That is, we send sample size to infinity and see what happens.
The detailed treatment of the topic seems to be the most theoretical part of the book; I just
skimmed over the chapter and will try to summarize only the big ideas here.
10.1
Point Estimation
10.1.1 Criteria
A point estimator’s asymptotic behavior is characterized by two properties;
• Consistency means the estimator converges to the correct value as the sample size becomes infinite.
lim Pθ (|Wn − θ| < ǫ) = 1
n→∞
• Efficiency looks at the variance of the estimator as the sample size becomes infinite. If
the variance reaches the lower bound defined by the Cramer-Rao, the estimator is called
asymptotically efficient.
Concretely, an asymptotically efficient estimator Wn for a parameter τ (θ) if
√
n [Wn − τ (θ)] → n [0, v (θ)]
and v (θ) achieves the Cramer-Rao Lower Bound.
It is notable that MLEs are in general best estimators: they are consistent estimators, and
asymptotically efficient at the same time. ”Some regularity conditions are required, but it
is explicitly said to hold in most common situations and you don’t really want to care about it.)
38
10.1.2 Comparing Consistent Estimators
MLE is everyone’s favorite estimator. However, other estimators may have other desirable
properties ”robustness, ease of calculation, etc) so we need to be able to see what we are giving
up in terms of efficiency. Comparing different asymptotically consistent estimators is done
by looking at their variances, through the idea of asymptotic relative efficiency ”ARE). If two
estimators Wn and Vn satisfy
√
n [Wn − τ (θ)]
√
n [Vn − τ (θ)]
Then, the ARE is defined by
2
→ n 0, σW
→ n 0, σV2
ARE (Vn , Wn ) =
2
σW
σV2
ARE is going to be a function of the parameters – so we will be able to see where it peaks,
where is it larger/smaller than 1, etc.
10.1.3 Asymptotic Behavior of Bootstrapping
Bootstraps are good ways to estimate variances of arbitrary estimators. For any estimator
ˆ
θˆ (x) = θ,
Var θˆ =
∗
nn
1 X ˆ∗ ¯ˆ∗ 2
θ −θ
n
n − 1 i=1 i
Of course, we cannot sample all nn possible samples. So we can always do a partial bootstrap by taking B random resamples. The book shows some examples where bootstrap yields a
better variance estimate than the Delta method ”which exploits that when estimates are asymptotically efficient, they will reach Cramer-Rao bound – so naturally it will underestimate.).
Parametric and Nonparametric Bootstrap
The usual type of bootstrapping, which draws data ”with replacement) to generate random
samples is called a nonparametric bootstrap. On the contrary, parametric bootstrapping asˆ and generate random samples
sumes a distribution. Say Xi ∼ f (x|θ). We take MLE estimate θ,
from there.
X1∗ , X2∗ , · · · , Xn∗ ∼ f x|θˆ
It is same as the usual nonparametric bootstrap from there.
39
Consistency and Efficiency
The textbook does not cover a lot of material about the evaluation of bootstrapping. In general,
it is an effective and reasonable way.
10.2
Robustness
1. It should have a reasonably good efficiency at the assumed model.
2. It should be robust in the sense that small deviations from the model assumptions should
impair the performance only slightly.
3. Somewhat larger deviations from the model should not cause a catastrophe.
10.2.1 Robustness of Mean and Median
The mean reaches the Cramer-Rao bound, so it is an efficient estimator. Also, if there is a
small variation from the normal distribution assumption, it will fare pretty well. We can try
to see this by a δ-contamination model. There, the distribution is the assumed model with a
probability of 1−δ, some other distribution with δ. Note when the two distributions are similar,
mean’s variance is still small. However, if the other distribution is Cauchy, for example, the
variance will go to infinity.
That brings us to the notion of breakdown value. A breakdown value b is the maximal portion of the sample which can go to infinity before the statistic goes infinity, or meaningless .
Of course, the mean’s breakdown value is 0, where the median has 0.5.
How can we compare the mean and the median? We can prove median is asymptotically
normal, and use ARE to compare those in different types of distributions. Some examples in
the book show it will fare better than the mean in double exponential distribution, but not in
normal or logistic. So it fares better with thicker tails, as we expect.
10.2.2 M-estimators
M-estimators is a generalized form of estimator. Most estimators minimize some type of criteria - for example, squared error gives the mean, absolute error gives the median, and the
negative log likelihood will give you MLE. M-estimators are a class of estimators which minimize
n
X
i=1
ρ (xi − a)
Huber’s Loss
Huber’s loss is a Frankenstein-style loss function created by patching squared and absolute
loss together.
40
ρ (x) =

 1 x2
2
k |x| − 1 k 2
2
if |x| ≤ k
otherwise
Works like quadratic around zero, and linear away. It’s differentiable, and continuous. k
is a tunable parameter. Increasing k would be like decreasing the robustness to outliers. The
minimizer to this loss function is called the Huber estimator and it is asymptotically normal
with mean θ. If you do an ARE comparison with mean and median;
• Huber is close to mean in normal, and better than mean at logistic or double exponential.
• Huber is worse than median in double exponential, much better than it in normal or
logistic.
10.3
Hypothesis Testing
10.3.1 Asymptotic Distribution of LRT
When H0 : θ = θ0 and H1 : θ 6= θ0 . Suppose Xi ∼ f (x|θ). Then under H0 , as n → ∞,
−2 log λ (X) → χ21 in distribution
Regardless of the original distribution! Kicks ass. A more general version, which do not
specify H0 and H1 explicitly, states that the quantity −2 log λ (X) will still converge to chi-
squared distribution with its df equal to the difference between the number of free parameters
in Θ0 and the number of free parameters in Θ1 .
How do we define the number of free parameters? Most often, Θ can be represented as a
subset of Rq and Θ0 can be represented as a subset of Rp . Then q − p = v is the df for the test
statistic.
10.3.2 Wald’s Test and Score Test
Two more types of tests are discussed.
Wald Test
When we have an estimator Wn for a parameter θ, which will asymtotically normal, we can
use this as a basis for testing θ = θ0 . In general, a Wald test is a test based on a statistic of the
form
Zn =
Wn − θ0
Sn
Sn is standard error for Wn . ”Last time I’ve seen this was in logistic regression; they did
Wald Test on regression coefficients to derive p-values of coefficient being nonzero.)
41
Score Test
10.4
Interval Estimation
11 Analysis of Variance and Regression
11.1
One-way ANOVA
ANOVA is a method of comparing means of several populations, often assumed to be normally
distributed. Normally, the data are assumed to be follow the model
Yij = θi + ǫij
where θi are unknown means and ǫij are error random variables. The classic oneway
ANOVA assumptions are as follows:
1. Eǫij = 0, Varǫij = σi2 < ∞, and all errors are uncorrelated.
2. ǫij are independent, normally distributed
3. σi2 = σ 2 for all i ”also known as homoscedascity)
11.1.1 Different ANOVA Hypothesis
The classic one-way ANOVA null hypothesis states
H0 : θ 1 = θ 2 = θ 3 = · · · = θ k
which is kind of uninformative, and doesn’t give us much information. The text starts with a
more useful type of hypothesis, using contrasts. A contrast is a linear combination of variables
/parameters where the weights sum up to 0. We now run the test with different hypothesis,
the null being
H0 :
k
X
ai θi = 0
i=1
Now, by choosing the weights carefully, we can ask other types of interesting questions.
a = (1, −1, 0, 0, · · · , 0) will ask if θ1 = θ2 . a = 1, − 21 , − 12 , 0, 0, · · · , 0 will ask if θ1 = (θ2 + θ3 ) /2,
etc.
11.1.2 Inference Regarding Linear Combination of Means
The means of each sample Y¯i are normal:
ni
1 X
Yij ∼ n θi , σ 2 /ni
Y¯i =
ni j=1
42
The linear combination of normal variables are once again, normal, with the following
parameters:
k
X
i=1
ai Y¯i ∼ n
k
X
ai θi , σ 2
i=1
k
X
a2
i
i=1
ni
!
Since we don’t know the variance, we can replace this as the sample variance. Si2 is the
regular sample variance from the ith sample. Then, the pooled estimator is given by
Sp2 =
k
k ni
1 XX
1 X
2
(ni − 1) Si2 =
(yij − y¯i· )
N − k i=1
N − k i=1 j=1
The estimator Sp2 has an interesting interpretation: it is the mean square within treatment
groups. This makes sense, since σ 2 only affects the variance within a group in our model. Then,
replacing the variance with Sp2 , we get the following which has usual Student’s t-distribution
with N − k degrees of freedom.
Pk
¯ Pk a i θi
i=1 ai Yi −
i=1
q P
∼ tN −k
k
2
Sp i=1 a2i /ni
Now, we can do the usual t-test. In cases where we are only checking the equivalence of two
means, this is equivalent to the two-sample t test is that here information from other groups
are factored in estimating Sp2 .
11.1.3 The ANOVA F Test
How do we test the classic hypothesis? We can think of it as an intersection of multiple hypothesis:
θ ∈ {θ : θ1 = θ2 = · · · = θk } ⇐⇒ θ ∈ Θa for all contrast a ⇐⇒ θ ∈ ∩a∈A Θa
We can reject this intersection if the test fails for any a. We can test an individual hypothesis
with H0a by the following statistic:
Pk
P
i=1 ai Y¯i − ki=1 ai θi q P
Ta = k
Sp2 i=1 a2i /n
where we will reject it if Ta > k for some k. Now, all tests will pass if and only if
sup Ta ≤ k
a∈A
Where A = {a :
P
a = 0}. How can we find this supremum? Lemma 11.2.7 gives us the
exact form of a and the value of the supremum. However, the important thing is that:
43
sup Ta2 =
a∈A
Pk
i=1 ni
2
Y¯i − Y¯ − θi − θ¯
Sp2
∼ (k − 1) Fk−1,N −k
the statistic of which is called the F -statistic, and it gives us the F-test. Now, we can reject
H0 when
2
¯ / (k − 1)
¯
n
Y
−
Y
i
i=1 i
Pk
Sp2
> Fk−1,N −k,α
What is the rationale behind looking at F statistic? The denominator is the estimated variance within groups. The numerator is the mean square between treatment groups, weighted
2
by the size of the group. Y¯i − Y¯ is the squared error between the group mean and the grand
mean. ni weights them by the size of the group. Dividing by k − 1 is getting the average error
per group. Now, the ratio between these two quantities should be higher when inter-group
variance is high relative to the intra-group variance.
11.1.4 Simultaneous Estimation of Contrasts
A couple of strategies for making inferences about multiple equalities are discussed; the Bonferroni procedure and the Scheffe’s procedure. The Bonferroni procedure allows you to make
inferences about m pairs of means being equal. You have to set m in advance, and adjust the
level of the test so the intersection tests will be of desired power.
Scheffe’s procedure is more notable, which allows you to construct confidence intervals
on any arbitrary contrasts after the procedure is done. It is noted as a legitimate use of data
snooping. However, at the cost of power of inference, the intervals are usually wider. It goes
like:
p
If M = (k − 1) Fk−1,N −k,α , then the probability is 1 − α that
v
v
u
u
k
k
k
k
2
u X
u X
X
X
a
a2i
i
t
2
¯
¯
a i Y i − M Sp
ai Yi + M tSp2
ai θi ≤
≤
n
n
i=1
i=1
i=1 i
i=1 i
i=1
k
X
11.1.5 Partitioning Sum of Squares
ANOVA provides a useful way of thinking about the way in which different treatments affect
a measured variable. We can allocate variation of the measured variable to different sources,
because:
ni
k X
X
i=1 j=1
2
(yij − y¯) =
k
X
i=1
2
ni (¯
yi· − y¯) +
ni
k X
X
i=1 j=1
(yij − y¯i· )
2
2
which can easily be proved because (yij − y¯) = ((yij − y¯i ) + (¯
yi − y¯)) and when you evalu2
ate the square, cross terms are zero. The sums of squares are also chi-square distributed, after
scaling.
44
11.2
Simple Linear Regression
Simple linear regression is discussed in three different contexts - as a minimizer to the leastsquares without any statistical assumptions, as a best linear unbiased estimators under some
variance assumptions, as an inference mechanism under distribution assumptions. Not surprisingly, we will be able to draw more powerful conclusions when we assume more.
11.2.1 General Model
In all three different contexts, the actual line stays the same. Say (xi , yi ) are pairs of examples,
where xi are the predictor variables, yi being response variables. Then x
¯ and y¯ are means of
xi and yi , respectively, and
Sxx =
n
X
i=1
Syy =
n
X
i=1
Sxy =
n
X
i=1
(xi − x
¯)
(yi − y¯)
2
2
(xi − x
¯) (yi − y¯)
And then we will fit the following line y = bx + a with:
b=
Sxy
Sxx
a = y¯ − b¯
x
Note the slope can be interpreted as Cov (X, Y ) /VarX.
11.2.2 Least Square Solution
Least square provides a way to fit a line to the given data. No statistical inferences can be
drawn here. Let’s say we want to minimize the residual sum of squares
RSS =
n
X
i=1
(yi − (c + dxi ))
2
Now c can be determined easily - rewrite the summand as
2
(yi − (c + dxi )) = ((y − dxi ) − c)
2
and the minimizer of this is just the average of y − dxi which is y¯ − d¯
x. d can be determined
from differentiating the quadratic formula and setting it to 0.
Also note that changing the direction of the regression ”using y to predict x) will give you a
different regression line: this is obvious since b becomes Syy /Sxy .
45
11.2.3 Best Linear Unbiased Estimators: BLUE
Let’s add some contexts: we now think of the values yi as observed values of uncorrelated
random variables Yi . xi are known, fixed values chosen by the experimenter. We assume the
following model:
EYi = α + βxi
where VarYi = σ 2 , which is a common variance across variables. Or equivalently, set Eǫi = 0,
Varǫi = σ 2 and have
Yi = α + βxi + ǫi
Now, let us estimate α and β as a linear combination of Yi s
n
X
di Yi
i=1
Furthermore, we only look at unbiased estimators. With an unbiased estimator of slope β
must satisfy
E
n
X
di Yi = β
i=1
We can transform LHS as
E
n
X
i=1
di Yi =
n
X
di EYi =
n
X
di (α + βxi ) = α
i=1
i=1
i=1
n
X
di
!
+β
For this to be β, we need the following conditions to hold true:
n
X
di = 0 and
i=1
n
X
n
X
i=1
di xi
!
di x i = 1
i=1
So the minimum variance estimator which satisfies the above conditions is called the best
linear unbiased estimator ”BLUE). The di s which satisfy this could be find using a similar strategy for maximizing Ta in section 11.1.3 above. After the dust settles, we have:
di =
(xi − x
¯)
Sxx
which seems to have an interesting interpretation. Higher Sxx make the coefficients smaller,
and xi deviating more from x
¯ makes coefficients larger.
What is the variance of β now?
Varb = σ 2
n
X
i=1
46
d2i =
σ2
Sxx
11.2.4 Normal Assumptions
Now, we can assume normality for the variables which let us make further claims regarding
the estimators. The text discusses two ways of doing this, which are practically equivalent.
The more common one is the conditional normal model, which states
Yi ∼ n α + βxi , σ 2
which is a special case of the model discussed above. Even less general is the bivarite normal model, which assumes the pair (Xi , Yi ) follows a bivariate normal distribution. However,
in general we don’t care about the distribution of Xi , but only the conditional distribution of
Yi . So bivariate normal assumptions are not used often.
Also, note both models satisfy the assumptions we have made in the above section.
Maximum Likelihood
Under this distribution assumption, we can try to find the MLE of β. We expect to find the same
formula - and we actually do. The log likelihood function is maximized at the same choice of
β and α.
What about the MLE of σ 2 ? It is given by
n
2
1 X
ˆ i
σ
ˆ =
yi − α
ˆ − βx
n i=1
2
which is the variance of the error ”RSS) - makes sense because RSS are effectively ǫi with
Varǫi = σ 2 ! However, note that σ
ˆ 2 is not an unbiased estimator of σ 2 .
Distributions of Estimators Under Normality Assumption
ˆ and S 2 are
The sampling distributions of the maximum likelihood estimates α
ˆ , β,
with
n
σ2 X 2
α
ˆ ∼ n α,
x
nSxx i=1 i
!
σ2
ˆ
β ∼ n β,
Sxx
−σ 2 x
¯
Cov α
ˆ , βˆ =
Sxx
Furthermore, α
ˆ , βˆ and S 2 are independent and
(n − 2) S 2
∼ χ2n−2
σ2
When σ 2 is unknown, we can still make inferences about them using S 2 since we get Student’s t-distribution using:
47
α
ˆ−α
p Pn
∼ tn−2
S ( i=1 x2i ) / (nSxx )
and
βˆ − β
√
∼ tn−2
S/ Sxx
Significance of the Slope
The t-test for significance of β will reject H0 : β = 0 when
βˆ − 0 > tn−2,α/2
√
S/ Sxx This is equivalent to the following, since t-distribution squared is distributed following the
F -distribution.
βˆ2
> F1,n−2,α
S 2 /Sxx
The LHS quantity, the F -statistic, can be interpreted as follows:
2
Sxy
/Sxx
Regression sum of squares
βˆ2
=
=
2
S /Sxx
RSS/ (n − 2)
Residual sum of squares/df
which is nicely summarized in an ANOVA table.
Partitioning Sum of Squares
As an another similarity to ANOVA, we can express the total sum of squares in the data set by
a sum of regression sum of squares and the residual sum of squares:
n
X
i=1
2
(yi − y) =
n
X
i=1
2
(ˆ
yi − y) +
n
X
i=1
(yi − yˆi )
2
When we split the sum of squares, we can take the ratio between the regression sum of
squares and the total sum of squares as the coefficient of determination, called r2 :
Pn
2
2
Sxy
(ˆ
yi − y)
r2 = Pni=1
=
2
Sxx Syy
ˆi )
i=1 (yi − y
The last portion of this inequality is not very obvious... look at Exercise 11.34 for more
intuition.
48
Prediction at Specified x0
We are able to discuss the distribution of the response variable at a specified position x0 . Call
ˆ 0 which is an
this Y0 . Under our assumptions, E (Y |x0 ) = α + βx0 which is estimated by α
ˆ + βx
unbiased estimator. What is the variance of this estimator?
ˆ 0 = Varˆ
Var α
ˆ + βx
α + Varβˆ x20 + 2x0 Cov α
ˆ , βˆ
n
σ 2 X 2 σ 2 x20
2σ 2 x0 x
¯
xi +
−
nSxx i=1
Sxx
Sxx
!
2
(x0 − x)
1
+
= σ2
n
Sxx
=
Now we have a normal distribution for Y0 . For inference, the following quantity follows a
Student’s t-distribution:
ˆ − (α + βx0 )
α
ˆ + βx
q0
∼ tn−2
2
S n1 + (x0S−x)
xx
which can be used to make confidence interval for EY0 = α + βx0 .
Prediction Interval
ˆ 0 , which is our estimate
The previous estimation and inference was done on the estimator α
ˆ +βx
for the mean of Y0 . Now, can we make intervals for Y0 itself? Obviously, the interval is going
to be larger - we should add variance from the distribution of Y0 as well.
Here’s the definition: a 100 (1 − α) % prediction interval for an unobserved random variable
Y based on the observed data X is a random interval [L (X) , U (X)] such that
Pθ (L (X) ≤ Y ≤ U (X)) ≥ 1 − α
for all θ. The variance of Y0 is given by summing up the variance of the mean estimator and
the common variance σ 2 .
Simulataneous Estimation and Confidence Bands
We can create confidence bands around the fitted line, which gives us confidence intervals
for the mean of Y at that x. This is similar to getting confidence bands in ANOVA, and the same
two processes apply: Bonferroni and Scheffe. Without further details, we state the Scheffe
band:
Under the conditional normal regression model, the probability is at least 1 − α that
ˆ − Mα S
α
ˆ + βx
where Mα =
s
p
2F2,n−2,α .
2
1
(x − x)
ˆ + Mα S
< α + βx < α
ˆ + βx
+
n
Sxx
49
s
1
(x − x)
+
n
Sxx
2
12 Regression Models
The last chapter! Yay! Can’t believe I made it so far. ”Well, yeah, I skipped over a good amount
of material...) Anyways, this chapter covers a number of different models for regression.
12.1
Errors in Variables (EIV) Models
In EIV models, contrary to the traditional regression methods, the xs, as well as ys, are realized
values of a random variable whose mean we cannot observe: EXi = ξi . The means of the two
families of variables are linked by a linear relationship. If EYi = ηi ,
ηi = α + βξi
In this model, there is really no distinction between the predictor variable and the response
variable.
12.1.1 Functional And Structural Relationship
There are two different types of EIV models. The more obvious one is the linear functional
relationship model, where ξi s are fixed, unknown parameters. Adding more
parameterization
2
gives us the linear structural relationship model, where ξi ∼ iid n ξ, σξ . In practice, they
share a lot of properties and the functional model is used more often.
12.1.2 Mathematical Solution: Orthogonal Least Squares
OLS regression measures the vertical distance between each point and the line, since we trust
xs to be correct. In EIV, there is no reason to do that and we switch to orthogonal regression.
Here, the deviation is the distance between the point and the regression line. The line segment
spanning this distance is orthogonal to the regression line, thus the name. The formula for this
in case of a simple regression is given in the book.
Orthogonal least squares line always lies between the two OLS regression lines - y on x and
x on y.
12.1.3 Maximum Likelihood Estimation
The MLE of the functional linear model is discussed. The obvious likelihood function, however,
does not have a finite maximum. ”Setting derivatives to zero results in a saddle point.) To avoid
this problem, we change the model where we do not know the variances of the two errors ”one
in x and one in y), but their ratio λ.
σδ2 = λσǫ2
Note since VarX = σδ2 , this includes the regular regression model when λ = 0 =⇒ VarX = 0.
The maximization can be done analytically. This MLE, when λ = 1, will be the result of the
50
orthogonal least squares. When we send λ → 0, it will become the regular OLS results. Cool
right?
Case for the structural model is discussed, but I’m going to just skip over it.
12.1.4 Confidence Sets
Omitted.
12.2
Logistic Regression And GLM
12.2.1 Generalized Linear Model
A GLM consists of three components: the random component ”response variables), the systematic component ”a function h (x) of predictor variables, linear in the parameter), and the link
function g (µ). Then the model states
g (EYi ) = h (x)
Important points: the response variables are supposed to come from a specified exponential
family.
12.2.2 Logistic Regression
In logistic regression, Yi ∼ Bernoulli (πi ), g is the logit function. Let us limit h to have the form
α + βxi for easier discussion. Then we have
log
πi
1 − πi
= α + βxi
Note that log (π/ (1 − π)) is the natural parameter of the Bernoulli family, since the pmf can
be represented as
y
π (1 − π)
1−y
π
= (1 − π) exp y log
1−π
when the natural parameter is used as the link function, as in this case, it is called the
canonical link. We can rewrite the link equation which gives us better intution about how the
probability and the linear combination is related.
πi =
eα+βxi
1 + eα+βxi
Estimating logistic regression is done by MLE, as we don’t have a clear equivalent of least
squares. This will be done numerically.
51
12.3
Robust Regression
Remind the relationship between mean ”minimizes L2 mean) and the median ”minimizes L1
mean). There is a median-equivalent for least squares; which is called LAD ”least absolute
deviation) regression. It minimizes
n
X
i=1
|yi − (a + bxi )|
This is L1 regression. ”Solvable by LP.) As can be expected, it is quite more robust against
outliers. However, the asymptotic normality analysis gives as the ARE of least squares and
LAD is 4f (0) ”f is the standard normal pdf) which gives us about 64%. So we give up a good
2
bit of efficiency against least squares.
12.3.1 Huber Loss
Analogeous to M-estimator, we can find regression functions that minimizes the Huber loss.
The analysis of this is complicated and is omitted from the book as well. However, it hits a good
middle ground between the two extreme regression techniques. The book demonstrates this
over three datasets; where the errors are generated from normal, logistic, and double exponential distributions. Then, the AREs are calculated between least squares, LAD and M-estimator.
The result is very good. Here I replicate the table:
Error
Normal
Logistic
Double Exponential
vs. least squares
0.98
1.03
1.07
vs. LAD
1.39
1.27
1.14
Almost as good as least squares in normal, completely kick arse in other cases. Very impressive!! Also, note LAD is worse off than least squares in everything. What a piece of crap.
Anyways, I sort of understand why professor Boyd said Huber loss will improve things greatly!
52
Linear Algebra Lecture Notes
jongman@gmail.com
January 19, 2015
This lecture note summarizes my takeaways from taking Gilbert Strang’s Linear Algebra course online.
1 Solving Linear Systems
1.1
Interpreting Linear Systems
Say we have the following linear system:

2
4

1
0
3
5
   
14
1 x1
   
7 x2  = 35
14
1 x3
There are two complementing ways to interpret this.
1.1.1 The Row-wise Interpretation
In the classical row-wise picture, each equation becomes a hyperplane ”or a line) in a hyperspace ”or space).
For example, 2x1 + 4x2 + x3 = 14. The solution is where the three hyperplanes meet.
1.1.2 The Column-wise Interpretation
In the column-wise picture, we think in terms of column vectors. We want to represent the right hand side
as a linear combination of column vectors of A.
1.2
Elimination
Elimination is a series of row operations that will change your given matrix into a upper-triangular matrix.
The allowed operations are as of follows.
• Adding a multiple of a row to another row
• Changing the order of rows
Elimination, combined with back substitution, is how software packages solve systems of linear equations.
1
1.2.1 Row Operations and Column Operations
Say you are multiplying a 1 × 3 row vector with a 3 × 3 matrix. What is the result? It is a linear combination
of rows of the matrix.
h
1
3

i a

2 d
g

b
c
e

f  = 1 × [a, b, c] + 3 × [d, e, f ] + 2 × [g, h, i]
h
i
Similarly, multiplying a matrix with a column vector on its right side gives us a liner combination of
columns of the matrix. We conclude that multiplying on the left will give us row operations; multiplying
on the right gives us column operations.
1.2.2 Representing Elimination With A Series of Matrices
Since elimination is a purely row-wise operation, we can represent it with a series of multiplication operations on the left of the matrix. The matrices that are multiplied to do the elimination are called the
elementary matrices or the permutation matrices, depending on what they are trying to do. Now, the
elimination process of a 3 × 3 matrix A can be represented as:
E23 E12 A = U
Now, keep an eye on E23 E12 : if you multiply these together, it will be a single matrix that will do the entire
elimination by itself!
1.2.3 Side: Multiple Interpretations of Matrix Multiplications
Say we are multiplying two matices A × B = C. Multiple ways to interpret this operation:
• Dot product approach: Cij =
P
Aik Bkj ”all indices are row first)
• Column-wise approach: C i = A × B i . Columns of C are linear combinations of columns in A.
• Row-wise approach: Ci = Ai × B. Rows of C are linear combinations of rows in B.
• Column multiplied by rows: Note that a column vector multiplied by a row vector is a full matrix.
Now, we can think of C as a sum of products between ith column of A and ith row of B!
• Blockwise: If we split up A and B into multiple blocks where the sizes would match, we can do regular
multiplication using those blocks! If A and B were both split into 2 × 2 chunks, each block being a
square. Then, C11 = A11 B11 + A12 B21 !
1.3
Finding Inverses with Gauss-Jordan
Say we want to find an inverse of A. We have the following equation:
A [c1 c2 · · · cn ] = In
2
where ci is the i-th column of A−1 . Now, each column in In are linear combinations of columns of A
- namely, ith column of In is Aci . So each column in In gives us a system of linear equations, that can
be solved by Gauss elimination. The way of solving n linear systems at once is called the Gauss-Jordan
method. We work with an augmented matrix of form [A|In ] and we eliminate A to be In . We can say:
Y
Ej [A|In ] = [In |?]
Say we found a set of Es that make the above equation hold. Then, we got:
Y
Ej A = In ⇐⇒
Y
Ej = A−1 =
Y
Ej I n
the last equality telling us that the right half of the augmented matrix after the elimination is A−1 , thus
proving the validity of the algorithm.
1.3.1 Inverses of Products and Transposes
What is (AB)
−1
? We can easily see B −1 A−1 is the answer because:
B −1 A−1 AB = I = ABB −1 A−1
Now, (AT )−1 ? We can start from
A−1 A = I
and transpose both sides. Now we get
A−1
So A−1
1.4
T
is the inverse of AT .
T
AT = I T = I
Elimination = Factorization; LU Decomposition
Doing a Gaussian elimination to a matrix A will reduce it to a upper triangular matrix U . How are A and
U related? LU decomposition tells us that there is a matrix L that connects the two matrices. ”Also, note
we ignore row exchanges for now)
We take all the elimination matrices used to transform A to E. Since row exchanges are not allowed, all
of these elements are lower triangular. ”We take an upper row, and add an multiple of it to a lower row).
Example: let
"
2
A=
8
#
1
7
We can represent the elimination with a elimination matrix:
"
#"
#
"
1 0 2
1 0
A=
−4 1 8
−4 1
Thus
3
# "
1
2
=
7
0
#
1
3
"
#
1
2
4
7
=
| {z }
A
"
|
1
0
#"
#
1
2
−4 1 0 3
{z } | {z }
U
E −1
We can factor out a diagonal matrix so U will only have ones on the diagonal:
A=
"
#"
1
0
−4
1
2
0
#"
0 1
3
0
1
2
1
#
1.4.1 A case for 3 × 3 matrices
Say n = 3. Then we can represent the elimination is
E32 E31 E21 A = U
Now, the following holds:
−1 −1 −1
A = E21
E31 E32 U = LU
and we call the product of elimination matrices L. Question: how is L always lower triangular? Let’s start
with E32 E31 E21 . Since each elimination matrix subtracts an upper from from lower rows - so everything
is moving downwards . A nonzero number cannot move up .
Now how do we calculate L? We go through an example, and we make the following claim: if you are
not using row exchanges, the multipliers will go into L! That means, the multiplier we used to make U21
zero will go into L21 . This is checked with an example.
1.4.2 Time Complexity for Gaussian elimination
Of course, the naive way gives O n3 . To eliminate using the i-th ”0-based) row, we would have to change
2
2
(n − i) cells. So, n2 + (n − 1) + · · · + 12 . Integrating n2 gives 31 n3 – which is the ballpark range for this sum.
1.5
Permutation Matrices
Say we have a list of all possible permutation matrices for a matrix of size n × n. There are n! possibilities:
we have n! different permutations.
What if we multiply two of those? The result is another permutation, so this set is closed under multiplication.
What if we invert one of those? The result is also a permutation - so this is closed under inversion as
well.
1.5.1 On Transpose and Symmetric Matrices
Trivial result: AT A is always symmetric for any A. Proof?
T
AT A = AT AT T = AT A
4
1.5.2 Permutation and LU matrices
How do we account for row exchanges in LU decomposition? We exchange rows before we start!
P A = LU
2 Vector Spaces and Subspaces
2.1
Definitions and Examples
• Closed under scalar multiplication
• Closed under vector addition
Some things are obvious:
• Closed under linear combination
• Contains 0
2.2
Subspaces
A subspace is a subset of a vector space, which is a vector space itself. For example, R2 has three kinds of
subsets: the plane itself, any line that goes through (0, 0), and {(0, 0)}.
If you take a union of two subspaces, in general, it is not a subspace. However, their intersection is still
a subspace.
2.3
Spaces of a matrix
Given a vector A, the set of all possible linear combinations of its vectors is called the column space of A:
C(A). A column space is inherently related to the solvability of linear systems. Say we want to solve Ax = b.
Any possible value of Ax is in the column space by definition; so Ax = b is solvable iff when b ∈ C(A).
The Null space is defined by the solution space of Ax = 0. Null and column spaces can have different
sizes: if A ∈ Rm×n , N (A) ∈ Rn and C (A) ∈ Rm . How do we find column and null spaces? Once again,
elimination.
2.4
Finding the Null Space: Solving Ax = 0
Say we eliminated A to get a row echelon form matrix U . Here are some definitions.
Pivot columns The columns that contain the pivots used in the elimination.
Free columns The rest of the columns.
Rank of a matrix Number of pivot columns.
5
Why are free columns called free ? In solving Ax = 0, we can assign arbitrary values to the variables
associated with free columns. The rest of the variables will be uniquely defined from those values.
To find the entire null space, we construct a particular solution for each of the free variables. We can
set its value 1, with rest of the free variables 0. Now we can get a particular solution. We repeat for all the
n − r free variables, and take their linear combination. We now know this set spans all possible values
for the free variables.
2.4.1 Reduced-row Echelon Form
Reduced-row echelon form does one more elimination upwards, and make the pivots 1. Making pivots
won’t change the solution since we are solving Ax = 0. Also, note that since every entry ”except the pivot
itself) of the pivot column is eliminated, if we take the pivot rows and columns we get an identity matrix
of size r.
The typical form of an rref can be shown as a block matrix:
R=
"
I
F
0
0
#
where I is the identity matrix, the pivot part, and F is the free part. Note that you can read off the
particular solutions directly off the matrix: now each row of the equation Rx = 0 takes the following form:
xp + axf1 + bxf2 + cxf3 + · · · = 0
where xp is a pivot variable, and xfi s are the free variables. And now getting the value of xp is extremely
easy.
We can abstract even further; think of a null space matrix N such that RN = 0. Each column of this
matrix is the particular solution. And we can set:
N =
"
−F
I
#
From the block multiplication, we know RN = 0 and how each column of N looks like.
2.5
Solving Ax = b
Now we solve a generic linear system. First, some solvability condition: we can solve it if b ∈ C (A).
Finding the solution space is pretty simple:
• Find a particular solution by setting all free vars to 0, and solving for pivot variables.
• Add it to the null space!
Is the solution space going to be a subspace? No, unless it goes through origin.
2.5.1 Rank and Number of Solutions
The key takeaway is that you can predict the number of solutions by only looking at the rank of the matrix.
Say A is a m × n matrix. What is the rank r?
6
• Full rank square matrix: When r = n = m, the rref becomes I and we have exactly one solution for
any b in Ax = b.
" #
I
and we
• Full column rank: This happens in tall matrices, where r = n < m. The rref looks like
0
n o
have no free variables, so N (A) = ~0 . Also, for any b, there might be 0 solutions ”when the zero row
should equal a nonzero bi ) or exactly one solution.
h
i
• Full row rank: This happens in wide matrices, where r = m < n. The rref looks like I F ”in prac-
tice, the columns of I and F are intertwined.). Since we have no zero rows, the number of solutions is
not going to be 0. Also, since there is a free variable, we are always getting a ∞ number of solutions.
#
"
I F
. We get 0 or ∞ solutions depending on b.
• Not-full column/row rank: The rref looks like
0 0
3 Linear Independence, Basis and Dimension
First, a lemma: when A ∈ Rm×n where m < n, there is a nonzero solution to Ax = 0 since we always get a
free variable.
3.1
Linear Independence
Linear independence A set of vectors v1 , v2 , · · · , vn is linearly independent when no linear combination
of them ”except for the 0 combination) result in a 0 vector.
i
h
Equivalently, say A = v1 v2 · · · vn . The column vectors are linearly independent:
• N (A) = {0}
• A is full column rank.
A corollary of the above lemma: 3 vectors in a 2-dimensional space cannot be independent.
3.2
Spanning, Basis, and Dimensions
Definition: A set of vectors {v1 , v2, · · · , vl } span a space iff the space consists of all linear combinations of
those vectors.
A basis for a space is an independent set of vectors {v1 , v2 , · · · , vd } which span the space. The number
of vectors d is called the dimension of the space. Here are some facts:
• A set of vectors in Rn {v1 , · · · , vn } gives a basis iff the n × n matrix with those as columns gives an
invertible matrix.
• Every basis has the same number of vectors, the number being the dimension of the space.
7
3.2.1 Relationship Between Rank and Dimension
The rank of matrix A is the number of pivot columns. At the same time, it is the dimension of the column
space of A - C (A).
OTOH, what is the dimension of N (A)? For each free variable, we get a special solution with that free
variable set to 1 and other free variable to 0. Each of these special solutions are independent, and span the
null space! So the dimension of N (A) = n − r.
3.3
Four Fundamental Spaces of A
Given a A ∈ Rn×m , here are the four fundamental spaces.
• Column space C (A) ∈ Rn
• Null space N (A) ∈ Rm
• Row space R (A) = all linear combinations of rows of A = C AT ∈ Rm
• Null space of AT - often called the left null space of A : N AT ∈ Rn
This is called the left null space because, N AT = y|AT y = 0 = y|y T A = 0 the latter equality
derived from taking the transpose.
We should understand what dimensions and basis do they have; and also how do they relate with each
other. First, the dimension and the basis:
• C (A) has dimension r, and the pivot columns are the basis.
• N (A) has dimension n − r , and the special solutions are the basis.
• R (A) has dimension r, and the first r rows in the rref are the basis.
• N AT has dimension m − r, and the last m − r rows in the elementary matrix E s.t. EA = R. IOW,
the E that comes out of Gauss-Jordan.
Note that summing the dimension of C (A) and N (A) gives n, where summing dimensions of R (A) and
N AT gives m!
3.3.1 Elimination and Spaces
What does taking rref do to the matrix’s spaces? We do know C (A) 6= C (R) since the last row of R can
potentially be zero. However, elimination does not change the row space; and its basis is the first r rows
in R.
3.3.2 Sets of Matrices as Vector Space
Suppose the following set: all sets of 3 × 3 matrices. It is a vector space because it is closed under addition,
multiplication by scalar!
What are some possible subspaces? All upper triangular matrices, all symmetric matrices, all diagonal
matrices, multiples of I, ..
8
We discuss an interesting properties of these different subspaces:
dim (A + B) = dim A + dim B − dim (A ∩ B)
3.3.3 Solutions As Vector Space
What are the solutions of the following differential equation?
∂2y
+y =0
∂x2
We can say: y = a sin x + b cos x. Now, we can think of the solution as a vector space with sin x and cos x
as their basis. Note these are functions, not vectors! So this is a good example why the idea of basis and
vector spaces plays a large role outside the world of vectors and matrices.
3.3.4 Rank One Matrices
Rank one matrices are special, as we can decompose it into a product of column vector and row vector
A = uv T . They are building blocks of other higher-rank matrices. A four-rank matrix can be constructed
by adding four rank-one matrices together.
4 Applications
4.1
Graphs and Networks
The most important model in applied math! A graph with m edges and n nodes can be represented by an
incidence matrix of size m × n, each row representing an edge. The entry Aij = 1 if edge i is coming into
node j, −1 if edge i leaves node j, and 0 otherwise. Note this is different from the adjacency matrix form I
am used to.
Some remarks about how notions in electric network and linear algebra concepts relate to each other.
• Loops: When 2 or more edges form a loop ”not necessarily a cycle), those rows are _not_ independent
and vice versa.
• Sparsity: This is a very sparse matrix! However, in applied linear algebra, it is way more common to
have structured matrices.
• Null space: Say x is a vector of potentials for each nodes - then Ax means differences between potentials of nodes. So Ax = 0 gives you pairs of nodes for which the potentials are the same. If the graph
is connected, the null space is a single dimensional space - c~1.
• N AT : What does AT y = 0 mean? Kirchhoff’s circuit law is like the flow preservation property in
electrical networks; net incoming/outgoing current is zero for any node. If y is a vector of currents,
AT y is a vector of net incoming currents.
– The basis of this null spaces are related to the loops in the graph. Suppose we pick a loop and
send a unit current along it. This gives us a basis.
9
– Repeat, and we can take all loops one by one and take all the basis!
– Say there are two loops: a − b − c − a and a − b − e − a. Is a − c − b − e − a a valid loop? No, it is the
sum ”or difference) of two loops and the special solution will be dependent of the previous two
special solutions.
– Now, realize that the number of the loop is dim N AT = m − r!
• R AT : What do the pivot columns represent? It is a fucking spanning tree! Whoa If there is a cycle
those rows are not going to be independent!
Taking all of this together, we can derive Euler’s formula!
• dim N AT = m − r
• r = n − 1 ”since the pivot columns represent a spanning tree which always have n − 1 edges!)
• Then
Number of loops = Number of edges − (Number of nodes − 1)
Now, in graph theory speak:
F = E − V + 1 ⇐⇒ V − E + F = 1
Ladies and gentlemen, let me introduce Euler’s formula. Holy crap.
Also, we can merge everything in a big equation. So far, we know:
• e = Ax ”potential differences)
• y = Ce ”Ohm’s law)
• AT y = f ”Kirchhoff’s law)
So we get:
AT CAx = f
whatever that means haha.
5 Orthogonality
5.1
Definition
What does it mean for two subspaces/vectors/basis to be orthogonal?
Vector orthogonality: Two vectors x and y are orthogonal iff xT y = 0. We can connect this to Pythagorean
2
2
2
theorem - kxk + kyk = kx + yk iff x and y are orthogonal.
2
2
2
kxk + kyk = xT x + y T y = kx + yk = xT x + y T y + 2xT y ⇐⇒ xT y = 0
10
Subspace orthogonality: two subspace S is orthogonal to subspace T when every vector in S is orthogonal to every vector in T . Examples:
• Are xz and yz planes orthogonal in R3 ? No they aren’t: they have a nonzero intersection! The vectors
in that intersection cannot be orthogonal to themselves.
Facts:
• Row space orthogonal to null space. Why? x ∈ N (A) iff
  
0
r1 · x
 .  .
.  .
Ax = 
 .  = .
0
rn · x

So x is orthogonal to all rows. And of course, it will be orthogonal to all linear combinations of the
rows.
• Row space and null spaces are orthogonal complements of R3 : nullspace contains all vectors orthogonal to the row space.
5.2
Projections
5.2.1 Why Project?
If Ax = b cannot be solved in general. So what do we do? We find the closest vector in C (A) that is closest
to b, which is a projection!
5.2.2 2D Case
Suppose we project vector b onto a subspace that is multiple of a. Say that the projected point is ax. We
know the vector from ax to b is orthogonal to a. We have:
0 = aT (ax − b) = aT ax − aT b ⇐⇒ x =
aT b
aT a
Look at the projected point ax:
ax = a
aT b
=
aT a
aaT
aT a
b
Note the last formula - that rank one matrix is the projection matrix P ! It has the following properties:
P 2 = P and P T = P
5.2.3 General Case
It’s the same derivation! If p is the projected point, we can write it as Aˆ
x. Then the error vector b − Aˆ
x is
perpendicular to the column space of A. So:
−1 T
0 = AT (b − Aˆ
x) = AT b − AT Aˆ
x ⇐⇒ x
ˆ = AT A
A b
11
Welcome pseudoinverse! Now, get p by
−1 T
p = Aˆ
x = A AT A
A b
{z
}
|
P
We got the projection matrix!
5.3
Least Squares
When A is tall, Ax = b is not generally solvable exactly. We multiply AT to both sides of the equation to get
AT Ax = AT b
Hoping AT A is invertible and we can solve this exactly. When is AT A invertible? It turns out rank of
AT A is equal to rank of A - so AT A is invertible only when A is full column rank.
5.3.1 Invertibility ofAT A
Assume A is full column rank. Let’s prove the following set of equivalent statements:
AT Ax = 0 ⇐⇒ x = 0 ⇐⇒ N AT A = {0} ⇐⇒ AT A is invertible
Take the first equation, and calculate dot products of each side with x:
T
xT AT Ax = 0 = (Ax) Ax ⇐⇒ Ax = 0
Since A is full column rank, N (A) = {0}. So x must be 0.
5.3.2 Least Squares as A Decomposition
If least square decomposes a vector b into p + e, p ∈ C (A) and e ∈ N (A). Now, if p = P b, what is e? Of course,
e = (I − P ) b. We get p + e = (P + I − P ) b = b.
5.4
Orthonormal Basis
A set of orthonormal vectors {q1 , q2, · · · } is a set of unit vectors where every pair is perpendicular. We can
write this with a matrix Q:
Q = [q1 q2, · · · ]
where qi s are column vectors. Now, the above requirement can be written as:
QT Q = I
This matrix Q is called a orthonormal matrix. ”Historically, we only call it orthogonal when it’s a
square...) What happens when Q is square? Since QT Q = I, Q = Q−1 .
12
5.4.1 Rationale
The projection matrix can be found as
P = Q QT Q
−1
QT = QQT
5.4.2 Gram-Schmidt
Given a set of vectors, how can we make them orthonormal? Well... I sorta do know... tedious process to
do by hand.
• Take an arbitrary vector, and normalize it. Include it in the result set.
• For each other vector v,
– For each vector u in the result set, subtract the projection of v onto u from v: v = v − uT v · u
– Normalize the resulting v and include it in the result set.
5.4.3 QR Decomposition
How do we write Gram-Schmidt in terms of matrices? We could write Gaussian elimination by
P A = LU
We write this as:
A = QR
Note that, because of how Gram-Schmidt works, R is going to be a lower triangular matrix! The first
column of Q is going to be the first column of A, scaled! Also, the second column is going to be a linear
combination of first two columns of A and etc.
6 Determinants
6.1
Properties
1. Identity matrix has determinant 1.
2. When you swap two rows, the sign of the determinant will change.
3. The determinant is a linear function of the matrix: ”I’m not saying det A + det B = det (A + B) – this
is not true)
”a) If you multiply a row by a scalar t, the determinant will be multiplied by t.
h
h
iT
h
iT
+ det a · · · y
= det a · · · x · · · b
”b) det a · · · x + y · · · b
The following can be derived from the above points:
13
···
b
iT
• If two rows are equal, determinant is 0.
• If we subtract a multiple of a row from another row, determinant doesn’t change. ”This can be proved
from 3b and 4)
• If there’s a row of zeros, determinant is 0.
• For a upper triangular matrix, determinant is the product of the diagonal products. ”Proof: do reverse
elimination to get a diagonal matrix with same determinant, use rule 3a repeatedly until we get I)
• det A ⇐⇒ A is singular
• Determinant is multiplicative: det AB = det A · det B
– So: det A−1 =
1
det A
• det A = det AT ”this means, swapping columns will flip the sign)
– Proof: let A = LU . det A = det (LU ) = det L det U . det L = 1, and det U = det U T since it is upper
triangular.
6.2
Calculation
6.2.1 Big Formula
X
det A =
p∈permute(1···n)
±
Y
aipi
Argh.. messy. Whatever.
6.2.2 Cofactor
Taking the big formula, and collecting terms by the number in the first row gives the cofactor expansion
formula. Say the matrix is 3 × 3:
det A =
a11 (a22 a33 − a23 a32 )
+
a12 (−a21 a33 + a23 a31 )
+
a13 (a21 a32 − a23 a31 )
Notice the quantities in parenthesis are either determinants of the 2 × 2 matrices, or their negatives.
Formally, Cij is defined to be a cofactor
Cij = ± det (smaller matrix with row iand col jremoved)
where the sign is + when i + j is even, - when i + j is even. This follows the checkerboard pattern.
”Formally, cofactors with the sign are called minors.) The resulting cofactor formula for the determinant
is:
det A = a11 C11 + a12 C12 + · · · + a1n C1n
14
6.3
Applications
6.3.1 Formula for A−1
A−1 =
1
CT
det A
where C is the matrix of cofactors in A. How do we verify it? Let’s check if
AC T = (det A) I
Expand the elements of the matrix - we actually see the diagonals are the determinants, from the cofactor formula!
But what about the off-diagonal elements? We claim those calculations are actually using cofactor
formula for a matrix with two equal rows! Aaaaah.....
6.3.2
Ax = b and Cramer’s Rule
Of course, we know
x=
1
CT b
det A
What are the entries of x in this formula? Since ci b is always a determinant of some matrix, we can
write it as:
xi =
det Bi
det A
for some matrix Bi . Cramer realized that Bi is A with column 1 replaced by b. This are beautiful, but
not practical ways of calculating stuff.
6.3.3 Determinant and Volume
The determinant of a n × n matrix A is the volume of a box in a n dimensional space with n sides of it
coinciding with the column/row vectors of A. Let’s look at some example cases to get ourselves convinced:
• A = I: it’s going to be a ”hyper)cube with volume 1.
• A = Q ”orthonormal): another cube, rotated. From here we can rediscover QT Q = I - take determi2
nants from both sides: (det Q) = 1. So det Q = ±1.
Also, we revisited many determinant properties and made sure they hold in this context as well.
7 Eigenvalues and Eigenvectors
7.1
Definition
You can think of a matrix as a linear mapping, using f (x) = Ax. For a matrix, we can find vector”s) x such
that f (x) is parallel to x. Formally:
15
Ax = λx
where λ is a scalar, called the eigenvalue. The xs are called the eigenvectors.
7.1.1 Example: Projection Matrix
Say there is a projection matrix, which projects stuff into a plane: when a vector is already in the plane,
it won’t change. So they will be eigenvector with eigenvalue 1. We can say there are two perpendicular
eigenvectors that span the plane. Are there any other eigenvalue? ”Intuitively, we expect to find one, since
we are in a 3D space) Yes, find the normal vector that goes through the origin. This will become a 0 vector
- so this one is an eigenvector with eigenvalue 0.
7.1.2 Example 2: Permutation Matrix
Let
A=
"
0
1
#
1
0
Trivially, [1, 1] is an eigenvector with eigenvalue 1. Also, [−1, 1] is an eigenvector with eigenvalue -1.
T
T
7.1.3 Properties
• n × n matrices will have n eigenvalues.
• Sum of the eigenvalues equal the trace of the matrix ”sum of the diagonal entries).
• The determinant is the product of the eigenvalues.
7.2
Calculation
Rewrite the equation as:
(A − λI) x = 0
Now, for this to be true for nontrivial x, A − λI has to be singular: this is equvalent to:
det (A − λI) = 0
We solve this to find λs. After that, we can use elimination to find x.
7.2.1 Properties
If we add aI to a matrix, each of its eigenvalues will increase by a. The eigenvectors will stay the same!
See: if Ax = λx,
(A + aI) x = λx + ax = (λ + a) x
16
What do we know about general addition? If we know eigenvalues and eigenvectors of A and B, what
do we know about A + B? First guess would be their eigenvalues be added, but false. Because they can
have different eigenvectors.
7.2.2 When Things Are Not So Rosy
Think of a rotation matrix; what vector is parallel to itself after rotating 90 degrees? None! When we
carry out the calculation, we get complex eigenvalues. Also, for some matrices we can have duplicate
eigenvalues, and no independent eigenvectors. Why is this important? We will see.
7.3
Diagonalization of A Matrix
Let us assume that A has n linearly independent eigenvectors, x1 to xn , each associated with eigenvalues
λ1 to λn . We can put them in columns of a n × n matrix, and call this S. Then, we can write what we know
as follows:
h
AS = A x1
x2
i h
x n = λ1 x 1
···
λ2 x 2
···
λn x n
i
How do we write the latter with a matrix representation? We can multiply S with a diagonal matrix
with eigenvalues along the diagonal:
h
λ1 x 1
λ2 x 2
···

···
λ1
0

i
0
λn x n = S 
 ..
.
λ2
..
.
···
..
.
0
..
.
0
···
λn
0
We call the diagonal matrix Λ. Now we have:
0



 = SΛ


AS = SΛ ⇐⇒ S −1 AS = Λ ⇐⇒ A = SΛS −1
Note since we assumed the eigenvectors are linearly independent, S is invertible.
From this, we can infer interesting properties of eigenvalues. What are the eigenvectors and eigenvalues of A2 ? Intuitively, of course, the eigenvectors are the same - for an eigenvector x, Ax is a scalar
multiplication of x. So A2 x will still be a scalar multiplication, with a factor of λ2 . However, we can see it
from diagonalization as well:
A2 = SΛS −1
2
= SΛS −1 SΛS −1 = SΛ2 S −1
In general, if you take powers of a matrix, it will have the same eigenvectors, but the eigenvalues will
get powered.
7.3.1 Understanding Powers of Matrix Via Eigenvalues
What does
lim An
n→∞
look like? Does it ever go to 0? We can find out, by looking at its eigenvalues:
17
An = SΛn S −1
For An to go to 0, Λn has to go to zero - so the absolute value of all eigenvalues are less than 1!
7.3.2 Understanding Diagonalizability
A is sure to have n independent eigenvectors ”and thus be diagonalizable) if all the eigenvalues are different. Note there are cases where there are repeated eigenvalues and there are independent eigenvectors
”well, take I for an example).
7.4
Applications
7.4.1 Solving Recurrences
Let’s solve a difference equation:
uk+1 = Auk
with a given u0 . How to find uk = Ak u0 , without actually powering the matrix? The idea is to rewrite u0
as a linear combination of normalized eigenvectors of A:
u0 = c 1 x 1 + c 2 x2 + · · · + c n x n
Now, Au0 is:
Au0 = Ac1 x1 + Ac2 x2 + · · · + Acn xn = c1 λ1 x1 + c2 λ2 x2 + · · ·
Yep, we got a pattern! Now we know:
Ak u0 =
X
ci λki xi
i
The idea of eigenvalues kind of sank in after looking at this example; they, sort of, decompose the linear
mapping represented by A into orthogonal basis. After you represent a random vector in this space, the
effects of A can be isolated in each direction. So it actually describes A pretty well! The name eigen now
kind of makes sense.
7.4.2 Deriving Closed Form Solution of Fibonacci Sequence
Let f0 = 0, f1 = 1, fn = fn−1 + fn−2 . I know that I can write
un =
"
fn
fn−1
#
and un+1
"
1
=
1
#
1
un = Aun
0
What are the eigenvalues of A? The characteristic polynomial comes out to be
(1 − λ) · (−λ) − 1 = λ2 − λ − 1 = 0
18
Plugging this into the quadratic formula, we get:
√
√
√
1± 5
1+ 5
1− 5
λ=
: λ1 =
, λ2 =
2
2
2
Since λ1 > 1, λ2 < 1, when n grows, λ1 will dominate the growth of the fibonacci number. So we know:
√ !100
1+ 5
2
f100 ≈ c ·
Why? Rewrite u0 as
u0 = c 1 x1 + c 2 x2
and we know
100
u100 = c1 λ100
1 x 1 + c 2 λ2 x 2
and with λ2 < 1, the second term is meaningless when k is large. Whoa whoa.... nice...
For the sake of completeness, let’s calculate the eigenvectors as well. The eigenvectors are:
x1 =
"
λ1
1
#
, x2 =
"
λ2
1
#
Solve for c1 and c2 by equating:
u0 =
" #
0
1
= c 1 x1 + c 2 x2 =
"
√ #
c1 +c2 +(c1 −c2 ) 5
2
c1 + c2
Now we have the formula for uk - thus a closed form solution for fk .
7.4.3 Differential Equations
Consider the following system of differential equations:
du1
= −u1 + 2u2
dt
du2
= u1 − 2u2
dt
with initial value u1 (0) = 1, u2 (0) = 0. Let u (t) = [u1 (t) , u2 (t)] and we can write above as:
T
du
= Au
dt
with
A=
"
−1
1
2
#
−2
The eigenvalues of A are 0 and -3, and the associated eigenvectors are x1 = [2, 1] and x2 = [1, −1] ,
T
respectively.
19
T
Now, the solution is a linear combination of two special solutions, each corresponding with an eigenvalue:
u (t) = c1 eλ1 t x1 + c2 eλ2 t x2
which we can check by plugging into the above representation. We can solve for c1 and c2 using the initial
condition - and we now know everything. So eigenvalues still give us insights about which parts of the
solution blows up, or goes to 0, etc. Of course, since eigenvalues now sit on the exponent, it has to be
negative to go to 0. If the eigenvalue is 0, the corresponding portion will stay constant.
Note: the eigenvalues might be imaginary - in that case, only the real portion counts in terms of asymptotic behavior. For example,
e(−3+6i)t = e−3t e6it = e−3t (cos t + i sin t)
6
and the latter part’s magnitude is 1.
7.4.4 Thinking In Terms of S and Λ
In
du
= Au
dt
A is a non-diagonal matrix, and represents the interaction, relation between the variables. This is coupling; we can decouple variables by using eigenvalues. Now, decompose u into a linear combination of
eigenvectors by setting u = Sv. We get:
du
dv
dv
=S
= ASv ⇐⇒
= S −1 ASv = Λv
dt
dt
dt
Wow, we now have a set of equations like: v1′ = λ1 v1 , v2′ = λ2 v2 , and so on. Same as in the difference
equation example. Now, how do we express the solution in this? Since
dv
= Λv
dt
we would like to express v as
v (t) = eΛt v (0)
which gives
u (t) = SeΛt S −1 u (0)
which gives the decoupling effect we are looking for. But WTF is that eΛt ? First let’s define eAt as:
eAt = I + At +
∞
2
n
X
(At)
(At)
+ ···+ =
2
n!
n=0
Just like the power series definition for eax = 1 + ax +
20
(ax)2
2!
+
(ax)3
3!
+ · · · . Now what is SeΛt S −1 ?
e
At
=e
SΛS −1
So it’s same as eAt !
7.5
X SΛS −1
=
n!
n
tn
=
X SΛn S −1 tn
n!
=S
X
Λ n tn
n!
S −1 = SeΛt S −1
Applications: Markov Matrices and Fourier Series
7.5.1 Markov Matrix
What is a Markov Matrix? Nonnegative square matrix, with all columns summing up to 1. Now, we know
those processes are never going to blow up. Maybe they will reach a steady state. We already know:
• Any eval cannot be greater than 1, because it will make things blow up.
• If an eval equals 1, the evec for that eval will be the steady state.
We will find out that Markov matrices always have 1 as an evalue. Also, it will never be a repeated eigenvalue.
7.5.2 Proof of 1 Being An Eigenvector
Since A has 1 as column sums, A − 1I has zero sum columns. Now, the rows are dependent: add up
all rows and they sum to 0. This leads us to the corollary that A and AT has same eigenvalues ”because
T
det (A − λI) = det (A − λI)
= det AT − λI )
7.5.3 Projections with Orthonormal Basis
If we have a set of orthonormal basis, arbitrary vector v can be represented as:
v = Qx
Since Q is a projection matrix, Q−1 = QT . So we can solve for x as: x = QT v
Nothing Fourier related so far. What now?
7.5.4 Fourier Series
We write a given function f as a linear combination of sin and cos:
f (x) = a0 + a1 cos x + b1 sin x + a2 cos 2x + b2 sin 2x + · · ·
This infinite series is called the Fourier series. We now work in function space; instead of orthogonal
vectors we use orthogonal functions: 1, cos x, sin x, cos 2x, and so on. We represent a function with a linear
combination of those basis functions.
But what does it mean that two functions are orthogonal? How is dot product defined between two
functions? We define:
f T g (x) =
ˆ
f (x) g (x) dx
21
We can calculate this between constants, sines and cosines because they are all periodic.
Now, a0 is easy to determine - take the average value. What is a1 ?
ˆ
2π
f (x) cos xdx = a1
0
ˆ
0
2π
1
cos xdx =⇒ a1 =
π
2
ˆ
2π
f (x) cos xdx
0
where the latter equality comes from the fact the basis functions are orthogonal. ”Btw, I’m not sure how
we can fix the bounds on the above integral. Maybe I should go back to the book.)
7.6
Symmetric Matrices and Positive Definiteness
When A ∈ Sn where Sn is the set of n × n symmetric matrices, we state:
• All eigenvalues are real
• We can choose eigenvectors such that they are all orthogonal.
The exact proof is left to the book.
The usual diagonalization, A = SΛS −1 now becomes A = QΛQ−1 = QΛQT , the latter equality coming
from the fact that Q has orthonormal columns, so QQT = I. This is called the spectral theorem; spectrum
means the eigenvalues of the matrix.
7.6.1 Proof
Why real eigenvalues? Let’s say
Ax = λx
We can take conjugates on both sides;
¯x
¯x = λ¯
A¯
However, since we assume A to be real, we know:
¯x
¯x = A¯
A¯
x = λ¯
Try using symmetry, by taking transposing stuff:
¯
x
¯T AT = xT A = x
¯T λ
The second equality coming from symmetry assumption. Now, multiply x
¯T to the both sides of the first
equation:
x
¯T Ax = λ¯
xT x
And we multiply x to the right side of the last equality:
¯ Tx
x
¯T Ax = λx
Then
22
λ¯
xT x = x
¯T Ax = λxT x
Now, we know:
¯
λ=λ
thus λ is real - if x
¯T x is nonzero.
x
¯T x =
X
i
”unless x is 0. But if x is 0, λ = 0)
(a + ib) (a − ib) =
X
a 2 + b2 > 0
i
7.6.2 When A is Complex
¯ The last equality can work if:
We repeat the above argument, without assuming A = A.
A=A
T
7.6.3 Rank One Decomposition
Recall, if A is symmetric, we can write:


A = QΛQT = [q1 q2 · · · ] 

λ1
λ2

q1T
 T
 q2  = λ1 q1 q1T + λ2 q2 q2T + · · ·
 
..
..
.
.

So every symmetric matrix can be decomposed as a linear combination of perpendicular projection
”rank one) matrix!
7.6.4 Pivots And Eigenvalues
Number of positive/negative eigenvalues for symmetric matrices can be determined from the signs of the
pivots. The number of positive eigenvalues is the same as the number of positive pivots.
7.6.5 Positive Definiteness
A PSD matrix is a symmetric matrix. If symmetric matrices are good matrices, PSD are excellent . It is
a symmetric matrix with all eigenvalues are positive. Of course, all the pivots are positive.
So, for 2 × 2 matrices, PSD matrices always have positive determinants and positive trace. However,
this is not a sufficient condition for positive definiteness, as can be demonstrated in the following matrix:
"
#
−1 0
0 −2
We state that a matrix is positive definite iff all its subdeterminants are positive; they determinants of
submatrices formed by taking a m × m submatrix from the left top corner.
23
To summarize:
• All eigenvalues are positive
• All pivots are positive
• All subdeterminants are positive
7.7
Complex Numbers and Examples
Introducing complex numbers and FFT.
7.7.1 Redefining Inner Products for Complex Vectors
If z ∈ Cn , z T z is not going to give me the length of the vector squared, as in real space. Because z T z =
P
P 2
2
2
j (aj + ibj ) =
j aj − bj 6= kzk. As seen in the proof of real eigenvalues of symmetric vectors, we need:
z¯T z.
For simplicity, we write:
z¯T z = z H z =
X
2
kzi k = kzk
2
where H stands for Hermitian. So, from now on, let’s use Hermitian instead of usual inner product.
7.7.2 Redefining Symmetric Matrices
We also claim our notion of symmetric matrix A = AT is no good for symmetrix matrix. We want A¯T = A
2
for obvious reasons ”so the diagonal elements of A¯T A are kai k ).
We define a Hermitian matrix A to satisfy:
AH , A¯T = A
7.7.3 Orthonormal Basis
Now, for a matrix Q with orthonormal columns, we say:
QH Q = I
Also we call these matrices unitary.
7.7.4 Fourier Matrices

1

1


Fn =  1
.
.
.
1
w
1
w2
w2
..
.
w4
..
.
1 wn−1
24
w2(n−1)

···

· · ·

· · ·



· · ·
···
to generalize, we have (Fn )ij = wij . Note both indices are 0-based. Also, we want wn = 1 - nth primitive
root of unity. So we use
w = cos
2π
2π
+ sin
= ei(2π/n)
n
n
”Of course I know we can use modular arithmetic to find a different primitive root of unity.. but meh.)
One remarkable thing about Fourier matrices is that their columns are orthonormal. Then the following
is true:
T
FnH Fn = I ⇐⇒ Fn−1 = F¯n
which makes it easy to invert Fourier transforms!
7.7.5 Idea Behind FFT
Of course, FFT is a divide-and-conquer algorithm. In the lecture, larger order Fourier matrices are connected to a smaller order matrice by noticing:
Fn =
"
I
D
I
−D
#"
Fn/2
0
0
Fn/2
#
P
for which P is a odd-even permutation matrix. Exploiting the structures of these matrices, this multiplication can be done in linear time.
7.8
Positive Definiteness and Quadratic Forms
7.8.1 Tests of Positive Definiteness
These are equivalence conditions of positive definiteness for a symmetric matrix A:
1. All eigenvalues are positive.
2. All subdeterminants are positive.
3. All pivots are positive.
4. xT Ax > 0 for all x.
”Also, if all the positiveness in above definition is swapped by nonnegativeness, we get a positive semidefinite matrix.)
Now what’s special with the new 4th property? Actually, property 4 is the definition of positive definiteness in most texts; property 1-3 are actually just the tests for it. What does the product xT Ax mean? If
we do it by hand we get:
xT Ax =
X
Aij xi xj
i,j
This is called the quadratic form. Now, the question is: is this positive for all x or not?
25
7.8.2 Graphs of Quadratic Forms
Say x ∈ R2 and let z = xT Ax; what’s the graph of this function? If x is not positive ”semi)definite, we get a
saddle point. A saddle point is a maximum in one direction whilest being a minimum in another. ”Actually
these directions are eigenvalue directions.)
What happens when we have a positive definite matrix? We know z will be 0 at the origin, so this must
be the global minimum. Therefore, we want the first derivative to be all 0. However, this is not enough to
ensure minimum point.
We want to refactor quadratic form as a sum of squares form ” the completing the squares trick). So
we can ensure that xT Ax is positive everywhere except 0, given a particular example. But how do we do
it in a general case? The course reveals that, actually, Gaussian elimination is equivalent to completing
the squares! Holy crap... Positive coefficients on squares mean positive pivots, which means positive
definiteness!
And if we try to recall Calculus we were presented with a magic formula called the second derivative
test - which was just checking if second derivative matrix was positive definite! Niiicee.
7.8.3 Geometry of Positive Definiteness; Ellipsoids
If a matrix is positive definite, we know xT Ax > 0 for except x = 0. If we set xT Ax = c for a constant c, this
is an equation of a ellipse - or an ellipsoid.
The major/middle/minor axis of this ellipsoid will be determined by the eigenvectors of A, their lengths
being determined by the eigenvalues.
7.8.4
AT A is Positive Definite!
Yep, AT A is always positive definite regardless of A. Because covariances.. but here’s a proof:
xT AT Ax = (Ax) (Ax) = kAxk > 0 unless x = 0
2
T
7.9
Similar Matrices
7.9.1 Definition
Two n × n matrices A and B are similar, if for some invertible M , A = M BM −1 . Example: A is similar to
Λ because Λ = S −1 AS. Why are they similar? They have the same eigenvalues. How do we prove it? Let
Ax = λx. Then Ax = M BM −1 x = λx ⇐⇒ BM −1 x = λM −1 x. Now, M −1 x is an eigenvector of B with the
same eigenvalue.
So this measure divides matrices into groups with identical set of eigenvalues. The most preferable
of them, obviously, are diagonal matrices. They have trivial eigenvectors.
7.9.2 Repeated Eigenvalues And Jordan Form
What if two or more eigenvalues are the same? Then the matrix might not be diagonalizable - it might
not have a full set of independent eigenvectors. What happens now? Using above definition of similar
matrices, there can be two families of matrices with same set of eigenvalues, but not similar to each
26
other. One is the diagonal matrix, in its own family. The other family contains all the other matrices with
given eigenvalues.
In the latter family, the matrix that is as close as possible to the diagonal matrix, is called the Jordan
Form for the family of matrices.
7.9.3 Updated Definition
Looking at some examples reveals that similar matrices actually have the same number of eigenvalues as
well. So when eigenvalues are repeated, diagonal matrices have n eigenvectors - other matrices don’t.
7.9.4 Jordan Theorem
Every square matrix A is similar to a Jordan matrix J which is composed of Jordan blocks in the main
diagonal. For an eigenvalue λi repeated r times, the Jordan block is a r × r matrix with λi on the diagonals,
and 1s on the above diagonal with rest of the entries 0.
7.10 Singular Value Decomposition
7.10.1 Introduction
We can factorize symmetric matrices as A = QΛQT . We can also factor a general matrix A as A = SΛS −1 but
this can’t be done in some cases, when S is not invertible. SVD is a generalized diagonalization approach
which defines a decomposition
A = U ΣV T
for any matrix A with a diagonal Σ and orthonormal U and V . In other words, we can say:
A = U ΣV T = U ΣV −1 ⇐⇒ AV = U Σ
where V is a orthnormal basis for the row space, U is an orthonormal basis for the column space. Hmm.
Do the dimensions even match?
7.10.2 Calculation
How can we find V ? Let’s think of AT A.
AT A = V ΣT U T U ΣV T = V Σ2 V T
Yes! We can factorize AT A ”which is posdef) to find its eigenvalues and orthnormal eigenvectors. The
eigenvectors will be V , and we can construct Σ by taking the square root of eigenvalues of AT A ”which is
possible, since AT A is posdef).
Similarly, looking at AAT will let us find the U s.
Of course, our assertion ”that U and V are basis of column and row spaces, respectively, and why the
eigenvalues of AT A and AAT are the same) needs to be proven. ”In the lecture, the latter fact is stated as
obvious.. where did I miss it?)
27
8 Linear Transformation
8.1
Linear Transformation and Matrices
A linear transformation is a function T with the following properties:
• T (u + v) = T (u) + T (v)
• T (cu) = cT (u)
Of course, all linear transformations have an unique matrix representation, and vice versa. How can we
find a matrix given a linear transform?
Let’s say we have a linear transform T : Rn → Rm . We have a basis for the input space v1 , · · · , vn and
a basis for the output space w1 , · · · , wm . How do we find the matrix for this transformation? Transform
the first basis v1 to get: T (v1 ) = c1 w1 + c2 w2 + · · · + cm wm . And those coefficients c1 · · · cm take up the first
column of the matrix.
8.2
Change of Basis
Image compression: say we are compressing a grayscale image. Using standard basis to represent this
image does not exploit the fact that neighboring pixels tend to have similar luminousity value. Are there
other bases that can give us good sparse approximations?
JPEG uses the Fourier basis. It divides images into 8 × 8 blocks and changes the basis to Fourier basis,
and we can threshold the coefficients. Hmm, actually, I want to try this out. Also, wavelets are a popular
choice recently.
8.2.1 Same Linear Transformation from Different Bases
Say we have a linear transformation T . If we put vectors in coordinate system defined by a set of basis
v1 · · · vn , we will get a matrix for the transformation. Let’s call it A. If we use another set of basis w1 · · · wn ,
we get another matrix which we call B.
A and B are similar as in 7.9!
8.2.2 Eigenvector basis
Say we have an eigenvector basis: T (vi ) = λi vi . Then, the change of basis matrix is diagonal!
8.3
Left, right, pseudoinverse
Let’s generalize the idea of an inverse. A 2-sided inverse is the usual inverse:
AA−1 = I = A−1 A
A matrix A has a 2-sided inverse when it’s square, and has full rank: n = m = r.
28
8.3.1 Left and Right Inverses
Suppose a full column-rank matrix n = r. The null space is just {0}, and for any Ax = b there is either 0 or
1 solution. Since A is full column rank, AT A is full rank and invertible. Note that we have a left inverse
−1 T
here: AT A
A . If we multiply this inverse and A we get:
−1 T
−1 T AT A
A A = AT A
A A =I
Resulting I is a n × n matrix.
Similarly, suppose a full row-rank matrix ”m = r). The null space has dimension n − r, so for every
−1
Ax = b there are infinitely many solutions. Now we have a right inverse: AT AAT
.
8.3.2 Left Inverse and Project
What do we get if we multiply the left inverse on the right?
−1 T
A AT A
A
This is a projection matrix! Similarly, if we multiply the right inverse on the left, we will get a projection
matrix that projects onto its row space.
8.3.3 Pseudoinverse
What is the closest thing to an inverse when A is neither full row or column rank? Think of this: take any
vector in the row space, and transform it by A, you’ll get something from a column space. ”In this regard,
A is a projection onto the column space.) The two spaces ”row and column) have the same dimension, so
they should have a 1:1 ”or invertible) relationship - so the matrix that undoes this transformation is called
the pseudoinverse, A+ .
8.3.4 Pseudoinverse from SVD
How do we find a pseudoinverse? Maybe we can start from SVD:
A = U ΣV −1
where Σ is a ”sort-of) diagonal matrix sized m × n with singular values on the diagonal. What is its
pseudoinverse? Well that’s easy - we can get a n×m matrix and put inverse singular values on the diagonal.
Then what happens in ΣΣ+ ? We get a m × m matrix with first r diagonal entries 1. Σ+ Σ works similarly.
How do we go from Σ+ to A+ ? Because U and V are orthonormal, we can just use:
A+ = V Σ+ U T
29
Singular Vector Decomposition
January 19, 2015
Abstract
This is my writeup for Strang’s Introduction to Linear Algebra, Chapter 6, and the accompanying video
lectures. I have never had an intuitive understanding about SVDs... now I sorta do. Let’s go over the basics,
derivation, and intuition.
1 Eigenvalues and Eigenvectors
1.1
Introduction
Let A be a n × n matrix. Then x ∈ Rn is an eigenvector of A iff:
Ax = λx
”1)
for some number λ. In laymen’s terms, x does not change its direction when multiplied by - it merely
scales by λ. λ is called the eigenvalue associated with x. Of course, any nonzero multiple of x is going to
be a eigenvector associated λ; it is customary we take unit vectors.
1.1.1 Calculation
How do we find eigenvalues and eigenvectors given A? Rewrite ”1) as:
Ax = λIx ⇐⇒ (A − λI) x = 0
In other words, x must be in null space of A − λI. In order to have nontrivial x, A − λI must be singular.
That’s how we find eigenvalues - solve:
det |A − λI| = 0
This boils down to solving a nth order polynomial. The roots can be anything - they can be repeated,
negative, or even complex.
1.1.2 Properties
These nice properties hold:
• If A is singular, 0 is an eigenvalue - of course, because det A = det |A − 0I| = 0.
1
• Determinant of A equals the product of all eigenvalues.
• Trace of A equals the sum of all eigenvalues.
1.1.3 Eigenvectors Are In C (A)
Something that was not immediately obvious was that each x was in the column space C (A). x comes from
null space of A − λI, which doesn’t seem to be related to A - so it sounded surprising to me first. However,
recall the definition: Ax = λx. Therefore, λx ∈ C (A) and so is x.
1.2
Diagonalization
Everything we want to do with eigenvectors work fine when we have n linearly independent eigenvectors.
For now, let us assume we can choose such eigenvectors. Then, we can perform something that is very
nice called diagonalization. Let x1 · · · xn be n eigenvectors, each associated with eigenvalues λ1 · · · λn . It is
defined:
Axi = λi xi for i = 1 · · · n
We can write this in matrix form. Let S be a matrix with xi as column vectors. This is a n × n matrix, and
the above LHS can be written as:
h
A x1
x2
···
x3
i
xn = AS
Let Λ be a diagonal matrix where Λii = λi , then RHS can be written as:
h
λ1 x 1
λ2 x 2
λ3 x 3
···
i
λn xn = SΛ
Joining the two gives
AS = SΛ
S is invertible since its columns are independent. Then we can write:
A = SΛS −1
which is called the diagonalization of A.
1.2.1 Matrices That Are Not Diagonalizable
When is A not diagonalizable? Say we have a repeated eigenvalue, λd which is repeated r times. A − λd
will be singular, but its rank might be greater than n − r. In such case, rank of the null space of the matrix
is less than r, and we won’t be able to choose r vectors that are independent.
1.3
Intuition
A n × n matrix can be seen as a function ”actually it’s a linear transformation): it takes a vector, and returns
a vector. Eigenvectors can be regarded as axis of this function, where those on this axis are not changed
2
”at least in terms of directions). If we use eigenvectors as axis, we can represent any vector in C (A) in this
coordinate system, by representing it as a linear combination of eigenvectors. After we do this change of
coordinates, the transformation is done by multiplying each coordinate with its eigenvalues individually.
There are no interactions between each axis anymore.
What happens when x ∈
/ C (A)? Of course, even if x ∈
/ C (A), Ax ∈ C (A). Say x = a + b where a ∈ C (A)
and b ∈
/ C (A) ⇐⇒ b ∈ N (A). Then Ax = Aa + Ab = Aa. Therefore we can just project x onto C (A) first
and proceed as if x was in C (A).
So, now we see x as: projecting x onto C (A) and do a change of coordinates, multiply each coordinate,
and convert back to the original coordinate system. Note the diagonalization:
A = SΛS −1
Multiplying x by A is multiplying 3 matrices - S −1 which maps x onto the eigenvector coordinate system,
Λ that multiplies each coordinates ”remember that Λ is diagonal!), and S that maps the result onto the
original coordinate system again.
1.3.1 Application: Powers of Matrix
Using this diagonalization, powers of matrices can be calculated easily. Let what is A100 ? Since A2 is
A2 = SΛS −1
SΛS −1 = SΛ2 S −1
A100 = SΛ100 S −1 . And exponential of a diagonal matrix is really trivial.
2 Symmetric And Positive Definite-ness
2.1
Orthonormal Eigenvectors
When A is symmetric, we have special characteristics that make the diagonalization even better. Those
are:
1. All eigenvalues are real.
2. We can always choose n orthonormal eigenvectors; not only they are independent, but they can be
chosen to be perpendicular to each other!
The latter point now means that S T S = I ⇐⇒ S = S −1 . We should actually call this matrix Q now, as in
QR decomposition. That gives the following diagonalization:
A = QΛQT
2.1.1 Proof
¯ The second
The first property is proven by taking complex conjugate of λ and discovering that λ = λ.
property is more complex to prove; it comes from Schur’s theorem. Schur’s theorem states that:
3
Every square matrix can be factored into QT Q−1 where T is upper triangular and Q is orthonormal.
I think this is very intuitive - it’s an analogue of using QR decomposition to find an orthonormal basis for
the column space. Now, R is upper triangular - because first column in A is represented as a multiple of
the first column of Q. Second column in A is represented as linear combinations of first two columns in R,
etc. Now, we can think of decomposing R into
R = T Q−1
where Q−1 maps x onto this coordinate system, and T working in this coordinate system. Then QT Q−1
works just like the diagonalization in 1.3.
Now we use induction and prove the first row and column of T are zeros, except for the top left element.
If we continue like this, we get a diagonal T which means we will have:
A = QΛQ−1 = QΛQT
2.1.2 Sum of Rank One Matrices
The symmetric diagonalization can be rewritten as
A = QΛQT = λ1 x1 xT1 + λ2 x2 xT2 + · · ·
Yep, it’s a weighted sum of rank one matrices! Nice! Even better, x1 xT1 are projection matrices! Because
kxi k = 1 since orthonormal, the projection matrix formula
−1 T
A AT A
A
becomes just xi xTi in this case.
So, we do have a great interpretation of this; multiplication by a symmetric matrix can be done by
mapping x onto different, orthogonal 1D spaces. Then, each coordinate will be multiplied by a certain
number. Later we can map it back to the original coordinates. This provides another insight regarding
what eigenvalues and eigenvectors actually mean. The eigenvector associated with the largest eigenvalue
is the axis where multiplying A makes the biggest change!
Since truncated SVD is just a generalization of this diagonalization, it is very intuitive how PCA can be
done by SVD.
2.2
Positive Definite Matrices
2.2.1 Definition
A special subset of symmetric matrices are called positive definite. The following facts are all equivalent;
if one holds, everything else will.
1. If we do elimination, all pivots will be positive.
2. All eigenvalues are positive.
4
3. All n upper-left determinants are positive.
4. xT Ax is positive except at x = 0.
2.2.2 Gaussian Elimination is Equivalent to Completing the Square
How on earth is property 4 related to other properties? Why are they equivalent? The course proceeds
with an example, and states the following without further generalization.
P
xT Ax is effectively a second order polynomial: ij xi xj Aij . How on earth do we know if this is bounded
at 0? We complete the squares! Represent that formula by a sum of squares - and since squares are non-
negative we know the polynomial will be positive unless all numbers are 0. The big revealation is that this
process is same as Gaussian elimination. Positive coefficients on squares mean positive pivots! Holy crap.
Thus, if 1-3 holds, 4 will hold. If 4 holds, the result of the completed squares will show that 1-3 holds.
2.2.3 Relation to Second Order Test in Calculus
Recall the awkward second order test in Multivariate Calculus, which was stated without proof? It actually
checks if the Hessian is positive definite. Hahaha!
2.2.4
AT A is Always PosDef
A special form, AT A, always yields positive definite matrix regarless of the shape of R - if R has independent
columns. This is why covariance matrices are always positive!
Actually, the converse always holds too: if B is posdef, there always exists an A such that AT A = B. This
can be found by the Cholesky decomposition of B!
Recall: all covariance matrices are positive definite, and all positive definite matrices are covariance
matrices of some distribution .
3 Singular Value Decomposition
3.1
Definition and Intuition
SVD is a generalization of the symmetric diagonalization. It is applicable to any matrix ”doesn’t have to
be symmetric, or even square). Therefore it is the most useful. How can this be done? Instead of using a
single set of orthonormal basis, we use two sets U and V . Then a matrix A can be factored as:
A = U ΣV T
where U and V are orthonormal. U is a basis of column space of A, V is a basis of row space of A. Σ is a
”sort-of) diagonal matrix with singular values on its diagonals.
3.1.1 Intuition
The same intuition from symmetric diagonalization applies here. Given x, we can first map it to the row
space. ”Since V is orthonormal, V T = V −1 - so think of it as the mapping to the row space.) Now, x is rep-
5
resented as a linear combination of an orthnormal basis of the row space of A. We now multiply each coordinate by a number, and convert back to original coordinates by U . Since Ax is always in C (A), columns
of U are the basis of C (A).
Hmm, sounds like magic. I know I can map to a space, and recover original coordinate system from
that space. But I am mapping x to row space and recovering from column space as if we mapped x to the
column space as if we mapped to column space in the first place. WTF? This holds because columns of V
are sort-of eigenvectors of A; if we transform vi , we get a multiple of ui . Those are called singular vectors,
and the relation goes like this:
Avi = σi ui
where σi are called singular values. Also, orthogonality gives that all vi and ui s are orthogonal to each
other! Woah, there’s too much magic in here.. :)
3.1.2 Analogue to Rank-one Decomposition
We can also write:
A = U ΣV T = σ1 u1 v1T + σ2 u2 v2T + · · ·
we can easily see how PCA works with SVD.
3.2
Derivation
When m 6= n for A ∈ Rm×n , the dimensions are kind of match up. But I will gloss over the details. ”It’s past
midnight)
Let U be the eigenvector matrix for AAT , V be the eigenvector matrix for AT A and everything works
out. We can choose these matrices to be orthnormal, because AT A and AAT are symmetric. Now we assert
we can find Σ such that:
A = U ΣV T
Multiply both sides by AT on the left:
AT A = U ΣV T
T
U ΣV T = V ΣU T U ΣV T = V Σ2 V T
the last equality coming from that we chose Σ to be orthonormal. Yep, RHS is the symmetric decomposition of AT A - I said it works out. So the diagonal entries of Σ will contain square roots of eigenvalues of
AT A.
Now we can close the loop by proving
AV Σ−1 = U
First, we prove that columns of AV Σ−1 have unit length:
AT AV = V Σ2 ⇐⇒ V T AT AV = V T V Σ2
T
⇐⇒ (AV ) AV = Σ2
6
”2)
The last equality shows that columns of AV are orthogonal, and the ith singular value gives its length
squared.
Next, we prove AV Σ−1 is indeed an eigenvector of AAT . Multiply ”2) by A on both sides, on the left, and
rearrange to get:
AAT AV = ΣAV
Note that column vectors of AV are actually eigenvectors of matrix AAT . ”Smart parenthesis will let
you realize this).
4 That Was It!
Wasn’t too hard right?
7
Multivariable Calculus Lecture Notes
jongman@gmail.com
January 21, 2015
Contents
1 Directional Derivatives
2
2 Lagrange Multipliers
2
3 Non-independent Variables
3
4 Double Integrals
4
5 Polar Coordinates
4
5.1 Moment of Inertia . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
6 Change Of Variables
4
4
6.1 General Method and Jacobian . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4
6.2 Outline of Proof . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5
6.3 Revisiting Polar Coordinates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5
6.4 Determining Boundaries with Changed Variables . . . . . . . . . . . . . . . . . . . . . . . . . .
5
7 Vector Fields and Line Integrals
5
7.1 Parameterized Representation of Trajectory . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
6
7.2 Geometric Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
6
7.3 Gradient Field . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
6
7.3.1 The Fundamental Theorem of Calculus for Line Integrals . . . . . . . . . . . . . . . . .
7
7.3.2 Testing If F Is A Gradient Field . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
7
7.3.3 Identifying Potentials . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
7.4 Curl . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
7
8
8 Green’s Theorem
8.1 Proof . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
8
8
9 Flux and Green’s Theorem in Normal Form
9
9.1 Interpretation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
9
9.2 Green’s Theorem in Normal Form . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
9
10 Triple Integrals and Spherical Coordinates
9
10.1 Cylinderical Coordinates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
9
10.2 Spherical Coordinates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
9
1
11 Vector Fields in Space
10
11.1 Flux in Space . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
11.1.1 Examples: Spheres and Cylinders . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
10
10
11.1.2 General Case . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
10
11.1.3 Even More General: Parametric Description . . . . . . . . . . . . . . . . . . . . . . . . .
11
11.1.4 When You Only Know The Normal Vector . . . . . . . . . . . . . . . . . . . . . . . . . . .
11.2 Divergence Theorem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
11
12
11.2.1 Del Notation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
12
11.2.2 Physical Interpretation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
13
11.2.3 Proof . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
13
11.2.4 Application: Diffusion Equations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
13
11.3 Line Integrals In The Space . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
11.3.1 Testing If A Vector Field Is A Gradient Field . . . . . . . . . . . . . . . . . . . . . . . . . .
14
14
11.3.2 Using Antiderivatives to Find The Potential . . . . . . . . . . . . . . . . . . . . . . . . . .
14
11.4 Curl in 3D . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
11.5 Stokke’s Theorem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
15
15
11.5.1 Orientation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
15
11.5.2 Relationship To Green’s Theorem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
16
11.5.3 Outline of Proof . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
16
11.6 Green and Related Theorems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
16
11.7 Simply-connectedness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
16
11.8 Path Independence and Surface Independence
16
. . . . . . . . . . . . . . . . . . . . . . . . . . .
12 Exam Review
17
13 Extra: Matrix Derivatives
17
13.1 Derivative Matrix and Gradient . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
17
13.2 The Chain Rule for Derivative Matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
18
13.3 Hessian Matrix
18
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1 Directional Derivatives
What is the derivative of function f along a vector ~v ? Simply,
∇f ·
~v
|~v |
which is the gradient vector projected onto the unit vector. Makes so much sense.
2 Lagrange Multipliers
Instead of solving regular minimization/maximization problems, one can solve such problems when the
inputs are constrained , where they are on a curve. Restating,
Maximize/minimize f (x, y, z) such that g (x, y, z) = c
2
How do we solve this? Note that when the extreme will happen on one of the boundary, or where the
level curve of f is parallel to the curve g = c. Here is a colloquial, informal argument; if they were not
parallel, working along the curve in the right direction will increase/decrease the value of f , thus it won’t
be a maximum/minimum.
Finding such point is easy: the gradient vector of f is perpendicular to the level curve, and the gradient
of g is perpendicular to the constraint curve. So we set
∇f = λ∇g
which gives us a system of equations. The λ is called the Lagrange Multiplier.
Note the Lagrange Multiplier condition being satisfied is only a necessary condition for minimum/maximum points; always check the boundary, as well as minimum/maximum criteria.
3 Non-independent Variables
What happens when some of the variables are related in a multivariate function? Let w = x2 + y 2 + z 2 ,
where z = x2 + y 2 . What is ∂w
∂x ?
Turns out this problem is ill-defined. All the materials so far assumed all the variables are independent;
however in this problem it is not. The restriction equation is a curve; we are differentiating the value of
w along a curve. It depends on which direction you are differentiating – in other words, which direction
you are holding constant.
A new notation is in order:
∂w
∂x
y
means differentiating w with respect to x, while holding y constant. Therefore, z will become a dependent
variable and change as we go. This has the effect of slicing the constraint surface with a xy plane and
differentiating on the resulting curve.
Now how do we calculate these?
• Total differentials: take total differentials, let the differentials for the fixed variables to be 0. Solve
for interested derivative.
dw = 2xdx + 2ydy + 2zdz
= 2xdx + 2zdz
dz = 2xdx + 2ydy
= 2xdx
Combining,
Solving for dw/dx
dw = 2xdx + 2 x2 + y 2 2xdx
dw
= 2x + 4x3 + 4xy 2
dx
• Chain rule:
3
∂w
∂x
y
∂y
∂z
∂x
+wy
+wz
= 2x + 4xz = 2x + 4x3 + 4xy 2
= wx
∂x
∂x
∂x
| {z }
| {z }
| {z }
1
0
2x
4 Double Integrals
Iterative integral method is introduced.
5 Polar Coordinates
Sometimes it is helpful to use polar coordinates, r and θ. When we use this change of variables, we cannot
say:
dxdy = drdθ
Why? We are dividing the region as a set of infinitesimal rectangles, one side of which is dr, but the
another side is r · dθ. So we have
dxdy = r · drdθ
5.1
Moment of Inertia
I=
¨
2
d (x, y) δ (x, y) dA
d being the distance from the axis of rotation, δ being the density.
6 Change Of Variables
6.1
General Method and Jacobian
Say we have a function of x and y, and want to integrate it by u (x, y), v (x, y). ”This usually happens when
this change simplifies the integrand greatly, or the boundaries of the integration) How do we do this in
general?
When we transform dxdy into dudv – what is the conversion factor for transformation? It is the absolute
value of the Jacobian, which is defined as:
∂ (u, v) ux
J=
=
∂ (x, y) vx
uy vy which is the determinant of the matrix of partial derivatives. We have:
ˆ
dudv =
ˆ
|J| dxdy
You can also remember it as
|J|∂ (x, y) = ∂ (u, v)
4
Note the absolute value: the area of the area element is always positive. Also note
∂ (u, v) ∂ (x, y)
·
=1
∂ (x, y) ∂ (u, v)
which is useful for easier calculation.
6.2
Outline of Proof
We take total derivatives of the relationship to get:
"
∆u
∆v
#
=
"
ux ∆x + uy ∆y
vx ∆x + vy ∆y
#
=
"
ux
vx
uy
vy
#"
∆x
∆y
#
Now transform(∆x, 0) and (0, ∆y) to uv-coordinate space. We get (ux ∆x, vx ∆y) and (uy ∆y, vy ∆y). Now,
take the cross product of these variables which is the area of the translated rectangle:
dA′ = dudv = (ux vy − vx uy ) ∆x∆y = Jdxdy
6.3
Revisiting Polar Coordinates
Let’s try to revalidate our polar coordinates formula using the Jacobian. We have the following transformation:
"
x
y
#
=
"
r cos θ
r sin θ
#
The Jacobian is:
x
r
J=
yr
So
xθ cos θ
=
yθ sin θ
−r sin θ =r
r cos θ dxdy = |r| dudv = rdudv
which justifies our formula for polar coordinates.
6.4
Determining Boundaries with Changed Variables
There are no general methods, a careful case-by-case analysis is needed. One good tip is transform the
integrated region into uv-coordinate space ”not always possible, when we are in higher dimension), and
consider each edge one by one.
7 Vector Fields and Line Integrals
Vector field is defined by a function from R2 → R2 :
F~ (x, y) = M (x, y) ˆi + N (x, y) ˆj
5
A line integral integrates a function of the vector field along a trajectory. For example, you can find
the work done by the forces in the vector field, to a particle moving along a given trajectory. The work is
defined as the dot product of the force and the displacement:
W = F~ · ∆r
So the quantity we want to calculate is:
ˆ
F~ (x, y) ∆r
C
which is very vague. What should we do?
7.1
Parameterized Representation of Trajectory
Let ~r be a function of time parameter, t. Then we can slice the trajectory by infinitesimal dt and have:
ˆ
F~ (x, y) dr =
C
ˆ
dr
F~ (x (t) , y (t)) dt
dt
the latter integral is just a single variable integral with respect to t. Effectively, we are integrating the
dot product of the force vector and the velocity vector.
The same calculation could be carried out using a different notation: letting dr = hdx, dyi and realizing
~
F = hM, N i gives
ˆ
M dx + N dy
C
We cannot integrate this directly; M and N both rely on each of the two parameters. We can get dx and
dy by using total differentials and represent them at t. This results in the same integral.
7.2
Geometric Approach
Notice:
d~r = T~ ds
where T~ is the tangent vector along the trajectory, and ds is the length element. Noticing this will let
you integrate some of the integrals just intuitively.
7.3
Gradient Field
In physics, we see something called gradient field; vector field where F is defined as a gradient of a function.
F~ = ∇f
The following three statements are equivalent.
1. F is a gradient field.
6
2. Path-dependency: if two paths have coinciding starting and ending points, the work done by the field
along the two paths are the same.
3. Conservativeness: the work done to any closed path is 0.
The proofs are not too involved. ”Don’t think we covered the proof from 3 to 1 though..)
7.3.1 The Fundamental Theorem of Calculus for Line Integrals
States
ˆ
∇f · d~r = f (P1 ) − f (P0 )
C
where P1 and P0 are ending and starting points of the path. Outline of proof:
ˆ
∇f · d~r =
C
ˆ fx
C
dy
dx
+ fy
dt
dt
with the second equality coming from the chain rule.
dt =
ˆ
C
df
dt =
dt
ˆ
df
C
7.3.2 Testing If F Is A Gradient Field
It is claimed that F~ = M ˆi + N ˆj is a gradient field iff:
My = N x
and the field is defined & differentiable in the entire plane.
7.3.3 Identifying Potentials
Say the above equation holds - can we recover the potential function from the gradients? Two ways are
discussed:
1. Set up an arbitrary path from (0, 0) to (x1 , y1 ) and do a line integral. The value will be f (x1 , y1 )−f (0, 0)
but as we don’t care about constants ”potentials are equivalent up to a constant) we can just calculate
f (x1 , y1 ) and take it to the potential function.
2. Use antiderivatives - we know fx = M and fy = N . So calculate
ˆ
M dx = u (x, y) + g (y)
x
note g (y) appears in place of the integration constant. We can differentiate this with regard to y and
try to match it with N . You generally don’t want to integrate N wrt y and match the two equations when trigs are involved it might not be easy to see the equivalence of two portions of the equations,
causing confusion.
I guess this kind of serves as a constructive proof for the equivalence in the beginning of 7.3.
7
7.4
Curl
The curl is defined as:
curlF = Nx − My
from the definition, we know the curl is 0 when F is a gradient field. In physics context, the curl measures
the angular/rotational velocity/torque. Of course, I don’t care about those applications.
8 Green’s Theorem
Green’s Thorem relates seemingly unrelated concepts, the curl and the line integral. If C is a closed curve
that encloses the region R in counterclockwise direction, and F~ is defined everywhere, Green’s theorem says:
˛
F~ · d~r =
C
˛
M dx + N dy =
C
¨
Nx − My dA =
R
¨
curlF dA
R
Note this can be used to prove that F~ is conservative if the curl is 0.
8.1
Proof
Here’s an outline of the proof.
First, we take the special case N = 0. Why does this suffice? The similar strategy can be used to prove
the case for M = 0, and then we have two equalities which can be added to get the desired equality.
˛
¨
M dx =
˛C
−My dA
¨R
N dy =
Nx dA
R
C
Second, we note that we are able to divide the region freely, calculate the integral separately, and sum
them. The line which separates the two regions will get counted twice; but if we keep the counterclockwise
direction, the two curves will cancel each other out. So, we say we will slice the region into vertically
simple slices, which is defined by a ≤ x ≤ b, f1 (x) ≤ y ≤ f2 (x).
Then
˛
M dx =
ˆ
b
M (x, f1 (x)) dx −
b
M (x, f2 (x)) dx
a
a
C
ˆ
Note the first equality holds because the line integrals are 0 in the vertical edges of the region, since
dx = 0. What about the curl integral?
¨
b
−My dA =
ˆ
b
=
ˆ
f2 (x)
−
f1 (x)
a
R
ˆ
∂M
dydx
∂y
M (x, f1 (x)) − M (x, f2 (x)) dx
a
which is equivalent to the expression for
¸
C
M dx above and thus gives the proof.
8
9 Flux and Green’s Theorem in Normal Form
Flux along a curve C is defined by
ˆ
F ·n
ˆ ds
C
where n
ˆ is a unit vector normal to C, rotated 90 degrees clockwise from the curve. If you think about it,
it is actually rotating all the vectors in the vector field and calculating the work done by the field. Therefore,
the Flux becomes, if F = hP, Qi, the rotated field F ′ = h−Q, P i ”note the field is rotated clockwise)
ˆ
F ·n
ˆ ds =
C
ˆ
−Qdx + P dy
C
and after that it’s just the regular line integral.
9.1
Interpretation
If F is a velocity field, the flux measures the amount that passes through the given line, each unit time.
9.2
Green’s Theorem in Normal Form
If F = hP, Qi,
˛
F ·n
ˆ ds =
C
˛
−Qdx + P dy =
C
¨
Px + Qy dA =
¨
divF dA
we know this is equivalent to the regular Green’s theorem. Also, Px + Qy = divF is called the divergence which measures the expanding-ness of the field.
10 Triple Integrals and Spherical Coordinates
This portion of the course deals with integrating in the 3D space. The most obvious way is to use (x, y, z)
coordinates – but that can be tricky at times. Often, using alternative coordinate system can make things
easier.
10.1 Cylinderical Coordinates
This is an extension of the polar coordinates, keeping z axis intact. Each point in space is represented by
the triplet (r, θ, z). The volume element dV becomes r · dr · dθ · dz. Similarly, dS = rdθdz
10.2 Spherical Coordinates
A point is represented by a triplet of coordinates (ρ, φ, θ). ρ represents the distance from the origin (ρ ≥ 0),
φ represents the angle from the positive z-axis (0 ≤ φ ≤ π), θ represents the usual angle from the positive
x-axis. First, we see z = ρ cos φ. r, the distance from the origin to the projection onto XY -plane, is given by
ρ sin φ; therefore we have
(x, y, z) = (ρ sin φ cos θ, ρ sin φ sin θ, ρ cos φ)
9
Mostly, they are integrated in the following order: dρdφdθ. What does a volume element dV look like?
Note it is approximately a cuboid. The depth of the cuboid is easily found; it is ∆ρ. What are the lengths of
the sides? The top-bottom sides ”parallel to the xy plane) are basically arcs of circles of length r = ρ sin φ.
Therefore, its length is given by ρ sin φ∆θ. The left-right sides are arcs of circles of length ρ, so they are
ρ∆φ. So we have:
dV = ρ2 sin φdρdφdθ
Similarly, the surface element can be found as
dS = ρ2 sin φdφdθ
11 Vector Fields in Space
Vector fields in spaces are defined. They are not that different from the 2D cousin.
11.1 Flux in Space
The flux is not defined for lines in space; they are defined for surfaces. You should set up a double integral
which represents the surface, and take the integral of F · n
ˆ dS where F is the vector field, n
ˆ is the normal
vector to the surface, and dS is the surface area element. Note that there is not a preset way to set the
orientation of n
ˆ as in 2D cases – you’ll have to set it explicitly. Then
Flux =
¨
F~ · ~ndS
S
If F is a velocity field, the flux of F through the surface S represents the amount of the matter that passes
through S per unit time.
11.1.1 Examples: Spheres and Cylinders
For a sphere of radius a centered at the origin, we have:
1
n
ˆ = ± hx, y, zi
a
For a cylinder of radius a with its center coinciding with the z axis, we have:
1
n
ˆ = ± hx, y, 0i
a
11.1.2 General Case
When z = f (x, y), you should set up bounds for x and y according to the shadow the surface casts on the
xy plane. Also, we have:
n
ˆ dS = ± h−fx , −fy , 1i dxdy
so if F = hP, Q, Si,
10
¨
F~ · n
ˆ dS = ±
R
¨
−P fx − Qfy − Rdxdy
R
What is the proof/intuition behind this? Find a rectangle in the shadow of the curve with two sides ∆x
and ∆y: the corresponding part of the curve will roughly look like a parallelogram, if the shadow rectangle
is small enough.
How do we find the area of a parallelogram? Of course: cross products. Also note the cross product
will give us a normal vector; so it gives two in one!
Say the lower-left point of the shadow rectangle is (x, y). The corresponding point in the curve will be
(x, y, f (x, y)). We now can find two vectors ~u and ~v which coincide with the sides of the parallelogram, and
their cross product will be:
~u × ~v = ~ndS
Fortunately, they are easy to find. ~u goes from (x, y, f (x, y)) to (x + ∆x, y, f (x + ∆x, y)). Note that f (x + ∆x, y) ≈
f (x, y) + ∆x · fx (x, y) . Using similar process for ~v , we have:
~u = h∆x, 0, fx (x, y) · ∆xi
= h1, 0, fx i ∆x
~v = h0, ∆y, fy (x, y) · ∆yi
= h0, 1, fy i ∆y
Now
ˆi ˆj kˆ ~u × ~v = 1 0 fx = h−fx , −fy , 1i dxdy
0 1 f y 11.1.3 Even More General: Parametric Description
S : (x, y, z) where each coordinate is a function of two parameters: x = x (u, v) , y = y (u, v) , z = z (u, v). Now,
we want to express n
ˆ dS as somethingdudv. Therefore, we find a position at ~r (u, v) and find a parallelogram
with two sides ending at ~r.
Using the similar technique, we get
n
ˆ ∆S = ±
∂~r
∂~r
∂~r
∂~r
∆u∆v
∆u ×
∆v = ±
×
∂u
∂v
∂u ∂v
Note the analogeous pattern of this to the above formula.
11.1.4 When You Only Know The Normal Vector
Say, if you know an equation of a plane, then the normal vector comes free. Another case; if your surface
is given by g (x, y, z) = 0, you have ∆g as the normal vector. You can still use the above methods, but there
is a shortcut.
Slanted Plane
If your surface is a slanted plane: find the angle the plane and the xy plane makes. Call this α. Then, the
area element of the shadow ∆A is related to the area element of the shadow ∆S by:
11
∆A = cos α∆S
which is trivial. However, note that we can find cos α by taking the inner product between their normal
vectors!
~ · kˆ
N
cos α = ~
N
So we have
∆S =
and
n
ˆ ∆S =
~
n
ˆ · N
~ · kˆ
N
~
N ~ · kˆ
N
∆A
∆A = ±
~
N
dxdy
~ · kˆ
N
g (x, y, z) = 0
Say we have a surface described by:
z − f (x, y) = 0
Then
~ = ∇g = h−fx , −fy , 1i
N
Apply the above method and it is equivalent to what we saw in 11.1.2.
11.2 Divergence Theorem
Also known as Gauss-Green theorem , it is a 3D analogue for Green in Normal Form. Goes like this:
If a closed surface S completely encloses a region D, and we choose to orient the normal vectors to point
outwards, and F~ is a vector field defined and differentiable everywhere in D:
‹
S
~=
F~ · dS
‹
F~ · n
ˆ dS =
˚
divF~ dV
D
S
Now what is the divergence in 3D? Very easy to remember:
div P ˆi + Qˆj + Rkˆ = Px + Qy + Rz
11.2.1 Del Notation
Before we proceed to proof, a del notation is introduced:
∇=
∂ ∂ ∂
,
,
∂x ∂y ∂z
12
which is a very informal notation, but we’ll just use this for now. So now we can think of divergence in
terms of:
∇ · F~ =
∂Q ∂R
∂P
+
+
∂x
∂y
∂z
This is not equivalent to a gradient; notice the dot there. The gradient is a vector; the divergence is a
function.
11.2.2 Physical Interpretation
A divergence of a vector field corresponds to a source rate , which is the amount of flux generated per
unit volume. If our vector field is a velocity field of an incompressible fluid, then the surface integral of
Flux is the amount of fluid leaving D per unit time.
11.2.3 Proof
We employ the exact same strategy we used for proving Green’s theorem. Here’s an outline.
• First, assume a vector field with only z component. ”We can later do the same thing to x and y, and
sum up three identities to get the theorem.)
• Assume a vertically simple region - something that lives between two graphs z1 (x, y) and z2 (x, y).
˝
‚
• Now evaluate the integral
Rz dzdxdy and h0, 0, Ri · n
ˆ dS and hope to meet in the middle.
– The latter integral can be done by dividing the entire surface into three pieces; the top, the bottom, and the sides. We can expect to have 0 flux for sides since our vector field doesn’t have any
x or y component.
– The top and the bottom integral can be done by using the formula in 11.1.2. ”Caveat: we’ll need
to take care of the different orientation of the normal vector on top and bottom!)
• Then we can move on to the general case; we can always slice any region into vertically simple slices,
take the flux integral, and sum them up and they will be the same..
11.2.4 Application: Diffusion Equations
Diffusion equation governs how a new liquid diffuses within the ambient ”immobile) liquid. Ink in a cup
of water, smoke in the air, ...
We want to find the function that represents the concentration at the given point, from its partial derivatives: u (x, y, z, t).
We have the set of equations:
∂u
=k
∂t
2
=k∇
· ∇u} = k
∇ u
| {z
|{z}
Laplacian
div(∇u)
∂2u ∂2u ∂2u
+ 2 + 2
∂x2
∂y
∂z
The same equation can be used in modeling heat in immobile air.
So how is the vector field defined? Given a concentration, every point moves towards the direction
where the concentration decreases the fastest; that is negative gradient. In fact:
13
F~ = −k∇u
Now, whats the relationship between F~ and
∂u
∂t ?
It’s the divergence theorem! Assume a small closed
region D with a surface S. The flux of D through S is given as:
˚
divF~ dV =
D
d
F~ · n
ˆ dS = amount of smoke/liquid through S = −
dt
S
‹
˚
udV
D
The first equality holding by the divergence theorem, and the latter equality justified by our definition
above. We can modify the RHS to state:
˚
divF~ dV = −
D
d
dt
˚
udV = −
D
˚
∂u
dV
∂t
The latter equality holds because the middle quantity is the derivative of sums; which is equivalent to
sums of derivatives. So we have:
divF~ = −
∂u
∂t
11.3 Line Integrals In The Space
Doesn’t change things much, but checking if a vector field is a gradient field becomes more complicated.
F~ = P ˆi + Qˆj + Rkˆ
Now then
Work =
ˆ
C
F~ · d~r =
ˆ
P dx + Qdy + Rdz
C
Similar to the planar case, we parameterize the line and express the integral in terms of the parameter.
”See above)
11.3.1 Testing If A Vector Field Is A Gradient Field
Contrary to the 2D case where we only checked My = Nx , we now have three different conditions to check.
We exploit that the second derivatives are the same regardless of the order you take them. So we check:
• Py = Q x
• Pz = R x
• Q z = Ry
The field is a gradient field iff these hold and the field is defined for a simply connected region.
11.3.2 Using Antiderivatives to Find The Potential
Say we want to solve:
• fx = 2xy
• fy = x2 + z 3
14
• fz = 3yz 2 − 4z 3
Calculate
ˆ
fx dx = x2 y + g (y, z)
Note the integration constant g (y, z) is a function only consists of y and z. Then differentiate by y and
compare with fy .
∂
fy = x + z = x + g y =
∂y
2
3
2
ˆ
fx dx
that tells us g = yz 3 + h (z). Of course, we will find h (z) with comparing the derivative with fz .
fz = 3yz 2 − 4z 3 = 3yz 2 + h′
which tells us h (z) = −z 4 + c then we have
f (x, y, z) = x2 y + yz 3 − z 4 + c
11.4 Curl in 3D
Curl measures how much a vector field fails to be conservative. It is a vector-valued function:
curlF~ = (Ry − Qz ) ˆi + (Pz − Rx ) ˆj + (Qx − Py ) kˆ
F~ is defined in a simply connected region and is conservative, iff curl is 0. How to remember this:
ˆi
∂
∇ × F~ = ∂x
P
ˆj
∂
∂y
Q
kˆ ∂ = curlF~
∂z R
Geometrically, curl measures the rotation component in a velocity field. Its direction points to the axis of
rotation, and its magnitude gives us twice the angular velocity.
11.5 Stokke’s Theorem
˛
C
F~ · d~r =
¨ ∇ × F~ ·ˆ
ndS
S|
{z }
curlF~
If C is a closed curve, and S is any surface bounded by C. Any surface, for real?
11.5.1 Orientation
The orientations of C and S needs to be compatible. What does it mean? Right-hand rule: if we walk along
C with S on our left side, the normal vector should be pointing up for us.
Right hand rule: point thumb along C in the positive direction. Point index finger towards interior of
S, tangent to S. Middle finger points n
ˆ.
15
11.5.2 Relationship To Green’s Theorem
When we limit a surface and the curve to a 2D plane, we get Green’s theorem.
11.5.3 Outline of Proof
• By Green’s theorem, we know it holds for C and S in x, y plane.
• By swapping coordinates accordingly, we can say the same for any coordinate plane; yz and xz planes.
• Actually, it holds for any right-handed coordinates; whatever rotation you use, it will hold by Green’
theorem. ”Using that work, flux, curl make sense independently of coordinates.)
• Given any S, we can decompose it into tiny, almost flat pieces. I guess we know where we are going!
• So using the same cancelling argument, the only line integrals that are calculated are the ones along
the edges.
• In each of the small surface, since it’s almost flat , Green’s theorem works and the line integral equals
the flux.
• Now: the sum of flux = the sum of line integrals = line integral along C.
11.6 Green and Related Theorems
Thesis
2 dimensions
Total work ”line integral) equals total curl/flux of curl ”surface integral)
Green’s theorem
Flux ”line/surface integral) equals total divergence ”surface/volume integral)
Green’s theorem in normal form
Note that for 2D
ˆ flux of curl equals total z component of 3D curl, which is
• since the normal vector of 2D xy plane is k,
the total 2D curl.
• the surface and the volume are the same things so the Green’s theorem in normal form works in
surface form.
The direction conventions are set that the theorems will be compatible between 2D and 3D.
11.7 Simply-connectedness
A region is simply-connected if every closed loop inside it bound at least one surface inside the region.
”The surface can be arbitrary.) Simply connectedness is critical to applying Green’s or Stoke’s theorem.
11.8 Path Independence and Surface Independence
Green’s theorem gives you path independence when F~ is a gradient vector; the line integral will be the
same whatever line you choose, when the endpoints are the same. Now note that Stoke’s theorem does
somesthing similar; we are getting surface independence , where the surface integrals are the same whenever the edges are the same.
16
S
Di
In the 2D case, the gradient field property gives this independence; what is its 3D equivalent that yields
surface independence?
Take two surfaces S1 and S2 bounded by a curve C. What are the difference between two surface
integrals? Merge the two surfaces S1 and S2 to get a surface S which is closed. We can calculate the flux
across that surface by divergence theorem! Jeez, my head is spinning.
˚
D
div ∇ × F~ dV
If this divergence integral is always 0, this gives you the surface independence. We can check:
∇ × F~ = hRy − Qz , Pz − Rx , Qx − Py i
The divergence of this is
div ∇ × F~ = (Ry − Qz )x + (Pz − Rx )y + (Qx − Py )z
= Rxy − Qxz + Pyz − Rxy + Qxz − Pyz
=0
Yes, the divergence of a curl is always zero!
12 Exam Review
We did triple, double, and single integrals in space:
13 Extra: Matrix Derivatives
This summarizes the treatment of multivariate calculus through gradient and Jacobian. Although they are
mostly technical matter, these have been important in many cases.
13.1 Derivative Matrix and Gradient
If f : Rn → Rm , we have

f1

 
 f2 
 
 
f (x1 , x2 , · · · , xn ) =  f3 
 . 
 . 
 . 
fm
the derivative matrix of f , Df is a m × n matrix consisting of first-order partial derivatives:
 ∂f1
∂x1
∂f2
∂x1
∂f1
∂x2
∂f2
∂x2
∂f1
∂x3
∂f2
∂x3
···
..
.
···
..
.
∂fm
∂x1
∂fm
∂x2
∂fm
∂x3
···


 .
 .
 .
..
.
17
∂f1
∂xn
∂f2
∂xn



.. 

. 
∂fm
∂xn
∂fi
.
Note Dfij = ∂x
j
When m = 1, which means a f is a real-valued function, we define ∇f , the gradient, to be a column
vector consisting of partial derivatives:

∂f
1
 ∂x
∂f
 ∂x
 2
 ∂f
 ∂x3


 
∂f ∂f ∂f
∂f

∇f =
,
,
,··· ,
=
 . 
∂x1 ∂x2 ∂x3
∂xn
 . 
 . 
∂f
∂xn
The latter round bracket notation is introduced for brevity on paper. Note Df is a row vector in this
case; Df = ∇f T .
13.2 The Chain Rule for Derivative Matrices
We have
D (f ◦ g) (x) = Df (g (x)) Dg (x)
where the multiplication is a matrix multiplication. Note the order is important! If we change the order it
might not work.
13.3 Hessian Matrix
When f is real-valued, we can define the Hessian matrix to be a n × n matrix with second-order partial
derivatives:

∂2f
2
 ∂∂x2 f1

 ∂x2 ∂x1
H (f ) = 


..
.
∂2f
∂xn ∂x1
∂2f
∂x1 ∂x2
∂2f
∂x2 ∂x2
···
..
.
···
..
.
∂2f
∂xn ∂x2
···
∂2f
∂x1 ∂xn
∂2f
∂x2 ∂xn
..
.
∂2f
∂xn ∂xn
Note this is a symmetric matrix. Note that

D∇f =
∂f
1
 ∂x
∂f
 ∂x
 2
 ∂f
D  ∂x3

 2
∂ f
2

  ∂∂x2 f1
 
  ∂x2 ∂x1
=
 .   ...
 .  
 . 
∂2f
∂f
∂xn
∂xn ∂x1
∂2f
∂x1 ∂x2
∂2f
∂x2 ∂x2
···
..
.
···
..
.
∂2f
∂xn ∂x2
···
18







∂2f
∂x1 ∂xn
∂2f
∂x2 ∂xn
..
.
∂2f
∂xn ∂xn




 = H(f )


Convex Optimization Lecture Notes ”Incomplete)
jongman@gmail.com
January 21, 2015
Contents
1 Introduction
5
2 Convex Sets
5
2.1 Types of Sets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.1.1 Affine Sets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5
5
2.1.2 Convex Sets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5
2.1.3 Cones . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.2 Important Sets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5
5
2.2.1 Norm Balls / Norm Cones . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
6
2.2.2 Positive Semidefinite Cone . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.3 Operations that Preserve Convexity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
6
6
2.3.1 Linear-fractional and Perspective Functions . . . . . . . . . . . . . . . . . . . . . . . . .
6
2.4 Generalized Inequalities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.4.1 Positive Semidefinite Cone . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.4.2 Minimum and minimal Elements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
6
7
7
2.5 Supporting Hyperplane Theorem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
7
2.6 Dual Cones . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.6.1 Dual Generalized Inequality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
7
7
2.6.2 Dual Generalized Inequality and Minimum/Minimal Elements . . . . . . . . . . . . . .
7
3 Convex Functions
8
3.1 Basic Properties and Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.1.1 Extended-value Extensions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
8
8
3.1.2 Equivalent Conditions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
8
3.1.3 Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
8
3.1.4 Epigraph . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
9
3.1.5 Jensen’s Inequality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.2 Operations That Preserve Convexity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
9
9
3.2.1 Composition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
10
3.3 The Conjugate Function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
10
3.4 Quasiconvex Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.4.1 Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
11
11
3.4.2 Operations That Preserve Quasiconvexity
. . . . . . . . . . . . . . . . . . . . . . . . . .
11
3.5 Log-concavity and Log-convexity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
11
1
4 Convex Optimization Problems
11
4.1 Optimization Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.1.1 Feasibility Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
11
11
4.1.2 Transformations and Equivalent Problems . . . . . . . . . . . . . . . . . . . . . . . . . .
4.2 Convex Optimization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
12
12
4.2.1 Optimality Conditions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
12
4.2.2 Equivalent Convex Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
13
4.3 Quasiconvex Optimization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
13
4.3.1 Solving Quasiconvex Optimization Problems . . . . . . . . . . . . . . . . . . . . . . . . .
13
4.3.2 Example: Convex over Concave
14
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.4 Linear Optimization Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
14
4.4.1 Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
14
4.4.2 Linear-fractional Programming . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
14
4.5 Quadratic Optimization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
15
4.5.1 Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
15
4.5.2 Second Order Cone Programming . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
15
4.5.3 SOCP: Robust Linear Programming
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
16
4.5.4 SOCP: Stochastic Inequality Constraints . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.6 Geometric Programming . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
16
16
4.6.1 Monomials and Posynomials . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
16
4.6.2 Geometric Programming . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
17
4.6.3 Convex Transformation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
17
4.7 Generalized Inequality Constraints . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.7.1 Conic Form Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
17
17
4.7.2 SDP: Semidefinite Programming . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
18
4.8 Vector Optimization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.8.1 Optimal Values . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.8.2 Pareto Optimality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
18
18
18
4.8.3 Scalarization for Pareto Optimality
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
18
4.8.4 Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
19
5 Duality
5.1 The Lagrangian And The Dual . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
19
19
5.1.1 The Lagrangian . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
19
5.1.2 The Lagrangian Dual Function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
19
5.1.3 Lagrangian Dual and Lower Bounds . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
20
5.1.4 Intuitions Behind Lagrangian Dual . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
20
5.1.5 LP example and Finite Dual Conditions . . . . . . . . . . . . . . . . . . . . . . . . . . . .
20
5.1.6 Conjugate Functions and Lagrange Dual . . . . . . . . . . . . . . . . . . . . . . . . . . .
20
5.2 The Lagrange Dual Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5.2.1 Duality Gap . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
20
21
5.2.2 Strong Duality And Slater’s Constraint Qualification . . . . . . . . . . . . . . . . . . . .
21
5.2.3 Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5.3 Geometric Interpretation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5.3.1 Strong Duality of Convex Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
21
21
21
5.4 Optimality Conditions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
22
2
5.4.1 Certificate of Suboptimality and Stopping Criteria . . . . . . . . . . . . . . . . . . . . . .
22
5.4.2 Complementary Slackness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5.4.3 KKT Optimality Conditions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
22
23
5.5 Solving The Primal Problem via The Dual . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5.6 Sensitivity Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
23
23
5.6.1 Global Sensitivity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5.6.2 Local Sensitivity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
24
24
5.7 Examples and Reformulating Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
24
5.7.1 Introducing variables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5.7.2 Making explicit constraints implicit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5.7.3 Transforming the objective . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
25
25
25
5.8 Generalized Inequailities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
25
6 Approximation and Fitting
6.1 Norm Approximation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
25
25
6.1.1 Examples . . . . . . . . . . . . . . . . . . . . . . . . . .
6.1.2 Different Penalty Functions and Their Consequences
6.1.3 Outliers and Robustness . . . . . . . . . . . . . . . . .
6.1.4 Least-norm Problems . . . . . . . . . . . . . . . . . . .
6.2 Regularized Approximations . . . . . . . . . . . . . . . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
25
26
26
27
27
6.2.1 Bi-criterion Formulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
6.2.2 Regularization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
6.2.3 ℓ1 Regularization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
27
28
28
6.2.4 Signal Reconstruction Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
6.3 Robust Approximation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
28
29
6.3.1 Stochastic Formulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
6.3.2 Worst-case Robust Approximation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
6.4 Function Fitting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
29
30
30
6.4.1 Constraints . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
6.4.2 Sparse Descriptions and Basis Pursuit . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
6.4.3 Checking Model Consistency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
30
30
31
7 Statistical Estimation
7.1 Parametric Distribution Estimation . . .
7.1.1 Logistic Regression Example . . .
7.1.2 MAP Estimation . . . . . . . . . . .
7.2 Nonparameteric Distribution Estimation
31
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
31
32
32
32
7.2.1 Priors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
7.2.2 Objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
32
33
7.3 Optimal Detector Design And Hypothesis Testing
7.4 Chebyshev and Chernoff Bounds . . . . . . . . . .
7.4.1 Chebyshev Bounds . . . . . . . . . . . . . .
7.4.2 Chernoff Bounds . . . . . . . . . . . . . . .
7.5 Experiment Design . . . . . . . . . . . . . . . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
33
33
33
34
34
7.5.1 Further Modeling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
34
3
.
.
.
.
.
.
.
.
.
.
.
.
.
.
8 Geometric Problems
34
8.1 Point-to-Set Distance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
8.1.1 PCA Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
34
35
8.2 Distance between Sets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
8.3 Euclidean Distance and Angle Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
35
35
8.3.1 Expressing Constraints in Terms of G . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
8.3.2 Well-Condition Constraints . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
8.3.3 Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
35
36
36
8.4 Extremal Volume Ellipsoids . . . . . . . . . .
8.4.1 Lowner-John Ellipsoid . . . . . . . . .
8.4.2 Maximum Volume Inscribed Ellipsoid
8.4.3 Affine Invariance . . . . . . . . . . . .
8.5 Centering . . . . . . . . . . . . . . . . . . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
36
36
36
37
37
8.5.1 Chebychev Center . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
8.5.2 Maximum Volume Ellipsoid Center . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
8.5.3 Analytic Center . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
37
37
38
8.6 Classification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
38
8.6.1 Linear Discrimination . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
8.6.2 Robust Linear Discrimination . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
8.6.3 Nonlinear Discrimination . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
38
38
39
8.7 Placement and Location . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
8.8 Floor Planning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
39
39
9 Numerical Linear Algebra Background
40
10 Unconstrained Optimization
40
10.1 Unconstrained Minimization Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
40
10.1.1 Strong Convexity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
10.1.2 Conditional Number of Sublevel Sets . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
40
41
10.2 Descent Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
10.3 Gradient Descent . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
10.3.1 Performance Analysis on Toy Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . .
41
41
42
10.4 Steepest Descent . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
42
10.4.1 Steepest Descent With an ℓ1 Norm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
10.4.2 Performance and Choice of Norm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
42
42
10.5 Newton’s Method . . . . . . . .
10.5.1 The Newton Decrement
10.5.2 Newton’s Method . . . .
10.5.3 Convergence Analysis .
10.5.4 Summary . . . . . . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
42
43
43
43
44
10.6 Self-Concordant Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
44
4
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
1 Introduction
2 Convex Sets
This chapter introduces numerous convex sets.
2.1
Types of Sets
2.1.1 Affine Sets
A set is affine when it contains any line that passes through any two points that belong to the set. Examples:
• Hyperplanes
• Solution set of linear equation systems.
An affine combination is defined to be a linear combination where the coefficients sum up to 1. An affine
set is closed under affine combination.
2.1.2 Convex Sets
A set is convex when it contains any line segment that passes through any two points that belong to the set.
A convex combination is defined to be an affine combination with nonnegative coefficients. A convex set
is closed under convex combination.
All affine sets are convex.
2.1.3 Cones
A set is a cone when a nonnegative scalar multiple of any element belongs to the set. A cone feels like a
ice-cream cone; it starts at the origin, and it gets wider going away from the origin. It could be analogous
to a pie slice. Given an point in a cone, the cone will contain the ray ”half-lines) originating from the origin
and going through the point.
A conic combination is a linear combination with nonnegative coefficients. A convex cone is closed
under conic combinations.
2.2
Important Sets
• Hyperplane is affine.
• Halfspaces are convex.
• Polyhedra/polyhedron are intersections of halfspaces, so are convex.
• Euclidean balls are defined by ||2 . They are convex. Elipsoids are related to balls, related by a positive
definite matrix P .
5
2.2.1 Norm Balls / Norm Cones
Given a norm, a norm ball replaces the L2 norm for Euclidean balls. A norm cone is a different beast; it is
the set
C = {(x, t) | kxk ≤ t, t ≥ 0} ∈ Rn+1
where x ∈ Rn . The norm cone with an L2 norm is called the second-order cone.
2.2.2 Positive Semidefinite Cone
The following notations are used:
• Sn is the set of all symmetric n × n matrices.
• Sn+ is the set of all positive semidefinite n × n matrices.
• Sn++ are positive definite.
Because
θ1 , θ2 ≥ 0, A, B ∈ Sn+ =⇒ θ1 A + θ2 B ∈ Sn+
the set is a convex cone.
2.3
Operations that Preserve Convexity
Some functions preserve convex-ness of the set.
• Intersections
• Taking image under an affine function
2.3.1 Linear-fractional and Perspective Functions
The perspective function
P (z, t) = z/t
preserves convexity. ”The domain needs that t > 0) Similarly, a linear-fractional function,
f (x) =
Ax + b
cT x + b
which is formed by combining the perspective function with an affine function, preserves convexity as
well.
2.4
Generalized Inequalities
A cone can be used to define inequalities, if it meets certain criteria. It goes like:
x K y ⇐⇒ y − x ∈ K
6
where K is a cone. It certainly makes sense; since K contains any rays, we get transitivity. If K is pointed
at the origin, we get asymmetry. Since K contains the origin, we get reflexivity. This is an useful concept
that will be exploited later in the course a lot.
2.4.1 Positive Semidefinite Cone
Actually, using the positive semidefinite cone to compare matrices is a standard practice so for the rest of
the book, matrix inequalities are automatically done used PSD cone.
2.4.2 Minimum and minimal Elements
Generalized inequalities do not always give you a single element that is the minimum; we sometimes get
a class of elements that are not smaller than any other elements, and are incomparable to each other.
2.5
Supporting Hyperplane Theorem
This has not proved to be useful in the course yet; when it is needed I will come back and fill it in...
2.6
Dual Cones
Let K be a cone. Then the dual cone is that
K ∗ = y|xT y ≥ 0 ∀x ∈ K
The idea of it is hard to express intuitively. When the cone is sharp, the dual cone will be obstuse, and
vice versa. Also, when K is a proper cone, K∗∗ = K.
2.6.1 Dual Generalized Inequality
The relationship K∗ is a generalized inequality induced by the dual cone K∗. In some ways it can relate
to the original inequality K . Most notably,
x K y ⇐⇒ y − x ∈ K ⇐⇒ λT (y − x) ≥ 0 ⇐⇒ λT x ≤ λT y
where λ is any element of K∗. The takeaway of this is that we can use dual cones to compare values
with respect to the original cone. See below for how this is used.
2.6.2 Dual Generalized Inequality and Minimum/Minimal Elements
Minimum Element From above, we can see x ∈ S will be a minimum element with respect to K, when
it is a unique minimizer λT x for all λ ∈ K∗. Geometrically, when x is a minimum element, the hyperplane
passing through x with λ as a normal vector z|λT (x − z) = 0 is a strict supporting hyperplane; it touches
S only at x. To see this: say x 6= y ∈ S. Since λT x < λT y by our assumption, we have λT (y − z) > 0.
Minimal Elements Here, we have a gap between necessary and sufficient conditions.
Necessity If x is a minimizer of λT x for all λ ∈ K∗, x is a minimal element.
Sufficiency Even if x is a minimal element, it is possible that x is not a minimizer of λT x for some λ ∈ K∗.
7
However, if S is convex, the two conditions are indeed equivalent.
3 Convex Functions
3.1
Basic Properties and Definitions
A function f is convex if the domain is convex, and for any x, y ∈ domf and θ ∈ [0, 1] we have
θ · f (x) + (1 − θ) · f (y) ≥ f (θx + (1 − θ) y)
Colloquially, you can say the chord is above the curve .
3.1.1 Extended-value Extensions
We can augment a function f :
f˜ (x) =

f (x)
∞
x ∈ domf
otherwise
which is called the extended-value extension of f . Care must be taken in ensuring the extension is still
convex/concave: the extension can break such properties.
3.1.2 Equivalent Conditions
The following conditions, paired with the domain being convex, is equivalent to convexity.
First Order Condition
T
f (y) ≥ f (x) + ∇f (x) (y − x)
Geometrically, you can explain this as the tangential hyperplane which meets a convex function at a
certain point, it will be a global lower bound in the entire domain. It has outstanding consequences; by
examining a single point, we get information about the entire function.
Second Order Condition
∇2 f (x) 0
which says the Hessian is positive semidefinite.
3.1.3 Examples
• ln x is concave on R++
• eax is convex on R
• xa is convex on R++ if a ≥ 1, concave 0 ≤ a ≤ 1
• |x| is convex on R
a
• Negative entropy: x log x is convex on R++
8
• Norms: every norm is convex on Rn
• Max functions: max {x1 , x2, x3 , · · · } is convex in Rn
• Quadratic-over-linear: x2 /y is convex in {y > 0}
• Log-sum-exp: as a soft approximation of the max function, f (x) = log (
• Geometric mean: (
Qn
i=1
xi )
1/n
is concave on Rn++
P
exi ) is convex on Rn .
• Log-determinant: log det X is convex on positive definite X.
3.1.4 Epigraph
An epigraph for a function f : Rn → R is a set of points above the graph. Naturally, it’s a subset of Rn+1
and sometimes is an useful way to think about convex functions. A function is convex iff its epigraph is
convex!
3.1.5 Jensen’s Inequality
The definition of convex function extends to multiple, even infinite, sums:
f
as long as
3.2
P
X
X
xi θi ≤
f (xi ) θi
θ = 1. This can be used to prove a swath of other inequalities.
Operations That Preserve Convexity
• Nonnegative weighted sums
• Composition with an affine mapping: if f is convex/concave, f (Ax + b) also is.
• Pointwise maximum/supremum: f (x) = maxi fi (x) is convex if all fi are convex. This can be used to
prove that:
– Sum of k largest elements is convex: it’s the max of
• Max eigenvalue of symmetric matrix is convex.
n
k
combinations.
• Minimization:
g (x) = inf f (x, y)
y∈C
is convex if f is convex. You can prove this by using epigraphs – you are slicing the epigraph of g.
• Perspective function
9
3.2.1 Composition
Setup: h: Rk → R and g: Rn → Rk . The composition f (x) = h (g (x)).
Then:
• f is convex:
– if h is convex and nondecreasing, and g is convex, or
– if h is convex and nonincreasing, and g is concave
• f is concave:
– if h is concave and nondecreasing, and g is concave, or
– if h is concave and nonincreasing, and g is convex
To summarize: if h is nondecreasing, and h and g has same curvature ”convexity/concavity), f follows. If
h is nonincreasing, and h and g have differing curvature, f follows h.
When determining nonincreasing/nondecreasing property of h, use its extended-value extension. Since
extended-value extension can break nonincreasing/nondecreasing properties, you might want to come up
with alternative definitions which are defined everywhere.
Composition Examples
• exp g (x) is convex if g is.
• log g (x) is concave if g is concave and positive.
•
1
g(x)
is convex if g is positive and concave.
• g (x) is convex if p ≥ 1 and g is convex and nonnegative.
p
Vector Composition If
f (x) = h (g1 (x) , g2 (x) , g3 (x) , · · · )
the above rules still hold, except:
• All gs need to have the same curvature.
• h need to be nonincreasing/nondecreasing with respect to every input.
3.3
The Conjugate Function
Given an f : Rn → R, the function f ∗: Rn → R is the conjugate:
f ∗ (y) = sup
x∈domf
y T x − f (x)
It is closely related to Lagrange Duals. I will omit further materials on this.
10
3.4
Quasiconvex Functions
Quasiconvex functions are defined by having all its sublevel sets convex. Colloquially, they are unimodal
functions. Quasiconvex functions are solved by bisection method + solving feasibility problems. The
linear-fractional function
f (x) =
aT x + b
cT x + b
is quasiconvex. ”Can prove: let f (x) = α and try to come up with the definition of the sublevel set.)
Remember how we solved this by bisection methods? :-)
3.4.1 Properties
• First-order condition: f (y) ≤ f (x) =⇒ ∇f (x) (y − x) ≤ 0. Note: the gradient defines a supporting
T
hyperplane. Since f is quasiconvex, all points y with f (y) ≤ f (x) must lie in one side of the hyperplane.
• Second-order condition: f ′′ (x) ≥ 0
3.4.2 Operations That Preserve Quasiconvexity
• Nonnegative weighted sum
• Composition: if h is nondecreasing and g is quasiconvex, then h (g (x)) is quasiconvex.
• Composition with affine/linear fractional transformation.
• Minimization along an axis: g (x) = miny f (x, y) is quasiconvex if f is.
3.5
Log-concavity and Log-convexity
The function is log-concave or log-convex if its log is concave/convex.
• The pdf of a Gaussian distribution is log-concave.
• The gamma function is log-concave.
4 Convex Optimization Problems
4.1
Optimization Problems
A typical optimization problem formulation looks like:
minimize f0 (x)
subject to fi (x) ≤ 0
(i = 1, · · · , m)
hi (x) = 0
(i = 1, · · · , p)
4.1.1 Feasibility Problem
There are cases where you want to find a single x which satisfies all equality and inequality constraints.
These are feasibility problems.
11
4.1.2 Transformations and Equivalent Problems
Each problem can have multiple representations which are same in nature but expressed differently. Different expression can have different properties.
• Nonzero equivalence constraints: move everything to LHS.
• Minimization/maximization: flip signs.
• Transformation of objective/constraint function: transforming objective functions through monotonic functions can yield an equivalent problem.
• Slack variables: f (x) ≤ 0 is swapped out by f (x) + s = 0 and s ≥ 0.
• Swapping implicit/explicit constraint: move an implicit constraint to explicit, by using extended value
extension.
4.2
Convex Optimization
A convex optimization problem looks just like the above definition, but have a few differences:
• All f s are convex.
• All gs are affine. Thus, the set of equivalence constraints can be expressed by aTi x = bi thus Ax = b
Note all convex optimization problems do not have any locally optimal points; all optimals are global. Also,
another important property arises from them: the feasible set of a convex optimization problem is convex,
because it is an intersection of convex and affine sets – a sublevel set of a convex function is convex, and
the feasible set is an intersection of sublevel sets and an affine set.
Another important thing to note is that convexity is a function of the problem description; different
formulations can make a non-convex problem convex and vice versa. It’s one of the most important points
of the course.
4.2.1 Optimality Conditions
If the objective function f0 is differentiable, x is the unique optimal solution if for all feasible y, we have
T
∇f0 (x) (y − x) ≥ 0
this comes trivially from the first order condition of a quasiconvex function ”a convex function is always
quasiconvex).
Optimality conditions for some special ”mostly trivial) cases of problems are discussed:
• Unconstrained problem: Set gradient to 0.
• Only equality constraint: We can derive the below from the general optimality condition described
above.
∇f0 (x) + AT ν = 0
(ν ∈ Rp )
Here’s a brief outline: for any feasible y, we need to have ∇f0 (x) (y − x) ≥ 0. Note that y − x ∈ N (A)
⊥
”the null space) because Ax = b = Ay =⇒ A (x − y) = 0. Now, this means ∇f0 (x) ∈ N (A) = R AT ,
T
the last term being the column space of AT . Now, we can let
12
∇f0 (x) = AT (−ν)
for some ν and we are done.
• Minimize over nonnegative orthant: for each i, we need to have

∇f (x) = 0
0
i
∇f (x) ≥ 0
0
i
if xi > 0
if xi = 0
This is both intuitive, and relates to a concept ”KKT conditions) which is discussed later. If xi is at
the boundary, it can have some positive gradient along that axis; we can still be optimal because
decreasing xi will make it infeasible. Otherwise, we need to have zero gradients.
4.2.2 Equivalent Convex Problems
Just like what we discussed for general optimization problems, but specially for convex problems.
• Eliminating affine constraint: Ax = b is equivalent to x ∈ F z + x0 where the column space of F is
N (A) and x0 is a particular solution of Ax = b. So we can just put f0 (F z + x0 ) in place of f0 (x).
• Uneliminating affine constraint: going in the opposite direction; if we are dealing with f0 (Ai x + bi ),
let Ai x + bi = yi .
– On eliminating/uneliminating affine constraint: on a naive view, eliminating affine constraint
always seem like a good idea. However, it isn’t so; it is usually better to keep the affine constraint,
and only do the elimination if it is immediately computationally advantageous. ”This will be
discussed later in the course.. but I don’t remember where)
• Slack variables
• Epigraph form: minimize t subject to f0 (x) − t ≤ 0 is effectively minimizing f0 (x). This seems stupid,
but this gives us a convenient framework, because we can make objectives linear.
4.3
Quasiconvex Optimization
When the objective function is quasiconvex, it is called a quasiconvex optimization problem. The biggest
difference is that we will now have local optimal points; quasiconvex functions are allowed to have flat
portions which give rise to local optimal points.
4.3.1 Solving Quasiconvex Optimization Problems
Quasiconvex optimization problems are solved by bisection methods; at each iteration we ask if the sublevel set empty for given threshold. We can solve this by a convex feasibility problem.
13
4.3.2 Example: Convex over Concave
Say p (x) is convex, q (x) is concave. Then f (x) = p (x) /q (x) is quasiconvex! How do you know? Consider
the sublevel set:
{x : f (x) ≤ t} = {x : p (x) /q (x) ≤ t} = {x : p (x) − t · q (x) ≤ 0}
and p (x) − t · q (x) is convex! So the sublevel sets are convex.
4.4
Linear Optimization Problems
In an LP problem, objectives and constraints are all affine functions. LP algorithms are very, very advanced
and all these problems are readily solvable in today’s computers. It is a very mature technology.
4.4.1 Examples
• Chebyshev Center of a Polyhedron: note that the ball lying inside a halfplane aTi x ≤ bi can be represented as
kuk2 ≤ r =⇒ aTi (xc + u) ≤ bi
Since
sup aTi u = r kai k2
kuk2 ≤r
we can rewrite the constraint as
aTi xc + r kai k2 ≤ bi
which is a linear constraint on xc and r. Therefore, having this inequality constraint for all sides of
the polyhedron gives a LP problem.
• Piecewise-linear minimization. Minimize:
max aTi x + bi
i
This is equivalent to LP: minimize t subject to aTi x + bi ≤ t! This can be a quick, dirty, cheap way to
solve convex optimization problems.
4.4.2 Linear-fractional Programming
If the objective is linear-fractional, while the constraints are affine, it becomes a LFP problem. This is a
quasiconvex problem, but it can also be translated into a LP problem. I will skip the formulation here.
14
4.5
Quadratic Optimization
QP is a special kind of convex optimization where the objective is a convex quadratic function, and the
constraint functions are affine.
1
minimize xT P x + q T x + r
2
subject to Gx h
Ax = b
and P ∈ Sn+ . When the inequality constraint is quadratic as well, it becomes a QCQP ”Quadratically Con-
strainted Quadratic Programming) problem.
4.5.1 Examples
• Least squares: needs no more introduction. When linear inequality constraints are added, it is no
longer analytically solvable, but still is very tractable.
• Isotonic regression: we add the following constraint to a least squares algorithm: x1 ≤ x2 ≤ · · · ≤ xn .
This is still very easy in QP!
• Distance between polyhedra: Minimizing Euclidean distance is a QP problem, and the constraints
”two polyhedras) are convex.
• Classic Markowitz Portfolio Optimization
– Given an expected return vector p¯ and the covariance matrix Σ, find the minimum variance
portfolio with expected return greater than, or equal to, rmin . This is trivially representable in
QP.
– Many extensions are possible; allow short positions, transaction costs, etc.
4.5.2 Second Order Cone Programming
SOCP is closely related to QP. It has a linear objective, but a second-order cone inequality constraint:
minimize f T x
subject to |Ai x + bi |2 ≤ cTi x + di
Fx = g
The inequality constraint forces the tuple Ai x + bi , cTi + di to lie in the second-order cone in Rn+1 .
When ci = 0 for all i, we can make them regular quadratic constraints and this becomes a QCQP. So basically,
using a second-order cone instead of a ”possibly open) polyhedra in LP.
t.
Note that the linear objective does not make SOCP weaker than QCQP. You can minimize t where f0 (x) ≤
15
4.5.3 SOCP: Robust Linear Programming
Suppose we have a LP
minimize cT x
subject to aTi x ≤ bi
but the numbers given in the problem could be inaccurate. As an example, let’s just assume that the
true value of ai can lie in a ellipsoid defined by Pi , centered at the given value:
ai ∈ E = {a¯i + Pi u| kuk2 ≤ 1}
and other values (c and bi ) are fixed. We want the inequalities to hold for all possible value of a. The
inequality constraint can be cast as
sup (a¯i + Pi u) = a¯i T x + PiT x2 ≤ bi
kuk2 ≤1
which is actually a SOCP constraint. Note that the additional norm term PiT x2 acts as a regularization
term; they prevent x from being large in directions with considerable uncertainty in the parameters ai .
4.5.4 SOCP: Stochastic Inequality Constraints
When ai are normally distributed vectors with mean a¯i and covariance matrix Σi , the following constraint
P aTi x ≤ bi ≥ η
says that a linear inequality constraint will hold with a probability of η or better. This can be cast as a SOCP
constraint as well. Since x will be concrete numbers, we can say aTi x ∼ n u¯i , σ 2 . Then
P aTi x ≤ bi = P
ui − u¯i
bi − u¯i
≤
σ
σ
=P
Z≤
bi − u¯i
σ
≥ η ⇐⇒ Φ
bi − u
¯i
σ
≥ η ⇐⇒
bi − µ¯i
≥ Φ−1 (η)
σ
The last condition can be rephrased as
1/2 a¯i T x + Φ−1 (η) Σi x ≤ bi
2
which is a SOCP constraint.
4.6
Geometric Programming
Geometric programming problems involve products of powers of variables, not weighted sums of variables.
4.6.1 Monomials and Posynomials
A monomial function f is a product of powers of variables in the form
16
f (x) = cxa1 1 xa2 2 · · · xann
where c > 0. A sum of monomials are called a posynomial; which looks like
f (x) =
K
X
k
ck xa1 1k xa2 2k · · · xannk
4.6.2 Geometric Programming
A GP problem looks like:
minimize f0 (x)
subject to fi (x) ≤ 1
(i = 1, 2, 3, · · · , m)
hi (x) = 1
(i = 1, 2, 3, · · · , p)
where f are posynomials and h are monomials. The domain of this problem is Rn++ .
4.6.3 Convex Transformation
GP problems are not convex in general, but a change of variables will turn a GP into a convex optimization
problem. Letting
yi = log xi ⇐⇒ xi = eyi
yields a monomial f (x) to
f (x1 , x2 , x3 , · · · ) = f (ey1 , ey2 , ey3 , · · · )
= c · e a 1 y1 · e a 2 y 2 · e a 3 y3 · · ·
= exp aT y + b
which is now an exponential of affine function. Similarly, a posynomial will be converted into a sum
of exponentials of affine functions. Now, taking log of the objective and the constraints. The posynomials
turn into log-sum-exp ”which are convex), the monomials will be become affine. Thus, this is our regular
convex problem now.
4.7
Generalized Inequality Constraints
4.7.1 Conic Form Problems
Conic form problem is a generalization of LP, replacing componentwise inequality with generalized linear
inequality with a cone K.
minimize cT x
subject to F x + g K 0
Ax = b
17
The SOCP can be expressed as a conic form problem if we set Ki to be a second-order cone in Rni +1 :
minimize cT x
subject to − Ai x + bi , cTi x + di Ki 0
(i = 1, · · · , m)
Fx = g
from which the name of SOCP comes.
4.7.2 SDP: Semidefinite Programming
A special form of conic program, where K is Sn+ , which is the set of positive semidefinite matrices, is called
a SDP. It has the form:
minimize cT x
subject to x1 F1 + x2 F2 + · · · + xn Fn 0
Ax = b
4.8
Vector Optimization
We can generalize the regular convex optimization by letting the objective function take vector values;
we can now use proper cones and generalized inequalities to find the best vector value. These are called
vector optimization problems.
4.8.1 Optimal Values
When a point x∗ is better or equal to than every other point in the domain of the problem, x∗ is called the
optimal. In a vector optimization problem, if an optimal exists, it is unique. ”Why? Vector optimization
requires a proper cone; proper cones are pointed – they do not contain lines. However, if x1 and x2 are
both optimal, p = x1 − x2 and −p are both in the cone, making it improper.)
4.8.2 Pareto Optimality
In many problems we do not have a minimum value achievable, but a set of minimal values. They are
incomparable to each other. A point x ∈ D is pareto-optimal when for all y that f0 (y) K f0 (x) implies
f0 (y) = f0 (x). Note that there can be multiple values with the same minimal value. Note that every pareto
value has to lie on the boundary of the set of achievable values.
4.8.3 Scalarization for Pareto Optimality
A standard technique for finding pareto optimal points is to scalarize vector objectives by taking a weighted
sum. This can be explained in terms of dual generalized inequality. Pick any λ ∈ K∗ and solve the following
problem:
18
minimize λT f0 (x)
subject to fi (x) ≤ 0
hi (x) = 0
By what we discussed in 2.6.2, a pareto optimal point must be a minimizer of this objective for any
λ ∈ K∗.
Now what happens when the problem is convex? Each λ with λ ≻K∗ 0 will likely give us a different
pareto point. Note λ K∗ 0 might not give us such guarantee, some elements might not be pareto optimal.
4.8.4 Examples
• Regularized linear regression tries to minimize RMSE and norm of the coefficient at the same time so we optimize kAx − bk2 + λxT x. Changing λ lets us explore all pareto optimal points.
2
5 Duality
This chapter explores many important ideas. Duality is introduced and used as a tool to derive optimality
conditions. KKT conditions are explained.
5.1
The Lagrangian And The Dual
5.1.1 The Lagrangian
The Lagrangian L associated with a convex optimization problem
minimize f0 (x)
subject to fi (x) ≤ 0
(i = 1, · · · , m)
hi (x) = 0
(i = 1, · · · , p)
is a function taking x and the weights as input, and returning a weighted sum of the objective and
constraints:
L (x, λ, ν) = f0 (x) +
m
X
λi fi (x) +
i=1
p
X
νi hi (x)
i=1
So positive values of fi (x) are going to penalize the objective function. The weights are called dual
variables or Lagrangian multiplier vectors.
5.1.2 The Lagrangian Dual Function
The Lagrangian dual function takes λ and ν, and minimizes L over all possible x.
g (λ, ν) = inf L (x, λ, ν) = inf
x∈D
x∈D
f0 (x) +
m
X
i=1
19
λi fi (x) +
p
X
i=1
νi hi (x)
!
5.1.3 Lagrangian Dual and Lower Bounds
It is easy to see that for any elementwise positive λ, the Lagrangian dual function provides a lower bound
on the optimal value p∗ of the original problem. This is very easy to see; if x is any feasible point, the
dual function value is the sum of ”possibly suboptimal) value p; if xp is the feasible optimal point, we have
negative values of fi (i ≥ 1) and zeros for hi . Then,
g (λ, ν) = inf L (x, λ, ν) ≤ L (xp , λ, ν) ≤ f (xp ) = p∗
x∈D
5.1.4 Intuitions Behind Lagrangian Dual
An alternative way to express constraints is to introduce indicator functions in the objectives:
minimize f0 (x) +
m
X
I− (fi (x)) +
i=1
p
X
I0 (hi (x))
i=1
the indicator functions will have a value of 0 when the constraint is met, ∞ otherwise. Now, these
represent how much you are irritated by a violated constraint. We can replace them with a linear function
- just a different set of preferences. Instead of hard constraints, we are imposing soft constraints.
5.1.5 LP example and Finite Dual Conditions
A linear program’s lagrange dual function is
g (λ, ν) = inf L (x, λ, ν) = −bT ν + inf c + AT ν − λ
x
x
T
x
The dual value can be found analytically, since it is a affine function of x. Whenever any element of
c + AT ν − λ is nonzero, we can manipulate x to make the dual value −∞. So it is finite only on a line where
c + AT ν − λ = 0, which is a surprisingly common occurrence.
5.1.6 Conjugate Functions and Lagrange Dual
The two functions are closely related, and Lagrangian dual can be expressed in terms of the conjugate
function of the objective function, which makes duals easier to derive if the conjugate is readily known.
5.2
The Lagrange Dual Problem
There’s one more thing which is named Lagrangian: the dual problem. The dual problem is the optimization problem
maximize g (λ, ν)
subject to λ 0
A pair of (λ, ν) is called dual feasible if it is a feasible point of this problem. The solution of this problem,
(λ∗, ν∗) is called dual optimal or optimal Lagrange multipliers. The dual problem is always convex; whether
or not the primal problem is convex. Why? g is a pointwise infimum of affine functions of λ and ν.
Note the langrange dual for many problems were bounded only for a subset of the domain. We can
bake this restriction into the problem explicitly, as a constraint.
20
5.2.1 Duality Gap
Langrangian dual’s solution d∗ are related to the solution of the primal problem p∗, notably:
d∗ ≤ p∗
Regardless of the original problem being convex. When the inequality is not strict, this is called a weak
duality. The difference p ∗ −d∗ is called the duality gap. Duality can be used to provide lower bound of the
primal p roblem.
5.2.2 Strong Duality And Slater’s Constraint Qualification
When the duality gap is 0, we say strong duality holds for the problem. That means the lower bound
obtained from the dual equals to the optimal solution of the problem; therefore solving the dual is ”sort
of) same as solving the primal. For obvious reasons, strong duality is very desirable but it doesn’t hold in
general. But for convex problems, we usually ”but not always) have strong duality.
Given a convex problem, how do we know if strong duality holds? There are many qualifications, which
ensures strong duality if the qualifications are satisfied. The text discusses one such qualification; Slater’s
constraint qualification. The condition is quiet simple: if there exists x ∈ relintD such that all inequality
conditions are strictly held, we have strong duality. Put another way:
fi (x) < 0 (i = 0, 1, 2, · · · ) , Ax = b
Also, it is noted that affine inequality constraints are allowed to held weakly.
5.2.3 Examples
• Least-squares: since there are no infeasibility constraints, Slater’s condition just equals feasibility: so
as long as the primal problem is feasible, strong duality holds.
• QCQP: The Lagrangian is a quadratic form. When all λs are nonnegative, we have a positive semidefinite form and we can solve minimization over x analytically.
• Nonconvex example: Minimizing a nonconvex quadratic function over the unit ball has strong duality.
5.3
Geometric Interpretation
This section introduces some ways to think about Lagrange dual functions, which offer some intuition
about why Slater’s condition works, and why most convex problems have strong duality.
5.3.1 Strong Duality of Convex Problems
Let’s try to explain figure 5.3 to 5.5 from the book. Consider following set G:
G = {(f1 (x) , · · · , fm (x) , h1 (x) , · · · , hp (x) , f0 (x)) |x ∈ D} = {(u, v, t) |x ∈ D}
Note ui = fi (x), vi = gi (x) and t = f0 (x). Now the Lagrangian of this problem
21
L (λ, ν, x) =
X
λi ui +
X
ν i vi + t
can be interpreted as a hyperplane passing through x with normal vector (λ, ν, 1)1 and that hyperplane will
meet t-axis at the value of the Lagrangian. ”See figure 5.3 from the book.)
Now, the Lagrange dual function
g (λ, ν) = inf L (λ, ν, x)
x∈D
will find x in the border of D: intuitively, the Lagrangian can still be decreased by wiggling x if x ∈
relintD. Therefore, the value of the Lagrange dual function can now be interpreted as a supporting hyperplane with normal vector (λ, ν, 1).
Next, we solve the Lagrange dual problem which maximizes the position where the hyperplane hits the
t-axis. Can we hit p∗, the optimal value? When G is convex, the feasible portion of G ”i.e. u 0 and ν = 0)
is convex again, and we can find a supporting hyperplane that meets G at the optimal point! But when G
is not, p∗ can hide in a nook inside G and the supporting hyperplane might not meet p∗ at all.
5.4
Optimality Conditions
5.4.1 Certificate of Suboptimality and Stopping Criteria
We know, without assuming strong duality,
g (λ, ν) ≤ p⋆ ≤ f0 (x)
Now, f0 (x) − g (λ, ν) gives a upper bound on f0 (x) − p⋆, the quantity which shows how suboptimal x is.
This gives us a stopping criteria for iterative algorithms; when f0 (x) − g (λ, ν) ≤ ǫ, it is a certificate that x is
less than ǫ suboptimal. The quantity will never drop below the duality gap, so if you want this to work for
arbitrarily small ǫ we would need strong duality.
5.4.2 Complementary Slackness
Suppose the primal and dual optimal values are attained and equal. Then,
f0 (x⋆ ) = g (λ⋆ , ν ⋆ )
X
X
= inf f0 (x) +
λi fi (x) +
νi hi (x)
x
X
X
≤ f0 (x⋆ ) +
λi fi (x⋆ ) +
νi hi (x⋆ )
≤ f0 (x⋆ )
”assumed 0 duality gap)
”definition of Lagrangian dual function)
”taking infimum is less than or equal to any x)
”λ⋆i are nonnegative, fi values are nonpositive and hi values are 0)
So all inequalities can be replaced by equalities! In particular, it means two things. First, x⋆ minimizes
the Lagrangian. Next,
X
λ⋆i fi (x⋆ ) = 0
Since each term in this sum is nonpositive, we can conclude all terms are 0: so for all i ∈ [1, m] we have:
1 Yep,
there are some notation abusing here since λ and ν themselves are vectors.
22

 λ⋆ = 0
i
either
f (x⋆ ) = 0
i
This condition is called complementary slackness.
5.4.3 KKT Optimality Conditions
KKT is a set of conditions for a tuple (x⋆ , λ⋆ , ν ⋆ ) which are primal and dual feasible. It is a necessary condition for x∗ and (λ∗, ν∗) being optimal points for their respective problems with zero duality gap. That is,
all optimal points must satisfy these conditions.
KKT condition is:
• x⋆ is prime feasible: fi (x⋆ ) ≤ 0 for all i, hi (x⋆ ) = 0 for all i.
• (λ⋆ , ν ⋆ ) is dual feasible: λ⋆i ≥ 0
• Complementary slackness: λ⋆i fi (x⋆i ) = 0
• Gradient of Lagrangian disappears: ∇f0 (x⋆ ) +
P
λ⋆i ∇fi (x⋆ ) +
P
νi⋆ ∇hi (x⋆ ) = 0
Note the last condition is something we didn’t see before. It makes intuitive sense though - the optimal point
for the dual problem must minimize the Lagrangian. Since the primal problem is convex, the Lagrangian
is convex - and the only point with 0 gradient is the minimum.
KKT and Convex Problems
When the primal problem is convex, KKT condition is necessary-sufficient for optimality. This has immense importance. We can frame solving convex optimization problems by solving KKT conditions. Sometimes KKT condition might be solvable analytically, giving us closed form solution for the optimization
problem.
The text also mentions that when Slater’s condition is satisfied for a convex problem, we can say that
arbitrary x is primal optimal iff there are (λ, ν) that satisfies KKT along with x. I actually am not sure about
why Slater’s condition is needed for this claim but the lecture doesn’t make a big deal out of it, so meh..
5.5
Solving The Primal Problem via The Dual
When we have strong duality and dual problem is easier to solve”due to some exploitable structure or
analytical solution), one might solve the dual first to find the dual optimal point (λ⋆ , ν ⋆ ) and find the x
that minimizes the Lagrangian. If this x is feasible, we have a solution! Otherwise, what do we do? If
the Lagrangian is strictly convex, then there is a unique minimum: if this minimum is infeasible, then we
conclude the primal optimal is unattainable.
5.6
Sensitivity Analysis
The Lagrange multipliers for the dual problem can be used to infer the sensitivity of the optimal value
with respect to perturbations of the constraints. What kind of perterbations? We can tighten or relax
constraints for an arbitrary optimization problem, by changing the constraints to:
23

f (x) ≤ u
i
i
h (x) = v
i
(i = 1, 2, · · · , m)
(i = 1, 2, · · · , p)
i
Letting ui > 0 means we have more freedom regarding the value of fi ; ui < 0 otherwise.
5.6.1 Global Sensitivity
The Lagrange multipliers will give you information about how the optimal value will change when we do
this. Let’s denote the optimal value of the perturbed problem as a function of u and v: p⋆ (u, v). We have a
lower bound for this value when strong duality holds:
f0 (x) ≥ p⋆ (0, 0) − λ⋆T u − ν ⋆T v
which can be obtained by manipulating the definitions.
Using this lower bound, we can make some inferences about how f0 (x) will change with respect to u and
v. Basically, when lower bound increases greatly, we make an inference the optimal value will increase
greatly. However, when lower bound decreases, we don’t have such an assurance2 . Examples:
• When λ⋆i is large, and we tighten the constraint (ui < 0), this will increase the lower bound a lot; the
optimal value will increase greatly.
• When λ⋆i is is small, and we loosen the constraint (ui > 0), this will decrease the lower bound a bit,
but this might not decrease the optimal value a lot.
5.6.2 Local Sensitivity
The text shows an interesting identity:
λ⋆i = −
∂p⋆ (0, 0)
∂ui
Now, λ⋆i gives you the slope of the optimal value with respect to the particular constraint. All these, along
with complementary slackness, can be used to interpret Lagrange multipliers; they tell you how tight a
given inequality constraint is. Suppose we found that λ⋆1 = 0.1, and λ⋆2 = 100 after solving a problem. By
complementary slackness, we know f1 (x⋆ ) = f2 (x⋆ ) = 0 and they are both tight. However, when we do
decrease u2 , we know p⋆ will move much more abruptly, because of the slope interpretation above. On
the other hand, what happens when we increase u2 ? Locally we know p⋆ will start to decrease fast; but it
doesn’t tell us how it will behave when we keep increasing u2 .
5.7
Examples and Reformulating Problems
As different formulations can change convex problems to non-convex and vice versa, dual problems are
affected by how the problem is exactly formulated. Because of this, a problem that looks unnecessarily
complicated might end up to be a better representation. The text gives some examples of this.
2 Actually I’m a bit curious regarding this as well - lower bound increasing might not increase the optimal value when optimal
value was well above the lower bound to begin with.
24
5.7.1 Introducing variables
5.7.2 Making explicit constraints implicit
5.7.3 Transforming the objective
5.8
Generalized Inequailities
How does the idea of Lagrangian dual extend to problems with vector inequalities? Well, it generalizes
pretty well - we can define everything pretty similarly. Except how the nonnegative restriction for λ becomes a nonnegative restriction with dual cone. Here are some intuition behind this difference. Say we
have the following problem:
minimize
f0 (x)
s.t.
fi (x) Ki 0
hi (x) = 0
(i = 1, 2, · · · , m)
(i = 1, 2, · · · , p)
Now the Lagrangian multiplier λ is vector valued. The Lagrangian becomes:
f0 (x) +
X
λTi fi (x) +
X
νi hi (x)
fi (x) is nonpositive with respect to Ki means that −fi (x) ∈ Ki . Remember we want each product
(x) to be nonpositive - otherwise this dual won’t be a lower bound anymore. Now we try to find the
set of λ that makes λT y negative for all −y ∈ K. We will need to make λT y positive for all y ∈ K. What is
λTi fi
this set? The dual cone.
The dual of SDP is also given as an example. The actual derivation involves more linear algebra than I
am comfortable with ”shameful) so I’m skipping things here.
6 Approximation and Fitting
With this chapter begins part II of the book on applications of convex optimization. Hopefully, I will be
less confused/frustrated by materials in this part. :-)
6.1
Norm Approximation
This section discusses various forms of the linear approximation problem:
minx |Ax − b|
with different norms and constraints. Without doubt, this is one of the most important optimization
problems.
6.1.1 Examples
• ℓ2 norm: we get least squares.
• ℓ∞ norm: Chebyshev approximation problem. Reduces to an LP which is as easy as least squares, but
no one discusses it!
25
• ℓ1 norm: Sum of absolute residuals norm. Also reduces to LP, extremely interesting, as we will discuss
further in this chapter.
6.1.2 Different Penalty Functions and Their Consequences
The shape of the norm used in the approximation affects the results tremendously. The most common
norms are ℓp norms - given a residual vector r,
X
i
|ri |
p
!1/p
We can ignore the powering by 1/p and just minimize the base of the exponentiation. Now we can think
of ℓp norms giving separate penalties to each of the residual vector. Note that most norms do the same - so
we can think in terms of a penalty function φ (r) when we think about norms.
The text examines a few notable penalty functions:
• Linear: sum of absolute values; associated with ℓ1 norm.
• Quadratic: sum of squared errors; associated with ℓ2 norm.
• Deadzone-linear: zero penalty for small enough residuals; grows linearly after the barrier.
• Log-barrier: grows infinitely as we get near the preset barrier.
Now how does these affect our solution? The penalty function measures our level of irritation with regard
to the residual. When φ (r) grows rapidly as r becomes large, we are immensly irritated. When φ (r) shrinks
rapidly as r becomes small, we don’t care as much.
This simple description actually explains the stark difference between ℓ1 and ℓ2 norms. With a ℓ1 norm,
the slope of the penalty does not change when the residual gets smaller. Therefore, we still have enough
urge to shrink the residual until it becomes 0. On the other hand, with a ℓ2 norm the penalty will quickly
get smaller when the residual gets smaller than 1. Now, once we go below 1, we do not have as much
motivation to shrink it further - the penalty does not decrease as much. What happens when the residual
is large? Then, ℓ1 is actually less irritated by ℓ2 ; the penalty grows much more rapidly.
These explanations let us predict how the residuals from both penalty functions will be distributed. ℓ1
will give us a lot of zeros, and a handful of very large residuals. ℓ2 will only have a very small number of
large residuals; and it won’t have as many zeros - many residuals will be near zero, but not exactly. The
figures in the text confirms this theory. This actually was one of the most valuable intuitions I got out of
this course. Awesome.
Another little gem discussed in the lecture is that, contrary to the classic approach to fitting problems,
the actual algorithms that find the x are not your tools anymore - they are standard now. The penalty
function is your tool - you shape your problem to fit your actual needs. This is a very interesting, and at
the same time very powerful perspective!
6.1.3 Outliers and Robustness
Different penalty functions behave differently when outliers are present. As we can guess, quadratic loss
functions are affected much worse than linear losses. When a penalty function is not sensitive to outliers,
it is called robust. Linear loss function is an obvious example of this. The text introduces another robust
penalty function, which is the Huber penalty function. It is a hybrid between quadratic and linear losses.
26
φhub (u) =

 u2
(|u| < M )
M (2 |u| − M )
otherwise
Huber function grows linearly after the preset barrier. It is the closest thing to a constant-beyondbarrier loss function, without losing convexity. When all the residuals are small, we get the exact least
square results - but if there are large residuals, we don’t go nuts with it. It is said in the lecture that 80%
of all applications of linear regression could benefit from this. A bold, but very interesting claim.
6.1.4 Least-norm Problems
A closely related problem is least-norm problem which has the following form:
minimize |x|
subject to Ax = b
which obviously is meaningful only when Ax = b is underdetermined. This can be cast as a norm
approximation problem by noting that the solution space is given by a particular solution, and the null
space of A. Let Z consist of column vectors that are basis for N (A), and we minimize:
|x0 + Zu|
Two concrete examples are discussed in the lecture.
• If we use ℓ2 norm, we have a closed form solution using the KKT conditions.
• If we use ℓ1 norm, it can be modeled as an LP. This approach is in vogue, say in last 10 years or so. We
are now looking for a sparse x.
6.2
Regularized Approximations
Regularization is a practice of minimizing the norm of the coefficient |x|, as well as the norm of the residual.
It is a popular practice in multiple disciplines.
Why do it? The text introduces a few examples. First of all, it can be a way to express our prior knowledge or preference towards smaller coefficients. There might be cases where our model is not a good
approximation of reality when x gets larger.
Personally, this made the most sense to me; it can be a way of taking variations/errors of the matrix A
into account. For example, say we assume an error ∆ in our matrix A. So we are minimizing (A + ∆) x−b =
Ax − b + ∆x; the error is multiplied by x! We don’t want a large x.
6.2.1 Bi-criterion Formulation
Regularization can be cast as a bi-criterion problem, as we have two objectives to minimize. We can trace
the optimal trade-off curve between the two objectives. On one end, where |x| = 0, we have Ax = 0 and the
residual norm is |b|. At the other end, there can be multiple Pareto-optimal points which minimize |Ax − b|.
”When both norms are ℓ2 , it is unique)
27
6.2.2 Regularization
The actual practice of regularization is more concrete than merely trying to minimize the two objectives;
it is a scalarization method. We minimize
|Ax − b| + γ |x|
where γ is a problem parameter. ”Which, in practice, is typically set by cross validation or manual intervention) Practically, γ is the knobs we turn to solve the problem.
Another common practice is taking the weighted sum of squared norms:
2
|Ax − b| + δ |x|
2
Note it is not obvious that the two problems sweep out the same tradeoff curve. ”They do, and you can
find the mapping between γ and δ given a specific problem).
The most prominent scheme is Tikhonov regularization/ridge regression. We minimize:
2
2
|Ax − b|2 + δ |x|2
which even has an analytic solution.
The text also mentions a smoothing regularization scheme - the penalty is on Dx instead of x. D can
change depending on your criteria of fitness of solutions. For example, if we want x to be smooth, we can
roughly penalize its second derivative by setting D as the Toeplitz matrix:

1 −2 1

0 1 −2
D=
0 0
1

..
..
..
.
.
.
0
0
−2
..
.
0 0
0 0
1
..
.
0
..
.

···

· · ·

· · ·

..
.
So that the elements of Dx are approximately the second derivatives (2xi − xi−1 − xi+1 ).
6.2.3
ℓ1 Regularization
ℓ1 regularization is introduced as a heuristic for finding sparse solutions. We minimize:
|Ax − b|2 + γ |x|1
The optimal tradeoff curve here can be an approximation of the optimal tradeoff curve between |Ax − b|2
and the cardinality cardx, which is the number of nonzero elements of x. ℓ1 regularization can be solved
as a SOCP problem.
6.2.4 Signal Reconstruction Problem
An important class of problem is introduced: signal reconstruction. There is an underlying signal x which
is observed with some noise; resulting in corrupted observation. x is assumed to be smooth. What is the
most plausible guess for the time series x?
This can be cast as a bicriterion problem: first we want to minimize |ˆ
x − xcor |2 where x
ˆ is our guess
and xcor is the corrupted observation. On the other hand, we think smooth x
ˆ are more likely, so we minimize a penalization function: φ (ˆ
x). Different penalization schemes are introduced, quadratic smoothing
28
and total variance smoothing. In short, they are ℓ2 and ℓ1 penalizers, respectively. When underlying process has some jumps, as you can expect, total variance smoothing preserves those jumps, while quadratic
smoothing tries to smooth out the transition.
Some more insights are shared in the lecture videos. Recall ℓ1 regularization gives you a small number
of nonzero regularized terms. So if you are penalizing
φ (ˆ
x) =
X
|ˆ
xi+1 − x
ˆi |
the first derivative is going to be sparse. What does the resulting function look like? Piecewise constant.
Similarly, say we take the approximate second derivative |2ˆ
xi − x
ˆi−1 − x
ˆi+1 |? We get piecewise linear! The
theme goes on - if we take the third difference, we get piecewise quadratic ”actually, splines).
6.3
Robust Approximation
How do we solve approximation when A is noisy? Let’s say,
A = A¯ + U
where A¯ represents the componentwise mean, and U represents the random component with zero
mean. How do we handle this? The prevalent method is to ignore that A has possible errors. It is okay,
as long as you do a posterior analysis on the method: try changing A by a small amount and try the
approximation again, see how it changes.
6.3.1 Stochastic Formulation
A reasonable formulation for an approximation problem is to minimize:
minimize E kAx − bk
This is untractable in general, but tractable in some special cases, including when we minimize the ℓ2
norm:
minimize E kAx − bk2
2
Then:
2
¯ − b + U x T Ax
¯ − b + Ux
E kAx − bk2 = E Ax
¯ − b + ExT U T U x
¯ − b T Ax
= Ax
¯ − b 2 + x T P x
= Ax
2
2
2 1/2 ¯ − b + = Ax
x
P
2
2
Tada, we got Tikhonov regularization! This makes perfect sense - increasing magnitudes of x will increase the variation of Ax, which in turn increase the average value of kAx − bk by Jensen’s inequality. So
¯ − b small with making the variance small. This is a nice interpretation
we are trying to balance making Ax
for Tikhonov regularization as well.
29
6.3.2 Worst-case Robust Approximation
Instead of taking expected value of the error, we can try to minimize the supremum of error across a set A
consisting possible values of A. The text describes several types of A we can use to come up with explicit
solutions. The following are those examples:
• When A is a finite set
• When A is mean plus U , where U is an error in a norm ball.
• When each row of A is mean plus Pi , where Pi describes an ellipsoid of possible values.
• more examples..
Worst-case robust least squares is mentioned in the lecture. This is not a convex problem, but it can be
solved exactly. In fact, any optimization problem with two quadratic functions can be solved exactly ”see
appendix of the book).
6.4
Function Fitting
In a function fitting problem, we try to approximate an unknown function by a linear combination of basis
functions. We determine the coefficient vector x which yields the following function:
f (u) =
X
xi fi (u)
where fi () is the i-th basis function. A typical basis function are powers of u: the possible set of f is the
set of polynomials. You can use piecewise linear and polynomial functions; using piecewise polynomial
will give you spline functions.
6.4.1 Constraints
We can impose various constraints on the function being fitted. The text introduces some tractable set of
constraints.
P
• Function value interpolation: the function value at a given point f (v) =
xi fi (v) is a linear function
of x. Therefore, equality constraints and inequality constraints are actually linear constraints.
• Derivative constraints: the derivative value at a given point f (v) =
of x.
P
xi ∇fi (v) is also a linear function
6.4.2 Sparse Descriptions and Basis Pursuit
In basis pursuit problems, we want to find a sparse f out of a very large number of basis functions. By
a sparse f , we mean there are a few nonzero entries in the coefficient vector x. Mathematically, this is
equivalent to the regressor selection problem ”quite unsurprisingly), so the similar set of heuristics can be
used. First, we can use ℓ1 regularization to approximate optimizing for cardx.
30
6.4.3 Checking Model Consistency
The text introduces an interesting problem - given a set of data points, is there a convex function that
satisfies all those data? Fortunately, recall the first order convexity condition from 3.1.2 - using this, ensuring the convexity of a function is as easy as finding the gradients at the data points so that the first order
condition is satisfied. We want to find g1 , · · · , gm so that:
yj ≥ yi + giT (uj − ui )
for any pair of i and j.
Fitting Convex Function To The Data
We can fit a convex function to the data by finding fitted values of y, and ensuring the above condition
holds for the fitted values. Formally, solve:
minimize (yi − yˆi )
2
subject toˆ
yj ≥ yˆi + giT (uj − ui ) for any pair of i, j
This is a regular QP. Note the result of this problem is not a functional representation of the fitted
function, as in regular regression problems. Rather, we get the value of the function - so it’s a point-value
representation.
Bounding Values
Say we want to find out if a new data point is irregular or not - is it consistent with what we saw earlier?
In other words, given a new unew , what is the range of values possible given the previous data? We can
minimize/maximize for yˆnew subject to the first order constraint, to find the range. These problems are LP.
7 Statistical Estimation
I was stuck in this chapter for too long. It’s time to finish this chapter no matter what. This chapter shows
some example applications of convex optimization in statistical settings.
7.1
Parametric Distribution Estimation
The first example is MLE fitting - the most obvious, but the most useful. We of course require the constraints
on x to be convex optimization friendly. A linear model with IID noise is discussed:
yi = aTi x + vi
The MLE is of course
xml = argmaxx l (x) = argmaxx log px (y)
px (y) depends on the distribution of vi s. Different assumptions on this distribution leads to different
fitting methods:
• Gaussian noise gives you OLS
31
• Laplacian noise gives you ℓ1 regularized regression ”of course, Laplacian distribution has a sharp
peak at 0, which equates to having high incentive to reduce residual when residual is really small)
• Uniform noise
Also note that we need to have log px (y) to be concave in x, not y: and exponential families of distribution
meet this criteria. Also, in many cases, your natural choice of parameters might not yield a log likelihood
function that is concave. Usually with a change of variables, we achieve this.
Also, we discuss that these distributions are equivalent to different penalty schemes - as demonstrated
by the equivalence of L2 with Gaussian, L1 with Laplacian. There are 1:1 correspondence. If you have a
penalty function p (v), the corresponding distribution is ep(v) normalized!
7.1.1 Logistic Regression Example
exp aT u+b
We model p = S a u + b = 1+exp(aT u+b) where u are the explanatory variables, a and b are model parameters. Say we have n = q + m examples, the first q of them having yi = 1 and next m of them having yi = 0.
T
Then, the likelihood function has:
q
Y
i=1
pi
n
Y
i=q+1
(1 − pi )
Take log and plug in above equation for p and we get the following concave function:
q
X
i=1
n
X
log 1 + exp aT ui + b
aT ui + b −
i=q+1
7.1.2 MAP Estimation
MAP is the Bayes equivalent of MLE. The underlying philosophy is vastly different, but the optimization
technicality remains more or less the same, except a term that describes the prior distribution.
7.2
Nonparameteric Distribution Estimation
A nonparameteric distribution is one which we don’t have any closed formula for. So we will estimate a
vector p where
prob (x = αk ) = pk
which lies in Rn .
7.2.1 Priors
• Expected value of any function are just linear equality in terms of p, so we can express them easily.
• The variance of the random variable is a concave function of p. Therefore, a lower bound on the
variance can be expressed within the convex setting.
• The entropy of X is a concave, so we can express the lower bound as well.
• The KL-divergence between p and q is convex. So we can impose upper bound here.
32
7.2.2 Objectives
• We can minimize/maximize expected values because they are affine to p.
• We can find MLE because log likelihood for p in this setting is always concave.
• We can find maximum entropy.
• We can find minimum KL-divergence between p and q.
7.3
Optimal Detector Design And Hypothesis Testing
I will only cover this section briefly.
Problem setup: the parameter θ can take m values. For each value of θ, we have a nonparameteric
distribution over n possible values α1 · · · αn . The probabilities can be represented by a matrix P = Rn×m .
We call each θ a different hypothesis. We want to find which θ generated given sample. So the detector we
want to design is a function from a sample to θ.
We can create either deterministic or probabilistic detectors; like in game theory, introducing extra
randomness can improve the detector in many ways. For a simple and convincing example; say we have
a binary problem. Draw an ROC curve which shows the tradeoff between false positive and false negative
errors. A deterministic detector might not be able to hit the sweet spot where pf n = pf p depending on the
θs - but probabilistic detectors can.
7.4
Chebyshev and Chernoff Bounds
7.4.1 Chebyshev Bounds
Chebyshev bounds give an upper bound on a probability of a set based on known quantities; many inequalities follow this form. For example, Markov’s inequality says: If X ∈ R+ has EX = µ then we have
prob (X ≥ 1) ≤ µ. ”Of course, this inequality is completely useless when µ > 1 but that’s how all these
inequalities are.) This section looks at cases where we can find such bounds using convex optimization.
In this setup, our prior knowledge is represented as a pair of functions and their expected values. The
set whose probability we want to find bounds for is given as C. We want something like:
prob (X ∈ C) ≤ Ef (X)
for some function f whose expectation we can take.
The recipe is to concoct an f which is a linear combination of the prior knowledges. Then Ef (X) is
simply a linear combination of the expectations. How do we ensure the EV is above prob (X ∈ C)? We
can impose that f (x) ≥ 1C (x) pointwise, where 1C is an indicator function for C. We can now state the
following problem:
minimize
X
ai xi =
Efi (X) xi = Ef (X)
X
subject tof (z) =
xi fi (z) ≥ 1
if z ∈ C
X
f (z) =
xi fi (z) ≥ 0
if z ∈ S\C
X
This is a convex optimization problem, since the constraints are convex. For example, the first constraint
can be recast as
33
g1 (x) = 1 − inf f (z) < 0
z∈C
which is surely convex. There is another formulation where we solve a case where the first two moments are specified; but I am omitting it.
7.4.2 Chernoff Bounds
This section deals with Chernoff bounds, which has a different form, but the same concept.
7.5
Experiment Design
We discuss various solutions to the experiment design problem as an application. The setup is as follows.
We have a fixed menu of p different experiments which is represented by ai (1 ≤ i ≤ p). We will perform
m experiments, each experiment taken from the menu. For each experiment, we get yi as the result which
is
yi = aTi x + wi
where wi are independent unit Gaussian noise. The maximum likelihood estimate is of course given by
least squares. Then, the associated error e = x
ˆ − x has zero mean and has covariance matrix E:
E = EeeT =
X
ai aTi
How do we minimize E? What kind of metrics do we use?
−1
7.5.1 Further Modeling
First, this is an offline problem and we don’t actually care about the order we perform. So the only thing
we care is that for each experiment on the menu, how many times do we perform it. So the optimization variables are a list of nonnegative integers mi which sum up to m. Of course, the above problem is
combinatorially hard and we relax it a bit, by modeling what fraction of m do we run each experiments.
Still, the objective E is a vector ”actually a matrix) so we need some scalarization scheme to minimize
it. The text discusses some strategies including:
• D-optimial design: minimize determinant of E. Since determinant is the volume of the box, we are
in effect minimizing the volume of the confidence ellipsoid.
• E-optimal design: we minimize the largest eigenvalue of E. Rationale: the diameter of the confidence
ellipsoid is proportional to norm of the matrix.
• A-optimal design: we minimize the trace. This is, effectively, minimizing the error squared.
8 Geometric Problems
8.1
Point-to-Set Distance
A project of the point x0 to a closed set C is defined as the closest point in C that minimizes the distance
from x0 . When C is closed and convex, and the norm is strictly convex ”e.g. Euclidean), we can prove the
34
projection is unique.
When the set C is convex, finding the projection is a convex optimization problem. Some examples are
discussed - planes, halfplanes, and a proper cone.
Finding the separating hyperplane between a point and a convex set is discussed as well. When we use
Euclidean norm, we have a geometric, intuitive way to find one - take x0 and its projection p (x0 ), and use
~ 0 ) − ~x and passes the mean of two points. However, for other norms,
the hyperplane which is normal to p (x
we have to construct such hyperplane using dual problem; if we find a particular Lagrangian multiplier
for which the dual problem is feasible, we know that multiplier constitutes a separating hyperplane.
8.1.1 PCA Example
Suppose the set C of m × n matrices with at most k rank. A projection of X0 onto C which minimizes the
Euclidean norm is achieved by a truncated SVD - yes PCA!
8.2
Distance between Sets
Distance between two convex sets is an convex optimization problem, of course. The dual of this problem
can be interpreted as a problem finding a separating hyperplane between the two sets. The argument can
be made: if strong duality holds, a positive distance implies an existence of a separating hyperplane.
8.3
Euclidean Distance and Angle Problems
This section deals with problems where Euclidean distances and angles between vectors are constrained.
Setup: n vectors in Rn , for which we assume their Euclidean lengths are known: li = kai k2 .
Distance and angular constraints can be cast as a constraint on G, which is the Gram matrix of A which
has ai as column vectors:
G = AT A
G will be our optimization variable; after the optimization we can back out the interested vectors by
Cholesky factorization. This is a SDP since G is always positive semidefinite.
8.3.1 Expressing Constraints in Terms of G
• Diagonal entries will give length squared: Gii = li2
• The distance between vector i and j dij can be written as:
kai − aj k2 = li2 + lj2 − 2aTi aj
which means Gij is an affine function of d2ij : Gij =
1/2
= li2 + lj2 − 2Gij
li2 +lj2 −d2ij
2
1/2
This means range constraints on d2ij can be a pair of linear constraints on Gij .
• Gij is an affine function of the correlation coefficient ρij .
• Gij is also an affine function of cosine of the angle between two vectors: cos α. Since cos−1 is monotonic, we can use this to constrain the range of α.
35
8.3.2 Well-Condition Constraints
The condition number of A, σ1 /σn , is a quasiconvex function of G. So we can impose a maximum value or
try to minimize it using quasiconvex optimization.
Two additional approaches to well-conditionness are discussed - dual basis and maximizing log det G .
8.3.3 Examples
• When we only care about angles between vectors ”or correlations) we can set li = 1 for all i.
• When we only care about distance between vectors, we can assume that the mean of the vectors are 0.
This can be solved using the squared lengths as the optimization variable. Since Gij = li2 + lj2 − 2d2ij /2,
we get:
G = z1T + 1z T − D /2
which should be PSD ”zi = li2 ).
8.4
Extremal Volume Ellipsoids
This section deals with problems which approximates given sets with ellipsoids.
8.4.1 Lowner-John Ellipsoid
The LJ ellipsoid ǫlj for a set C is defined as the minimum-volume ellipsoid that contains C. This can be
cast as a convex optimization problem, however is only tractable when C is tractable. ”Of course, C has an
infinite number of points or whatever, it’s not going to be tractable..) We set our optimization variable A
and b such that:
εlj = {v| kAv + bk2 ≤ 1}
The volume of the LJ ellipsoid is proportional to det A−1 , so that’s what we optimize for. We minimize:
log det A−1
subject to supv∈C kAv + bk2 ≤ 1. As a trivial example, consider when C is a finite set of size m; then the
constraints translate into m convex constraints on A and b.
A notable feature of LJ ellipsoid is that its efficiency can be bounded; if you shrink an LJ ellipsoid by a
factor or n ”the dimension), it is guaranteed to fit inside C ”of course, when C is bounded and has nonempty
√
interior). So roughly we have a factor of n approximation. ”Argh.. the proof is tricky. Uses modified
problem’s KKT conditions.)
√
When the set is symmetric about a point x0 , the factor 1/n can be improved to 1/ n.
8.4.2 Maximum Volume Inscribed Ellipsoid
A related problem tries to find the maximum volume ellipsoid which lies inside a bounded, convex set C
with nonempty interrior. We use a different formulation of the ellipsoid now; it’s a forward projection of
a unit ball.
36
ε = {Bu + d| kuk2 ≤ 1}
Now its volume is proportional to det B. The constraint would be:
sup IC (Bu + d) ≤ 0
kuk2 ≤1
Max Ellipsoid Inside A Polyhedron A polyhedron is described by a set of m linear inequalities:
C = x|aTi x ≤ bi
We can now optimize regarding B and d. We can translate the constraint as:
sup IC (Bu + d) ≤ 0 ⇐⇒
kuk2 ≤1
which is a convex constraint on B and d.
sup aTi (Bu + d) ≤ bi
kuk2 ≤1
⇐⇒ BaTi 2 + aTi d ≤ bi
8.4.3 Affine Invariance
If T is an invertible matrix, it is stated that the transformed LJ ellipsoid will still cover the set C after
transformation. It holds for maximum volume inscribed ellipsoid as well.
8.5
Centering
8.5.1 Chebychev Center
Given a bounded, nonempty-interior set C ∈ Rn , a Chebychev centering problem finds a point where the
depth is maximized, which is defined as:
depth (x, C) = dist (x, Rn \C)
So it’s a point which is farthest from the exterior of C. This is not always tractable; suppose C is defined
as a set of convex inequalities fi (x) ≤ 0. Then, Chebychev center could be found by solving:
maximize
subject to
R
gi (x, R) ≤ 0 (i = 0, 1, · · · )
where gi is a pointwise maximum of fi (x + Ru) where kuk2 ≤ 1. Since fi is convex and x + Ru is affine
in x and R, gi is a convex function. However, it’s hard to evaluate gi ; since we have to find a pointwise
maximum of convex functions. Therefore, Chebychev center problem is feasible only for specific classes
of C. For example, when C is a polyhedron. ”An LP can solve this case)
8.5.2 Maximum Volume Ellipsoid Center
A generalization of Chebychev center is MVE center; the center of the maximum volume inscribed ellipsoid.
When the MVE problem is solvable, MVE center is trivially attained.
37
8.5.3 Analytic Center
An analytic center works with the logarithmic barrier − log x. If C is defined as
fi (x) ≤ 0
for all i, the analytic center minimizes
−
X
log (−fi (x))
i
This makes sense; when x is feasible, the absolute value of fi (x) kind of denotes the margin between x
and infeasible regions. Analytic center tries to maximize the product of those margins. The analytic center
is not invariant under different representations of the same set C, obviously.
8.6
Classification
This section deals with two sets of data {x1 , x2 , · · · , xN } and {y1 , y2 , · · · , yM }. We want to find a function
f (x) such that f (xi ) > 0 and f (yi ) < 0.
8.6.1 Linear Discrimination
Linear discrimination finds an affine function f (x) = aT x − b which satisfies the above requirements.
Since these requirements are homogeneous in a and b, we can scale them arbitrarily so the following are
satisfied:

 aT x − b ≥ 1
i
aT y − b ≤ −1
i
8.6.2 Robust Linear Discrimination
If two sets can be linearly discriminated, there will always be multiple functions that separate them. One
way to choose among them is to maximize the minimum distance from the line to each sample; in other
words, maximum margin or the thickest slab . This leads to the following problem:
maximize t
subject to aT xi − b ≥ t
aT yi − b ≤ −t
kak2 ≤ 1
Note the last requirement; we normalize a, since we will be able to arbitrarily increase t unless we normalize a.
Support Vector Classifier When two sets cannot be linearly separated, we can relax the constraints
f (xi ) > 1 and f (yi ) < −1 by rewriting them as:
f (xi ) > 1 − ui , and f (yi ) < −1 + vi
38
where ui and vi are nonnegative. Those numbers can be interpreted as a measure of how much each
constraint is violated. We can try to make these sparse by optimizing for the sum; this is an ℓ1 norm and u
and v will ”hopefully) be sparse.
Support Vector Machine The above are two approaches for robust linear discrimination. First tries to
maximize the width of the slab. Second tries to minimize the number of misclassified points ”actually, it
optimizes its proxy). We can consider the trade-off between the two. Note the width of the slab
z| − 1 ≤ aT z − b ≤ 1
Can be calculated by the distance between the two hyperplanes aT z = b − 1 and aT z = b + 1. Let
aT z1 = b − 1 and aT z2 = b + 1. aT (z1 − z2 ) = 2. It follows kz1 − z2 k2 = 2/ kak2 . Now we can solve the
following multicriterion optimization problem:
minimize kak2 + γ 1T u + 1T v
subject to aT xi − b ≥ 1 − ui
aT yi − b ≤ −1 + vi
u 0, v 0
We have SVM!
Logistic Regression
Another way to do approximate linear discrimination is logistic regression. This should be very familiar
now; the negative log likelihood function is convex.
8.6.3 Nonlinear Discrimination
We can create nonlinear separation space by introducing quadratic and polynomial features. For polynomial discrimination, we can do a bisection on the degree to find the smallest polynomial that can separate
the input.
8.7
Placement and Location
Placement problem deals with n points in Rk , where some locations are given, and the rest of the problems
are the optimization variables. The treatment is rather basic. In essense,
• You can minimize the sum of distances between connected nodes when the distance metric is convex.
• You can place upper bounds on distance between pairs of points, or lengths of certain paths.
• When the underlying connectivity represents a DAG, you can also minimize the max distance from a
source node to a sink node using a DP-like argument.
8.8
Floor Planning
A floor planning tries to place a number of axis-aligned rectangles without overlaps, optimizing for the size
of the bounding rectangle. This is a hard combinatorial optimization problem in general, but specifying
39
relative positioning of the boxes can make these problems convex. A relative positioning constraint gives
how individual pairs of rectangles are positioned. For example, rectangle i must be either above, below,
left to, right to rectangle j. These can be cast as linear inequalities. For example, we can specify that
rectangle i is left to rectangle j by specifying:
xi + w i ≤ xj
Some other constraints we can use:
• Minimum area for each rectangle
• Aspect ratio constraints are simple linear ”in)equalities.
• Alignment constraints: for example, two rectangles are centered at the same line
• Symmetry constraints
• Distance constraints: given relative positioning constraints, ℓ1 or ℓ∞ constraints can be cast pretty
easily.
Optimizing for the area of the bounding box gives you a geometric programming problem.
9 Numerical Linear Algebra Background
10 Unconstrained Optimization
Welcome to part 3! For the rest of the material I plan to skip over the theoretical parts, only covering the
motivation and rationale of the algorithms.
10.1 Unconstrained Minimization Problems
An unconstrained minimization doesn’t have any constraints but a function f () we need to minimize. We
assume f to be convex and differentiable, so the optimality can be checked by looking at the gradient ∇f .
This can sometimes be solved analytically ”for example, least squares), but in general we need to resort to
an iterative method ”for example, geometric programming or analytic center).
10.1.1 Strong Convexity
For most of this chapter we assume that f is strongly convex; which means that there exists m > 0 such
that
∇2 f (x) mI
for all symmetric x. This feels like an analogue of having a positive second order coefficient - is this different
from having posdef Hessian? ”Hopefully video lecture provides some insights)
Anyways, this is an extremely strong assumption, and we can’t, in general, expect our functions to be
strongly convex. Then why assume this? We are looking at theoretical convergence, which is already not
attainable ”because no algorithm is going to run infinitely). Professor says it’s more of a feel good stuff so let’s make assumptions that can shorten the proof.
40
Strong convexity has interesting consequences; the usual convexity bound can be improved so we have:
T
f (y) ≥ f (x) + ∇f (x) (y − x) +
m
2
ky − xk2
2
We can analytically find the minimum point of RHS of this equation, and plug it back into the RHS to
get the lower bound of f (y):
f (y) ≥ f (x) −
1
2
k∇f (x)k2
2m
So this practically means that we have a near-optimal point when the gradient is small. When we know m
2
this can be a hard guarantee, but m is not attainable in general. Therefore, we resort to making k∇f (x)k2
small enough so that we have a high chance of being near optimal.
10.1.2 Conditional Number of Sublevel Sets
Conditional numbers of sublevel sets have a strong effect on efficiency of some algorithms. Conditional
number of a set is defined as the ratio between maximum and minimum width of the set. The width for a
convex set C along a direction q (kqk2 = 1) is defined by:
W (C, q) = sup q T z − inf q T z
z∈C
z∈C
10.2 Descent Methods
The family of iterative optimization algorithms which generate a new solution x(k+1) from x(k) by taking
x(k+1) = x(k) + t(k) ∆x(k)
where t(k) is called the step size, and ∆x(k) is called the search direction. Depending on how we choose t
and ∆x, we get different algorithms. There are two popular ways of choosing t:
• Exact line search minimizes f (t) = f x(k) + t · ∆x(k) exactly, by either analytic or iterative means.
This is used when this minimization problem can be solved efficiently.
• Backtracking search tries to find a t where the objective function sufficiently decreases. The exact
details isn’t very important; it is employed when the minimization problem is harder to solve. The
algorithm is governed by two parameters, which, practically, does not drastically change the performance of the search.
10.3 Gradient Descent
Taking ∆x(k) = −∇f (x) will give you gradient descent algorithm. Some results for the convergence analysis are displayed. The lower bound for iteration is given as:
log ((f (x0 ) − p∗ ) /ǫ)
log (1/c)
where p∗ is the optimal value, and we stop when we have f x(k) −p∗ < ǫ. The numerator is intuitive; the
denominator involves the condition number and roughly equal to m/M . Therefore, as condition number
increases, the number of required iterations will grow linearly. Given a constant condition number, this
41
bound shows that the error will decrease exponentially. For some reason this is called linear convergence
in optimization context.
10.3.1 Performance Analysis on Toy Problems
Exact search and backtracking line search is compared on toy problems; the number of iteration can differ
by a factor of 2 or something like that. Also, we look at an example where we play with the condition
number of the Hessian of f ; and the number of iteration can really blow up.
10.4 Steepest Descent
Steepest descent algorithm generalizes gradient descent by employing a different norm. Given a norm, the
normalized steepest descent direction is given by
n
o
T
∆x = argmin ∇f (x) v| kvk = 1
Geometrically, we look at all vectors in a unit ball centered at current x and try to minimize f (x). When
we use Euclidean norm, we regain gradient descent. Also, in some cases, we can think of SD as GD after a
change of coordinates ”intuitively this makes sense, because using a different norm is essentially employing
a different view on the coordinate system).
10.4.1 Steepest Descent With an ℓ1 Norm
When we use ℓ1 norm, SD essentially becomes the coordinate descent algorithm. It can be trivially shown:
take the basis vector with the largest gradient component, and minimize along that direction. Since we
are using ℓ1 norm, we can never take a steeper descent.
10.4.2 Performance and Choice of Norm
Without any problem-specific assumptions, essentially same as GD. However, remember that the condition number greatly affects the performance of GD - and change of coordinates can change the sublevel
set’s condition number. Therefore, if we can choose a norm such that the sublevel sets will approximate
an ellipsoid/sphere, SD works very well. A Hessian at the optimal point, if attainable, will minimize the
condition number greatly.
10.5 Newton’s Method
Newton’s method is the workhorse of convex optimization. The major motivation is that it tries to minimize
the quadratic approximation of f () at x. To do this, we choose a Newton step ∆xnt :
∆xnt = −∇2 f (x)
−1
∇f (x)
Several properties and interpretations are discussed.
• The Newton step minimizes the second-order Taylor approximation of f . So, when f () roughly follows
a quadratic form, Newton’s method is tremendously efficient.
42
• It’s the steepest descent direction for the quadratic norm defined by the Hessian. Recall that the
Hessian at the optimal point is a great choice for a norm for SD - so when we have a near-optimal
point, this choice minimizes the condition number greatly.
• Solution of linearized optimality condition: we want to find v such that ∇f (x + v) = 0. And approximately:
∇f (x + v) ≈ ∇f (x) + ∇2 f (x) v = 0
and the Newton update is a solution for this.
• Newton step is affinely invariant; so multiplying only a single coordinate by a constant factor will
not change convergence. This is a big advantage over the usual gradient descent. Therefore, Newton’s method is much more resistant to high condition number sublevel sts. In practice, extremely
high condition number can still hinder us because of finite precision arithmetic, yet it is still a big
improvement.
10.5.1 The Newton Decrement
The Newton decrement is a scalar value which is closely related; it is used as a stopping criterion as well:
1/2
T
−1
λ (x) = ∇f (x) ∇2 f (x) ∇f (x)
This is related to our estimate of our error f (x) − p∗ by the following relationship:
f (x) − p∗ ≈ f (x) − inf fˆ (y) =
y
1 2
λ
2
We stop when this value ”λ2 /2) is less than ǫ.
10.5.2 Newton’s Method
Newton’s method closesly follows the gradient descent algorithm, except it uses the Newton decrement for
stopping criterion, which is checked before making the update.
10.5.3 Convergence Analysis
The story told by the convergence analysis is interesting. There exists a threshold for k∇f (x)k2 , when
broken, makes the algorithm converge quadratically. This condition ”broken threshold for gradient), once
attained, will hold in all further iterations. Therefore, the algorithm works in two separate stages.
• In the dampen Newton phase, line search can give us an update size t < 1, and f will decrease by at
least γ, another constant.
• Pure Newton phase will follow, where we will only use full updates ”t = 1) and we get quadratic
convergence.
43
10.5.4 Summary
• Very fast convergence: especially, quadratic convergence when we reach the optimal point.
• Affine invariant: much more resistant to high condition numbers.
• Performance does not depend much on the correct choice of parameters, unlike SD.
10.6 Self-Concordant Functions
This section covers an alternative assumption on f which allows us a better ”or more elegant) analysis on
the performance of Newton method. This seems like more of an aesthetic, theoretic result, so unless some
insights come up in the video lectures, I am going to skip it.
44
Murphy Ch13: Sparse Linear Models
September 11, 2014
1.3
Useful for cases where we have more features than examples. Real world applications, kernel basis functions,
representing signals in terms of wavelet basis functions.
1
1. There are lots of heuristics. Forward selection, backward selection. Orthogonal least squares (where we
try to fit residuals from previous models). Orthogonal matching pursuits (variant of orthogonal least
squares, but we do not refit linear model after we
choose a new feature). And on, and on, and on.
Bayesian variable selection
We want to determine a binary vector γ which means
“relevance” of each variable. A Bayesian approach dictates:
p (γ|D) = P
2. Stochastic search: use MCMC. Metropolis Hastings
algorithm? However it is unnecessarily inefficient,
and there are better approximations.
p (D|γ)
′
γ ′ p (D|γ )
3. EM: EM cannot be applied to spike-and-slab model
because we cannot back out γ from w, or something like that. We can make another approximation to make EM work (gah). Bernoulli-Gaussian is
intractable in EM, but there’s another approximation.
If dimension is small enough, we can either
1. take the combination with largest posterior
2. take features with individual posterior probability
bigger than 0.5
1.1
Algorithms
Spike and slab model
2
The above approach is intractable when dimension is
large, of course. Let us approximate it by a model.
2.1
p (γ|D) ∝ p (γ) p (D|γ)
ℓ1 regularization: basics
Sparsity argument
(1) A Laplacian prior results in ℓ1 regularization (yeah I already know this). Then why is this sparse? The peThe spike and slab model represents the prior part
nalized optimization is the lagrangian dual of the conp (γ) by treating each variable individually, and giving
strained optimization problem. (And this should have
each variable a fixed probability to be relevant: π0 . This
zero duality gap, I guess) Then we get this famous figure
is a dial we use to control sparcity. The likelihood part is
modeled as a linear model of variables with normal error.
The weight of the linear model comes from γ, and also
from a normal prior we also get to set. (when rj = 0, wj
is set to be fixed at given parameter.)
The prior part peaks sharply at sparse models, and the
normal part is kind of spreads further... so this is called
the spike and slab model.
Then for a given bound on kwk1 , we can draw the
feasible region as a diamond around the origin. Also,
1.2 Bernoulli-Gaussian
for each value of the objective, we can draw a contour
This is essentially similar to spike and slab. Instead, w of weights that results in that objective. Note the two
is
P not determined by γ - the input to the likelihood is regions are in a tradeoff relationship. If we increase the
bound, the diamond will get bigger, and the contour will
j γj wj xij .
Also it is shown that this can be massaged into ℓ0 be smaller (we will go closer to the least squares soluregularization, where the penalization is on the number tion). Note that at the optimal point, the contour and
the diamond will touch each other (if it strictly overlaps,
of features. Meh.
1
we can always make the diamond smaller but retain the 2.5 Bayesian Inference
same objective value). And geometrically it is more likely
to hit the corner of the diamond because it is spiky in There are some results stating that the spike-and-slab
model results in better prediction accuracy than lasso.
higher dimensions.
2.2
Soft Thresholding
3
Another notable property is the soft thresholding effect.
We use subgradients to prove optimality of the ℓ1 loss.
This involves a quantity called cj , which is proportional
to the correlation between the jth feature and the residual due to the other features. Then, the final weight wj
is given as:


(cj + λ) /aj if cj < −λ
w
ˆj = 0
if cj ∈ [−λ, λ]


(cj − λ) /aj if cj > λ
3.1
3.2
LARS and homotopy methods
Homotopy methods exploit the fact that it is sometimes
faster to calculate the weights with given λk when λk ≈
λk−1 . LARS is a special case of such methods - starts
from a large enough λ so that only one variable will be
chosen. From there, we can add one variable at a time
fast, using an analytical result.
LARS cannot be used for ℓ1 -regularized GLMs.
Regularization paths
3.3
For different values of λ, we can plot how the weight
of jth feature changes - this is called the regularization
path. The figure in the book shows that the weights for
ridge regression deviates gradually from 0; but for lasso
they stay at 0 until it becomes nonzero suddenly.
It is noted that the regularization path is piecewise
linear, and LARS exploits this to solve this problem
faster(roughly, the same time required for a single least
squares).
2.4
Coordinate descent
When all coordinates are fixed except one, the optimal
solution might be calculated analytically. This gives rise
to the coordinate descent algorithm.
So, the weights will be zero if the absolute value of the
correlation power is smaller than λ. Also notable is that
the weight in other cases are also “shrunk”, which makes
lasso a biased estimator. So this is the problem we deal
with Hierarchical Adaptive Lasso.
2.3
ℓ1 regularization: algorithms
Proximal and gradient projection
methods
FISTA belongs to this category! When the objective f
can be decomposed of two convex functions L and R,
L differentiable and R _not_ differentiable, ISTA optimizes f in an iterative fashion. It approximates L with
a quadratic function Lq (θ) around last guess. So the
update looks like
θk+1 = argminz R (z) + Lq (z)
Model selection
Random facts and strategies.
For some popular forms of R we can solve this easily.
FISTA approximates L around a point other than the
most recent guess. Intuitively, it was said (during the
study group session) that this is effectively the “heavyball” method.
• Naive strategy: use lasso to pick features, and do
a least squares on the chosen variables. (well, in
practice, lasso is also used to fight collinearity as
well - so I think this is a poor idea)
• Theoretical considerations: using cross-validation
for model selection might not be optimal. Cross validation optimizes model prediction accuracy which
might be different from finding the correct model.
3.4
EM for lasso
Gah. Approximate Laplace prior with a Gaussian scale
mixture and we have a setup for EM. Why? The
variance-gamma distribution, which is a normal distribution with its variance following a gamma distribution,
is also called the generalized Laplace distribution. It actually includes the usual Laplace as a special case.
I initially thought, WTF? But turns out the formulation is useful for extending ℓ1 .
• Problem: ℓ1 regularization is unstable which means
small perturbations in data can result in drastically
different results. (Therefore Bayesian is more stable)
Bagging can help - do a number of bootstrapped
lasso and find the probability of each variable being
included. This is called bolasso.
2
4
Extensions of ℓ1 regularization
come by giving separate penalty parameter for each variable. Using cross-validation to tune those D parameters might be infeasible in traditional sense, but using a
Bayesian modeling we can just use EM to fit them. The
model looks like: γj → τj2 → wj . If we integrate τ s out,
we get a Laplace distribution (wow). γ follows inverse
Gamma, which is the conjugate for Laplace (wow), so
EM works kind of smoothly.. Impressive.
So this is like a generalized version of lasso, I guess.
You are searching through different “Laplace-like” priors
which include Laplace. The contour of the penalty looks
more like a starfish rather than a diamond.
• Group lasso solves the problem of one variable being related to multiple variables. For instance, when
we have a categorical variable. The approach is
to group coefficients together, and penalize each
group’s square root of sum of squared coefficients.
The text claims that the square root gives you sparsity. Intuitively, if each group consists of just one
variable, this is the same as ℓ1 so that kind of make
sense. However, still not sure why they do not just
use ℓ1 . This can be fit using proximal methods or
EM.
• Fused lasso is useful for denoising stuff. Usually, the
Automatic relevance determinadesign matrix is set to I and we have one weight per 6
one observation. The penalty is given by the ℓ1 norm
tion & sparse Bayesian learning
of the coefficients and ℓ1 norm of differences between
adjacent coefficients. You can also use EM to fit this, All approaches so far had the form: τj2 → wj → y ← X
again using the GSM formulation of Laplace.
- and we integrated τ out. Instead, we can integrate out
w and maximize the likelihood with regard to τ 2 . This
• Elastic net is a mixture of lasso and ridge. There are gives rise to a method called ARD or SBL.
results that state lasso works poorly with correlated
variables - lasso will only pick one of them. Elastic
net supposedly solves all these problems.
6.1 ARD for linear regression
Conventionally, we denote weight precision (stdev) by
αj = 1/τ 2 , and measurement
precision by βj = 1/σ2 .
−1
and y ∼ N y|wT x, 1/β .
Then, w ∼ N w|0, A
(A = diag(α)) Now we can integrate out w, and put
a Gamma prior on both α and β. Now the likelihood is
– Also mentionable is the grouping effect, where
a function of parameters to the distribution of α and β,
highly correlated variables split up the weights
which is maximized.
that lasso would have given to only one.
Why is it sparse? Not really sure, the sparsity argu– However, the vanilla elastic net is known to ment here is definitely one of the crappier arguments in
give you rather unimpressive results. It is this book. :-(
said it’s because the shrinkage is performed
ARD is different from regular MAP in the sense that
twice - and you have to undo the ℓ2 shrink- we integrate out the prior parameter (in this case, α)
age. Hmmm........ where can I find research but here we integrate out w. However, there are some
regarding this? Maybe Hastie?
results that they are connected - ARD is in fact MAP of
a certain problem.
– You can fit elastic net using a standard lasso algorithm by augmenting the design matrix and
response vector (the same trick we used for
nnlasso).
5
Non-convex regularizers
6.2
Algorithms for ARD
Two problems with Laplace prior is claimed: the peak
and the tails are not heavy enough. (?) So the noise ARD objective is not convex, so the results will depend
are insufficiently shrunk, and the relevant coefficients are on the initial value.
shrunk too much. The text says there are more flexible
• Of course, EM can be used.
priors that have larger spike at 0 and heavier tails. Practically, it is said that they often outperform ℓ1 regular• Or a fixed-point algorithm can be used as well. We
ization. (!)
know that at the optimal point the gradients disapdℓ
dℓ
= 0), so we enforce it iteratively
= dβ
pear ( dα
j
5.1 Hierarchical adaptive lasso
until everything converges.
So the tradeoff: higher λs suppress noise better, but
• Iteratively reweighted ℓ1 algorithm
shrinks relevant parameters too much. This can be over-
3
6.2.1
For Logistic Regression
When we do binary logistic regression, we cannot use
EM because Gaussian prior is not conjugate to the logistic likelihood. We can do some approximations (such as
Laplace approximation) to use EM though.
7
Sparse Coding
Sparse coding is similar to sparse PCA/ICA in idea.
However, sparse coding generates lots of factors, usually more than the number of dimensions; however, the
coefficients are promoted to be sparse. So out of a lot of
factors, we choose a small number of them to represent
each data.
7.1
Learning a sparse factor loading matrix
We can maximize the following likelihood:
log p (D|W) =
N
X
i=1
log
ˆ
zi
N xi |Wzi , σ 2 I p (zi ) dzi
Two problems - this is too hard to optimize, and sparsity is not being promoted. We approximate it by summing maximum likelihood for each zi instead. Also we
use a Laplacian prior for z. After the approximation and
massaging the formula, this is optimized in the following
fashion - pick a guess for Z, find optimal W. Now estimate Z using W, and repeat. These two steps can be
done by least squares and lasso, respectively.
There are different variants of this - trying to approximate X by WZ (non-negative sparse coding), adding
sparsity constraint on both Z and W (sparse matrix factorization).
7.2
Compressed sensing
The rough idea: we do not get to observe x, but its low
dimensional projection y = Rx + ǫ. How do we optimize
it? We model x = Wz with a sparsity promoting prior on
z. Picking the right basis is important - wavelet, domain
specific.
4
Gaussian Processes
January 19, 2015
1 Introduction
In supervised learning, we try to learn an unknown function f for some inputs xi and outputs yi , such
that f (xi ) = yi , possibly corrupted by noise. Then, we work with the distribution of function over given
data: p (f |X, y). Usually, we assume a specific form of f , so we learn a parameter set θ and work with the
distribution p (θ|X, y). Nonparametric regression techniques, such as LOWESS (locally weighted scatterplot smoother), overcome this problem by using the given data as our knowledge representation directly.
Gaussian Process regression is a Bayesian equivalent of such technique.
How is GP different from LOWESS? We make an important assumption: any vector [f (xi )] for any
xi is jointly Gaussian. Also, the covariance between two function values f (x1 ) and f (x2 ) relies on the
user-provided kernel κ (x1 , x2 ).
2 GP Regression
2.1
Noise-free Observations
We usually assume the mean function E [f (x)] = 0 because GPs are usually flexible enough that we don’t
need an intercept. (However, µ is not omitted from below discussion because there are some modeling
techniques which employ a parametric representation of µ)
Say we have the training examples X and y. Since observations are noiseless, we assume yi = f (xi )
and write y = f. Now we can take a test example X∗ and predict the distribution of the corresponding
output f∗ . How do we do it? By our assumption of f , the vector of fs are jointly Gaussian, and follows the
distribution:
f
f∗
!
∼N
µ
µ∗
!
,
K
K∗
KT∗
K∗∗
!!
where K, K∗ and KT∗ are covariance matrices calculated from user-specified κ. Then, by standard rules for
conditioning Gaussians (of course I don’t know how to do this, but see section 4.3), we can get a distribution
for f∗ conditioned on the rest of the observed variables:
p (f∗ |X∗ , X, f) = N (f∗ |µ∗ , Σ∗ )
where µ∗ = µ (X∗ ) + KT∗ K−1 (f − µ (X)) and Σ∗ = K∗∗ − KT K−1 K∗ . We now got a posterior distribution of
output values for X∗ .
1
2.2
Noisy Observations
We now have:
cov [yp , yq ] = κ (xp , xq ) + σy2 δpq
where δpq is the Kronecker delta.
2.3
Kernel Parameters
Different kernel parameters will result in different functions. Say we use a squared-exponential kernel:
κ (xp , xq ) =
σf2
1
2
exp − 2 (xp − xq ) + σy2 δpq
2ℓ
• Increasing ℓ will decrease covariance between any two points by discounting distances; the function
will be more wiggly as a result.
• Increasing σf will increase the vertical scale of the function.
• Increasing σy will increase the uncertainty around near points.
This basically means we encode our belief on the function’s behavior in the kernel.
2.4
Estimating Kernel Parameters
Kernel parameters are chosen by maximizing the marginal likelihood p(y|X).
2
Murphy Ch16: Adaptive Basis Function
January 21, 2015
1.1.2 Pros and Cons
Instead of manually concocting kernels, we try to
learn the features from the input data. Methods
CART are easy to interpret, but they do not predict
based on ABF use weak learners as a way to come
very accurately, and it has a high variance.
up with features from the data.
1.2
1 Trees
1.1
Random Forests
In a random forest, we fit M different trees each
trained by a different subset of the data ”bootstrapped samples, and choosing a subset of the fea-
CART
tures). So this is bagged decision trees. For performance of RF compared with different tools, see
CART ”Classification and Regression Trees) is a clas-
Caruana”2006).
sical decision tree model. Basically, at each node of
Why does RF perform well? Recall the error can
the tree, it splits the input space into two chunks by
an axis-parallel hyperplane. When we reach the leaf
be decomposed into bias and variance. Decision
trees are very high variance and low bias. And tak-
of the tree, the final prediction/classification is made
ing average of multiple decision tree will keep the
bias ”if the bias is same across all trees) and lowers
by a constant, or a simple learner such as linear regression.
variance by a square root factor. ”See mathematicalmonk for proof)
In practice, not only we take subsets of data points
1.1.1 Growing and Shrinking Trees
and attributes, we add random types of splitting operators to increase the randomness.
Given a loss function, we can always try to find
the pair of optimal parameter and the threshold
which minimizes the average error, assuming the
2 Generalized Additive Models
next nodes will be leaves. We can try all parameters; and for each parameter we sort the possible
Generalized additive model employs multiple ”pos-
values and try each threshold between two adja-
sibly nonparameteric) models, one for each feature.
cent observed values. We can stop growing the tree
Each model j tries to predict yi from feature xij ; so
when some stopping criteria is met; this criteria is a
the model looks like:
heuristic. For example, you can limit the depth, or
have a threshold in reduction of cost, etc. Growing
f (x) = α + f1 (x1 ) + · · · + fD (xD )
a tree sometimes is sometimes to myopic; we might
not make any splits at all. An alternative is grow a
fj () can be any scatterplot smoother; regularized
splines are common. ”Hmm, why not use something
full tree and prune.
1
nonparametric like LOESS?)
3.2
2.1
Different loss functions result in different boosting
algorithms. Examples:
Backfitting
GAMs can be fit by an iterative scheme; at each stage
• Squared loss results in L2Boosting
each fj tries to fit the residual from all models except
• Exponential loss results in AdaBoost
fj . Also, note α is not identifiable in the above form;
PN
we need to set i=1 fj (xij ) = 0 to ensure this.
2.2
• Logloss results in LogitBoost ”since exponential
loss explodes for outliers, this is more robust)
MARS
These approaches typically require that you do a
weighted fit at each stage.
Multivariate Adaptive Regression Splines introduces interaction terms to GAMs. Interaction terms
are generated by multiplying basis functions in the
form of (xj − t) , (t − xj ) where t is an observed value
of xj .
3.3
Functional Gradient Boosting
So far, we tried to predict the residual between the
target and our prediction in each stage of the boosting process. Instead, we can try to approximate the
gradient of the loss function. The target of the regression g looks like this:
3 Boosting
3.1
Different Flavors of Boosting
Forward Stagewise Additive Boosting
gim =
Boosting tries to optimize the sum of losses given a
∂L (yi , fm−1 (xi ))
∂fm−1 (xi )
And we do a line search to find ρm which mini-
loss function L():
mizes L (fm−1 − ρgm ). This does not require the re-
f ∗ = min
f
X
gression process to work on weighted sets, which is
an advantage. Also it can be easily extended to different types of losses, such as Huber loss.
L (yi , f (xi ))
i
We will have different functions for different
types of L(). Typical choices include the squared
loss ”in regression contexts), log loss and exponen-
3.4
tial loss ”in classification contexts). At each stage,
Sparse boosting uses simple linear regression as the
adaptive basis function. This is known as forward
stagewise linear regression ”how is this different
from stepwise regression?).
we try to fit the residual between y and the current
prediction, by minimizing:
(βm , γm ) = argminβ,γ
X
Sparse Boosting
L (yi , fm−1 (xi ) + βφ (xi ; γ))
i
3.5
where γ is the parameter set for the weak learner;
and β is the weight for it. Then fm is taken to be:
MART
When shallow CART models are used as the weak
learner for gradient boosting, it is called MART. This
fm (x) = fm−1 (x) + βm φ (x; γm )
has a slight modification to the gradient boosting al-
In practice, shrinking β by multiplying it with 0 <
ν ≤ 1 works better than the above update.
the leaves of the tree to minimize the loss, instead of
minimizing the gradient.
gorithm though; we re-estimate the parameters at
2
3.6
Interpretations
Boosting can be interpreted as an ℓ1 regularization.
Also, there’s a Bayesian interpretation ”well, sort of
broken IMO) about boosting as well.
4 Notes
• Subsampling: empirically max is preferred
over avg for some reason
• Feeding artificially distorted input ”hmm)
• Convolutional neural net uses human ”domain
specific) expertise in feature generation
3
Murphy Ch14: Kernel Methods
January 21, 2015
Kernels are measures of similarity defined for two elements in an abstract space. Of course, this is
hugely useful.
1 Kernel Functions
A kernel function κ (x, x′ ) ≥ 0 is a real-valued function for x, x′ ∈ χ. Typically this function is symmetric.
This section goes through different types of kernels and their properties.
This page also seems to provide valuable insights: http://mlg.eng.cam.ac.uk/duvenaud/cookbook/index.html
1.1
RBF Kernels
The Radial Basis Function kernel, or the squared exponential kernel, or the Gaussian kernel looks like:
1
T
κ (x, x′ ) = exp − (x − x′ ) Σ−1 (x − x′ )
2
Σ is commonly diagonal, so the function ends up looking like:
1X 1
2
(xi − x′i )
κ (x, x ) = exp −
2 i σi2
′
!
where σi is called the characteristic length scale of dimension i. When all dimensions share the characteristic length (you probably want to standardize the data beforehand), we get:
kx − x′ k
κ (x, x′ ) = exp −
2σ 2
2
!
which only depends on the distance between two vectors (the radial basis), so this is called the RBF kernel.
1.2
Cosine Similarity
The cosine similarity between TF/IDF vectors for document similarity can also be thought of a kernel.
1
1.3
Mercer Kernels
Mercer Kernel is a class of kernels which a very desirable property. Many kernels we use are Mercer
kernels.
A Mercer kernel satisfies the following: for any sets of input xi , the Gram matrix K with Kij = κ (xi , xj )
is positive definite kernel. For this reason, Mercer is also called positive definite kernel.
The Mercer kernel’s importance comes from the fact that Mercer kernel can be expressed by an inner
product. We can decompose the Gram matrix K as:
K = U T ΛU
And we can now set:
where
1
φ (xi ) = Λ 2 U:,i
T
kij = φ (xi ) φ (xj )
So you can find the feature vector implied by the kernel function. This fact is very important to apply
the kernel trick, described below.
1.4
Linear Kernels
κ (x, x′ ) = xT x′
Wat.
1.5
Matern Kernel
The Matern kernel is commonly used in GP regression.... but don’t know what is going on
1.6
Domain Specific Kernels
An example of a highly domain-specific kernel, we define a kernel measuring the similarity between two
strings:
κ (x, x′ ) =
X
ws φs (x) φs (x′ )
s
where s is any possible substring, φs (x) is the number of times s occurs in x.
Another example is the pyramid matching kernel; when spatial matching of two sets of points are desired, we can group points into a multi-sized histogram and each histogram are compared.
2
1.7
Kernels From Probabilistic Generative Models
1.7.1 Probability Product Kernels
Say we have a generative model: p (x|θ). Given two examples xi and xj , we can try to estimate the parameter from one and see how the other is probable. This gives rise to the probability product kernel.
1.7.2 Fisher Kernel
I don’t fully understand details, but Fisher kernel relies on this idea: if two observations are similar, they
will want to update the MLE in a similar direction. So their gradient should look similar.
2 Using Kernels Inside GLMs
We can easily use kernels in any GLM, by using a feature vector comprised of distances from some centroids:
φ (x) = [κ (x, µ1 ) , κ (x, µ2 ) , κ (x, µ3 ) , · · ·]
µi ∈ χ is a set of K centroids. Of course, picking the right set of centroids is very tricky. If we have a low
dimension input space, we can make a grid of centroids, but that’s plain impossible in a high dimensional
space.
One way of solving this problem is making every observation a centroid of its own, and use a sparsitypromoting prior to make sure we don’t get a hairy overfitted monster. There are a few approaches;
• L1VM, ℓ1 -regularized vector machine uses ℓ1 regularization.
• RVM, relevance-vector machine uses techniques similar to ARD/SBL algorithms.
• SVM modifies the likelihood term (into hinge loss, I guess) to promote sparsity.
3 The Kernel Trick
Instead of replacing all feature vector with its kernel-based representation φ (x) = [κ (x, µ1 ) , κ (x, µ2 ) , κ (x, µ3 ) , · · ·],
we can rewrite algorithms to replace inner product with a call to κ. This is called the Kernel trick.
For example, we can replace the following calculation:
2
kxi − xj k2 = xTi xi + xTj xj − 2xTi xj = κ (xi , xi ) + κ (xj , xj ) − 2κ (xi , xj )
The latter equivalence requires that the kernel is Mercer. (Note that x in the first two parts are actually
φ (x) - don’t know why the book uses x)
3.1
Kernelized Ridge Regression
Unlike k-means or nearest neighbor, ridge regression is not based on the notion of similarity. So how do
we kernelize them? The closed form solution for ordinary ridge looks like this:
3
w = X T X + λID
−1
XT y
Using matrix inversion lemma, this can be rewritten as:
w = X T XX T + λIN
−1
y
See that XX T ? We’re going to replace it with the Gram matrix K!
3.2
Kernel PCA
The detailed treatment is kind of complex; here’s a brief description of the idea.
We also want to replace XX T with the Gram matrix. We usually calculate eigenvectors from X T X, but
to kernelize we need to get them from XX T (so we can put K in place). Fortunately, getting eigenvectors of
X T X from that of XX T requires some simple algebraic manipulation. After that, we have to take care of
normalization of eigenvectors and centering; but I leave it to future myself to understand if need arises....
4 Support Vector Machines
This section gives a brief treatment of SVMs, both for regression and classification. There seems to be
two different ways to derive SVM; the large-margin approach (which coincides with the introduction from
2
Boyd) and the loss-function approach. The loss-function approach will treat the kwk2 term as an ℓ2 regularization term.
4.1
Loss Function Treatment
SVM regression uses the epsilon insensitive loss function which is flat 0 around 0, and grows linearly after
that. It’s a symmetric version of the hinge loss. Obviously this is not differentiable, but we can introduce
slack variables - and we have a constrained problem with a differentiable objective. Also, one can show
that the optimal weights w
ˆ is given by
w
ˆ=
X
α i xi
i
where α is a nonnegative, sparse vector. So the weight is a weighted sum of input samples; these are
the support vectors which are the namesake.
SVM classification uses the hinge loss, obviously.
4.2
Multiclass Classification
There’s no elegant generalization of SVM for multiclass problems, unlike the Bayesian approaches. (It’s
quite evident that Murphy doesn’t think highly of SVMs ;)
4
4.3
Choosing Parameters
We need to both choose the kernel parameter and the regularization multiplier. This can be done by a
grid-search fashion, or a LARS-like fashion - exploiting narrow kernels need high regularization, and vice
versa.
5 Comparison of Discriminative Kernel Methods
This section describes the difference between different kernel-based methods. The conclusion seems to be
that most methods have comparable performance - and L1VM should be the fastest. SVM doesn’t give you
well-calibrated probabilities, so avoid it unless you have a structured output.
6 Kernels for Building Generative Models
This section deals with a different type of kernels; smoothing kernels. Topics like Nadaraya-Watson, or
LOESS are discussed.
6.1
Smoothing Kernel
Obviously, smoothing kernel is more or less a pdf; they sum to 1. Gaussian is a popular smoothing kernel.
We Epanechnikov kernel, tri-cube kernel is also mentioned.
6.2
Kernel Density Estimation
KDE is a nonparametric method to come up with a distribution given samples:
p (x|D) =
1 X
κh (x − xi )
N
which is also called the Parzen window density estimator. h is a bandwidth parameter you get to set.
If you want to do this to estimate a class conditional density, you might want to let each point have a
different bandwidth, and expand it until you hit K neighboring samples. (This doesn’t mean much if you
are using Gaussian kernels; something with a compact support would be needed.) Anyways, this is similar
to K-means.
6.3
Kernel Regression
Kernel regression is a natural extension of nearest neighbors:
f (x) =
X
wi (x) yi
i
and wi is just normalized distance between xi and x:
κh (x − xi )
wi (x) = P
j κh (x − xj )
5
So the regression takes the weighted mean of observed ys.
You don’t have to use smoothing kernel for this; replace κh (x − xi ) with κ (x, xi ) and you got yourself a
LOESS, locally-weighted regression.
6

Sample Question Paper - jsunil tutorial cbse maths science

Assignment 04 (No need to hand in your solutions)

13.4 THE CROSS PRODUCT The Area of a Parallelogram Definition

102M Lab Manual - Department of Physics - The University of Texas

How to Estimate Eigenvalues and Eigenvectors of Covariance

The Dantzig selector - Department of Statistics

Point Estimation: properties of estimators 1 FINITE-SAMPLE

Vector Webinar: AUTOSAR Configuration Process - How to handle

Sample Solutions of Assignment 9 for MATH3270A for ODE

NEURAL NETWORKS and PATTERN RECOGNITION

Econometrics 2. Sample Questions - Personal pages of the CEU

register

Sample Exam #1 Elementary Statistics