Studies in linear and non-linear programming pdf download

Nonlinear Optimization text-book. Cornelis Roos. Download PDF. A short summary of this paper. Roos, and T. Terlaky T. Delft , U. Euler — 0. The contours of the objective function are shown. This NLO problem has two optimal solutions, namely 0. Index sets will be denoted by I, J and K. We now define some classes of NLO problems. We will abuse this terminology a bit by sometimes referring to affine functions as linear.

Linear opti- mization is often called linear programming. The reader is probably familiar with the simplex algorithm for solving LO problems. It contains the following optimization problem. Exercise 0. A unified methodology for solving nonlinear optimization problems would have to wait until the development of calculus in the 17th century.

This modern methodology is due to Fermat — Because of this work, Lagrange — stated clearly that he considered Fermat to be the inventor of calculus as opposed to Newton — and Liebnitz — who were later locked in a bitter struggle for this honour. In the words of Lagrange himself: One can state the following general principle.

If one is looking for the max- imum or minimum of some function of many variables subject to the con- dition that these variables are related by a constraint given by one or more equations, then one should add to the function whose extremum is sought the functions that yield the constraint equations each multiplied by undeter- mined multipliers and seek the maximum or minimum of the resulting sum as if the variables were independent.

The resulting equations, combined with the constraint equations, will serve to determine all unknowns. We can therefore solve the nonlinear system 3 and the x-part of one of the solutions of 3 will yield the optimal solution of NLO provided it exists. Note that it is difficult to know beforehand whether an optimal solution of NLO exists.

This brings us to the problem of solving a system of nonlinear equations. Isaac New- ton lent his name to perhaps the most widely known algorithm for this problem. The study of nonlinear optimization in the time of Fermat, Lagrange, Euler and Newton was driven by the realization that many physical principles in nature can be explained via optimization extremum principles.

For example, the well known prin- ciple of Fermat for the refraction of light may be stated as: in an inhomogeneous medium, light travels from one point to another along the path requiring the shortest time. Similarly, it was known that many problems in celestial mechanics could be formu- lated as extremum problems.

We return to the problem of deciding whether NLO has an optimal solution at all. In the 19th century, Karl Weierstrass — proved the famous result — known to any student of analysis — that a continuous function attains its infimum and supremum on a compact set. This gave a practical sufficient condition for the existence of optimal solutions.

In modern times, nonlinear optimization is used in optimal engineering design, fi- nance, statistics and many other fields. It has been said that we live in the age of optimization, where everything has to be better and faster than before. Think of de- signing a car with minimal air resistance, a bridge of minimal weight that still meets essential specifications, a stock portfolio where the risk is minimal and the expected return high; the list is endless. If you can make a mathematical model of your decision problem, then you can optimize it!

In Chapters 1 and 2 we will study the methods of Fermat and Lagrange. So- called duality theory based on the methodology of Lagrange will follow in Chapter 3. Then we will turn our attention to optimization algorithms in the remaining chapters. Finally, we conclude with a chapter on special classes of structured nonlinear optimization problems that can be solved very efficiently by interior point algorithms. In the rest of this chapter we give a few more examples of historical and practical problems to give some idea of the field of nonlinear optimization.

Tikhomirov AMS, For more background and details the reader is referred to that book. We only state the prob- lems here — the solutions will be presented later in this book in the form of exercises, once the reader has studied optimality conditions. Of course, the mathematical tools we will employ were not available to the originators of the problems.

For the ingenious original historical solutions the reader is again referred to the book by Tikhomirov. How can one prove that this is correct? Solution via Exercise 2.

In this work he considers the following problem among others. Given a sphere, inscribe a cylinder of maximal volume. And of course the diagonal of the cylinder has the same length as the diameter of the sphere.

How can one show that this is correct? Solution via Exercises 0. This problem was included in the first book on optimization, namely On maximal and minimal values by Viviani in How can one find the Torricelli point for any given triangle? Below we give examples of the least squares and other quadratic optimization prob- lems. One of the important characteristics of the concrete are its sand-and-gravel composition, i.

For each sort concrete the civil engineers can give an ideal sand-and- gravel composition that ensures the desired strength with minimal cement content.

Unfortunately, in the sand-and-gravel mines, such ideal composition can not be found in general. The solution is to mix different sand-and-gravel mixtures in order to ap- proximate the desired quality as closely as possible. Mathematical model Let us assume that we have n different stone size categories. The components ci indicate what fraction of the sand- P and-gravel mixture belongs to the i-th stone size category.

The goal is to find the best possible approximation P of the ideal mixture by using the material offered by the m mines. We can further sim- plify this problem by eliminating variable z. The distance of two vectors can be measured alternatively by e. Restate the mixing problem by using these norms and show that this way, in both cases, pure linear optimization problems can be obtained. Given assets with expected return ri and covariances vij , the problem is to find portfolios of the assets that have minimal variance given a level of total return, and maximal return given a level of total variance.

The so-called efficient frontier is then just the optimal value function. Example 0. One can check e. In doing this you are free to set the level of risk you are prepared to take.

Give the mathematical description of the problem you have solved and report on your results. For example, one can formulate necessary and sufficient conditions for the existence of optimal solutions in this case. It is therefore important to study convexity in some detail. Formally we have the following definition. Definition 1. In other words, the line segment connecting two arbitrary points of a convex set is contained in the set.

Figure 1. Theorem 1. Exercise 1. Then cl C is a convex set. We now turn our attention to so-called convex functions. Intuitively, it is easier to characterize minima of convex functions than minima of more general functions, and for this reason we will study convex functions in some detail.

Prove that f is a convex function. These are convex functions with the nice property that — if a minimum of the function exists — then this minimum is unique. We have seen in Exercise 1. Also, the next exercise shows that a quadratic function is convex if and only if the matrix Q in its definition is positive-semidefinite PSD. Hint: use the result from the previous exercise. Note that we can change the problem of maximizing a concave function into the problem of minimizing a convex function.

Now we review some further properties of convex sets and convex functions that are necessary to understand and analyze convex optimization problems. First we review some elementary properties of convex sets. Observe that conv S is generated by taking all possible convex combinations of points from S. We now define some important convex subsets of a given convex set C, namely the so-called extremal sets, that play an important role in convex analysis.

The faces of a polytope are familiar examples of extreme sets of the polytope. An extremal set consisting of only one point is called an extremal point. Observe that extremal sets are convex by definition, and the convex set C itself is always an extremal set of C. It is easy to verify the following result. Lemma 1. The epigraph itself is the only two dimensional extremal set. Then all extremal sets of C are closed.

In the above examples we have pointed out extremal sets of different dimension with- out giving a formal definition of what the dimension of a convex set is. The dimension of C is defined as the dimension of aff C.

Give the extreme points of C; Motivate your answer. What is aff C in this case? What is the dimension of C? Motivate your answer. Hint: Use Exercise 1. Further, by Exercise 1. Then C has at least one extremal point. The set of such points is not empty, because C is bounded and closed and the norm function is continuous. This is a contradiction, because these three different points lie on the same line as well. The lemma is proved. The following theorem shows that a compact convex set is completely determined by its extremal points.

Then C is the convex hull of its extreme points. Show that the minimum of f is attained at an extreme point of C. Hint: Use the Krein- Milman Theorem. Example 1. A pointed convex cone could be defined equivalently as a convex cone that does not contain any line. If the convex cone C is pointed, then it does not contain any subspace, except the origin 0. In that case we show that 0 is an extremal point of C. This 1 2 implies that the line through x , the origin 0 and x is in C, contradicting the assumption that C is pointed.

By definition, every vector in the dual cone has a nonnegative inner product with every vector in C. Sn is a self-dual cone. An important cone in the study of convex optimization is the so-called recession cone. Surprisingly, the recession cone does not depend on the choice of x. The unit sphere is a closed convex, i. This claim follows from the following three observations: 1.

The proof of the first statement is complete. The convexity of R x easily follows from the convexity of C. The other part of the proof follows form item i of Lemma 1. Then by item iv of Lemma 1. By statement iii of the same lemma, this line is contained in the recession cone of C as well.

Applying iv of Lemma 1. This is a contradiction, the lemma is proved. Let us assume that RE is not an extremal set of R. This leads us to a generalized concept of the interior of a convex set, namely the relative interior. If the convex set is full-dimension i. The set of relative interior points of the set C will be denoted by C 0.

The relative interior C 0 of a convex set C is obviously a subset of the convex set. We will show that the relative interior C 0 is a relatively open i. The proof is constructive. A direct consequence of the above lemma is the following. Moreover, if C is nonempty then its relative interior C 0 is nonempty as well. For a proof the reader is referred to the excellent books of Rockafellar [38] and Stoer and Witzgall [42]. Then f is continuous on the relative interior C 0 of C. Let us first consider the one dimensional case.

We use an analogous construction for the n-dimensional case. Thus the function f x is continuous at the point 0. P Then k k! The proof is complete. We leave the proofs as exercises. Give a concrete example that in this case the statement of the lemma fails. Here the first inequality followed from the convexity of the function f.

We will work towards generalizing these results to multivariate convex functions. Further we introduce the concept of directional derivative. This implies the following result. The proof of the lemma is complete. Then the following statements are equivalent. The function f is convex on C. As one interchanges the role x and x, the right-hand-side inequality is obtained analogously. Now we prove that 2 implies 3.

Finally we prove that 3 implies 1. The proof of the Lemma is complete. The theorem is proved. Let f be a twice continuously differentiable function on the open convex set C.

First optimality conditions for unconstrained op- timization are considered. Then optimality conditions for some special constrained optimization problems are derived. First we define local and global minima of the above problem. Definition 2. Convex functions possess the appealing property that a local minimum is global.

Example 2. Exercise 2. Theorem 2. The same proof applies if we choose ei , the standard unit vectors instead of the arbitrary direction s. Show that the solution given by Kepler is indeed optimal by using Theorem 2. Such points are not necessarily minimum points. These points are called stationary points. We will now see that this is also a sufficient condition if f is convex. Proof: As x is a minimum of f then by Theorem 2. Show that the objective function is convex.

Give necessary and sufficient conditions for a minimum of 2. In other words, give the equations that determine the Torricelli point. Find the Torricelli point of the triangle with vertices 0, 0 , 3, 0 and 1, 2. If we assume positive definiteness then we get a strict local minimum. Proof: Since f is twice continuously differentiable, it follows from the positive def- initeness of the Hesse matrix at x that it is positive definite in a neighborhood of x.

Hence the claim follows from theorem 2. Here aff C denotes the affine hull of C. In general this is not the case because the level sets of convex optimization problems are closed. However as we will see later the barrier function approach will result in such relatively open feasible sets. This is an important feature of interior point methods that will be discussed later on.

If the set of feasible solutions is not relatively open, similar results by using similar techniques can be derived see Theorem 2. Eliminate one of the variables and show that the resulting problem can be written as the problem of minimizing a univariate convex function on an open interval.

Show that the answer given by Tartaglia is indeed the optimal solution, by applying Theorem 2. Almost always we will assume that the functions are differentiable. Due to the convexity of F the right hand side of the above equation is in F , hence the convexity of F D x follows. Note that the cone of feasible directions is not necessarily closed even if the set of feasible solutions F is closed. Figure 2. We will now formulate an optimality condition in terms of the cone of feasible di- rections.

It states that a feasible solution is optimal if and only if the gradient of the objective in that point has an acute angle with all feasible directions at that point no feasible descent direction exists. In doing so, we will look at where we are going in the rest of this chapter, and what we would like to prove.

The results in this section are not essential or even necessary in developing the theory further, but should provide some geometric insight and a taste of things to come. This condition will depend only on the constraint functions that are zero active at x. We now give a sufficient condition for 2. The condition 2. The proof that 2. Hint: Use Lemma 1. Use the previous exercise and Theorem 2. Prove that the new problem is a convex optimization problem CO. If we write out condition 2.

Almost always we will assume that the functions f and gj are differentiable. We now introduce the assumption on F that we referred to in the previous section, namely the Slater condition.

Let us consider the optimization problem min f x s. The feasible region F contains only one point, 2, 0 , for which the non-linear constraint becomes an equality. Hence, the problem is not Slater regular. For this point the non-linear con- straint holds with strict inequality. However, this point does not lie in the relative interior of C. Some constraint functions gj x might take the value zero for all feasible points. Such constraints are called singular while the others are called regular.

Remark: Note, that if CO is Slater regular, then all singular functions must be linear. First we show an elementary property. Lemma 2. This last statement follows from the convexity of the functions gj. Hence, this point is an ideal Slater point. It will play an essential role in developing the KKT theory. Before stating the convex Farkas lemma, we present a simple separation theorem.

It essentially states that disjoint convex sets can be separated by a hyper plane geomet- rically, in IR2 or IR3 , the convex sets lie on different sides of the separating plane. Its proof can be found in most textbooks see e. Now we are ready to prove the convex Farkas Lemma. The proof here is a simplified version of the proofs in the books [38, 42].

The systems 2. Proof: If the system 2. This is the trivial part of the lemma. Note that this part is true without any regularity condition. To prove the other side let us assume that 2. Clearly the set U is convex note that due to the Slater condition singular functions are linear and due to the infeasibility of 2. Hence ac- cording to Theorem 2. Secondly we establish that 2.

Then we prove that y0 must be positive. Hence by 2. The proof is by contradiction. Hence from 2. At this point we have 2. The system 2. Adding a sufficiently large positive multiple of 2. The general inductive step goes analogously. By construction the system 2. Adding a sufficiently large multiple of 2. We have proved that the multipliers of all the singular constraints can be made strictly positive. Let us consider the convex optimization problem CO min x s.

Then CO is not Slater regular. Hence, the Farkas Lemma does not hold for CO. Then CO is Slater regular 0 is an ideal Slater point. Apply the convex Farkas Lemma 2. With a proper reformulation, apply the convex Farkas Lemma 2.

Note that for fixed y the Lagrange function is convex in x. One easily sees that 2. We will see in the proof of Theorem 2. On the other hand, for feasible x, i. This example does not have an optimal solution, and, as we have mentioned, the x part of a saddle point is always an optimal solution of CO. Assuming Slater regularity, the next result states that L x, y has a saddle point if and only if CO has an optimal solution. Assume that the Slater regularity condition is satisfied.

The vector x is an optimal solution of CO if and only if there is a vector y such that x, y is a saddle point of the Lagrange function L. Proof: The easy part of the theorem is to prove that if x, y is a saddle point of L x, y then x is optimal for CO.

The proof of this part does not need any regularity condition. From the saddle point inequality 2. To prove the other direction we need Slater regularity and the Convex Farkas Lemma 2. Let us take an optimal solution x of the convex optimization problem CO. By the Convex Farkas Lemma 2. Corollary 2. Under the assumptions of Theorem 2. To check the validity of these statements is left to the reader.

It is important to understand that — under the assumptions of Corollary 2. The proof is left as an exercise. Thus we have derived necessary and sufficient optimality conditions for the convex optimization problem CO under the Slater regularity assumption. Note that if an optimization problem is not convex, or does not satisfy any regularity condition, then only weaker results can be proven. Under some assumptions, a convex optimization problem CO and its dual have the same optimal objective values.

We can therefore use the dual problem to show that a certain solution of CO is in fact optimal. Moreover, some optimization algorithms solve CO and its dual problem at the same time, and when the objective values are the same then optimality has been proved. First we define the more general Lagrange dual and then we specialize it to get the so-called Wolfe dual for convex problems. Lemma 3. It is easy to prove the so-called weak duality theorem, which states that the duality gap is always nonnegative.

Theorem 3. Corollary 3. To prove the so-called strong duality theorem one needs a regularity condition. Let x be a feasible solution of CO. Proof: Directly follows from Corollary 2. It is also not always true even not under regularity condition that the convex optimization problem has an optimal solution. Frequently only the supremum or the infimum of the objective function exists.

Example 3. For this reason, the Wolfe dual does not have a concave objective function in general, but it is still very useful tool, as we will see. Using the results of the previous section one easily proves weak and strong duality results, as we will now show.

A more detailed discussion of duality theory can be found in [2, 28]. In other words, weak duality holds for CO and WD. Proof: Let x, y be a feasible solution for WD. Since the functions f and g1 ,. This completes the proof. Also assume that CO satisfies the Slater regularity condition. Proof: Follows directly from Corollary 2. Remember, we are only allowed to form the Wolfe dual of a nonlinear optimization problem if it is a convex optimization problem.

In such cases no duality relation holds in general. For nonconvex problems one has to work with the Lagrange dual. Note that the Slater condition holds for this example. Lagrange dual We can double check this answer by using the Lagrange dual. Obviously all the constraints are continuously differen- tiable. Here aj denotes the jth row of matrix A. Quadratic optimization The quadratic optimization problem is considered in the symmetric form.

Observe that the vector x occurring in this dual is not necessarily feasible for QO! To eliminate the x variables another form of the dual can be presented. This problem can also be used to illustrate duality in convex optimization.

The most probable density values at the sample points are to be determined that satisfy some linear e. Also note that for simplicity we did not split the equality constraint into two inequalities but we used immediately that its multiplier is a free variable. Observe further that due to the logarithm in the primal objective, the primal optimal solution is necessarily strictly positive, hence the dual variable s must be zero at the optimum.

It shows that positive duality gap might occur for convex problems when the problem does not satisfy the Slater regularity condition. Moreover, it makes clear that the Wolfe dual might be significantly weaker than the Lagrange dual. The only constraint is non-linear and singular, thus CO is not Slater regular. The optimal value of the object function is 1. Thus the Wolfe dual is infeasible, yielding an infinitely large duality gap.

This gives a nonzero duality gap that equals to 1. Here the convex set C where the above functions are defined is IR2. Let us make the Lagrange dual of 3. Thus for this problem both the primal and the dual problems have optimal solutions with equal zero optimal objective values. As we will see the new formulation results in a quite different dual. The new dual has also an optimal solution but now the duality gap is positive.

Exercise 3. Prove that the set C is convex. Prove that problem 3. The last equality requires some explanation.

Note that although the problem is not convex, and does not satisfy the Slater regularity condition, the duality gap is zero.

The primal semidefinite optimization problem is defined as P SO min cT x 3. The dual problem of the semidefinite optimization problem, as given e. Again, the dual of the semidefinite opti- mization problem is convex. The trace of a matrix is a linear function of the matrix and the convex combination of positive semidefinite matrices is also positive semidefinite. Since the minimization in 3. If we minimize in S all the terms in 3. The reader readily verifies that this is identical to 3.

Give the dual semidefinite problem. Prove that there is a positive duality gap for this primal—dual semidefinite optimization pair. If we minimize in s all the terms in 3. If we minimize 3. A generic algorithm for minimizing the function f x can be presented as follows. The crucial elements of all algorithms, besides the selection of a starting point are printed boldface in the scheme, given above.

Once a search direction is obtained, then one performs the line search procedure. Before we discuss these aspects in detail we turn to the question of the convergence rate of an algorithm. In order to quantify the concept of rate or order of convergence, we need the following definition. Definition 4. Exercise 4. Lemma 4. There are many ways to do this. This 2 k is called Dichotomous Search. It is based on a geometric concept called the Golden section.

The golden section of a line segment is its division into two unequal segments, such that the ratio of the longer of the two segments to the whole segment is equal to the ratio of the shorter segment to the longer segment.

With reference to Figure 4. Figure 4. Which of the two bounds is better? The algorithm is similar to the Dichotomous and Golden section search ones, in the sense that it too uses an interval of uncertainty that is reduced at each iteration. The algorithm proceeds as follows. Note that the two interpretations are indeed equivalent. It converges quadratically if the following conditions are met: 1.

The next two examples illustrate the possible scenarios. Example 4. This will be shown in Exercise 6. Although it works fine in many applications, several theoretical and practical disadvantages can be mentioned. First, the minimization of a convex quadratic function by the gradient method is not a finite process in general. This is an ideal book for teaching a one or two semester masters-level course in optimization — it broadly covers linear and nonlinear programming effectively balancing modeling, algorithmic theory, computation, implementation, illuminating historical facts, and numerous interesting examples and exercises.

Due to the clarity of the exposition, this book also serves as a valuable reference for self-study. This volume presents the essential concepts of linear and nonlinear programming in an accessible format filled with anecdotes, examples, and exercises that bring the topic to life.

The authors plumb their decades of experience in optimization to provide an enriching layer of historical context. Suitable for advanced undergraduates and masters students in management science, operations research, and related fields.

This book is an introduction to nonlinear programming. A comprehensive bibliography including diverse web sites with information about nonlinear programming, in particular software, is presented. Without sacrificing the necessary mathematical rigor, excessive formalisms are avoided. Several examples, exercises with detailed solutions, and applications are provided, making the text adequate for individual studies.

The book is written for students from the fields of applied mathematics, engineering, economy, and computation. Author : Katta G. Optimization is one of the most important areas of modern applied mathematics, with applications in fields from engineering and economics to finance, statistics, management science, and medicine.

While many books have addressed its various aspects, Nonlinear Optimization is the first comprehensive treatment that will allow graduate students and researchers to understand its modern ideas, principles, and methods within a reasonable time, but without sacrificing mathematical precision.

Andrzej Ruszczynski, a leading expert in the optimization of nonlinear stochastic systems, integrates the theory and the methods of nonlinear optimization in a unified, clear, and mathematically rigorous fashion, with detailed and easy-to-follow proofs illustrated by numerous examples and figures. The book covers convex analysis, the theory of optimality conditions, duality theory, and numerical methods for solving unconstrained and constrained optimization problems.

It addresses not only classical material but also modern topics such as optimality conditions and numerical methods for problems involving nondifferentiable functions, semidefinite programming, metric regularity and stability theory of set-constrained systems, and sensitivity analysis of optimization problems. Based on a decade's worth of notes the author compiled in successfully teaching the subject, this book will help readers to understand the mathematical foundations of the modern theory and methods of nonlinear optimization and to analyze new problems, develop optimality theory for them, and choose or construct numerical solution methods.

It is a must for anyone seriously interested in optimization. Author : Mokhtar S. Bazaraa,Hanif D. The Third Edition begins with a general introduction to nonlinear programming with illustrative examples and guidelines for model construction. The logical and self-contained format uniquely covers nonlinear programming techniques with a great depth of information and an abundance of valuable examples and illustrations that showcase the most current advances in nonlinear problems.

Author : Igor Griva,Stephen G. Flexible graduate textbook that introduces the applications, theory, and algorithms of linear and nonlinear optimization in a clear succinct style, supported by numerous examples and exercises.

It introduces important realistic applications and explains how optimization can address them. Author : William P. Optimization is the act of obtaining the "best" result under given circumstances. In design, construction, and maintenance of any engineering system, engineers must make technological and managerial decisions to minimize either the effort or cost required or to maximize benefits. There is no single method available for solving all optimization problems efficiently.

Several optimization methods have been developed for different types of problems. The optimum-seeking methods are mathematical programming techniques specifically, nonlinear programming techniques. Nonlinear Optimization: Models and Applications presents the concepts in several ways to foster understanding.

Geometric interpretation: is used to re-enforce the concepts and to foster understanding of the mathematical procedures.

The student sees that many problems can be analyzed, and approximate solutions found before analytical solutions techniques are applied. Numerical approximations: early on, the student is exposed to numerical techniques. These numerical procedures are algorithmic and iterative.

Algorithms: all algorithms are provided with a step-by-step format. Examples follow the summary to illustrate its use and application. Nonlinear Optimization: Models and Applications: Emphasizes process and interpretation throughout Presents a general classification of optimization problems Addresses situations that lead to models illustrating many types of optimization problems Emphasizes model formulations Addresses a special class of problems that can be solved using only elementary calculus Emphasizes model solution and model sensitivity analysis About the author: William P.

He received his Ph. He has written many publications, including over 20 books and over journal articles. Currently, he is an adjunct professor in the Department of Mathematics at the College of William and Mary.

Author : Dimitri P. Arrow,Leonid Hurwicz,Hirofumi Uzawa. Linear and Nonlinear Programming David G. Luenberger,Yinyu Ye. Non-Linear Programming Nita H. Shah,Poonam Prakash Mishra. Studies in linear and non-linear programming ArrowKenneth Joseph. Studies in linear and non-linear programming, by K.

Arrow, L. Hurwicz Kenneth Joseph Arrow. Linear and Nonlinear Optimization Richard W. Nonlinear Optimization Andrzej Ruszczynski. Nonlinear Programming Mokhtar S. Nash,Ariela Sofer.