On generalized derivatives, optimality conditions and uniqueness of solutions in nonsmooth optimization

En optimisation les conditions d’optimalité jouent un rôle primordial pour détecter les solutions optimales et leur étude occupe une place significative dans la recherche actuelle. Afin d’exprimer adéquatement des conditions d’optimalité les chercheurs ont introduit diverses notions de dérivées généralisées non seulement pour des fonctions non lisses, mais aussi pour des fonctions à valeurs ensemblistes, dites applications multivoques ou multifonctions. Cette thèse porte sur l’application des deux nouveaux concepts de dérivées généralisées: les ensembles variationnels de Khanh-Tuan et les approximations de Jourani-Thibault, aux problèmes d’optimisation multiobjectif et aux problèmes d’équilibre vectoriel. L’enjeu principal est d’obtenir des conditions d’optimalité du premier et du second ordre pour les problèmes ayant des données multivoques ou univoques non lisses et pas forcément continues, et des conditions assurant l’unicité des solutions dans les problèmes d’équilibre vectoriel.Optimality conditions for nonsmooth optimization have become one of the most important topics in the study of optimization-related problems. Various notions of generalized derivatives have been introduced to establish optimality conditions. Besides establishing optimality conditions, generalized derivatives also is an important tool for studying the local uniqueness of solutions. During the last three decades, these topics have been being developed, generalized and applied to many elds of mathematics by many authors all over the world. The purpose of this thesis is to investigate the above topics. It consists of ve chapters. In Chapter 1, we develop elements of calculus of variational sets for set-valued mappings, which were recently introduced in Khanh and Tuan (2008). Most of the usual calculus rules, from chain and sum rules to rules for unions, intersections, products and other operations on mappings, are established. As applications we provide a direct employment of sum rules to establishing an explicit formula for a variational set of the solution map to a parametrized variational inequality in terms of variational sets of the data. Furthermore, chain rules and sum or product rules are also used to prove optimality conditions for weak solutions of some vector optimization problems. In Chapter 2, we propose notions of higher-order outer and inner radial derivatives of set-valued maps and obtain main calculus rules. Some direct applications of these rules in proving optimality conditions for particular optimization problems are provided. Then, we establish higher-order optimality necessary conditions and sufficient ones for a general set-valued vector optimization problem with inequality constraints. Chapter 3 is devoted to using first and second-order approximations, which were introduced by Jourani and Thibault (1993) and Allali and Amaroq (1997), as generalized derivatives, to establish both necessary and sufficient optimality conditions for various kinds of solutions to nonsmooth vector equilibrium problems with functional constraints. Our rst-order conditions are shown to be applicable in many cases, where existing ones cannot be applied. The second-order conditions are new. In Chapter 4, we consider nonsmooth multi-objective fractional programming on normed spaces. Using rst and second-order approximations as generalized derivatives, rst and second-order optimality conditions are established. For sufficient conditions no convexity is needed. Our results can be applied even in innite dimensional cases involving innitely discontinuousmaps. In Chapter 5, we establish sufficient conditions for the local uniqueness of solutions to nonsmooth strong and weak vector equilibrium problems. Also by using approximations, our results are valid even in cases where the maps involved in the problems suffer innite discontinuity at the considered point