Europt 2024
21st Conference on Advances in
Continuous Optimization

Conservation Laws for Gradient Flows

Understanding the geometric properties of gradient descent dynamics is a key ingredient in deciphering the recent success of very large machine learning models. A striking observation is that trained over-parameterized models retain some properties of the optimization initialization. This "implicit bias" is believed to be responsible for some favorable properties of the trained models and could explain their good generalization properties. In this talk I will first rigorously expose the definition and basic properties of "conservation laws", which are maximal sets of independent quantities conserved during gradient flows of a given model (e.g. of a ReLU network with a given architecture) with any training data and any loss. Then I will explain how to find the exact number of these quantities by performing finite-dimensional algebraic manipulations on the Lie algebra generated by the Jacobian of the model. In the specific case of linear and ReLu networks, this procedure recovers the conservation laws known in the literature, and prove that there are no other laws. The associated paper can be found here and the open source code is here. This is a joint work with Sibylle Marcotte and Rémi Gribonval.

Talk title

Voluptatem perferendis sed assumenda voluptatibus. Laudantium molestiae sint. Doloremque odio dolore dolore sit. Quae labore alias ea omnis ex expedita sapiente molestias atque. Optio voluptas et.

Aboriosam inventore dolorem inventore nam est esse. Aperiam voluptatem nisi molestias laborum ut. Porro dignissimos eum. Tempore dolores minus unde est voluptatum incidunt ut aperiam.

Et dolore blanditiis officiis non quod id possimus. Optio non commodi alias sint culpa sapiente nihil ipsa magnam. Qui eum alias provident omnis incidunt aut. Eius et officia corrupti omnis error vel quia omnis velit. In qui debitis autem aperiam voluptates unde sunt et facilis.

Talk title

Voluptatem perferendis sed assumenda voluptatibus. Laudantium molestiae sint. Doloremque odio dolore dolore sit. Quae labore alias ea omnis ex expedita sapiente molestias atque. Optio voluptas et.

Aboriosam inventore dolorem inventore nam est esse. Aperiam voluptatem nisi molestias laborum ut. Porro dignissimos eum. Tempore dolores minus unde est voluptatum incidunt ut aperiam.

Et dolore blanditiis officiis non quod id possimus. Optio non commodi alias sint culpa sapiente nihil ipsa magnam. Qui eum alias provident omnis incidunt aut. Eius et officia corrupti omnis error vel quia omnis velit. In qui debitis autem aperiam voluptates unde sunt et facilis.

Multiobjective optimization, uncertainty, and a bit of set optimization

In many applications one has to deal with various difficulties at the same time, like uncertain data or several competing objective functions. For instance, for the integration of neighborhood networks into overarching distributing energy networks, a pure optimization of the neighborhood networks under an externally defined weighting of the relevant targets does not adequately model the problem. Moreover, uncertainties in the form of fluctuations or other disturbances can appear and a found solution has to be robust against that. A robust approach to uncertain multiobjective optimization corresponds to solving a set-valued optimization problem. However, it is a very difficult task to solve these optimization problems even for specific cases..

In this talk, we give a short introduction and motivation for multiobjective optimization. In case of uncertainties, a family of parametrized multiobjective optimization problems has to be examined and we discuss approaches on how this is done in the literature. Thereby, basic concepts of set optimization are needed, i.e., of optimization with a set-valued objective function. For that reason, we also provide a short introduction to the main optimality concepts and to new solution approaches used in set optimization. We combine everything for obtaining a solvable reformulation of an uncertain multiobjective problem using a robust approach which extends the classical epigraphical reformulation from single-objective optimization to multiple objectives.