Statistical Analysis Stack Exchange
Statistical Analysis Stack Exchange

Questions tagged [non-convex]

19 questions
12
votes
3 answers

Gradient descent on non-convex functions

What situations do we know of where gradient descent can be shown to converge (either to a critical point or to a local/global minima) for non-convex functions? For SGD on non-convex functions, one kind of proof has been reviewed here,…
gradstudent
  • 271
  • 2
  • 9
5
votes
1 answer

Are there any "convex neural networks"?

Are there any neural network training procedure that involves solving a convex problem? Note that I am referring more to MLPs, instead of (multi-class) logistic regression which is a neural network with no hidden layers. I know that for MLPs, if…
Fraïssé
  • 961
  • 2
  • 13
  • 29
4
votes
0 answers

Optimization textbooks for statistics and data analytics

Any statistical analysis, machine learning or data science involves some sort of optimization at the end of the day. I'm looking for good linear and nonlinear optimization textbooks for self learning. Here are some of the criteria (subjective) for…
forecaster
  • 7,349
  • 9
  • 43
  • 81
3
votes
1 answer

Why are K-means and GMM (Gaussian Mixture Models) not suitable for discovering clusters with non-convexs shapes?

I have seen that mainly here and from a lot of resources that K-means and Hello all! Gaussian mixtures are not suitable for detecting clusters with non-convex shapes. I know that because both methods have an assumption about clusters being…
yer
  • 115
  • 5
3
votes
0 answers

If $\ell_0$ regularization can be done via the proximal operator, why are people still using LASSO?

I have just learned that a general framework in constrained optimization is called "proximal gradient optimization". It is interesting that the $\ell_0$ "norm" is also associated with a proximal operator. Hence, one can apply iterative hard…
ArtificiallyIntelligence
  • 558
  • 3
  • 10
3
votes
1 answer

Non-convex optimization without using gradient descent

If we want to optimize a convex function, we could use methods like gradient descent or computing the derivative of the function and equalize to zero so we can obtain the global minimum. But I wonder if it's possible to compute the derivative of a…
dotcsv
  • 35
  • 6
2
votes
0 answers

How to solve a non-convex with equality constraint optimization problem?

I have a non-convex optimization problem with equality constraint, I can derive the KKT conditions, but it seems just one of the KKT conditions is valid. Could you please give some advice on how to determine the optimum value and get the solution?
Andrew
  • 21
  • 1
1
vote
1 answer

How do linear constraints affect the convexity of my OLS-like optimisation problem?

I would like to augment a linear regression (so a convex OLS problem) with some additional constraints on the coefficients to match the subject I'm working on. Having $x\in \mathbb{R}^n$, the solution of my linear regression, and my constraints…
quentin
  • 11
  • 1
1
vote
1 answer

Convergence under large set of learning rates

What is the interpretation of a stochastic optimization problem where a gradient descent algorithm is converging under a wide range of learning rate schedules (including ones with quite large initial values, i.e. when taking large gradient steps)?…
Dionysis M
  • 794
  • 6
  • 17
1
vote
0 answers

How to minimize the sum of Frobenius norm and Nuclear norm

I have to minimize an objective function of the the form : $||X_{s} - Y_{s}D_{s}||_{F}^{2} + ||D_{s}||_{F}^{2} + ||D_{s}||_{*}^{2}$ where $||.||_{F}$ denotes the Frobenius norm and $||.||_{*}$ denotes the nuclear norm given by $||D_{s}||_{*}^{2}…
Upendra01
  • 1,566
  • 4
  • 18
  • 28
1
vote
0 answers

Solving constrained optimization problems with Adam

The adam algorithm has been very successful for solving non-convex optimization problems that appear in deep learning. Are there ways to extend adam to solve constrained optimization problems? Among the papers citing Adam, there are papers based on…
Hari
  • 235
  • 1
  • 9
1
vote
0 answers

How to optimize ratiometric loss function with variance term in it?

I'm training a neural network (or any ML model with non-convex gradient-based optimization) to predict a continuous outcome variable. Currently, I use the mean squared error loss function, i.e., if $y$ is the true outcome and $\hat{y}$ is the model…
adpbw
  • 11
  • 1
1
vote
0 answers

Minimal requirement for reaching a feasible solution in non-convex constrained gradient descent

The problem is as follows: $\max_x f(x) \enspace , \enspace \text{s.t.} \enspace g(x) \leq \alpha $ We can not assume that either $f$ nor $g$ are convex, on the contrary - we can assume they are strictly non-convex. The functions are unknown, but we…
Chen
  • 121
  • 4
0
votes
1 answer

Cost function of neural networks can be non-convex, then why do we use it?

I saw a thread here (Cost function of neural network is non-convex?). After I read this, I am really confused. I am wondering that if the cost function is not convex, and we do backpropagation, then can it be "well-backpropagated?" Because it is not…
JAEMTO
  • 103
  • 3
0
votes
0 answers

Convergence analysis for federated learning using DNN model

In convergence analysis of federated learning (FL), usually, we have an assumption that the loss function is strongly convex. However, when the loss function model is non-convex, e.g., using DNN, I rarely find the convergence analysis for this…
bnbfreak
  • 101
  • 1
1
2