As an experiment I am building a keras model to approximate the determinant of a matrix. However, when I run it the loss goes down at every epoch and the validation loss goes up! For example:
8s - loss: 7573.9168 - val_loss: 21831.5428 Epoch 21/50 8s - loss: 7345.0197 - val_loss: 23594.8540 Epoch 22/50 13s - loss: 7087.7454 - val_loss: 24718.3967 Epoch 23/50 7s - loss: 6851.8714 - val_loss: 25624.8609 Epoch 24/50 6s - loss: 6637.8168 - val_loss: 26616.7835 Epoch 25/50 7s - loss: 6446.8898 - val_loss: 28856.9654 Epoch 26/50 7s - loss: 6255.7414 - val_loss: 30122.7924 Epoch 27/50 7s - loss: 6054.5280 - val_loss: 32458.5306 Epoch 28/50 Here is the complete code:
import numpy as np import sys from scipy.stats import pearsonr from scipy.linalg import det from sklearn.model_selection import train_test_split from tqdm import tqdm from sklearn.preprocessing import StandardScaler from sklearn.pipeline import Pipeline import math import tensorflow as tf from keras.models import Sequential from keras.layers import Dense from keras.wrappers.scikit_learn import KerasRegressor from keras import backend as K def baseline_model(): # create model model = Sequential() model.add(Dense(200, input_dim=n**2, kernel_initializer='normal', activation='relu')) model.add(Dense(1, input_dim=n**2)) # model.add(Dense(1, kernel_initializer='normal')) # Compile model model.compile(loss='mean_squared_error', optimizer='adam') return model n = 15 print("Making the input data using seed 7", file=sys.stderr) np.random.seed(7) U = np.random.choice([0, 1], size=(n**2,n)) #U is a random orthogonal matrix X =[] Y =[] # print(U) for i in tqdm(range(100000)): I = np.random.choice(n**2, size = n) # Pick out the random rows and sort the rows of the matrix lexicographically. A = U[I][np.lexsort(np.rot90(U[I]))] X.append(A.ravel()) Y.append(det(A)) X = np.array(X) Y = np.array(Y) print("Data created") estimators = [] estimators.append(('standardize', StandardScaler())) estimators.append(('mlp', KerasRegressor(build_fn=baseline_model, epochs=50, batch_size=32, verbose=2))) pipeline = Pipeline(estimators) X_train, X_test, y_train, y_test = train_test_split(X, Y, train_size=0.75, test_size=0.25) pipeline.fit(X_train, y_train, mlp__validation_split=0.3) How can I stop it overfitting so badly?
Update 1
I tried adding more layers and L_2 regularization. However, it makes little or no difference.
def baseline_model(): # create model model = Sequential() model.add(Dense(n**2, input_dim=n**2, kernel_initializer='glorot_normal', activation='relu')) model.add(Dense(int((n**2)/2.0), kernel_initializer='glorot_normal', activation='relu', kernel_regularizer=regularizers.l2(0.01))) model.add(Dense(int((n**2)/2.0), kernel_initializer='glorot_normal', activation='relu', kernel_regularizer=regularizers.l2(0.01))) model.add(Dense(int((n**2)/2.0), kernel_initializer='glorot_normal', activation='relu', kernel_regularizer=regularizers.l2(0.01))) model.add(Dense(1, kernel_initializer='glorot_normal')) # Compile model model.compile(loss='mean_squared_error', optimizer='adam') return model I increased the number of epochs to 100 and it finishes with:
19s - loss: 788.9504 - val_loss: 18423.2807 Epoch 97/100 24s - loss: 760.2046 - val_loss: 18305.9273 Epoch 98/100 20s - loss: 806.0941 - val_loss: 18174.8706 Epoch 99/100 24s - loss: 780.0487 - val_loss: 18356.7482 Epoch 100/100 27s - loss: 749.2595 - val_loss: 18331.5859 Is it possible to approximate the determinant of a matrix using keras?
1 Answers
Answers 1
I tested your code and got the same result. But let's go into basic understanding of matrix determinant (DET). DET consists of n! products, so you cannot really approximate it with n*n weights in few layers of neural network. This requires number of weights that would not scale to n=15, since 15! is 1307674368000 terms for multiplication in the DET.
0 comments:
Post a Comment