Skip to content

Add a tracking of global loss in each epoch #23

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
ducovrossem wants to merge 6 commits into
base: master
Choose a base branch
from
Open

Add a tracking of global loss in each epoch #23

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
a5c1038
Add global loss tracker
ducovrossem Dec 19, 2014
6bce0da
Add global loss computation
ducovrossem Dec 19, 2014
af11285
Verbose dependency and return fit_vectors
ducovrossem Dec 19, 2014
f6c4f9f
fit_vectors returns global_loss
ducovrossem Dec 19, 2014
0ac1497
Indentation level in fit_vector function
ducovrossem Jan 2, 2015
654eb4f
Calculate weightless loss seperately
ducovrossem Jan 2, 2015
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 7 additions & 2 deletions glove/glove.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,8 @@ def __init__(self, no_components=30, learning_rate=0.05,

self.dictionary = None
self.inverse_dictionary = None

self.global_loss = None

def fit(self, matrix, epochs=5, no_threads=2, verbose=False):
"""
Expand Down Expand Up @@ -86,8 +88,8 @@ def fit(self, matrix, epochs=5, no_threads=2, verbose=False):

# Shuffle the coocurrence matrix
np.random.shuffle(shuffle_indices)

fit_vectors(self.word_vectors,
self.global_loss = fit_vectors(self.word_vectors,
self.vectors_sum_gradients,
Copy link
Copy Markdown
Owner

@maciejkula maciejkula Dec 23, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you just fix the indentation of these lines? self.vectors_sum_gradients should be indented to the same level as self.word_vectors.

self.word_biases,
self.biases_sum_gradients,
Expand All @@ -99,6 +101,9 @@ def fit(self, matrix, epochs=5, no_threads=2, verbose=False):
self.max_count,
self.alpha,
int(no_threads))

if verbose:
print('Global loss: %d' % self.global_loss)

if not np.isfinite(self.word_vectors).all():
raise Exception('Non-finite values in word vectors. '
Expand Down
9 changes: 8 additions & 1 deletion glove/glove_cython.pyx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,9 @@ def fit_vectors(double[:, ::1] wordvec,

# Loss and gradient variables.
cdef double prediction, entry_weight, loss

# Define global loss
cdef double global_loss

# Iteration variables
cdef int i, j, shuffle_index
Expand All @@ -75,6 +78,9 @@ def fit_vectors(double[:, ::1] wordvec,
# Compute loss and the example weight.
entry_weight = double_min(1.0, (count / max_count)) ** alpha
loss = entry_weight * (prediction - c_log(count))

# Update the weighted global loss
global_loss += 0.5 * entry_weight * (prediction - c_log(count)) **2
Copy link
Copy Markdown

@piskvorky piskvorky Dec 23, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not factor out the log call? Is this relying on a compiler optimization?

Also, and I'm probably missing something obvious, but why the `0.5 *``?


# Update step: apply gradients and reproject
# onto the unit sphere.
Expand All @@ -100,7 +106,8 @@ def fit_vectors(double[:, ::1] wordvec,
learning_rate = initial_learning_rate / sqrt(wordbias_sum_gradients[word_b])
wordbias[word_b] -= learning_rate * loss
wordbias_sum_gradients[word_b] += loss ** 2


return global_loss

def transform_paragraph(double[:, ::1] wordvec,
double[::1] wordbias,
Expand Down