Daily Blog - December 4, 2025

This morning, the kids woke up really early. They were up before me. M keeps saying he wants to wake up earlier so he has more time to play before school. He’s doing it himself. He also is getting better at dressing himself and brushing his hair. It’s nice to see him becoming more independent.

After dropping the kids off at school, I went to the store to pick up a few things. After I got home, I relaxed a bit then made lunch. I had a bacon lettuce wrap. It was pretty good.

In the afternoon, I worked on some Kaggle learning. I read “Approaching (Almost) Any NLP problem on Kaggle” and followed through the notebook. My notes are below.

Notes on “Approaching (Almost) Any NLP problem on Kaggle”

These are my notes from the article “Approaching (Almost) Any NLP problem on Kaggle”. It can be found here.

Problem Setup

This notebook uses a dataset that contains some text from a novel and the author of the novel. The goal is to be able to predict the author when given some text. The dataset has three authors: EAP, HPL, and MWS, so the problem is basically a 3-class classification problem.

Building Basic Models

The first step is to build some basic models to get a baseline. The notebook uses TF-IDF vectorization followed by a simple Logistic Regression as a first attempt:

tfv = TfidfVectorizer(min_df=3, max_features=None, 
                      strip_accents='unicode', analyzer='word', 
                      token_pattern=r'\w{1,}', ngram_range=(1, 3), 
                      use_idf=1, smooth_idf=1, sublinear_tf=1, 
                      stop_words = 'english')
tfv.fit(list(xtrain) + list(xvalid))
xtrain_tfv =  tfv.transform(xtrain)
xvalid_tfv = tfv.transform(xvalid)
clf = LogisticRegression(C=1.0)
clf.fit(xtrain_tfv, ytrain)
predictions = clf.predict(xvalid_tfv)
print("logloss: %0.3f " % multiclass_logloss(yvalid, predictions))

This gives a log loss of 0.626, which is a decent starting point. But we can do better. Instead of using IF-IDF, we can use word counts as features by using CountVectorizer:

ctv = CountVectorizer(analyzer='word', token_pattern=r'\w{1,}', 
                      ngram_range=(1, 3), stop_words = 'english')
ctv.fit(list(xtrain) + list(xvalid))
xtrain_ctv =  ctv.transform(xtrain)
xvalid_ctv = ctv.transform(xvalid)
clf = LogisticRegression(C=1.0)
clf.fit(xtrain_ctv, ytrain)
predictions = clf.predict(xvalid_ctv)
print("logloss: %0.3f " % multiclass_logloss(yvalid, predictions))

Wallah! The log loss improves to 0.528! This is basically a 0.1 improvement just by changing the feature representation from TF-IDF to word counts.

What happens if we try Native Bayes?

clf = MultinomialNB()
clf.fit(xtrain_ctv, ytrain)
predictions = clf.predict(xvalid_ctv)
print("logloss: %0.3f " % multiclass_logloss(yvalid, predictions))

This gives a log loss of 0.578, which is worse than word counts but better than TF-IDF. What if we combine Native Bayes on word counts?

clf = MultinomialNB()
clf.fit(xtrain_ctv, ytrain)
predictions = clf.predict(xvalid_ctv)
print("logloss: %0.3f " % multiclass_logloss(yvalid, predictions))

This gives a log loss of 0.485, which is even better! One more popular algorithm is SVM:

SVM

SVMs are popular for text classification. Let’s see how they do with the TF-IDF features.

svd = decomposition.TruncatedSVD(n_components=120)
svd.fit(xtrain_tfv)
xtrain_svd = svd.transform(xtrain_tfv)
xvalid_svd = svd.transform(xvalid_tfv)

# Scale the data obtained from SVD
scl = preprocessing.StandardScaler()
scl.fit(xtrain_svd)
xtrain_svd_scl = scl.transform(xtrain_svd)
xvalid_svd_scl = scl.transform(xvalid_svd)
clf = SVC(C=1.0, probability=True)
clf.fit(xtrain_svd_scl, ytrain)
predictions = clf.predict_proba(xvalid_svd_scl)
print("logloss: %0.3f " % multiclass_logloss(yvalid, predictions))

This takes twenty minutes to run but gives a log loss of 0.740. So SVM is not going to work with this dataset. Before we finish this section, let’s try one of the most popular algorithms: XGBoost.

XGBoost

XGBoost is a popular algorithm for tabular data. Let’s see how it does with TF-IDF features.

clf = xgb.XGBClassifier(max_depth=7, n_estimators=200, 
                        learning_rate=0.1, nthread=10, 
                        subsample=0.8, colsample_bytree=0.8, 
                       )
clf.fit(xtrain_tfv.tocsc(), ytrain)
predictions = clf.predict_proba(xvalid_tfv.tocsc())
print("logloss: %0.3f " % multiclass_logloss(yvalid, predictions))

This gives a log loss of 0.816, which is worse than all the other models! Of course, this is without hyperparameter tuning.

Grid Search for Hyperparameter Tuning

To improve the XGBoost model, we can do hyperparameter tuning using a grid search. First we get a scorer, and then a pipeline consisting of SVD, scaling and then logistic regression.

mll_scorer = metrics.make_scorer(multiclass_logloss, greater_is_better=False, needs_proba=True)

# Initialize SVD
svd = TruncatedSVD()

# Initialize StandardScaler
scl = preprocessing.StandardScaler()

lr_model = LogisticRegression()

# Create the pipeline
clf = pipeline.Pipeline([('svd', svd),
                              ('scl', scl),
                              ('lr', lr_model)])

Now we can set up the parameter grid for grid search.