Scikit-learn: How to calculate the True Negative

I think you should treat this multi-class classification in a one-vs-the-rest manner (so each 2x2 table i measures the performance of a binary classification problem that whether each obs belongs to label i or not). Consequently, you can calculate the TP, FP, FN, TN for each individual label.

import numpy as np

confusion_matrix = np.array([[2,0,3,4],
                             [0,4,5,1],
                             [1,0,3,2],
                             [5,0,0,4]])

def process_cm(confusion_mat, i=0, to_print=True):
    # i means which class to choose to do one-vs-the-rest calculation
    # rows are actual obs whereas columns are predictions
    TP = confusion_mat[i,i]  # correctly labeled as i
    FP = confusion_mat[:,i].sum() - TP  # incorrectly labeled as i
    FN = confusion_mat[i,:].sum() - TP  # incorrectly labeled as non-i
    TN = confusion_mat.sum().sum() - TP - FP - FN
    if to_print:
        print('TP: {}'.format(TP))
        print('FP: {}'.format(FP))
        print('FN: {}'.format(FN))
        print('TN: {}'.format(TN))
    return TP, FP, FN, TN

for i in range(4):
    print('Calculating 2x2 contigency table for label{}'.format(i))
    process_cm(confusion_matrix, i, to_print=True)

Calculating 2x2 contigency table for label0
TP: 2
FP: 6
FN: 7
TN: 19
Calculating 2x2 contigency table for label1
TP: 4
FP: 0
FN: 6
TN: 24
Calculating 2x2 contigency table for label2
TP: 3
FP: 8
FN: 3
TN: 20
Calculating 2x2 contigency table for label3
TP: 4
FP: 7
FN: 5
TN: 18