Darkon Documentation

About Darkon

Build Status codecov License PyPI Gitter Codacy Badge

Darkon: Toolkit to Hack Your Deep Learning Models

Darkon is an open source toolkit to understand deep learning models better. Deep learning is often referred as a black-box that is difficult to understand. But, accountability and controllability could be critical to commercialize deep learning models. People often think that high accuracy on prepared dataset is enough to use the model for commercial products. However, well-performing models on prepared dataset often fail in real world usages and cause corner cases to be fixed. Moreover, it is necessary to explain the result to trust the system in some applications such as medical diagnosis, financial decisions, etc. We hope
 Darkon can help you to understand the trained models, which could be used to debug failures, interpret decisions, and so on.

Here, we provide functions to analyze deep learning model decisions easily applicable to any Tensorflow models (other models to be supported later). Influence score can be useful to understand the model through training samples. The score can be used for filtering bad training samples that affects test performance negatively. It is useful to prioritize potential mislabeled examples to be fixed, and debug distribution mismatch between train and test samples. In this version, we have added Grad-CAM and Guided Grad-CAM, which are useful to understand decisions of CNN models.

We will gradually enable technologies to analyze deep learning models easily applicable to your existing projects. More features will be released soon. Feedback and feature request are always welcome, which help us to manage priorities. Please keep your eyes on Darkon.

Dependencies

Installation

Install Darkon alone

pip install darkon

Install with TensorFlow CPU

pip install darkon[tensorflow]

Install with TensorFlow GPU

pip install darkon[tensorflow-gpu]

Communication

License

Apache License 2.0

References

[1] Cook, R. D. and Weisberg, S. “Residuals and influence in regression”, New York: Chapman and Hall, 1982

[2] Koh, P. W. and Liang, P. “Understanding Black-box Predictions via Influence Functions” ICML2017

[3] Pearlmutter, B. A. “Fast exact multiplication by the hessian” Neural Computation, 1994

[4] Agarwal, N., Bullins, B., and Hazan, E. “Second order stochastic optimization in linear time” arXiv preprint arXiv:1602.03943

[5] Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, Dhruv Batra “Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization” ICCV2017

About Darkon

Build Status codecov License PyPI Gitter Codacy Badge

Darkon: Toolkit to Hack Your Deep Learning Models

Darkon is an open source toolkit to understand deep learning models better. Deep learning is often referred as a black-box that is difficult to understand. But, accountability and controllability could be critical to commercialize deep learning models. People often think that high accuracy on prepared dataset is enough to use the model for commercial products. However, well-performing models on prepared dataset often fail in real world usages and cause corner cases to be fixed. Moreover, it is necessary to explain the result to trust the system in some applications such as medical diagnosis, financial decisions, etc. We hope
 Darkon can help you to understand the trained models, which could be used to debug failures, interpret decisions, and so on.

Here, we provide functions to analyze deep learning model decisions easily applicable to any Tensorflow models (other models to be supported later). Influence score can be useful to understand the model through training samples. The score can be used for filtering bad training samples that affects test performance negatively. It is useful to prioritize potential mislabeled examples to be fixed, and debug distribution mismatch between train and test samples. In this version, we have added Grad-CAM and Guided Grad-CAM, which are useful to understand decisions of CNN models.

We will gradually enable technologies to analyze deep learning models easily applicable to your existing projects. More features will be released soon. Feedback and feature request are always welcome, which help us to manage priorities. Please keep your eyes on Darkon.

Dependencies
Installation

Install Darkon alone

pip install darkon

Install with TensorFlow CPU

pip install darkon[tensorflow]

Install with TensorFlow GPU

pip install darkon[tensorflow-gpu]
Communication
License

Apache License 2.0

References

[1] Cook, R. D. and Weisberg, S. “Residuals and influence in regression”, New York: Chapman and Hall, 1982

[2] Koh, P. W. and Liang, P. “Understanding Black-box Predictions via Influence Functions” ICML2017

[3] Pearlmutter, B. A. “Fast exact multiplication by the hessian” Neural Computation, 1994

[4] Agarwal, N., Bullins, B., and Hazan, E. “Second order stochastic optimization in linear time” arXiv preprint arXiv:1602.03943

[5] Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, Dhruv Batra “Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization” ICCV2017

API

influence.influence

References

[1]Pang Wei Koh and Percy Liang “Understanding Black-box Predictions via Influence Functions” ICML2017
class Influence(workspace, feeder, loss_op_train, loss_op_test, x_placeholder, y_placeholder, test_feed_options=None, train_feed_options=None, trainable_variables=None)[source]

Influence Class

Parameters:
  • workspace (str) – Path for workspace directory
  • feeder (InfluenceFeeder) – Dataset feeder
  • loss_op_train (tf.Operation) – Tensor for loss function used for training. it may includes regularization.
  • loss_op_test (tf.Operation) – Tensor for loss function for inference.
  • x_placeholder (tf.Tensor) – Data place holder Tensor from tf.placeholder()
  • y_placeholder (tf.Tensor) – Target place holder Tensor from tf.placeholder()
  • test_feed_options (dict) – Optional parameters to run loss operation in testset
  • train_feed_options (dict) – Optional parameters to run loss operation in trainset
  • trainable_variables (tuple, or list) – Trainable variables to be used If None, all variables are trainable Default: None
upweighting_influence(*args, **kwargs)[source]
Calculate influence score of given training samples that affect on the test samples
Negative value indicates bad effect on the test loss
Parameters:
  • sess (tf.Session) – Tensorflow session
  • test_indices (list) – Test samples to be used. Influence on these samples are calculated.
  • test_batch_size (int) – batch size for test samples
  • approx_params (dict) –

    Parameters for inverse hessian vector product approximation Default:

    {‘scale’: 1e4, ‘damping’: 0.01, ‘num_repeats’: 1, ‘recursion_batch_size’: 10, ‘recursion_depth’: 10000}
  • train_indices (list) – Training samples indices to be calculated.
  • num_total_train_example (int) – Number of total training samples used for training, which might be different from the size of train_indices
  • force_refresh (bool) – If False, it calculates only when test samples and parameters are changed. Default: False
Returns:

numpy.ndarray
upweighting_influence_batch(*args, **kwargs)[source]

Iteratively calculate influence scores for training data sampled by batch sampler Negative value indicates bad effect on the test loss

Parameters:
  • sess (tf.Session) – Tensorflow session
  • test_indices (list) – Test samples to be used. Influence on these samples are calculated.
  • test_batch_size (int) – batch size for test samples
  • approx_params (dict) –

    Parameters for inverse hessian vector product approximation Default:

    {‘scale’: 1e4, ‘damping’: 0.01, ‘num_repeats’: 1, ‘recursion_batch_size’: 10, ‘recursion_depth’: 10000}
  • train_batch_size (int) – Batch size of training samples
  • train_iterations (int) – Number of iterations
  • subsamples (int) – Number of training samples in a batch to be calculated. If -1, all samples are calculated (no subsampling). Default: -1
  • force_refresh (bool) – If False, it calculates only when test samples and parameters are changed. Default: False
Returns:

numpy.ndarray
influence.feeder

Feeding interface for Influence class

class InfluenceFeeder[source]

Bases: object

reset()[source]

reset dataset

test_indices(indices)[source]

test data feeder

Parameters:indices (testing sample index) –
Returns:
  • x (feed input values)
  • y (feed label values)
train_batch(batch_size)[source]

training data feeder by batch sampling

Parameters:batch_size (batch size) –
Returns:
  • xs (feed input values)
  • ys (feed label values)
train_one(index)[source]

single training data feeder

Parameters:index (training sample index) –
Returns:
  • x (feed one input value)
  • y (feed one label value)
gradcam.gradcam

References

[1]Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, Dhruv Batra “Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization” ICCV2017
class Gradcam(x_placeholder, num_classes, featuremap_op_name, predict_op_name=None, graph=None)[source]

Gradcam Class

Parameters:
  • x_placeholder (tf.Tensor) – Data place holder Tensor from tf.placeholder()
  • num_classes (int) – number of classes
  • featuremap_op_name (str) – Operation name of CNN feature map layer To get the list of candidate names, use Gradcam.candidate_featuremap_op_names()
  • predict_op_name (str) – Operation name of prediction layer (decision output) To get the list of candidate names, use Gradcam.candidate_predict_op_names()
  • graph (tf.Graph) – Tensorflow graph
static candidate_featuremap_op_names(sess, graph=None, feed_options=None)[source]

Returns the list of candidates for operation names of CNN feature map layer

Parameters:
  • sess (tf.Session) – Tensorflow session
  • graph (tf.Graph) – Tensorflow graph
  • feed_options (dict) – Optional parameters to graph
Returns:

list – String list of candidates
static candidate_predict_op_names(sess, num_classes, graph=None, feed_options=None)[source]

Returns the list of candidate for operation names of prediction layer

Parameters:
  • sess (tf.Session) – Tensorflow session
  • num_classes (int) – Number of prediction classes
  • graph (tf.Graph) – Tensorflow graph
  • feed_options (dict) – Optional parameters to graph
Returns:

list – String list of candidates
gradcam(sess, input_data, target_index=None, feed_options={})[source]

Calculate Grad-CAM (class activation map) and Guided Grad-CAM for given input on target class

Parameters:
  • sess (tf.Session) – Tensorflow session
  • input_data (numpy.ndarray) – A single input instance
  • target_index (int) – Target class index If None, predicted class index is used
  • feed_options (dict) – Optional parameters to graph
Returns:

dict

Note

Keys in return:
  • gradcam_img: Heatmap overlayed on input
  • guided_gradcam_img: Guided Grad-CAM result
  • heatmap: Heatmap of input on the target class
  • guided_backprop: Guided backprop result
static overlay_gradcam(image, heatmap)[source]

Overlay heatmap on input data

Examples

All of examples in here

darkon.Gradcam

ImageNet, ResNet

Sentence polarity dataset, cnn text classification