model package

The Model Package handles interfacing with the neural network backend and holds custom objects.

model.backup_restore module

Functions for backing up, restoring and creating model snapshots.

class lib.model.backup_restore.Backup(model_dir, model_name)

Bases: object

Performs the back up of models at each save iteration, and the restoring of models from their back up location.

Parameters:
  • model_dir (str) – The folder that contains the model to be backed up

  • model_name (str) – The name of the model that is to be backed up

static backup_model(full_path)

Backup a model file.

The backed up file is saved with the original filename in the original location with .bk appended to the end of the name.

Parameters:

full_path (str) – The full path to a .h5 model file or a .json state file

restore()

Restores a model from backup.

The original model files are migrated into a folder within the original model folder named <model_name>_archived_<timestamp>. The .bk backup files are then moved to the location of the previously existing model files. Logs that were generated after the the last backup was taken are removed.

snapshot_models(iterations)

Take a snapshot of the model at the current state and back it up.

The snapshot is a copy of the model folder located in the same root location as the original model file, with the number of iterations appended to the end of the folder name.

Parameters:

iterations (int) – The number of iterations that the model has trained when performing the snapshot.

model.initializers module

Module Summary

ConvolutionAware

Initializer that generates orthogonal convolution filters in the Fourier space.

ICNR

ICNR initializer for checkerboard artifact free sub pixel convolution

compute_fans

Computes the number of input and output units for a weight shape.

Custom Initializers for faceswap.py

class lib.model.initializers.ConvolutionAware(eps_std=0.05, seed=None, initialized=False)

Bases: Initializer

Initializer that generates orthogonal convolution filters in the Fourier space. If this initializer is passed a shape that is not 3D or 4D, orthogonal initialization will be used.

Adapted, fixed and optimized from: https://github.com/keras-team/keras-contrib/blob/master/keras_contrib/initializers/convaware.py

Parameters:
  • eps_std (float, optional) – The Standard deviation for the random normal noise used to break symmetry in the inverse Fourier transform. Default: 0.05

  • seed (int, optional) – Used to seed the random generator. Default: None

  • initialized (bool, optional) – This should always be set to False. To avoid Keras re-calculating the values every time the model is loaded, this parameter is internally set on first time initialization. Default:False

Returns:

The modified kernel weights

Return type:

tensor

References

Armen Aghajanyan, https://arxiv.org/abs/1702.06295

get_config()

Return the Convolutional Aware Initializer configuration.

Returns:

The configuration for ICNR Initialization

Return type:

dict

class lib.model.initializers.ICNR(initializer, scale=2)

Bases: Initializer

ICNR initializer for checkerboard artifact free sub pixel convolution

Parameters:
  • initializer (keras.initializers.Initializer) – The initializer used for sub kernels (orthogonal, glorot uniform, etc.)

  • scale (int, optional) – scaling factor of sub pixel convolution (up sampling from 8x8 to 16x16 is scale 2). Default: 2

Returns:

The modified kernel weights

Return type:

tensor

Example

>>> x = conv2d(... weights_initializer=ICNR(initializer=he_uniform(), scale=2))

References

Andrew Aitken et al. Checkerboard artifact free sub-pixel convolution https://arxiv.org/pdf/1707.02937.pdf, https://distill.pub/2016/deconv-checkerboard/

get_config()

Return the ICNR Initializer configuration.

Returns:

The configuration for ICNR Initialization

Return type:

dict

lib.model.initializers.compute_fans(shape, data_format='channels_last')

Computes the number of input and output units for a weight shape.

Ported directly from Keras as the location moves between keras and tensorflow-keras

Parameters:
  • shape (tuple) – shape tuple of integers

  • data_format (str) – Image data format to use for convolution kernels. Note that all kernels in Keras are standardized on the “channels_last” ordering (even when inputs are set to “channels_first”).

Returns:

A tuple of scalars, (fan_in, fan_out).

Return type:

tuple

Raises:

ValueError – In case of invalid data_format argument.

model.layers module

Module Summary

GlobalMinPooling2D

Global minimum pooling operation for spatial data.

GlobalStdDevPooling2D

Global standard deviation pooling operation for spatial data.

KResizeImages

A custom upscale function that uses keras.backend.resize_images to upsample.

L2_normalize

Normalizes a tensor w.r.t.

PixelShuffler

PixelShuffler layer for Keras.

QuickGELU

Applies GELU approximation that is fast but somewhat inaccurate.

ReflectionPadding2D

Reflection-padding layer for 2D input (e.g.

SubPixelUpscaling

Sub-pixel convolutional up-scaling layer.

Swish

Swish Activation Layer implementation for Keras.

Custom Layers for faceswap.py.

class lib.model.layers.GlobalMinPooling2D(*args, **kwargs)

Bases: _GlobalPooling2D

Global minimum pooling operation for spatial data.

call(inputs: Tensor, *args, **kwargs) Tensor

This is where the layer’s logic lives.

Parameters:

inputs (tf.Tensor) – Input tensor, or list/tuple of input tensors

Returns:

A tensor or list/tuple of tensors

Return type:

tensor

class lib.model.layers.GlobalStdDevPooling2D(*args, **kwargs)

Bases: _GlobalPooling2D

Global standard deviation pooling operation for spatial data.

call(inputs: Tensor, *args, **kwargs) Tensor

This is where the layer’s logic lives.

Parameters:

inputs (tensor) – Input tensor, or list/tuple of input tensors

Returns:

A tensor or list/tuple of tensors

Return type:

tensor

class lib.model.layers.KResizeImages(*args, **kwargs)

Bases: Layer

A custom upscale function that uses keras.backend.resize_images to upsample.

Parameters:
  • size (int or float, optional) – The scale to upsample to. Default: 2

  • interpolation (["nearest", "bilinear"], optional) – The interpolation to use. Default: “nearest”

  • kwargs (dict) – The standard Keras Layer keyword arguments (if any)

call(inputs: Tensor, *args, **kwargs) Tensor

Call the upsample layer

Parameters:

inputs (tf.Tensor) – Input tensor, or list/tuple of input tensors

Returns:

A tensor or list/tuple of tensors

Return type:

tf.Tensor

compute_output_shape(input_shape: tuple[int, ...]) tuple[int, ...]

Computes the output shape of the layer.

This is the input shape with size dimensions multiplied by size

Parameters:

input_shape (tuple or list of tuples) – Shape tuple (tuple of integers) or list of shape tuples (one per output tensor of the layer). Shape tuples can include None for free dimensions, instead of an integer.

Returns:

An input shape tuple

Return type:

tuple

get_config() dict[str, Any]

Returns the config of the layer.

Returns:

A python dictionary containing the layer configuration

Return type:

dict

class lib.model.layers.L2_normalize(*args, **kwargs)

Bases: Layer

Normalizes a tensor w.r.t. the L2 norm alongside the specified axis.

Parameters:
  • axis (int) – The axis to perform normalization across

  • kwargs (dict) – The standard Keras Layer keyword arguments (if any)

call(inputs: Tensor, *args, **kwargs) Tensor

This is where the layer’s logic lives.

Parameters:

inputs (tf.Tensor) – Input tensor, or list/tuple of input tensors

Returns:

A tensor or list/tuple of tensors

Return type:

tf.Tensor

get_config() dict[str, Any]

Returns the config of the layer.

A layer config is a Python dictionary (serializable) containing the configuration of a layer. The same layer can be reinstated later (without its trained weights) from this configuration.

The configuration of a layer does not include connectivity information, nor the layer class name. These are handled by Network (one layer of abstraction above).

Returns:

A python dictionary containing the layer configuration

Return type:

dict

class lib.model.layers.PixelShuffler(*args, **kwargs)

Bases: Layer

PixelShuffler layer for Keras.

This layer requires a Convolution2D prior to it, having output filters computed according to the formula \(filters = k * (scale_factor * scale_factor)\) where k is a user defined number of filters (generally larger than 32) and scale_factor is the up-scaling factor (generally 2).

This layer performs the depth to space operation on the convolution filters, and returns a tensor with the size as defined below.

Notes

In practice, it is useful to have a second convolution layer after the PixelShuffler layer to speed up the learning process. However, if you are stacking multiple PixelShuffler blocks, it may increase the number of parameters greatly, so the Convolution layer after PixelShuffler layer can be removed.

Example

>>> # A standard sub-pixel up-scaling block
>>> x = Convolution2D(256, 3, 3, padding="same", activation="relu")(...)
>>> u = PixelShuffler(size=(2, 2))(x)
[Optional]
>>> x = Convolution2D(256, 3, 3, padding="same", activation="relu")(u)
Parameters:
  • size (tuple, optional) – The (h, w) scaling factor for up-scaling. Default: (2, 2)

  • data_format ([“channels_first”, “channels_last”, None], optional) – The data format for the input. Default: None

  • kwargs (dict) – The standard Keras Layer keyword arguments (if any)

References

https://gist.github.com/t-ae/6e1016cc188104d123676ccef3264981

call(inputs: Tensor, *args, **kwargs) Tensor

This is where the layer’s logic lives.

Parameters:

inputs (tf.Tensor) – Input tensor, or list/tuple of input tensors

Returns:

A tensor or list/tuple of tensors

Return type:

tf.Tensor

compute_output_shape(input_shape: tuple[int, ...]) tuple[int, ...]

Computes the output shape of the layer.

Assumes that the layer will be built to match that input shape provided.

Parameters:

input_shape (tuple or list of tuples) – Shape tuple (tuple of integers) or list of shape tuples (one per output tensor of the layer). Shape tuples can include None for free dimensions, instead of an integer.

Returns:

An input shape tuple

Return type:

tuple

get_config() dict[str, Any]

Returns the config of the layer.

A layer config is a Python dictionary (serializable) containing the configuration of a layer. The same layer can be reinstated later (without its trained weights) from this configuration.

The configuration of a layer does not include connectivity information, nor the layer class name. These are handled by Network (one layer of abstraction above).

Returns:

A python dictionary containing the layer configuration

Return type:

dict

class lib.model.layers.QuickGELU(*args, **kwargs)

Bases: Layer

Applies GELU approximation that is fast but somewhat inaccurate.

Parameters:
  • name (str, optional) – The name for the layer. Default: “QuickGELU”

  • kwargs (dict) – The standard Keras Layer keyword arguments (if any)

call(inputs: Tensor, *args, **kwargs) Tensor

Call the QuickGELU layerr

Parameters:

inputs (tf.Tensor) – The input Tensor

Returns:

The output Tensor

Return type:

tf.Tensor

class lib.model.layers.ReflectionPadding2D(*args, **kwargs)

Bases: Layer

Reflection-padding layer for 2D input (e.g. picture).

This layer can add rows and columns at the top, bottom, left and right side of an image tensor.

Parameters:
  • stride (int, optional) – The stride of the following convolution. Default: 2

  • kernel_size (int, optional) – The kernel size of the following convolution. Default: 5

  • kwargs (dict) – The standard Keras Layer keyword arguments (if any)

build(input_shape: Tensor) None

Creates the layer weights.

Must be implemented on all layers that have weights.

Parameters:

input_shape (tf.Tensor) – Keras tensor (future input to layer) or list/tuple of Keras tensors to reference for weight shape computations.

call(inputs: Tensor, *args, **kwargs) Tensor

This is where the layer’s logic lives.

Parameters:

inputs (tf.Tensor) – Input tensor, or list/tuple of input tensors

Returns:

A tensor or list/tuple of tensors

Return type:

tf.Tensor

compute_output_shape(input_shape: tuple[int, ...]) tuple[int, ...]

Computes the output shape of the layer.

Assumes that the layer will be built to match that input shape provided.

Parameters:

input_shape (tuple or list of tuples) – Shape tuple (tuple of integers) or list of shape tuples (one per output tensor of the layer). Shape tuples can include None for free dimensions, instead of an integer.

Returns:

An input shape tuple

Return type:

tuple

get_config() dict[str, Any]

Returns the config of the layer.

A layer config is a Python dictionary (serializable) containing the configuration of a layer. The same layer can be reinstated later (without its trained weights) from this configuration.

The configuration of a layer does not include connectivity information, nor the layer class name. These are handled by Network (one layer of abstraction above).

Returns:

A python dictionary containing the layer configuration

Return type:

dict

class lib.model.layers.SubPixelUpscaling(*args, **kwargs)

Bases: Layer

Sub-pixel convolutional up-scaling layer.

This layer requires a Convolution2D prior to it, having output filters computed according to the formula \(filters = k * (scale_factor * scale_factor)\) where k is a user defined number of filters (generally larger than 32) and scale_factor is the up-scaling factor (generally 2).

This layer performs the depth to space operation on the convolution filters, and returns a tensor with the size as defined below.

Notes

This method is deprecated as it just performs the same as PixelShuffler using explicit Tensorflow ops. The method is kept in the repository to support legacy models that have been created with this layer.

In practice, it is useful to have a second convolution layer after the SubPixelUpscaling layer to speed up the learning process. However, if you are stacking multiple SubPixelUpscaling blocks, it may increase the number of parameters greatly, so the Convolution layer after SubPixelUpscaling layer can be removed.

Example

>>> # A standard sub-pixel up-scaling block
>>> x = Convolution2D(256, 3, 3, padding="same", activation="relu")(...)
>>> u = SubPixelUpscaling(scale_factor=2)(x)
[Optional]
>>> x = Convolution2D(256, 3, 3, padding="same", activation="relu")(u)
Parameters:
  • size (int, optional) – The up-scaling factor. Default: 2

  • data_format ([“channels_first”, “channels_last”, None], optional) – The data format for the input. Default: None

  • kwargs (dict) – The standard Keras Layer keyword arguments (if any)

References

based on the paper “Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network” (https://arxiv.org/abs/1609.05158).

build(input_shape: tuple[int, ...]) None

Creates the layer weights.

Must be implemented on all layers that have weights.

Parameters:

input_shape (tensor) – Keras tensor (future input to layer) or list/tuple of Keras tensors to reference for weight shape computations.

call(inputs: Tensor, *args, **kwargs) Tensor

This is where the layer’s logic lives.

Parameters:

inputs (tf.Tensor) – Input tensor, or list/tuple of input tensors

Returns:

A tensor or list/tuple of tensors

Return type:

tf.Tensor

compute_output_shape(input_shape: tuple[int, ...]) tuple[int, ...]

Computes the output shape of the layer.

Assumes that the layer will be built to match that input shape provided.

Parameters:

input_shape (tuple or list of tuples) – Shape tuple (tuple of integers) or list of shape tuples (one per output tensor of the layer). Shape tuples can include None for free dimensions, instead of an integer.

Returns:

An input shape tuple

Return type:

tuple

get_config() dict[str, Any]

Returns the config of the layer.

A layer config is a Python dictionary (serializable) containing the configuration of a layer. The same layer can be reinstated later (without its trained weights) from this configuration.

The configuration of a layer does not include connectivity information, nor the layer class name. These are handled by Network (one layer of abstraction above).

Returns:

A python dictionary containing the layer configuration

Return type:

dict

class lib.model.layers.Swish(*args, **kwargs)

Bases: Layer

Swish Activation Layer implementation for Keras.

Parameters:
  • beta (float, optional) – The beta value to apply to the activation function. Default: 1.0

  • kwargs (dict) – The standard Keras Layer keyword arguments (if any)

References

Swish: a Self-Gated Activation Function: https://arxiv.org/abs/1710.05941v1

call(inputs, *args, **kwargs)

Call the Swish Activation function.

Parameters:

inputs (tensor) – Input tensor, or list/tuple of input tensors

Returns:

A tensor or list/tuple of tensors

Return type:

tf.Tensor

get_config()

Returns the config of the layer.

Adds the beta to config.

Returns:

A python dictionary containing the layer configuration

Return type:

dict

model.losses module

Module Summary

model.nets module

Module Summary

model.nn_blocks module

Module Summary

Conv2D

A standard Keras Convolution 2D layer with parameters updated to be more appropriate for Faceswap architecture.

Conv2DBlock

A standard Convolution 2D layer which applies user specified configuration to the layer.

Conv2DOutput

A Convolution 2D layer that separates out the activation layer to explicitly set the data type on the activation to float 32 to fully support mixed precision training.

ResidualBlock

Residual block from dfaker.

SeparableConv2DBlock

Seperable Convolution Block.

Upscale2xBlock

Custom hybrid upscale layer for sub-pixel up-scaling.

UpscaleBlock

An upscale layer for sub-pixel up-scaling.

set_config

Set the global configuration parameters from the user's config file.

Neural Network Blocks for faceswap.py.

class lib.model.nn_blocks.Conv2D(*args, **kwargs)

Bases: Conv2D

A standard Keras Convolution 2D layer with parameters updated to be more appropriate for Faceswap architecture.

Parameters are the same, with the same defaults, as a standard keras.layers.Conv2D except where listed below. The default initializer is updated to he_uniform or convolutional aware based on user configuration settings.

Parameters:
  • padding (str, optional) – One of “valid” or “same” (case-insensitive). Default: “same”. Note that “same” is slightly inconsistent across backends with strides != 1, as described here.

  • is_upscale (bool, optional) – True if the convolution is being called from an upscale layer. This causes the instance to check the user configuration options to see if ICNR initialization has been selected and should be applied. This should only be passed in as True from UpscaleBlock layers. Default: False

class lib.model.nn_blocks.Conv2DBlock(filters: int, kernel_size: int | tuple[int, int] = 5, strides: int | tuple[int, int] = 2, padding: str = 'same', normalization: str | None = None, activation: str | None = 'leakyrelu', use_depthwise: bool = False, relu_alpha: float = 0.1, **kwargs)

Bases: object

A standard Convolution 2D layer which applies user specified configuration to the layer.

Adds reflection padding if it has been selected by the user, and other post-processing if requested by the plugin.

Adds instance normalization if requested. Adds a LeakyReLU if a residual block follows.

Parameters:
  • filters (int) – The dimensionality of the output space (i.e. the number of output filters in the convolution)

  • kernel_size (int, optional) – An integer or tuple/list of 2 integers, specifying the height and width of the 2D convolution window. Can be a single integer to specify the same value for all spatial dimensions. NB: If use_depthwise is True then a value must still be provided here, but it will be ignored. Default: 5

  • strides (tuple or int, optional) – An integer or tuple/list of 2 integers, specifying the strides of the convolution along the height and width. Can be a single integer to specify the same value for all spatial dimensions. Default: 2

  • padding (["valid", "same"], optional) – The padding to use. NB: If reflect padding has been selected in the user configuration options, then this argument will be ignored in favor of reflect padding. Default: “same”

  • normalization (str or None, optional) – Normalization to apply after the Convolution Layer. Select one of “batch” or “instance”. Set to None to not apply normalization. Default: None

  • activation (str or None, optional) – The activation function to use. This is applied at the end of the convolution block. Select one of “leakyrelu”, “prelu” or “swish”. Set to None to not apply an activation function. Default: “leakyrelu”

  • use_depthwise (bool, optional) – Set to True to use a Depthwise Convolution 2D layer rather than a standard Convolution 2D layer. Default: False

  • relu_alpha (float) – The alpha to use for LeakyRelu Activation. Default=`0.1`

  • kwargs (dict) – Any additional Keras standard layer keyword arguments to pass to the Convolutional 2D layer

class lib.model.nn_blocks.Conv2DOutput(filters: int, kernel_size: int | tuple[int], activation: str = 'sigmoid', padding: str = 'same', **kwargs)

Bases: object

A Convolution 2D layer that separates out the activation layer to explicitly set the data type on the activation to float 32 to fully support mixed precision training.

The Convolution 2D layer uses default parameters to be more appropriate for Faceswap architecture.

Parameters are the same, with the same defaults, as a standard keras.layers.Conv2D except where listed below. The default initializer is updated to he_uniform or convolutional aware based on user config settings.

Parameters:
  • filters (int) – The dimensionality of the output space (i.e. the number of output filters in the convolution)

  • kernel_size (int or tuple/list of 2 ints) – The height and width of the 2D convolution window. Can be a single integer to specify the same value for all spatial dimensions.

  • activation (str, optional) – The activation function to apply to the output. Default: “sigmoid”

  • padding (str, optional) –

    One of “valid” or “same” (case-insensitive). Default: “same”. Note that “same” is slightly inconsistent across backends with strides != 1, as described here.

  • kwargs (dict) – Any additional Keras standard layer keyword arguments to pass to the Convolutional 2D layer

class lib.model.nn_blocks.DepthwiseConv2D(*args, **kwargs)

Bases: DepthwiseConv2D

A standard Keras Depthwise Convolution 2D layer with parameters updated to be more appropriate for Faceswap architecture.

Parameters are the same, with the same defaults, as a standard keras.layers.DepthwiseConv2D except where listed below. The default initializer is updated to he_uniform or convolutional aware based on user configuration settings.

Parameters:
  • padding (str, optional) –

    One of “valid” or “same” (case-insensitive). Default: “same”. Note that “same” is slightly inconsistent across backends with strides != 1, as described here.

  • is_upscale (bool, optional) – True if the convolution is being called from an upscale layer. This causes the instance to check the user configuration options to see if ICNR initialization has been selected and should be applied. This should only be passed in as True from UpscaleBlock layers. Default: False

class lib.model.nn_blocks.ResidualBlock(filters: int, kernel_size: int | tuple[int, int] = 3, padding: str = 'same', **kwargs)

Bases: object

Residual block from dfaker.

Parameters:
  • filters (int) – The dimensionality of the output space (i.e. the number of output filters in the convolution)

  • kernel_size (int, optional) – An integer or tuple/list of 2 integers, specifying the height and width of the 2D convolution window. Can be a single integer to specify the same value for all spatial dimensions. Default: 3

  • padding (["valid", "same"], optional) – The padding to use. Default: “same”

  • kwargs (dict) – Any additional Keras standard layer keyword arguments to pass to the Convolutional 2D layer

Returns:

The output tensor from the Upscale layer

Return type:

tensor

class lib.model.nn_blocks.SeparableConv2DBlock(filters: int, kernel_size: int | tuple[int, int] = 5, strides: int | tuple[int, int] = 2, **kwargs)

Bases: object

Seperable Convolution Block.

Parameters:
  • filters (int) – The dimensionality of the output space (i.e. the number of output filters in the convolution)

  • kernel_size (int, optional) – An integer or tuple/list of 2 integers, specifying the height and width of the 2D convolution window. Can be a single integer to specify the same value for all spatial dimensions. Default: 5

  • strides (tuple or int, optional) – An integer or tuple/list of 2 integers, specifying the strides of the convolution along the height and width. Can be a single integer to specify the same value for all spatial dimensions. Default: 2

  • kwargs (dict) – Any additional Keras standard layer keyword arguments to pass to the Separable Convolutional 2D layer

class lib.model.nn_blocks.Upscale2xBlock(filters: int, kernel_size: int | tuple[int, int] = 3, padding: str = 'same', activation: str | None = 'leakyrelu', interpolation: str = 'bilinear', sr_ratio: float = 0.5, scale_factor: int = 2, fast: bool = False, **kwargs)

Bases: object

Custom hybrid upscale layer for sub-pixel up-scaling.

Most of up-scaling is approximating lighting gradients which can be accurately achieved using linear fitting. This layer attempts to improve memory consumption by splitting with bilinear and convolutional layers so that the sub-pixel update will get details whilst the bilinear filter will get lighting.

Adds reflection padding if it has been selected by the user, and other post-processing if requested by the plugin.

Parameters:
  • filters (int) – The dimensionality of the output space (i.e. the number of output filters in the convolution)

  • kernel_size (int, optional) – An integer or tuple/list of 2 integers, specifying the height and width of the 2D convolution window. Can be a single integer to specify the same value for all spatial dimensions. Default: 3

  • padding (["valid", "same"], optional) – The padding to use. Default: “same”

  • activation (str or None, optional) – The activation function to use. This is applied at the end of the convolution block. Select one of “leakyrelu”, “prelu” or “swish”. Set to None to not apply an activation function. Default: “leakyrelu”

  • interpolation (["nearest", "bilinear"], optional) – Interpolation to use for up-sampling. Default: “bilinear”

  • scale_factor (int, optional) – The amount to upscale the image. Default: 2

  • sr_ratio (float, optional) – The proportion of super resolution (pixel shuffler) filters to use. Non-fast mode only. Default: 0.5

  • fast (bool, optional) – Use a faster up-scaling method that may appear more rugged. Default: False

  • kwargs (dict) – Any additional Keras standard layer keyword arguments to pass to the Convolutional 2D layer

class lib.model.nn_blocks.UpscaleBlock(filters: int, kernel_size: int | tuple[int, int] = 3, padding: str = 'same', scale_factor: int = 2, normalization: str | None = None, activation: str | None = 'leakyrelu', **kwargs)

Bases: object

An upscale layer for sub-pixel up-scaling.

Adds reflection padding if it has been selected by the user, and other post-processing if requested by the plugin.

Parameters:
  • filters (int) – The dimensionality of the output space (i.e. the number of output filters in the convolution)

  • kernel_size (int, optional) – An integer or tuple/list of 2 integers, specifying the height and width of the 2D convolution window. Can be a single integer to specify the same value for all spatial dimensions. Default: 3

  • padding (["valid", "same"], optional) – The padding to use. NB: If reflect padding has been selected in the user configuration options, then this argument will be ignored in favor of reflect padding. Default: “same”

  • scale_factor (int, optional) – The amount to upscale the image. Default: 2

  • normalization (str or None, optional) – Normalization to apply after the Convolution Layer. Select one of “batch” or “instance”. Set to None to not apply normalization. Default: None

  • activation (str or None, optional) – The activation function to use. This is applied at the end of the convolution block. Select one of “leakyrelu”, “prelu” or “swish”. Set to None to not apply an activation function. Default: “leakyrelu”

  • kwargs (dict) – Any additional Keras standard layer keyword arguments to pass to the Convolutional 2D layer

class lib.model.nn_blocks.UpscaleDNYBlock(filters: int, kernel_size: int | tuple[int, int] = 3, padding: str = 'same', activation: str | None = 'leakyrelu', size: int = 2, interpolation: str = 'bilinear', **kwargs)

Bases: object

Upscale block that implements methodology similar to the Disney Research Paper using an upsampling2D block and 2 x convolutions

Adds reflection padding if it has been selected by the user, and other post-processing if requested by the plugin.

References

https://studios.disneyresearch.com/2020/06/29/high-resolution-neural-face-swapping-for-visual-effects/

Parameters:
  • filters (int) – The dimensionality of the output space (i.e. the number of output filters in the convolution)

  • kernel_size (int, optional) – An integer or tuple/list of 2 integers, specifying the height and width of the 2D convolution window. Can be a single integer to specify the same value for all spatial dimensions. Default: 3

  • activation (str or None, optional) – The activation function to use. This is applied at the end of the convolution block. Select one of “leakyrelu”, “prelu” or “swish”. Set to None to not apply an activation function. Default: “leakyrelu”

  • size (int, optional) – The amount to upscale the image. Default: 2

  • interpolation (["nearest", "bilinear"], optional) – Interpolation to use for up-sampling. Default: “bilinear”

  • kwargs (dict) – Any additional Keras standard layer keyword arguments to pass to the Convolutional 2D layers

class lib.model.nn_blocks.UpscaleResizeImagesBlock(filters: int, kernel_size: int | tuple[int, int] = 3, padding: str = 'same', activation: str | None = 'leakyrelu', scale_factor: int = 2, interpolation: Literal['nearest', 'bilinear'] = 'bilinear')

Bases: object

Upscale block that uses the Keras Backend function resize_images to perform the up scaling Similar in methodology to the Upscale2xBlock

Adds reflection padding if it has been selected by the user, and other post-processing if requested by the plugin.

Parameters:
  • filters (int) – The dimensionality of the output space (i.e. the number of output filters in the convolution)

  • kernel_size (int, optional) – An integer or tuple/list of 2 integers, specifying the height and width of the 2D convolution window. Can be a single integer to specify the same value for all spatial dimensions. Default: 3

  • padding (["valid", "same"], optional) – The padding to use. Default: “same”

  • activation (str or None, optional) – The activation function to use. This is applied at the end of the convolution block. Select one of “leakyrelu”, “prelu” or “swish”. Set to None to not apply an activation function. Default: “leakyrelu”

  • scale_factor (int, optional) – The amount to upscale the image. Default: 2

  • interpolation (["nearest", "bilinear"], optional) – Interpolation to use for up-sampling. Default: “bilinear”

  • kwargs (dict) – Any additional Keras standard layer keyword arguments to pass to the Convolutional 2D layer

lib.model.nn_blocks.set_config(configuration: dict) None

Set the global configuration parameters from the user’s config file.

These options are used when creating layers for new models.

Parameters:

configuration (dict) – The configuration options that exist in the training configuration files that pertain specifically to Custom Faceswap Layers. The keys should be: icnr_init, conv_aware_init and ‘reflect_padding’

model.normalization module

Module Summary

InstanceNormalization

Instance normalization layer (Lei Ba et al, 2016, Ulyanov et al., 2016).

Normalization methods for faceswap.py specific to Tensorflow backend

class lib.model.normalization.AdaInstanceNormalization(*args, **kwargs)

Bases: Layer

Adaptive Instance Normalization Layer for Keras.

Parameters:
  • axis (int, optional) – The axis that should be normalized (typically the features axis). For instance, after a Conv2D layer with data_format=”channels_first”, set axis=1 in InstanceNormalization. Setting axis=None will normalize all values in each instance of the batch. Axis 0 is the batch dimension. axis cannot be set to 0 to avoid errors. Default: None

  • momentum (float, optional) – Momentum for the moving mean and the moving variance. Default: 0.99

  • epsilon (float, optional) – Small float added to variance to avoid dividing by zero. Default: 1e-3

  • center (bool, optional) – If True, add offset of beta to normalized tensor. If False, beta is ignored. Default: True

  • scale (bool, optional) – If True, multiply by gamma. If False, gamma is not used. When the next layer is linear (also e.g. relu), this can be disabled since the scaling will be done by the next layer. Default: True

References

Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization - https://arxiv.org/abs/1703.06868

build(input_shape)

Creates the layer weights.

Parameters:

input_shape (tensor) – Keras tensor (future input to layer) or list/tuple of Keras tensors to reference for weight shape computations.

call(inputs, training=None)

This is where the layer’s logic lives.

Parameters:

inputs (tensor) – Input tensor, or list/tuple of input tensors

Returns:

A tensor or list/tuple of tensors

Return type:

tensor

compute_output_shape(input_shape)

Calculate the output shape from this layer.

Parameters:

input_shape (tuple) – The input shape to the layer

Returns:

The output shape to the layer

Return type:

int

get_config()

Returns the config of the layer.

The Keras configuration for the layer.

Returns:

A python dictionary containing the layer configuration

Return type:

dict

class lib.model.normalization.GroupNormalization(*args, **kwargs)

Bases: Layer

Group Normalization

Parameters:
  • axis (int, optional) – The axis that should be normalized (typically the features axis). For instance, after a Conv2D layer with data_format=”channels_first”, set axis=1 in InstanceNormalization. Setting axis=None will normalize all values in each instance of the batch. Axis 0 is the batch dimension. axis cannot be set to 0 to avoid errors. Default: None

  • gamma_init (str, optional) – Initializer for the gamma weight. Default: “one”

  • beta_init (str, optional) – Initializer for the beta weight. Default “zero”

  • gamma_regularizer (varies, optional) – Optional regularizer for the gamma weight. Default: None

  • beta_regularizer (varies, optional) – Optional regularizer for the beta weight. Default None

  • epsilon (float, optional) – Small float added to variance to avoid dividing by zero. Default: 1e-3

  • group (int, optional) – The group size. Default: 32

  • data_format (["channels_first", "channels_last"], optional) – The required data format. Optional. Default: None

  • kwargs (dict) – Any additional standard Keras Layer key word arguments

References

Shaoanlu GAN: https://github.com/shaoanlu/faceswap-GAN

build(input_shape)

Creates the layer weights.

Parameters:

input_shape (tensor) – Keras tensor (future input to layer) or list/tuple of Keras tensors to reference for weight shape computations.

call(inputs, *args, **kwargs)

This is where the layer’s logic lives.

Parameters:

inputs (tensor) – Input tensor, or list/tuple of input tensors

Returns:

A tensor or list/tuple of tensors

Return type:

tensor

get_config()

Returns the config of the layer.

The Keras configuration for the layer.

Returns:

A python dictionary containing the layer configuration

Return type:

dict

class lib.model.normalization.InstanceNormalization(*args, **kwargs)

Bases: Layer

Instance normalization layer (Lei Ba et al, 2016, Ulyanov et al., 2016).

Normalize the activations of the previous layer at each step, i.e. applies a transformation that maintains the mean activation close to 0 and the activation standard deviation close to 1.

Parameters:
  • axis (int, optional) – The axis that should be normalized (typically the features axis). For instance, after a Conv2D layer with data_format=”channels_first”, set axis=1 in InstanceNormalization. Setting axis=None will normalize all values in each instance of the batch. Axis 0 is the batch dimension. axis cannot be set to 0 to avoid errors. Default: None

  • epsilon (float, optional) – Small float added to variance to avoid dividing by zero. Default: 1e-3

  • center (bool, optional) – If True, add offset of beta to normalized tensor. If False, beta is ignored. Default: True

  • scale (bool, optional) – If True, multiply by gamma. If False, gamma is not used. When the next layer is linear (also e.g. relu), this can be disabled since the scaling will be done by the next layer. Default: True

  • beta_initializer (str, optional) – Initializer for the beta weight. Default: “zeros”

  • gamma_initializer (str, optional) – Initializer for the gamma weight. Default: “ones”

  • beta_regularizer (str, optional) – Optional regularizer for the beta weight. Default: None

  • gamma_regularizer (str, optional) – Optional regularizer for the gamma weight. Default: None

  • beta_constraint (float, optional) – Optional constraint for the beta weight. Default: None

  • gamma_constraint (float, optional) – Optional constraint for the gamma weight. Default: None

References

build(input_shape)

Creates the layer weights.

Parameters:

input_shape (tensor) – Keras tensor (future input to layer) or list/tuple of Keras tensors to reference for weight shape computations.

call(inputs, training=None)

This is where the layer’s logic lives.

Parameters:

inputs (tensor) – Input tensor, or list/tuple of input tensors

Returns:

A tensor or list/tuple of tensors

Return type:

tensor

get_config()

Returns the config of the layer.

A layer config is a Python dictionary (serializable) containing the configuration of a layer. The same layer can be reinstated later (without its trained weights) from this configuration.

The configuration of a layer does not include connectivity information, nor the layer class name. These are handled by Network (one layer of abstraction above).

Returns:

A python dictionary containing the layer configuration

Return type:

dict

class lib.model.normalization.RMSNormalization(*args, **kwargs)

Bases: Layer

Root Mean Square Layer Normalization (Biao Zhang, Rico Sennrich, 2019)

RMSNorm is a simplification of the original layer normalization (LayerNorm). LayerNorm is a regularization technique that might handle the internal covariate shift issue so as to stabilize the layer activations and improve model convergence. It has been proved quite successful in NLP-based model. In some cases, LayerNorm has become an essential component to enable model optimization, such as in the SOTA NMT model Transformer.

RMSNorm simplifies LayerNorm by removing the mean-centering operation, or normalizing layer activations with RMS statistic.

Parameters:
  • axis (int) – The axis to normalize across. Typically this is the features axis. The left-out axes are typically the batch axis/axes. This argument defaults to -1, the last dimension in the input.

  • epsilon (float, optional) – Small float added to variance to avoid dividing by zero. Default: 1e-8

  • partial (float, optional) – Partial multiplier for calculating pRMSNorm. Valid values are between 0.0 and 1.0. Setting to 0.0 or 1.0 disables. Default: 0.0

  • bias (bool, optional) – Whether to use a bias term for RMSNorm. Disabled by default because RMSNorm does not enforce re-centering invariance. Default False

  • kwargs (dict) – Standard keras layer kwargs

References

build(input_shape)

Validate and populate axis

Parameters:

input_shape (tensor) – Keras tensor (future input to layer) or list/tuple of Keras tensors to reference for weight shape computations.

call(inputs, *args, **kwargs)

Call Root Mean Square Layer Normalization

Parameters:

inputs (tensor) – Input tensor, or list/tuple of input tensors

Returns:

A tensor or list/tuple of tensors

Return type:

tensor

compute_output_shape(input_shape)

The output shape of the layer is the same as the input shape.

Parameters:

input_shape (tuple) – The input shape to the layer

Returns:

The output shape to the layer

Return type:

tuple

get_config()

Returns the config of the layer.

A layer config is a Python dictionary (serializable) containing the configuration of a layer. The same layer can be reinstated later (without its trained weights) from this configuration.

The configuration of a layer does not include connectivity information, nor the layer class name. These are handled by Network (one layer of abstraction above).

Returns:

A python dictionary containing the layer configuration

Return type:

dict

model.optimizers module

Module Summary

model.session module

Settings manager for Keras Backend

class lib.model.session.KSession(name: str, model_path: str, model_kwargs: dict | None = None, allow_growth: bool = False, exclude_gpus: list[int] | None = None, cpu_mode: bool = False)

Bases: object

Handles the settings of backend sessions for inference models.

This class acts as a wrapper for various keras.Model() functions, ensuring that actions performed on a model are handled consistently and can be performed in parallel in separate threads.

This is an early implementation of this class, and should be expanded out over time.

Notes

The documentation refers to keras. This is a pseudonym for either keras or tensorflow.keras depending on the backend in use.

Parameters:
  • name (str) – The name of the model that is to be loaded

  • model_path (str) – The path to the keras model file

  • model_kwargs (dict, optional) – Any kwargs that need to be passed to keras.models.load_models(). Default: None

  • allow_growth (bool, optional) – Enable the Tensorflow GPU allow_growth configuration option. This option prevents Tensorflow from allocating all of the GPU VRAM, but can lead to higher fragmentation and slower performance. Default: False

  • exclude_gpus (list, optional) – A list of indices correlating to connected GPUs that Tensorflow should not use. Pass None to not exclude any GPUs. Default: None

  • cpu_mode (bool, optional) – True run the model on CPU. Default: False

append_softmax_activation(layer_index: int = -1) None

Append a softmax activation layer to a model

Occasionally a softmax activation layer needs to be added to a model’s output. This is a convenience function to append this layer to the loaded model.

Parameters:

layer_index (int, optional) – The layer index of the model to select the output from to use as an input to the softmax activation layer. Default: -1 (The final layer of the model)

define_model(function: Callable) None

Defines a model from the given function.

This method acts as a wrapper for keras.models.Model().

Parameters:

function (function) – A function that defines a keras.Model and returns it’s inputs and outputs. The function that generates these results should be passed in, NOT the results themselves, as the function needs to be executed within the correct context.

load_model() None

Loads a model.

This method is a wrapper for keras.models.load_model(). Loads a model and its weights from model_path defined during initialization of this class. Any additional kwargs to be passed to keras.models.load_model() should also be defined during initialization of the class.

For Tensorflow backends, the make_predict_function method is called on the model to make it thread safe.

load_model_weights() None

Load model weights for a defined model inside the correct session.

This method is a wrapper for keras.load_weights(). Once a model has been defined in define_model() this method can be called to load its weights from the model_path defined during initialization of this class.

For Tensorflow backends, the make_predict_function method is called on the model to make it thread safe.

predict(feed: list[numpy.ndarray] | ndarray, batch_size: int | None = None) list[numpy.ndarray] | ndarray

Get predictions from the model.

This method is a wrapper for keras.predict() function. For Tensorflow backends this is a straight call to the predict function.

Parameters:
  • feed (numpy.ndarray or list) – The feed to be provided to the model as input. This should be a numpy.ndarray for single inputs or a list of numpy.ndarray objects for multiple inputs.

  • batchsize (int, optional) – The batch size to run prediction at. Default None

Returns:

The predictions from the model

Return type:

numpy.ndarray