extract package

The Extract Package handles the various plugins available for extracting face sets in Faceswap.

plugins.extract.base Module

Interfaces for Faceswap extract plugins

class plugins.extract.base.ExtractPlugin(input_size: int, batch_size: int = 1, is_rgb: bool = False, dtype: str = 'float32', scale: tuple[int, int] = (0, 1), force_cpu: bool = False)

Base extract plugin that all plugins must inherit from.

Parameters:

• input_size (int) – The size of the input required by the plugin. The input will always be square at these dimensions

• batch_size (int) – The batch size that the plugin processes data at. Note: Only the process method is guaranteed to receive data at this batch size (or less). The other processes may receive higher batch sizes for re-processing reasons. Do not rely on this when processing data. Default: 1

• is_rgb (bool) – True if the plugin expects input images to be RGB rather than BGR. Default: False

• dtype (str) – A valid datatype that the plugin expects to receive the image at. Default: “float32”

• scale (tuple[int, int]) – The scale that the plugin expects to receive the image at eg: (0, 255) for uint8 images. Default: (0, 1)

• force_cpu (bool) – For Torch models, force running on the CPU, rather than the accelerated device. Sets the torch.device to device. Default: False

batch_size

The maximum batch size that this plugin’s ‘process’ method will receive

property device: device

The selected device to run torch ops on

dtype

The datatype that the plugin expects images at

from_torch(batch: ndarray) → ndarray

Run inference on a PyTorch model.

This function does not need to be used, however it handles torch backend for better throughput, so it is recommended. Must have used self.load_torch_model to load the Torch model to use this function.

Parameters:

batch (ndarray) – The batch array to feed to the PyTorch model

Return type:

The result from the PyTorch model

input_size

The size of the plugin’s input in pixels

is_rgb

True if the plugin expects RGB images. False for BGR

abstractmethod load_model() → torch.nn.Module | cv2.dnn.Net | T.Any

Override to perform any model initialization code

Return type:

The loaded model that will be accessible from Model

load_torch_model(model: Module, weights_path: str, return_indices: list[int] | None = None) → Module

Load a PyTorch model, apply the weights and pass a warmup batch through

This function does not need to be used, but some default Faceswap optimizations are performed here, so without using this function you will either need to apply them yourself or not have them applied

Parameters:

• model (Module) – The Torch model to load

• weights_path (str) – Full path to the weights file to load

• return_indices (list[int] | None) – If the model outputs multiple items, but you only require some of them, the indices of the required items can be placed here so that when calling from_torch any extra data is not copied from the GPU. Default: None (return all data)

Return type:

The loaded model ready for inference

model: torch.nn.Module | cv2.dnn.Net | T.Any

The loaded model for the plugin

name

The name of the plugin. Derived from the module name

post_process(batch: ndarray) → ndarray[tuple[Any, ...], dtype[float32]]

Override to perform post-processing

Parameters:

batch (ndarray) – This will be the output from the previous ‘process’ step

Returns:

• For detect plugins this must be an (N, M, left, top, right, bottom) bounding boxes for

• detected faces scaled to model input size as float32. N is the batch size, M is the number

• of detections per batch

• For align plugins this must be an (N, 68, 2) float32 array for each (x, y) landmark point

• for each face in the batch. co-ordinates should be normalized to 0.0 to 1.0 range

• For mask plugins this must be an (N, size, size) float32 image in range 0. - 1.0 for each

• face in the batch

• For identity plugins this must be an (N, M) float32 identity embedding

Return type:

ndarray[tuple[Any, …], dtype[float32]]

pre_process(batch: ndarray) → ndarray

Override to perform pre-processing

Parameters:

batch (ndarray) –

For detection plugins, this will be a batch of square, padded, images at model input size in the plugin’s color order, image format and data range.

For align plugins this will be a face detection ROI bounding box (batch size, left, top, right, bottom) as INT32.

For all other plugins this will be a batch of aligned face images at model input size in the plugin’s color order, image format and data range

Returns:

• For align plugins, this should be an adjustment of the detected face’s bounding box to cut

• a square out of the original image for feeding the model. Out of bounds values are allowed,

• as these will be handled. This bounding box will be used to prepare the image at the

• correct size for feeding the model.

• For all other plugins, any pre-processing (eg normalization) should be applied ready for

• feeding the model.

Return type:

ndarray

abstractmethod process(batch: ndarray) → ndarray

Override to perform processing. This is where the model should be called

Parameters:

batch (ndarray) –

For detection plugins, this will be a batch of square, padded, images at model input size in the correct format for feeding the model

For align, mask and identity plugins this will be a batch of square face patches at model input size in the correct format for feeding the model

Returns:

• This can return any numpy array, but it must be a numpy array. For detect plugins that can

• return several results, usually in a list, then this must be an object array

Return type:

ndarray

scale

The numeric range that the plugin expects images to be in

class plugins.extract.base.FacePlugin(input_size: int, batch_size: int = 1, is_rgb: bool = False, dtype: str = 'float32', scale: tuple[int, int] = (0, 1), force_cpu: bool = False, centering: Literal['face', 'head', 'legacy'] = 'face')

Base extract plugin that all plugins that work with aligned faces must inherit from.

Parameters:

• input_size (int) – The size of the input required by the plugin. The input will always be square at these dimensions

• batch_size (int) – The batch size that the plugin processes data at. Note: Only the process method is guaranteed to receive data at this batch size (or less). The other processes may receive higher batch sizes for re-processing reasons. Do not rely on this when processing data. Default: 1

• is_rgb (bool) – True if the plugin expects input images to be RGB rather than BGR. Default: False

• dtype (str) – A valid datatype that the plugin expects to receive the image at. Default: “float32”

• scale (tuple[int, int]) – The scale that the plugin expects to receive the image at eg: (0, 255) for uint8 images. Default: (0, 1)

• force_cpu (bool) – For Torch models, force running on the CPU, rather than the accelerated device. Sets the torch.device to device. Default: False

• centering (CenteringType) – The centering that the mask should be stored at

centering: CenteringType

The aligned centering of the image patch to feed the model

storage_name

Dictionary safe name for storing the serialized data

Type:

str

Classes

ExtractPlugin(input_size[, batch_size, ...])	Base extract plugin that all plugins must inherit from.
FacePlugin(input_size[, batch_size, is_rgb, ...])	Base extract plugin that all plugins that work with aligned faces must inherit from.

Class Inheritance Diagram

align package

plugins.extract.align.cv2_dnn Module

CV2 DNN landmarks extractor for faceswap.py Adapted from: https://github.com/yinguobing/cnn-facial-landmark MIT License

Copyright (c) 2017 Yin Guobing

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the “Software”), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

class plugins.extract.align.cv2_dnn.CV2DNNAlign

CV2 DNN Plugin for face alignment

load_model() → Net

Load the CV2 DNN Aligner Model

Return type:

The loaded cv2-DNN model

model: cv2.dnn.Net

The loaded model for the plugin

pre_process(batch: ndarray) → ndarray

Format the ROI faces detection boxes for prediction

Parameters:

batch (ndarray) – The batch of face detection bounding boxes as (bs, l, t, r, b)

Return type:

The face detection bounding boxes formatted to take an image patch for prediction

process(batch: ndarray) → ndarray

Predict the 68 point landmarks

Parameters:

• feed – The batch to feed into the aligner

• batch (ndarray)

Return type:

The predictions from the aligner

Classes

CV2DNNAlign()

CV2 DNN Plugin for face alignment

Class Inheritance Diagram

plugins.extract.align.dark_decoder Module

DARK heatmap decoding for heatmap based aligners.

class plugins.extract.align.dark_decoder.Dark(num_points: int, size: int, blur_kernel: int = 11)

Dark heatmap decoding

https://github.com/ilovepose/DarkPose

Parameters:

• num_points (int) – The number of landmarks output from the model

• size (int) – The size of the heatmap

• blur_kernel (int)

__call__(heatmap: ndarray)

Call self as a function.

Parameters:

heatmap (ndarray)

gaussian_blur(heatmap: ndarray) → ndarray

Perform gaussian blurring on the heatmaps

Parameters:

heatmap (ndarray) – Batch of heatmaps to blur (N, points, size, size)

Return type:

The blurred heatmaps

get_max_preds(batch_heatmaps: ndarray) → ndarray

get predictions from score maps

Parameters:

• heatmaps – Heatmap to derive points from ([batch_size, num_joints, height, width])

• batch_heatmaps (ndarray)

Returns:

The derived points from the heatmaps (B, N, 2)

Return type:

coords

taylor(heatmap: ndarray, coords: ndarray) → ndarray

Sub-pixel refine the predictions

Parameters:

• heatmap (ndarray) – The processed heatmaps for refinement

• coords (ndarray) – The coordinates to be refined

Return type:

The refined coordinates

Classes

Dark(num_points, size[, blur_kernel])

Dark heatmap decoding

Variables

logger

A standard logging.logger with additional "verbose" and "trace" levels added.

Class Inheritance Diagram

plugins.extract.align.fan Module

Facial landmarks extractor for faceswap.py Code adapted and modified from: https://github.com/1adrianb/face-alignment

class plugins.extract.align.fan.ConvBlock(num_in: int, num_out: int)

Convolution block for FAN

Parameters:

• num_in (int) – The number of in channels

• num_out (int) – The number of out channels

forward(inputs: Tensor) → Tensor

Forward pass through FAN’s conv block

Parameters:

inputs (Tensor) – Input to the conv block

Return type:

Output from the conv block

class plugins.extract.align.fan.FAN

FAN Face alignment

load_model() → FaceAlignmentNetwork

Load the FAN model

Return type:

The loaded FAN model

model: FaceAlignmentNetwork

The loaded model for the plugin

post_process(batch: ndarray) → ndarray

Process the output from the model

Parameters:

batch (ndarray) – The predictions from the aligner

Return type:

The final landmarks in 0-1 space

pre_process(batch: ndarray) → ndarray

Format the ROI faces detection boxes for prediction

Parameters:

batch (ndarray) – The batch of face detection bounding boxes as (bs, l, t, r, b)

Return type:

The face detection bounding boxes formatted to take an image patch for prediction

process(batch: ndarray) → ndarray

Predict the 68 point landmarks

Parameters:

batch (ndarray) – The batch to feed into the aligner

Return type:

The predictions from the aligner

class plugins.extract.align.fan.FaceAlignmentNetwork(num_stack: int = 4, num_modules: int = 1, hg_depth: int = 4, num_features: int = 256, num_classes: int = 68)

2D FAN alignment for faceswap

Parameters:

• num_stack (int)

• num_modules (int)

• hg_depth (int)

• num_features (int)

• num_classes (int)

forward(inputs: Tensor) → list[Tensor]

Forward pass through FAN face alignment

Parameters:

inputs (Tensor) – Input to FAN

Return type:

Output from FAN

class plugins.extract.align.fan.HourGlass(num_modules: int, depth: int, num_features: int)

Hour-glass module for FAN

Parameters:

• num_modules (int) – The number of modules in the hour-glass network

• depth (int) – The depth of the hour-glass network

• num_features (int) – The number of features to generate

forward(inputs: Tensor) → Tensor

Forward pass through FAN’s hour-glass network

Parameters:

inputs (Tensor) – Input to the hour-glass network

Return type:

Output from the hour-glass network

Classes

ConvBlock(num_in, num_out)	Convolution block for FAN
FAN()	FAN Face alignment
FaceAlignmentNetwork([num_stack, ...])	2D FAN alignment for faceswap
HourGlass(num_modules, depth, num_features)	Hour-glass module for FAN

Class Inheritance Diagram

plugins.extract.align.hrnet Module

Facial landmarks extractor for faceswap.pnt_y Code adapted and modified from: https://github.com/1adrianb/face-alignment

class plugins.extract.align.hrnet.BasicBlock(in_channels: int, out_channels: int, stride: int = 1, downsample: Module | None = None)

Basic block for HRNet

Parameters:

• in_channels (int) – The number of in channels

• out_channels (int) – The number of out channels

• stride (int) – The stride for the first 3x3 conv block. Default: 1

• downsample (nn.Module | None) – The module to use for downsampling or None for no downsample. Default: None

forward(inputs: Tensor) → Tensor

Forward pass through HRNet’s basic block

Parameters:

inputs (Tensor) – Input to the conv block

Return type:

Output from the conv block

class plugins.extract.align.hrnet.BasicBlockAttention(in_channels, out_channels, stride=1, downsample=None)

Custom Basic block for HRNet with Attention

Parameters:

• in_channels – The number of in channels

• out_channels – The number of out channels

• stride – The stride for the first 3x3 conv block. Default: 1

• downsample – The module to use for downsampling or None for no downsample. Default: None

forward(inputs: Tensor) → Tensor

Forward pass through HRNet’s basic block with attention

Parameters:

inputs (Tensor) – Input to the conv block

Return type:

Output from the conv block

class plugins.extract.align.hrnet.Bottleneck(in_channels: int, out_channels: int, stride: int = 1, downsample: Module | None = None)

Bottleneck for HRNet

Parameters:

• in_channels (int) – The number of in channels

• out_channels (int) – The number of out channels

• stride (int) – The stride for the first 3x3 conv block. Default: 1

• downsample (nn.Module | None) – The module to use for downsampling or None for no downsample. Default: None

forward(inputs: Tensor) → Tensor

Forward pass through HRNet’s basic block

Parameters:

inputs (Tensor) – Input to the conv block

Return type:

Output from the conv block

class plugins.extract.align.hrnet.HRNet

HRNet Face alignment

load_model() → HighResolutionNet

Load the HRNet model

Return type:

The loaded HRNet model

model: HighResolutionNet

The loaded model for the plugin

post_process(batch: ndarray) → ndarray

Process the output from the model

Parameters:

batch (ndarray) – The predictions from the aligner

Return type:

The final landmarks in 0-1 space

pre_process(batch: ndarray) → ndarray

Format the ROI faces detection boxes for prediction

Parameters:

batch (ndarray) – The batch of face detection bounding boxes as (bs, l, t, r, b)

Return type:

The face detection bounding boxes formatted to take an image patch for prediction

process(batch: ndarray) → ndarray

Predict the 68 point landmarks

Parameters:

batch (ndarray) – The batch to feed into the aligner

Return type:

The predictions from the aligner

class plugins.extract.align.hrnet.HRNetStageConfig(num_modules: int, num_branches: int, num_blocks: list[int], num_channels: list[int], block: Literal['ATTENTION', 'BLOCK', 'BOTTLENECK'])

Configuration settings for each stage of HRNet

Parameters:

• num_modules (int)

• num_branches (int)

• num_blocks (list[int])

• num_channels (list[int])

• block (Literal['ATTENTION', 'BLOCK', 'BOTTLENECK'])

class plugins.extract.align.hrnet.HighResolutionModule(num_branches: int, block: type[BasicBlockAttention] | type[BasicBlock] | type[Bottleneck], num_blocks: list[int], num_in_channels: list[int], num_channels: list[int], multi_scale_output: bool = True)

High Resolution Module for HRNet

Parameters:

• num_branches (int) – The number of branches to use

• blocks – The block object to use

• num_blocks (list[int]) – The number of blocks in each branch

• num_in_channels (list[int]) – The number of input channels in each branch

• num_channels (list[int]) – The number of channels in each branch

• multi_scale_output (bool) – True to output multi-scaled

• block (type[BasicBlockAttention] | type[BasicBlock] | type[Bottleneck])

forward(inputs: list[Tensor]) → list[Tensor]

Forward pass through the HR Module

Parameters:

inputs (list[Tensor]) – Input to the HR Module

Return type:

Output from the HR Module

get_num_in_channels() → list[int]

Obtain the number of input channels to the module

Return type:

The number of input channels to the module

class plugins.extract.align.hrnet.HighResolutionNet(num_joints: int, final_conv_kernel: int, stage_2_config: HRNetStageConfig, stage_3_config: HRNetStageConfig, stage_4_config: HRNetStageConfig)

The HRNet Landmark Detection model

Parameters:

• num_joints (int) – The number of joints in the model

• final_conv_kernel (int) – Kernel size of the final convolution

• stage_2_config (HRNetStageConfig) – Configuration settings for stage 2 layers

• stage_3_config (HRNetStageConfig) – Configuration settings for stage 3 layers

• stage_4_config (HRNetStageConfig) – Configuration settings for stage 4 layers

forward(inputs: Tensor) → Tensor

Forward pass through HRNet

Parameters:

inputs (Tensor) – Input to HRNet

Return type:

Output from HRNet

Classes

BasicBlock(in_channels, out_channels[, ...])	Basic block for HRNet
BasicBlockAttention(in_channels, out_channels)	Custom Basic block for HRNet with Attention
Bottleneck(in_channels, out_channels[, ...])	Bottleneck for HRNet
HRNet()	HRNet Face alignment
HRNetStageConfig(num_modules, num_branches, ...)	Configuration settings for each stage of HRNet
HighResolutionModule(num_branches, block, ...)	High Resolution Module for HRNet
HighResolutionNet(num_joints, ...)	The HRNet Landmark Detection model

Class Inheritance Diagram

detect package

plugins.extract.detect.cv2_dnn Module

OpenCV DNN Face detection plugin

class plugins.extract.detect.cv2_dnn.CV2DNNDetect

CV2 DNN detector for face recognition

load_model() → Net

Load the CV2 DNN Detector Model

Return type:

The loaded cv2-DNN model

model: cv2.dnn.Net

The loaded model for the plugin

post_process(batch: ndarray) → ndarray

Compile found faces for output

Parameters:

batch (ndarray) – The detection results for the model

Return type:

The processed detection bounding box from the model at model input size

pre_process(batch: ndarray) → ndarray

Compile the detection image(s) for prediction

Parameters:

batch (ndarray) – The input batch of images at model input size in the correct color order

Return type:

The batch of images ready for feeding the model

process(batch: ndarray) → ndarray

Run model to get predictions

Parameters:

batch (ndarray) – A batch of images ready to feed the model

Return type:

The batch of detection results from the model

Classes

CV2DNNDetect()

CV2 DNN detector for face recognition

Class Inheritance Diagram

plugins.extract.detect.mtcnn Module

MTCNN Face detection plugin

class plugins.extract.detect.mtcnn.MTCNN

MTCNN detector for face recognition.

load_model() → MTCNNModel

Load the model

Return type:

The loaded MTCNN model

model: MTCNNModel

The loaded model for the plugin

post_process(batch: ndarray) → ndarray

Remove confidences from output

Parameters:

batch (ndarray) – The detection results for the model

Return type:

The processed detection bounding box from the model at model input size

pre_process(batch: ndarray) → ndarray

Compile the detection image(s) for prediction. No further pre-processing required for MTCNN

Parameters:

batch (ndarray) – The input batch of images at model input size in the correct color order

Return type:

The batch of images ready for feeding the model

process(batch: ndarray) → ndarray

Run model to get predictions

Parameters:

batch (ndarray) – A batch of images ready to feed the model

Return type:

The batch of detection results from the model

class plugins.extract.detect.mtcnn.MTCNNModel(weights_path: list[str], device: device, input_size: int = 640, min_size: int = 20, threshold: list[float] | None = None, factor: float = 0.709)

MTCNN Detector for face alignment

Parameters:

• weights_path (list[str]) – List of paths to the 3 MTCNN subnet weights

• device (torch.device) – The device to run inference on

• input_size (int) – The height, width input size to the model. Default: 640

• min_size (int) – The minimum size of a face to accept as a detection. Default: 20

• threshold (list[float] | None) – List of floats for the three steps, Default: [0.6, 0.7, 0.7]

• factor (float) – The factor used to create a scaling pyramid of face sizes to detect in the image. Default: 0.709

detect_faces(batch: ndarray) → tuple[ndarray, tuple[ndarray]]

Detects faces in an image, and returns bounding boxes and points for them.

Parameters:

batch (ndarray) – The input batch of images to detect face in

Return type:

List of numpy arrays containing the bounding box and 5 point landmarks of detected faces

class plugins.extract.detect.mtcnn.ONet(weights_path: str)

PyTorch O-Net model Definition for MTCNN

Parameters:

weights_path (str) – The path to the torch weights file

forward(inputs: Tensor) → tuple[Tensor, Tensor, Tensor]

Keras O-Network Definition for MTCNN

Parameters:

inputs (Tensor) – The input to ONet

Returns:

• classifier – The result from ONet classifier

• bbox_regress – The result from ONet bbox regression

• landmark_regress – The result from ONet landmark regression

Return type:

tuple[Tensor, Tensor, Tensor]

class plugins.extract.detect.mtcnn.ONetRunner(weights_path: str, device: device, input_size: int, threshold: float)

Keras O-Net model for MTCNN

Parameters:

• weights_path (str) – The path to the torch model file

• device (torch.device) – The device to run inference on

• input_size (int) – The input size of the model

• threshold (float) – Threshold for O-Net

__call__(images: ndarray, rectangle_batch: list[ndarray]) → list[tuple[ndarray, ndarray]]

Third stage - further refinement and facial landmarks positions with o-net

Parameters:

• images (ndarray) – The batch of images to detect faces in

• rectangle_batch (list[ndarray]) – List of numpy.ndarray face candidates from R-Net

Return type:

List of refined final candidates, scores and landmark points from O-Net

class plugins.extract.detect.mtcnn.PNet(weights_path: str)

PyTorch P-Net model for MTCNN

Parameters:

weights_path (str) – The path to the torch model file

forward(inputs: Tensor) → tuple[Tensor, Tensor]

PyTorch P-Network Definition for MTCNN

Parameters:

inputs (Tensor) – The input tensor to PNet

Returns:

• classifier – The result from PNet classifier

• bbox_regress – The result from PNet bbox regression

Return type:

tuple[Tensor, Tensor]

class plugins.extract.detect.mtcnn.PNetRunner(weights_path: str, device: device, input_size: int, min_size: int, factor: float, threshold: float)

Runner for PyTorch P-Net model for MTCNN

Parameters:

• weights_path (str) – The path to the torch model file

• device (torch.device) – The device to use for model inference

• input_size (int) – The input size of the model

• min_size (int) – The minimum size of a face to accept as a detection. Default: 20

• threshold (float) – Threshold for P-Net

• factor (float)

__call__(images: ndarray) → list[ndarray]

first stage - fast proposal network (p-net) to obtain face candidates

Parameters:

images (ndarray) – The batch of images to detect faces in

Return type:

List of face candidates from P-Net

class plugins.extract.detect.mtcnn.RNet(weights_path: str)

PyTorch R-Net model Definition for MTCNN

Parameters:

weights_path (str) – The path to the torch weights file

forward(inputs: Tensor) → tuple[Tensor, Tensor]

Keras R-Network Definition for MTCNN

Parameters:

inputs (Tensor) – The input to RNet

Returns:

• classifier – The result from RNet classifier

• bbox_regress – The result from RNet bbox regression

Return type:

tuple[Tensor, Tensor]

class plugins.extract.detect.mtcnn.RNetRunner(weights_path: str, device: device, input_size: int, threshold: float)

Runner for PyTorch R-Net for MTCNN

Parameters:

• weights_path (str) – The path to the torch model file

• device (torch.device) – The device to run inference on

• input_size (int) – The input size of the model

• threshold (float) – Threshold for R-Net

__call__(images: ndarray, rectangle_batch: list[ndarray]) → list[ndarray]

second stage - refinement of face candidates with r-net

Parameters:

• images (ndarray) – The batch of images to detect faces in

• rectangle_batch (list[ndarray]) – face candidates from P-Net

Return type:

Refined face candidates from R-Net

plugins.extract.detect.mtcnn.nms(rectangles: ndarray, scores: ndarray, threshold: float, method: str = 'iom') → tuple[ndarray, ndarray]

Apply non-maximum suppression on ROIs in same scale(matrix version)

Parameters:

• rectangles (ndarray) – The [b, l, t, r, b] bounding box detection candidates

• threshold (float) – Threshold for successful match

• method (str) – “iom” method or default. default: “iom”

• scores (ndarray)

Returns:

• rectangles – The [b, l, t, r, b] bounding boxes

• scores – The associated scores for the rectangles

Return type:

tuple[ndarray, ndarray]

plugins.extract.detect.mtcnn.rect2square(rectangles: ndarray) → ndarray

change rectangles into squares (matrix version)

Parameters:

rectangles (ndarray) – [b, x, y, x1, y1] rectangles

Return type:

Original rectangle changed to a square

Functions

nms(rectangles, scores, threshold[, method])	Apply non-maximum suppression on ROIs in same scale(matrix version)
rect2square(rectangles)	change rectangles into squares (matrix version)

Classes

MTCNN()	MTCNN detector for face recognition.
MTCNNModel(weights_path, device[, ...])	MTCNN Detector for face alignment
ONet(weights_path)	PyTorch O-Net model Definition for MTCNN
ONetRunner(weights_path, device, input_size, ...)	Keras O-Net model for MTCNN
PNet(weights_path)	PyTorch P-Net model for MTCNN
PNetRunner(weights_path, device, input_size, ...)	Runner for PyTorch P-Net model for MTCNN
RNet(weights_path)	PyTorch R-Net model Definition for MTCNN
RNetRunner(weights_path, device, input_size, ...)	Runner for PyTorch R-Net for MTCNN

Class Inheritance Diagram

plugins.extract.detect.retinaface Module

Retina face detector adapted from: https://github.com/biubug6/Pytorch_Retinaface

MIT License

Copyright (c) 2019 Sefik Ilkin Serengil

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the “Software”), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

class plugins.extract.detect.retinaface.BboxHead(in_channels: int = 512, num_anchors: int = 3)

Bounding Box Head Module for RetinaFace

Parameters:

• in_channels (int) – The number of input channels. Default: 512

• num_anchors (int) – The number of anchors. Default: 3

forward(inputs: Tensor) → Tensor

Forward pass through BboxHead Module

Parameters:

inputs (Tensor) – The input to the BboxHead Module

Return type:

The output from BboxHead Module

class plugins.extract.detect.retinaface.ClassHead(in_channels: int = 512, num_anchors: int = 3)

Class Head Module for RetinaFace

Parameters:

• in_channels (int) – The number of input channels. Default: 512

• num_anchors (int) – The number of anchors. Default: 3

forward(inputs: Tensor) → Tensor

Forward pass through ClassHead Module

Parameters:

inputs (Tensor) – The input to the ClassHead Module

Return type:

The output from ClassHead Module

class plugins.extract.detect.retinaface.FPN(in_channels_list: list[int], out_channels: int)

FPN Module for RetinaFace

Parameters:

• in_channels_list (list[int]) – The number of input channels

• out_channels (int) – The number of output channels

forward(inputs: Tensor) → list[Tensor]

Forward pass through FPN Module

Parameters:

inputs (Tensor) – The input to the FPN Module

Return type:

The output from FPN Module

class plugins.extract.detect.retinaface.MobileNetV1

MobileNet V1 for use with RetinaFace

forward(inputs: Tensor) → Tensor

Forward pass through MobileNetV1

Parameters:

inputs (Tensor) – The input to MobileNetV1

Return type:

The output from MobileNetV1

class plugins.extract.detect.retinaface.RetinaFace

RetinaFace detector for face detection

load_model() → RetinaFaceModel

Initialize RetinaFace Model

Return type:

The loaded RetinaFace model

model: RetinaFaceModel

The loaded model for the plugin

post_process(batch: ndarray) → ndarray

Process the output from the model to bounding boxes

Parameters:

batch (ndarray) – The output predictions from the S3FD model

Return type:

The processed detection bounding box from the model at model input size

pre_process(batch: ndarray) → ndarray

Compile the detection image(s) for prediction

Parameters:

batch (ndarray) – The input batch of images at model input size in the correct color order, dtype and scale

Return type:

The batch of images ready for feeding the model

process(batch: ndarray) → ndarray

Run model to get predictions

Parameters:

batch (ndarray) – A batch of images ready to feed the model

Return type:

The batch of detection results from the model

class plugins.extract.detect.retinaface.RetinaFaceModel(backbone: Literal['mobilenet', 'resnet'])

RetinaFace Model

Parameters:

backbone (T.Literal['mobilenet', 'resnet']) – The backbone to use for RetinaFace

forward(inputs: Tensor) → tuple[Tensor, Tensor]

Forward pass through RetinaFace

Parameters:

inputs (Tensor) – The input to the RetinaFace Module

Return type:

The output from RetinaFace Module

class plugins.extract.detect.retinaface.SSH(in_channels: int, out_channel: int)

SSH Module for RetinaFace

Parameters:

• in_channels (int) – The number of input channels

• out_channels – The number of output channels

• out_channel (int)

forward(inputs: Tensor) → Tensor

Forward pass through SSH Module

Parameters:

inputs (Tensor) – The input to the SSH Module

Return type:

The output from SSH Module

plugins.extract.detect.retinaface.conv_bn(in_channels: int, out_channels: int, kernel: int = 3, stride: int = 1, padding: int = 1, use_relu: bool = False, leaky: float = 0.0) → Sequential

Generates a Conv Batch Norm sequential module for RetinaFace

Parameters:

• in_channels (int) – The number of input channels

• out_channels (int) – The number of output channels

• kernel (int) – The kernel size. Default: 3

• stride (int) – The number of strides. Default: 1

• padding (int) – The padding to apply. Default: 1

• use_relu (bool) – True to use LeakyReLU activation

• leaky (float) – The negative float value for the LeakyReLU. Default: 0.0

Return type:

The built sequential module

plugins.extract.detect.retinaface.conv_dw(in_channels: int, out_channels: int, stride: int, leaky=0.1) → Sequential

Generates a double Conv Batch Norm sequential module for RetinaFace

Parameters:

• in_channels (int) – The number of input channels

• out_channels (int) – The number of output channels

• stride (int) – The number of strides. Default: 1

• leaky – The negative float value for the LeakyReLU. Default: 0.0

Return type:

The built sequential module

Functions

conv_bn(in_channels, out_channels[, kernel, ...])	Generates a Conv Batch Norm sequential module for RetinaFace
conv_dw(in_channels, out_channels, stride[, ...])	Generates a double Conv Batch Norm sequential module for RetinaFace

Classes

BboxHead([in_channels, num_anchors])	Bounding Box Head Module for RetinaFace
ClassHead([in_channels, num_anchors])	Class Head Module for RetinaFace
FPN(in_channels_list, out_channels)	FPN Module for RetinaFace
MobileNetV1()	MobileNet V1 for use with RetinaFace
RetinaFace()	RetinaFace detector for face detection
RetinaFaceModel(backbone)	RetinaFace Model
SSH(in_channels, out_channel)	SSH Module for RetinaFace

Class Inheritance Diagram

plugins.extract.detect.s3fd Module

S3FD Face detection plugin https://arxiv.org/abs/1708.05237

Adapted from S3FD Port in FAN: https://github.com/1adrianb/face-alignment

class plugins.extract.detect.s3fd.L2Norm(n_channels: int, scale: float)

L2 Normalization layer for S3FD.

Parameters:

• n_channels (int) – The number of channels to normalize

• scale (float) – The scaling for initial weights. Default: 1.0

forward(inputs: Tensor)

Call the L2 Normalization Layer.

Parameters:

inputs (Tensor) – The input to the L2 Normalization Layer

Return type:

The output from the L2 Normalization Layer

class plugins.extract.detect.s3fd.S3FD

S3FD detector for face detection

static decode(location: ndarray, priors: ndarray) → ndarray

Decode locations from predictions using priors to undo the encoding we did for offset regression at train time.

Parameters:

• location (ndarray) – location predictions for location layers,

• priors (ndarray) – Prior boxes in center-offset form.

Return type:

Decoded bounding box predictions

load_model() → S3FDModel

Load the S3FD Model

Return type:

The loaded S3FD model

model: S3FDModel

The loaded model for the plugin

post_process(batch: ndarray) → ndarray

Process the output from the model to bounding boxes

Parameters:

batch (ndarray) – The output predictions from the S3FD model

Return type:

The processed detection bounding box from the model at model input size

pre_process(batch: ndarray) → ndarray

Compile the detection image(s) for prediction

Parameters:

batch (ndarray) – The input batch of images at model input size in the correct color order, dtype and scale

Return type:

The batch of images ready for feeding the model

process(batch: ndarray) → ndarray

Run model to get predictions

Parameters:

batch (ndarray) – A batch of images ready to feed the model

Return type:

The batch of detection results from the model

class plugins.extract.detect.s3fd.S3FDModel

The S3FD Model, adapted from https://github.com/1adrianb/face-alignment

forward(inputs: Tensor) → list[Tensor]

Run the forward pass through S3FD

Parameters:

inputs (Tensor) – The (N, C, H, W) batch of images to process

Return type:

The predictions from the S3FD model

Classes

L2Norm(n_channels, scale)	L2 Normalization layer for S3FD.
S3FD()	S3FD detector for face detection
S3FDModel()	The S3FD Model, adapted from https://github.com/1adrianb/face-alignment

Class Inheritance Diagram

mask package

plugins.extract.mask.bisenet_fp Module

BiSeNet Face-Parsing mask plugin

Architecture and Pre-Trained Model ported from PyTorch to Keras by TorzDF from https://github.com/zllrunning/face-parsing.PyTorch

class plugins.extract.mask.bisenet_fp.AttentionRefinementModule(in_channels: int, filters: int)

The Attention Refinement block for BiSeNet Face Parsing

Parameters:

• in_channels (int) – The number of input channels to the block

• filters (int) – The dimensionality of the output space (i.e. the number of output filters in the convolution).

forward(inputs: Tensor) → Tensor

Call the Attention Refinement block.

Parameters:

inputs (Tensor) – The input to the block

Return type:

The output from the block

class plugins.extract.mask.bisenet_fp.BasicBlock(in_channels: int, filters: int, stride: int = 1)

The basic building block for ResNet 18.

Parameters:

• in_channels (int) – The number of input channels

• filters (int) – The dimensionality of the output space (i.e. the number of output filters in the convolution).

• stride (int) – The strides of the convolution along the height and width. Default: 1

forward(inputs: Tensor) → Tensor

Call the ResNet 18 basic block.

Parameters:

inputs (Tensor) – The input to the block

Return type:

The output from the block

class plugins.extract.mask.bisenet_fp.BiSeNet(num_classes: int)

BiSeNet Face-Parsing Mask from https://github.com/zllrunning/face-parsing.PyTorch

PyTorch model implemented in Keras and then back to pytorch by TorzDF, because why not?

Parameters:

num_classes (int) – The number of segmentation classes to create

forward(inputs: Tensor) → tuple[Tensor, Tensor, Tensor]

Get predictions from the BiSeNet-FP model

Parameters:

inputs (Tensor) – The input to BiSeNet-FP

Return type:

The outputs from BiSeNet-FP

class plugins.extract.mask.bisenet_fp.BiSeNetFP

Neural network to process face image into a segmentation mask of the face

load_model() → BiSeNet

Initialize the BiSeNet Face Parsing model.

Return type:

The loaded BiSeNetFP model

model: BiSeNet

The loaded model for the plugin

post_process(batch: ndarray) → ndarray

Process the output from the model

Parameters:

batch (ndarray) – The predictions from the masker

Return type:

The final masks

pre_process(batch: ndarray) → ndarray

Format the detected faces for prediction

Parameters:

batch (ndarray) – The batch of aligned faces in the correct format for the model

Return type:

The updated images for feeding the model

process(batch: ndarray) → ndarray

Get the masks from the model

Parameters:

batch (ndarray) – The batch to feed into the masker

Return type:

The predicted masks from the plugin

class plugins.extract.mask.bisenet_fp.BiSeNetOutput(in_channels: int, filters: int, num_classes: int)

The BiSeNet Output block for Face Parsing

Parameters:

• in_channels (int) – The number of input channels

• filters (int) – The dimensionality of the output space (i.e. the number of output filters in the convolution).

• num_class – The number of classes to generate

• num_classes (int)

forward(inputs: Tensor) → Tensor

Call the BiSeNet Output block.

Parameters:

inputs (Tensor) – The input to the block

Return type:

The output from the block

class plugins.extract.mask.bisenet_fp.ContextPath

The Context Path block for BiSeNet Face Parsing.

forward(inputs: Tensor) → tuple[Tensor, Tensor, Tensor]

Call the Context Path block.

Parameters:

inputs (Tensor) – The input to the block

Return type:

The feature outputs from ResNet 18

class plugins.extract.mask.bisenet_fp.ConvBNReLU(in_channels: int, filters: int, kernel_size: int = 3, strides: int = 1, padding: int = 1)

Convolutional 3D with Batch Normalization block.

Parameters:

• in_channels (int) – The number of input channels

• filters (int) – The dimensionality of the output space (i.e. the number of output filters in the convolution).

• kernel_size (int) – The height and width of the 2D convolution window. Default: 3

• strides (int) – The strides of the convolution along the height and width. Default: 1

• padding (int) – The amount of padding to apply prior to the first Convolutional Layer. Default: 1

forward(inputs: Tensor) → Tensor

Call the Convolutional Batch Normalization block.

Parameters:

inputs (Tensor) – The input to the block

Return type:

The output from the block

class plugins.extract.mask.bisenet_fp.FeatureFusionModule(in_channels: int, filters: int)

The Feature Fusion block for BiSeNet Face Parsing

Parameters:

• in_channels (int) – The number of input channels to the module

• filters (int) – The dimensionality of the output space (i.e. the number of output filters in the convolution).

forward(feat_spatial: Tensor, feat_context: Tensor) → Tensor

Call the Feature Fusion block.

Parameters:

• feat_spatial (Tensor) – The spatial features input to the block

• feat_context (Tensor) – The context features input to the block

Return type:

The output from the block

class plugins.extract.mask.bisenet_fp.ResNet18

ResNet 18 block. Used at the start of BiSeNet Face Parsing.

forward(inputs: Tensor) → tuple[Tensor, Tensor, Tensor]

Call the ResNet 18 block.

Parameters:

inputs (Tensor) – The input to the ResNet 18

Return type:

The feature outputs from ResNet 18

Classes

AttentionRefinementModule(in_channels, filters)	The Attention Refinement block for BiSeNet Face Parsing
BasicBlock(in_channels, filters[, stride])	The basic building block for ResNet 18.
BiSeNet(num_classes)	BiSeNet Face-Parsing Mask from https://github.com/zllrunning/face-parsing.PyTorch
BiSeNetFP()	Neural network to process face image into a segmentation mask of the face
BiSeNetOutput(in_channels, filters, num_classes)	The BiSeNet Output block for Face Parsing
ContextPath()	The Context Path block for BiSeNet Face Parsing.
ConvBNReLU(in_channels, filters[, ...])	Convolutional 3D with Batch Normalization block.
FeatureFusionModule(in_channels, filters)	The Feature Fusion block for BiSeNet Face Parsing
ResNet18()	ResNet 18 block.

Class Inheritance Diagram

plugins.extract.mask.custom Module

Custom Mask for faceswap.py

class plugins.extract.mask.custom.Custom

A mask that fills the whole face area with 1s or 0s (depending on user selected settings) for custom editing.

load_model() → None

No model to load, just return

Return type:

None

pre_process(batch: ndarray) → ndarray

Return a zero array of the same shape and dtype as the input array

Parameters:

batch (ndarray) – The batch of aligned faces in the correct format for the model

Return type:

A zero’d array of the same shape and dtype as the input

process(batch: ndarray) → ndarray

Get the masks from the model

Parameters:

batch (ndarray) – The batch to process

Return type:

The processed empty masks

Classes

Custom()

A mask that fills the whole face area with 1s or 0s (depending on user selected settings) for custom editing.

Class Inheritance Diagram

plugins.extract.mask.unet_dfl Module

UNET DFL face mask plugin

Architecture and Pre-Trained Model based on… TernausNet: U-Net with VGG11 Encoder Pre-Trained on ImageNet for Image Segmentation https://arxiv.org/abs/1801.05746 https://github.com/ternaus/TernausNet

Source Implementation and fine-tune training…. https://github.com/iperov/DeepFaceLab/blob/master/nnlib/TernausNet.py

Model file sourced from… https://github.com/iperov/DeepFaceLab/blob/master/nnlib/FANSeg_256_full_face.h5

class plugins.extract.mask.unet_dfl.ConvBlock(in_channels: int, filters: int, recursions: int)

Convolution block for UnetDFL down-scales

Parameters:

• in_channels (int) – The number of input channels to the block

• filters (int) – The number of filters for the convolution

• recursions (int) – The number of convolutions to run

forward(inputs: Tensor) → Tensor

Convolution Block forward pass

Parameters:

inputs (Tensor) – The input to the convolution block

Return type:

The output from the convolution block

class plugins.extract.mask.unet_dfl.DecoderBlock(in_channels: int, middle_channels: int, out_channels: int, relu: bool)

Decoder Block for UnetDFL

Parameters:

• in_channels (int) – The number of input channels to the block

• middle_channels (int) – The number of filters for the first convolution

• out_channels (int) – The number of filters for the second convolution

• relu (bool) – True to use ReLU activation on the first conv. False to use no activation

forward(inputs: Tensor) → Tensor

Decoder block forward pass

Parameters:

inputs (Tensor) – The input to the decoder block

Return type:

The output from the decoder block

class plugins.extract.mask.unet_dfl.UNetDFL

Neural network to process face image into a segmentation mask of the face

load_model() → UnetDFL

Initialize the UNet-DFL Model

Return type:

The loaded UnetDFL model

model: UnetDFL

The loaded model for the plugin

process(batch: ndarray) → ndarray

Get the masks from the model

Parameters:

batch (ndarray) – The batch to feed into the masker

Return type:

The predicted masks from the plugin

class plugins.extract.mask.unet_dfl.UnetDFL

UNet DFL Definition for PyTorch

forward(inputs: Tensor) → Tensor

UnetDFL forward pass

Parameters:

inputs (Tensor) – The input to UnetDFL

Return type:

The output from UnetDFL

Classes

ConvBlock(in_channels, filters, recursions)	Convolution block for UnetDFL down-scales
DecoderBlock(in_channels, middle_channels, ...)	Decoder Block for UnetDFL
UNetDFL()	Neural network to process face image into a segmentation mask of the face
UnetDFL()	UNet DFL Definition for PyTorch

Class Inheritance Diagram

plugins.extract.mask.vgg_clear Module

VGG Clear face mask plugin.

class plugins.extract.mask.vgg_clear.ConvBlock(in_channels: int, filters: int, iterations: int, padding: int = 1)

Convolutional loop with max pooling layer for VGG Clear.

Parameters:

• in_channels (int) – The number of input channels to the model

• filters (int) – The number of filters that should appear in each Conv2D layer

• iterations (int) – The number of consecutive Conv2D layers to create

• padding (int) – The amount of padding to apply to the first convolution

forward(inputs: Tensor) → Tensor

Call the convolutional loop.

Parameters:

inputs (Tensor) – The input tensor to the block

Return type:

The output tensor from the convolutional block

class plugins.extract.mask.vgg_clear.ScorePool(in_channels: int, scale: float, crop: tuple[int, int])

Cropped scaling of the pooling layer.

Parameters:

• in_channels (int) – The number of input channels to the model

• scale (float) – The scaling to apply to the pool

• crop (tuple[int, int]) – The amount of 2D cropping to apply. Tuple of (Left/Top, Right/Bottom) ints

forward(inputs: Tensor) → Tensor

Call the score pool layer.

Parameters:

inputs (Tensor) – The input tensor to the block

Return type:

The output tensor from the block

class plugins.extract.mask.vgg_clear.VGGClear

Neural network to process face image into a segmentation mask of the face

load_model() → VGGClearModel

Initialize the VGG Clear Mask model.

Return type:

The loaded VGGClear model

model: VGGClearModel

The loaded model for the plugin

pre_process(batch: ndarray) → ndarray

Format the detected faces for prediction

Parameters:

batch (ndarray) – The batch of aligned faces in the correct format for the model

Return type:

The updated images for feeding the model

process(batch: ndarray) → ndarray

Get the masks from the model

Parameters:

batch (ndarray) – The batch to feed into the masker

Return type:

The predicted masks from the plugin

class plugins.extract.mask.vgg_clear.VGGClearModel

VGG Clear mask for Faceswap.

Caffe model re-implemented in Keras by Kyle Vrooman. Re-implemented for torch by TorzDF

References

On Face Segmentation, Face Swapping, and Face Perception (https://arxiv.org/abs/1704.06729)

Source Implementation: https://github.com/YuvalNirkin/face_segmentation

Model file sourced from: https://github.com/YuvalNirkin/face_segmentation/releases/download/1.1/face_seg_fcn8s_300_no_aug.zip

forward(inputs: Tensor) → Tensor

Call the VGG Clear Model.

Parameters:

inputs (Tensor) – The input to the model

Return type:

The output from the VGG-Clear model

Classes

ConvBlock(in_channels, filters, iterations)	Convolutional loop with max pooling layer for VGG Clear.
ScorePool(in_channels, scale, crop)	Cropped scaling of the pooling layer.
VGGClear()	Neural network to process face image into a segmentation mask of the face
VGGClearModel()	VGG Clear mask for Faceswap.

Class Inheritance Diagram

plugins.extract.mask.vgg_obstructed Module

VGG Obstructed face mask plugin

class plugins.extract.mask.vgg_obstructed.ConvBlock(in_channels: int, filters: int, iterations: int, padding: int = 1, pool_padding: int = 1)

Convolutional loop with max pooling layer for VGG Obstructed.

Parameters:

• in_channels (int) – The number of input channels to the model

• filters (int) – The number of filters that should appear in each Conv2D layer

• iterations (int) – The number of consecutive Conv2D layers to create

• padding (int) – The amount of padding to apply to the first convolution. Default: 1

• pool_padding (int) – The amount of padding to apply to the max pooling layer. Default: 1

forward(inputs: Tensor) → Tensor

Call the convolutional loop.

Parameters:

inputs (Tensor) – The input tensor to the block

Return type:

The output tensor from the convolutional block

class plugins.extract.mask.vgg_obstructed.ScorePool(in_channels: int, scale: float, crop: tuple[int, int])

Cropped scaling of the pooling layer.

Parameters:

• in_channels (int) – The number of input channels to the model

• scale (float) – The scaling to apply to the pool

• crop (tuple[int, int]) – The amount of 2D cropping to apply. Tuple of (Left/Top, Right/Bottom) ints

forward(inputs: Tensor) → Tensor

Call the score pool layer.

Parameters:

inputs (Tensor) – The input tensor to the block

Return type:

The output tensor from the block

class plugins.extract.mask.vgg_obstructed.VGGObstructed

Neural network to process face image into a segmentation mask of the face

load_model() → VGGObstructedModel

Initialize the VGGObstructed Mask model.

Return type:

The loaded VGGObstructed model

model: VGGObstructedModel

The loaded model for the plugin

pre_process(batch: ndarray) → ndarray

Format the detected faces for prediction

Parameters:

batch (ndarray) – The batch of aligned faces in the correct format for the model

Return type:

The updated images for feeding the model

process(batch: ndarray) → ndarray

Get the masks from the model

Parameters:

batch (ndarray) – The batch to feed into the masker

Return type:

The predicted masks from the plugin

class plugins.extract.mask.vgg_obstructed.VGGObstructedModel

VGG Obstructed mask for Faceswap.

Caffe model re-implemented in Keras by Kyle Vrooman. Re-implemented for Pytorch by TorzDF

References

On Face Segmentation, Face Swapping, and Face Perception (https://arxiv.org/abs/1704.06729) Source Implementation: https://github.com/YuvalNirkin/face_segmentation Model file sourced from: https://github.com/YuvalNirkin/face_segmentation/releases/download/1.0/face_seg_fcn8s.zip

forward(inputs: Tensor) → Tensor

Call the VGG Obstructed Model.

Parameters:

inputs (Tensor) – The input to the model

Return type:

The output from the VGG Obstructed model

Classes

ConvBlock(in_channels, filters, iterations)	Convolutional loop with max pooling layer for VGG Obstructed.
ScorePool(in_channels, scale, crop)	Cropped scaling of the pooling layer.
VGGObstructed()	Neural network to process face image into a segmentation mask of the face
VGGObstructedModel()	VGG Obstructed mask for Faceswap.

Class Inheritance Diagram

identity package

plugins.extract.identity.vggface2 Module

VGGFace inference

class plugins.extract.identity.vggface2.ConvBlock(in_channels: int, filters: int, kernel: int, stride: int = 2)

Convolution block for ResNet50

Parameters:

• in_channels (int) – The number of input channels

• filters (int) – The filters for the 1st and 2nd conv layers in the main path

• kernel (int) – The kernel size of middle conv layer of the block

• stride (int) – The stride length for the first and last convolution

forward(inputs: Tensor) → Tensor

Call the resnet50 ConvBlock

Parameters:

inputs (Tensor) – Input tensor

Return type:

Output tensor from the ConvBlock

class plugins.extract.identity.vggface2.IdentityBlock(in_channels: int, filters: int, kernel: int)

Identity block for ResNet50

Parameters:

• in_channels (int) – The number of input channels

• filters (int) – The filters for the 1st and 2nd conv layers in the main path

• kernel (int) – The kernel size of middle conv layer of the block

forward(inputs: Tensor) → Tensor

Call the resnet50 Identity block

Parameters:

inputs (Tensor) – Input tensor

Return type:

Output tensor from the Identity block

class plugins.extract.identity.vggface2.ResNet50

ResNet50 imported for VGG-Face2 adapted from https://github.com/WeidiXie/Keras-VGGFace2-ResNet50

forward(inputs: Tensor) → Tensor

Call the resnet50 Network

Parameters:

inputs (Tensor) – Input tensor

Return type:

Output tensor from resnet50

class plugins.extract.identity.vggface2.VGGFace2

VGGFace2 feature extraction.

Extracts feature vectors from faces in order to compare similarity.

Notes

Input images should be in BGR Order

Model exported from: https://github.com/WeidiXie/Keras-VGGFace2-ResNet50 which is based on: https://www.robots.ox.ac.uk/~vgg/software/vgg_face/

Licensed under Creative Commons Attribution License. https://creativecommons.org/licenses/by-nc/4.0/

load_model() → VGGFace2Model

Initialize VGG Face 2 Model.

Return type:

The loaded VGGFace2 model

model: VGGFace2Model

The loaded model for the plugin

pre_process(batch: ndarray) → ndarray

Format the detected faces for prediction

Parameters:

batch (ndarray) – The batch of aligned faces in the correct format for the model

Return type:

The updated images for feeding the model

process(batch: ndarray) → ndarray

Get the identity matrix from the model

Parameters:

batch (ndarray) – The batch to feed into the recognition plugin

Return type:

The predictions from the plugin

class plugins.extract.identity.vggface2.VGGFace2Model

VGG-Face 2 model with resnet 50 backbone. Adapted from https://github.com/WeidiXie/Keras-VGGFace2-ResNet50

forward(inputs: Tensor) → Tensor

Forward pass through the VGGFace2 model

Parameters:

inputs (Tensor) – Input to the VGGFace2 Model

Return type:

Output from the VGGFace2 Model

Classes

ConvBlock(in_channels, filters, kernel[, stride])	Convolution block for ResNet50
IdentityBlock(in_channels, filters, kernel)	Identity block for ResNet50
ResNet50()	ResNet50 imported for VGG-Face2 adapted from https://github.com/WeidiXie/Keras-VGGFace2-ResNet50
VGGFace2()	VGGFace2 feature extraction.
VGGFace2Model()	VGG-Face 2 model with resnet 50 backbone.

Class Inheritance Diagram

plugins.extract.identity.t_face Module

Tencent TFace inference

class plugins.extract.identity.t_face.TFace

Tencent TFace with the IR 50 and IR 101 backbones

Extracts feature vectors from faces in order to compare similarity.

From: https://github.com/Tencent/TFace

load_model() → IRNet

Initialize TFace Model

Return type:

The loaded TFace model

model: IRNet

The loaded model for the plugin

pre_process(batch: ndarray) → ndarray

Format the detected faces for prediction

Parameters:

batch (ndarray) – The batch of aligned faces in the correct format for the model

Return type:

The updated images for feeding the model

process(batch: ndarray) → ndarray

Get the identity matrix from the model

Parameters:

batch (ndarray) – The batch to feed into the recognition plugin

Return type:

The predictions from the plugin

Classes

TFace()

Tencent TFace with the IR 50 and IR 101 backbones

Class Inheritance Diagram

extract package

plugins.extract.extract_config Module

Default configurations for extract

plugins.extract.extract_config.load_config(config_file: str | None = None) → _Config

Load the Extraction configuration .ini file

Parameters:

• Default (Path to a custom .ini configuration file to load.)

• file)

• config_file (str | None)

Return type:

The loaded convert config object

Functions

load_config([config_file])

Load the Extraction configuration .ini file

Variables

aligner_distance	A dataclass for storing config items loaded from config.ini files and dynamically assigning and validating that the correct datatype is used.
aligner_features	A dataclass for storing config items loaded from config.ini files and dynamically assigning and validating that the correct datatype is used.
aligner_max_scale	A dataclass for storing config items loaded from config.ini files and dynamically assigning and validating that the correct datatype is used.
aligner_min_scale	A dataclass for storing config items loaded from config.ini files and dynamically assigning and validating that the correct datatype is used.
aligner_roll	A dataclass for storing config items loaded from config.ini files and dynamically assigning and validating that the correct datatype is used.
logger	A standard logging.logger with additional "verbose" and "trace" levels added.
mask_storage_size	A dataclass for storing config items loaded from config.ini files and dynamically assigning and validating that the correct datatype is used.
profile_max_vram	A dataclass for storing config items loaded from config.ini files and dynamically assigning and validating that the correct datatype is used.
profile_num_faces	A dataclass for storing config items loaded from config.ini files and dynamically assigning and validating that the correct datatype is used.
profile_save_config	A dataclass for storing config items loaded from config.ini files and dynamically assigning and validating that the correct datatype is used.
profile_test_time	A dataclass for storing config items loaded from config.ini files and dynamically assigning and validating that the correct datatype is used.
profile_warmup_time	A dataclass for storing config items loaded from config.ini files and dynamically assigning and validating that the correct datatype is used.