extract package

The Extract Package handles the various plugins available for extracting face sets in Faceswap.

pipeline module

Module Summary

ExtractMedia An object that passes through the Extractor pipeline.
Extractor Creates a detect/align`/ mask pipeline and yields results frame by frame from the detected_faces generator

Module

Return a requested detector/aligner/masker pipeline

Tensorflow does not like to release GPU VRAM, so parallel plugins need to be managed to work together.

This module sets up a pipeline for the extraction workflow, loading detect, align and mask plugins either in parallel or in series, giving easy access to input and output.

class plugins.extract.pipeline.ExtractMedia(filename, image, detected_faces=None)

Bases: object

An object that passes through the Extractor pipeline.

Parameters:
  • filename (str) – The base name of the original frame’s filename
  • image (numpy.ndarray) – The original frame
  • detected_faces (list, optional) – A list of DetectedFace objects. Detected faces can be added later with add_detected_faces(). Default: None
add_detected_faces(faces)

Add detected faces to the object. Called at the end of each extraction phase.

Parameters:faces (list) – A list of DetectedFace objects
detected_faces

A list of DetectedFace objects in the image.

Type:list
filename

The base name of the image filename.

Type:str
get_image_copy(color_format)

Get a copy of the image in the requested color format.

Parameters:color_format (['BGR', 'RGB', 'GRAY']) – The requested color format of image
Returns:A copy of image in the requested color_format
Return type:numpy.ndarray
image

The source frame for this object.

Type:numpy.ndarray
image_shape

The shape of the stored image.

Type:tuple
image_size

The (height, width) of the stored image.

Type:tuple
remove_image()

Delete the image and reset image to None.

Required for multi-phase extraction to avoid the frames stacking RAM.

set_image(image)

Add the image back into image

Required for multi-phase extraction adds the image back to this object.

Parameters:image (numpy.ndarry) – The original frame to be re-applied to for this filename
class plugins.extract.pipeline.Extractor(detector, aligner, masker, configfile=None, multiprocess=False, exclude_gpus=None, rotate_images=None, min_size=20, normalize_method=None, re_feed=0, image_is_aligned=False)

Bases: object

Creates a detect/align`/ mask pipeline and yields results frame by frame from the detected_faces generator

input_queue is dynamically set depending on the current phase of extraction

Parameters:
  • detector (str) – The name of a detector plugin as exists in plugins.extract.detect
  • aligner (str) – The name of an aligner plugin as exists in plugins.extract.align
  • masker (str or list) – The name of a masker plugin(s) as exists in plugins.extract.mask. This can be a single masker or a list of multiple maskers
  • configfile (str, optional) – The path to a custom extract.ini configfile. If None then the system config/extract.ini file will be used.
  • multiprocess (bool, optional) – Whether to attempt processing the plugins in parallel. This may get overridden internally depending on the plugin combination. Default: False
  • exclude_gpus (list, optional) – A list of indices correlating to connected GPUs that Tensorflow should not use. Pass None to not exclude any GPUs. Default: None
  • rotate_images (str, optional) – Used to set the plugins.extract.detect.rotation attribute. Pass in a single number to use increments of that size up to 360, or pass in a list of ints to enumerate exactly what angles to check. Can also pass in 'on' to increment at 90 degree intervals. Default: None
  • min_size (int, optional) – Used to set the plugins.extract.detect.min_size attribute Filters out faces detected below this size. Length, in pixels across the diagonal of the bounding box. Set to 0 for off. Default: 0
  • normalize_method ({None, ‘clahe’, ‘hist’, ‘mean’}, optional) – Used to set the plugins.extract.align.normalize_method attribute. Normalize the images fed to the aligner.Default: None
  • re_feed (int) – The number of times to re-feed a slightly adjusted bounding box into the aligner. Default: 0
  • image_is_aligned (bool, optional) – Used to set the plugins.extract.mask.image_is_aligned attribute. Indicates to the masker that the fed in image is an aligned face rather than a frame. Default: False
phase

The current phase that the pipeline is running. Used in conjunction with passes and final_pass to indicate to the caller which phase is being processed

Type:str
detected_faces()

Generator that returns results, frame by frame from the extraction pipeline

This is the exit point for the extraction pipeline and is used to obtain the output of any pipeline phase

Yields:faces (ExtractMedia) – The populated extracted media object.

Example

>>> for extract_media in extractor.detected_faces():
>>>     filename = extract_media.filename
>>>     image = extract_media.image
>>>     detected_faces = extract_media.detected_faces
final_pass

bool, Return True if this is the final extractor pass otherwise False

Useful for iterating over the pipeline passes or detected_faces() and handling accordingly.

Example

>>> for face in extractor.detected_faces():
>>>     if extractor.final_pass:
>>>         <do final processing>
>>>     else:
>>>         extract_media.set_image(image)
>>>         <do intermediate processing>
>>>         extractor.input_queue.put(extract_media)
input_queue

Return the correct input queue depending on the current phase

The input queue is the entry point into the extraction pipeline. An ExtractMedia object should be put to the queue.

For detect/single phase operations the ExtractMedia.filename and image attributes should be populated.

For align/mask (2nd/3rd pass operations) the ExtractMedia.detected_faces should also be populated by calling ExtractMedia.set_detected_faces().

Type:queue
launch()

Launches the plugin(s)

This launches the plugins held in the pipeline, and should be called at the beginning of each phase. To ensure VRAM is conserved, It will only launch the plugin(s) required for the currently running phase

Example

>>> for phase in extractor.passes:
>>>     extractor.launch():
>>>         <do processing>
passes

Returns the total number of passes the extractor needs to make.

This is calculated on several factors (vram available, plugin choice, multiprocess etc.). It is useful for iterating over the pipeline and handling accordingly.

Example

>>> for phase in extractor.passes:
>>>     if phase == 1:
>>>         extract_media = ExtractMedia("path/to/image/file", image)
>>>         extractor.input_queue.put(extract_media)
>>>     else:
>>>         extract_media.set_image(image)
>>>         extractor.input_queue.put(extract_media)
Type:int
phase_text

The plugins that are running in the current phase, formatted for info text output.

Type:str
set_aligner_normalization_method(method)

Change the normalization method for faces fed into the aligner.

Parameters:method ({"none", "clahe", "hist", "mean"}) – The normalization method to apply to faces prior to feeding into the aligner’s model
set_batchsize(plugin_type, batchsize)

Set the batch size of a given plugin_type to the given batchsize.

This should be set prior to launch() if the batch size is to be manually overridden

Parameters:
  • plugin_type ({'aligner', 'detector'}) – The plugin_type to be overridden
  • batchsize (int) – The batch size to use for this plugin type

extract plugins package

_base module

Base class for Faceswap detect, align and mask Plugins

class plugins.extract._base.Extractor(git_model_id=None, model_filename=None, exclude_gpus=None, configfile=None, instance=0)

Bases: object

Extractor Plugin Object

All _base classes for Aligners, Detectors and Maskers inherit from this class.

This class sets up a pipeline for working with ML plugins.

Plugins are split into 3 threads, to utilize Numpy and CV2s parallel processing, as well as allow the predict function of the model to sit in a dedicated thread. A plugin is expected to have 3 core functions, each in their own thread: - process_input() - Prepare the data for feeding into a model - predict() - Feed the data through the model - process_output() - Perform any data post-processing

Parameters:
  • git_model_id (int) – The second digit in the github tag that identifies this model. See https://github.com/deepfakes-models/faceswap-models for more information
  • model_filename (str) – The name of the model file to be loaded
  • exclude_gpus (list, optional) – A list of indices correlating to connected GPUs that Tensorflow should not use. Pass None to not exclude any GPUs. Default: None
  • configfile (str, optional) – Path to a custom configuration ini file. Default: Use system configfile
  • instance (int, optional) – If this plugin is being executed multiple times (i.e. multiple pipelines have been launched), the instance of the plugin must be passed in for naming convention reasons. Default: 0

The following attributes should be set in the plugin’s __init__() method after initializing the parent.

name

Name of this plugin. Used for display purposes.

Type:str
input_size

The input size to the model in pixels across one edge. The input size should always be square.

Type:int
color_format

Color format for model. Must be 'BGR', 'RGB' or 'GRAY'. Defaults to 'BGR' if not explicitly set.

Type:str
vram

Approximate VRAM used by the model at input_size. Used to calculate the batchsize. Be conservative to avoid OOM.

Type:int
vram_warnings

Approximate VRAM used by the model at input_size that will still run, but generates warnings. Used to calculate the batchsize. Be conservative to avoid OOM.

Type:int
vram_per_batch

Approximate additional VRAM used by the model for each additional batch. Used to calculate the batchsize. Be conservative to avoid OOM.

Type:int

See also

plugins.extract.detect._base
Detector parent class for extraction plugins.
plugins.extract.align._base
Aligner parent class for extraction plugins.
plugins.extract.mask._base
Masker parent class for extraction plugins.
plugins.extract.pipeline
The extract pipeline that configures and calls all plugins
batchsize = None

Batchsize for feeding this model. The number of images the model should feed through at once.

Type:int
check_and_raise_error()

Check all threads for errors

Exposed for pipeline to check plugin’s threads for errors

config = None

Config for this plugin, loaded from extract.ini configfile

Type:dict
finalize(batch)

Override method (at <plugin_type> level)

This method should be overridden at the <plugin_type> level (IE. plugins.extract.detect._base, plugins.extract.align._base or plugins.extract.mask._base) and should not be overridden within plugins themselves.

Handles consistent finalization for all plugins that exist within that plugin type. Its input is always the output from process_output()

Parameters:batch (dict) – Contains the batch that is currently being passed through the plugin process
get_batch(queue)

Override method (at <plugin_type> level)

This method should be overridden at the <plugin_type> level (IE. plugins.extract.detect._base, plugins.extract.align._base or plugins.extract.mask._base) and should not be overridden within plugins themselves.

Get ExtractMedia items from the queue in batches of batchsize

Parameters:queue (queue.Queue()) – The queue that the batch will be fed from. This will be the input to the plugin.
init_model()

Override method

Override this method to execute the specific model initialization method

initialize(*args, **kwargs)

Initialize the extractor plugin

Should be called from pipeline

join()

Join all threads

Exposed for pipeline to join plugin’s threads

model = None

The model for this plugin. Set in the plugin’s init_model() method

Type:varies
model_path = None

Path to the model file(s) (if required). Multiple model files should be a list of strings

Type:str or list
predict(batch)

Override method

Override this method for specific extractor model prediction function

Parameters:batch (dict) – Contains the batch that is currently being passed through the plugin process

Notes

Input for predict() should have been set in process_input() with the addition of a feed key to the batch dict.

Output from the model should add the key prediction to the batch dict.

For Detect:
the expected output for the prediction key of the batch dict should be a list of batchsize of detected face points. These points should be either a list, tuple or numpy.ndarray with the first 4 items being the left, top, right, bottom points, in that order
process_input(batch)

Override method

Override this method for specific extractor pre-processing of image

Parameters:batch (dict) – Contains the batch that is currently being passed through the plugin process

Notes

When preparing an input to the model a key feed must be added to the batch dict which contains this input.

process_output(batch)

Override method

Override this method for specific extractor model post predict function

Parameters:batch (dict) – Contains the batch that is currently being passed through the plugin process

Notes

For Align:
The key landmarks must be returned in the batch dict from this method. This should be a list or numpy.ndarray of batchsize containing a list, tuple or numpy.ndarray of (x, y) coordinates of the 68 point landmarks as calculated from the model.
queue_size = None

Queue size for all internal queues. Set in initialize()

Type:int
start()

Start all threads

Exposed for pipeline to start plugin’s threads

detect._base module

Base class for Face Detector plugins

All Detector Plugins should inherit from this class. See the override methods for which methods are required.

The plugin will receive a ExtractMedia object.

For each source frame, the plugin must pass a dict to finalize containing:

>>> {'filename': <filename of source frame>,
>>>  'detected_faces': <list of DetectedFace objects containing bounding box points}}

To get a DetectedFace object use the function:

>>> face = self.to_detected_face(<face left>, <face top>, <face right>, <face bottom>)
class plugins.extract.detect._base.Detector(git_model_id=None, model_filename=None, configfile=None, instance=0, rotation=None, min_size=0, **kwargs)

Bases: plugins.extract._base.Extractor

Detector Object

Parent class for all Detector plugins

Parameters:
  • git_model_id (int) – The second digit in the github tag that identifies this model. See https://github.com/deepfakes-models/faceswap-models for more information
  • model_filename (str) – The name of the model file to be loaded
  • rotation (str, optional) – Pass in a single number to use increments of that size up to 360, or pass in a list of ints to enumerate exactly what angles to check. Can also pass in 'on' to increment at 90 degree intervals. Default: None
  • min_size (int, optional) – Filters out faces detected below this size. Length, in pixels across the diagonal of the bounding box. Set to 0 for off. Default: 0
Other Parameters:
 

configfile (str, optional) – Path to a custom configuration ini file. Default: Use system configfile

See also

plugins.extract.pipeline
The extraction pipeline for calling plugins
plugins.extract.detect
Detector plugins
plugins.extract._base
Parent class for all extraction plugins
plugins.extract.align._base
Aligner parent class for extraction plugins.
plugins.extract.mask._base
Masker parent class for extraction plugins.
finalize(batch)

Finalize the output from Detector

This should be called as the final task of each plugin.

Parameters:batch (dict) – The final dict from the plugin process. It must contain the keys filename, faces
Yields:ExtractMedia – The DetectedFaces list will be populated for this class with the bounding boxes for the detected faces found in the frame.
get_batch(queue)

Get items for inputting to the detector plugin in batches

Items are received as ExtractMedia objects and converted to dict for internal processing.

Items are returned from the queue in batches of batchsize

Remember to put 'EOF' to the out queue after processing the final batch

Outputs items in the following format. All lists are of length batchsize:

>>> {'filename': [<filenames of source frames>],
>>>  'image': <numpy.ndarray of images standardized for prediction>,
>>>  'scale': [<scaling factors for each image>],
>>>  'pad': [<padding for each image>],
>>>  'detected_faces': [[<lib.align.DetectedFace objects]]}
Parameters:queue (queue.Queue()) – The queue that the batch will be fed from. This will be a queue that loads images.
Returns:
  • exhausted, boolTrue if queue is exhausted, False if not.
  • batch, dict – A dictionary of lists of batchsize.
static to_detected_face(left, top, right, bottom)

Return a DetectedFace object for the bounding box

align._base module

Base class for Face Aligner plugins

All Aligner Plugins should inherit from this class. See the override methods for which methods are required.

The plugin will receive a ExtractMedia object.

For each source item, the plugin must pass a dict to finalize containing:

>>> {"filename": [<filename of source frame>],
>>>  "landmarks": [list of 68 point face landmarks]
>>>  "detected_faces": [<list of DetectedFace objects>]}
class plugins.extract.align._base.Aligner(git_model_id=None, model_filename=None, configfile=None, instance=0, normalize_method=None, re_feed=0, **kwargs)

Bases: plugins.extract._base.Extractor

Aligner plugin _base Object

All Aligner plugins must inherit from this class

Parameters:
  • git_model_id (int) – The second digit in the github tag that identifies this model. See https://github.com/deepfakes-models/faceswap-models for more information
  • model_filename (str) – The name of the model file to be loaded
  • normalize_method ({None, ‘clahe’, ‘hist’, ‘mean’}, optional) – Normalize the images fed to the aligner. Default: None
  • re_feed (int) – The number of times to re-feed a slightly adjusted bounding box into the aligner. Default: 0
Other Parameters:
 

configfile (str, optional) – Path to a custom configuration ini file. Default: Use system configfile

See also

plugins.extract.pipeline
The extraction pipeline for calling plugins
plugins.extract.align
Aligner plugins
plugins.extract._base
Parent class for all extraction plugins
plugins.extract.detect._base
Detector parent class for extraction plugins.
plugins.extract.mask._base
Masker parent class for extraction plugins.
finalize(batch)

Finalize the output from Aligner

This should be called as the final task of each plugin.

Pairs the detected faces back up with their original frame before yielding each frame.

Parameters:batch (dict) – The final dict from the plugin process. It must contain the keys: detected_faces, landmarks, filename
Yields:ExtractMedia – The DetectedFaces list will be populated for this class with the bounding boxes and landmarks for the detected faces found in the frame.
get_batch(queue)

Get items for inputting into the aligner from the queue in batches

Items are returned from the queue in batches of batchsize

Items are received as ExtractMedia objects and converted to dict for internal processing.

To ensure consistent batch sizes for aligner the items are split into separate items for each DetectedFace object.

Remember to put 'EOF' to the out queue after processing the final batch

Outputs items in the following format. All lists are of length batchsize:

>>> {'filename': [<filenames of source frames>],
>>>  'image': [<source images>],
>>>  'detected_faces': [[<lib.align.DetectedFace objects]]}
Parameters:queue (queue.Queue()) – The queue that the plugin will be fed from.
Returns:
  • exhausted, boolTrue if queue is exhausted, False if not
  • batch, dict – A dictionary of lists of batchsize:
set_normalize_method(method)

Set the normalization method for feeding faces into the aligner.

Parameters:method ({"none", "clahe", "hist", "mean"}) – The normalization method to apply to faces prior to feeding into the model

mask._base module

Base class for Face Masker plugins

Plugins should inherit from this class

See the override methods for which methods are required.

The plugin will receive a ExtractMedia object.

For each source item, the plugin must pass a dict to finalize containing:

>>> {"filename": <filename of source frame>,
>>>  "detected_faces": <list of bounding box dicts from lib/plugins/extract/detect/_base>}
class plugins.extract.mask._base.Masker(git_model_id=None, model_filename=None, configfile=None, instance=0, image_is_aligned=False, **kwargs)

Bases: plugins.extract._base.Extractor

Masker plugin _base Object

All Masker plugins must inherit from this class

Parameters:
  • git_model_id (int) – The second digit in the github tag that identifies this model. See https://github.com/deepfakes-models/faceswap-models for more information
  • model_filename (str) – The name of the model file to be loaded
  • image_is_aligned (bool, optional) – Indicates that the passed in image is an aligned face rather than a frame. Default: False
Other Parameters:
 

configfile (str, optional) – Path to a custom configuration ini file. Default: Use system configfile

See also

plugins.extract.pipeline
The extraction pipeline for calling plugins
plugins.extract.align
Aligner plugins
plugins.extract._base
Parent class for all extraction plugins
plugins.extract.detect._base
Detector parent class for extraction plugins.
plugins.extract.align._base
Aligner parent class for extraction plugins.
finalize(batch)

Finalize the output from Masker

This should be called as the final task of each plugin.

Pairs the detected faces back up with their original frame before yielding each frame.

Parameters:batch (dict) – The final dict from the plugin process. It must contain the keys: detected_faces, filename, feed_faces
Yields:ExtractMedia – The DetectedFaces list will be populated for this class with the bounding boxes, landmarks and masks for the detected faces found in the frame.
get_batch(queue)

Get items for inputting into the masker from the queue in batches

Items are returned from the queue in batches of batchsize

Items are received as ExtractMedia objects and converted to dict for internal processing.

To ensure consistent batch sizes for masker the items are split into separate items for each DetectedFace object.

Remember to put 'EOF' to the out queue after processing the final batch

Outputs items in the following format. All lists are of length batchsize:

>>> {'filename': [<filenames of source frames>],
>>>  'detected_faces': [[<lib.align.DetectedFace objects]]}
Parameters:queue (queue.Queue()) – The queue that the plugin will be fed from.
Returns:
  • exhausted, boolTrue if queue is exhausted, False if not
  • batch, dict – A dictionary of lists of batchsize:

vgg_face2_keras module

VGG_Face2 inference and sorting

class plugins.extract.recognition.vgg_face2_keras.VGGFace2(*args, **kwargs)

Bases: plugins.extract._base.Extractor

VGG Face feature extraction.

Extracts feature vectors from faces in order to compare similarity.

Notes

Input images should be in BGR Order

Model exported from: https://github.com/WeidiXie/Keras-VGGFace2-ResNet50 which is based on: https://www.robots.ox.ac.uk/~vgg/software/vgg_face/

Licensed under Creative Commons Attribution License. https://creativecommons.org/licenses/by-nc/4.0/

static find_cosine_similiarity(source_face, test_face)

Find the cosine similarity between two faces.

Parameters:
  • source_face (numpy.ndarray) – The first face to test against test_face
  • test_face (numpy.ndarray) – The second face to test against source_face
Returns:

The cosine similarity between the two faces

Return type:

float

init_model()

Initialize VGG Face 2 Model.

predict(batch)

Return encodings for given image from vgg_face2.

Parameters:batch (numpy.ndarray) – The face to be fed through the predictor. Should be in BGR channel order
Returns:The encodings for the face
Return type:numpy.ndarray
sorted_similarity(predictions, method='ward')

Sort a matrix of predictions by similarity.

Transforms a distance matrix into a sorted distance matrix according to the order implied by the hierarchical tree (dendrogram).

Parameters:
  • predictions (numpy.ndarray) – A stacked matrix of vgg_face2 predictions of the shape (N, D) where N is the number of observations and D are the number of dimensions. NB: The given predictions will be overwritten to save memory. If you still require the original values you should take a copy prior to running this method
  • method (['single','centroid','median','ward']) – The clustering method to use.
Returns:

List of indices with the order implied by the hierarchical tree

Return type:

list