extract package

The Extract Package handles the various plugins available for extracting face sets in Faceswap.

extract_media module 

Object for holding and manipulating media passing through a faceswap extraction pipeline

class plugins.extract.extract_media.ExtractMedia(filename: str, image: np.ndarray, detected_faces: list[DetectedFace] | None = None, is_aligned: bool = False)

Bases: object

An object that passes through the Extractor pipeline.

Parameters:

filename (str) – The base name of the original frame’s filename
image (numpy.ndarray) – The original frame or a faceswap aligned face image
detected_faces (list, optional) – A list of DetectedFace objects. Detected faces can be added later with add_detected_faces(). Setting None will default to an empty list. Default: None
is_aligned (bool, optional) – True if the image is an aligned faceswap image otherwise False. Used for face filtering with vggface2. Aligned faceswap images will automatically skip detection, alignment and masking. Default: False

add_detected_faces(faces: list[DetectedFace]) → None

Add detected faces to the object. Called at the end of each extraction phase.

Parameters:: faces (list) – A list of DetectedFace objects

add_frame_metadata(metadata: PNGHeaderSourceDict) → None

Add the source frame metadata from an aligned PNG’s header data.

metadata: dict: The contents of the ‘source’ field in the PNG header

add_sub_folders(folders: list[str | None]) → None

Add detected faces to the object. Called at the end of each extraction phase.

Parameters:: folders (list) – A list of str sub folder names or None if no sub folder is required. Should correspond to the detected faces list

property detected_faces: list[DetectedFace]

A list of DetectedFace objects in the image.

Type:: list

property filename: str

The base name of the image filename.

Type:: str

property frame_metadata: PNGHeaderSourceDict

The frame metadata that has been added from an aligned image. This property should only be called after add_frame_metadata() has been called when processing an aligned face. For all other instances an assertion error will be raised.

Raises:: AssertionError – If frame metadata has not been populated from an aligned image
Type:: dict

get_image_copy(color_format: T.Literal['BGR', 'RGB', 'GRAY']) → np.ndarray

Get a copy of the image in the requested color format.

Parameters:: color_format (['BGR', 'RGB', 'GRAY']) – The requested color format of image
Returns:: A copy of image in the requested color_format
Return type:: numpy.ndarray

property image: np.ndarray

The source frame for this object.

Type:: numpy.ndarray

property image_shape: tuple[int, int, int]

The shape of the stored image.

Type:: tuple

property image_size: tuple[int, int]

The (height, width) of the stored image.

Type:: tuple

property is_aligned: bool: bool. True if image is an aligned faceswap image otherwise False

remove_image() → None

Delete the image and reset image to None.

Required for multi-phase extraction to avoid the frames stacking RAM.

set_image(image: np.ndarray) → None

Add the image back into image

Required for multi-phase extraction adds the image back to this object.

Parameters:: image (numpy.ndarry) – The original frame to be re-applied to for this filename

property sub_folders: list[str | None]

The sub_folders that the faces should be output to. Used when binning filter output is enabled. The list corresponds to the list of detected faces

Type:: list

pipeline module 

Return a requested detector/aligner/masker pipeline

Tensorflow does not like to release GPU VRAM, so parallel plugins need to be managed to work together.

This module sets up a pipeline for the extraction workflow, loading detect, align and mask plugins either in parallel or in series, giving easy access to input and output.

class plugins.extract.pipeline.Extractor(detector: str | None, aligner: str | None, masker: str | list[str] | None, recognition: str | None = None, configfile: str | None = None, multiprocess: bool = False, exclude_gpus: list[int] | None = None, rotate_images: str | None = None, min_size: int = 0, normalize_method: Literal['none', 'clahe', 'hist', 'mean'] | None = None, re_feed: int = 0, re_align: bool = False, disable_filter: bool = False)

Bases: object

Creates a detect/align`/ mask pipeline and yields results frame by frame from the detected_faces generator

input_queue is dynamically set depending on the current phase of extraction

Parameters:

detector (str or None) – The name of a detector plugin as exists in plugins.extract.detect
aligner (str or None) – The name of an aligner plugin as exists in plugins.extract.align
masker (str or list or None) – The name of a masker plugin(s) as exists in plugins.extract.mask. This can be a single masker or a list of multiple maskers
recognition (str or None) – The name of the recognition plugin to use. None to not do face recognition. Default: None
configfile (str, optional) – The path to a custom extract.ini configfile. If None then the system config/extract.ini file will be used.
multiprocess (bool, optional) – Whether to attempt processing the plugins in parallel. This may get overridden internally depending on the plugin combination. Default: False
exclude_gpus (list, optional) – A list of indices correlating to connected GPUs that Tensorflow should not use. Pass None to not exclude any GPUs. Default: None
rotate_images (str, optional) – Used to set the plugins.extract.detect.rotation attribute. Pass in a single number to use increments of that size up to 360, or pass in a list of ints to enumerate exactly what angles to check. Can also pass in 'on' to increment at 90 degree intervals. Default: None
min_size (int, optional) – Used to set the plugins.extract.detect.min_size attribute. Filters out faces detected below this size. Length, in pixels across the diagonal of the bounding box. Set to 0 for off. Default: 0
normalize_method ({None, ‘clahe’, ‘hist’, ‘mean’}, optional) – Used to set the plugins.extract.align.normalize_method attribute. Normalize the images fed to the aligner.Default: None
re_feed (int) – The number of times to re-feed a slightly adjusted bounding box into the aligner. Default: 0
re_align (bool, optional) – True to obtain landmarks by passing the initially aligned face back through the aligner. Default False
disable_filter (bool, optional) – Disable all aligner filters regardless of config option. Default: False

phase

The current phase that the pipeline is running. Used in conjunction with passes and final_pass to indicate to the caller which phase is being processed

Type:: str

property aligner: Aligner: The currently selected aligner plugin

detected_faces() → Generator[ExtractMedia, None, None]

Generator that returns results, frame by frame from the extraction pipeline

This is the exit point for the extraction pipeline and is used to obtain the output of any pipeline phase

Yields:: faces (ExtractMedia) – The populated extracted media object.

Example

>>> for extract_media in extractor.detected_faces():
>>>     filename = extract_media.filename
>>>     image = extract_media.image
>>>     detected_faces = extract_media.detected_faces

property final_pass: bool

bool, Return True if this is the final extractor pass otherwise False

Useful for iterating over the pipeline passes or detected_faces() and handling accordingly.

Example

>>> for face in extractor.detected_faces():
>>>     if extractor.final_pass:
>>>         <do final processing>
>>>     else:
>>>         extract_media.set_image(image)
>>>         <do intermediate processing>
>>>         extractor.input_queue.put(extract_media)

import_data(input_location: str) → None

Import json data to the detector and/or aligner if ‘import’ plugin has been selected

Parameters:: input_location (str) – Full path to the input location for the extract process

property input_queue: EventQueue

Return the correct input queue depending on the current phase

The input queue is the entry point into the extraction pipeline. An ExtractMedia object should be put to the queue.

For detect/single phase operations the ExtractMedia.filename and image attributes should be populated.

For align/mask (2nd/3rd pass operations) the ExtractMedia.detected_faces should also be populated by calling ExtractMedia.set_detected_faces().

Type:: queue

launch() → None

Launches the plugin(s)

This launches the plugins held in the pipeline, and should be called at the beginning of each phase. To ensure VRAM is conserved, It will only launch the plugin(s) required for the currently running phase

Example

>>> for phase in extractor.passes:
>>>     extractor.launch():
>>>         <do processing>

property passes: int

Returns the total number of passes the extractor needs to make.

This is calculated on several factors (vram available, plugin choice, multiprocess etc.). It is useful for iterating over the pipeline and handling accordingly.

Example

>>> for phase in extractor.passes:
>>>     if phase == 1:
>>>         extract_media = ExtractMedia("path/to/image/file", image)
>>>         extractor.input_queue.put(extract_media)
>>>     else:
>>>         extract_media.set_image(image)
>>>         extractor.input_queue.put(extract_media)

Type:: int

property phase_text: str

The plugins that are running in the current phase, formatted for info text output.

Type:: str

property recognition: Identity: The currently selected recognition plugin

reset_phase_index() → None: Reset the current phase index back to 0. Used for when batch processing is used in extract.

set_batchsize(plugin_type: Literal['align', 'detect'], batchsize: int) → None

Set the batch size of a given plugin_type to the given batchsize.

This should be set prior to launch() if the batch size is to be manually overridden

Parameters:

plugin_type ({'align', 'detect'}) – The plugin_type to be overridden
batchsize (int) – The batch size to use for this plugin type

_base module 

Base class for Faceswap detect, align and mask Plugins

Bases: object

Extractor Plugin Object

All _base classes for Aligners, Detectors and Maskers inherit from this class.

This class sets up a pipeline for working with ML plugins.

Plugins are split into 3 threads, to utilize Numpy and CV2s parallel processing, as well as allow the predict function of the model to sit in a dedicated thread. A plugin is expected to have 3 core functions, each in their own thread: - process_input() - Prepare the data for feeding into a model - predict() - Feed the data through the model - process_output() - Perform any data post-processing

Parameters:

git_model_id (int) – The second digit in the github tag that identifies this model. See https://github.com/deepfakes-models/faceswap-models for more information
model_filename (str) – The name of the model file to be loaded
exclude_gpus (list, optional) – A list of indices correlating to connected GPUs that Tensorflow should not use. Pass None to not exclude any GPUs. Default: None
configfile (str, optional) – Path to a custom configuration ini file. Default: Use system configfile
instance (int, optional) – If this plugin is being executed multiple times (i.e. multiple pipelines have been launched), the instance of the plugin must be passed in for naming convention reasons. Default: 0

The following attributes should be set in the plugin’s __init__() method after initializing the parent.

name

Name of this plugin. Used for display purposes.

Type:: str

input_size

The input size to the model in pixels across one edge. The input size should always be square.

Type:: int

color_format

Color format for model. Must be 'BGR', 'RGB' or 'GRAY'. Defaults to 'BGR' if not explicitly set.

Type:: str

vram

Approximate VRAM used by the model at input_size. Used to calculate the batchsize. Be conservative to avoid OOM.

Type:: int

vram_warnings

Approximate VRAM used by the model at input_size that will still run, but generates warnings. Used to calculate the batchsize. Be conservative to avoid OOM.

Type:: int

vram_per_batch

Approximate additional VRAM used by the model for each additional batch. Used to calculate the batchsize. Be conservative to avoid OOM.

Type:: int

See also

plugins.extract.detect._base: Detector parent class for extraction plugins.
plugins.extract.align._base: Aligner parent class for extraction plugins.
plugins.extract.mask._base: Masker parent class for extraction plugins.
plugins.extract.pipeline: The extract pipeline that configures and calls all plugins

batchsize

Batchsize for feeding this model. The number of images the model should feed through at once.

Type:: int

check_and_raise_error() → None

Check all threads for errors

Exposed for pipeline to check plugin’s threads for errors

config

Config for this plugin, loaded from extract.ini configfile

Type:: dict

finalize(batch: BatchType) → Generator[ExtractMedia, None, None]

Override method (at <plugin_type> level)

This method should be overridden at the <plugin_type> level (IE. plugins.extract.detect._base, plugins.extract.align._base or plugins.extract.mask._base) and should not be overridden within plugins themselves.

Handles consistent finalization for all plugins that exist within that plugin type. Its input is always the output from process_output()

Parameters:: batch (ExtractorBatch) – Contains the batch that is currently being passed through the plugin process

get_batch(queue: Queue) → tuple[bool, BatchType]

Override method (at <plugin_type> level)

This method should be overridden at the <plugin_type> level (IE. plugins.extract.detect._base, plugins.extract.align._base or plugins.extract.mask._base) and should not be overridden within plugins themselves.

Get ExtractMedia items from the queue in batches of batchsize

Parameters:: queue (queue.Queue()) – The queue that the batch will be fed from. This will be the input to the plugin.

init_model() → None

Override method

Override this method to execute the specific model initialization method

initialize(*args, **kwargs) → None

Initialize the extractor plugin

Should be called from pipeline

join() → None

Join all threads

Exposed for pipeline to join plugin’s threads

model: KSession | cv2.dnn.Net | None

The model for this plugin. Set in the plugin’s init_model() method

Type:: varies

model_path

Path to the model file(s) (if required). Multiple model files should be a list of strings

Type:: str or list

on_completion() → None: Override to perform an action when the extract process has completed. By default, no action is undertaken

predict(feed: ndarray) → ndarray

Override method

Override this method for specific extractor model prediction function

Parameters:: feed (numpy.ndarray) – The feed images for the batch

Notes

Input for predict() should have been set in process_input()

Output from the model should populate the key prediction of the batch.

For Detect:: the expected output for the prediction of the batch should be a list of batchsize of detected face points. These points should be either a list, tuple or numpy.ndarray with the first 4 items being the left, top, right, bottom points, in that order

process_input(batch: BatchType) → None

Override method

Override this method for specific extractor pre-processing of image

Parameters:: batch (ExtractorBatch) – Contains the batch that is currently being passed through the plugin process

process_output(batch: BatchType) → None

Override method

Override this method for specific extractor model post predict function

Parameters:: batch (ExtractorBatch) – Contains the batch that is currently being passed through the plugin process

Notes

For Align:: The landmarks must be populated in batch from this method. This should be a list or numpy.ndarray of batchsize containing a list, tuple or numpy.ndarray of (x, y) coordinates of the 68 point landmarks as calculated from the model.

queue_size

Queue size for all internal queues. Set in initialize()

Type:: int

rollover_collector(queue: Queue) → T.Literal['EOF'] | ExtractMedia

For extractors after the Detectors, the number of detected faces per frame vs extractor batch size mean that faces will need to be split/re-joined with frames. The rollover collector can be used to rollover items that don’t fit in a batch.

Collect the item from the _rollover dict or from the queue. Add face count per frame to self._faces_per_filename for joining batches back up in finalize

Parameters:: queue (queue.Queue) – The input queue to the aligner. Should contain ExtractMedia objects
Returns:: The next extract media object, or EOF if pipe has ended
Return type:: ExtractMedia or EOF

start() → None

Start all threads

Exposed for pipeline to start plugin’s threads

class plugins.extract._base.ExtractorBatch(image: list[np.ndarray] = <factory>, detected_faces: Sequence[DetectedFace | list[DetectedFace]] = <factory>, filename: list[str] = <factory>, feed: np.ndarray = array([], dtype=float64), prediction: np.ndarray = array([], dtype=float64), data: list[dict[str, T.Any]] = <factory>)

Bases: object

Dataclass for holding a batch flowing through post Detector plugins.

The batch size for post Detector plugins is not the same as the overall batch size. An image may contain 0 or more detected faces, and these need to be split and recombined to be able to utilize a plugin’s internal batch size.

Plugin types will inherit from this class and add required keys.

Parameters:

image (list) – List of numpy.ndarray containing the original frames
detected_faces (list) – List of DetectedFace objects
filename (list) – List of original frame filenames for the batch
feed (numpy.ndarray) – Batch of feed images to feed the net with
prediction (numpy.nd.array) – Batch of predictions. Direct output from the aligner net
data (dict) – Any specific data required during the processing phase for a particular plugin

data: list[dict[str, T.Any]]

detected_faces: Sequence[DetectedFace | list[DetectedFace]]

feed: np.ndarray = array([], dtype=float64)

filename: list[str]

image: list[np.ndarray]

prediction: np.ndarray = array([], dtype=float64)

align plugins package 

align._base.aligner module 

Base class for Face Aligner plugins

All Aligner Plugins should inherit from this class. See the override methods for which methods are required.

The plugin will receive a ExtractMedia object.

For each source item, the plugin must pass a dict to finalize containing:

>>> {"filename": [<filename of source frame>],
>>>  "landmarks": [list of 68 point face landmarks]
>>>  "detected_faces": [<list of DetectedFace objects>]}

class plugins.extract.align._base.aligner.Aligner(git_model_id: int | None = None, model_filename: str | None = None, configfile: str | None = None, instance: int = 0, normalize_method: Literal['none', 'clahe', 'hist', 'mean'] | None = None, re_feed: int = 0, re_align: bool = False, disable_filter: bool = False, **kwargs)

Bases: Extractor

Aligner plugin _base Object

All Aligner plugins must inherit from this class

Parameters:

git_model_id (int) – The second digit in the github tag that identifies this model. See https://github.com/deepfakes-models/faceswap-models for more information
model_filename (str) – The name of the model file to be loaded
normalize_method ({None, ‘clahe’, ‘hist’, ‘mean’}, optional) – Normalize the images fed to the aligner. Default: None
re_feed (int, optional) – The number of times to re-feed a slightly adjusted bounding box into the aligner. Default: 0
re_align (bool, optional) – True to obtain landmarks by passing the initially aligned face back through the aligner. Default False
disable_filter (bool, optional) – Disable all aligner filters regardless of config option. Default: False
configfile (str, optional) – Path to a custom configuration ini file. Default: Use system configfile

See also

plugins.extract.pipeline: The extraction pipeline for calling plugins
plugins.extract.align: Aligner plugins
plugins.extract._base: Parent class for all extraction plugins
plugins.extract.detect._base: Detector parent class for extraction plugins.
plugins.extract.mask._base: Masker parent class for extraction plugins.

faces_to_feed(faces: ndarray) → ndarray

Overide for specific plugin processing to convert a batch of face images from UINT8 (0-255) into the correct format for the plugin’s inference

Parameters:: faces (numpy.ndarray) – The batch of faces in UINT8 format
Returns:: class – The batch of faces in the format to feed through the plugin
Return type:: numpy.ndarray

finalize(batch: BatchType) → Generator[ExtractMedia, None, None]

Finalize the output from Aligner

This should be called as the final task of each plugin.

Pairs the detected faces back up with their original frame before yielding each frame.

Parameters:: batch (AlignerBatch) – The final batch item from the plugin process.
Yields:: ExtractMedia – The DetectedFaces list will be populated for this class with the bounding boxes and landmarks for the detected faces found in the frame.

get_batch(queue: Queue) → tuple[bool, AlignerBatch]

Get items for inputting into the aligner from the queue in batches

Items are returned from the queue in batches of batchsize

Items are received as ExtractMedia objects and converted to dict for internal processing.

To ensure consistent batch sizes for aligner the items are split into separate items for each DetectedFace object.

Remember to put 'EOF' to the out queue after processing the final batch

Outputs items in the following format. All lists are of length batchsize:

>>> {'filename': [<filenames of source frames>],
>>>  'image': [<source images>],
>>>  'detected_faces': [[<lib.align.DetectedFace objects]]}

Parameters:

queue (queue.Queue()) – The queue that the plugin will be fed from.

Returns:

exhausted, bool – True if queue is exhausted, False if not
batch, ExtractorBatch – The batch object for the current batch

initialize(*args, **kwargs) → None: Add a call to add model input size to the re-aligner

on_completion() → None: Output the filter counts when process has completed

set_normalize_method(method: Literal['none', 'clahe', 'hist', 'mean'] | None) → None

Set the normalization method for feeding faces into the aligner.

Parameters:: method ({"none", "clahe", "hist", "mean"}) – The normalization method to apply to faces prior to feeding into the model

class plugins.extract.align._base.aligner.AlignerBatch(image: list[np.ndarray] = <factory>, detected_faces: list[DetectedFace] = <factory>, filename: list[str] = <factory>, feed: np.ndarray = array([], dtype=float64), prediction: np.ndarray = array([], dtype=float64), data: list[dict[str, T.Any]] = <factory>, batch_id: int = 0, landmarks: np.ndarray = array([], dtype=float64), refeeds: list[np.ndarray] = <factory>, second_pass: bool = False, second_pass_masks: np.ndarray = array([], dtype=float64))

Bases: ExtractorBatch

Dataclass for holding items flowing through the aligner.

Inherits from ExtractorBatch

Parameters:

batch_id (int) – A unique integer for tracking this batch
landmarks (list) – List of 68 point numpy.ndarray landmark points returned from the aligner
refeeds (list) – List of numpy.ndarrays for holding each of the feeds that will be put through the model for each refeed
second_pass (bool, optional) – True if this batch is passing through the aligner for a second time as re-align has been selected otherwise False. Default: False
second_pass_masks (numpy.ndarray, optional) – The masks used to filter out re-feed values for passing to the re-aligner.

batch_id: int = 0

detected_faces: list[DetectedFace]

landmarks: np.ndarray = array([], dtype=float64)

refeeds: list[np.ndarray]

second_pass: bool = False

second_pass_masks: np.ndarray = array([], dtype=float64)

align._base.processing module 

Processing methods for aligner plugins

class plugins.extract.align._base.processing.AlignedFilter(feature_filter: bool, min_scale: float, max_scale: float, distance: float, roll: float, save_output: bool, disable: bool = False)

Bases: object

Applies filters on the output of the aligner

Parameters:

feature_filter (bool) – True to enable filter to check relative position of eyes/eyebrows and mouth. False to disable.
min_scale (float) – Filters out faces that have been aligned at below this value as a multiplier of the minimum frame dimension. Set to 0 for off.
max_scale (float) – Filters out faces that have been aligned at above this value as a multiplier of the minimum frame dimension. Set to 0 for off.
distance (float) – Filters out faces that are further than this distance from an “average” face. Set to 0 for off.
roll (float) – Filters out faces with a roll value outside of 0 +/- the value given here. Set to 0 for off.
save_output (bool) – True if the filtered faces should be kept as they are being saved. False if they should be deleted
disable (bool, Optional) – True to disable the filter regardless of config options. Default: False

filtered_mask(batch: AlignerBatch, skip: np.ndarray | list[int] | None = None) → np.ndarray

Obtain a list of boolean values for the given batch indicating whether they pass the filter test.

Parameters:

batch (AlignerBatch) – The batch of face to obtain masks for
skip (list or numpy.ndarray, optional) – List or 1D numpy array of indices indicating faces that have already been filter masked and so should not be filtered again. Values in these index positions will be returned as True

Returns:

Boolean mask array corresponding to any of the input DetectedFace objects that passed a test. False the face passed the test. True it failed

Return type:

numpy.ndarray

output_counts(): Output the counts of filtered items

class plugins.extract.align._base.processing.ReAlign(active: bool, do_refeeds: bool, do_filter: bool)

Bases: object

Holds data and methods for 2nd pass re-aligns

Parameters:

active (bool) – True if re-alignment has been requested otherwise False
do_refeeds (bool) – True if re-feeds should be re-aligned, False if just the final output of the re-feeds should be aligned
do_filter (bool) – True if aligner filtered out faces should not be re-aligned. False if all faces should be re-aligned

property active: bool

True if re_aligns have been selected otherwise False

Type:: bool

add_batch(batch: AlignerBatch) → None

Add first pass alignments to the queue for picking up for re-alignment, update their second_pass attribute to True and clear attributes not required.

Parameters:: batch (AlignerBatch) – aligner batch to perform re-alignment on

property do_filter: bool

True if re-aligning is active and faces which failed the aligner filter test should not be re-aligned otherwise False

Type:: bool

property do_refeeds: bool

True if re-aligning is active and re-aligning re-feeds has been selected otherwise False

Type:: bool

get_batch() → AlignerBatch

Retrieve the next batch currently queued for re-alignment

Returns:: The next AlignerBatch for re-alignment
Return type:: AlignerBatch

property items_queued: bool

True if re-align is active and items are queued for a 2nd pass otherwise False

Type:: bool

property items_tracked: bool

True if items exist in the tracker so still need to be processed

Type:: bool

process_batch(batch: AlignerBatch) → list[np.ndarray]

Pre process a batch object for re-aligning through the aligner.

Parameters:: batch (AlignerBatch) – aligner batch to perform pre-processing on
Returns:: List of UINT8 aligned faces batch for each selected refeed
Return type:: list

process_output(subbatches: list[AlignerBatch], batch_masks: np.ndarray) → None

Process the output from the re-align pass.

Transform landmarks from aligned face space to face space
Re-insert faces that were filtered out from the re-align process back into the landmarks list

Parameters:

subbatches (list) – List of sub-batch results for each re-aligned re-feed performed
batch_masks (numpy.ndarray) – The original re-feed filter masks from the first pass

set_input_size_and_centering(input_size: int, centering: CenteringType) → None

Set the input size of the loaded plugin once the model has been loaded

Parameters:

input_size (int) – The input size, in pixels, of the aligner plugin
centering (["face", "head" or "legacy"]) – The centering to align the image at for re-aligning

track_batch(batch_id: int) → None

Add newly seen batch id from the aligner to the batch tracker, so that we can keep track of whether there are still batches to be processed when the aligner hits ‘EOF’

Parameters:: batch_id (int) – The batch id to add to batch tracking

untrack_batch(batch_id: int) → None

Remove the tracked batch from the tracker once the batch has been fully processed

Parameters:: batch_id (int) – The batch id to remove from batch tracking

align.cv2_dnn module 

CV2 DNN landmarks extractor for faceswap.py Adapted from: https://github.com/yinguobing/cnn-facial-landmark MIT License

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the “Software”), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

class plugins.extract.align.cv2_dnn.Align(**kwargs)

Bases: Aligner

Perform transformation to align and get landmarks

align_image(batch: AlignerBatch) → tuple[list[numpy.ndarray], list[list[int]], list[tuple[int, int]]]

Align the incoming image for prediction

Parameters:

batch (AlignerBatch) – The current batch to align the input for

Returns:

faces (list) – List of feed faces for the aligner
rois (list) – List of roi’s for the faces
offsets (list) – List of offsets for the faces

faces_to_feed(faces: ndarray) → ndarray

Convert a batch of face images from UINT8 (0-255) to fp32 (0.0-255.0)

Parameters:: faces (numpy.ndarray) – The batch of faces in UINT8 format
Returns:: class – The batch of faces as fp32
Return type:: numpy.ndarray

get_pts_from_predict(batch: AlignerBatch)

Get points from predictor and populates the landmarks property

Parameters:: batch (AlignerBatch) – The current batch from the model with predictions populated

static get_square_box(box: list[int]) → list[int]

Get a square box out of the given box, by expanding it.

Parameters:: box (list) – The (left, top, right, bottom) box positions
Returns:: The original box but made square
Return type:: list

init_model() → None: Initialize CV2 DNN Detector Model

classmethod move_box(box: list[int], offset: tuple[int, int]) → list[int]

Move the box to direction specified by vector offset

Parameters:

box (list) – The (left, top, right, bottom) box positions
offset (tuple) – (x, y) offset to move the box

Returns:

The original box shifted by the offset

Return type:

list

classmethod pad_image(box: list[int], image: ndarray) → tuple[numpy.ndarray, tuple[int, int]]

Pad image if face-box falls outside of boundaries

Parameters:

box (list) – The (left, top, right, bottom) roi box positions
image (numpy.ndarray) – The image to be padded

Returns:

The padded image

Return type:

numpy.ndarray

predict(feed: ndarray) → ndarray

Predict the 68 point landmarks

Parameters:: feed (numpy.ndarray) – The batch to feed into the aligner
Returns:: The predictions from the aligner
Return type:: numpy.ndarray

process_input(batch: BatchType) → None

Compile the detected faces for prediction

Parameters:: batch (AlignerBatch) – The current batch to process input for
Returns:: The batch item with the feed populated and any required data added
Return type:: AlignerBatch

process_output(batch: BatchType) → None

Process the output from the model

Parameters:: batch (AlignerBatch) – The current batch from the model with predictions populated

align.fan module 

Facial landmarks extractor for faceswap.py Code adapted and modified from: https://github.com/1adrianb/face-alignment

class plugins.extract.align.fan.Align(**kwargs)

Bases: Aligner

Perform transformation to align and get landmarks

crop(batch: AlignerBatch, center_scale: ndarray) → list[numpy.ndarray]

Crop image around the center point

Parameters:

batch (AlignerBatch) – The current batch to crop the image for
center_scale (numpy.ndarray) – The center and scale for the bounding box

Returns:

List of cropped images for the batch

Return type:

list

faces_to_feed(faces: ndarray) → ndarray

Convert a batch of face images from UINT8 (0-255) to fp32 (0.0-1.0)

Parameters:: faces (numpy.ndarray) – The batch of faces in UINT8 format
Returns:: class – The batch of faces as fp32 in 0.0 to 1.0 range
Return type:: numpy.ndarray

get_center_scale(detected_faces: list[DetectedFace]) → np.ndarray

Get the center and set scale of bounding box

Parameters:: detected_faces (list) – List of DetectedFace objects for the batch
Returns:: The center and scale of the bounding box
Return type:: numpy.ndarray

get_pts_from_predict(batch: AlignerBatch) → None

Get points from predictor and populate the landmarks property of the AlignerBatch

Parameters:: batch (AlignerBatch) – The current batch from the model with predictions populated

init_model() → None: Initialize FAN model

predict(feed: ndarray) → ndarray

Predict the 68 point landmarks

Parameters:: batch (numpy.ndarray) – The batch to feed into the aligner
Returns:: The predictions from the aligner
Return type:: numpy.ndarray

process_input(batch: BatchType) → None

Compile the detected faces for prediction

Parameters:: batch (AlignerBatch) – The current batch to process input for

process_output(batch: BatchType) → None

Process the output from the model

Parameters:: batch (AlignerBatch) – The current batch from the model with predictions populated

classmethod transform(points: ndarray, center_scales: ndarray, resolutions: ndarray) → ndarray

Transform Image

Parameters:

points (numpy.ndarray) – The points to transform
center_scales (numpy.ndarray) – The calculated centers and scales for the batch
resolutions (numpy.ndarray) – The resolutions

detect plugins package 

detect._base module 

Base class for Face Detector plugins

All Detector Plugins should inherit from this class. See the override methods for which methods are required.

The plugin will receive a ExtractMedia object.

For each source frame, the plugin must pass a dict to finalize containing:

>>> {'filename': <filename of source frame>,
>>>  'detected_faces': <list of DetectedFace objects containing bounding box points}}

To get a DetectedFace object use the function:

>>> face = self._to_detected_face(<face left>, <face top>, <face right>, <face bottom>)

Bases: Extractor

Detector Object

Parent class for all Detector plugins

Parameters:

git_model_id (int) – The second digit in the github tag that identifies this model. See https://github.com/deepfakes-models/faceswap-models for more information
model_filename (str) – The name of the model file to be loaded
rotation (str, optional) – Pass in a single number to use increments of that size up to 360, or pass in a list of ints to enumerate exactly what angles to check. Can also pass in 'on' to increment at 90 degree intervals. Default: None
min_size (int, optional) – Filters out faces detected below this size. Length, in pixels across the diagonal of the bounding box. Set to 0 for off. Default: 0
configfile (str, optional) – Path to a custom configuration ini file. Default: Use system configfile

See also

plugins.extract.pipeline: The extraction pipeline for calling plugins
plugins.extract.detect: Detector plugins
plugins.extract._base: Parent class for all extraction plugins
plugins.extract.align._base: Aligner parent class for extraction plugins.
plugins.extract.mask._base: Masker parent class for extraction plugins.

finalize(batch: BatchType) → Generator[ExtractMedia, None, None]

Finalize the output from Detector

This should be called as the final task of each plugin.

Parameters:: batch (ExtractorBatch) – The batch object for the current batch
Yields:: ExtractMedia – The DetectedFaces list will be populated for this class with the bounding boxes for the detected faces found in the frame.

get_batch(queue: Queue) → tuple[bool, DetectorBatch]

Get items for inputting to the detector plugin in batches

Items are received as ExtractMedia objects and converted to dict for internal processing.

Items are returned from the queue in batches of batchsize

Remember to put 'EOF' to the out queue after processing the final batch

Outputs items in the following format. All lists are of length batchsize:

>>> {'filename': [<filenames of source frames>],
>>>  'image': <numpy.ndarray of images standardized for prediction>,
>>>  'scale': [<scaling factors for each image>],
>>>  'pad': [<padding for each image>],
>>>  'detected_faces': [[<lib.align.DetectedFace objects]]}

Parameters:

queue (queue.Queue()) – The queue that the batch will be fed from. This will be a queue that loads images.

Returns:

exhausted, bool – True if queue is exhausted, False if not.
batch, ExtractorBatch – The batch object for the current batch

class plugins.extract.detect._base.DetectorBatch(image: list[numpy.ndarray] = <factory>, detected_faces: list[list['DetectedFace']] = <factory>, filename: list[str] = <factory>, feed: ~numpy.ndarray = array([], dtype=float64), prediction: ~numpy.ndarray = array([], dtype=float64), data: list[dict[str, typing.Any]] = <factory>, rotation_matrix: list[numpy.ndarray] = <factory>, scale: list[float] = <factory>, pad: list[tuple[int, int]] = <factory>, initial_feed: ~numpy.ndarray = array([], dtype=float64))

Bases: ExtractorBatch

Dataclass for holding items flowing through the aligner.

Inherits from ExtractorBatch

Parameters:

rotation_matrix (numpy.ndarray) – The rotation matrix for any requested rotations
scale (float) – The scaling factor to take the input image back to original size
pad (tuple) – The amount of padding to apply to the image to feed the network
initial_feed (numpy.ndarray) – Used to hold the initial feed when rotate images is enabled

detected_faces: list[list['DetectedFace']]

initial_feed: np.ndarray = array([], dtype=float64)

pad: list[tuple[int, int]]

rotation_matrix: list[np.ndarray]

scale: list[float]

detect.mtcnn module 

MTCNN Face detection plugin

class plugins.extract.detect.mtcnn.Detect(**kwargs)

Bases: Detector

MTCNN detector for face recognition.

init_model() → None: Initialize MTCNN Model.

predict(feed: ndarray) → ndarray

Run model to get predictions

Parameters:: batch (DetectorBatch) – Contains the batch to pass through the MTCNN model
Returns:: The batch with the predictions added to the dictionary
Return type:: dict

process_input(batch: BatchType) → None

Compile the detection image(s) for prediction

Parameters:: batch (DetectorBatch) – Contains the batch that is currently being passed through the plugin process

process_output(batch: BatchType) → None

MTCNN performs no post processing so the original batch is returned

Parameters:: batch (DetectorBatch) – Contains the batch to apply postprocessing to

class plugins.extract.detect.mtcnn.MTCNN(model_path: list[str], allow_growth: bool, exclude_gpus: list[int] | None, cpu_mode: bool, input_size: int = 640, minsize: int = 20, threshold: list[float] | None = None, factor: float = 0.709)

Bases: object

MTCNN Detector for face alignment

Parameters:

model_path (list) – List of paths to the 3 MTCNN subnet weights
allow_growth (bool, optional) – Enable the Tensorflow GPU allow_growth configuration option. This option prevents Tensorflow from allocating all of the GPU VRAM, but can lead to higher fragmentation and slower performance. Default: False
exclude_gpus (list, optional) – A list of indices correlating to connected GPUs that Tensorflow should not use. Pass None to not exclude any GPUs. Default: None
cpu_mode (bool, optional) – True run the model on CPU. Default: False
input_size (int, optional) – The height, width input size to the model. Default: 640
minsize (int, optional) – The minimum size of a face to accept as a detection. Default: 20
threshold (list, optional) – List of floats for the three steps, Default: [0.6, 0.7, 0.7]
factor (float, optional) – The factor used to create a scaling pyramid of face sizes to detect in the image. Default: 0.709

detect_faces(batch: ndarray) → tuple[numpy.ndarray, tuple[numpy.ndarray]]

Detects faces in an image, and returns bounding boxes and points for them.

Parameters:: batch (numpy.ndarray) – The input batch of images to detect face in
Returns:: list of numpy arrays containing the bounding box and 5 point landmarks of detected faces
Return type:: List

class plugins.extract.detect.mtcnn.ONet(model_path: str, allow_growth: bool, exclude_gpus: list[int] | None, cpu_mode: bool, input_size: int, threshold: float)

Bases: KSession

Keras O-Net model for MTCNN

Parameters:

model_path (str) – The path to the keras model file
allow_growth (bool, optional) – Enable the Tensorflow GPU allow_growth configuration option. This option prevents Tensorflow from allocating all of the GPU VRAM, but can lead to higher fragmentation and slower performance. Default: False
exclude_gpus (list, optional) – A list of indices correlating to connected GPUs that Tensorflow should not use. Pass None to not exclude any GPUs. Default: None
cpu_mode (bool, optional) – True run the model on CPU. Default: False
input_size (int) – The input size of the model
threshold (list, optional) – Threshold for O-Net

static model_definition() → tuple[list[Tensor], list[Tensor]]: Keras O-Network for MTCNN

class plugins.extract.detect.mtcnn.PNet(model_path: str, allow_growth: bool, exclude_gpus: list[int] | None, cpu_mode: bool, input_size: int, min_size: int, factor: float, threshold: float)

Bases: KSession

Keras P-Net model for MTCNN

Parameters:

model_path (str) – The path to the keras model file
allow_growth (bool, optional) – Enable the Tensorflow GPU allow_growth configuration option. This option prevents Tensorflow from allocating all of the GPU VRAM, but can lead to higher fragmentation and slower performance. Default: False
exclude_gpus (list, optional) – A list of indices correlating to connected GPUs that Tensorflow should not use. Pass None to not exclude any GPUs. Default: None
cpu_mode (bool, optional) – True run the model on CPU. Default: False
input_size (int) – The input size of the model
minsize (int, optional) – The minimum size of a face to accept as a detection. Default: 20
threshold (list, optional) – Threshold for P-Net

static model_definition() → tuple[list[Tensor], list[Tensor]]: Keras P-Network Definition for MTCNN

class plugins.extract.detect.mtcnn.RNet(model_path: str, allow_growth: bool, exclude_gpus: list[int] | None, cpu_mode: bool, input_size: int, threshold: float)

Bases: KSession

Keras R-Net model Definition for MTCNN

Parameters:

model_path (str) – The path to the keras model file
allow_growth (bool, optional) – Enable the Tensorflow GPU allow_growth configuration option. This option prevents Tensorflow from allocating all of the GPU VRAM, but can lead to higher fragmentation and slower performance. Default: False
exclude_gpus (list, optional) – A list of indices correlating to connected GPUs that Tensorflow should not use. Pass None to not exclude any GPUs. Default: None
cpu_mode (bool, optional) – True run the model on CPU. Default: False
input_size (int) – The input size of the model
threshold (list, optional) – Threshold for R-Net

static model_definition() → tuple[list[Tensor], list[Tensor]]: Keras R-Network Definition for MTCNN

plugins.extract.detect.mtcnn.nms(rectangles: ndarray, scores: ndarray, threshold: float, method: str = 'iom') → tuple[numpy.ndarray, numpy.ndarray]

apply non-maximum suppression on ROIs in same scale(matrix version)

Parameters:

rectangles (np.ndarray) – The [b, l, t, r, b] bounding box detection candidates
threshold (float) – Threshold for succesful match
method (str, optional) – “iom” method or default. Defalt: “iom”

Returns:

rectangles (np.ndarray) – The [b, l, t, r, b] bounding boxes
scores np.ndarray – The associated scores for the rectangles

plugins.extract.detect.mtcnn.rect2square(rectangles: ndarray) → ndarray

change rectangles into squares (matrix version)

Parameters:: rectangles (numpy.ndarray) – [b, x, y, x1, y1] rectangles
Returns:: Original rectangle changed to a square
Return type:: list

mask plugins package 

mask._base module 

Base class for Face Masker plugins

Plugins should inherit from this class

See the override methods for which methods are required.

The plugin will receive a ExtractMedia object.

For each source item, the plugin must pass a dict to finalize containing:

>>> {"filename": <filename of source frame>,
>>>  "detected_faces": <list of bounding box dicts from lib/plugins/extract/detect/_base>}

class plugins.extract.mask._base.Masker(git_model_id: int | None = None, model_filename: str | None = None, configfile: str | None = None, instance: int = 0, **kwargs)

Bases: Extractor

Masker plugin _base Object

All Masker plugins must inherit from this class

Parameters:

git_model_id (int) – The second digit in the github tag that identifies this model. See https://github.com/deepfakes-models/faceswap-models for more information
model_filename (str) – The name of the model file to be loaded
configfile (str, optional) – Path to a custom configuration ini file. Default: Use system configfile

See also

plugins.extract.pipeline: The extraction pipeline for calling plugins
plugins.extract.align: Aligner plugins
plugins.extract._base: Parent class for all extraction plugins
plugins.extract.detect._base: Detector parent class for extraction plugins.
plugins.extract.align._base: Aligner parent class for extraction plugins.

finalize(batch: BatchType) → Generator[ExtractMedia, None, None]

Finalize the output from Masker

This should be called as the final task of each plugin.

Pairs the detected faces back up with their original frame before yielding each frame.

Parameters:: batch (dict) – The final dict from the plugin process. It must contain the keys: detected_faces, filename, feed_faces, roi_masks
Yields:: ExtractMedia – The DetectedFaces list will be populated for this class with the bounding boxes, landmarks and masks for the detected faces found in the frame.

get_batch(queue: Queue) → tuple[bool, MaskerBatch]

Get items for inputting into the masker from the queue in batches

Items are returned from the queue in batches of batchsize

Items are received as ExtractMedia objects and converted to dict for internal processing.

To ensure consistent batch sizes for masker the items are split into separate items for each DetectedFace object.

Remember to put 'EOF' to the out queue after processing the final batch

Outputs items in the following format. All lists are of length batchsize:

>>> {'filename': [<filenames of source frames>],
>>>  'detected_faces': [[<lib.align.DetectedFace objects]]}

Parameters:

queue (queue.Queue()) – The queue that the plugin will be fed from.

Returns:

exhausted, bool – True if queue is exhausted, False if not
batch, ExtractorBatch – The batch object for the current batch

class plugins.extract.mask._base.MaskerBatch(image: list[np.ndarray] = <factory>, detected_faces: list[DetectedFace] = <factory>, filename: list[str] = <factory>, feed: np.ndarray = array([], dtype=float64), prediction: np.ndarray = array([], dtype=float64), data: list[dict[str, T.Any]] = <factory>, roi_masks: list[np.ndarray] = <factory>, feed_faces: list[AlignedFace] = <factory>)

Bases: ExtractorBatch

Dataclass for holding items flowing through the aligner.

Inherits from ExtractorBatch

Parameters:: roi_masks (list) – The region of interest masks for the batch

detected_faces: list[DetectedFace]

feed_faces: list[AlignedFace]

roi_masks: list[np.ndarray]

mask.bisenet_fp module 

BiSeNet Face-Parsing mask plugin

Architecture and Pre-Trained Model ported from PyTorch to Keras by TorzDF from https://github.com/zllrunning/face-parsing.PyTorch

class plugins.extract.mask.bisenet_fp.AttentionRefinementModule(filters: int)

Bases: object

The Attention Refinement block for BiSeNet Face Parsing

Parameters:: filters (int) – The dimensionality of the output space (i.e. the number of output filters in the convolution).

class plugins.extract.mask.bisenet_fp.BiSeNet(model_path: str, allow_growth: bool, exclude_gpus: list[int] | None, input_size: int, num_classes: int, cpu_mode: bool)

Bases: KSession

BiSeNet Face-Parsing Mask from https://github.com/zllrunning/face-parsing.PyTorch

PyTorch model implemented in Keras by TorzDF

Parameters:

model_path (str) – The path to the keras model file
allow_growth (bool) – Enable the Tensorflow GPU allow_growth configuration option. This option prevents Tensorflow from allocating all of the GPU VRAM, but can lead to higher fragmentation and slower performance
exclude_gpus (list) – A list of indices correlating to connected GPUs that Tensorflow should not use. Pass None to not exclude any GPUs
input_size (int) – The input size to the model
num_classes (int) – The number of segmentation classes to create
cpu_mode (bool, optional) – True run the model on CPU. Default: False

class plugins.extract.mask.bisenet_fp.BiSeNetOutput(filters: int, num_classes: int, label: str = '')

Bases: object

The BiSeNet Output block for Face Parsing

Parameters:

filters (int) – The dimensionality of the output space (i.e. the number of output filters in the convolution).
num_class (int) – The number of classes to generate
label – The label for this output (for naming). Default: “” (i.e. empty string, or no label)
str – The label for this output (for naming). Default: “” (i.e. empty string, or no label)
optional – The label for this output (for naming). Default: “” (i.e. empty string, or no label)

class plugins.extract.mask.bisenet_fp.ContextPath

Bases: object

The Context Path block for BiSeNet Face Parsing.

class plugins.extract.mask.bisenet_fp.ConvBn(filters: int, kernel_size: int = 3, strides: int = 1, padding: int = 1, activation: int = True, prefix: str = '', start_idx: int = 1)

Bases: object

Convolutional 3D with Batch Normalization block.

Parameters:

filters (int) – The dimensionality of the output space (i.e. the number of output filters in the convolution).
kernel_size (int, optional) – The height and width of the 2D convolution window. Default: 3
strides (int, optional) – The strides of the convolution along the height and width. Default: 1
padding (int, optional) – The amount of padding to apply prior to the first Convolutional Layer. Default: 1
activation (bool) – Whether to include ReLu Activation at the end of the block. Default: True
prefix (str, optional) – The prefix to name the layers within the block. Default: "" (empty string, i.e. no prefix)
start_idx (int, optional) – The starting index for naming the layers within the block. See _get_name() for more information. Default: 1

class plugins.extract.mask.bisenet_fp.FeatureFusionModule(filters: int)

Bases: object

The Feature Fusion block for BiSeNet Face Parsing

Parameters:: filters (int) – The dimensionality of the output space (i.e. the number of output filters in the convolution).

class plugins.extract.mask.bisenet_fp.Mask(**kwargs)

Bases: Masker

Neural network to process face image into a segmentation mask of the face

init_model() → None: Initialize the BiSeNet Face Parsing model.

predict(feed: ndarray) → ndarray: Run model to get predictions

process_input(batch: BatchType) → None: Compile the detected faces for prediction

process_output(batch: BatchType) → None: Compile found faces for output

class plugins.extract.mask.bisenet_fp.ResNet18

Bases: object

ResNet 18 block. Used at the start of BiSeNet Face Parsing.

recognition plugins package 

recognition._base module 

Base class for Face Recognition plugins

All Recognition Plugins should inherit from this class. See the override methods for which methods are required.

The plugin will receive a ExtractMedia object.

For each source frame, the plugin must pass a dict to finalize containing:

>>> {'filename': <filename of source frame>,
>>>  'detected_faces': <list of DetectedFace objects containing bounding box points}}

To get a DetectedFace object use the function:

>>> face = self.to_detected_face(<face left>, <face top>, <face right>, <face bottom>)

class plugins.extract.recognition._base.Identity(git_model_id: int | None = None, model_filename: str | None = None, configfile: str | None = None, instance: int = 0, **kwargs)

Bases: Extractor

Face Recognition Object

Parent class for all Recognition plugins

Parameters:

git_model_id (int) – The second digit in the github tag that identifies this model. See https://github.com/deepfakes-models/faceswap-models for more information
model_filename (str) – The name of the model file to be loaded
configfile (str, optional) – Path to a custom configuration ini file. Default: Use system configfile

See also

plugins.extract.pipeline: The extraction pipeline for calling plugins
plugins.extract.detect: Detector plugins
plugins.extract._base: Parent class for all extraction plugins
plugins.extract.align._base: Aligner parent class for extraction plugins.
plugins.extract.mask._base: Masker parent class for extraction plugins.

add_identity_filters(filters: ndarray, nfilters: ndarray, threshold: float) → None

Add identity encodings to filter by identity in the recognition plugin

Parameters:

filters (numpy.ndarray) – The array of filter embeddings to use
nfilters (numpy.ndarray) – The array of nfilter embeddings to use
threshold (float) – The threshold for a positive filter match

finalize(batch: BatchType) → Generator[ExtractMedia, None, None]

Finalize the output from Masker

This should be called as the final task of each plugin.

Pairs the detected faces back up with their original frame before yielding each frame.

Parameters:: batch (RecogBatch) – The final batch item from the plugin process.
Yields:: ExtractMedia – The DetectedFaces list will be populated for this class with the bounding boxes, landmarks and masks for the detected faces found in the frame.

get_batch(queue: Queue) → tuple[bool, RecogBatch]

Get items for inputting into the recognition from the queue in batches

Items are returned from the queue in batches of batchsize

Items are received as ExtractMedia objects and converted to RecogBatch for internal processing.

To ensure consistent batch sizes for masker the items are split into separate items for each DetectedFace object.

Remember to put 'EOF' to the out queue after processing the final batch

Outputs items in the following format. All lists are of length batchsize:

>>> {'filename': [<filenames of source frames>],
>>>  'detected_faces': [[<lib.align.DetectedFace objects]]}

Parameters:

queue (queue.Queue()) – The queue that the plugin will be fed from.

Returns:

exhausted, bool – True if queue is exhausted, False if not
batch, ExtractorBatch – The batch object for the current batch

class plugins.extract.recognition._base.IdentityFilter(save_output: bool)

Bases: object

Applies filters on the output of the recognition plugin

Parameters:: save_output (bool) – True if the filtered faces should be kept as they are being saved. False if they should be deleted

add_filters(filters: ndarray, nfilters: ndarray, threshold) → None

Add identity encodings to the filter and set whether each filter is enabled

Parameters:

filters (numpy.ndarray) – The array of filter embeddings to use
nfilters (numpy.ndarray) – The array of nfilter embeddings to use
threshold (float) – The threshold for a positive filter match

output_counts(): Output the counts of filtered items

class plugins.extract.recognition._base.RecogBatch(image: list[numpy.ndarray] = <factory>, detected_faces: list[lib.align.detected_face.DetectedFace] = <factory>, filename: list[str] = <factory>, feed: ~numpy.ndarray = array([], dtype=float64), prediction: ~numpy.ndarray = array([], dtype=float64), data: list[dict[str, typing.Any]] = <factory>, feed_faces: list[lib.align.aligned_face.AlignedFace] = <factory>)

Bases: ExtractorBatch

Dataclass for holding items flowing through the aligner.

Inherits from ExtractorBatch

detected_faces: list[DetectedFace]

feed_faces: list[AlignedFace]

recognition.vgg_face2 module 

VGG_Face2 inference and sorting

class plugins.extract.recognition.vgg_face2.Cluster(predictions: ndarray, method: Literal['single', 'centroid', 'median', 'ward'], threshold: float | None = None)

Bases: object

Cluster the outputs from a VGG-Face 2 Model

Parameters:

predictions (numpy.ndarray) – A stacked matrix of vgg_face2 predictions of the shape (N, D) where N is the number of observations and D are the number of dimensions. NB: The given predictions will be overwritten to save memory. If you still require the original values you should take a copy prior to running this method
method (['single','centroid','median','ward']) – The clustering method to use.
threshold (float, optional) – The threshold to start creating bins for. Set to None to disable binning

class plugins.extract.recognition.vgg_face2.Recognition(*args, **kwargs)

Bases: Identity

VGG Face feature extraction.

Extracts feature vectors from faces in order to compare similarity.

Notes

Input images should be in BGR Order

Model exported from: https://github.com/WeidiXie/Keras-VGGFace2-ResNet50 which is based on: https://www.robots.ox.ac.uk/~vgg/software/vgg_face/

Licensed under Creative Commons Attribution License. https://creativecommons.org/licenses/by-nc/4.0/

init_model() → None: Initialize VGG Face 2 Model.

predict(feed: ndarray) → ndarray

Return encodings for given image from vgg_face2.

Parameters:: batch (numpy.ndarray) – The face to be fed through the predictor. Should be in BGR channel order
Returns:: The encodings for the face
Return type:: numpy.ndarray

process_input(batch: BatchType) → None: Compile the detected faces for prediction

process_output(batch: BatchType) → None: No output processing for vgg_face2