extract package
The Extract Package handles the various plugins available for extracting face sets in Faceswap.
extract_media module
Object for holding and manipulating media passing through a faceswap extraction pipeline
- class plugins.extract.extract_media.ExtractMedia(filename: str, image: np.ndarray, detected_faces: list[DetectedFace] | None = None, is_aligned: bool = False)
Bases:
object
An object that passes through the
Extractor
pipeline.- Parameters:
filename (str) – The base name of the original frame’s filename
image (
numpy.ndarray
) – The original frame or a faceswap aligned face imagedetected_faces (list, optional) – A list of
DetectedFace
objects. Detected faces can be added later withadd_detected_faces()
. SettingNone
will default to an empty list. Default:None
is_aligned (bool, optional) –
True
if theimage
is an aligned faceswap image otherwiseFalse
. Used for face filtering with vggface2. Aligned faceswap images will automatically skip detection, alignment and masking. Default:False
- add_detected_faces(faces: list[DetectedFace]) None
Add detected faces to the object. Called at the end of each extraction phase.
- Parameters:
faces (list) – A list of
DetectedFace
objects
- add_frame_metadata(metadata: PNGHeaderSourceDict) None
Add the source frame metadata from an aligned PNG’s header data.
- metadata: dict
The contents of the ‘source’ field in the PNG header
- add_sub_folders(folders: list[str | None]) None
Add detected faces to the object. Called at the end of each extraction phase.
- Parameters:
folders (list) – A list of str sub folder names or
None
if no sub folder is required. Should correspond to the detected faces list
- property detected_faces: list[DetectedFace]
A list of
DetectedFace
objects in theimage
.- Type:
list
- property frame_metadata: PNGHeaderSourceDict
The frame metadata that has been added from an aligned image. This property should only be called after
add_frame_metadata()
has been called when processing an aligned face. For all other instances an assertion error will be raised.- Raises:
AssertionError – If frame metadata has not been populated from an aligned image
- Type:
dict
- get_image_copy(color_format: T.Literal['BGR', 'RGB', 'GRAY']) np.ndarray
Get a copy of the image in the requested color format.
- property image: np.ndarray
The source frame for this object.
- Type:
numpy.ndarray
- remove_image() None
Delete the image and reset
image
toNone
.Required for multi-phase extraction to avoid the frames stacking RAM.
- set_image(image: np.ndarray) None
Add the image back into
image
Required for multi-phase extraction adds the image back to this object.
- Parameters:
image (
numpy.ndarry
) – The original frame to be re-applied to for thisfilename
- property sub_folders: list[str | None]
The sub_folders that the faces should be output to. Used when binning filter output is enabled. The list corresponds to the list of detected faces
- Type:
list
pipeline module
Return a requested detector/aligner/masker pipeline
Tensorflow does not like to release GPU VRAM, so parallel plugins need to be managed to work together.
This module sets up a pipeline for the extraction workflow, loading detect, align and mask plugins either in parallel or in series, giving easy access to input and output.
- class plugins.extract.pipeline.Extractor(detector: str | None, aligner: str | None, masker: str | list[str] | None, recognition: str | None = None, configfile: str | None = None, multiprocess: bool = False, exclude_gpus: list[int] | None = None, rotate_images: str | None = None, min_size: int = 0, normalize_method: Literal['none', 'clahe', 'hist', 'mean'] | None = None, re_feed: int = 0, re_align: bool = False, disable_filter: bool = False)
Bases:
object
Creates a
detect
/align`
/mask
pipeline and yields results frame by frame from thedetected_faces
generatorinput_queue
is dynamically set depending on the currentphase
of extraction- Parameters:
detector (str or
None
) – The name of a detector plugin as exists inplugins.extract.detect
aligner (str or
None
) – The name of an aligner plugin as exists inplugins.extract.align
masker (str or list or
None
) – The name of a masker plugin(s) as exists inplugins.extract.mask
. This can be a single masker or a list of multiple maskersrecognition (str or
None
) – The name of the recognition plugin to use.None
to not do face recognition. Default:None
configfile (str, optional) – The path to a custom
extract.ini
configfile. IfNone
then the systemconfig/extract.ini
file will be used.multiprocess (bool, optional) – Whether to attempt processing the plugins in parallel. This may get overridden internally depending on the plugin combination. Default:
False
exclude_gpus (list, optional) – A list of indices correlating to connected GPUs that Tensorflow should not use. Pass
None
to not exclude any GPUs. Default:None
rotate_images (str, optional) – Used to set the
plugins.extract.detect.rotation
attribute. Pass in a single number to use increments of that size up to 360, or pass in alist
ofints
to enumerate exactly what angles to check. Can also pass in'on'
to increment at 90 degree intervals. Default:None
min_size (int, optional) – Used to set the
plugins.extract.detect.min_size
attribute. Filters out faces detected below this size. Length, in pixels across the diagonal of the bounding box. Set to0
for off. Default:0
normalize_method ({None, ‘clahe’, ‘hist’, ‘mean’}, optional) – Used to set the
plugins.extract.align.normalize_method
attribute. Normalize the images fed to the aligner.Default:None
re_feed (int) – The number of times to re-feed a slightly adjusted bounding box into the aligner. Default: 0
re_align (bool, optional) –
True
to obtain landmarks by passing the initially aligned face back through the aligner. DefaultFalse
disable_filter (bool, optional) – Disable all aligner filters regardless of config option. Default:
False
- phase
The current phase that the pipeline is running. Used in conjunction with
passes
andfinal_pass
to indicate to the caller which phase is being processed- Type:
str
- detected_faces() Generator[ExtractMedia, None, None]
Generator that returns results, frame by frame from the extraction pipeline
This is the exit point for the extraction pipeline and is used to obtain the output of any pipeline
phase
- Yields:
faces (
ExtractMedia
) – The populated extracted media object.
Example
>>> for extract_media in extractor.detected_faces(): >>> filename = extract_media.filename >>> image = extract_media.image >>> detected_faces = extract_media.detected_faces
- property final_pass: bool
bool, Return
True
if this is the final extractor pass otherwiseFalse
Useful for iterating over the pipeline
passes
ordetected_faces()
and handling accordingly.Example
>>> for face in extractor.detected_faces(): >>> if extractor.final_pass: >>> <do final processing> >>> else: >>> extract_media.set_image(image) >>> <do intermediate processing> >>> extractor.input_queue.put(extract_media)
- import_data(input_location: str) None
Import json data to the detector and/or aligner if ‘import’ plugin has been selected
- Parameters:
input_location (str) – Full path to the input location for the extract process
- property input_queue: EventQueue
Return the correct input queue depending on the current phase
The input queue is the entry point into the extraction pipeline. An
ExtractMedia
object should be put to the queue.For detect/single phase operations the
ExtractMedia.filename
andimage
attributes should be populated.For align/mask (2nd/3rd pass operations) the
ExtractMedia.detected_faces
should also be populated by callingExtractMedia.set_detected_faces()
.- Type:
queue
- launch() None
Launches the plugin(s)
This launches the plugins held in the pipeline, and should be called at the beginning of each
phase
. To ensure VRAM is conserved, It will only launch the plugin(s) required for the currently running phaseExample
>>> for phase in extractor.passes: >>> extractor.launch(): >>> <do processing>
- property passes: int
Returns the total number of passes the extractor needs to make.
This is calculated on several factors (vram available, plugin choice,
multiprocess
etc.). It is useful for iterating over the pipeline and handling accordingly.Example
>>> for phase in extractor.passes: >>> if phase == 1: >>> extract_media = ExtractMedia("path/to/image/file", image) >>> extractor.input_queue.put(extract_media) >>> else: >>> extract_media.set_image(image) >>> extractor.input_queue.put(extract_media)
- Type:
int
- property phase_text: str
The plugins that are running in the current phase, formatted for info text output.
- Type:
str
- reset_phase_index() None
Reset the current phase index back to 0. Used for when batch processing is used in extract.
- set_batchsize(plugin_type: Literal['align', 'detect'], batchsize: int) None
Set the batch size of a given
plugin_type
to the givenbatchsize
.This should be set prior to
launch()
if the batch size is to be manually overridden- Parameters:
plugin_type ({'align', 'detect'}) – The plugin_type to be overridden
batchsize (int) – The batch size to use for this plugin type
_base module
Base class for Faceswap detect
, align
and
mask
Plugins
- class plugins.extract._base.Extractor(git_model_id: int | None = None, model_filename: str | list[str] | None = None, exclude_gpus: list[int] | None = None, configfile: str | None = None, instance: int = 0)
Bases:
object
Extractor Plugin Object
All
_base
classes for Aligners, Detectors and Maskers inherit from this class.This class sets up a pipeline for working with ML plugins.
Plugins are split into 3 threads, to utilize Numpy and CV2s parallel processing, as well as allow the predict function of the model to sit in a dedicated thread. A plugin is expected to have 3 core functions, each in their own thread: -
process_input()
- Prepare the data for feeding into a model -predict()
- Feed the data through the model -process_output()
- Perform any data post-processing- Parameters:
git_model_id (int) – The second digit in the github tag that identifies this model. See https://github.com/deepfakes-models/faceswap-models for more information
model_filename (str) – The name of the model file to be loaded
exclude_gpus (list, optional) – A list of indices correlating to connected GPUs that Tensorflow should not use. Pass
None
to not exclude any GPUs. Default:None
configfile (str, optional) – Path to a custom configuration
ini
file. Default: Use system configfileinstance (int, optional) – If this plugin is being executed multiple times (i.e. multiple pipelines have been launched), the instance of the plugin must be passed in for naming convention reasons. Default: 0
The following attributes should be set in the plugin’s
__init__()
method after initializing the parent.- name
Name of this plugin. Used for display purposes.
- Type:
str
- input_size
The input size to the model in pixels across one edge. The input size should always be square.
- Type:
int
- color_format
Color format for model. Must be
'BGR'
,'RGB'
or'GRAY'
. Defaults to'BGR'
if not explicitly set.- Type:
str
- vram
Approximate VRAM used by the model at
input_size
. Used to calculate thebatchsize
. Be conservative to avoid OOM.- Type:
int
- vram_warnings
Approximate VRAM used by the model at
input_size
that will still run, but generates warnings. Used to calculate thebatchsize
. Be conservative to avoid OOM.- Type:
int
- vram_per_batch
Approximate additional VRAM used by the model for each additional batch. Used to calculate the
batchsize
. Be conservative to avoid OOM.- Type:
int
See also
plugins.extract.detect._base
Detector parent class for extraction plugins.
plugins.extract.align._base
Aligner parent class for extraction plugins.
plugins.extract.mask._base
Masker parent class for extraction plugins.
plugins.extract.pipeline
The extract pipeline that configures and calls all plugins
- batchsize
Batchsize for feeding this model. The number of images the model should feed through at once.
- Type:
int
- check_and_raise_error() None
Check all threads for errors
Exposed for
pipeline
to check plugin’s threads for errors
- config
Config for this plugin, loaded from
extract.ini
configfile- Type:
dict
- finalize(batch: BatchType) Generator[ExtractMedia, None, None]
Override method (at <plugin_type> level)
This method should be overridden at the <plugin_type> level (IE.
plugins.extract.detect._base
,plugins.extract.align._base
orplugins.extract.mask._base
) and should not be overridden within plugins themselves.Handles consistent finalization for all plugins that exist within that plugin type. Its input is always the output from
process_output()
- Parameters:
batch (
ExtractorBatch
) – Contains the batch that is currently being passed through the plugin process
- get_batch(queue: Queue) tuple[bool, BatchType]
Override method (at <plugin_type> level)
This method should be overridden at the <plugin_type> level (IE.
plugins.extract.detect._base
,plugins.extract.align._base
orplugins.extract.mask._base
) and should not be overridden within plugins themselves.Get
ExtractMedia
items from the queue in batches ofbatchsize
- Parameters:
queue (queue.Queue()) – The
queue
that the batch will be fed from. This will be the input to the plugin.
- init_model() None
Override method
Override this method to execute the specific model initialization method
- model: KSession | cv2.dnn.Net | None
The model for this plugin. Set in the plugin’s
init_model()
method- Type:
varies
- model_path
Path to the model file(s) (if required). Multiple model files should be a list of strings
- Type:
str or list
- on_completion() None
Override to perform an action when the extract process has completed. By default, no action is undertaken
- predict(feed: ndarray) ndarray
Override method
Override this method for specific extractor model prediction function
- Parameters:
feed (
numpy.ndarray
) – The feed images for the batch
Notes
Input for
predict()
should have been set inprocess_input()
Output from the model should populate the key
prediction
of thebatch
.- For Detect:
the expected output for the
prediction
of thebatch
should be alist
ofbatchsize
of detected face points. These points should be either alist
,tuple
ornumpy.ndarray
with the first 4 items being the left, top, right, bottom points, in that order
- process_input(batch: BatchType) None
Override method
Override this method for specific extractor pre-processing of image
- Parameters:
batch (
ExtractorBatch
) – Contains the batch that is currently being passed through the plugin process
- process_output(batch: BatchType) None
Override method
Override this method for specific extractor model post predict function
- Parameters:
batch (
ExtractorBatch
) – Contains the batch that is currently being passed through the plugin process
Notes
- queue_size
Queue size for all internal queues. Set in
initialize()
- Type:
int
- rollover_collector(queue: Queue) T.Literal['EOF'] | ExtractMedia
For extractors after the Detectors, the number of detected faces per frame vs extractor batch size mean that faces will need to be split/re-joined with frames. The rollover collector can be used to rollover items that don’t fit in a batch.
Collect the item from the
_rollover
dict or from the queue. Add face count per frame to self._faces_per_filename for joining batches back up in finalize- Parameters:
queue (
queue.Queue
) – The input queue to the aligner. Should containExtractMedia
objects- Returns:
The next extract media object, or EOF if pipe has ended
- Return type:
ExtractMedia
or EOF
- class plugins.extract._base.ExtractorBatch(image: list[np.ndarray] = <factory>, detected_faces: Sequence[DetectedFace | list[DetectedFace]] = <factory>, filename: list[str] = <factory>, feed: np.ndarray = array([], dtype=float64), prediction: np.ndarray = array([], dtype=float64), data: list[dict[str, T.Any]] = <factory>)
Bases:
object
Dataclass for holding a batch flowing through post Detector plugins.
The batch size for post Detector plugins is not the same as the overall batch size. An image may contain 0 or more detected faces, and these need to be split and recombined to be able to utilize a plugin’s internal batch size.
Plugin types will inherit from this class and add required keys.
- Parameters:
image (list) – List of
numpy.ndarray
containing the original framesdetected_faces (list) – List of
DetectedFace
objectsfilename (list) – List of original frame filenames for the batch
feed (
numpy.ndarray
) – Batch of feed images to feed the net withprediction (
numpy.nd.array
) – Batch of predictions. Direct output from the aligner netdata (dict) – Any specific data required during the processing phase for a particular plugin
- data: list[dict[str, T.Any]]
- detected_faces: Sequence[DetectedFace | list[DetectedFace]]
- feed: np.ndarray = array([], dtype=float64)
- filename: list[str]
- image: list[np.ndarray]
- prediction: np.ndarray = array([], dtype=float64)
align plugins package
align._base.aligner module
Base class for Face Aligner plugins
All Aligner Plugins should inherit from this class. See the override methods for which methods are required.
The plugin will receive a ExtractMedia
object.
For each source item, the plugin must pass a dict to finalize containing:
>>> {"filename": [<filename of source frame>],
>>> "landmarks": [list of 68 point face landmarks]
>>> "detected_faces": [<list of DetectedFace objects>]}
- class plugins.extract.align._base.aligner.Aligner(git_model_id: int | None = None, model_filename: str | None = None, configfile: str | None = None, instance: int = 0, normalize_method: Literal['none', 'clahe', 'hist', 'mean'] | None = None, re_feed: int = 0, re_align: bool = False, disable_filter: bool = False, **kwargs)
Bases:
Extractor
Aligner plugin _base Object
All Aligner plugins must inherit from this class
- Parameters:
git_model_id (int) – The second digit in the github tag that identifies this model. See https://github.com/deepfakes-models/faceswap-models for more information
model_filename (str) – The name of the model file to be loaded
normalize_method ({None, ‘clahe’, ‘hist’, ‘mean’}, optional) – Normalize the images fed to the aligner. Default:
None
re_feed (int, optional) – The number of times to re-feed a slightly adjusted bounding box into the aligner. Default: 0
re_align (bool, optional) –
True
to obtain landmarks by passing the initially aligned face back through the aligner. DefaultFalse
disable_filter (bool, optional) – Disable all aligner filters regardless of config option. Default:
False
configfile (str, optional) – Path to a custom configuration
ini
file. Default: Use system configfile
See also
plugins.extract.pipeline
The extraction pipeline for calling plugins
plugins.extract.align
Aligner plugins
plugins.extract._base
Parent class for all extraction plugins
plugins.extract.detect._base
Detector parent class for extraction plugins.
plugins.extract.mask._base
Masker parent class for extraction plugins.
- faces_to_feed(faces: ndarray) ndarray
Overide for specific plugin processing to convert a batch of face images from UINT8 (0-255) into the correct format for the plugin’s inference
- Parameters:
faces (
numpy.ndarray
) – The batch of faces in UINT8 format- Returns:
class – The batch of faces in the format to feed through the plugin
- Return type:
numpy.ndarray
- finalize(batch: BatchType) Generator[ExtractMedia, None, None]
Finalize the output from Aligner
This should be called as the final task of each plugin.
Pairs the detected faces back up with their original frame before yielding each frame.
- Parameters:
batch (
AlignerBatch
) – The final batch item from the plugin process.- Yields:
ExtractMedia
– TheDetectedFaces
list will be populated for this class with the bounding boxes and landmarks for the detected faces found in the frame.
- get_batch(queue: Queue) tuple[bool, AlignerBatch]
Get items for inputting into the aligner from the queue in batches
Items are returned from the
queue
in batches ofbatchsize
Items are received as
ExtractMedia
objects and converted todict
for internal processing.To ensure consistent batch sizes for aligner the items are split into separate items for each
DetectedFace
object.Remember to put
'EOF'
to the out queue after processing the final batchOutputs items in the following format. All lists are of length
batchsize
:>>> {'filename': [<filenames of source frames>], >>> 'image': [<source images>], >>> 'detected_faces': [[<lib.align.DetectedFace objects]]}
- Parameters:
queue (queue.Queue()) – The
queue
that the plugin will be fed from.- Returns:
exhausted, bool –
True
if queue is exhausted,False
if notbatch,
ExtractorBatch
– The batch object for the current batch
- initialize(*args, **kwargs) None
Add a call to add model input size to the re-aligner
- on_completion() None
Output the filter counts when process has completed
- set_normalize_method(method: Literal['none', 'clahe', 'hist', 'mean'] | None) None
Set the normalization method for feeding faces into the aligner.
- Parameters:
method ({"none", "clahe", "hist", "mean"}) – The normalization method to apply to faces prior to feeding into the model
- class plugins.extract.align._base.aligner.AlignerBatch(image: list[np.ndarray] = <factory>, detected_faces: list[DetectedFace] = <factory>, filename: list[str] = <factory>, feed: np.ndarray = array([], dtype=float64), prediction: np.ndarray = array([], dtype=float64), data: list[dict[str, T.Any]] = <factory>, batch_id: int = 0, landmarks: np.ndarray = array([], dtype=float64), refeeds: list[np.ndarray] = <factory>, second_pass: bool = False, second_pass_masks: np.ndarray = array([], dtype=float64))
Bases:
ExtractorBatch
Dataclass for holding items flowing through the aligner.
Inherits from
ExtractorBatch
- Parameters:
batch_id (int) – A unique integer for tracking this batch
landmarks (list) – List of 68 point
numpy.ndarray
landmark points returned from the alignerrefeeds (list) – List of
numpy.ndarrays
for holding each of the feeds that will be put through the model for each refeedsecond_pass (bool, optional) –
True
if this batch is passing through the aligner for a second time as re-align has been selected otherwiseFalse
. Default:False
second_pass_masks (
numpy.ndarray
, optional) – The masks used to filter out re-feed values for passing to the re-aligner.
- batch_id: int = 0
- detected_faces: list[DetectedFace]
- landmarks: np.ndarray = array([], dtype=float64)
- refeeds: list[np.ndarray]
- second_pass: bool = False
- second_pass_masks: np.ndarray = array([], dtype=float64)
align._base.processing module
Processing methods for aligner plugins
- class plugins.extract.align._base.processing.AlignedFilter(feature_filter: bool, min_scale: float, max_scale: float, distance: float, roll: float, save_output: bool, disable: bool = False)
Bases:
object
Applies filters on the output of the aligner
- Parameters:
feature_filter (bool) –
True
to enable filter to check relative position of eyes/eyebrows and mouth.False
to disable.min_scale (float) – Filters out faces that have been aligned at below this value as a multiplier of the minimum frame dimension. Set to
0
for off.max_scale (float) – Filters out faces that have been aligned at above this value as a multiplier of the minimum frame dimension. Set to
0
for off.distance (float) – Filters out faces that are further than this distance from an “average” face. Set to
0
for off.roll (float) – Filters out faces with a roll value outside of 0 +/- the value given here. Set to
0
for off.save_output (bool) –
True
if the filtered faces should be kept as they are being saved.False
if they should be deleteddisable (bool, Optional) –
True
to disable the filter regardless of config options. Default:False
- filtered_mask(batch: AlignerBatch, skip: np.ndarray | list[int] | None = None) np.ndarray
Obtain a list of boolean values for the given batch indicating whether they pass the filter test.
- Parameters:
batch (
AlignerBatch
) – The batch of face to obtain masks forskip (list or
numpy.ndarray
, optional) – List or 1D numpy array of indices indicating faces that have already been filter masked and so should not be filtered again. Values in these index positions will be returned asTrue
- Returns:
Boolean mask array corresponding to any of the input DetectedFace objects that passed a test.
False
the face passed the test.True
it failed- Return type:
numpy.ndarray
- output_counts()
Output the counts of filtered items
- class plugins.extract.align._base.processing.ReAlign(active: bool, do_refeeds: bool, do_filter: bool)
Bases:
object
Holds data and methods for 2nd pass re-aligns
- Parameters:
active (bool) –
True
if re-alignment has been requested otherwiseFalse
do_refeeds (bool) –
True
if re-feeds should be re-aligned,False
if just the final output of the re-feeds should be aligneddo_filter (bool) –
True
if aligner filtered out faces should not be re-aligned.False
if all faces should be re-aligned
- property active: bool
True
if re_aligns have been selected otherwiseFalse
- Type:
bool
- add_batch(batch: AlignerBatch) None
Add first pass alignments to the queue for picking up for re-alignment, update their
second_pass
attribute toTrue
and clear attributes not required.- Parameters:
batch (
AlignerBatch
) – aligner batch to perform re-alignment on
- property do_filter: bool
True
if re-aligning is active and faces which failed the aligner filter test should not be re-aligned otherwiseFalse
- Type:
bool
- property do_refeeds: bool
True
if re-aligning is active and re-aligning re-feeds has been selected otherwiseFalse
- Type:
bool
- get_batch() AlignerBatch
Retrieve the next batch currently queued for re-alignment
- Returns:
The next
AlignerBatch
for re-alignment- Return type:
AlignerBatch
- property items_queued: bool
True
if re-align is active and items are queued for a 2nd pass otherwiseFalse
- Type:
bool
- property items_tracked: bool
True
if items exist in the tracker so still need to be processed- Type:
bool
- process_batch(batch: AlignerBatch) list[np.ndarray]
Pre process a batch object for re-aligning through the aligner.
- Parameters:
batch (
AlignerBatch
) – aligner batch to perform pre-processing on- Returns:
List of UINT8 aligned faces batch for each selected refeed
- Return type:
list
- process_output(subbatches: list[AlignerBatch], batch_masks: np.ndarray) None
Process the output from the re-align pass.
Transform landmarks from aligned face space to face space
Re-insert faces that were filtered out from the re-align process back into the landmarks list
- Parameters:
subbatches (list) – List of sub-batch results for each re-aligned re-feed performed
batch_masks (
numpy.ndarray
) – The original re-feed filter masks from the first pass
- set_input_size_and_centering(input_size: int, centering: CenteringType) None
Set the input size of the loaded plugin once the model has been loaded
- Parameters:
input_size (int) – The input size, in pixels, of the aligner plugin
centering (["face", "head" or "legacy"]) – The centering to align the image at for re-aligning
- track_batch(batch_id: int) None
Add newly seen batch id from the aligner to the batch tracker, so that we can keep track of whether there are still batches to be processed when the aligner hits ‘EOF’
- Parameters:
batch_id (int) – The batch id to add to batch tracking
- untrack_batch(batch_id: int) None
Remove the tracked batch from the tracker once the batch has been fully processed
- Parameters:
batch_id (int) – The batch id to remove from batch tracking
align.cv2_dnn module
CV2 DNN landmarks extractor for faceswap.py Adapted from: https://github.com/yinguobing/cnn-facial-landmark MIT License
Copyright (c) 2017 Yin Guobing
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the “Software”), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
- class plugins.extract.align.cv2_dnn.Align(**kwargs)
Bases:
Aligner
Perform transformation to align and get landmarks
- align_image(batch: AlignerBatch) tuple[list[numpy.ndarray], list[list[int]], list[tuple[int, int]]]
Align the incoming image for prediction
- Parameters:
batch (
AlignerBatch
) – The current batch to align the input for- Returns:
faces (list) – List of feed faces for the aligner
rois (list) – List of roi’s for the faces
offsets (list) – List of offsets for the faces
- faces_to_feed(faces: ndarray) ndarray
Convert a batch of face images from UINT8 (0-255) to fp32 (0.0-255.0)
- Parameters:
faces (
numpy.ndarray
) – The batch of faces in UINT8 format- Returns:
class – The batch of faces as fp32
- Return type:
numpy.ndarray
- get_pts_from_predict(batch: AlignerBatch)
Get points from predictor and populates the
landmarks
property- Parameters:
batch (
AlignerBatch
) – The current batch from the model withpredictions
populated
- static get_square_box(box: list[int]) list[int]
Get a square box out of the given box, by expanding it.
- Parameters:
box (list) – The (left, top, right, bottom) box positions
- Returns:
The original box but made square
- Return type:
list
- init_model() None
Initialize CV2 DNN Detector Model
- classmethod move_box(box: list[int], offset: tuple[int, int]) list[int]
Move the box to direction specified by vector offset
- Parameters:
box (list) – The (left, top, right, bottom) box positions
offset (tuple) – (x, y) offset to move the box
- Returns:
The original box shifted by the offset
- Return type:
list
- classmethod pad_image(box: list[int], image: ndarray) tuple[numpy.ndarray, tuple[int, int]]
Pad image if face-box falls outside of boundaries
- Parameters:
box (list) – The (left, top, right, bottom) roi box positions
image (
numpy.ndarray
) – The image to be padded
- Returns:
The padded image
- Return type:
numpy.ndarray
- predict(feed: ndarray) ndarray
Predict the 68 point landmarks
- Parameters:
feed (
numpy.ndarray
) – The batch to feed into the aligner- Returns:
The predictions from the aligner
- Return type:
numpy.ndarray
- process_input(batch: BatchType) None
Compile the detected faces for prediction
- Parameters:
batch (
AlignerBatch
) – The current batch to process input for- Returns:
The batch item with the
feed
populated and any requireddata
added- Return type:
AlignerBatch
- process_output(batch: BatchType) None
Process the output from the model
- Parameters:
batch (
AlignerBatch
) – The current batch from the model withpredictions
populated
align.fan module
Facial landmarks extractor for faceswap.py Code adapted and modified from: https://github.com/1adrianb/face-alignment
- class plugins.extract.align.fan.Align(**kwargs)
Bases:
Aligner
Perform transformation to align and get landmarks
- crop(batch: AlignerBatch, center_scale: ndarray) list[numpy.ndarray]
Crop image around the center point
- Parameters:
batch (
AlignerBatch
) – The current batch to crop the image forcenter_scale (
numpy.ndarray
) – The center and scale for the bounding box
- Returns:
List of cropped images for the batch
- Return type:
list
- faces_to_feed(faces: ndarray) ndarray
Convert a batch of face images from UINT8 (0-255) to fp32 (0.0-1.0)
- Parameters:
faces (
numpy.ndarray
) – The batch of faces in UINT8 format- Returns:
class – The batch of faces as fp32 in 0.0 to 1.0 range
- Return type:
numpy.ndarray
- get_center_scale(detected_faces: list[DetectedFace]) np.ndarray
Get the center and set scale of bounding box
- Parameters:
detected_faces (list) – List of
DetectedFace
objects for the batch- Returns:
The center and scale of the bounding box
- Return type:
numpy.ndarray
- get_pts_from_predict(batch: AlignerBatch) None
Get points from predictor and populate the
landmarks
property of theAlignerBatch
- Parameters:
batch (
AlignerBatch
) – The current batch from the model withpredictions
populated
- init_model() None
Initialize FAN model
- predict(feed: ndarray) ndarray
Predict the 68 point landmarks
- Parameters:
batch (
numpy.ndarray
) – The batch to feed into the aligner- Returns:
The predictions from the aligner
- Return type:
numpy.ndarray
- process_input(batch: BatchType) None
Compile the detected faces for prediction
- Parameters:
batch (
AlignerBatch
) – The current batch to process input for
- process_output(batch: BatchType) None
Process the output from the model
- Parameters:
batch (
AlignerBatch
) – The current batch from the model withpredictions
populated
- classmethod transform(points: ndarray, center_scales: ndarray, resolutions: ndarray) ndarray
Transform Image
- Parameters:
points (
numpy.ndarray
) – The points to transformcenter_scales (
numpy.ndarray
) – The calculated centers and scales for the batchresolutions (
numpy.ndarray
) – The resolutions
detect plugins package
detect._base module
Base class for Face Detector plugins
All Detector Plugins should inherit from this class. See the override methods for which methods are required.
The plugin will receive a ExtractMedia
object.
For each source frame, the plugin must pass a dict to finalize containing:
>>> {'filename': <filename of source frame>,
>>> 'detected_faces': <list of DetectedFace objects containing bounding box points}}
To get a DetectedFace
object use the function:
>>> face = self._to_detected_face(<face left>, <face top>, <face right>, <face bottom>)
- class plugins.extract.detect._base.Detector(git_model_id: int | None = None, model_filename: str | list[str] | None = None, configfile: str | None = None, instance: int = 0, rotation: str | None = None, min_size: int = 0, **kwargs)
Bases:
Extractor
Detector Object
Parent class for all Detector plugins
- Parameters:
git_model_id (int) – The second digit in the github tag that identifies this model. See https://github.com/deepfakes-models/faceswap-models for more information
model_filename (str) – The name of the model file to be loaded
rotation (str, optional) – Pass in a single number to use increments of that size up to 360, or pass in a
list
ofints
to enumerate exactly what angles to check. Can also pass in'on'
to increment at 90 degree intervals. Default:None
min_size (int, optional) – Filters out faces detected below this size. Length, in pixels across the diagonal of the bounding box. Set to
0
for off. Default:0
configfile (str, optional) – Path to a custom configuration
ini
file. Default: Use system configfile
See also
plugins.extract.pipeline
The extraction pipeline for calling plugins
plugins.extract.detect
Detector plugins
plugins.extract._base
Parent class for all extraction plugins
plugins.extract.align._base
Aligner parent class for extraction plugins.
plugins.extract.mask._base
Masker parent class for extraction plugins.
- finalize(batch: BatchType) Generator[ExtractMedia, None, None]
Finalize the output from Detector
This should be called as the final task of each
plugin
.- Parameters:
batch (
ExtractorBatch
) – The batch object for the current batch- Yields:
ExtractMedia
– TheDetectedFaces
list will be populated for this class with the bounding boxes for the detected faces found in the frame.
- get_batch(queue: Queue) tuple[bool, DetectorBatch]
Get items for inputting to the detector plugin in batches
Items are received as
ExtractMedia
objects and converted todict
for internal processing.Items are returned from the
queue
in batches ofbatchsize
Remember to put
'EOF'
to the out queue after processing the final batchOutputs items in the following format. All lists are of length
batchsize
:>>> {'filename': [<filenames of source frames>], >>> 'image': <numpy.ndarray of images standardized for prediction>, >>> 'scale': [<scaling factors for each image>], >>> 'pad': [<padding for each image>], >>> 'detected_faces': [[<lib.align.DetectedFace objects]]}
- Parameters:
queue (queue.Queue()) – The
queue
that the batch will be fed from. This will be a queue that loads images.- Returns:
exhausted, bool –
True
if queue is exhausted,False
if not.batch,
ExtractorBatch
– The batch object for the current batch
- class plugins.extract.detect._base.DetectorBatch(image: list[numpy.ndarray] = <factory>, detected_faces: list[list['DetectedFace']] = <factory>, filename: list[str] = <factory>, feed: ~numpy.ndarray = array([], dtype=float64), prediction: ~numpy.ndarray = array([], dtype=float64), data: list[dict[str, typing.Any]] = <factory>, rotation_matrix: list[numpy.ndarray] = <factory>, scale: list[float] = <factory>, pad: list[tuple[int, int]] = <factory>, initial_feed: ~numpy.ndarray = array([], dtype=float64))
Bases:
ExtractorBatch
Dataclass for holding items flowing through the aligner.
Inherits from
ExtractorBatch
- Parameters:
rotation_matrix (
numpy.ndarray
) – The rotation matrix for any requested rotationsscale (float) – The scaling factor to take the input image back to original size
pad (tuple) – The amount of padding to apply to the image to feed the network
initial_feed (
numpy.ndarray
) – Used to hold the initialfeed
when rotate images is enabled
- detected_faces: list[list['DetectedFace']]
- initial_feed: np.ndarray = array([], dtype=float64)
- pad: list[tuple[int, int]]
- rotation_matrix: list[np.ndarray]
- scale: list[float]
detect.mtcnn module
MTCNN Face detection plugin
- class plugins.extract.detect.mtcnn.Detect(**kwargs)
Bases:
Detector
MTCNN detector for face recognition.
- init_model() None
Initialize MTCNN Model.
- predict(feed: ndarray) ndarray
Run model to get predictions
- Parameters:
batch (
DetectorBatch
) – Contains the batch to pass through the MTCNN model- Returns:
The batch with the predictions added to the dictionary
- Return type:
dict
- process_input(batch: BatchType) None
Compile the detection image(s) for prediction
- Parameters:
batch (
DetectorBatch
) – Contains the batch that is currently being passed through the plugin process
- process_output(batch: BatchType) None
MTCNN performs no post processing so the original batch is returned
- Parameters:
batch (
DetectorBatch
) – Contains the batch to apply postprocessing to
- class plugins.extract.detect.mtcnn.MTCNN(model_path: list[str], allow_growth: bool, exclude_gpus: list[int] | None, cpu_mode: bool, input_size: int = 640, minsize: int = 20, threshold: list[float] | None = None, factor: float = 0.709)
Bases:
object
MTCNN Detector for face alignment
- Parameters:
model_path (list) – List of paths to the 3 MTCNN subnet weights
allow_growth (bool, optional) – Enable the Tensorflow GPU allow_growth configuration option. This option prevents Tensorflow from allocating all of the GPU VRAM, but can lead to higher fragmentation and slower performance. Default:
False
exclude_gpus (list, optional) – A list of indices correlating to connected GPUs that Tensorflow should not use. Pass
None
to not exclude any GPUs. Default:None
cpu_mode (bool, optional) –
True
run the model on CPU. Default:False
input_size (int, optional) – The height, width input size to the model. Default: 640
minsize (int, optional) – The minimum size of a face to accept as a detection. Default: 20
threshold (list, optional) – List of floats for the three steps, Default: [0.6, 0.7, 0.7]
factor (float, optional) – The factor used to create a scaling pyramid of face sizes to detect in the image. Default: 0.709
- detect_faces(batch: ndarray) tuple[numpy.ndarray, tuple[numpy.ndarray]]
Detects faces in an image, and returns bounding boxes and points for them.
- Parameters:
batch (
numpy.ndarray
) – The input batch of images to detect face in- Returns:
list of numpy arrays containing the bounding box and 5 point landmarks of detected faces
- Return type:
List
- class plugins.extract.detect.mtcnn.ONet(model_path: str, allow_growth: bool, exclude_gpus: list[int] | None, cpu_mode: bool, input_size: int, threshold: float)
Bases:
KSession
Keras O-Net model for MTCNN
- Parameters:
model_path (str) – The path to the keras model file
allow_growth (bool, optional) – Enable the Tensorflow GPU allow_growth configuration option. This option prevents Tensorflow from allocating all of the GPU VRAM, but can lead to higher fragmentation and slower performance. Default:
False
exclude_gpus (list, optional) – A list of indices correlating to connected GPUs that Tensorflow should not use. Pass
None
to not exclude any GPUs. Default:None
cpu_mode (bool, optional) –
True
run the model on CPU. Default:False
input_size (int) – The input size of the model
threshold (list, optional) – Threshold for O-Net
- static model_definition() tuple[list[Tensor], list[Tensor]]
Keras O-Network for MTCNN
- class plugins.extract.detect.mtcnn.PNet(model_path: str, allow_growth: bool, exclude_gpus: list[int] | None, cpu_mode: bool, input_size: int, min_size: int, factor: float, threshold: float)
Bases:
KSession
Keras P-Net model for MTCNN
- Parameters:
model_path (str) – The path to the keras model file
allow_growth (bool, optional) – Enable the Tensorflow GPU allow_growth configuration option. This option prevents Tensorflow from allocating all of the GPU VRAM, but can lead to higher fragmentation and slower performance. Default:
False
exclude_gpus (list, optional) – A list of indices correlating to connected GPUs that Tensorflow should not use. Pass
None
to not exclude any GPUs. Default:None
cpu_mode (bool, optional) –
True
run the model on CPU. Default:False
input_size (int) – The input size of the model
minsize (int, optional) – The minimum size of a face to accept as a detection. Default: 20
threshold (list, optional) – Threshold for P-Net
- static model_definition() tuple[list[Tensor], list[Tensor]]
Keras P-Network Definition for MTCNN
- class plugins.extract.detect.mtcnn.RNet(model_path: str, allow_growth: bool, exclude_gpus: list[int] | None, cpu_mode: bool, input_size: int, threshold: float)
Bases:
KSession
Keras R-Net model Definition for MTCNN
- Parameters:
model_path (str) – The path to the keras model file
allow_growth (bool, optional) – Enable the Tensorflow GPU allow_growth configuration option. This option prevents Tensorflow from allocating all of the GPU VRAM, but can lead to higher fragmentation and slower performance. Default:
False
exclude_gpus (list, optional) – A list of indices correlating to connected GPUs that Tensorflow should not use. Pass
None
to not exclude any GPUs. Default:None
cpu_mode (bool, optional) –
True
run the model on CPU. Default:False
input_size (int) – The input size of the model
threshold (list, optional) – Threshold for R-Net
- static model_definition() tuple[list[Tensor], list[Tensor]]
Keras R-Network Definition for MTCNN
- plugins.extract.detect.mtcnn.nms(rectangles: ndarray, scores: ndarray, threshold: float, method: str = 'iom') tuple[numpy.ndarray, numpy.ndarray]
apply non-maximum suppression on ROIs in same scale(matrix version)
- Parameters:
rectangles (
np.ndarray
) – The [b, l, t, r, b] bounding box detection candidatesthreshold (float) – Threshold for succesful match
method (str, optional) – “iom” method or default. Defalt: “iom”
- Returns:
rectangles (
np.ndarray
) – The [b, l, t, r, b] bounding boxesscores
np.ndarray
– The associated scores for the rectangles
- plugins.extract.detect.mtcnn.rect2square(rectangles: ndarray) ndarray
change rectangles into squares (matrix version)
- Parameters:
rectangles (
numpy.ndarray
) – [b, x, y, x1, y1] rectangles- Returns:
Original rectangle changed to a square
- Return type:
list
mask plugins package
mask._base module
Base class for Face Masker plugins
Plugins should inherit from this class
See the override methods for which methods are required.
The plugin will receive a ExtractMedia
object.
For each source item, the plugin must pass a dict to finalize containing:
>>> {"filename": <filename of source frame>,
>>> "detected_faces": <list of bounding box dicts from lib/plugins/extract/detect/_base>}
- class plugins.extract.mask._base.Masker(git_model_id: int | None = None, model_filename: str | None = None, configfile: str | None = None, instance: int = 0, **kwargs)
Bases:
Extractor
Masker plugin _base Object
All Masker plugins must inherit from this class
- Parameters:
git_model_id (int) – The second digit in the github tag that identifies this model. See https://github.com/deepfakes-models/faceswap-models for more information
model_filename (str) – The name of the model file to be loaded
configfile (str, optional) – Path to a custom configuration
ini
file. Default: Use system configfile
See also
plugins.extract.pipeline
The extraction pipeline for calling plugins
plugins.extract.align
Aligner plugins
plugins.extract._base
Parent class for all extraction plugins
plugins.extract.detect._base
Detector parent class for extraction plugins.
plugins.extract.align._base
Aligner parent class for extraction plugins.
- finalize(batch: BatchType) Generator[ExtractMedia, None, None]
Finalize the output from Masker
This should be called as the final task of each plugin.
Pairs the detected faces back up with their original frame before yielding each frame.
- Parameters:
batch (dict) – The final
dict
from the plugin process. It must contain the keys:detected_faces
,filename
,feed_faces
,roi_masks
- Yields:
ExtractMedia
– TheDetectedFaces
list will be populated for this class with the bounding boxes, landmarks and masks for the detected faces found in the frame.
- get_batch(queue: Queue) tuple[bool, MaskerBatch]
Get items for inputting into the masker from the queue in batches
Items are returned from the
queue
in batches ofbatchsize
Items are received as
ExtractMedia
objects and converted todict
for internal processing.To ensure consistent batch sizes for masker the items are split into separate items for each
DetectedFace
object.Remember to put
'EOF'
to the out queue after processing the final batchOutputs items in the following format. All lists are of length
batchsize
:>>> {'filename': [<filenames of source frames>], >>> 'detected_faces': [[<lib.align.DetectedFace objects]]}
- Parameters:
queue (queue.Queue()) – The
queue
that the plugin will be fed from.- Returns:
exhausted, bool –
True
if queue is exhausted,False
if notbatch,
ExtractorBatch
– The batch object for the current batch
- class plugins.extract.mask._base.MaskerBatch(image: list[np.ndarray] = <factory>, detected_faces: list[DetectedFace] = <factory>, filename: list[str] = <factory>, feed: np.ndarray = array([], dtype=float64), prediction: np.ndarray = array([], dtype=float64), data: list[dict[str, T.Any]] = <factory>, roi_masks: list[np.ndarray] = <factory>, feed_faces: list[AlignedFace] = <factory>)
Bases:
ExtractorBatch
Dataclass for holding items flowing through the aligner.
Inherits from
ExtractorBatch
- Parameters:
roi_masks (list) – The region of interest masks for the batch
- detected_faces: list[DetectedFace]
- feed_faces: list[AlignedFace]
- roi_masks: list[np.ndarray]
mask.bisenet_fp module
BiSeNet Face-Parsing mask plugin
Architecture and Pre-Trained Model ported from PyTorch to Keras by TorzDF from https://github.com/zllrunning/face-parsing.PyTorch
- class plugins.extract.mask.bisenet_fp.AttentionRefinementModule(filters: int)
Bases:
object
The Attention Refinement block for BiSeNet Face Parsing
- Parameters:
filters (int) – The dimensionality of the output space (i.e. the number of output filters in the convolution).
- class plugins.extract.mask.bisenet_fp.BiSeNet(model_path: str, allow_growth: bool, exclude_gpus: list[int] | None, input_size: int, num_classes: int, cpu_mode: bool)
Bases:
KSession
BiSeNet Face-Parsing Mask from https://github.com/zllrunning/face-parsing.PyTorch
PyTorch model implemented in Keras by TorzDF
- Parameters:
model_path (str) – The path to the keras model file
allow_growth (bool) – Enable the Tensorflow GPU allow_growth configuration option. This option prevents Tensorflow from allocating all of the GPU VRAM, but can lead to higher fragmentation and slower performance
exclude_gpus (list) – A list of indices correlating to connected GPUs that Tensorflow should not use. Pass
None
to not exclude any GPUsinput_size (int) – The input size to the model
num_classes (int) – The number of segmentation classes to create
cpu_mode (bool, optional) –
True
run the model on CPU. Default:False
- class plugins.extract.mask.bisenet_fp.BiSeNetOutput(filters: int, num_classes: int, label: str = '')
Bases:
object
The BiSeNet Output block for Face Parsing
- Parameters:
filters (int) – The dimensionality of the output space (i.e. the number of output filters in the convolution).
num_class (int) – The number of classes to generate
label – The label for this output (for naming). Default: “” (i.e. empty string, or no label)
str – The label for this output (for naming). Default: “” (i.e. empty string, or no label)
optional – The label for this output (for naming). Default: “” (i.e. empty string, or no label)
- class plugins.extract.mask.bisenet_fp.ContextPath
Bases:
object
The Context Path block for BiSeNet Face Parsing.
- class plugins.extract.mask.bisenet_fp.ConvBn(filters: int, kernel_size: int = 3, strides: int = 1, padding: int = 1, activation: int = True, prefix: str = '', start_idx: int = 1)
Bases:
object
Convolutional 3D with Batch Normalization block.
- Parameters:
filters (int) – The dimensionality of the output space (i.e. the number of output filters in the convolution).
kernel_size (int, optional) – The height and width of the 2D convolution window. Default: 3
strides (int, optional) – The strides of the convolution along the height and width. Default: 1
padding (int, optional) – The amount of padding to apply prior to the first Convolutional Layer. Default: 1
activation (bool) – Whether to include ReLu Activation at the end of the block. Default:
True
prefix (str, optional) – The prefix to name the layers within the block. Default:
""
(empty string, i.e. no prefix)start_idx (int, optional) – The starting index for naming the layers within the block. See
_get_name()
for more information. Default: 1
- class plugins.extract.mask.bisenet_fp.FeatureFusionModule(filters: int)
Bases:
object
The Feature Fusion block for BiSeNet Face Parsing
- Parameters:
filters (int) – The dimensionality of the output space (i.e. the number of output filters in the convolution).
- class plugins.extract.mask.bisenet_fp.Mask(**kwargs)
Bases:
Masker
Neural network to process face image into a segmentation mask of the face
- init_model() None
Initialize the BiSeNet Face Parsing model.
- predict(feed: ndarray) ndarray
Run model to get predictions
- process_input(batch: BatchType) None
Compile the detected faces for prediction
- process_output(batch: BatchType) None
Compile found faces for output
- class plugins.extract.mask.bisenet_fp.ResNet18
Bases:
object
ResNet 18 block. Used at the start of BiSeNet Face Parsing.
recognition plugins package
recognition._base module
Base class for Face Recognition plugins
All Recognition Plugins should inherit from this class. See the override methods for which methods are required.
The plugin will receive a ExtractMedia
object.
For each source frame, the plugin must pass a dict to finalize containing:
>>> {'filename': <filename of source frame>,
>>> 'detected_faces': <list of DetectedFace objects containing bounding box points}}
To get a DetectedFace
object use the function:
>>> face = self.to_detected_face(<face left>, <face top>, <face right>, <face bottom>)
- class plugins.extract.recognition._base.Identity(git_model_id: int | None = None, model_filename: str | None = None, configfile: str | None = None, instance: int = 0, **kwargs)
Bases:
Extractor
Face Recognition Object
Parent class for all Recognition plugins
- Parameters:
git_model_id (int) – The second digit in the github tag that identifies this model. See https://github.com/deepfakes-models/faceswap-models for more information
model_filename (str) – The name of the model file to be loaded
configfile (str, optional) – Path to a custom configuration
ini
file. Default: Use system configfile
See also
plugins.extract.pipeline
The extraction pipeline for calling plugins
plugins.extract.detect
Detector plugins
plugins.extract._base
Parent class for all extraction plugins
plugins.extract.align._base
Aligner parent class for extraction plugins.
plugins.extract.mask._base
Masker parent class for extraction plugins.
- add_identity_filters(filters: ndarray, nfilters: ndarray, threshold: float) None
Add identity encodings to filter by identity in the recognition plugin
- Parameters:
filters (
numpy.ndarray
) – The array of filter embeddings to usenfilters (
numpy.ndarray
) – The array of nfilter embeddings to usethreshold (float) – The threshold for a positive filter match
- finalize(batch: BatchType) Generator[ExtractMedia, None, None]
Finalize the output from Masker
This should be called as the final task of each plugin.
Pairs the detected faces back up with their original frame before yielding each frame.
- Parameters:
batch (
RecogBatch
) – The final batch item from the plugin process.- Yields:
ExtractMedia
– TheDetectedFaces
list will be populated for this class with the bounding boxes, landmarks and masks for the detected faces found in the frame.
- get_batch(queue: Queue) tuple[bool, RecogBatch]
Get items for inputting into the recognition from the queue in batches
Items are returned from the
queue
in batches ofbatchsize
Items are received as
ExtractMedia
objects and converted toRecogBatch
for internal processing.To ensure consistent batch sizes for masker the items are split into separate items for each
DetectedFace
object.Remember to put
'EOF'
to the out queue after processing the final batchOutputs items in the following format. All lists are of length
batchsize
:>>> {'filename': [<filenames of source frames>], >>> 'detected_faces': [[<lib.align.DetectedFace objects]]}
- Parameters:
queue (queue.Queue()) – The
queue
that the plugin will be fed from.- Returns:
exhausted, bool –
True
if queue is exhausted,False
if notbatch,
ExtractorBatch
– The batch object for the current batch
- class plugins.extract.recognition._base.IdentityFilter(save_output: bool)
Bases:
object
Applies filters on the output of the recognition plugin
- Parameters:
save_output (bool) –
True
if the filtered faces should be kept as they are being saved.False
if they should be deleted
- add_filters(filters: ndarray, nfilters: ndarray, threshold) None
Add identity encodings to the filter and set whether each filter is enabled
- Parameters:
filters (
numpy.ndarray
) – The array of filter embeddings to usenfilters (
numpy.ndarray
) – The array of nfilter embeddings to usethreshold (float) – The threshold for a positive filter match
- output_counts()
Output the counts of filtered items
- class plugins.extract.recognition._base.RecogBatch(image: list[numpy.ndarray] = <factory>, detected_faces: list[lib.align.detected_face.DetectedFace] = <factory>, filename: list[str] = <factory>, feed: ~numpy.ndarray = array([], dtype=float64), prediction: ~numpy.ndarray = array([], dtype=float64), data: list[dict[str, typing.Any]] = <factory>, feed_faces: list[lib.align.aligned_face.AlignedFace] = <factory>)
Bases:
ExtractorBatch
Dataclass for holding items flowing through the aligner.
Inherits from
ExtractorBatch
- detected_faces: list[DetectedFace]
- feed_faces: list[AlignedFace]
recognition.vgg_face2 module
VGG_Face2 inference and sorting
- class plugins.extract.recognition.vgg_face2.Cluster(predictions: ndarray, method: Literal['single', 'centroid', 'median', 'ward'], threshold: float | None = None)
Bases:
object
Cluster the outputs from a VGG-Face 2 Model
- Parameters:
predictions (numpy.ndarray) – A stacked matrix of vgg_face2 predictions of the shape (N, D) where N is the number of observations and D are the number of dimensions. NB: The given
predictions
will be overwritten to save memory. If you still require the original values you should take a copy prior to running this methodmethod (['single','centroid','median','ward']) – The clustering method to use.
threshold (float, optional) – The threshold to start creating bins for. Set to
None
to disable binning
- class plugins.extract.recognition.vgg_face2.Recognition(*args, **kwargs)
Bases:
Identity
VGG Face feature extraction.
Extracts feature vectors from faces in order to compare similarity.
Notes
Input images should be in BGR Order
Model exported from: https://github.com/WeidiXie/Keras-VGGFace2-ResNet50 which is based on: https://www.robots.ox.ac.uk/~vgg/software/vgg_face/
Licensed under Creative Commons Attribution License. https://creativecommons.org/licenses/by-nc/4.0/
- init_model() None
Initialize VGG Face 2 Model.
- predict(feed: ndarray) ndarray
Return encodings for given image from vgg_face2.
- Parameters:
batch (numpy.ndarray) – The face to be fed through the predictor. Should be in BGR channel order
- Returns:
The encodings for the face
- Return type:
numpy.ndarray
- process_input(batch: BatchType) None
Compile the detected faces for prediction
- process_output(batch: BatchType) None
No output processing for vgg_face2