align package

The align Package handles detected faces, their alignments and masks.

aligned_face module

Handles aligned faces and corresponding pose estimates

Module Summary

AlignedFace

Class to align a face.

get_matrix_scaling

Given a matrix, return the cv2 Interpolation method and inverse interpolation method for applying the matrix on an image.

PoseEstimate

Estimates pose from a generic 3D head model for the given 2D face landmarks.

transform_image

Perform transformation on an image, applying the given size and padding to the matrix.

Module

Aligner for faceswap.py

class lib.align.aligned_face.AlignedFace(landmarks: ndarray, image: ndarray | None = None, centering: Literal['face', 'head', 'legacy'] = 'face', size: int = 64, coverage_ratio: float = 1.0, dtype: str | None = None, is_aligned: bool = False, is_legacy: bool = False)

Bases: object

Class to align a face.

Holds the aligned landmarks and face image, as well as associated matrices and information about an aligned face.

Parameters:
  • landmarks (numpy.ndarray) – The original 68 point landmarks that pertain to the given image for this face

  • image (numpy.ndarray, optional) – The original frame that contains the face that is to be aligned. Pass None if the aligned face is not to be generated, and just the co-ordinates should be calculated.

  • centering (["legacy", "face", "head"], optional) – The type of extracted face that should be loaded. “legacy” places the nose in the center of the image (the original method for aligning). “face” aligns for the nose to be in the center of the face (top to bottom) but the center of the skull for left to right. “head” aligns for the center of the skull (in 3D space) being the center of the extracted image, with the crop holding the full head. Default: “face”

  • size (int, optional) – The size in pixels, of each edge of the final aligned face. Default: 64

  • coverage_ratio (float, optional) – The amount of the aligned image to return. A ratio of 1.0 will return the full contents of the aligned image. A ratio of 0.5 will return an image of the given size, but will crop to the central 50%% of the image.

  • dtype (str, optional) – Set a data type for the final face to be returned as. Passing None will return a face with the same data type as the original image. Default: None

  • is_aligned_face (bool, optional) – Indicates that the image is an aligned face rather than a frame. Default: False

  • is_legacy (bool, optional) – Only used if is_aligned is True. True indicates that the aligned image being loaded is a legacy extracted face rather than a current head extracted face

property adjusted_matrix: ndarray

The 3x2 transformation matrix for extracting and aligning the core face area out of the original frame with padding and sizing applied.

Type:

numpy.ndarray

property average_distance: float

The average distance of the core landmarks (18-67) from the mean face that was used for aligning the image.

Type:

float

property centering: Literal['legacy', 'head', 'face']

The centering of the Aligned Face. One of “legacy”, “head”, “face”.

Type:

str

extract_face(image: ndarray | None) ndarray | None

Extract the face from a source image and populate face. If an image is not provided then None is returned.

Parameters:

image (numpy.ndarray or None) – The original frame to extract the face from. None if the face should not be extracted

Returns:

The extracted face at the given size, with the given coverage of the given dtype or None if no image has been provided.

Return type:

numpy.ndarray or None

property face: ndarray | None

The aligned face at the given size at the specified coverage in the given dtype. If an image has not been provided then an the attribute will return None.

Type:

numpy.ndarray

get_cropped_roi(image_size: int, target_size: int, centering: Literal['face', 'head', 'legacy']) ndarray

Obtain the region of interest within an aligned face set to centered coverage for an alternative centering

Parameters:
  • image_size (int) – The size of the full head extracted image loaded from disk

  • target_size (int) – The size of the target centered face with coverage ratio applied in relation to the original image size

  • centering (["legacy", "face"]) – The type of centering to obtain the region of interest for. “legacy” places the nose in the center of the image (the original method for aligning). “face” aligns for the nose to be in the center of the face (top to bottom) but the center of the skull for left to right.

Returns:

The (left, top, right, bottom location of the region of interest within an aligned face centered on the head for the given centering

Return type:

numpy.ndarray

property interpolators: tuple[int, int]

(interpolator and reverse interpolator) for the adjusted matrix.

Type:

tuple

property landmarks: ndarray

The 68 point facial landmarks aligned to the extracted face box.

Type:

numpy.ndarray

property matrix: ndarray

The 3x2 transformation matrix for extracting and aligning the core face area out of the original frame, with no padding or sizing applied. The returned matrix is offset for the given centering.

Type:

numpy.ndarray

property normalized_landmarks: ndarray

The 68 point facial landmarks normalized to 0.0 - 1.0 as aligned by Umeyama.

Type:

numpy.ndarray

property original_roi: ndarray

The location of the extracted face box within the original frame.

Type:

numpy.ndarray

property padding: int

The amount of padding (in pixels) that is applied to each side of the extracted face image for the selected extract type.

Type:

int

property pose: PoseEstimate

The estimated pose in 3D space.

Type:

lib.align.PoseEstimate

property relative_eye_mouth_position: float

Value representing the relative position of the lowest eye/eye-brow point to the highest mouth point. Positive values indicate that eyes/eyebrows are aligned above the mouth, negative values indicate that eyes/eyebrows are misaligned below the mouth.

Type:

float

property size: int

The size (in pixels) of one side of the square extracted face image.

Type:

int

transform_points(points: ndarray, invert: bool = False) ndarray

Perform transformation on a series of (x, y) co-ordinates in world space into aligned face space.

Parameters:
  • points (numpy.ndarray) – The points to transform

  • invert (bool, optional) – True to reverse the transformation (i.e. transform the points into world space from aligned face space). Default: False

Returns:

The transformed points

Return type:

numpy.ndarray

class lib.align.aligned_face.PoseEstimate(landmarks: ndarray)

Bases: object

Estimates pose from a generic 3D head model for the given 2D face landmarks.

Parameters:

landmarks (numpy.ndarry) – The original 68 point landmarks aligned to 0.0 - 1.0 range

References

Head Pose Estimation using OpenCV and Dlib - https://www.learnopencv.com/tag/solvepnp/ 3D Model points - http://aifi.isr.uc.pt/Downloads/OpenGL/glAnthropometric3DModel.cpp

property offset: dict[Literal['face', 'head', 'legacy'], numpy.ndarray]

The amount to offset a standard 0.0 - 1.0 umeyama transformation matrix for a from the center of the face (between the eyes) or center of the head (middle of skull) rather than the nose area.

Type:

dict

property pitch: float

The pitch of the aligned face in eular angles

Type:

float

property roll: float

The roll of the aligned face in eular angles

Type:

float

property xyz_2d: ndarray

numpy.ndarray projected (x, y) coordinates for each x, y, z point at a constant distance from adjusted center of the skull (0.5, 0.5) in the 2D space.

property yaw: float

The yaw of the aligned face in eular angles

Type:

float

lib.align.aligned_face.get_adjusted_center(image_size: int, source_offset: ndarray, target_offset: ndarray, source_centering: Literal['face', 'head', 'legacy']) ndarray

Obtain the correct center of a face extracted image to translate between two different extract centerings.

Parameters:
  • image_size (int) – The size of the image at the given source_centering

  • source_offset (numpy.ndarray) – The pose offset to translate a base extracted face to source centering

  • target_offset (numpy.ndarray) – The pose offset to translate a base extracted face to target centering

  • source_centering (["face", "head", "legacy"]) – The centering of the source image

Returns:

The center point of the image at the given size for the target centering

Return type:

numpy.ndarray

lib.align.aligned_face.get_centered_size(source_centering: Literal['face', 'head', 'legacy'], target_centering: Literal['face', 'head', 'legacy'], size: int, coverage_ratio: float = 1.0) int

Obtain the size of a cropped face from an aligned image.

Given an image of a certain dimensions, returns the dimensions of the sub-crop within that image for the requested centering at the requested coverage ratio

Notes

“legacy” places the nose in the center of the image (the original method for aligning). “face” aligns for the nose to be in the center of the face (top to bottom) but the center of the skull for left to right. “head” places the center in the middle of the skull in 3D space.

The ROI in relation to the source image is calculated by rounding the padding of one side to the nearest integer then applying this padding to the center of the crop, to ensure that any dimensions always have an even number of pixels.

Parameters:
  • source_centering (["head", "face", "legacy"]) – The centering that the original image is aligned at

  • target_centering (["head", "face", "legacy"]) – The centering that the sub-crop size should be obtained for

  • size (int) – The size of the source image to obtain the cropped size for

  • coverage_ratio (float, optional) – The coverage ratio to be applied to the target image. Default: 1.0

Returns:

The pixel size of a sub-crop image from a full head aligned image with the given coverage ratio

Return type:

int

lib.align.aligned_face.get_matrix_scaling(matrix: ndarray) tuple[int, int]

Given a matrix, return the cv2 Interpolation method and inverse interpolation method for applying the matrix on an image.

Parameters:

matrix (numpy.ndarray) – The transform matrix to return the interpolator for

Returns:

The interpolator and inverse interpolator for the given matrix. This will be (Cubic, Area) for an upscale matrix and (Area, Cubic) for a downscale matrix

Return type:

tuple

lib.align.aligned_face.transform_image(image: ndarray, matrix: ndarray, size: int, padding: int = 0) ndarray

Perform transformation on an image, applying the given size and padding to the matrix.

Parameters:
  • image (numpy.ndarray) – The image to transform

  • matrix (numpy.ndarray) – The transformation matrix to apply to the image

  • size (int) – The final size of the transformed image

  • padding (int, optional) – The amount of padding to apply to the final image. Default: 0

Returns:

The transformed image

Return type:

numpy.ndarray

alignments module

Handles alignments stored in a serialized alignments.fsa file

Module Summary

Alignments

The alignments file is a custom serialized .fsa file that holds information for each frame for a video or series of images.

Thumbnails

Thumbnail images stored in the alignments file.

Module

Alignments file functions for reading, writing and manipulating the data stored in a serialized alignments file.

class lib.align.alignments.AlignmentDict

Bases: TypedDict

Dictionary for holding all of the alignment information within a single alignment file

faces: list[lib.align.alignments.AlignmentFileDict]
video_meta: dict[str, float | int]
class lib.align.alignments.AlignmentFileDict

Bases: dict

Typed Dictionary for storing a single faces’ Alignment Information in alignments files.

h: int
identity: dict[str, list[float]]
landmarks_xy: list[float] | ndarray
mask: dict[str, lib.align.alignments.MaskAlignmentsFileDict]
thumb: ndarray | None
w: int
x: int
y: int
class lib.align.alignments.Alignments(folder: str, filename: str = 'alignments')

Bases: object

The alignments file is a custom serialized .fsa file that holds information for each frame for a video or series of images.

Specifically, it holds a list of faces that appear in each frame. Each face contains information detailing their detected bounding box location within the frame, the 68 point facial landmarks and any masks that have been extracted.

Additionally it can also hold video meta information (timestamp and whether a frame is a key frame.)

Parameters:
  • folder (str) – The folder that contains the alignments .fsa file

  • filename (str, optional) – The filename of the .fsa alignments file. If not provided then the given folder will be checked for a default alignments file filename. Default: “alignments”

add_face(frame_name: str, face: AlignmentFileDict) int

Add a new face for the given frame_name in data and return it’s index.

Parameters:
  • frame_name (str) – The frame name to add the face to. This should be the base name of the frame, not the full path

  • face (dict) – The face information to add to the given frame_name, correctly formatted for storing in data

Returns:

The index of the newly added face within data for the given frame_name

Return type:

int

backup() None

Create a backup copy of the alignments file.

Creates a copy of the serialized alignments file appending a timestamp onto the end of the file name and storing in the same folder as the original file.

count_faces_in_frame(frame_name: str) int

Return number of faces that appear within data for the given frame_name.

Parameters:

frame_name (str) – The frame name to return the count for. This should be the base name of the frame, not the full path

Returns:

The number of faces that appear in the given frame_name

Return type:

int

property data: dict[str, lib.align.alignments.AlignmentDict]

The loaded alignments file in dictionary form.

Type:

dict

delete_face_at_index(frame_name: str, face_index: int) bool

Delete the face for the given frame_name at the given face index from data.

Parameters:
  • frame_name (str) – The frame name to remove the face from. This should be the base name of the frame, not the full path

  • face_index (int) – The index number of the face within the given frame_name to remove

Returns:

True if a face was successfully deleted otherwise False

Return type:

bool

property faces_count: int

The total number of faces that appear in the alignments data.

Type:

int

property file: str

The full path to the currently loaded alignments file.

Type:

str

filter_faces(filter_dict: dict[str, list[int]], filter_out: bool = False) None

Remove faces from data based on a given filter list.

Parameters:
  • filter_dict (dict) – Dictionary of source filenames as key with a list of face indices to filter as value.

  • filter_out (bool, optional) – True if faces should be removed from data when there is a corresponding match in the given filter_dict. False if faces should be kept in data when there is a corresponding match in the given filter_dict, but removed if there is no match. Default: False

frame_exists(frame_name: str) bool

Check whether a given frame_name exists within the alignments data.

Parameters:

frame_name (str) – The frame name to check. This should be the base name of the frame, not the full path

Returns:

True if the given frame_name exists within the alignments data otherwise False

Return type:

bool

frame_has_faces(frame_name: str) bool

Check whether a given frame_name exists within the alignments data and contains at least 1 face.

Parameters:

frame_name (str) – The frame name to check. This should be the base name of the frame, not the full path

Returns:

True if the given frame_name exists within the alignments data and has at least 1 face associated with it, otherwise False

Return type:

bool

frame_has_multiple_faces(frame_name: str) bool

Check whether a given frame_name exists within the alignments data and contains more than 1 face.

Parameters:

frame_name (str) – The frame_name name to check. This should be the base name of the frame, not the full path

Returns:

True if the given frame_name exists within the alignments data and has more than 1 face associated with it, otherwise False

Return type:

bool

property frames_count: int

The number of frames that appear in the alignments data.

Type:

int

get_faces_in_frame(frame_name: str) list[lib.align.alignments.AlignmentFileDict]

Obtain the faces from data associated with a given frame_name.

Parameters:

frame_name (str) – The frame name to return faces for. This should be the base name of the frame, not the full path

Returns:

The list of face dictionaries that appear within the requested frame_name

Return type:

list

property hashes_to_alignment: dict[str, lib.align.alignments.AlignmentFileDict]

The SHA1 hash of the face mapped to the alignment for the face that the hash corresponds to. The structure of the dictionary is:

Notes

This method is depractated and exists purely for updating legacy hash based alignments to new png header storage in lib.align.update_legacy_png_header.

Type:

dict

property hashes_to_frame: dict[str, dict[str, int]]

The SHA1 hash of the face mapped to the frame(s) and face index within the frame that the hash corresponds to.

Notes

This method is depractated and exists purely for updating legacy hash based alignments to new png header storage in lib.align.update_legacy_png_header.

Type:

dict

property have_alignments_file: bool

True if an alignments file exists at location file otherwise False.

Type:

bool

mask_is_valid(mask_type: str) bool

Ensure the given mask_type is valid for the alignments data.

Every face in the alignments data must have the given mask type to successfully pass the test.

Parameters:

mask_type (str) – The mask type to check against the current alignments data

Returns:

True if all faces in the current alignments possess the given mask_type otherwise False

Return type:

bool

property mask_summary: dict[str, int]

The mask type names stored in the alignments data as key with the number of faces which possess the mask type as value.

Type:

dict

save() None

Write the contents of data and _meta to a serialized .fsa file at the location file.

save_video_meta_data(pts_time: list[float], keyframes: list[int]) None

Save video meta data to the alignments file.

If the alignments file does not have an entry for every frame (e.g. if Extract Every N was used) then the frame is added to the alignments file with no faces, so that they video meta data can be stored.

Parameters:
  • pts_time (list) – A list of presentation timestamps (float) in frame index order for every frame in the input video

  • keyframes (list) – A list of frame indices corresponding to the key frames in the input video

property thumbnails: Thumbnails

The low resolution thumbnail images that exist within the alignments file

Type:

Thumbnails

update_face(frame_name: str, face_index: int, face: AlignmentFileDict) None

Update the face for the given frame_name at the given face index in data.

Parameters:
  • frame_name (str) – The frame name to update the face for. This should be the base name of the frame, not the full path

  • face_index (int) – The index number of the face within the given frame_name to update

  • face (dict) – The face information to update to the given frame_name at the given face_index, correctly formatted for storing in data

update_from_dict(data: dict[str, lib.align.alignments.AlignmentDict]) None

Replace all alignments with the contents of the given dictionary

Parameters:

data (dict[str, AlignmentDict]) – The alignments, in correctly formatted dictionary form, to be populated into this Alignments

property version: float

The alignments file version number.

Type:

float

property video_meta_data: dict[str, list[int] | list[float] | None]

The frame meta data stored in the alignments file. If data does not exist in the alignments file then None is returned for each Key

Type:

dict

yield_faces() Generator[tuple[str, list[AlignmentFileDict], int, str], None, None]

Generator to obtain all faces with meta information from data. The results are yielded by frame.

Notes

The yielded order is non-deterministic.

Yields:
  • frame_name (str) – The frame name that the face belongs to. This is the base name of the frame, as it appears in data, not the full path

  • faces (list) – The list of face dict objects that exist for this frame

  • face_count (int) – The number of faces that exist within data for this frame

  • frame_fullname (str) – The full path (folder and filename) for the yielded frame

class lib.align.alignments.MaskAlignmentsFileDict

Bases: TypedDict

Typed Dictionary for storing Masks.

affine_matrix: list[float] | np.ndarray
interpolator: int
mask: bytes
stored_centering: CenteringType
stored_size: int
class lib.align.alignments.PNGHeaderAlignmentsDict

Bases: TypedDict

Base Dictionary for storing a single faces’ Alignment Information in Alignments files and PNG Headers.

h: int
identity: dict[str, list[float]]
landmarks_xy: list[float] | ndarray
mask: dict[str, lib.align.alignments.MaskAlignmentsFileDict]
w: int
x: int
y: int
class lib.align.alignments.PNGHeaderDict

Bases: TypedDict

Dictionary for storing all alignment and meta information in PNG Headers

alignments: PNGHeaderAlignmentsDict
source: PNGHeaderSourceDict
class lib.align.alignments.PNGHeaderSourceDict

Bases: TypedDict

Dictionary for storing additional meta information in PNG headers

alignments_version: float
face_index: int
original_filename: str
source_filename: str
source_frame_dims: tuple[int, int] | None
source_is_video: bool
class lib.align.alignments.Thumbnails(alignments: Alignments)

Bases: object

Thumbnail images stored in the alignments file.

The thumbnails are stored as low resolution (64px), low quality jpg in the alignments file and are used for the Manual Alignments tool.

Parameters:

alignments (:class:'~lib.align.Alignments`) – The parent alignments class that these thumbs belong to

add_thumbnail(frame: str, face_index: int, thumb: ndarray) None

Add a thumbnail for the given face index for the given frame.

Parameters:
  • frame (str) – The name of the frame to add the thumbnail for

  • face_index (int) – The face index within the given frame to add the thumbnail for

  • thumb (numpy.ndarray) – The encoded jpg thumbnail at 64px to add to the alignments file

get_thumbnail_by_index(frame_index: int, face_index: int) ndarray

Obtain a jpg thumbnail from the given frame index for the given face index

Parameters:
  • frame_index (int) – The frame index that contains the thumbnail

  • face_index (int) – The face index within the frame to retrieve the thumbnail for

Returns:

The encoded jpg thumbnail

Return type:

numpy.ndarray

property has_thumbnails: bool

True if all faces in the alignments file contain thumbnail images otherwise False.

Type:

bool

detected_face module

Handles detected face objects and their associated masks.

Module Summary

BlurMask

Factory class to return the correct blur object for requested blur type.

DetectedFace

Detected face and landmark information

Mask

Face Mask information and convenience methods

update_legacy_png_header

Update a legacy extracted face from pre v2.1 alignments by placing the alignment data for the face in the png exif header for the given filename with the given alignment data.

Module

Face and landmarks detection for faceswap.py

class lib.align.detected_face.BlurMask(blur_type: Literal['gaussian', 'normalized'], mask: ndarray, kernel: int | float, is_ratio: bool = False, passes: int = 1)

Bases: object

Factory class to return the correct blur object for requested blur type.

Works for square images only. Currently supports Gaussian and Normalized Box Filters.

Parameters:
  • blur_type (["gaussian", "normalized"]) – The type of blur to use

  • mask (numpy.ndarray) – The mask to apply the blur to

  • kernel (int or float) – Either the kernel size (in pixels) or the size of the kernel as a ratio of mask size

  • is_ratio (bool, optional) – Whether the given kernel parameter is a ratio or not. If True then the actual kernel size will be calculated from the given ratio and the mask size. If False then the kernel size will be set directly from the kernel parameter. Default: False

  • passes (int, optional) – The number of passes to perform when blurring. Default: 1

Example

>>> print(mask.shape)
(128, 128, 1)
>>> new_mask = BlurMask("gaussian", mask, 3, is_ratio=False, passes=1).blurred
>>> print(new_mask.shape)
(128, 128, 1)
property blurred: ndarray

The final mask with blurring applied.

Type:

numpy.ndarray

class lib.align.detected_face.DetectedFace(image: ndarray | None = None, left: int | None = None, width: int | None = None, top: int | None = None, height: int | None = None, landmarks_xy: ndarray | None = None, mask: dict[str, 'Mask'] | None = None, filename: str | None = None)

Bases: object

Detected face and landmark information

Holds information about a detected face, it’s location in a source image and the face’s 68 point landmarks.

Methods for aligning a face are also callable from here.

Parameters:
  • image (numpy.ndarray, optional) – Original frame that holds this face. Optional (not required if just storing coordinates)

  • left (int) – The left most point (in pixels) of the face’s bounding box as discovered in plugins.extract.detect

  • width (int) – The width (in pixels) of the face’s bounding box as discovered in plugins.extract.detect

  • top (int) – The top most point (in pixels) of the face’s bounding box as discovered in plugins.extract.detect

  • height (int) – The height (in pixels) of the face’s bounding box as discovered in plugins.extract.detect

  • landmarks_xy (list) – The 68 point landmarks as discovered in plugins.extract.align. Should be a list of 68 (x, y) tuples with each of the landmark co-ordinates.

  • mask (dict) – The generated mask(s) for the face as generated in plugins.extract.mask. Must be a dict of {name (str): Mask}.

image

This is a generic image placeholder that should not be relied on to be holding a particular image. It may hold the source frame that holds the face, a cropped face or a scaled image depending on the method using this object.

Type:

numpy.ndarray, optional

left

The left most point (in pixels) of the face’s bounding box as discovered in plugins.extract.detect

Type:

int

width

The width (in pixels) of the face’s bounding box as discovered in plugins.extract.detect

Type:

int

top

The top most point (in pixels) of the face’s bounding box as discovered in plugins.extract.detect

Type:

int

height

The height (in pixels) of the face’s bounding box as discovered in plugins.extract.detect

Type:

int

landmarks_xy

The 68 point landmarks as discovered in plugins.extract.align.

Type:

list

mask

The generated mask(s) for the face as generated in plugins.extract.mask. Is a dict of {name (str): Mask}.

Type:

dict

add_identity(name: str, embedding: ndarray) None

Add an identity embedding to this detected face. If an identity already exists for the given name it will be overwritten

Parameters:
  • name (str) – The name of the mechanism that calculated the identity

  • embedding (numpy.ndarray) – The identity embedding

add_landmarks_xy(landmarks: ndarray) None

Add landmarks to the detected face object. If landmarks alread exist, they will be overwritten.

Parameters:

landmarks (numpy.ndarray) – The 68 point face landmarks to add for the face

add_mask(name: str, mask: np.ndarray, affine_matrix: np.ndarray, interpolator: int, storage_size: int = 128, storage_centering: CenteringType = 'face') None

Add a Mask to this detected face

The mask should be the original output from plugins.extract.mask If a mask with this name already exists it will be overwritten by the given mask.

Parameters:
  • name (str) – The name of the mask as defined by the plugins.extract.mask._base.name parameter.

  • mask (numpy.ndarray) – The mask that is to be added as output from plugins.extract.mask It should be in the range 0.0 - 1.0 ideally with a dtype of float32

  • affine_matrix (numpy.ndarray) – The transformation matrix required to transform the mask to the original frame.

  • interpolator – The CV2 interpolator required to transform this mask to it’s original frame.

  • int – The CV2 interpolator required to transform this mask to it’s original frame.

  • storage_size – The size the mask is to be stored at. Default: 128

  • (optional) (str) – The size the mask is to be stored at. Default: 128

  • storage_centering – The centering to store the mask at. One of “legacy”, “face”, “head”. Default: “face”

  • (optional) – The centering to store the mask at. One of “legacy”, “face”, “head”. Default: “face”

property aligned: AlignedFace

The aligned face connected to this detected face.

property bottom: int

Bottom point (in pixels) of face detection bounding box within the parent image

Type:

int

clear_all_identities() None

Remove all stored identity embeddings

from_alignment(alignment: AlignmentFileDict, image: ndarray | None = None, with_thumb: bool = False) None

Set the attributes of this class from an alignments file and optionally load the face into the image attribute.

Parameters:
  • alignment (dict) – A dictionary entry for a face from an alignments file containing the keys x, w, y, h, landmarks_xy. Optionally the key thumb will be provided. This is for use in the manual tool and contains the compressed jpg thumbnail of the face to be allocated to thumbnail. Optionally the key ``mask` will be provided, but legacy alignments will not have this key.

  • image (numpy.ndarray, optional) – If an image is passed in, then the image attribute will be set to the cropped face based on the passed in bounding box co-ordinates

  • with_thumb (bool, optional) – Whether to load the jpg thumbnail into the detected face object, if provided. Default: False

from_png_meta(alignment: PNGHeaderAlignmentsDict) None

Set the attributes of this class from alignments stored in a png exif header.

Parameters:

alignment (dict) – A dictionary entry for a face from alignments stored in a png exif header containing the keys x, w, y, h, landmarks_xy and mask

get_landmark_mask(area: Literal['eye', 'face', 'mouth'], blur_kernel: int, dilation: float) ndarray

Add a LandmarksMask to this detected face

Landmark based masks are generated from face Aligned Face landmark points. An aligned face must be loaded. As the data is coming from the already aligned face, no further mask cropping is required.

Parameters:
  • area (["face", "mouth", "eye"]) – The type of mask to obtain. face is a full face mask the others are masks for those specific areas

  • blur_kernel (int) – The size of the kernel for blurring the mask edges

  • dilation (float) – The amount of dilation to apply to the mask. as a percentage of the mask size

Returns:

The generated landmarks mask for the selected area

Return type:

numpy.ndarray

get_training_masks() ndarray | None

Obtain the decompressed combined training masks.

Returns:

A 3D array containing the decompressed training masks as uint8 in 0-255 range if training masks are present otherwise None

Return type:

numpy.ndarray

property identity: dict[str, numpy.ndarray]

Identity mechanism as key, identity embedding as value.

Type:

dict

property landmarks_xy: ndarray

The aligned face connected to this detected face.

load_aligned(image: np.ndarray | None, size: int = 256, dtype: str | None = None, centering: CenteringType = 'head', coverage_ratio: float = 1.0, force: bool = False, is_aligned: bool = False, is_legacy: bool = False) None

Align a face from a given image.

Aligning a face is a relatively expensive task and is not required for all uses of the DetectedFace object, so call this function explicitly to load an aligned face.

This method plugs into lib.align.AlignedFace to perform face alignment based on this face’s landmarks_xy. If the face has already been aligned, then this function will return having performed no action.

Parameters:
  • image (numpy.ndarray) – The image that contains the face to be aligned

  • size (int) – The size of the output face in pixels

  • dtype (str, optional) – Optionally set a dtype for the final face to be formatted in. Default: None

  • centering (["legacy", "face", "head"], optional) – The type of extracted face that should be loaded. “legacy” places the nose in the center of the image (the original method for aligning). “face” aligns for the nose to be in the center of the face (top to bottom) but the center of the skull for left to right. “head” aligns for the center of the skull (in 3D space) being the center of the extracted image, with the crop holding the full head. Default: “head”

  • coverage_ratio (float, optional) – The amount of the aligned image to return. A ratio of 1.0 will return the full contents of the aligned image. A ratio of 0.5 will return an image of the given size, but will crop to the central 50%% of the image. Default: 1.0

  • force (bool, optional) – Force an update of the aligned face, even if it is already loaded. Default: False

  • is_aligned (bool, optional) – Indicates that the image is an aligned face rather than a frame. Default: False

  • is_legacy (bool, optional) – Only used if is_aligned is True. True indicates that the aligned image being loaded is a legacy extracted face rather than a current head extracted face

Notes

This method must be executed to get access to the following an AlignedFace object

property right: int

Right point (in pixels) of face detection bounding box within the parent image

Type:

int

store_training_masks(masks: list[numpy.ndarray | None], delete_masks: bool = False) None

Concatenate and compress the given training masks and store for retrieval.

Parameters:
  • masks (list) – A list of training mask. Must be all be uint-8 3D arrays of the same size in 0-255 range

  • delete_masks (bool, optional) – True to delete any of the Mask objects owned by this detected face. Use to free up unrequired memory usage. Default: False

to_alignment() AlignmentFileDict

Return the detected face formatted for an alignments file

Returns:

alignment – The alignment dict will be returned with the keys x, w, y, h, landmarks_xy, mask. The additional key thumb will be provided if the detected face object contains a thumbnail.

Return type:

dict

to_png_meta() PNGHeaderAlignmentsDict

Return the detected face formatted for insertion into a png itxt header.

returns: dict

The alignments dict will be returned with the keys x, w, y, h, landmarks_xy and mask

class lib.align.detected_face.LandmarksMask(points: list[np.ndarray], storage_size: int = 128, storage_centering: CenteringType = 'face', dilation: float = 0.0)

Bases: Mask

Create a single channel mask from aligned landmark points.

Landmarks masks are created on the fly, so the stored centering and size should be the same as the aligned face that the mask will be applied to. As the masks are created on the fly, blur + dilation is applied to the mask at creation (prior to compression) rather than after decompression when requested.

Note

Threshold is not used for Landmarks mask as the mask is binary

Parameters:
  • points (list) – A list of landmark points that correspond to the given storage_size to create the mask. Each item in the list should be a numpy.ndarray that a filled convex polygon will be created from

  • storage_size (int, optional) – The size (in pixels) that the compressed mask should be stored at. Default: 128.

  • storage_centering – The centering to store the mask at. One of “legacy”, “face”, “head”. Default: “face”

  • (optional) (str) – The centering to store the mask at. One of “legacy”, “face”, “head”. Default: “face”

  • dilation (float, optional) – The amount of dilation to apply to the mask. as a percentage of the mask size. Default: 0.0

generate_mask(affine_matrix: ndarray, interpolator: int) None

Generate the mask.

Creates the mask applying any requested dilation and blurring and assigns compressed mask to _mask

Parameters:
  • affine_matrix (numpy.ndarray) – The transformation matrix required to transform the mask to the original frame.

  • interpolator – The CV2 interpolator required to transform this mask to it’s original frame

  • int – The CV2 interpolator required to transform this mask to it’s original frame

property mask: ndarray

Overrides the default mask property, creating the processed mask at first call and compressing it. The decompressed mask is returned from this property.

Type:

numpy.ndarray

class lib.align.detected_face.Mask(storage_size: int = 128, storage_centering: CenteringType = 'face')

Bases: object

Face Mask information and convenience methods

Holds a Faceswap mask as generated from plugins.extract.mask and the information required to transform it to its original frame.

Holds convenience methods to handle the warping, storing and retrieval of the mask.

Parameters:
  • storage_size (int, optional) – The size (in pixels) that the mask should be stored at. Default: 128.

  • storage_centering – The centering to store the mask at. One of “legacy”, “face”, “head”. Default: “face”

  • (optional) (str) – The centering to store the mask at. One of “legacy”, “face”, “head”. Default: “face”

stored_size

The size, in pixels, of the stored mask across its height and width.

Type:

int

stored_centering

The centering that the mask is stored at. One of “legacy”, “face”, “head”

Type:

str

add(mask: ndarray, affine_matrix: ndarray, interpolator: int) None

Add a Faceswap mask to this Mask.

The mask should be the original output from plugins.extract.mask

Parameters:
  • mask (numpy.ndarray) – The mask that is to be added as output from plugins.extract.mask It should be in the range 0.0 - 1.0 ideally with a dtype of float32

  • affine_matrix (numpy.ndarray) – The transformation matrix required to transform the mask to the original frame.

  • interpolator – The CV2 interpolator required to transform this mask to it’s original frame

  • int – The CV2 interpolator required to transform this mask to it’s original frame

property affine_matrix: ndarray

numpy.ndarray: The affine matrix to transpose the mask to a full frame.

Type:

class

from_dict(mask_dict: MaskAlignmentsFileDict) None

Populates the Mask from a dictionary loaded from an alignments file.

Parameters:

mask_dict (dict) – A dictionary stored in an alignments file containing the keys mask, affine_matrix, interpolator, stored_size, stored_centering

get_full_frame_mask(width: int, height: int) ndarray

Return the stored mask in a full size frame of the given dimensions

Parameters:
  • width (int) – The width of the original frame that the mask was extracted from

  • height (int) – The height of the original frame that the mask was extracted from

Returns:

:class:`numpy.ndarray`

Return type:

The mask affined to the original full frame of the given dimensions

property interpolator: int

The cv2 interpolator required to transpose the mask to a full frame.

Type:

int

property mask: ndarray

The mask at the size of stored_size with any requested blurring, threshold amount and centering applied.

Type:

numpy.ndarray

property original_roi: ndarray

numpy.ndarray: The original region of interest of the mask in the source frame.

Type:

class

replace_mask(mask: ndarray) None

Replace the existing _mask with the given mask.

Parameters:

mask (numpy.ndarray) – The mask that is to be added as output from plugins.extract.mask. It should be in the range 0.0 - 1.0 ideally with a dtype of float32

set_blur_and_threshold(blur_kernel: int = 0, blur_type: Literal['gaussian', 'normalized'] | None = 'gaussian', blur_passes: int = 1, threshold: int = 0) None

Set the internal blur kernel and threshold amount for returned masks

Parameters:
  • blur_kernel (int, optional) – The kernel size, in pixels to apply gaussian blurring to the mask. Set to 0 for no blurring. Should be odd, if an even number is passed in (outside of 0) then it is rounded up to the next odd number. Default: 0

  • blur_type (["gaussian", "normalized"], optional) – The blur type to use. gaussian or normalized box filter. Default: gaussian

  • blur_passes (int, optional) – The number of passed to perform when blurring. Default: 1

  • threshold (int, optional) – The threshold amount to minimize/maximize mask values to 0 and 100. Percentage value. Default: 0

set_dilation(amount: float) None

Set the internal dilation object for returned masks

Parameters:

amount (float) – The amount of erosion/dilation to apply as a percentage of the total mask size. Negative values erode the mask. Positive values dilate the mask

set_sub_crop(source_offset: np.ndarray, target_offset: np.ndarray, centering: CenteringType, coverage_ratio: float = 1.0) None

Set the internal crop area of the mask to be returned.

This impacts the returned mask from mask if the requested mask is required for different face centering than what has been stored.

Parameters:
  • source_offset (numpy.ndarray) – The (x, y) offset for the mask at its stored centering

  • target_offset (numpy.ndarray) – The (x, y) offset for the mask at the requested target centering

  • centering (str) – The centering to set the sub crop area for. One of “legacy”, “face”. “head”

  • coverage_ratio (float, optional) – The coverage ratio to be applied to the target image. None for default (1.0). Default: None

property stored_mask: ndarray

The mask at the size of stored_size as it is stored (i.e. with no blurring/centering applied).

Type:

numpy.ndarray

to_dict(is_png=False) MaskAlignmentsFileDict

Convert the mask to a dictionary for saving to an alignments file

Parameters:

is_png (bool) – True if the dictionary is being created for storage in a png header otherwise False. Default: False

Returns:

The Mask for saving to an alignments file. Contains the keys mask, affine_matrix, interpolator, stored_size, stored_centering

Return type:

dict

to_png_meta() MaskAlignmentsFileDict

Convert the mask to a dictionary supported by png itxt headers.

Returns:

The Mask for saving to an alignments file. Contains the keys mask, affine_matrix, interpolator, stored_size, stored_centering

Return type:

dict

lib.align.detected_face.update_legacy_png_header(filename: str, alignments: Alignments) PNGHeaderDict | None

Update a legacy extracted face from pre v2.1 alignments by placing the alignment data for the face in the png exif header for the given filename with the given alignment data.

If the given file is not a .png then a png is created and the original file is removed

Parameters:
  • filename (str) – The image file to update

  • alignments (lib.align.alignments.Alignments) – The alignments data the contains the information to store in the image header. This must be a v2.0 or less alignments file as later versions no longer store the face hash (not required)

Returns:

The metadata that has been applied to the given image

Return type:

dict