align package
The align Package handles detected faces, their alignments and masks.
aligned_face module
Handles aligned faces and corresponding pose estimates
Module Summary
Class to align a face. |
|
Given a matrix, return the cv2 Interpolation method and inverse interpolation method for applying the matrix on an image. |
|
Estimates pose from a generic 3D head model for the given 2D face landmarks. |
|
Perform transformation on an image, applying the given size and padding to the matrix. |
Module
Aligner for faceswap.py
- class lib.align.aligned_face.AlignedFace(landmarks: ndarray, image: ndarray | None = None, centering: Literal['face', 'head', 'legacy'] = 'face', size: int = 64, coverage_ratio: float = 1.0, dtype: str | None = None, is_aligned: bool = False, is_legacy: bool = False)
Bases:
object
Class to align a face.
Holds the aligned landmarks and face image, as well as associated matrices and information about an aligned face.
- Parameters:
landmarks (
numpy.ndarray
) – The original 68 point landmarks that pertain to the given image for this faceimage (
numpy.ndarray
, optional) – The original frame that contains the face that is to be aligned. Pass None if the aligned face is not to be generated, and just the co-ordinates should be calculated.centering (["legacy", "face", "head"], optional) – The type of extracted face that should be loaded. “legacy” places the nose in the center of the image (the original method for aligning). “face” aligns for the nose to be in the center of the face (top to bottom) but the center of the skull for left to right. “head” aligns for the center of the skull (in 3D space) being the center of the extracted image, with the crop holding the full head. Default: “face”
size (int, optional) – The size in pixels, of each edge of the final aligned face. Default: 64
coverage_ratio (float, optional) – The amount of the aligned image to return. A ratio of 1.0 will return the full contents of the aligned image. A ratio of 0.5 will return an image of the given size, but will crop to the central 50%% of the image.
dtype (str, optional) – Set a data type for the final face to be returned as. Passing
None
will return a face with the same data type as the originalimage
. Default:None
is_aligned_face (bool, optional) – Indicates that the
image
is an aligned face rather than a frame. Default:False
is_legacy (bool, optional) – Only used if is_aligned is
True
.True
indicates that the aligned image being loaded is a legacy extracted face rather than a current head extracted face
- property adjusted_matrix: ndarray
The 3x2 transformation matrix for extracting and aligning the core face area out of the original frame with padding and sizing applied.
- Type:
numpy.ndarray
- property average_distance: float
The average distance of the core landmarks (18-67) from the mean face that was used for aligning the image.
- Type:
float
- property centering: Literal['legacy', 'head', 'face']
The centering of the Aligned Face. One of “legacy”, “head”, “face”.
- Type:
str
- extract_face(image: ndarray | None) ndarray | None
Extract the face from a source image and populate
face
. If an image is not provided thenNone
is returned.- Parameters:
image (
numpy.ndarray
orNone
) – The original frame to extract the face from.None
if the face should not be extracted- Returns:
The extracted face at the given size, with the given coverage of the given dtype or
None
if no image has been provided.- Return type:
numpy.ndarray
orNone
- property face: ndarray | None
The aligned face at the given
size
at the specifiedcoverage
in the givendtype
. If animage
has not been provided then an the attribute will returnNone
.- Type:
numpy.ndarray
- get_cropped_roi(image_size: int, target_size: int, centering: Literal['face', 'head', 'legacy']) ndarray
Obtain the region of interest within an aligned face set to centered coverage for an alternative centering
- Parameters:
image_size (int) – The size of the full head extracted image loaded from disk
target_size (int) – The size of the target centered face with coverage ratio applied in relation to the original image size
centering (["legacy", "face"]) – The type of centering to obtain the region of interest for. “legacy” places the nose in the center of the image (the original method for aligning). “face” aligns for the nose to be in the center of the face (top to bottom) but the center of the skull for left to right.
- Returns:
The (left, top, right, bottom location of the region of interest within an aligned face centered on the head for the given centering
- Return type:
numpy.ndarray
- property interpolators: tuple[int, int]
(interpolator and reverse interpolator) for the
adjusted matrix
.- Type:
tuple
- property landmarks: ndarray
The 68 point facial landmarks aligned to the extracted face box.
- Type:
numpy.ndarray
- property matrix: ndarray
The 3x2 transformation matrix for extracting and aligning the core face area out of the original frame, with no padding or sizing applied. The returned matrix is offset for the given
centering
.- Type:
numpy.ndarray
- property normalized_landmarks: ndarray
The 68 point facial landmarks normalized to 0.0 - 1.0 as aligned by Umeyama.
- Type:
numpy.ndarray
- property original_roi: ndarray
The location of the extracted face box within the original frame.
- Type:
numpy.ndarray
- property padding: int
The amount of padding (in pixels) that is applied to each side of the extracted face image for the selected extract type.
- Type:
int
- property pose: PoseEstimate
The estimated pose in 3D space.
- Type:
lib.align.PoseEstimate
- property relative_eye_mouth_position: float
Value representing the relative position of the lowest eye/eye-brow point to the highest mouth point. Positive values indicate that eyes/eyebrows are aligned above the mouth, negative values indicate that eyes/eyebrows are misaligned below the mouth.
- Type:
float
- property size: int
The size (in pixels) of one side of the square extracted face image.
- Type:
int
- transform_points(points: ndarray, invert: bool = False) ndarray
Perform transformation on a series of (x, y) co-ordinates in world space into aligned face space.
- Parameters:
points (
numpy.ndarray
) – The points to transforminvert (bool, optional) –
True
to reverse the transformation (i.e. transform the points into world space from aligned face space). Default:False
- Returns:
The transformed points
- Return type:
numpy.ndarray
- class lib.align.aligned_face.PoseEstimate(landmarks: ndarray)
Bases:
object
Estimates pose from a generic 3D head model for the given 2D face landmarks.
- Parameters:
landmarks (
numpy.ndarry
) – The original 68 point landmarks aligned to 0.0 - 1.0 range
References
Head Pose Estimation using OpenCV and Dlib - https://www.learnopencv.com/tag/solvepnp/ 3D Model points - http://aifi.isr.uc.pt/Downloads/OpenGL/glAnthropometric3DModel.cpp
- property offset: dict[Literal['face', 'head', 'legacy'], numpy.ndarray]
The amount to offset a standard 0.0 - 1.0 umeyama transformation matrix for a from the center of the face (between the eyes) or center of the head (middle of skull) rather than the nose area.
- Type:
dict
- property pitch: float
The pitch of the aligned face in eular angles
- Type:
float
- property roll: float
The roll of the aligned face in eular angles
- Type:
float
- property xyz_2d: ndarray
numpy.ndarray
projected (x, y) coordinates for each x, y, z point at a constant distance from adjusted center of the skull (0.5, 0.5) in the 2D space.
- property yaw: float
The yaw of the aligned face in eular angles
- Type:
float
- lib.align.aligned_face.get_adjusted_center(image_size: int, source_offset: ndarray, target_offset: ndarray, source_centering: Literal['face', 'head', 'legacy']) ndarray
Obtain the correct center of a face extracted image to translate between two different extract centerings.
- Parameters:
image_size (int) – The size of the image at the given
source_centering
source_offset (
numpy.ndarray
) – The pose offset to translate a base extracted face to source centeringtarget_offset (
numpy.ndarray
) – The pose offset to translate a base extracted face to target centeringsource_centering (["face", "head", "legacy"]) – The centering of the source image
- Returns:
The center point of the image at the given size for the target centering
- Return type:
numpy.ndarray
- lib.align.aligned_face.get_centered_size(source_centering: Literal['face', 'head', 'legacy'], target_centering: Literal['face', 'head', 'legacy'], size: int, coverage_ratio: float = 1.0) int
Obtain the size of a cropped face from an aligned image.
Given an image of a certain dimensions, returns the dimensions of the sub-crop within that image for the requested centering at the requested coverage ratio
Notes
“legacy” places the nose in the center of the image (the original method for aligning). “face” aligns for the nose to be in the center of the face (top to bottom) but the center of the skull for left to right. “head” places the center in the middle of the skull in 3D space.
The ROI in relation to the source image is calculated by rounding the padding of one side to the nearest integer then applying this padding to the center of the crop, to ensure that any dimensions always have an even number of pixels.
- Parameters:
source_centering (["head", "face", "legacy"]) – The centering that the original image is aligned at
target_centering (["head", "face", "legacy"]) – The centering that the sub-crop size should be obtained for
size (int) – The size of the source image to obtain the cropped size for
coverage_ratio (float, optional) – The coverage ratio to be applied to the target image. Default: 1.0
- Returns:
The pixel size of a sub-crop image from a full head aligned image with the given coverage ratio
- Return type:
int
- lib.align.aligned_face.get_matrix_scaling(matrix: ndarray) tuple[int, int]
Given a matrix, return the cv2 Interpolation method and inverse interpolation method for applying the matrix on an image.
- Parameters:
matrix (
numpy.ndarray
) – The transform matrix to return the interpolator for- Returns:
The interpolator and inverse interpolator for the given matrix. This will be (Cubic, Area) for an upscale matrix and (Area, Cubic) for a downscale matrix
- Return type:
tuple
- lib.align.aligned_face.transform_image(image: ndarray, matrix: ndarray, size: int, padding: int = 0) ndarray
Perform transformation on an image, applying the given size and padding to the matrix.
- Parameters:
image (
numpy.ndarray
) – The image to transformmatrix (
numpy.ndarray
) – The transformation matrix to apply to the imagesize (int) – The final size of the transformed image
padding (int, optional) – The amount of padding to apply to the final image. Default: 0
- Returns:
The transformed image
- Return type:
numpy.ndarray
alignments module
Handles alignments stored in a serialized alignments.fsa file
Module Summary
The alignments file is a custom serialized |
|
Thumbnail images stored in the alignments file. |
Module
Alignments file functions for reading, writing and manipulating the data stored in a serialized alignments file.
- class lib.align.alignments.AlignmentDict
Bases:
TypedDict
Dictionary for holding all of the alignment information within a single alignment file
- faces: list[lib.align.alignments.AlignmentFileDict]
- video_meta: dict[str, float | int]
- class lib.align.alignments.AlignmentFileDict
Bases:
dict
Typed Dictionary for storing a single faces’ Alignment Information in alignments files.
- h: int
- identity: dict[str, list[float]]
- landmarks_xy: list[float] | ndarray
- mask: dict[str, lib.align.alignments.MaskAlignmentsFileDict]
- thumb: ndarray | None
- w: int
- x: int
- y: int
- class lib.align.alignments.Alignments(folder: str, filename: str = 'alignments')
Bases:
object
The alignments file is a custom serialized
.fsa
file that holds information for each frame for a video or series of images.Specifically, it holds a list of faces that appear in each frame. Each face contains information detailing their detected bounding box location within the frame, the 68 point facial landmarks and any masks that have been extracted.
Additionally it can also hold video meta information (timestamp and whether a frame is a key frame.)
- Parameters:
folder (str) – The folder that contains the alignments
.fsa
filefilename (str, optional) – The filename of the
.fsa
alignments file. If not provided then the given folder will be checked for a default alignments file filename. Default: “alignments”
- add_face(frame_name: str, face: AlignmentFileDict) int
Add a new face for the given frame_name in
data
and return it’s index.- Parameters:
frame_name (str) – The frame name to add the face to. This should be the base name of the frame, not the full path
face (dict) – The face information to add to the given frame_name, correctly formatted for storing in
data
- Returns:
The index of the newly added face within
data
for the given frame_name- Return type:
int
- backup() None
Create a backup copy of the alignments
file
.Creates a copy of the serialized alignments
file
appending a timestamp onto the end of the file name and storing in the same folder as the originalfile
.
- count_faces_in_frame(frame_name: str) int
Return number of faces that appear within
data
for the given frame_name.- Parameters:
frame_name (str) – The frame name to return the count for. This should be the base name of the frame, not the full path
- Returns:
The number of faces that appear in the given frame_name
- Return type:
int
- property data: dict[str, lib.align.alignments.AlignmentDict]
The loaded alignments
file
in dictionary form.- Type:
dict
- delete_face_at_index(frame_name: str, face_index: int) bool
Delete the face for the given frame_name at the given face index from
data
.- Parameters:
frame_name (str) – The frame name to remove the face from. This should be the base name of the frame, not the full path
face_index (int) – The index number of the face within the given frame_name to remove
- Returns:
True
if a face was successfully deleted otherwiseFalse
- Return type:
bool
- property file: str
The full path to the currently loaded alignments file.
- Type:
str
- filter_faces(filter_dict: dict[str, list[int]], filter_out: bool = False) None
Remove faces from
data
based on a given filter list.- Parameters:
filter_dict (dict) – Dictionary of source filenames as key with a list of face indices to filter as value.
filter_out (bool, optional) –
True
if faces should be removed fromdata
when there is a corresponding match in the given filter_dict.False
if faces should be kept indata
when there is a corresponding match in the given filter_dict, but removed if there is no match. Default:False
- frame_exists(frame_name: str) bool
Check whether a given frame_name exists within the alignments
data
.- Parameters:
frame_name (str) – The frame name to check. This should be the base name of the frame, not the full path
- Returns:
True
if the given frame_name exists within the alignmentsdata
otherwiseFalse
- Return type:
bool
- frame_has_faces(frame_name: str) bool
Check whether a given frame_name exists within the alignments
data
and contains at least 1 face.- Parameters:
frame_name (str) – The frame name to check. This should be the base name of the frame, not the full path
- Returns:
True
if the given frame_name exists within the alignmentsdata
and has at least 1 face associated with it, otherwiseFalse
- Return type:
bool
- frame_has_multiple_faces(frame_name: str) bool
Check whether a given frame_name exists within the alignments
data
and contains more than 1 face.- Parameters:
frame_name (str) – The frame_name name to check. This should be the base name of the frame, not the full path
- Returns:
True
if the given frame_name exists within the alignmentsdata
and has more than 1 face associated with it, otherwiseFalse
- Return type:
bool
- get_faces_in_frame(frame_name: str) list[lib.align.alignments.AlignmentFileDict]
Obtain the faces from
data
associated with a given frame_name.- Parameters:
frame_name (str) – The frame name to return faces for. This should be the base name of the frame, not the full path
- Returns:
The list of face dictionaries that appear within the requested frame_name
- Return type:
list
- property hashes_to_alignment: dict[str, lib.align.alignments.AlignmentFileDict]
The SHA1 hash of the face mapped to the alignment for the face that the hash corresponds to. The structure of the dictionary is:
Notes
This method is depractated and exists purely for updating legacy hash based alignments to new png header storage in
lib.align.update_legacy_png_header
.- Type:
dict
- property hashes_to_frame: dict[str, dict[str, int]]
The SHA1 hash of the face mapped to the frame(s) and face index within the frame that the hash corresponds to.
Notes
This method is depractated and exists purely for updating legacy hash based alignments to new png header storage in
lib.align.update_legacy_png_header
.- Type:
dict
- property have_alignments_file: bool
True
if an alignments file exists at locationfile
otherwiseFalse
.- Type:
bool
- mask_is_valid(mask_type: str) bool
Ensure the given
mask_type
is valid for the alignmentsdata
.Every face in the alignments
data
must have the given mask type to successfully pass the test.- Parameters:
mask_type (str) – The mask type to check against the current alignments
data
- Returns:
True
if all faces in the current alignments possess the givenmask_type
otherwiseFalse
- Return type:
bool
- property mask_summary: dict[str, int]
The mask type names stored in the alignments
data
as key with the number of faces which possess the mask type as value.- Type:
dict
- save_video_meta_data(pts_time: list[float], keyframes: list[int]) None
Save video meta data to the alignments file.
If the alignments file does not have an entry for every frame (e.g. if Extract Every N was used) then the frame is added to the alignments file with no faces, so that they video meta data can be stored.
- Parameters:
pts_time (list) – A list of presentation timestamps (float) in frame index order for every frame in the input video
keyframes (list) – A list of frame indices corresponding to the key frames in the input video
- property thumbnails: Thumbnails
The low resolution thumbnail images that exist within the alignments file
- Type:
Thumbnails
- update_face(frame_name: str, face_index: int, face: AlignmentFileDict) None
Update the face for the given frame_name at the given face index in
data
.- Parameters:
frame_name (str) – The frame name to update the face for. This should be the base name of the frame, not the full path
face_index (int) – The index number of the face within the given frame_name to update
face (dict) – The face information to update to the given frame_name at the given face_index, correctly formatted for storing in
data
- update_from_dict(data: dict[str, lib.align.alignments.AlignmentDict]) None
Replace all alignments with the contents of the given dictionary
- Parameters:
data (dict[str, AlignmentDict]) – The alignments, in correctly formatted dictionary form, to be populated into this
Alignments
- property version: float
The alignments file version number.
- Type:
float
- property video_meta_data: dict[str, list[int] | list[float] | None]
The frame meta data stored in the alignments file. If data does not exist in the alignments file then
None
is returned for each Key- Type:
dict
- yield_faces() Generator[tuple[str, list[AlignmentFileDict], int, str], None, None]
Generator to obtain all faces with meta information from
data
. The results are yielded by frame.Notes
The yielded order is non-deterministic.
- Yields:
frame_name (str) – The frame name that the face belongs to. This is the base name of the frame, as it appears in
data
, not the full pathfaces (list) – The list of face dict objects that exist for this frame
face_count (int) – The number of faces that exist within
data
for this frameframe_fullname (str) – The full path (folder and filename) for the yielded frame
- class lib.align.alignments.MaskAlignmentsFileDict
Bases:
TypedDict
Typed Dictionary for storing Masks.
- affine_matrix: list[float] | np.ndarray
- interpolator: int
- mask: bytes
- stored_centering: CenteringType
- stored_size: int
- class lib.align.alignments.PNGHeaderAlignmentsDict
Bases:
TypedDict
Base Dictionary for storing a single faces’ Alignment Information in Alignments files and PNG Headers.
- h: int
- identity: dict[str, list[float]]
- landmarks_xy: list[float] | ndarray
- mask: dict[str, lib.align.alignments.MaskAlignmentsFileDict]
- w: int
- x: int
- y: int
- class lib.align.alignments.PNGHeaderDict
Bases:
TypedDict
Dictionary for storing all alignment and meta information in PNG Headers
- alignments: PNGHeaderAlignmentsDict
- source: PNGHeaderSourceDict
- class lib.align.alignments.PNGHeaderSourceDict
Bases:
TypedDict
Dictionary for storing additional meta information in PNG headers
- alignments_version: float
- face_index: int
- original_filename: str
- source_filename: str
- source_frame_dims: tuple[int, int] | None
- source_is_video: bool
- class lib.align.alignments.Thumbnails(alignments: Alignments)
Bases:
object
Thumbnail images stored in the alignments file.
The thumbnails are stored as low resolution (64px), low quality jpg in the alignments file and are used for the Manual Alignments tool.
- Parameters:
alignments (:class:'~lib.align.Alignments`) – The parent alignments class that these thumbs belong to
- add_thumbnail(frame: str, face_index: int, thumb: ndarray) None
Add a thumbnail for the given face index for the given frame.
- Parameters:
frame (str) – The name of the frame to add the thumbnail for
face_index (int) – The face index within the given frame to add the thumbnail for
thumb (
numpy.ndarray
) – The encoded jpg thumbnail at 64px to add to the alignments file
- get_thumbnail_by_index(frame_index: int, face_index: int) ndarray
Obtain a jpg thumbnail from the given frame index for the given face index
- Parameters:
frame_index (int) – The frame index that contains the thumbnail
face_index (int) – The face index within the frame to retrieve the thumbnail for
- Returns:
The encoded jpg thumbnail
- Return type:
numpy.ndarray
- property has_thumbnails: bool
True
if all faces in the alignments file contain thumbnail images otherwiseFalse
.- Type:
bool
detected_face module
Handles detected face objects and their associated masks.
Module Summary
Factory class to return the correct blur object for requested blur type. |
|
Detected face and landmark information |
|
Face Mask information and convenience methods |
|
Update a legacy extracted face from pre v2.1 alignments by placing the alignment data for the face in the png exif header for the given filename with the given alignment data. |
Module
Face and landmarks detection for faceswap.py
- class lib.align.detected_face.BlurMask(blur_type: Literal['gaussian', 'normalized'], mask: ndarray, kernel: int | float, is_ratio: bool = False, passes: int = 1)
Bases:
object
Factory class to return the correct blur object for requested blur type.
Works for square images only. Currently supports Gaussian and Normalized Box Filters.
- Parameters:
blur_type (["gaussian", "normalized"]) – The type of blur to use
mask (
numpy.ndarray
) – The mask to apply the blur tokernel (int or float) – Either the kernel size (in pixels) or the size of the kernel as a ratio of mask size
is_ratio (bool, optional) – Whether the given
kernel
parameter is a ratio or not. IfTrue
then the actual kernel size will be calculated from the given ratio and the mask size. IfFalse
then the kernel size will be set directly from thekernel
parameter. Default:False
passes (int, optional) – The number of passes to perform when blurring. Default:
1
Example
>>> print(mask.shape) (128, 128, 1) >>> new_mask = BlurMask("gaussian", mask, 3, is_ratio=False, passes=1).blurred >>> print(new_mask.shape) (128, 128, 1)
- property blurred: ndarray
The final mask with blurring applied.
- Type:
numpy.ndarray
- class lib.align.detected_face.DetectedFace(image: ndarray | None = None, left: int | None = None, width: int | None = None, top: int | None = None, height: int | None = None, landmarks_xy: ndarray | None = None, mask: dict[str, 'Mask'] | None = None, filename: str | None = None)
Bases:
object
Detected face and landmark information
Holds information about a detected face, it’s location in a source image and the face’s 68 point landmarks.
Methods for aligning a face are also callable from here.
- Parameters:
image (numpy.ndarray, optional) – Original frame that holds this face. Optional (not required if just storing coordinates)
left (int) – The left most point (in pixels) of the face’s bounding box as discovered in
plugins.extract.detect
width (int) – The width (in pixels) of the face’s bounding box as discovered in
plugins.extract.detect
top (int) – The top most point (in pixels) of the face’s bounding box as discovered in
plugins.extract.detect
height (int) – The height (in pixels) of the face’s bounding box as discovered in
plugins.extract.detect
landmarks_xy (list) – The 68 point landmarks as discovered in
plugins.extract.align
. Should be alist
of 68 (x, y)tuples
with each of the landmark co-ordinates.mask (dict) – The generated mask(s) for the face as generated in
plugins.extract.mask
. Must be a dict of {name (str):Mask
}.
- image
This is a generic image placeholder that should not be relied on to be holding a particular image. It may hold the source frame that holds the face, a cropped face or a scaled image depending on the method using this object.
- Type:
numpy.ndarray, optional
- left
The left most point (in pixels) of the face’s bounding box as discovered in
plugins.extract.detect
- Type:
int
- width
The width (in pixels) of the face’s bounding box as discovered in
plugins.extract.detect
- Type:
int
- top
The top most point (in pixels) of the face’s bounding box as discovered in
plugins.extract.detect
- Type:
int
- height
The height (in pixels) of the face’s bounding box as discovered in
plugins.extract.detect
- Type:
int
- landmarks_xy
The 68 point landmarks as discovered in
plugins.extract.align
.- Type:
list
- mask
The generated mask(s) for the face as generated in
plugins.extract.mask
. Is a dict of {name (str):Mask
}.- Type:
dict
- add_identity(name: str, embedding: ndarray) None
Add an identity embedding to this detected face. If an identity already exists for the given
name
it will be overwritten- Parameters:
name (str) – The name of the mechanism that calculated the identity
embedding (numpy.ndarray) – The identity embedding
- add_landmarks_xy(landmarks: ndarray) None
Add landmarks to the detected face object. If landmarks alread exist, they will be overwritten.
- Parameters:
landmarks (
numpy.ndarray
) – The 68 point face landmarks to add for the face
- add_mask(name: str, mask: np.ndarray, affine_matrix: np.ndarray, interpolator: int, storage_size: int = 128, storage_centering: CenteringType = 'face') None
Add a
Mask
to this detected faceThe mask should be the original output from
plugins.extract.mask
If a mask with this name already exists it will be overwritten by the given mask.- Parameters:
name (str) – The name of the mask as defined by the
plugins.extract.mask._base.name
parameter.mask (numpy.ndarray) – The mask that is to be added as output from
plugins.extract.mask
It should be in the range 0.0 - 1.0 ideally with adtype
offloat32
affine_matrix (numpy.ndarray) – The transformation matrix required to transform the mask to the original frame.
interpolator – The CV2 interpolator required to transform this mask to it’s original frame.
int – The CV2 interpolator required to transform this mask to it’s original frame.
storage_size – The size the mask is to be stored at. Default: 128
(optional) (str) – The size the mask is to be stored at. Default: 128
storage_centering – The centering to store the mask at. One of “legacy”, “face”, “head”. Default: “face”
(optional) – The centering to store the mask at. One of “legacy”, “face”, “head”. Default: “face”
- property aligned: AlignedFace
The aligned face connected to this detected face.
- property bottom: int
Bottom point (in pixels) of face detection bounding box within the parent image
- Type:
int
- clear_all_identities() None
Remove all stored identity embeddings
- from_alignment(alignment: AlignmentFileDict, image: ndarray | None = None, with_thumb: bool = False) None
Set the attributes of this class from an alignments file and optionally load the face into the
image
attribute.- Parameters:
alignment (dict) – A dictionary entry for a face from an alignments file containing the keys
x
,w
,y
,h
,landmarks_xy
. Optionally the keythumb
will be provided. This is for use in the manual tool and contains the compressed jpg thumbnail of the face to be allocated tothumbnail. Optionally the key ``mask`
will be provided, but legacy alignments will not have this key.image (numpy.ndarray, optional) – If an image is passed in, then the
image
attribute will be set to the cropped face based on the passed in bounding box co-ordinateswith_thumb (bool, optional) – Whether to load the jpg thumbnail into the detected face object, if provided. Default:
False
- from_png_meta(alignment: PNGHeaderAlignmentsDict) None
Set the attributes of this class from alignments stored in a png exif header.
- Parameters:
alignment (dict) – A dictionary entry for a face from alignments stored in a png exif header containing the keys
x
,w
,y
,h
,landmarks_xy
andmask
- get_landmark_mask(area: Literal['eye', 'face', 'mouth'], blur_kernel: int, dilation: float) ndarray
Add a
LandmarksMask
to this detected faceLandmark based masks are generated from face Aligned Face landmark points. An aligned face must be loaded. As the data is coming from the already aligned face, no further mask cropping is required.
- Parameters:
area (["face", "mouth", "eye"]) – The type of mask to obtain. face is a full face mask the others are masks for those specific areas
blur_kernel (int) – The size of the kernel for blurring the mask edges
dilation (float) – The amount of dilation to apply to the mask. as a percentage of the mask size
- Returns:
The generated landmarks mask for the selected area
- Return type:
numpy.ndarray
- get_training_masks() ndarray | None
Obtain the decompressed combined training masks.
- Returns:
A 3D array containing the decompressed training masks as uint8 in 0-255 range if training masks are present otherwise
None
- Return type:
numpy.ndarray
- property identity: dict[str, numpy.ndarray]
Identity mechanism as key, identity embedding as value.
- Type:
dict
- property landmarks_xy: ndarray
The aligned face connected to this detected face.
- load_aligned(image: np.ndarray | None, size: int = 256, dtype: str | None = None, centering: CenteringType = 'head', coverage_ratio: float = 1.0, force: bool = False, is_aligned: bool = False, is_legacy: bool = False) None
Align a face from a given image.
Aligning a face is a relatively expensive task and is not required for all uses of the
DetectedFace
object, so call this function explicitly to load an aligned face.This method plugs into
lib.align.AlignedFace
to perform face alignment based on this face’slandmarks_xy
. If the face has already been aligned, then this function will return having performed no action.- Parameters:
image (numpy.ndarray) – The image that contains the face to be aligned
size (int) – The size of the output face in pixels
dtype (str, optional) – Optionally set a
dtype
for the final face to be formatted in. Default:None
centering (["legacy", "face", "head"], optional) – The type of extracted face that should be loaded. “legacy” places the nose in the center of the image (the original method for aligning). “face” aligns for the nose to be in the center of the face (top to bottom) but the center of the skull for left to right. “head” aligns for the center of the skull (in 3D space) being the center of the extracted image, with the crop holding the full head. Default: “head”
coverage_ratio (float, optional) – The amount of the aligned image to return. A ratio of 1.0 will return the full contents of the aligned image. A ratio of 0.5 will return an image of the given size, but will crop to the central 50%% of the image. Default: 1.0
force (bool, optional) – Force an update of the aligned face, even if it is already loaded. Default:
False
is_aligned (bool, optional) – Indicates that the
image
is an aligned face rather than a frame. Default:False
is_legacy (bool, optional) – Only used if is_aligned is
True
.True
indicates that the aligned image being loaded is a legacy extracted face rather than a current head extracted face
Notes
This method must be executed to get access to the following an
AlignedFace
object
- property right: int
Right point (in pixels) of face detection bounding box within the parent image
- Type:
int
- store_training_masks(masks: list[numpy.ndarray | None], delete_masks: bool = False) None
Concatenate and compress the given training masks and store for retrieval.
- Parameters:
masks (list) – A list of training mask. Must be all be uint-8 3D arrays of the same size in 0-255 range
delete_masks (bool, optional) –
True
to delete any of theMask
objects owned by this detected face. Use to free up unrequired memory usage. Default:False
- to_alignment() AlignmentFileDict
Return the detected face formatted for an alignments file
- Returns:
alignment – The alignment dict will be returned with the keys
x
,w
,y
,h
,landmarks_xy
,mask
. The additional keythumb
will be provided if the detected face object contains a thumbnail.- Return type:
dict
- to_png_meta() PNGHeaderAlignmentsDict
Return the detected face formatted for insertion into a png itxt header.
- returns: dict
The alignments dict will be returned with the keys
x
,w
,y
,h
,landmarks_xy
andmask
- class lib.align.detected_face.LandmarksMask(points: list[np.ndarray], storage_size: int = 128, storage_centering: CenteringType = 'face', dilation: float = 0.0)
Bases:
Mask
Create a single channel mask from aligned landmark points.
Landmarks masks are created on the fly, so the stored centering and size should be the same as the aligned face that the mask will be applied to. As the masks are created on the fly, blur + dilation is applied to the mask at creation (prior to compression) rather than after decompression when requested.
Note
Threshold is not used for Landmarks mask as the mask is binary
- Parameters:
points (list) – A list of landmark points that correspond to the given storage_size to create the mask. Each item in the list should be a
numpy.ndarray
that a filled convex polygon will be created fromstorage_size (int, optional) – The size (in pixels) that the compressed mask should be stored at. Default: 128.
storage_centering – The centering to store the mask at. One of “legacy”, “face”, “head”. Default: “face”
(optional) (str) – The centering to store the mask at. One of “legacy”, “face”, “head”. Default: “face”
dilation (float, optional) – The amount of dilation to apply to the mask. as a percentage of the mask size. Default: 0.0
- generate_mask(affine_matrix: ndarray, interpolator: int) None
Generate the mask.
Creates the mask applying any requested dilation and blurring and assigns compressed mask to
_mask
- Parameters:
affine_matrix (
numpy.ndarray
) – The transformation matrix required to transform the mask to the original frame.interpolator – The CV2 interpolator required to transform this mask to it’s original frame
int – The CV2 interpolator required to transform this mask to it’s original frame
- property mask: ndarray
Overrides the default mask property, creating the processed mask at first call and compressing it. The decompressed mask is returned from this property.
- Type:
numpy.ndarray
- class lib.align.detected_face.Mask(storage_size: int = 128, storage_centering: CenteringType = 'face')
Bases:
object
Face Mask information and convenience methods
Holds a Faceswap mask as generated from
plugins.extract.mask
and the information required to transform it to its original frame.Holds convenience methods to handle the warping, storing and retrieval of the mask.
- Parameters:
storage_size (int, optional) – The size (in pixels) that the mask should be stored at. Default: 128.
storage_centering – The centering to store the mask at. One of “legacy”, “face”, “head”. Default: “face”
(optional) (str) – The centering to store the mask at. One of “legacy”, “face”, “head”. Default: “face”
- stored_size
The size, in pixels, of the stored mask across its height and width.
- Type:
int
- stored_centering
The centering that the mask is stored at. One of “legacy”, “face”, “head”
- Type:
str
- add(mask: ndarray, affine_matrix: ndarray, interpolator: int) None
Add a Faceswap mask to this
Mask
.The mask should be the original output from
plugins.extract.mask
- Parameters:
mask (
numpy.ndarray
) – The mask that is to be added as output fromplugins.extract.mask
It should be in the range 0.0 - 1.0 ideally with adtype
offloat32
affine_matrix (
numpy.ndarray
) – The transformation matrix required to transform the mask to the original frame.interpolator – The CV2 interpolator required to transform this mask to it’s original frame
int – The CV2 interpolator required to transform this mask to it’s original frame
- property affine_matrix: ndarray
numpy.ndarray: The affine matrix to transpose the mask to a full frame.
- Type:
class
- from_dict(mask_dict: MaskAlignmentsFileDict) None
Populates the
Mask
from a dictionary loaded from an alignments file.- Parameters:
mask_dict (dict) – A dictionary stored in an alignments file containing the keys
mask
,affine_matrix
,interpolator
,stored_size
,stored_centering
- get_full_frame_mask(width: int, height: int) ndarray
Return the stored mask in a full size frame of the given dimensions
- Parameters:
width (int) – The width of the original frame that the mask was extracted from
height (int) – The height of the original frame that the mask was extracted from
- Returns:
:class:`numpy.ndarray`
- Return type:
The mask affined to the original full frame of the given dimensions
- property interpolator: int
The cv2 interpolator required to transpose the mask to a full frame.
- Type:
int
- property mask: ndarray
The mask at the size of
stored_size
with any requested blurring, threshold amount and centering applied.- Type:
numpy.ndarray
- property original_roi: ndarray
numpy.ndarray: The original region of interest of the mask in the source frame.
- Type:
class
- replace_mask(mask: ndarray) None
Replace the existing
_mask
with the given mask.- Parameters:
mask (
numpy.ndarray
) – The mask that is to be added as output fromplugins.extract.mask
. It should be in the range 0.0 - 1.0 ideally with adtype
offloat32
- set_blur_and_threshold(blur_kernel: int = 0, blur_type: Literal['gaussian', 'normalized'] | None = 'gaussian', blur_passes: int = 1, threshold: int = 0) None
Set the internal blur kernel and threshold amount for returned masks
- Parameters:
blur_kernel (int, optional) – The kernel size, in pixels to apply gaussian blurring to the mask. Set to 0 for no blurring. Should be odd, if an even number is passed in (outside of 0) then it is rounded up to the next odd number. Default: 0
blur_type (["gaussian", "normalized"], optional) – The blur type to use.
gaussian
ornormalized
box filter. Default:gaussian
blur_passes (int, optional) – The number of passed to perform when blurring. Default: 1
threshold (int, optional) – The threshold amount to minimize/maximize mask values to 0 and 100. Percentage value. Default: 0
- set_dilation(amount: float) None
Set the internal dilation object for returned masks
- Parameters:
amount (float) – The amount of erosion/dilation to apply as a percentage of the total mask size. Negative values erode the mask. Positive values dilate the mask
- set_sub_crop(source_offset: np.ndarray, target_offset: np.ndarray, centering: CenteringType, coverage_ratio: float = 1.0) None
Set the internal crop area of the mask to be returned.
This impacts the returned mask from
mask
if the requested mask is required for different face centering than what has been stored.- Parameters:
source_offset (
numpy.ndarray
) – The (x, y) offset for the mask at its stored centeringtarget_offset (
numpy.ndarray
) – The (x, y) offset for the mask at the requested target centeringcentering (str) – The centering to set the sub crop area for. One of “legacy”, “face”. “head”
coverage_ratio (float, optional) – The coverage ratio to be applied to the target image.
None
for default (1.0). Default:None
- property stored_mask: ndarray
The mask at the size of
stored_size
as it is stored (i.e. with no blurring/centering applied).- Type:
numpy.ndarray
- to_dict(is_png=False) MaskAlignmentsFileDict
Convert the mask to a dictionary for saving to an alignments file
- Parameters:
is_png (bool) –
True
if the dictionary is being created for storage in a png header otherwiseFalse
. Default:False
- Returns:
The
Mask
for saving to an alignments file. Contains the keysmask
,affine_matrix
,interpolator
,stored_size
,stored_centering
- Return type:
dict
- to_png_meta() MaskAlignmentsFileDict
Convert the mask to a dictionary supported by png itxt headers.
- Returns:
The
Mask
for saving to an alignments file. Contains the keysmask
,affine_matrix
,interpolator
,stored_size
,stored_centering
- Return type:
dict
- lib.align.detected_face.update_legacy_png_header(filename: str, alignments: Alignments) PNGHeaderDict | None
Update a legacy extracted face from pre v2.1 alignments by placing the alignment data for the face in the png exif header for the given filename with the given alignment data.
If the given file is not a .png then a png is created and the original file is removed
- Parameters:
filename (str) – The image file to update
alignments (
lib.align.alignments.Alignments
) – The alignments data the contains the information to store in the image header. This must be a v2.0 or less alignments file as later versions no longer store the face hash (not required)
- Returns:
The metadata that has been applied to the given image
- Return type:
dict