Thanks for the the efforts of contributors at Deepfakes(https://github.com/deepfakes/faceswap). Scene Selector is not required in generating face-swapped videos. It’s just a helper for first and last step described in part 1 (https://scene-selector.com/2019/10/03/play-deepfake-on-mac/). Though you might not a fan of keying commands in a terminal as a Mac user, I found there are some problems in Faseswap GUI running on my Mac. I will use the command line commands in this example.

Assuming you have set up the environment in part 1. Start Scene Selector, please use Version 3.6.3 or later. To make it quick, I choose the target clip from Blackpink’s MV and swap the face of Jennie to Rose.

First step is to prepare data for Faceswap.

  • Call out Menu->Collection Manager and choose the “Face” segment.
  • Add a new Collection. I named it “faceswap”.
  • Before importing, change the alignment method to Center.
  • You need 2 sets of faces for swapping from and to. I was lazy to collect more high-resolution photos for them. The suggestion from Faceswap is thousands of them including different expressions and orientations. I used the faces from the video instead.
Skip the vector and manually sort the face collection if you don’t have recognition model to sort them or it composes a few different faces. It saves a lot of time in importing.
  • Create new groups for each person, and relocate the faces to them. Just like this. Drop the low quality and incorrectly recognized ones. I got around 300 faces for each from this video.
  • Next step is to export the faces for training.
Check “Alignment File” only when you use models like “dfaker” with “-warp-to-landmarks” options. It needs to access original files to map the alignments to different coordinate systems and calcute hash for each one. The process significantly slows down the exporting speed.
  • In your terminal, change directory to “faceswap”. Set Option “-A”, “-B” to the folder you just exported and “-M” to the location model will be created. You can continue training any time later with the same command for previous interrupted training. I decreased the batch size to 32( default is 64 I think) because the computer became irresponsive at high batch size. I used the following command to start the training.
python faceswap.py train -A /Volumes/Extended/faceswap/faceswap-jennie/ -B /Volumes/Extended/faceswap/faceswap\ rose/ -m /Volumes/Extended/dfaker_model_jennie_rose/ -t dfaker -ss 500 -bs 32 -p

* faceswap used to take alignments.json as default alignment file. The latest version has changed it to “alignments.fsa” by default. If you would like to use mask to train the model, you have to generate mask and modify “mask_type” in the config/train.ini to the types have been generated. Here is the example to generated mask based on the alignment.json generated from Scene Selector.

python tools.py mask -p all -M components -it faces -i /Volumes/Extended/faceswap/faceswap-jennie/ -a /Volumes/Extended/faceswap/faceswap-jennie/alignments.json

  • Wait
  • Wait for more. My AMD RX580, though much faster than running CPU, still takes 5-7 seconds for an iteration. The training won’t converge. General speaking, the loss below 0.02 implies good results. You should judge by looking at the preview.
This is the preview window after 20000 iterations.
Faceswap generates tensorboard logs, use tensorboard –logdir=” your model log directory” to visualized the loss trends. I found It did not improve much after certain iteration. Not sure if a carefulIy chosen face set will help, but I think it’s time to stop.
  • Start to work on the video clips. Use Collection Manager to create a new Movie collection. In movie collection, you can only use one video clip copied from Scene Selector movie player. Select one clip. Copy it and use “Paste Copied Video Clip” to Movie Collection.
Face recognition is not as smart as you(yet). For example, I found 2 frames showing no faces between 2 similar frames. I have to add them back by manual aligning. Focusing on fitting face contour here is enough.
  • Once you have the collection manager showing only the face and some “no need to change” frames, choose “Export Originals”. You will get a folder of images of original frames and a alignment file. Also, the Timing file and audio file will be used later.
  • We have to generate mask file again on the output images. This time we use “frames” for “-it” as the alignments was made in coordinate of extire frame.
python tools.py mask -M components -it frames -i /Volumes/Extended/faceswap/Others/ -a /Volume/Extended/faceswap/Others/alignments.json

There are many options to convert, I used the default setting as following.

python faceswap.py convert -i /Volumes/Extended/faceswap/Others/ -o /Volumes/Extended/faceswap/output/ -m /Volumes/Extended/dfaker_model_jennie_rose -M components -s arg-color

You can use FFMPEG option to merge movie frames to a video or GIF. Alternately use Scene Selector’ Movie Generator to merge them into a video. When you use Movie Generator, move the audio and timing file to the output folder and select it or remove other image files in the top level folder.

  • The video will pop out in a new window. Enjoy.

Faceswap is not the promoting function of Scene Selector because the major task depends on the Faceswap open source project. However, it provides an easy way to prepare the data – takes advantage of the improving face landmark recognization technology from Apple and manual adjustment for extreme cases. It is recommended to upgrade your Mac to Catalina which comes with version 3 of face tool that will save your time in adjusting faces. You spend 1% of the time in importing video and filtering the faces and computer do the rest for you. It still meets our target.

Leave a comment

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: