MVStylizer: An Efficient Edge-Assisted Video Photorealistic Style
  Transfer System for Mobile Phones
  Recent research has made great progress in realizing neural style transfer of
images, which denotes transforming an image to a desired style. Many users
start to use their mobile phones to record their daily life, and then edit and
share the captured images and videos with other users. However, directly
applying existing style transfer approaches on videos, i.e., transferring the
style of a video frame by frame, requires an extremely large amount of
computation resources. It is still technically unaffordable to perform style
transfer of videos on mobile phones. To address this challenge, we propose
MVStylizer, an efficient edge-assisted photorealistic video style transfer
system for mobile phones. Instead of performing stylization frame by frame,
only key frames in the original video are processed by a pre-trained deep
neural network (DNN) on edge servers, while the rest of stylized intermediate
frames are generated by our designed optical-flow-based frame interpolation
algorithm on mobile phones. A meta-smoothing module is also proposed to
simultaneously upscale a stylized frame to arbitrary resolution and remove
style transfer related distortions in these upscaled frames. In addition, for
the sake of continuously enhancing the performance of the DNN model on the edge
server, we adopt a federated learning scheme to keep retraining each DNN model
on the edge server with collected data from mobile clients and syncing with a
global DNN model on the cloud server. Such a scheme effectively leverages the
diversity of collected data from various mobile clients and efficiently
improves the system performance. Our experiments demonstrate that MVStylizer
can generate stylized videos with an even better visual quality compared to the
state-of-the-art method while achieving 75.5$\times$ speedup for
1920$\times$1080 videos.
