Video Frame Interpolation (VFI)#

Important Note: Video frame interpolation is enabled through configuration files, not command-line parameters. Please add a video_frame_interpolation configuration block to your JSON config file to enable this feature.

Overview#

Video Frame Interpolation (VFI) is a technique that generates intermediate frames between existing frames to increase the frame rate and create smoother video playback. LightX2V integrates the RIFE (Real-Time Intermediate Flow Estimation) model to provide high-quality frame interpolation capabilities.

What is RIFE?#

RIFE is a state-of-the-art video frame interpolation method that uses optical flow estimation to generate intermediate frames. It can effectively:

Increase video frame rate (e.g., from 16 FPS to 32 FPS)
Create smooth motion transitions
Maintain high visual quality with minimal artifacts
Process videos in real-time

Installation and Setup#

Download RIFE Model#

First, download the RIFE model weights using the provided script:

python tools/download_rife.py <target_directory>

For example, to download to the location:

python tools/download_rife.py /path/to/rife/train_log

This script will:

Download RIFEv4.26 model from HuggingFace
Extract and place the model files in the correct directory
Clean up temporary files

Usage#

Configuration File Setup#

Video frame interpolation is enabled through configuration files. Add a video_frame_interpolation configuration block to your JSON config file:

{
    "infer_steps": 50,
    "target_video_length": 81,
    "target_height": 480,
    "target_width": 832,
    "fps": 16,
    "video_frame_interpolation": {
        "algo": "rife",
        "target_fps": 32,
        "model_path": "/path/to/rife/train_log"
    }
}

Command Line Interface#

Run inference using a configuration file that includes VFI settings:

python lightx2v/infer.py \
    --model_cls wan2.1 \
    --task t2v \
    --model_path /path/to/model \
    --config_json ./configs/video_frame_interpolation/wan_t2v.json \
    --prompt "A beautiful sunset over the ocean" \
    --save_result_path ./output.mp4

Configuration Parameters#

In the video_frame_interpolation configuration block:

algo: Frame interpolation algorithm, currently supports “rife”
target_fps: Target frame rate for the output video
model_path: RIFE model path, typically “/path/to/rife/train_log”

Other related configurations:

fps: Source video frame rate (default 16)

Configuration Priority#

The system automatically handles video frame rate configuration with the following priority:

video_frame_interpolation.target_fps - If video frame interpolation is enabled, this frame rate is used as the output frame rate
fps (default 16) - If video frame interpolation is not enabled, this frame rate is used; it’s always used as the source frame rate

How It Works#

Frame Interpolation Process#

Source Video Generation: The base model generates video frames at the source FPS
Frame Analysis: RIFE analyzes adjacent frames to estimate optical flow
Intermediate Frame Generation: New frames are generated between existing frames
Temporal Smoothing: The interpolated frames create smooth motion transitions

Technical Details#

Input Format: ComfyUI Image tensors [N, H, W, C] in range [0, 1]
Output Format: Interpolated ComfyUI Image tensors [M, H, W, C] in range [0, 1]
Processing: Automatic padding and resolution handling
Memory Optimization: Efficient GPU memory management

Example Configurations#

Basic Frame Rate Doubling#

Create configuration file wan_t2v_vfi_32fps.json:

{
    "infer_steps": 50,
    "target_video_length": 81,
    "target_height": 480,
    "target_width": 832,
    "seed": 42,
    "sample_guide_scale": 6,
    "enable_cfg": true,
    "fps": 16,
    "video_frame_interpolation": {
        "algo": "rife",
        "target_fps": 32,
        "model_path": "/path/to/rife/train_log"
    }
}

Run command:

python lightx2v/infer.py \
    --model_cls wan2.1 \
    --task t2v \
    --model_path ./models/wan2.1 \
    --config_json ./wan_t2v_vfi_32fps.json \
    --prompt "A cat playing in the garden" \
    --save_result_path ./output_32fps.mp4

Higher Frame Rate Enhancement#

Create configuration file wan_i2v_vfi_60fps.json:

{
    "infer_steps": 30,
    "target_video_length": 81,
    "target_height": 480,
    "target_width": 832,
    "seed": 42,
    "sample_guide_scale": 6,
    "enable_cfg": true,
    "fps": 16,
    "video_frame_interpolation": {
        "algo": "rife",
        "target_fps": 60,
        "model_path": "/path/to/rife/train_log"
    }
}

Run command:

python lightx2v/infer.py \
    --model_cls wan2.1 \
    --task i2v \
    --model_path ./models/wan2.1 \
    --config_json ./wan_i2v_vfi_60fps.json \
    --image_path ./input.jpg \
    --prompt "Smooth camera movement" \
    --save_result_path ./output_60fps.mp4

Performance Considerations#

Memory Usage#

RIFE processing requires additional GPU memory
Memory usage scales with video resolution and length
Consider using lower resolutions for longer videos

Processing Time#

Frame interpolation adds processing overhead
Higher target frame rates require more computation
Processing time is roughly proportional to the number of interpolated frames

Quality vs Speed Trade-offs#

Higher interpolation ratios may introduce artifacts
Optimal range: 2x to 4x frame rate increase
For extreme interpolation (>4x), consider multiple passes

Best Practices#

Optimal Use Cases#

Motion-heavy videos: Benefit most from frame interpolation
Camera movements: Smoother panning and zooming
Action sequences: Reduced motion blur perception
Slow-motion effects: Create fluid slow-motion videos

Recommended Settings#

Source FPS: 16-24 FPS (generated by base model)
Target FPS: 32-60 FPS (2x to 4x increase)
Resolution: Up to 720p for best performance

Troubleshooting#

Common Issues#

Out of Memory: Reduce video resolution or target FPS
Artifacts in output: Lower the interpolation ratio
Slow processing: Check GPU memory and consider using CPU offloading

Solutions#

Solve issues by modifying the configuration file:

{
    // For memory issues, use lower resolution
    "target_height": 480,
    "target_width": 832,

    // For quality issues, use moderate interpolation
    "video_frame_interpolation": {
        "target_fps": 24  // instead of 60
    },

    // For performance issues, enable offloading
    "cpu_offload": true
}

Technical Implementation#

The RIFE integration in LightX2V includes:

RIFEWrapper: ComfyUI-compatible wrapper for RIFE model
Automatic Model Loading: Seamless integration with the inference pipeline
Memory Optimization: Efficient tensor management and GPU memory usage
Quality Preservation: Maintains original video quality while adding frames

Video Frame Interpolation (VFI)

Contents