My graduation project. An interactive terminal learning platform that provides guided courses, visual file tree functionality, and error message guidance. Each course creates a container on the server side for user interaction.

Stack: TypeScript, Vue, Express, WebSocket, Docker in Docker
License: GPL

GitHub

2024 - 2025

Multichannel video synchronization tool that works across devices and platforms, with mobile support.

Video: Lin Pei-Yao. Triangular Relationship / Three-Channel Loop Video / 03'39" / 2021

Background

Multichannel video playing is a common needs in modern art exhibitions. The usual way to implement it is using dual monitors connected to the same computer, and use proprietary video player to control it.

However, this method requires different monitors to be physically connected, and doesn’t allow using mobile devices as display.

Therefore, I developed a Web-based system to allow multidevice multichannel video synchronisation.

Structure

The system consists of two parts: Server and Client

• Server
  • NodeJS, TypeScript, Electron, tRPC, Express, Socket.io
  • Store the video files
  • Manage the current playhead
  • Calculate the latency to each connected client
  • Periodically update current playhead to every client via WebSocket
  • Provide a tRPC and RESTful interface for control
  • Wrapped inside an Electron desktop application to simplify starting.
• Clients
  • TypeScript, ReactJS, Vite, tRPC, Socket.io, Redux
  • Connect to the server instance, load available videos
  • Dynamically syncing current playhead to the remote
  • Provide Admin interface when open inside Electron

Easy-to-Use Interface

Admin Interface of Synchan

In an art exhibition environment, the exhibition managers usually are not from tech background. It is required to provide an easy-to-use interface to start the whole system. Also, the installations usually don’t have mouse and keyboard connected once the set-up is completed. Thus, the unattended start-up is important to minimise the effort for the exhibition managers.

There are many limitations for a Web-based application to achieve this. Modern browsers require user interaction to play the sound, auto full-screen is also blocked.

Therefore, I wrapped the backend and the admin interface as an Electron application. The server is spawned once the application starts, requiring no terminal or background service configuration. Furthermore, Electron can be configured to allow autoplay and open-to-fullscreen. The application also memorises last played video, and automatically play it on the start.

As a result, exhibition managers can just turn on the computer, and the whole installation is up and running.

Real-Time Sync Engine

Latency between devices is the biggest problem for cross-device synchronisation, especially when both have audio tracks, the difference becomes trivial.

The latency mainly comes from the network latency between server and clients, while different clients have different latency. In the time code synchronisation protocol, I implemented a round-trip latency measurement. Every time the server sends a new time code, it starts a stopwatch, and wait for client’s ping back. Then server uses a moving-window algorithm to calculate the median of recent transmissions, so the spikes in networking can be eliminated. Finally server includes the calculated latency in the next time code packet sent to the client.

Once the client receives the time code with latency, it can calculate the real time code. But the next problem arises:

The video seek operation in the browser takes long time (>0.5s) on low-end devices like Raspberry Pi. So it is not possible to do precise control using seek, and it also breaks the continuity of the video playback.

Therefore, I use micro speed adjustment to align the target playhead, by a linear speed control between 1.05x to 0.95x. The number chosen is a balance between adjustment balance and user experience. It is unnoticeable by the users, and it doesn’t breaks the video playing or lag the browser.

After the fix, latency can be minimised to under 5ms, which is indistinguishable by human ear.

Performance Optimisation via Preloading

The networking may not be stable in an exhibition environment. While playing on low-end devices like Raspberry Pi, mid-playback buffering often causes little lags. Besides, exhibition installations usually play the same videos, and the set-up time is before the exhibition starts, which can be ignored.

Therefore, instead of streaming, it has many benefits to preload the whole video before playing.

I implemented a cache layer in the video player, which downloads the whole file and saves into IndexedDB. This method works in every modern browser out-of-box.

By using preloading, it solves the mid-playback buffering issue and significantly improves the stability.

Cross-Platform Clients

Synchan itself ships a Web-based client, which can run on every platform with a modern browser, including both desktop and mobile devices. Its server-client structure also allows it to be used by different clients.

On low-end devices like Raspberry Pi, running a Chromium instance to play video may be too heavy. Thus, I developed a headless client VLChan using VLC Python SDK. It connects to a Synchan Server and plays local files with synchronised playhead, allows best performance on a Raspberry Pi.

Show Cases

Lin Pei-Yao Solo Exhibition: The Dual Double-Channel (2024)

Used in work Gaze Triangle

• 3 video channels + 2 audio channels
• Server: Raspberry Pi 4
• Clients
  • Raspberry Pi 4 (same machine, connected monitor)
  • Mac Mini (projector + Bluetooth headset)
  • Android Phone (play audio by built-in speaker)

Lin Pei-Yao Solo Exhibition: Who is the speaker? (2025)

Used in work Inter-view with a Philosopher

• 1 video + 2 audio channels
• Server: Mac Mini
• Clients
  • Mac Mini (same machine): Video + audio
  • Raspberry Pi 4 (audio only VLC client)
• Extra time-code control with Actionwire

Want to Try?

Currently available by invitation only. For inquiries, please contact wancat@wancat.cc

This project is developed specific for Lin Pei-Yao Solo Exhibition: Who is the speaker? (2025).

In this exhibition, it required speech recognition for selected keywords, and perform specific actions, including smart light control and video playhead control. The speech recognition is done in real-time, deployed locally on a Raspberry Pi.

For example, in the command “Drink Tea”, it blinks one set of lights and seeks the video to a specific time (00:25) and jumps back to original position after 10 seconds.

Different voice commands have different actions, and some of them may depends on each other.

To make the concurrent events manageable, I used Reactive Programming design pattern via RxPy.

Structure

The program is divided into three parts: Events, Commands, and Actions.

Events are the input to the system, including microphone and WebSocket inputs. It will be transform into an Observable stream.

Actions are the output behaviours. Including light control and video playhead control.

Commands are the business logic. Freely connecting, composing, mixing all the inputs, and producing one output. Can be easily customised by user needs.

• Events (inputs)
  • Microphone -> Vosk -> Keyword extraction
  • WebSocket -> Current timecode
• Commands
  • Define the pipeline logic for every command
  • Written in Reactive Programming styles
  • No hidden state management. Easy to update
• Actions (outputs)
  • Light control -> LIFX LAN API
  • Video playhead control -> HTTP request

Keywords Recognition

I used Vosk as the offline speech recognition model, because it is small enough to run on a Raspberry Pi.

The original accuracy of the model is not good, and it is designed as a speech-to-text model, not for recognise specific keywords. I customised the vocabulary list to make it only select tokens that appears the keyword list. It’s also important to include [unk] in the list, to prevent the model output unknown words.

Synchan Integration

The video playing system is Synchan, a multichannel multidevice synchronised video playing system. It allows control via HTTP requests, and it updates the current time code to every clients via WebSocket. The time code is parsed as an Observable stream, and used to perform action according to video time code.

For example, in the beginning of the video, it turns on the light in the exhibition as the light is turned on in the video. And in the command “Drink Tea”, it seeks the video to back to 00:25, where the performer asked “Would you like some tea?”, and seeks back to the original playhead after 10 seconds.

Tech Stack

• Python
• RxPy
• Vosk
• Socket.IO
• lifxlan: Smart Light Control in LAN

Gallery

Want to Try?

Currently available by invitation only. For inquiries, please contact wancat@wancat.cc

2025

Stock traffic light analysis tool. Supports dividend yield calculations, import/export of watchlists, and stock filtering by yield.

Stack: TypeScript, React, Redux, Cheerio, Cloudflare Worker