Facebook Open-sources Its Newest Hardware, the Time Appliance Project
The Time Appliance Project (TAP) is Facebook’s newest sub-microsecond network time synchronization system. Designed to enhance data centers, it is open-source, well documented, and ready to review.
Data centers require synchronized timing between servers, databases, and services in order to facilitate a wide range of applications like electronic trading, voice and video, and wireless sensor networks.
Recently, Facebook released the Time Card, an open-source x86 PCIe interface card designed to provide accurate synchronization at tens of nanoseconds. The Time Appliance Project includes all of the hardware on the Time Card and is intended to speed up data centers.
Facebook’s Time Card/Time Appliance solution. Image used courtesy of Ahmad Byagowi
Central to the design of this card is the RCB-F9T timing board from u-blox. u-blox’s ZED-F9T multi-band timing module is also said to be a key technology in Facebook’s newest hardware project.
In collaboration with Open Compute, Facebook has launched the Time Appliances Project (TAP) with the intent to fully open-source this project and address many of the issues involved with proprietary time devices.
Diving Into the Hardware of the Time Card
Available as a GIT repository, the Time Appliance Project consists of hardware files, software files, BoMs, and details on the Precision Time Protocol (PTP) profile developed within the OCP.
The Time Card itself, which makes up the interface portion of the Time Appliance Project, was built in Altium Designer. It is not a high-density design. However, because the design is intended to interface relatively high-frequency GNSS systems, signal integrity and EMC data would be a welcome addition to the GIT files.
The PCB is a 62-mil thick six-layer board, using two ground reference planes on layer two and layer five. Layer three and layer four are orthogonally routed to avoid the significant effects of broadside coupling.
Time Card PCB layout (reference planes disabled) details layout design and density. Image from Facebook design files
Building the Time Appliance Project
The Time Appliance is built using three major hardware components:
- A GNSS timing source (u-blox’s RCB-F9T board)
- A miniaturized rubidium-based atomic clock (Microchip’s MAC-SA5X)
- A system-on-module FPGA (Alinx’s Artix-7 AC7100B SoM)
Mock-up of the early prototype model for the Time Card. Image used courtesy of Facebook
The FPGA system receives two primary signals from the GNSS: a pulse-per-second (PPS) reference signal and the time-of-day (ToD) signal. The system is clocked either by the rubidium oscillator or externally at 10 MHz.
After filtering and processing the two synchronization signals, the resulting reference signal is output to the PCI bridge to the x86 system network interface card (NIC).
High-level block diagram for the Time Appliance FPGA. Image used courtesy of Facebook
Digging deeper in the FPGA sub-systems, Facebook overviews the filtering (DCM) of the GNSS input signals.
The FPGA digital filtering sub-system. Image used courtesy of Facebook
Ultimately fed into the ToD unit, these filtered signals represent an accumulated increment of 8-nanosecond digital values. The intent is to use the GNSS signal to snapshot the incremental digital counts to result in an ideal of exactly 1-second PPS.
This system is said to build in redundancy, allowing for the MAC to take over in the absence of a GNSS signal for a period of time; specifically, the incremental digital count should be within one microsecond in 24 hours for defined loss-of-time accuracy.
The card is designed to interface with any x86 host system with a NIC capable of hardware time-stamping, which turns it into a “time appliance.” The system's synchronization is protocol-agnostic, allowing for the use of Network Time Protocol (NTP), Precision Time Protocol, or other alternative synchronization protocols.
Why Did Facebook Choose Open-source for TAP?
TAP is not the first open-source project Facebook has released into the wider community. In 2011 Facebook publicly released designs for a state-of-the-art data center, and alongside other industry members, launched the Open Compute Project Foundation.
Facebook has recognized the difficulty of working with proprietary systems: long development times for patching security vulnerabilities, closed-system software with limited configurability, hardware that is not user-serviceable, and high costs.
Breaking with the mold, the Time Card is said to be an inexpensive timing card that can be built with COATS components and PCB design to benefit anyone interested in sub-microsecond synchronization.
All said, as an open-source project, there is a great deal more information available for designers who are interested in either the hardware, software, or available protocol stacks.