Open Source in the Enterprise: How Qualcomm Contributes to the Linux Kernel Through Linaro
The open-source development model is crucial for advancing technological innovation among Linux developers creating low-level kernel and driver solutions. This development model facilitates leveraging features and updates from the broader Linux community and allows individual contributors to showcase their solutions. Many developers are now embracing open-source development, acknowledging its vast benefits. This white paper will outline how Qualcomm Technologies, Inc. (QTI) leverages and contributes to open-source technology, focusing on its collaboration with Linaro, which oversees open-source software for the Arm® instruction set architecture.
Qualcomm's Open-Source Initiatives in Collaboration with Linaro
Linaro plays a vital role in streamlining and harmonizing Arm architecture open-source software contributions. Arm architectures, which form the backbone of numerous devices today, benefit immensely from standardized software solutions. Qualcomm's collaboration with Linaro consolidates hardware and software solutions, facilitating mutual benefits for the two entities and the larger developer community by ensuring consistency in open-source contributions and helping to push forward key innovations.
Robotics in IoT and Upstreaming
In the ever-evolving Internet of Things (IoT) sphere, robots represent the intersection of machine intelligence and human interactions. One of Qualcomm Technologies’ leading contributions to open source is the Qualcomm® Robotics RB5 platform. This platform delivers high-speed connectivity and high-accuracy AI/ML inferencing to accelerate robot and drone development for enterprise, industrial, and professional service applications.
Voice-Activated Robots
Voice-activated robots promise to redefine how we perceive and interact with machines. Robots can now dynamically respond to voice commands, adjusting their movement speed or executing specific actions based solely on vocal cues, offering users a more natural interaction method.
The robot below was built using Qualcomm Robotics RB5 running ROS and is controlled using voice-activated Alexa commands:
Figure 1. Left: Alexa device to capture voice commands; Right: Qualcomm Robotics RB5 responding to the voice commands — in this example, increasing its movement speed across the floor
Image Recognition Robots
With its advanced computational capabilities, the Qualcomm Robotics RB5 platform allows robots to utilize cutting-edge image recognition to interact with their surroundings. The example below shows a powered prototype using the Qualcomm Robotics RB5 that can detect, approach, and grasp objects, such as a paper cup:
Figure 2. Robot powered by the Qualcomm Robotics RB5 using image recognition to identify and pick up an object
In this application, an OpenManipulator arm was mounted onto a TurtleBot3 and equipped with an Intel RealSense D435i camera for depth sensing. These components were then integrated with the Qualcomm Robotics RB5 platform via USB connections.
During operation, the camera streams the robot’s view as it navigates, and the Qualcomm Robotics RB5 platform runs object detection to find objects of interest, such as plastic cups in each frame. Once identified, the base unit receives a command to move into position while the arm is instructed to grab the cup. The robotic arm movements are controlled using the DYNAMIXEL SDK.
See below for the flow of information making this possible:
Figure 3. The flow of data from image recognition to actuator movements
In Figure 3, the camera stream is analyzed by a computer vision algorithm (represented by the Camera Node), which uses an RCNN to identify if a cup was found. Once identified, coordinates to reach and pick up the cup are computed. Subsequently, commands are sent from the Qualcomm Robotics RB5 to the OpenManipulator and TurtleBot3 to perform the appropriate actuator movements.
Upstreaming for the Qualcomm Robotics RB5 Development Kit
The team behind the Qualcomm Robotics RB5 Development Kit upstreams Linux enhancements and fixes for the platform's Qualcomm® QRB5165 processor, ensuring synchronization with the mainline Linux kernel as it evolves. The team provides software evaluation builds based on the Yocto project and recent Linux kernel versions. All non-HLOS (high-level OS) binaries are derived from our release infrastructure.
Figure 4. Overview of the release process for the Qualcomm Robotics RB5
The mainline kernel from kernel.org, the primary distribution point for the Linux kernel’s source code, continually evolves and grows. The team uses a version of this mainline kernel, makes enhancements, and creates the downstream version of the kernel. After this, it undergoes an internal review before being shared with the developer community through Code Linaro, and is simultaneously released to external OEMs. Finally, Linaro integrates (upstreams) the updated kernel back into the mainline so that the community and partners can benefit from the team’s enhancements and fixes.
Qualcomm Cloud AI 100: Production Upstream Linux
Another product that underscores Qualcomm Technologies' commitment to open source is the Qualcomm® Cloud AI 100. It is designed as a PCIe-based card and runs cloud-based inference, such as natural language processing, image classification, and recommendation systems. As a niche product, Qualcomm Cloud AI 100 is discretely made available to customers who want to run ML workloads across different regions, edge deployments, and data centers, with the option to run multiple cards in parallel.
The team behind the Qualcomm Cloud AI 100 produces the Direct Rendering Manager (DRM) accelerator driver for the Linux kernel and the Modern Host Interface (MHI) stack used on the Qualcomm Cloud AI 100. These offerings support x86, Arm64, and Linux 3.10-5.16 across multiple distros (CentOS, Red Hat Enterprise Linux, Ubuntu, etc.). Currently, Qualcomm Cloud AI consists of about 10,000 lines of code across 14 files and 300 commits, and the driver has been implemented as a miscellaneous device. It uses several kernel facilities, including PCI, DMA-buf, HWMON, sysfs, debugfs, and the DMA API.
The release procedure for the driver uses Dynamic Kernel Module Support on GitHub. The team deploys it as source code to customers, who compile the driver on their deployed systems, utilizing back-port logic to work with existing systems. On the MHI side, the code is upstreamed and maintained by Linaro. It currently has 24 unique commit authors as of v5.19-rc4, including two from Linaro and five from Qualcomm Technologies.
Collaboration: Why Open Source?
Open Source represents a paradigm shift in how software is developed and disseminated for technological advancement in robotics. By collaborating with initiatives like Linaro, Qualcomm Technologies aims to push the boundaries of what's possible while promoting an open and shared approach to software development.
Transparency and Trust
By making the code accessible to everyone, open-source software allows for increased scrutiny. This transparency ensures that vulnerabilities or bugs can be identified and addressed promptly, enhancing the software's trustworthiness and reliability. While open source doesn't always mean free, the open model can reduce costs, as organizations can harness and customize Qualcomm Technologies and Linaro’s pre-existing solutions to fit their needs instead of building software from scratch.
Flexibility and Customization
Qualcomm Technologies and Linaro’s open-source solutions are modular, allowing companies to tailor them to specific requirements. This adaptability ensures that the software evolves with end-user needs without being tied to the limitations or roadmaps of proprietary software vendors.
Conclusion
Open-source development is pivotal for technological growth and community-driven innovation. As we forge ahead in this era of rapid technological evolution, strategic collaborations such as that between Qualcomm Technologies and Linaro will play a crucial role. Qualcomm Technologies harnesses the immense potential of community-driven Linux developments while its contributions benefit the larger tech community.
For more information on the Qualcomm Robotics RB5 platform, check out the Qualcomm Robotics RB5 developer portal.
All images courtesy of Qualcomm Technologies, Inc.
Note: Qualcomm branded products are products of Qualcomm Technologies, Inc. and/or its subsidiaries. Arm is a registered trademark of Arm Limited (or its subsidiaries or affiliates) in the US and/or elsewhere.
