26 Mar 2019
Optimization, Stabilization, and Tuning of BSPs for Production
Optimization, Stabilization, and Tuning of BSPs for Production
Intrinsyc Technologies Corporation has a 20-year history of developing Board Support Packages (BSP’s) for many embedded products that have had success in the market. Over the past 7 years, Intrinsyc has focused BSP development on Qualcomm Snapdragon-based devices.
This blog describes concepts and considerations that are important for BSP’s for Snapdragon-based devices.
Intrinsyc Development Kits and SoMs (System-on-Modules) are used by customers to rapidly design embedded products in different domains including Internet of Things, wearable, computer vision, smart home hub/appliances, and robotics. In many cases, new carrier hardware is created for the SoMs, and a custom BSP developed, based on the standard Open-QTM development BSP that comes with the boards. Board-specific modifications are also made for GPIOs, power rails, peripheral drivers, and applications to match the product configuration and needs. These steps comprise typical initial rapid prototype development activities for proof-of-concepts.
Once the hardware is validated and feature development is over, there are several steps of optimization and tuning necessary in the productization process to make the end products energy-efficient, optimized for speed, data-secure, easy to manufacture, simple to upgrade, and aesthetically customized. Some of the optimization and tuning topics that are important for product designs, and where Intrinsyc's software development services have expertise, are detailed below.
Software Over-the-air Updates
Intrinsyc has a long history of device management solutions for customers, with particular focus on remote software update; including patented solutions for Internet-enabled device provisioning, upgrade, and recovery mechanism ().
Over the air update is a key technology to help manage embedded devices deployed in the field. Over their lifetime, embedded products based on Android will need their software to be updated for new features, bug fixes and latest software releases. Deploying software updates remotely to a large set of devices that are networked (or have access to networks) brings great advantages in terms of cost, effort, scalability, and time.
Android provides the software framework needed to support Over-the-Air (OTA) updates. In addition, from Android 8.0 (Oreo) release, a new OTA framework enables seamless updates (A/B updates) that reduce downtime by allowing downloads to happen in the background and eliminate the likelihood of post update failure. Product implementations will need customization to fully integrate OTA, and Intrinsyc's software services can provide a working reference starting point including sample interaction application.
There are 3 aspects/phases of OTA upgrade – Creating upgrade packages, deploying them via servers, and on-device installation. Below is the basic OTA update flow in non-A/B (pre-Oreo) and A/B (post-Oreo) architecture.
Google provides reference implementation for most of the components, but changes will be needed in Update Client, Update Server and Update package creation based on specific customer needs.A reference implementation of the Update Server can be provided by Intrinsyc. Typically customers will want to consider aspects like high availability, security, multiple product and version support, and volume of requests when planning and implementing the Update Server.
With over 15 years of experience in integrating Intrinsyc’s patented Remote Update technology as well as other industry standard remote update solutions to numerous platforms and operating systems, our engineering services can support to make Android product designs OTA-ready quickly and efficiently.
Splash Screen Customization
When an embedded Android device with a display is powered on, what displays on the screen will be typically an initial static Splash screen, followed by a dynamic Android boot logo, which is followed by Android lock screen/Home screen/Application Screen.
The BSP must be customized for any new display. Customer devices usually integrate different display panels than the development kit, and so customization will be needed. Customers can modify the kernel themselves, but will need to engage Intrinsyc for assistance on bootloader modifications and integration due to proprietary signing of the bootloader components.
Step One The static splash screen can appear almost instantaneously after power on, if the display driver is integrated to the bootloader. This will also need a preferred splash image - depending on the platform it could be a png file, or a header file to be pre-built into the bootloader. This is one part that customers want to modify, since they can add their own logo for branding. It should be noted that bootloaders vary between platforms (UEFI vs LK bootloader, for instance), and the customization steps also change correspondingly.
Step Two Android Boot Logo – Once the kernel boots and initializes, the splash display goes off for a short duration. Once Android comes up and its display framework is initialized, the display comes back on and shows the scrolling Android logo. This can also be customized to show a different video or scrolling screen, and the method differs between platforms.
Step Three By default, once boot is complete, the Android Home screen comes up. For the final product, customers may prefer instead to have their application launched directly, or to add their own launcher screen instead of the standard Android launcher, which aesthetically differentiates the final product from the development platform. As mentioned in the next section, Intrinsyc can assist with this integration including having an application run as a system app.
Power Optimization and Battery Operation
Products built using Intrinsyc’s Snapdragon-based platforms may be powered from a DC supply, or by a battery (which might be charged while the device operates). If DC powered and working 24/7, it is important that the power consumption is optimized, to reduce power wastage, prevent thermal issues, and increase device longevity. If they are battery powered and have periods of usage interspersed with periods of inactivity, the current consumption in operational, idle (when unused modules are switched off) and suspend (when the entire system in low power mode, waiting for designated interrupt events to wake the cores up) need to be optimized for longevity of operation between charges.
Reducing power consumption involves optimizing (or in some cases implementing) suspend/resume routines for each module/driver, taking care of their dependencies - clocks, power rails, GPIOs, memory usage, dependent libraries, firmware etc. It can also involve tweaking the power management framework for the platform; things such as Dynamic Frequency/Voltage scaling, choice of power governors for different scenarios, and efficient usage of wake locks. For this task, it is essential to have good understanding of Linux and Android driver and power management frameworks, chipset and platform architecture, hardware design, and the methodology/tools to measure current and power usage accurately.
Devices running on battery power will need tuning for the battery charger driver, and in some cases, the fuel gauge driver. Battery charging can be based on PMIC charger circuit (included in many Open-Q SOMs) or external charger ICs. The driver and charging algorithm must be calibrated for battery chemistry, battery type/size, thermal characteristics and charging current. Based on these parameters, it is also necessary to measure and set correct trigger levels to move between trickle, CC, CV and float charging modes. Based on requirements of device behavior, changes may also be needed in bootloaders for handling of deep discharged batteries, animation for battery charging etc.
Intrinsyc's software development services can assist with all aspects of power optimization and battery tuning, as a later phase of product development.
Integration and Stress Testing
Different modules of a product would have been developed separately but will need to work together in the device. Once individual modules are unit tested and validated, the product will undergo the phases of system integration and stress testing. This phase includes integration with external systems, and is when defects that went unnoticed in the individual modules during unit testing will be identified. Issues related to resource contention, API/interface mismatch, timing, thermal/memory load and throughput start to become apparent only when the whole system starts to be operational and working at its peak capabilities. This is also the phase where system stability issues such as crashes, reboots and freezes tend to show up.
At this stage of development, the right kind of validation tests and tools can identify and eliminate major integration issues. These are the kind of issues that can be difficult debug and resolve once the product has been deployed in field. Choosing the appropriate tests for this phase depends on the integrator’s experience in the productization phase, and capability to do system level debugging. Intrinsyc's engineers can assist with stress and performance test development, as well as BSP stabilization during system integration in final phases of BSP development.
Android customization and Optimization
By default, Android Open Source Platform (AOSP) has applications and framework components that cater to varied use cases. For a product, much of this is redundant and can be disabled or removed. This optimization step serves 2 purposes.
It reduces the boot time of the device. Normally android initializes all framework components before boot is complete. Depending on the platform, disabling or removing unnecessary components at kernel and framework levels can enable the device to complete the boot to Home Screen significantly faster. As an example, a device that does not have mobile connectivity will benefit from removing components related to mobile connectivity, and applications such as dialer used by phones. Optimization is also usually done in the kernel at Device tree and module levels.
It reduces the memory footprint of the device – Once the unnecessary components are removed, the amount of memory needed for the device is reduced leading to potential memory cost savings, and it also provides an option to increase the partition size for user to store data needed by user applications.
This optimization can be done using tools such as boot chart, and also needs understanding of the various components and their inter-working, since the framework has many dependencies that need to be carefully managed. Optimizing the device tree also needs good knowledge of kernel internals.
Device-specific tuning/trimming of the Android framework must be balanced with code maintenance impact; when framework modifications are invasive then any future upgrades to a new Android baseline are a larger task. Using best practices for this footprint tuning and overlay customization, is critical.
To further tune the BSP size and performance for production it is important to set logging/diagnostics to the debug levels required for a final product. There is an Android build mechanism to help in that. Customers can use Debug Build or Engineering Build, occupying more memory and being longer to boot (and less reactive in certain cases) during development, and User Build, which has less logging options but is faster to boot and needs less memory, for the final product. Apart from these the logging and debug levels can be customized so that some critical software stacks will still be logged in the final product. The development kit BSPs, intended for prototyping and R&D, are by nature debug/engineering builds. User Build integration, test, and stabilization are a key step in productization once pre-production prototypes are ready.
Another aspect related to diagnostics and productization is the ADB (Android Debug Bridge) and system console. In final products, ADB is either completely or partially disabled. Some customers wish to customize this, so that ADB can be enabled in the final product using secure keys, for debug or upgrade purposes. There are various levels of device security, and the level appropriate to any product depends on features such as proprietary/sensitive technology, vulnerable cloud access, sensitive customer data, etc. For devices which need to be secured against any malicious intrusion, Intrinsyc can integrate the complete Qualcomm Secure Boot chain solution as described below.
Device Security (secure boot chain)
Preventing any malicious access to a production device requires a full secure boot solution. For Android this currently consists of three separate technologies, Qualcomm Secure Boot, Verified Boot, and dm-verity.
Qualcomm Secure Boot
A proprietary technology in which the processor ROM-based bootstrap validates the signed low-level bootloader against customer security fuses blown into each processor. Other binaries including secure TrustZone images, hypervisor, and graphic processor binaries are also signed with the proprietary certificate technology. Intrinsyc engineers can assist with the complete integration and image signing. Secure Boot insures integrity of the highest-level bootloader.
Secure Boot is also a requirement for certain device security technologies such as hardware-backed key stores within the TrustZone technology. Some of these pieces are required for features such as Android Full-Disk or File-Based Encryption (to lock down sensitive user data).
The integrity of the kernel and rootfs are ensured by Verified Boot, in which the boot image is signed by customer keys. Access to the appropriate keystore is programmed into the highest-level bootloader.
The integrity of the Android system binaries is ensured by dm-verity. This feature must be built into the kernel, and appropriately configured.
In summary, the BSPs provided with the and are richly-featured, tested source which is an ideal starting point for R&D and product design, development, and customization. Once production prototyping is complete, the next phase of BSP development is to address the various optimization, stabilization, and security features described here. Intrinsyc software services can assist with all these aspects in the final product development phase.
Dinesh RV is Software Engineering Manager for Intrinsyc Technology's India operations, and is based in Bangalore. Dinesh has over 20 years of embedded system software development, helping bring products to market in such fields as telecom, connected camera, and robotics.