Hack w/ Limbic Media

Interactive Electronica and IoT

Visit Limbic Media

hack.limbicmedia.ca was created by the developers and engineers at Limbic Media to share their knowledge and experiences with DIY and hacker community-at-large.

by Steven Bjornson

Let's Get BitBake(d)

BitBake, a part of the Yocto Project, is a tool for automating the compilation of Linux kernels and packaging custom distributions. It is primarily intended for embedded systems. Therefore, you may only find the following useful if you are developing for Linux based embedded systems.

The goal of the project is to give developers access to a solid set of tools and components which ease the development time for their specific embedded hardware requirements. This is accomplished through collaboration with hardware vendors who create and maintain build recipes for their specific hardware sets. These recipes can then be easily extended and modified by embedded developers for their specific needs. Modifications can include control over components, services, device trees, and services as well as fine tuning of system footprint sizes. Through the modularity of the system, end developers can extend existing recipes for their platform which reduces the impact of upstream changes on development. In other words, as vendors develop and extend their systems, these upstream changes can easily be integrated.

The BitBake tool itself operates along the same principals as Make: different recipes are written to execute sets of required tasks. However, there are a few things about BitBake which may not be obvious when starting out.

Dependencies are Handled Automagically in BitBake

There are a large set of tools necessary to create a Linux distribution (such as the proper cross compiler for an embedded platform or the tool for uploading an image onto the hardware). These tools are setup in recipes which can be linked together as dependencies. These dependencies are automatically resolved by BitBake during the build process. In essence, this makes BitBake the only tool necessary in the build environment since all other tools are prepared and launched by it.

When first starting with BitBake it is important to know that there are two main environment variables that, when set, allow BitBake to resolve dependencies for specific platforms. Without this knowledge it can be difficult to determine what BitBake is doing under the hood (if you try using the command bitbake -vDDD <recipe>, you will see some 600,000 lines of text).

These two environment variables are MACHINE and DISTRO. With these set to your embedded machine and desired Linux distribution, BitBake will call on all the appropriate recipes for your desired build. Basically, once these are set, BitBake knows how to build the kernel for you. No extra work required. In other words, don't worry about what BitBake is doing. As long as you set these variables you are likely to get the build you desire.

Recipes in BitBake Run in Parallel

BitBake handles the scheduling and creation of the different components of the build. All recipes are built independently and in non-deterministic order. This is part of what makes the whole system confusing. If you try to trace all of the actions (the 600,000 lines mentioned above) you will get lost very quickly. When you build your own recipe it will, in most cases, run independent of the rest.

Extend Recipes with .bbappend Files

Because of the parallel nature of BitBake recipes, if you want to modify the build the only way to inject changes in the correct order is to create .bbappend files. These files share the name of their related recipe file (e.g. foo.bb and foo.bbappend) and are kept in your own meta- folder. Append files, as their name suggests, act as a means for appending functions, variables, and file modifications into an existing recipe. Functions can be added as tasks in a build (see here for details), and these tasks can be ordered around other tasks in a recipe (because recipe tasks run in sequence, not parallel). An append file is the only reliable way to augment a build.

An example of when this can be useful is in the case of patch files. If you were to create your own recipe, there would be no way to guarantee that the patch you created would be applied at the correct time (and therefore not overwritten by another recipe). The best solution is to create an append file for an existing recipe. This append file can add the patch file to the main recipe's list of files. In most cases there is a do_patch task in the recipe file (which looks for .patch files and applies them). By creating an append file for this recipe, the original recipe remains unedited allowing the patch file to be non-invasively added.

Creating a separate recipe to augment a task in an existing recipe is not likely to work (again, recipes run independently) and editing an existing recipe directly can break a build in the event of upstream changes.

There is one caveat to creating append files: it can be difficult to find the recipe responsible for the tasks you wish to augment. In other words, creating a .bbappend file is easy, but it can be difficult to trace the behaviour you wish to augment to a specific recipe. Try grepping through the verbose output of a build (using -vDDD) to find the specific files or behaviours you're hoping to augment.

BitBake Caches Previous Builds

BitBake keeps track of the files used in the build and in subsequent builds only updates files which have changed. Thus, after the first build of a Kernel, which can take several hours, the time to build is reduced significantly.

For teams of developers, this cache can be put onto a network file server (NFS) where it can be used as if it was a local directory.

Conclusion

After spending some time with BitBake, I can say that the information above would have been very useful when I first began. There are a tonne of resources on BitBake, but these resources are quite dense. When starting out, the immense amount of information on BitBake became a bit of a hindrance. The information is this article may be a little confusing for those just starting out. However, just try to remember, BitBake wants to do all the work and so if you set the right path variables and write non-invasive appends (basically, stay out of its way) it will treat you well.

Need help with a similar project?

Innovative Digital Art

Limbic Media

Consulting Engineering

Limbic Consulting