diff options
Diffstat (limited to 'documentation/overview-manual/overview-manual-concepts.xml')
| -rw-r--r-- | documentation/overview-manual/overview-manual-concepts.xml | 3235 |
1 files changed, 0 insertions, 3235 deletions
diff --git a/documentation/overview-manual/overview-manual-concepts.xml b/documentation/overview-manual/overview-manual-concepts.xml deleted file mode 100644 index 58b64bd269..0000000000 --- a/documentation/overview-manual/overview-manual-concepts.xml +++ /dev/null @@ -1,3235 +0,0 @@ -<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN" -"http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd" -[<!ENTITY % poky SYSTEM "../poky.ent"> %poky; ] > -<!--SPDX-License-Identifier: CC-BY-2.0-UK--> - -<chapter id=' overview-manual-concepts'> -<title>Yocto Project Concepts</title> - - <para> - This chapter provides explanations for Yocto Project concepts that - go beyond the surface of "how-to" information and reference (or - look-up) material. - Concepts such as components, the - <ulink url='&YOCTO_DOCS_REF_URL;#build-system-term'>OpenEmbedded build system</ulink> - workflow, cross-development toolchains, shared state cache, and so - forth are explained. - </para> - - <section id='yocto-project-components'> - <title>Yocto Project Components</title> - - <para> - The - <ulink url='&YOCTO_DOCS_REF_URL;#bitbake-term'>BitBake</ulink> - task executor together with various types of configuration files - form the - <ulink url='&YOCTO_DOCS_REF_URL;#oe-core'>OpenEmbedded-Core</ulink>. - This section overviews these components by describing their use and - how they interact. - </para> - - <para> - BitBake handles the parsing and execution of the data files. - The data itself is of various types: - <itemizedlist> - <listitem><para> - <emphasis>Recipes:</emphasis> - Provides details about particular pieces of software. - </para></listitem> - <listitem><para> - <emphasis>Class Data:</emphasis> - Abstracts common build information (e.g. how to build a - Linux kernel). - </para></listitem> - <listitem><para> - <emphasis>Configuration Data:</emphasis> - Defines machine-specific settings, policy decisions, and - so forth. - Configuration data acts as the glue to bind everything - together. - </para></listitem> - </itemizedlist> - </para> - - <para> - BitBake knows how to combine multiple data sources together and - refers to each data source as a layer. - For information on layers, see the - "<ulink url='&YOCTO_DOCS_DEV_URL;#understanding-and-creating-layers'>Understanding and Creating Layers</ulink>" - section of the Yocto Project Development Tasks Manual. - </para> - - <para> - Following are some brief details on these core components. - For additional information on how these components interact during - a build, see the - "<link linkend='openembedded-build-system-build-concepts'>OpenEmbedded Build System Concepts</link>" - section. - </para> - - <section id='usingpoky-components-bitbake'> - <title>BitBake</title> - - <para> - BitBake is the tool at the heart of the - <ulink url='&YOCTO_DOCS_REF_URL;#build-system-term'>OpenEmbedded build system</ulink> - and is responsible for parsing the - <ulink url='&YOCTO_DOCS_REF_URL;#metadata'>Metadata</ulink>, - generating a list of tasks from it, and then executing those - tasks. - </para> - - <para> - This section briefly introduces BitBake. - If you want more information on BitBake, see the - <ulink url='&YOCTO_DOCS_BB_URL;'>BitBake User Manual</ulink>. - </para> - - <para> - To see a list of the options BitBake supports, use either of - the following commands: - <literallayout class='monospaced'> - $ bitbake -h - $ bitbake --help - </literallayout> - </para> - - <para> - The most common usage for BitBake is - <filename>bitbake <replaceable>packagename</replaceable></filename>, - where <filename>packagename</filename> is the name of the - package you want to build (referred to as the "target"). - The target often equates to the first part of a recipe's - filename (e.g. "foo" for a recipe named - <filename>foo_1.3.0-r0.bb</filename>). - So, to process the - <filename>matchbox-desktop_1.2.3.bb</filename> recipe file, you - might type the following: - <literallayout class='monospaced'> - $ bitbake matchbox-desktop - </literallayout> - Several different versions of - <filename>matchbox-desktop</filename> might exist. - BitBake chooses the one selected by the distribution - configuration. - You can get more details about how BitBake chooses between - different target versions and providers in the - "<ulink url='&YOCTO_DOCS_BB_URL;#bb-bitbake-preferences'>Preferences</ulink>" - section of the BitBake User Manual. - </para> - - <para> - BitBake also tries to execute any dependent tasks first. - So for example, before building - <filename>matchbox-desktop</filename>, BitBake would build a - cross compiler and <filename>glibc</filename> if they had not - already been built. - </para> - - <para> - A useful BitBake option to consider is the - <filename>-k</filename> or <filename>--continue</filename> - option. - This option instructs BitBake to try and continue processing - the job as long as possible even after encountering an error. - When an error occurs, the target that failed and those that - depend on it cannot be remade. - However, when you use this option other dependencies can - still be processed. - </para> - </section> - - <section id='overview-components-recipes'> - <title>Recipes</title> - - <para> - Files that have the <filename>.bb</filename> suffix are - "recipes" files. - In general, a recipe contains information about a single piece - of software. - This information includes the location from which to download - the unaltered source, any source patches to be applied to that - source (if needed), which special configuration options to - apply, how to compile the source files, and how to package the - compiled output. - </para> - - <para> - The term "package" is sometimes used to refer to recipes. - However, since the word "package" is used for the packaged - output from the OpenEmbedded build system (i.e. - <filename>.ipk</filename> or <filename>.deb</filename> files), - this document avoids using the term "package" when referring - to recipes. - </para> - </section> - - <section id='overview-components-classes'> - <title>Classes</title> - - <para> - Class files (<filename>.bbclass</filename>) contain information - that is useful to share between recipes files. - An example is the - <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-autotools'><filename>autotools</filename></ulink> - class, which contains common settings for any application that - Autotools uses. - The - "<ulink url='&YOCTO_DOCS_REF_URL;#ref-classes'>Classes</ulink>" - chapter in the Yocto Project Reference Manual provides - details about classes and how to use them. - </para> - </section> - - <section id='overview-components-configurations'> - <title>Configurations</title> - - <para> - The configuration files (<filename>.conf</filename>) define - various configuration variables that govern the OpenEmbedded - build process. - These files fall into several areas that define machine - configuration options, distribution configuration options, - compiler tuning options, general common configuration options, - and user configuration options in - <filename>conf/local.conf</filename>, which is found in the - <ulink url='&YOCTO_DOCS_REF_URL;#build-directory'>Build Directory</ulink>. - </para> - </section> - </section> - - <section id='overview-layers'> - <title>Layers</title> - - <para> - Layers are repositories that contain related metadata (i.e. - sets of instructions) that tell the OpenEmbedded build system how - to build a target. - Yocto Project's - <link linkend='the-yocto-project-layer-model'>layer model</link> - facilitates collaboration, sharing, customization, and reuse - within the Yocto Project development environment. - Layers logically separate information for your project. - For example, you can use a layer to hold all the configurations - for a particular piece of hardware. - Isolating hardware-specific configurations allows you to share - other metadata by using a different layer where that metadata - might be common across several pieces of hardware. - </para> - - <para> - Many layers exist that work in the Yocto Project development - environment. - The - <ulink url='https://caffelli-staging.yoctoproject.org/software-overview/layers/'>Yocto Project Curated Layer Index</ulink> - and - <ulink url='http://layers.openembedded.org/layerindex/branch/master/layers/'>OpenEmbedded Layer Index</ulink> - both contain layers from which you can use or leverage. - </para> - - <para> - By convention, layers in the Yocto Project follow a specific form. - Conforming to a known structure allows BitBake to make assumptions - during builds on where to find types of metadata. - You can find procedures and learn about tools (i.e. - <filename>bitbake-layers</filename>) for creating layers suitable - for the Yocto Project in the - "<ulink url='&YOCTO_DOCS_DEV_URL;#understanding-and-creating-layers'>Understanding and Creating Layers</ulink>" - section of the Yocto Project Development Tasks Manual. - </para> - </section> - - <section id="openembedded-build-system-build-concepts"> - <title>OpenEmbedded Build System Concepts</title> - - <para> - This section takes a more detailed look inside the build - process used by the - <ulink url='&YOCTO_DOCS_REF_URL;#build-system-term'>OpenEmbedded build system</ulink>, - which is the build system specific to the Yocto Project. - At the heart of the build system is BitBake, the task executor. - </para> - - <para> - The following diagram represents the high-level workflow of a - build. - The remainder of this section expands on the fundamental input, - output, process, and metadata logical blocks that make up the - workflow. - </para> - - <para id='general-workflow-figure'> - <imagedata fileref="figures/YP-flow-diagram.png" format="PNG" align='center' width="8in"/> - </para> - - <para> - In general, the build's workflow consists of several functional - areas: - <itemizedlist> - <listitem><para> - <emphasis>User Configuration:</emphasis> - metadata you can use to control the build process. - </para></listitem> - <listitem><para> - <emphasis>Metadata Layers:</emphasis> - Various layers that provide software, machine, and - distro metadata. - </para></listitem> - <listitem><para> - <emphasis>Source Files:</emphasis> - Upstream releases, local projects, and SCMs. - </para></listitem> - <listitem><para> - <emphasis>Build System:</emphasis> - Processes under the control of - <ulink url='&YOCTO_DOCS_REF_URL;#bitbake-term'>BitBake</ulink>. - This block expands on how BitBake fetches source, applies - patches, completes compilation, analyzes output for package - generation, creates and tests packages, generates images, - and generates cross-development tools. - </para></listitem> - <listitem><para> - <emphasis>Package Feeds:</emphasis> - Directories containing output packages (RPM, DEB or IPK), - which are subsequently used in the construction of an - image or Software Development Kit (SDK), produced by the - build system. - These feeds can also be copied and shared using a web - server or other means to facilitate extending or updating - existing images on devices at runtime if runtime package - management is enabled. - </para></listitem> - <listitem><para> - <emphasis>Images:</emphasis> - Images produced by the workflow. - </para></listitem> - <listitem><para> - <emphasis>Application Development SDK:</emphasis> - Cross-development tools that are produced along with - an image or separately with BitBake. - </para></listitem> - </itemizedlist> - </para> - - <section id="user-configuration"> - <title>User Configuration</title> - - <para> - User configuration helps define the build. - Through user configuration, you can tell BitBake the - target architecture for which you are building the image, - where to store downloaded source, and other build properties. - </para> - - <para> - The following figure shows an expanded representation of the - "User Configuration" box of the - <link linkend='general-workflow-figure'>general workflow figure</link>: - </para> - - <para> - <imagedata fileref="figures/user-configuration.png" align="center" width="8in" depth="4.5in" /> - </para> - - <para> - BitBake needs some basic configuration files in order to - complete a build. - These files are <filename>*.conf</filename> files. - The minimally necessary ones reside as example files in the - <filename>build/conf</filename> directory of the - <ulink url='&YOCTO_DOCS_REF_URL;#source-directory'>Source Directory</ulink>. - For simplicity, this section refers to the Source Directory as - the "Poky Directory." - </para> - - <para> - When you clone the - <ulink url='&YOCTO_DOCS_REF_URL;#poky'>Poky</ulink> - Git repository or you download and unpack a Yocto Project - release, you can set up the Source Directory to be named - anything you want. - For this discussion, the cloned repository uses the default - name <filename>poky</filename>. - <note> - The Poky repository is primarily an aggregation of existing - repositories. - It is not a canonical upstream source. - </note> - </para> - - <para> - The <filename>meta-poky</filename> layer inside Poky contains - a <filename>conf</filename> directory that has example - configuration files. - These example files are used as a basis for creating actual - configuration files when you source - <ulink url='&YOCTO_DOCS_REF_URL;#structure-core-script'><filename>&OE_INIT_FILE;</filename></ulink>, - which is the build environment script. - </para> - - <para> - Sourcing the build environment script creates a - <ulink url='&YOCTO_DOCS_REF_URL;#build-directory'>Build Directory</ulink> - if one does not already exist. - BitBake uses the Build Directory for all its work during - builds. - The Build Directory has a <filename>conf</filename> directory - that contains default versions of your - <filename>local.conf</filename> and - <filename>bblayers.conf</filename> configuration files. - These default configuration files are created only if versions - do not already exist in the Build Directory at the time you - source the build environment setup script. - </para> - - <para> - Because the Poky repository is fundamentally an aggregation of - existing repositories, some users might be familiar with - running the <filename>&OE_INIT_FILE;</filename> script - in the context of separate - <ulink url='&YOCTO_DOCS_REF_URL;#oe-core'>OpenEmbedded-Core</ulink> - and BitBake repositories rather than a single Poky repository. - This discussion assumes the script is executed from - within a cloned or unpacked version of Poky. - </para> - - <para> - Depending on where the script is sourced, different - sub-scripts are called to set up the Build Directory - (Yocto or OpenEmbedded). - Specifically, the script - <filename>scripts/oe-setup-builddir</filename> inside the - poky directory sets up the Build Directory and seeds the - directory (if necessary) with configuration files appropriate - for the Yocto Project development environment. - <note> - The <filename>scripts/oe-setup-builddir</filename> script - uses the <filename>$TEMPLATECONF</filename> variable to - determine which sample configuration files to locate. - </note> - </para> - - <para> - The <filename>local.conf</filename> file provides many - basic variables that define a build environment. - Here is a list of a few. - To see the default configurations in a - <filename>local.conf</filename> file created by the build - environment script, see the - <ulink url='&YOCTO_GIT_URL;/cgit/cgit.cgi/poky/tree/meta-poky/conf/local.conf.sample'><filename>local.conf.sample</filename></ulink> - in the <filename>meta-poky</filename> layer: - <itemizedlist> - <listitem><para> - <emphasis>Target Machine Selection:</emphasis> - Controlled by the - <ulink url='&YOCTO_DOCS_REF_URL;#var-MACHINE'><filename>MACHINE</filename></ulink> - variable. - </para></listitem> - <listitem><para> - <emphasis>Download Directory:</emphasis> - Controlled by the - <ulink url='&YOCTO_DOCS_REF_URL;#var-DL_DIR'><filename>DL_DIR</filename></ulink> - variable. - </para></listitem> - <listitem><para> - <emphasis>Shared State Directory:</emphasis> - Controlled by the - <ulink url='&YOCTO_DOCS_REF_URL;#var-SSTATE_DIR'><filename>SSTATE_DIR</filename></ulink> - variable. - </para></listitem> - <listitem><para> - <emphasis>Build Output:</emphasis> - Controlled by the - <ulink url='&YOCTO_DOCS_REF_URL;#var-TMPDIR'><filename>TMPDIR</filename></ulink> - variable. - </para></listitem> - <listitem><para> - <emphasis>Distribution Policy:</emphasis> - Controlled by the - <ulink url='&YOCTO_DOCS_REF_URL;#var-DISTRO'><filename>DISTRO</filename></ulink> - variable. - </para></listitem> - <listitem><para> - <emphasis>Packaging Format:</emphasis> - Controlled by the - <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGE_CLASSES'><filename>PACKAGE_CLASSES</filename></ulink> - variable. - </para></listitem> - <listitem><para> - <emphasis>SDK Target Architecture:</emphasis> - Controlled by the - <ulink url='&YOCTO_DOCS_REF_URL;#var-SDKMACHINE'><filename>SDKMACHINE</filename></ulink> - variable. - </para></listitem> - <listitem><para> - <emphasis>Extra Image Packages:</emphasis> - Controlled by the - <ulink url='&YOCTO_DOCS_REF_URL;#var-EXTRA_IMAGE_FEATURES'><filename>EXTRA_IMAGE_FEATURES</filename></ulink> - variable. - </para></listitem> - </itemizedlist> - <note> - Configurations set in the - <filename>conf/local.conf</filename> file can also be set - in the <filename>conf/site.conf</filename> and - <filename>conf/auto.conf</filename> configuration files. - </note> - </para> - - <para> - The <filename>bblayers.conf</filename> file tells BitBake what - layers you want considered during the build. - By default, the layers listed in this file include layers - minimally needed by the build system. - However, you must manually add any custom layers you have - created. - You can find more information on working with the - <filename>bblayers.conf</filename> file in the - "<ulink url='&YOCTO_DOCS_DEV_URL;#enabling-your-layer'>Enabling Your Layer</ulink>" - section in the Yocto Project Development Tasks Manual. - </para> - - <para> - The files <filename>site.conf</filename> and - <filename>auto.conf</filename> are not created by the - environment initialization script. - If you want the <filename>site.conf</filename> file, you - need to create that yourself. - The <filename>auto.conf</filename> file is typically created by - an autobuilder: - <itemizedlist> - <listitem><para> - <emphasis><filename>site.conf</filename>:</emphasis> - You can use the <filename>conf/site.conf</filename> - configuration file to configure multiple - build directories. - For example, suppose you had several build environments - and they shared some common features. - You can set these default build properties here. - A good example is perhaps the packaging format to use - through the - <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGE_CLASSES'><filename>PACKAGE_CLASSES</filename></ulink> - variable.</para> - - <para>One useful scenario for using the - <filename>conf/site.conf</filename> file is to extend - your - <ulink url='&YOCTO_DOCS_REF_URL;#var-BBPATH'><filename>BBPATH</filename></ulink> - variable to include the path to a - <filename>conf/site.conf</filename>. - Then, when BitBake looks for Metadata using - <filename>BBPATH</filename>, it finds the - <filename>conf/site.conf</filename> file and applies - your common configurations found in the file. - To override configurations in a particular build - directory, alter the similar configurations within - that build directory's - <filename>conf/local.conf</filename> file. - </para></listitem> - <listitem><para> - <emphasis><filename>auto.conf</filename>:</emphasis> - The file is usually created and written to by - an autobuilder. - The settings put into the file are typically the - same as you would find in the - <filename>conf/local.conf</filename> or the - <filename>conf/site.conf</filename> files. - </para></listitem> - </itemizedlist> - </para> - - <para> - You can edit all configuration files to further define - any particular build environment. - This process is represented by the "User Configuration Edits" - box in the figure. - </para> - - <para> - When you launch your build with the - <filename>bitbake <replaceable>target</replaceable></filename> - command, BitBake sorts out the configurations to ultimately - define your build environment. - It is important to understand that the - <ulink url='&YOCTO_DOCS_REF_URL;#build-system-term'>OpenEmbedded build system</ulink> - reads the configuration files in a specific order: - <filename>site.conf</filename>, <filename>auto.conf</filename>, - and <filename>local.conf</filename>. - And, the build system applies the normal assignment statement - rules as described in the - "<ulink url='&YOCTO_DOCS_BB_URL;#bitbake-user-manual-metadata'>Syntax and Operators</ulink>" - chapter of the BitBake User Manual. - Because the files are parsed in a specific order, variable - assignments for the same variable could be affected. - For example, if the <filename>auto.conf</filename> file and - the <filename>local.conf</filename> set - <replaceable>variable1</replaceable> to different values, - because the build system parses <filename>local.conf</filename> - after <filename>auto.conf</filename>, - <replaceable>variable1</replaceable> is assigned the value from - the <filename>local.conf</filename> file. - </para> - </section> - - <section id="metadata-machine-configuration-and-policy-configuration"> - <title>Metadata, Machine Configuration, and Policy Configuration</title> - - <para> - The previous section described the user configurations that - define BitBake's global behavior. - This section takes a closer look at the layers the build system - uses to further control the build. - These layers provide Metadata for the software, machine, and - policies. - </para> - - <para> - In general, three types of layer input exists. - You can see them below the "User Configuration" box in the - <link linkend='general-workflow-figure'>general workflow figure</link>: - <itemizedlist> - <listitem><para> - <emphasis>Metadata (<filename>.bb</filename> + Patches):</emphasis> - Software layers containing user-supplied recipe files, - patches, and append files. - A good example of a software layer might be the - <ulink url='https://github.com/meta-qt5/meta-qt5'><filename>meta-qt5</filename></ulink> - layer from the - <ulink url='http://layers.openembedded.org/layerindex/branch/master/layers/'>OpenEmbedded Layer Index</ulink>. - This layer is for version 5.0 of the popular - <ulink url='https://wiki.qt.io/About_Qt'>Qt</ulink> - cross-platform application development framework for - desktop, embedded and mobile. - </para></listitem> - <listitem><para> - <emphasis>Machine BSP Configuration:</emphasis> - Board Support Package (BSP) layers (i.e. "BSP Layer" - in the following figure) providing machine-specific - configurations. - This type of information is specific to a particular - target architecture. - A good example of a BSP layer from the - <link linkend='gs-reference-distribution-poky'>Poky Reference Distribution</link> - is the - <ulink url='&YOCTO_GIT_URL;/cgit/cgit.cgi/poky/tree/meta-yocto-bsp'><filename>meta-yocto-bsp</filename></ulink> - layer. - </para></listitem> - <listitem><para> - <emphasis>Policy Configuration:</emphasis> - Distribution Layers (i.e. "Distro Layer" in the - following figure) providing top-level or general - policies for the images or SDKs being built for a - particular distribution. - For example, in the Poky Reference Distribution the - distro layer is the - <ulink url='&YOCTO_GIT_URL;/cgit/cgit.cgi/poky/tree/meta-poky'><filename>meta-poky</filename></ulink> - layer. - Within the distro layer is a - <filename>conf/distro</filename> directory that - contains distro configuration files (e.g. - <ulink url='&YOCTO_GIT_URL;/cgit/cgit.cgi/poky/tree/meta-poky/conf/distro/poky.conf'><filename>poky.conf</filename></ulink> - that contain many policy configurations for the - Poky distribution. - </para></listitem> - </itemizedlist> - </para> - - <para> - The following figure shows an expanded representation of - these three layers from the - <link linkend='general-workflow-figure'>general workflow figure</link>: - </para> - - <para> - <imagedata fileref="figures/layer-input.png" align="center" width="8in" depth="8in" /> - </para> - - <para> - In general, all layers have a similar structure. - They all contain a licensing file - (e.g. <filename>COPYING.MIT</filename>) if the layer is to be - distributed, a <filename>README</filename> file as good - practice and especially if the layer is to be distributed, a - configuration directory, and recipe directories. - You can learn about the general structure for layers used with - the Yocto Project in the - "<ulink url='&YOCTO_DOCS_DEV_URL;#creating-your-own-layer'>Creating Your Own Layer</ulink>" - section in the Yocto Project Development Tasks Manual. - For a general discussion on layers and the many layers from - which you can draw, see the - "<link linkend='overview-layers'>Layers</link>" and - "<link linkend='the-yocto-project-layer-model'>The Yocto Project Layer Model</link>" - sections both earlier in this manual. - </para> - - <para> - If you explored the previous links, you discovered some - areas where many layers that work with the Yocto Project - exist. - The - <ulink url="http://git.yoctoproject.org/">Source Repositories</ulink> - also shows layers categorized under "Yocto Metadata Layers." - <note> - Layers exist in the Yocto Project Source Repositories that - cannot be found in the OpenEmbedded Layer Index. - These layers are either deprecated or experimental - in nature. - </note> - </para> - - <para> - BitBake uses the <filename>conf/bblayers.conf</filename> file, - which is part of the user configuration, to find what layers it - should be using as part of the build. - </para> - - <section id="distro-layer"> - <title>Distro Layer</title> - - <para> - The distribution layer provides policy configurations for - your distribution. - Best practices dictate that you isolate these types of - configurations into their own layer. - Settings you provide in - <filename>conf/distro/<replaceable>distro</replaceable>.conf</filename> override - similar settings that BitBake finds in your - <filename>conf/local.conf</filename> file in the Build - Directory. - </para> - - <para> - The following list provides some explanation and references - for what you typically find in the distribution layer: - <itemizedlist> - <listitem><para> - <emphasis>classes:</emphasis> - Class files (<filename>.bbclass</filename>) hold - common functionality that can be shared among - recipes in the distribution. - When your recipes inherit a class, they take on the - settings and functions for that class. - You can read more about class files in the - "<ulink url='&YOCTO_DOCS_REF_URL;#ref-classes'>Classes</ulink>" - chapter of the Yocto Reference Manual. - </para></listitem> - <listitem><para> - <emphasis>conf:</emphasis> - This area holds configuration files for the - layer (<filename>conf/layer.conf</filename>), - the distribution - (<filename>conf/distro/<replaceable>distro</replaceable>.conf</filename>), - and any distribution-wide include files. - </para></listitem> - <listitem><para> - <emphasis>recipes-*:</emphasis> - Recipes and append files that affect common - functionality across the distribution. - This area could include recipes and append files - to add distribution-specific configuration, - initialization scripts, custom image recipes, - and so forth. - Examples of <filename>recipes-*</filename> - directories are <filename>recipes-core</filename> - and <filename>recipes-extra</filename>. - Hierarchy and contents within a - <filename>recipes-*</filename> directory can vary. - Generally, these directories contain recipe files - (<filename>*.bb</filename>), recipe append files - (<filename>*.bbappend</filename>), directories - that are distro-specific for configuration files, - and so forth. - </para></listitem> - </itemizedlist> - </para> - </section> - - <section id="bsp-layer"> - <title>BSP Layer</title> - - <para> - The BSP Layer provides machine configurations that - target specific hardware. - Everything in this layer is specific to the machine for - which you are building the image or the SDK. - A common structure or form is defined for BSP layers. - You can learn more about this structure in the - <ulink url='&YOCTO_DOCS_BSP_URL;'>Yocto Project Board Support Package (BSP) Developer's Guide</ulink>. - <note> - In order for a BSP layer to be considered compliant - with the Yocto Project, it must meet some structural - requirements. - </note> - </para> - - <para> - The BSP Layer's configuration directory contains - configuration files for the machine - (<filename>conf/machine/<replaceable>machine</replaceable>.conf</filename>) - and, of course, the layer - (<filename>conf/layer.conf</filename>). - </para> - - <para> - The remainder of the layer is dedicated to specific recipes - by function: <filename>recipes-bsp</filename>, - <filename>recipes-core</filename>, - <filename>recipes-graphics</filename>, - <filename>recipes-kernel</filename>, and so forth. - Metadata can exist for multiple formfactors, graphics - support systems, and so forth. - <note> - While the figure shows several - <filename>recipes-*</filename> directories, not all - these directories appear in all BSP layers. - </note> - </para> - </section> - - <section id="software-layer"> - <title>Software Layer</title> - - <para> - The software layer provides the Metadata for additional - software packages used during the build. - This layer does not include Metadata that is specific to - the distribution or the machine, which are found in their - respective layers. - </para> - - <para> - This layer contains any recipes, append files, and - patches, that your project needs. - </para> - </section> - </section> - - <section id="sources-dev-environment"> - <title>Sources</title> - - <para> - In order for the OpenEmbedded build system to create an - image or any target, it must be able to access source files. - The - <link linkend='general-workflow-figure'>general workflow figure</link> - represents source files using the "Upstream Project Releases", - "Local Projects", and "SCMs (optional)" boxes. - The figure represents mirrors, which also play a role in - locating source files, with the "Source Materials" box. - </para> - - <para> - The method by which source files are ultimately organized is - a function of the project. - For example, for released software, projects tend to use - tarballs or other archived files that can capture the - state of a release guaranteeing that it is statically - represented. - On the other hand, for a project that is more dynamic or - experimental in nature, a project might keep source files in a - repository controlled by a Source Control Manager (SCM) such as - Git. - Pulling source from a repository allows you to control - the point in the repository (the revision) from which you - want to build software. - Finally, a combination of the two might exist, which would - give the consumer a choice when deciding where to get - source files. - </para> - - <para> - BitBake uses the - <ulink url='&YOCTO_DOCS_REF_URL;#var-SRC_URI'><filename>SRC_URI</filename></ulink> - variable to point to source files regardless of their location. - Each recipe must have a <filename>SRC_URI</filename> variable - that points to the source. - </para> - - <para> - Another area that plays a significant role in where source - files come from is pointed to by the - <ulink url='&YOCTO_DOCS_REF_URL;#var-DL_DIR'><filename>DL_DIR</filename></ulink> - variable. - This area is a cache that can hold previously downloaded - source. - You can also instruct the OpenEmbedded build system to create - tarballs from Git repositories, which is not the default - behavior, and store them in the <filename>DL_DIR</filename> - by using the - <ulink url='&YOCTO_DOCS_REF_URL;#var-BB_GENERATE_MIRROR_TARBALLS'><filename>BB_GENERATE_MIRROR_TARBALLS</filename></ulink> - variable. - </para> - - <para> - Judicious use of a <filename>DL_DIR</filename> directory can - save the build system a trip across the Internet when looking - for files. - A good method for using a download directory is to have - <filename>DL_DIR</filename> point to an area outside of your - Build Directory. - Doing so allows you to safely delete the Build Directory - if needed without fear of removing any downloaded source file. - </para> - - <para> - The remainder of this section provides a deeper look into the - source files and the mirrors. - Here is a more detailed look at the source file area of the - <link linkend='general-workflow-figure'>general workflow figure</link>: - </para> - - <para> - <imagedata fileref="figures/source-input.png" width="6in" depth="6in" align="center" /> - </para> - - <section id='upstream-project-releases'> - <title>Upstream Project Releases</title> - - <para> - Upstream project releases exist anywhere in the form of an - archived file (e.g. tarball or zip file). - These files correspond to individual recipes. - For example, the figure uses specific releases each for - BusyBox, Qt, and Dbus. - An archive file can be for any released product that can be - built using a recipe. - </para> - </section> - - <section id='local-projects'> - <title>Local Projects</title> - - <para> - Local projects are custom bits of software the user - provides. - These bits reside somewhere local to a project - perhaps - a directory into which the user checks in items (e.g. - a local directory containing a development source tree - used by the group). - </para> - - <para> - The canonical method through which to include a local - project is to use the - <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-externalsrc'><filename>externalsrc</filename></ulink> - class to include that local project. - You use either the <filename>local.conf</filename> or a - recipe's append file to override or set the - recipe to point to the local directory on your disk to pull - in the whole source tree. - </para> - </section> - - <section id='scms'> - <title>Source Control Managers (Optional)</title> - - <para> - Another place from which the build system can get source - files is with - <ulink url='&YOCTO_DOCS_BB_URL;#bb-fetchers'>fetchers</ulink> - employing various Source Control Managers (SCMs) such as - Git or Subversion. - In such cases, a repository is cloned or checked out. - The - <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-fetch'><filename>do_fetch</filename></ulink> - task inside BitBake uses - the <ulink url='&YOCTO_DOCS_REF_URL;#var-SRC_URI'><filename>SRC_URI</filename></ulink> - variable and the argument's prefix to determine the correct - fetcher module. - <note> - For information on how to have the OpenEmbedded build - system generate tarballs for Git repositories and place - them in the - <ulink url='&YOCTO_DOCS_REF_URL;#var-DL_DIR'><filename>DL_DIR</filename></ulink> - directory, see the - <ulink url='&YOCTO_DOCS_REF_URL;#var-BB_GENERATE_MIRROR_TARBALLS'><filename>BB_GENERATE_MIRROR_TARBALLS</filename></ulink> - variable in the Yocto Project Reference Manual. - </note> - </para> - - <para> - When fetching a repository, BitBake uses the - <ulink url='&YOCTO_DOCS_REF_URL;#var-SRCREV'><filename>SRCREV</filename></ulink> - variable to determine the specific revision from which to - build. - </para> - </section> - - <section id='source-mirrors'> - <title>Source Mirror(s)</title> - - <para> - Two kinds of mirrors exist: pre-mirrors and regular - mirrors. - The - <ulink url='&YOCTO_DOCS_REF_URL;#var-PREMIRRORS'><filename>PREMIRRORS</filename></ulink> - and - <ulink url='&YOCTO_DOCS_REF_URL;#var-MIRRORS'><filename>MIRRORS</filename></ulink> - variables point to these, respectively. - BitBake checks pre-mirrors before looking upstream for any - source files. - Pre-mirrors are appropriate when you have a shared - directory that is not a directory defined by the - <ulink url='&YOCTO_DOCS_REF_URL;#var-DL_DIR'><filename>DL_DIR</filename></ulink> - variable. - A Pre-mirror typically points to a shared directory that is - local to your organization. - </para> - - <para> - Regular mirrors can be any site across the Internet - that is used as an alternative location for source - code should the primary site not be functioning for - some reason or another. - </para> - </section> - </section> - - <section id="package-feeds-dev-environment"> - <title>Package Feeds</title> - - <para> - When the OpenEmbedded build system generates an image or an - SDK, it gets the packages from a package feed area located - in the - <ulink url='&YOCTO_DOCS_REF_URL;#build-directory'>Build Directory</ulink>. - The - <link linkend='general-workflow-figure'>general workflow figure</link> - shows this package feeds area in the upper-right corner. - </para> - - <para> - This section looks a little closer into the package feeds - area used by the build system. - Here is a more detailed look at the area: - <imagedata fileref="figures/package-feeds.png" align="center" width="7in" depth="6in" /> - </para> - - <para> - Package feeds are an intermediary step in the build process. - The OpenEmbedded build system provides classes to generate - different package types, and you specify which classes to - enable through the - <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGE_CLASSES'><filename>PACKAGE_CLASSES</filename></ulink> - variable. - Before placing the packages into package feeds, - the build process validates them with generated output quality - assurance checks through the - <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-insane'><filename>insane</filename></ulink> - class. - </para> - - <para> - The package feed area resides in the Build Directory. - The directory the build system uses to temporarily store - packages is determined by a combination of variables and the - particular package manager in use. - See the "Package Feeds" box in the illustration and note the - information to the right of that area. - In particular, the following defines where package files are - kept: - <itemizedlist> - <listitem><para> - <ulink url='&YOCTO_DOCS_REF_URL;#var-DEPLOY_DIR'><filename>DEPLOY_DIR</filename></ulink>: - Defined as <filename>tmp/deploy</filename> in the Build - Directory. - </para></listitem> - <listitem><para> - <filename>DEPLOY_DIR_*</filename>: - Depending on the package manager used, the package type - sub-folder. - Given RPM, IPK, or DEB packaging and tarball creation, - the - <ulink url='&YOCTO_DOCS_REF_URL;#var-DEPLOY_DIR_RPM'><filename>DEPLOY_DIR_RPM</filename></ulink>, - <ulink url='&YOCTO_DOCS_REF_URL;#var-DEPLOY_DIR_IPK'><filename>DEPLOY_DIR_IPK</filename></ulink>, - <ulink url='&YOCTO_DOCS_REF_URL;#var-DEPLOY_DIR_DEB'><filename>DEPLOY_DIR_DEB</filename></ulink>, - or - <ulink url='&YOCTO_DOCS_REF_URL;#var-DEPLOY_DIR_TAR'><filename>DEPLOY_DIR_TAR</filename></ulink>, - variables are used, respectively. - </para></listitem> - <listitem><para> - <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGE_ARCH'><filename>PACKAGE_ARCH</filename></ulink>: - Defines architecture-specific sub-folders. - For example, packages could exist for the i586 or - qemux86 architectures. - </para></listitem> - </itemizedlist> - </para> - - <para> - BitBake uses the - <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package_write_deb'><filename>do_package_write_*</filename></ulink> - tasks to generate packages and place them into the package - holding area (e.g. <filename>do_package_write_ipk</filename> - for IPK packages). - See the - "<ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package_write_deb'><filename>do_package_write_deb</filename></ulink>", - "<ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package_write_ipk'><filename>do_package_write_ipk</filename></ulink>", - "<ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package_write_rpm'><filename>do_package_write_rpm</filename></ulink>", - and - "<ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package_write_tar'><filename>do_package_write_tar</filename></ulink>" - sections in the Yocto Project Reference Manual - for additional information. - As an example, consider a scenario where an IPK packaging - manager is being used and package architecture support for - both i586 and qemux86 exist. - Packages for the i586 architecture are placed in - <filename>build/tmp/deploy/ipk/i586</filename>, while packages - for the qemux86 architecture are placed in - <filename>build/tmp/deploy/ipk/qemux86</filename>. - </para> - </section> - - <section id='bitbake-dev-environment'> - <title>BitBake</title> - - <para> - The OpenEmbedded build system uses - <ulink url='&YOCTO_DOCS_REF_URL;#bitbake-term'>BitBake</ulink> - to produce images and Software Development Kits (SDKs). - You can see from the - <link linkend='general-workflow-figure'>general workflow figure</link>, - the BitBake area consists of several functional areas. - This section takes a closer look at each of those areas. - <note> - Separate documentation exists for the BitBake tool. - See the - <ulink url='&YOCTO_DOCS_BB_URL;'>BitBake User Manual</ulink> - for reference material on BitBake. - </note> - </para> - - <section id='source-fetching-dev-environment'> - <title>Source Fetching</title> - - <para> - The first stages of building a recipe are to fetch and - unpack the source code: - <imagedata fileref="figures/source-fetching.png" align="center" width="6.5in" depth="5in" /> - </para> - - <para> - The - <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-fetch'><filename>do_fetch</filename></ulink> - and - <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-unpack'><filename>do_unpack</filename></ulink> - tasks fetch the source files and unpack them into the - <ulink url='&YOCTO_DOCS_REF_URL;#build-directory'>Build Directory</ulink>. - <note> - For every local file (e.g. <filename>file://</filename>) - that is part of a recipe's - <ulink url='&YOCTO_DOCS_REF_URL;#var-SRC_URI'><filename>SRC_URI</filename></ulink> - statement, the OpenEmbedded build system takes a - checksum of the file for the recipe and inserts the - checksum into the signature for the - <filename>do_fetch</filename> task. - If any local file has been modified, the - <filename>do_fetch</filename> task and all tasks that - depend on it are re-executed. - </note> - By default, everything is accomplished in the Build - Directory, which has a defined structure. - For additional general information on the Build Directory, - see the - "<ulink url='&YOCTO_DOCS_REF_URL;#structure-core-build'><filename>build/</filename></ulink>" - section in the Yocto Project Reference Manual. - </para> - - <para> - Each recipe has an area in the Build Directory where the - unpacked source code resides. - The - <ulink url='&YOCTO_DOCS_REF_URL;#var-S'><filename>S</filename></ulink> - variable points to this area for a recipe's unpacked source - code. - The name of that directory for any given recipe is defined - from several different variables. - The preceding figure and the following list describe - the Build Directory's hierarchy: - <itemizedlist> - <listitem><para> - <ulink url='&YOCTO_DOCS_REF_URL;#var-TMPDIR'><filename>TMPDIR</filename></ulink>: - The base directory where the OpenEmbedded build - system performs all its work during the build. - The default base directory is the - <filename>tmp</filename> directory. - </para></listitem> - <listitem><para> - <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGE_ARCH'><filename>PACKAGE_ARCH</filename></ulink>: - The architecture of the built package or packages. - Depending on the eventual destination of the - package or packages (i.e. machine architecture, - <ulink url='&YOCTO_DOCS_REF_URL;#hardware-build-system-term'>build host</ulink>, - SDK, or specific machine), - <filename>PACKAGE_ARCH</filename> varies. - See the variable's description for details. - </para></listitem> - <listitem><para> - <ulink url='&YOCTO_DOCS_REF_URL;#var-TARGET_OS'><filename>TARGET_OS</filename></ulink>: - The operating system of the target device. - A typical value would be "linux" (e.g. - "qemux86-poky-linux"). - </para></listitem> - <listitem><para> - <ulink url='&YOCTO_DOCS_REF_URL;#var-PN'><filename>PN</filename></ulink>: - The name of the recipe used to build the package. - This variable can have multiple meanings. - However, when used in the context of input files, - <filename>PN</filename> represents the the name - of the recipe. - </para></listitem> - <listitem><para> - <ulink url='&YOCTO_DOCS_REF_URL;#var-WORKDIR'><filename>WORKDIR</filename></ulink>: - The location where the OpenEmbedded build system - builds a recipe (i.e. does the work to create the - package). - <itemizedlist> - <listitem><para> - <ulink url='&YOCTO_DOCS_REF_URL;#var-PV'><filename>PV</filename></ulink>: - The version of the recipe used to build the - package. - </para></listitem> - <listitem><para> - <ulink url='&YOCTO_DOCS_REF_URL;#var-PR'><filename>PR</filename></ulink>: - The revision of the recipe used to build the - package. - </para></listitem> - </itemizedlist> - </para></listitem> - <listitem><para> - <ulink url='&YOCTO_DOCS_REF_URL;#var-S'><filename>S</filename></ulink>: - Contains the unpacked source files for a given - recipe. - <itemizedlist> - <listitem><para> - <ulink url='&YOCTO_DOCS_REF_URL;#var-BPN'><filename>BPN</filename></ulink>: - The name of the recipe used to build the - package. - The <filename>BPN</filename> variable is - a version of the <filename>PN</filename> - variable but with common prefixes and - suffixes removed. - </para></listitem> - <listitem><para> - <ulink url='&YOCTO_DOCS_REF_URL;#var-PV'><filename>PV</filename></ulink>: - The version of the recipe used to build the - package. - </para></listitem> - </itemizedlist> - </para></listitem> - </itemizedlist> - <note> - In the previous figure, notice that two sample - hierarchies exist: one based on package architecture (i.e. - <filename>PACKAGE_ARCH</filename>) and one based on a - machine (i.e. <filename>MACHINE</filename>). - The underlying structures are identical. - The differentiator being what the OpenEmbedded build - system is using as a build target (e.g. general - architecture, a build host, an SDK, or a specific - machine). - </note> - </para> - </section> - - <section id='patching-dev-environment'> - <title>Patching</title> - - <para> - Once source code is fetched and unpacked, BitBake locates - patch files and applies them to the source files: - <imagedata fileref="figures/patching.png" align="center" width="7in" depth="6in" /> - </para> - - <para> - The - <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-patch'><filename>do_patch</filename></ulink> - task uses a recipe's - <ulink url='&YOCTO_DOCS_REF_URL;#var-SRC_URI'><filename>SRC_URI</filename></ulink> - statements and the - <ulink url='&YOCTO_DOCS_REF_URL;#var-FILESPATH'><filename>FILESPATH</filename></ulink> - variable to locate applicable patch files. - </para> - - <para> - Default processing for patch files assumes the files have - either <filename>*.patch</filename> or - <filename>*.diff</filename> file types. - You can use <filename>SRC_URI</filename> parameters to - change the way the build system recognizes patch files. - See the - <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-patch'><filename>do_patch</filename></ulink> - task for more information. - </para> - - <para> - BitBake finds and applies multiple patches for a single - recipe in the order in which it locates the patches. - The <filename>FILESPATH</filename> variable defines the - default set of directories that the build system uses to - search for patch files. - Once found, patches are applied to the recipe's source - files, which are located in the - <ulink url='&YOCTO_DOCS_REF_URL;#var-S'><filename>S</filename></ulink> - directory. - </para> - - <para> - For more information on how the source directories are - created, see the - "<link linkend='source-fetching-dev-environment'>Source Fetching</link>" - section. - For more information on how to create patches and how the - build system processes patches, see the - "<ulink url='&YOCTO_DOCS_DEV_URL;#new-recipe-patching-code'>Patching Code</ulink>" - section in the Yocto Project Development Tasks Manual. - You can also see the - "<ulink url='&YOCTO_DOCS_SDK_URL;#sdk-devtool-use-devtool-modify-to-modify-the-source-of-an-existing-component'>Use <filename>devtool modify</filename> to Modify the Source of an Existing Component</ulink>" - section in the Yocto Project Application Development and - the Extensible Software Development Kit (SDK) manual and - the - "<ulink url='&YOCTO_DOCS_KERNEL_DEV_URL;#using-traditional-kernel-development-to-patch-the-kernel'>Using Traditional Kernel Development to Patch the Kernel</ulink>" - section in the Yocto Project Linux Kernel Development - Manual. - </para> - </section> - - <section id='configuration-compilation-and-staging-dev-environment'> - <title>Configuration, Compilation, and Staging</title> - - <para> - After source code is patched, BitBake executes tasks that - configure and compile the source code. - Once compilation occurs, the files are copied to a holding - area (staged) in preparation for packaging: - <imagedata fileref="figures/configuration-compile-autoreconf.png" align="center" width="7in" depth="5in" /> - </para> - - <para> - This step in the build process consists of the following - tasks: - <itemizedlist> - <listitem><para> - <emphasis><ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-prepare_recipe_sysroot'><filename>do_prepare_recipe_sysroot</filename></ulink></emphasis>: - This task sets up the two sysroots in - <filename>${</filename><ulink url='&YOCTO_DOCS_REF_URL;#var-WORKDIR'><filename>WORKDIR</filename></ulink><filename>}</filename> - (i.e. <filename>recipe-sysroot</filename> and - <filename>recipe-sysroot-native</filename>) so that - during the packaging phase the sysroots can contain - the contents of the - <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-populate_sysroot'><filename>do_populate_sysroot</filename></ulink> - tasks of the recipes on which the recipe - containing the tasks depends. - A sysroot exists for both the target and for the - native binaries, which run on the host system. - </para></listitem> - <listitem><para> - <emphasis><filename>do_configure</filename></emphasis>: - This task configures the source by enabling and - disabling any build-time and configuration options - for the software being built. - Configurations can come from the recipe itself as - well as from an inherited class. - Additionally, the software itself might configure - itself depending on the target for which it is - being built.</para> - - <para>The configurations handled by the - <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-configure'><filename>do_configure</filename></ulink> - task are specific to configurations for the source - code being built by the recipe.</para> - - <para>If you are using the - <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-autotools'><filename>autotools</filename></ulink> - class, you can add additional configuration options - by using the - <ulink url='&YOCTO_DOCS_REF_URL;#var-EXTRA_OECONF'><filename>EXTRA_OECONF</filename></ulink> - or - <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGECONFIG_CONFARGS'><filename>PACKAGECONFIG_CONFARGS</filename></ulink> - variables. - For information on how this variable works within - that class, see the - <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-autotools'><filename>autotools</filename></ulink> - class - <ulink url='&YOCTO_GIT_URL;/cgit/cgit.cgi/poky/tree/meta/classes/autotools.bbclass'>here</ulink>. - </para></listitem> - <listitem><para> - <emphasis><filename>do_compile</filename></emphasis>: - Once a configuration task has been satisfied, - BitBake compiles the source using the - <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-compile'><filename>do_compile</filename></ulink> - task. - Compilation occurs in the directory pointed to by - the - <ulink url='&YOCTO_DOCS_REF_URL;#var-B'><filename>B</filename></ulink> - variable. - Realize that the <filename>B</filename> directory - is, by default, the same as the - <ulink url='&YOCTO_DOCS_REF_URL;#var-S'><filename>S</filename></ulink> - directory. - </para></listitem> - <listitem><para> - <emphasis><filename>do_install</filename></emphasis>: - After compilation completes, BitBake executes the - <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-install'><filename>do_install</filename></ulink> - task. - This task copies files from the - <filename>B</filename> directory and places them - in a holding area pointed to by the - <ulink url='&YOCTO_DOCS_REF_URL;#var-D'><filename>D</filename></ulink> - variable. - Packaging occurs later using files from this - holding directory. - </para></listitem> - </itemizedlist> - </para> - </section> - - <section id='package-splitting-dev-environment'> - <title>Package Splitting</title> - - <para> - After source code is configured, compiled, and staged, the - build system analyzes the results and splits the output - into packages: - <imagedata fileref="figures/analysis-for-package-splitting.png" align="center" width="7in" depth="7in" /> - </para> - - <para> - The - <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package'><filename>do_package</filename></ulink> - and - <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-packagedata'><filename>do_packagedata</filename></ulink> - tasks combine to analyze the files found in the - <ulink url='&YOCTO_DOCS_REF_URL;#var-D'><filename>D</filename></ulink> - directory and split them into subsets based on available - packages and files. - Analysis involves the following as well as other items: - splitting out debugging symbols, looking at shared library - dependencies between packages, and looking at package - relationships. - </para> - - <para> - The <filename>do_packagedata</filename> task creates - package metadata based on the analysis such that the - build system can generate the final packages. - The - <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-populate_sysroot'><filename>do_populate_sysroot</filename></ulink> - task stages (copies) a subset of the files installed by - the - <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-install'><filename>do_install</filename></ulink> - task into the appropriate sysroot. - Working, staged, and intermediate results of the analysis - and package splitting process use several areas: - <itemizedlist> - <listitem><para> - <ulink url='&YOCTO_DOCS_REF_URL;#var-PKGD'><filename>PKGD</filename></ulink>: - The destination directory - (i.e. <filename>package</filename>) for packages - before they are split into individual packages. - </para></listitem> - <listitem><para> - <ulink url='&YOCTO_DOCS_REF_URL;#var-PKGDESTWORK'><filename>PKGDESTWORK</filename></ulink>: - A temporary work area (i.e. - <filename>pkgdata</filename>) used by the - <filename>do_package</filename> task to save - package metadata. - </para></listitem> - <listitem><para> - <ulink url='&YOCTO_DOCS_REF_URL;#var-PKGDEST'><filename>PKGDEST</filename></ulink>: - The parent directory (i.e. - <filename>packages-split</filename>) for packages - after they have been split. - </para></listitem> - <listitem><para> - <ulink url='&YOCTO_DOCS_REF_URL;#var-PKGDATA_DIR'><filename>PKGDATA_DIR</filename></ulink>: - A shared, global-state directory that holds - packaging metadata generated during the packaging - process. - The packaging process copies metadata from - <filename>PKGDESTWORK</filename> to the - <filename>PKGDATA_DIR</filename> area where it - becomes globally available. - </para></listitem> - <listitem><para> - <ulink url='&YOCTO_DOCS_REF_URL;#var-STAGING_DIR_HOST'><filename>STAGING_DIR_HOST</filename></ulink>: - The path for the sysroot for the system on which - a component is built to run (i.e. - <filename>recipe-sysroot</filename>). - </para></listitem> - <listitem><para> - <ulink url='&YOCTO_DOCS_REF_URL;#var-STAGING_DIR_NATIVE'><filename>STAGING_DIR_NATIVE</filename></ulink>: - The path for the sysroot used when building - components for the build host (i.e. - <filename>recipe-sysroot-native</filename>). - </para></listitem> - <listitem><para> - <ulink url='&YOCTO_DOCS_REF_URL;#var-STAGING_DIR_TARGET'><filename>STAGING_DIR_TARGET</filename></ulink>: - The path for the sysroot used when a component that - is built to execute on a system and it generates - code for yet another machine (e.g. cross-canadian - recipes). - </para></listitem> - </itemizedlist> - The - <ulink url='&YOCTO_DOCS_REF_URL;#var-FILES'><filename>FILES</filename></ulink> - variable defines the files that go into each package in - <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGES'><filename>PACKAGES</filename></ulink>. - If you want details on how this is accomplished, you can - look at - <ulink url='&YOCTO_GIT_URL;/cgit/cgit.cgi/poky/tree/meta/classes/package.bbclass'><filename>package.bbclass</filename></ulink>. - </para> - - <para> - Depending on the type of packages being created (RPM, DEB, - or IPK), the - <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package_write_deb'><filename>do_package_write_*</filename></ulink> - task creates the actual packages and places them in the - Package Feed area, which is - <filename>${TMPDIR}/deploy</filename>. - You can see the - "<link linkend='package-feeds-dev-environment'>Package Feeds</link>" - section for more detail on that part of the build process. - <note> - Support for creating feeds directly from the - <filename>deploy/*</filename> directories does not - exist. - Creating such feeds usually requires some kind of feed - maintenance mechanism that would upload the new - packages into an official package feed (e.g. the - Ångström distribution). - This functionality is highly distribution-specific - and thus is not provided out of the box. - </note> - </para> - </section> - - <section id='image-generation-dev-environment'> - <title>Image Generation</title> - - <para> - Once packages are split and stored in the Package Feeds - area, the build system uses BitBake to generate the root - filesystem image: - <imagedata fileref="figures/image-generation.png" align="center" width="7.5in" depth="7.5in" /> - </para> - - <para> - The image generation process consists of several stages and - depends on several tasks and variables. - The - <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-rootfs'><filename>do_rootfs</filename></ulink> - task creates the root filesystem (file and directory - structure) for an image. - This task uses several key variables to help create the - list of packages to actually install: - <itemizedlist> - <listitem><para> - <ulink url='&YOCTO_DOCS_REF_URL;#var-IMAGE_INSTALL'><filename>IMAGE_INSTALL</filename></ulink>: - Lists out the base set of packages from which to - install from the Package Feeds area. - </para></listitem> - <listitem><para> - <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGE_EXCLUDE'><filename>PACKAGE_EXCLUDE</filename></ulink>: - Specifies packages that should not be installed - into the image. - </para></listitem> - <listitem><para> - <ulink url='&YOCTO_DOCS_REF_URL;#var-IMAGE_FEATURES'><filename>IMAGE_FEATURES</filename></ulink>: - Specifies features to include in the image. - Most of these features map to additional packages - for installation. - </para></listitem> - <listitem><para> - <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGE_CLASSES'><filename>PACKAGE_CLASSES</filename></ulink>: - Specifies the package backend (e.g. RPM, DEB, or - IPK) to use and consequently helps determine where - to locate packages within the Package Feeds area. - </para></listitem> - <listitem><para> - <ulink url='&YOCTO_DOCS_REF_URL;#var-IMAGE_LINGUAS'><filename>IMAGE_LINGUAS</filename></ulink>: - Determines the language(s) for which additional - language support packages are installed. - </para></listitem> - <listitem><para> - <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGE_INSTALL'><filename>PACKAGE_INSTALL</filename></ulink>: - The final list of packages passed to the package - manager for installation into the image. - </para></listitem> - </itemizedlist> - </para> - - <para> - With - <ulink url='&YOCTO_DOCS_REF_URL;#var-IMAGE_ROOTFS'><filename>IMAGE_ROOTFS</filename></ulink> - pointing to the location of the filesystem under - construction and the <filename>PACKAGE_INSTALL</filename> - variable providing the final list of packages to install, - the root file system is created. - </para> - - <para> - Package installation is under control of the package - manager (e.g. dnf/rpm, opkg, or apt/dpkg) regardless of - whether or not package management is enabled for the - target. - At the end of the process, if package management is not - enabled for the target, the package manager's data files - are deleted from the root filesystem. - As part of the final stage of package installation, - post installation scripts that are part of the packages - are run. - Any scripts that fail to run on the build host are run on - the target when the target system is first booted. - If you are using a - <ulink url='&YOCTO_DOCS_DEV_URL;#creating-a-read-only-root-filesystem'>read-only root filesystem</ulink>, - all the post installation scripts must succeed on the - build host during the package installation phase since the - root filesystem on the target is read-only. - </para> - - <para> - The final stages of the <filename>do_rootfs</filename> task - handle post processing. - Post processing includes creation of a manifest file and - optimizations. - </para> - - <para> - The manifest file (<filename>.manifest</filename>) resides - in the same directory as the root filesystem image. - This file lists out, line-by-line, the installed packages. - The manifest file is useful for the - <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-testimage*'><filename>testimage</filename></ulink> - class, for example, to determine whether or not to run - specific tests. - See the - <ulink url='&YOCTO_DOCS_REF_URL;#var-IMAGE_MANIFEST'><filename>IMAGE_MANIFEST</filename></ulink> - variable for additional information. - </para> - - <para> - Optimizing processes that are run across the image include - <filename>mklibs</filename>, <filename>prelink</filename>, - and any other post-processing commands as defined by the - <ulink url='&YOCTO_DOCS_REF_URL;#var-ROOTFS_POSTPROCESS_COMMAND'><filename>ROOTFS_POSTPROCESS_COMMAND</filename></ulink> - variable. - The <filename>mklibs</filename> process optimizes the size - of the libraries, while the <filename>prelink</filename> - process optimizes the dynamic linking of shared libraries - to reduce start up time of executables. - </para> - - <para> - After the root filesystem is built, processing begins on - the image through the - <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-image'><filename>do_image</filename></ulink> - task. - The build system runs any pre-processing commands as - defined by the - <ulink url='&YOCTO_DOCS_REF_URL;#var-IMAGE_PREPROCESS_COMMAND'><filename>IMAGE_PREPROCESS_COMMAND</filename></ulink> - variable. - This variable specifies a list of functions to call before - the build system creates the final image output files. - </para> - - <para> - The build system dynamically creates - <filename>do_image_*</filename> tasks as needed, based - on the image types specified in the - <ulink url='&YOCTO_DOCS_REF_URL;#var-IMAGE_FSTYPES'><filename>IMAGE_FSTYPES</filename></ulink> - variable. - The process turns everything into an image file or a set of - image files and can compress the root filesystem image to - reduce the overall size of the image. - The formats used for the root filesystem depend on the - <filename>IMAGE_FSTYPES</filename> variable. - Compression depends on whether the formats support - compression. - </para> - - <para> - As an example, a dynamically created task when creating a - particular image <replaceable>type</replaceable> would - take the following form: - <literallayout class='monospaced'> - do_image_<replaceable>type</replaceable> - </literallayout> - So, if the <replaceable>type</replaceable> as specified by - the <filename>IMAGE_FSTYPES</filename> were - <filename>ext4</filename>, the dynamically generated task - would be as follows: - <literallayout class='monospaced'> - do_image_ext4 - </literallayout> - </para> - - <para> - The final task involved in image creation is the - <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-image-complete'><filename>do_image_complete</filename></ulink> - task. - This task completes the image by applying any image - post processing as defined through the - <ulink url='&YOCTO_DOCS_REF_URL;#var-IMAGE_POSTPROCESS_COMMAND'><filename>IMAGE_POSTPROCESS_COMMAND</filename></ulink> - variable. - The variable specifies a list of functions to call once the - build system has created the final image output files. - <note> - The entire image generation process is run under - <link linkend='fakeroot-and-pseudo'>Pseudo</link>. - Running under Pseudo ensures that the files in the - root filesystem have correct ownership. - </note> - </para> - </section> - - <section id='sdk-generation-dev-environment'> - <title>SDK Generation</title> - - <para> - The OpenEmbedded build system uses BitBake to generate the - Software Development Kit (SDK) installer scripts for both - the standard SDK and the extensible SDK (eSDK): - </para> - - <para> - <imagedata fileref="figures/sdk-generation.png" width="9in" align="center" /> - <note> - For more information on the cross-development toolchain - generation, see the - "<link linkend='cross-development-toolchain-generation'>Cross-Development Toolchain Generation</link>" - section. - For information on advantages gained when building a - cross-development toolchain using the - <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-populate_sdk'><filename>do_populate_sdk</filename></ulink> - task, see the - "<ulink url='&YOCTO_DOCS_SDK_URL;#sdk-building-an-sdk-installer'>Building an SDK Installer</ulink>" - section in the Yocto Project Application Development - and the Extensible Software Development Kit (eSDK) - manual. - </note> - </para> - - <para> - Like image generation, the SDK script process consists of - several stages and depends on many variables. - The - <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-populate_sdk'><filename>do_populate_sdk</filename></ulink> - and - <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-populate_sdk_ext'><filename>do_populate_sdk_ext</filename></ulink> - tasks use these key variables to help create the list of - packages to actually install. - For information on the variables listed in the figure, - see the - "<link linkend='sdk-dev-environment'>Application Development SDK</link>" - section. - </para> - - <para> - The <filename>do_populate_sdk</filename> task helps create - the standard SDK and handles two parts: a target part and a - host part. - The target part is the part built for the target hardware - and includes libraries and headers. - The host part is the part of the SDK that runs on the - <ulink url='&YOCTO_DOCS_REF_URL;#var-SDKMACHINE'><filename>SDKMACHINE</filename></ulink>. - </para> - - <para> - The <filename>do_populate_sdk_ext</filename> task helps - create the extensible SDK and handles host and target parts - differently than its counter part does for the standard SDK. - For the extensible SDK, the task encapsulates the build - system, which includes everything needed (host and target) - for the SDK. - </para> - - <para> - Regardless of the type of SDK being constructed, the - tasks perform some cleanup after which a cross-development - environment setup script and any needed configuration files - are created. - The final output is the Cross-development - toolchain installation script (<filename>.sh</filename> - file), which includes the environment setup script. - </para> - </section> - - <section id='stamp-files-and-the-rerunning-of-tasks'> - <title>Stamp Files and the Rerunning of Tasks</title> - - <para> - For each task that completes successfully, BitBake writes a - stamp file into the - <ulink url='&YOCTO_DOCS_REF_URL;#var-STAMPS_DIR'><filename>STAMPS_DIR</filename></ulink> - directory. - The beginning of the stamp file's filename is determined - by the - <ulink url='&YOCTO_DOCS_REF_URL;#var-STAMP'><filename>STAMP</filename></ulink> - variable, and the end of the name consists of the task's - name and current - <link linkend='overview-checksums'>input checksum</link>. - <note> - This naming scheme assumes that - <ulink url='&YOCTO_DOCS_BB_URL;#var-BB_SIGNATURE_HANDLER'><filename>BB_SIGNATURE_HANDLER</filename></ulink> - is "OEBasicHash", which is almost always the case in - current OpenEmbedded. - </note> - To determine if a task needs to be rerun, BitBake checks - if a stamp file with a matching input checksum exists - for the task. - If such a stamp file exists, the task's output is - assumed to exist and still be valid. - If the file does not exist, the task is rerun. - <note> - <para>The stamp mechanism is more general than the - shared state (sstate) cache mechanism described in the - "<link linkend='setscene-tasks-and-shared-state'>Setscene Tasks and Shared State</link>" - section. - BitBake avoids rerunning any task that has a valid - stamp file, not just tasks that can be accelerated - through the sstate cache.</para> - - <para>However, you should realize that stamp files only - serve as a marker that some work has been done and that - these files do not record task output. - The actual task output would usually be somewhere in - <ulink url='&YOCTO_DOCS_REF_URL;#var-TMPDIR'><filename>TMPDIR</filename></ulink> - (e.g. in some recipe's - <ulink url='&YOCTO_DOCS_REF_URL;#var-WORKDIR'><filename>WORKDIR</filename></ulink>.) - What the sstate cache mechanism adds is a way to cache - task output that can then be shared between build - machines.</para> - </note> - Since <filename>STAMPS_DIR</filename> is usually a - subdirectory of <filename>TMPDIR</filename>, removing - <filename>TMPDIR</filename> will also remove - <filename>STAMPS_DIR</filename>, which means tasks will - properly be rerun to repopulate - <filename>TMPDIR</filename>. - </para> - - <para> - If you want some task to always be considered "out of - date", you can mark it with the - <ulink url='&YOCTO_DOCS_BB_URL;#variable-flags'><filename>nostamp</filename></ulink> - varflag. - If some other task depends on such a task, then that - task will also always be considered out of date, which - might not be what you want. - </para> - - <para> - For details on how to view information about a task's - signature, see the - "<ulink url='&YOCTO_DOCS_DEV_URL;#dev-viewing-task-variable-dependencies'>Viewing Task Variable Dependencies</ulink>" - section in the Yocto Project Development Tasks Manual. - </para> - </section> - - <section id='setscene-tasks-and-shared-state'> - <title>Setscene Tasks and Shared State</title> - - <para> - The description of tasks so far assumes that BitBake needs - to build everything and no available prebuilt objects - exist. - BitBake does support skipping tasks if prebuilt objects are - available. - These objects are usually made available in the form of a - shared state (sstate) cache. - <note> - For information on variables affecting sstate, see the - <ulink url='&YOCTO_DOCS_REF_URL;#var-SSTATE_DIR'><filename>SSTATE_DIR</filename></ulink> - and - <ulink url='&YOCTO_DOCS_REF_URL;#var-SSTATE_MIRRORS'><filename>SSTATE_MIRRORS</filename></ulink> - variables. - </note> - </para> - - <para> - The idea of a setscene task (i.e - <filename>do_</filename><replaceable>taskname</replaceable><filename>_setscene</filename>) - is a version of the task where - instead of building something, BitBake can skip to the end - result and simply place a set of files into specific - locations as needed. - In some cases, it makes sense to have a setscene task - variant (e.g. generating package files in the - <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package_write_deb'><filename>do_package_write_*</filename></ulink> - task). - In other cases, it does not make sense (e.g. a - <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-patch'><filename>do_patch</filename></ulink> - task or a - <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-unpack'><filename>do_unpack</filename></ulink> - task) since the work involved would be equal to or greater - than the underlying task. - </para> - - <para> - In the build system, the common tasks that have setscene - variants are - <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package'><filename>do_package</filename></ulink>, - <filename>do_package_write_*</filename>, - <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-deploy'><filename>do_deploy</filename></ulink>, - <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-packagedata'><filename>do_packagedata</filename></ulink>, - and - <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-populate_sysroot'><filename>do_populate_sysroot</filename></ulink>. - Notice that these tasks represent most of the tasks whose - output is an end result. - </para> - - <para> - The build system has knowledge of the relationship between - these tasks and other preceding tasks. - For example, if BitBake runs - <filename>do_populate_sysroot_setscene</filename> for - something, it does not make sense to run any of the - <filename>do_fetch</filename>, - <filename>do_unpack</filename>, - <filename>do_patch</filename>, - <filename>do_configure</filename>, - <filename>do_compile</filename>, and - <filename>do_install</filename> tasks. - However, if <filename>do_package</filename> needs to be - run, BitBake needs to run those other tasks. - </para> - - <para> - It becomes more complicated if everything can come - from an sstate cache because some objects are simply - not required at all. - For example, you do not need a compiler or native tools, - such as quilt, if nothing exists to compile or patch. - If the <filename>do_package_write_*</filename> packages - are available from sstate, BitBake does not need the - <filename>do_package</filename> task data. - </para> - - <para> - To handle all these complexities, BitBake runs in two - phases. - The first is the "setscene" stage. - During this stage, BitBake first checks the sstate cache - for any targets it is planning to build. - BitBake does a fast check to see if the object exists - rather than a complete download. - If nothing exists, the second phase, which is the setscene - stage, completes and the main build proceeds. - </para> - - <para> - If objects are found in the sstate cache, the build system - works backwards from the end targets specified by the user. - For example, if an image is being built, the build system - first looks for the packages needed for that image and the - tools needed to construct an image. - If those are available, the compiler is not needed. - Thus, the compiler is not even downloaded. - If something was found to be unavailable, or the - download or setscene task fails, the build system then - tries to install dependencies, such as the compiler, from - the cache. - </para> - - <para> - The availability of objects in the sstate cache is - handled by the function specified by the - <ulink url='&YOCTO_DOCS_BB_URL;#var-BB_HASHCHECK_FUNCTION'><filename>BB_HASHCHECK_FUNCTION</filename></ulink> - variable and returns a list of available objects. - The function specified by the - <ulink url='&YOCTO_DOCS_BB_URL;#var-BB_SETSCENE_DEPVALID'><filename>BB_SETSCENE_DEPVALID</filename></ulink> - variable is the function that determines whether a given - dependency needs to be followed, and whether for any given - relationship the function needs to be passed. - The function returns a True or False value. - </para> - </section> - </section> - - <section id='images-dev-environment'> - <title>Images</title> - - <para> - The images produced by the build system are compressed forms - of the root filesystem and are ready to boot on a target - device. - You can see from the - <link linkend='general-workflow-figure'>general workflow figure</link> - that BitBake output, in part, consists of images. - This section takes a closer look at this output: - <imagedata fileref="figures/images.png" align="center" width="5.5in" depth="5.5in" /> - </para> - - <note> - For a list of example images that the Yocto Project provides, - see the - "<ulink url='&YOCTO_DOCS_REF_URL;#ref-images'>Images</ulink>" - chapter in the Yocto Project Reference Manual. - </note> - - <para> - The build process writes images out to the - <ulink url='&YOCTO_DOCS_REF_URL;#build-directory'>Build Directory</ulink> - inside the - <filename>tmp/deploy/images/<replaceable>machine</replaceable>/</filename> - folder as shown in the figure. - This folder contains any files expected to be loaded on the - target device. - The - <ulink url='&YOCTO_DOCS_REF_URL;#var-DEPLOY_DIR'><filename>DEPLOY_DIR</filename></ulink> - variable points to the <filename>deploy</filename> directory, - while the - <ulink url='&YOCTO_DOCS_REF_URL;#var-DEPLOY_DIR_IMAGE'><filename>DEPLOY_DIR_IMAGE</filename></ulink> - variable points to the appropriate directory containing images - for the current configuration. - <itemizedlist> - <listitem><para> - <replaceable>kernel-image</replaceable>: - A kernel binary file. - The - <ulink url='&YOCTO_DOCS_REF_URL;#var-KERNEL_IMAGETYPE'><filename>KERNEL_IMAGETYPE</filename></ulink> - variable determines the naming scheme for the - kernel image file. - Depending on this variable, the file could begin with - a variety of naming strings. - The - <filename>deploy/images/</filename><replaceable>machine</replaceable> - directory can contain multiple image files for the - machine. - </para></listitem> - <listitem><para> - <replaceable>root-filesystem-image</replaceable>: - Root filesystems for the target device (e.g. - <filename>*.ext3</filename> or - <filename>*.bz2</filename> files). - The - <ulink url='&YOCTO_DOCS_REF_URL;#var-IMAGE_FSTYPES'><filename>IMAGE_FSTYPES</filename></ulink> - variable determines the root filesystem image type. - The - <filename>deploy/images/</filename><replaceable>machine</replaceable> - directory can contain multiple root filesystems for the - machine. - </para></listitem> - <listitem><para> - <replaceable>kernel-modules</replaceable>: - Tarballs that contain all the modules built for the - kernel. - Kernel module tarballs exist for legacy purposes and - can be suppressed by setting the - <ulink url='&YOCTO_DOCS_REF_URL;#var-MODULE_TARBALL_DEPLOY'><filename>MODULE_TARBALL_DEPLOY</filename></ulink> - variable to "0". - The - <filename>deploy/images/</filename><replaceable>machine</replaceable> - directory can contain multiple kernel module tarballs - for the machine. - </para></listitem> - <listitem><para> - <replaceable>bootloaders</replaceable>: - If applicable to the target machine, bootloaders - supporting the image. - The <filename>deploy/images/</filename><replaceable>machine</replaceable> - directory can contain multiple bootloaders for the - machine. - </para></listitem> - <listitem><para> - <replaceable>symlinks</replaceable>: - The - <filename>deploy/images/</filename><replaceable>machine</replaceable> - folder contains a symbolic link that points to the - most recently built file for each machine. - These links might be useful for external scripts that - need to obtain the latest version of each file. - </para></listitem> - </itemizedlist> - </para> - </section> - - <section id='sdk-dev-environment'> - <title>Application Development SDK</title> - - <para> - In the - <link linkend='general-workflow-figure'>general workflow figure</link>, - the output labeled "Application Development SDK" represents an - SDK. - The SDK generation process differs depending on whether you - build an extensible SDK (e.g. - <filename>bitbake -c populate_sdk_ext</filename> <replaceable>imagename</replaceable>) - or a standard SDK (e.g. - <filename>bitbake -c populate_sdk</filename> <replaceable>imagename</replaceable>). - This section takes a closer look at this output: - <imagedata fileref="figures/sdk.png" align="center" width="9in" depth="7.25in" /> - </para> - - <para> - The specific form of this output is a set of files that - includes a self-extracting SDK installer - (<filename>*.sh</filename>), host and target manifest files, - and files used for SDK testing. - When the SDK installer file is run, it installs the SDK. - The SDK consists of a cross-development toolchain, a set of - libraries and headers, and an SDK environment setup script. - Running this installer essentially sets up your - cross-development environment. - You can think of the cross-toolchain as the "host" - part because it runs on the SDK machine. - You can think of the libraries and headers as the "target" - part because they are built for the target hardware. - The environment setup script is added so that you can - initialize the environment before using the tools. - </para> - - <note><title>Notes</title> - <itemizedlist> - <listitem><para> - The Yocto Project supports several methods by which - you can set up this cross-development environment. - These methods include downloading pre-built SDK - installers or building and installing your own SDK - installer. - </para></listitem> - <listitem><para> - For background information on cross-development - toolchains in the Yocto Project development - environment, see the - "<link linkend='cross-development-toolchain-generation'>Cross-Development Toolchain Generation</link>" - section. - </para></listitem> - <listitem><para> - For information on setting up a cross-development - environment, see the - <ulink url='&YOCTO_DOCS_SDK_URL;'>Yocto Project Application Development and the Extensible Software Development Kit (eSDK)</ulink> - manual. - </para></listitem> - </itemizedlist> - </note> - - <para> - All the output files for an SDK are written to the - <filename>deploy/sdk</filename> folder inside the - <ulink url='&YOCTO_DOCS_REF_URL;#build-directory'>Build Directory</ulink> - as shown in the previous figure. - Depending on the type of SDK, several variables exist that help - configure these files. - The following list shows the variables associated with an - extensible SDK: - <itemizedlist> - <listitem><para> - <ulink url='&YOCTO_DOCS_REF_URL;#var-DEPLOY_DIR'><filename>DEPLOY_DIR</filename></ulink>: - Points to the <filename>deploy</filename> directory. - </para></listitem> - <listitem><para> - <ulink url='&YOCTO_DOCS_REF_URL;#var-SDK_EXT_TYPE'><filename>SDK_EXT_TYPE</filename></ulink>: - Controls whether or not shared state artifacts are - copied into the extensible SDK. - By default, all required shared state artifacts are - copied into the SDK. - </para></listitem> - <listitem><para> - <ulink url='&YOCTO_DOCS_REF_URL;#var-SDK_INCLUDE_PKGDATA'><filename>SDK_INCLUDE_PKGDATA</filename></ulink>: - Specifies whether or not packagedata is included in the - extensible SDK for all recipes in the "world" target. - </para></listitem> - <listitem><para> - <ulink url='&YOCTO_DOCS_REF_URL;#var-SDK_INCLUDE_TOOLCHAIN'><filename>SDK_INCLUDE_TOOLCHAIN</filename></ulink>: - Specifies whether or not the toolchain is included - when building the extensible SDK. - </para></listitem> - <listitem><para> - <ulink url='&YOCTO_DOCS_REF_URL;#var-SDK_LOCAL_CONF_WHITELIST'><filename>SDK_LOCAL_CONF_WHITELIST</filename></ulink>: - A list of variables allowed through from the build - system configuration into the extensible SDK - configuration. - </para></listitem> - <listitem><para> - <ulink url='&YOCTO_DOCS_REF_URL;#var-SDK_LOCAL_CONF_BLACKLIST'><filename>SDK_LOCAL_CONF_BLACKLIST</filename></ulink>: - A list of variables not allowed through from the build - system configuration into the extensible SDK - configuration. - </para></listitem> - <listitem><para> - <ulink url='&YOCTO_DOCS_REF_URL;#var-SDK_INHERIT_BLACKLIST'><filename>SDK_INHERIT_BLACKLIST</filename></ulink>: - A list of classes to remove from the - <ulink url='&YOCTO_DOCS_REF_URL;#var-INHERIT'><filename>INHERIT</filename></ulink> - value globally within the extensible SDK configuration. - </para></listitem> - </itemizedlist> - This next list, shows the variables associated with a standard - SDK: - <itemizedlist> - <listitem><para> - <ulink url='&YOCTO_DOCS_REF_URL;#var-DEPLOY_DIR'><filename>DEPLOY_DIR</filename></ulink>: - Points to the <filename>deploy</filename> directory. - </para></listitem> - <listitem><para> - <ulink url='&YOCTO_DOCS_REF_URL;#var-SDKMACHINE'><filename>SDKMACHINE</filename></ulink>: - Specifies the architecture of the machine on which the - cross-development tools are run to create packages for - the target hardware. - </para></listitem> - <listitem><para> - <ulink url='&YOCTO_DOCS_REF_URL;#var-SDKIMAGE_FEATURES'><filename>SDKIMAGE_FEATURES</filename></ulink>: - Lists the features to include in the "target" part - of the SDK. - </para></listitem> - <listitem><para> - <ulink url='&YOCTO_DOCS_REF_URL;#var-TOOLCHAIN_HOST_TASK'><filename>TOOLCHAIN_HOST_TASK</filename></ulink>: - Lists packages that make up the host part of the SDK - (i.e. the part that runs on the - <filename>SDKMACHINE</filename>). - When you use - <filename>bitbake -c populate_sdk <replaceable>imagename</replaceable></filename> - to create the SDK, a set of default packages apply. - This variable allows you to add more packages. - </para></listitem> - <listitem><para> - <ulink url='&YOCTO_DOCS_REF_URL;#var-TOOLCHAIN_TARGET_TASK'><filename>TOOLCHAIN_TARGET_TASK</filename></ulink>: - Lists packages that make up the target part of the SDK - (i.e. the part built for the target hardware). - </para></listitem> - <listitem><para> - <ulink url='&YOCTO_DOCS_REF_URL;#var-SDKPATH'><filename>SDKPATH</filename></ulink>: - Defines the default SDK installation path offered by - the installation script. - </para></listitem> - <listitem><para> - <ulink url='&YOCTO_DOCS_REF_URL;#var-SDK_HOST_MANIFEST'><filename>SDK_HOST_MANIFEST</filename></ulink>: - Lists all the installed packages that make up the host - part of the SDK. - This variable also plays a minor role for extensible - SDK development as well. - However, it is mainly used for the standard SDK. - </para></listitem> - <listitem><para> - <ulink url='&YOCTO_DOCS_REF_URL;#var-SDK_TARGET_MANIFEST'><filename>SDK_TARGET_MANIFEST</filename></ulink>: - Lists all the installed packages that make up the - target part of the SDK. - This variable also plays a minor role for extensible - SDK development as well. - However, it is mainly used for the standard SDK. - </para></listitem> - </itemizedlist> - </para> - </section> - </section> - - <section id="cross-development-toolchain-generation"> - <title>Cross-Development Toolchain Generation</title> - - <para> - The Yocto Project does most of the work for you when it comes to - creating - <ulink url='&YOCTO_DOCS_REF_URL;#cross-development-toolchain'>cross-development toolchains</ulink>. - This section provides some technical background on how - cross-development toolchains are created and used. - For more information on toolchains, you can also see the - <ulink url='&YOCTO_DOCS_SDK_URL;'>Yocto Project Application Development and the Extensible Software Development Kit (eSDK)</ulink> - manual. - </para> - - <para> - In the Yocto Project development environment, cross-development - toolchains are used to build images and applications that run - on the target hardware. - With just a few commands, the OpenEmbedded build system creates - these necessary toolchains for you. - </para> - - <para> - The following figure shows a high-level build environment regarding - toolchain construction and use. - </para> - - <para> - <imagedata fileref="figures/cross-development-toolchains.png" width="8in" depth="6in" align="center" /> - </para> - - <para> - Most of the work occurs on the Build Host. - This is the machine used to build images and generally work within - the the Yocto Project environment. - When you run - <ulink url='&YOCTO_DOCS_REF_URL;#bitbake-term'>BitBake</ulink> - to create an image, the OpenEmbedded build system - uses the host <filename>gcc</filename> compiler to bootstrap a - cross-compiler named <filename>gcc-cross</filename>. - The <filename>gcc-cross</filename> compiler is what BitBake uses to - compile source files when creating the target image. - You can think of <filename>gcc-cross</filename> simply as an - automatically generated cross-compiler that is used internally - within BitBake only. - <note> - The extensible SDK does not use - <filename>gcc-cross-canadian</filename> since this SDK - ships a copy of the OpenEmbedded build system and the sysroot - within it contains <filename>gcc-cross</filename>. - </note> - </para> - - <para> - The chain of events that occurs when <filename>gcc-cross</filename> is - bootstrapped is as follows: - <literallayout class='monospaced'> - gcc -> binutils-cross -> gcc-cross-initial -> linux-libc-headers -> glibc-initial -> glibc -> gcc-cross -> gcc-runtime - </literallayout> - <itemizedlist> - <listitem><para> - <filename>gcc</filename>: - The build host's GNU Compiler Collection (GCC). - </para></listitem> - <listitem><para> - <filename>binutils-cross</filename>: - The bare minimum binary utilities needed in order to run - the <filename>gcc-cross-initial</filename> phase of the - bootstrap operation. - </para></listitem> - <listitem><para> - <filename>gcc-cross-initial</filename>: - An early stage of the bootstrap process for creating - the cross-compiler. - This stage builds enough of the <filename>gcc-cross</filename>, - the C library, and other pieces needed to finish building the - final cross-compiler in later stages. - This tool is a "native" package (i.e. it is designed to run on - the build host). - </para></listitem> - <listitem><para> - <filename>linux-libc-headers</filename>: - Headers needed for the cross-compiler. - </para></listitem> - <listitem><para> - <filename>glibc-initial</filename>: - An initial version of the Embedded GNU C Library - (GLIBC) needed to bootstrap <filename>glibc</filename>. - </para></listitem> - <listitem><para> - <filename>glibc</filename>: - The GNU C Library. - </para></listitem> - <listitem><para> - <filename>gcc-cross</filename>: - The final stage of the bootstrap process for the - cross-compiler. - This stage results in the actual cross-compiler that - BitBake uses when it builds an image for a targeted - device. - <note> - If you are replacing this cross compiler toolchain - with a custom version, you must replace - <filename>gcc-cross</filename>. - </note> - This tool is also a "native" package (i.e. it is - designed to run on the build host). - </para></listitem> - <listitem><para> - <filename>gcc-runtime</filename>: - Runtime libraries resulting from the toolchain bootstrapping - process. - This tool produces a binary that consists of the - runtime libraries need for the targeted device. - </para></listitem> - </itemizedlist> - </para> - - <para> - You can use the OpenEmbedded build system to build an installer for - the relocatable SDK used to develop applications. - When you run the installer, it installs the toolchain, which - contains the development tools (e.g., - <filename>gcc-cross-canadian</filename>, - <filename>binutils-cross-canadian</filename>, and other - <filename>nativesdk-*</filename> tools), - which are tools native to the SDK (i.e. native to - <ulink url='&YOCTO_DOCS_REF_URL;#var-SDK_ARCH'><filename>SDK_ARCH</filename></ulink>), - you need to cross-compile and test your software. - The figure shows the commands you use to easily build out this - toolchain. - This cross-development toolchain is built to execute on the - <ulink url='&YOCTO_DOCS_REF_URL;#var-SDKMACHINE'><filename>SDKMACHINE</filename></ulink>, - which might or might not be the same - machine as the Build Host. - <note> - If your target architecture is supported by the Yocto Project, - you can take advantage of pre-built images that ship with the - Yocto Project and already contain cross-development toolchain - installers. - </note> - </para> - - <para> - Here is the bootstrap process for the relocatable toolchain: - <literallayout class='monospaced'> - gcc -> binutils-crosssdk -> gcc-crosssdk-initial -> linux-libc-headers -> - glibc-initial -> nativesdk-glibc -> gcc-crosssdk -> gcc-cross-canadian - </literallayout> - <itemizedlist> - <listitem><para> - <filename>gcc</filename>: - The build host's GNU Compiler Collection (GCC). - </para></listitem> - <listitem><para> - <filename>binutils-crosssdk</filename>: - The bare minimum binary utilities needed in order to run - the <filename>gcc-crosssdk-initial</filename> phase of the - bootstrap operation. - </para></listitem> - <listitem><para> - <filename>gcc-crosssdk-initial</filename>: - An early stage of the bootstrap process for creating - the cross-compiler. - This stage builds enough of the - <filename>gcc-crosssdk</filename> and supporting pieces so that - the final stage of the bootstrap process can produce the - finished cross-compiler. - This tool is a "native" binary that runs on the build host. - </para></listitem> - <listitem><para> - <filename>linux-libc-headers</filename>: - Headers needed for the cross-compiler. - </para></listitem> - <listitem><para> - <filename>glibc-initial</filename>: - An initial version of the Embedded GLIBC needed to bootstrap - <filename>nativesdk-glibc</filename>. - </para></listitem> - <listitem><para> - <filename>nativesdk-glibc</filename>: - The Embedded GLIBC needed to bootstrap the - <filename>gcc-crosssdk</filename>. - </para></listitem> - <listitem><para> - <filename>gcc-crosssdk</filename>: - The final stage of the bootstrap process for the - relocatable cross-compiler. - The <filename>gcc-crosssdk</filename> is a transitory - compiler and never leaves the build host. - Its purpose is to help in the bootstrap process to create - the eventual <filename>gcc-cross-canadian</filename> - compiler, which is relocatable. - This tool is also a "native" package (i.e. it is - designed to run on the build host). - </para></listitem> - <listitem><para> - <filename>gcc-cross-canadian</filename>: - The final relocatable cross-compiler. - When run on the - <ulink url='&YOCTO_DOCS_REF_URL;#var-SDKMACHINE'><filename>SDKMACHINE</filename></ulink>, - this tool - produces executable code that runs on the target device. - Only one cross-canadian compiler is produced per architecture - since they can be targeted at different processor optimizations - using configurations passed to the compiler through the - compile commands. - This circumvents the need for multiple compilers and thus - reduces the size of the toolchains. - </para></listitem> - </itemizedlist> - </para> - - <note> - For information on advantages gained when building a - cross-development toolchain installer, see the - "<ulink url='&YOCTO_DOCS_SDK_URL;#sdk-building-an-sdk-installer'>Building an SDK Installer</ulink>" - appendix in the Yocto Project Application Development and the - Extensible Software Development Kit (eSDK) manual. - </note> - </section> - - <section id="shared-state-cache"> - <title>Shared State Cache</title> - - <para> - By design, the OpenEmbedded build system builds everything from - scratch unless - <ulink url='&YOCTO_DOCS_REF_URL;#bitbake-term'>BitBake</ulink> - can determine that parts do not need to be rebuilt. - Fundamentally, building from scratch is attractive as it means all - parts are built fresh and no possibility of stale data exists that - can cause problems. - When developers hit problems, they typically default back to - building from scratch so they have a know state from the - start. - </para> - - <para> - Building an image from scratch is both an advantage and a - disadvantage to the process. - As mentioned in the previous paragraph, building from scratch - ensures that everything is current and starts from a known state. - However, building from scratch also takes much longer as it - generally means rebuilding things that do not necessarily need - to be rebuilt. - </para> - - <para> - The Yocto Project implements shared state code that supports - incremental builds. - The implementation of the shared state code answers the following - questions that were fundamental roadblocks within the OpenEmbedded - incremental build support system: - <itemizedlist> - <listitem><para> - What pieces of the system have changed and what pieces have - not changed? - </para></listitem> - <listitem><para> - How are changed pieces of software removed and replaced? - </para></listitem> - <listitem><para> - How are pre-built components that do not need to be rebuilt - from scratch used when they are available? - </para></listitem> - </itemizedlist> - </para> - - <para> - For the first question, the build system detects changes in the - "inputs" to a given task by creating a checksum (or signature) of - the task's inputs. - If the checksum changes, the system assumes the inputs have changed - and the task needs to be rerun. - For the second question, the shared state (sstate) code tracks - which tasks add which output to the build process. - This means the output from a given task can be removed, upgraded - or otherwise manipulated. - The third question is partly addressed by the solution for the - second question assuming the build system can fetch the sstate - objects from remote locations and install them if they are deemed - to be valid. - <note><title>Notes</title> - <itemizedlist> - <listitem><para> - The build system does not maintain - <ulink url='&YOCTO_DOCS_REF_URL;#var-PR'><filename>PR</filename></ulink> - information as part of the shared state packages. - Consequently, considerations exist that affect - maintaining shared state feeds. - For information on how the build system works with - packages and can track incrementing - <filename>PR</filename> information, see the - "<ulink url='&YOCTO_DOCS_DEV_URL;#automatically-incrementing-a-binary-package-revision-number'>Automatically Incrementing a Binary Package Revision Number</ulink>" - section in the Yocto Project Development Tasks Manual. - </para></listitem> - <listitem><para> - The code in the build system that supports incremental - builds is not simple code. - For techniques that help you work around issues related - to shared state code, see the - "<ulink url='&YOCTO_DOCS_DEV_URL;#dev-viewing-metadata-used-to-create-the-input-signature-of-a-shared-state-task'>Viewing Metadata Used to Create the Input Signature of a Shared State Task</ulink>" - and - "<ulink url='&YOCTO_DOCS_DEV_URL;#dev-invalidating-shared-state-to-force-a-task-to-run'>Invalidating Shared State to Force a Task to Run</ulink>" - sections both in the Yocto Project Development Tasks - Manual. - </para></listitem> - </itemizedlist> - </note> - </para> - - <para> - The rest of this section goes into detail about the overall - incremental build architecture, the checksums (signatures), and - shared state. - </para> - - <section id='concepts-overall-architecture'> - <title>Overall Architecture</title> - - <para> - When determining what parts of the system need to be built, - BitBake works on a per-task basis rather than a per-recipe - basis. - You might wonder why using a per-task basis is preferred over - a per-recipe basis. - To help explain, consider having the IPK packaging backend - enabled and then switching to DEB. - In this case, the - <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-install'><filename>do_install</filename></ulink> - and - <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package'><filename>do_package</filename></ulink> - task outputs are still valid. - However, with a per-recipe approach, the build would not - include the <filename>.deb</filename> files. - Consequently, you would have to invalidate the whole build and - rerun it. - Rerunning everything is not the best solution. - Also, in this case, the core must be "taught" much about - specific tasks. - This methodology does not scale well and does not allow users - to easily add new tasks in layers or as external recipes - without touching the packaged-staging core. - </para> - </section> - - <section id='overview-checksums'> - <title>Checksums (Signatures)</title> - - <para> - The shared state code uses a checksum, which is a unique - signature of a task's inputs, to determine if a task needs to - be run again. - Because it is a change in a task's inputs that triggers a - rerun, the process needs to detect all the inputs to a given - task. - For shell tasks, this turns out to be fairly easy because - the build process generates a "run" shell script for each task - and it is possible to create a checksum that gives you a good - idea of when the task's data changes. - </para> - - <para> - To complicate the problem, there are things that should not be - included in the checksum. - First, there is the actual specific build path of a given - task - the - <ulink url='&YOCTO_DOCS_REF_URL;#var-WORKDIR'><filename>WORKDIR</filename></ulink>. - It does not matter if the work directory changes because it - should not affect the output for target packages. - Also, the build process has the objective of making native - or cross packages relocatable. - <note> - Both native and cross packages run on the - <ulink url='&YOCTO_DOCS_REF_URL;#hardware-build-system-term'>build host</ulink>. - However, cross packages generate output for the target - architecture. - </note> - The checksum therefore needs to exclude - <filename>WORKDIR</filename>. - The simplistic approach for excluding the work directory is to - set <filename>WORKDIR</filename> to some fixed value and - create the checksum for the "run" script. - </para> - - <para> - Another problem results from the "run" scripts containing - functions that might or might not get called. - The incremental build solution contains code that figures out - dependencies between shell functions. - This code is used to prune the "run" scripts down to the - minimum set, thereby alleviating this problem and making the - "run" scripts much more readable as a bonus. - </para> - - <para> - So far, solutions for shell scripts exist. - What about Python tasks? - The same approach applies even though these tasks are more - difficult. - The process needs to figure out what variables a Python - function accesses and what functions it calls. - Again, the incremental build solution contains code that first - figures out the variable and function dependencies, and then - creates a checksum for the data used as the input to the task. - </para> - - <para> - Like the <filename>WORKDIR</filename> case, situations exist - where dependencies should be ignored. - For these situations, you can instruct the build process to - ignore a dependency by using a line like the following: - <literallayout class='monospaced'> - PACKAGE_ARCHS[vardepsexclude] = "MACHINE" - </literallayout> - This example ensures that the - <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGE_ARCHS'><filename>PACKAGE_ARCHS</filename></ulink> - variable does not depend on the value of - <ulink url='&YOCTO_DOCS_REF_URL;#var-MACHINE'><filename>MACHINE</filename></ulink>, - even if it does reference it. - </para> - - <para> - Equally, there are cases where you need to add dependencies - BitBake is not able to find. - You can accomplish this by using a line like the following: - <literallayout class='monospaced'> - PACKAGE_ARCHS[vardeps] = "MACHINE" - </literallayout> - This example explicitly adds the <filename>MACHINE</filename> - variable as a dependency for - <filename>PACKAGE_ARCHS</filename>. - </para> - - <para> - As an example, consider a case with in-line Python where - BitBake is not able to figure out dependencies. - When running in debug mode (i.e. using - <filename>-DDD</filename>), BitBake produces output when it - discovers something for which it cannot figure out dependencies. - The Yocto Project team has currently not managed to cover - those dependencies in detail and is aware of the need to fix - this situation. - </para> - - <para> - Thus far, this section has limited discussion to the direct - inputs into a task. - Information based on direct inputs is referred to as the - "basehash" in the code. - However, the question of a task's indirect inputs still - exits - items already built and present in the - <ulink url='&YOCTO_DOCS_REF_URL;#build-directory'>Build Directory</ulink>. - The checksum (or signature) for a particular task needs to add - the hashes of all the tasks on which the particular task - depends. - Choosing which dependencies to add is a policy decision. - However, the effect is to generate a master checksum that - combines the basehash and the hashes of the task's - dependencies. - </para> - - <para> - At the code level, a variety of ways exist by which both the - basehash and the dependent task hashes can be influenced. - Within the BitBake configuration file, you can give BitBake - some extra information to help it construct the basehash. - The following statement effectively results in a list of - global variable dependency excludes (i.e. variables never - included in any checksum): - <literallayout class='monospaced'> - BB_HASHBASE_WHITELIST ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH DL_DIR \ - SSTATE_DIR THISDIR FILESEXTRAPATHS FILE_DIRNAME HOME LOGNAME SHELL TERM \ - USER FILESPATH STAGING_DIR_HOST STAGING_DIR_TARGET COREBASE PRSERV_HOST \ - PRSERV_DUMPDIR PRSERV_DUMPFILE PRSERV_LOCKDOWN PARALLEL_MAKE \ - CCACHE_DIR EXTERNAL_TOOLCHAIN CCACHE CCACHE_DISABLE LICENSE_PATH SDKPKGSUFFIX" - </literallayout> - The previous example excludes - <ulink url='&YOCTO_DOCS_REF_URL;#var-WORKDIR'><filename>WORKDIR</filename></ulink> - since that variable is actually constructed as a path within - <ulink url='&YOCTO_DOCS_REF_URL;#var-TMPDIR'><filename>TMPDIR</filename></ulink>, - which is on the whitelist. - </para> - - <para> - The rules for deciding which hashes of dependent tasks to - include through dependency chains are more complex and are - generally accomplished with a Python function. - The code in <filename>meta/lib/oe/sstatesig.py</filename> shows - two examples of this and also illustrates how you can insert - your own policy into the system if so desired. - This file defines the two basic signature generators - <ulink url='&YOCTO_DOCS_REF_URL;#oe-core'>OE-Core</ulink> - uses: "OEBasic" and "OEBasicHash". - By default, a dummy "noop" signature handler is enabled - in BitBake. - This means that behavior is unchanged from previous versions. - OE-Core uses the "OEBasicHash" signature handler by default - through this setting in the <filename>bitbake.conf</filename> - file: - <literallayout class='monospaced'> - BB_SIGNATURE_HANDLER ?= "OEBasicHash" - </literallayout> - The "OEBasicHash" <filename>BB_SIGNATURE_HANDLER</filename> - is the same as the "OEBasic" version but adds the task hash to - the - <link linkend='stamp-files-and-the-rerunning-of-tasks'>stamp files</link>. - This results in any metadata change that changes the task hash, - automatically causing the task to be run again. - This removes the need to bump - <ulink url='&YOCTO_DOCS_REF_URL;#var-PR'><filename>PR</filename></ulink> - values, and changes to metadata automatically ripple across - the build. - </para> - - <para> - It is also worth noting that the end result of these - signature generators is to make some dependency and hash - information available to the build. - This information includes: - <itemizedlist> - <listitem><para> - <filename>BB_BASEHASH_task-</filename><replaceable>taskname</replaceable>: - The base hashes for each task in the recipe. - </para></listitem> - <listitem><para> - <filename>BB_BASEHASH_</filename><replaceable>filename</replaceable><filename>:</filename><replaceable>taskname</replaceable>: - The base hashes for each dependent task. - </para></listitem> - <listitem><para> - <filename>BBHASHDEPS_</filename><replaceable>filename</replaceable><filename>:</filename><replaceable>taskname</replaceable>: - The task dependencies for each task. - </para></listitem> - <listitem><para> - <filename>BB_TASKHASH</filename>: - The hash of the currently running task. - </para></listitem> - </itemizedlist> - </para> - </section> - - <section id='shared-state'> - <title>Shared State</title> - - <para> - Checksums and dependencies, as discussed in the previous - section, solve half the problem of supporting a shared state. - The other half of the problem is being able to use checksum - information during the build and being able to reuse or rebuild - specific components. - </para> - - <para> - The - <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-sstate'><filename>sstate</filename></ulink> - class is a relatively generic implementation of how to - "capture" a snapshot of a given task. - The idea is that the build process does not care about the - source of a task's output. - Output could be freshly built or it could be downloaded and - unpacked from somewhere. - In other words, the build process does not need to worry about - its origin. - </para> - - <para> - Two types of output exist. - One type is just about creating a directory in - <ulink url='&YOCTO_DOCS_REF_URL;#var-WORKDIR'><filename>WORKDIR</filename></ulink>. - A good example is the output of either - <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-install'><filename>do_install</filename></ulink> - or - <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package'><filename>do_package</filename></ulink>. - The other type of output occurs when a set of data is merged - into a shared directory tree such as the sysroot. - </para> - - <para> - The Yocto Project team has tried to keep the details of the - implementation hidden in <filename>sstate</filename> class. - From a user's perspective, adding shared state wrapping to a - task is as simple as this - <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-deploy'><filename>do_deploy</filename></ulink> - example taken from the - <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-deploy'><filename>deploy</filename></ulink> - class: - <literallayout class='monospaced'> - DEPLOYDIR = "${WORKDIR}/deploy-${PN}" - SSTATETASKS += "do_deploy" - do_deploy[sstate-inputdirs] = "${DEPLOYDIR}" - do_deploy[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}" - - python do_deploy_setscene () { - sstate_setscene(d) - } - addtask do_deploy_setscene - do_deploy[dirs] = "${DEPLOYDIR} ${B}" - do_deploy[stamp-extra-info] = "${MACHINE_ARCH}" - </literallayout> - The following list explains the previous example: - <itemizedlist> - <listitem><para> - Adding "do_deploy" to <filename>SSTATETASKS</filename> - adds some required sstate-related processing, which is - implemented in the - <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-sstate'><filename>sstate</filename></ulink> - class, to before and after the - <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-deploy'><filename>do_deploy</filename></ulink> - task. - </para></listitem> - <listitem><para> - The - <filename>do_deploy[sstate-inputdirs] = "${DEPLOYDIR}"</filename> - declares that <filename>do_deploy</filename> places its - output in <filename>${DEPLOYDIR}</filename> when run - normally (i.e. when not using the sstate cache). - This output becomes the input to the shared state cache. - </para></listitem> - <listitem><para> - The - <filename>do_deploy[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}"</filename> - line causes the contents of the shared state cache to be - copied to <filename>${DEPLOY_DIR_IMAGE}</filename>. - <note> - If <filename>do_deploy</filename> is not already in - the shared state cache or if its input checksum - (signature) has changed from when the output was - cached, the task runs to populate the shared - state cache, after which the contents of the shared - state cache is copied to - <filename>${DEPLOY_DIR_IMAGE}</filename>. - If <filename>do_deploy</filename> is in the shared - state cache and its signature indicates that the - cached output is still valid (i.e. if no - relevant task inputs have changed), then the - contents of the shared state cache copies - directly to - <filename>${DEPLOY_DIR_IMAGE}</filename> by the - <filename>do_deploy_setscene</filename> task - instead, skipping the - <filename>do_deploy</filename> task. - </note> - </para></listitem> - <listitem><para> - The following task definition is glue logic needed to - make the previous settings effective: - <literallayout class='monospaced'> - python do_deploy_setscene () { - sstate_setscene(d) - } - addtask do_deploy_setscene - </literallayout> - <filename>sstate_setscene()</filename> takes the flags - above as input and accelerates the - <filename>do_deploy</filename> task through the - shared state cache if possible. - If the task was accelerated, - <filename>sstate_setscene()</filename> returns True. - Otherwise, it returns False, and the normal - <filename>do_deploy</filename> task runs. - For more information, see the - "<ulink url='&YOCTO_DOCS_BB_URL;#setscene'>setscene</ulink>" - section in the BitBake User Manual. - </para></listitem> - <listitem><para> - The <filename>do_deploy[dirs] = "${DEPLOYDIR} ${B}"</filename> - line creates <filename>${DEPLOYDIR}</filename> and - <filename>${B}</filename> before the - <filename>do_deploy</filename> task runs, and also sets - the current working directory of - <filename>do_deploy</filename> to - <filename>${B}</filename>. - For more information, see the - "<ulink url='&YOCTO_DOCS_BB_URL;#variable-flags'>Variable Flags</ulink>" - section in the BitBake User Manual. - <note> - In cases where - <filename>sstate-inputdirs</filename> and - <filename>sstate-outputdirs</filename> would be the - same, you can use - <filename>sstate-plaindirs</filename>. - For example, to preserve the - <filename>${PKGD}</filename> and - <filename>${PKGDEST}</filename> output from the - <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package'><filename>do_package</filename></ulink> - task, use the following: - <literallayout class='monospaced'> - do_package[sstate-plaindirs] = "${PKGD} ${PKGDEST}" - </literallayout> - </note> - </para></listitem> - <listitem><para> - The <filename>do_deploy[stamp-extra-info] = "${MACHINE_ARCH}"</filename> - line appends extra metadata to the - <link linkend='stamp-files-and-the-rerunning-of-tasks'>stamp file</link>. - In this case, the metadata makes the task specific - to a machine's architecture. - See - "<ulink url='&YOCTO_DOCS_BB_URL;#ref-bitbake-tasklist'>The Task List</ulink>" - section in the BitBake User Manual for more - information on the <filename>stamp-extra-info</filename> - flag. - </para></listitem> - <listitem><para> - <filename>sstate-inputdirs</filename> and - <filename>sstate-outputdirs</filename> can also be used - with multiple directories. - For example, the following declares - <filename>PKGDESTWORK</filename> and - <filename>SHLIBWORK</filename> as shared state - input directories, which populates the shared state - cache, and <filename>PKGDATA_DIR</filename> and - <filename>SHLIBSDIR</filename> as the corresponding - shared state output directories: - <literallayout class='monospaced'> - do_package[sstate-inputdirs] = "${PKGDESTWORK} ${SHLIBSWORKDIR}" - do_package[sstate-outputdirs] = "${PKGDATA_DIR} ${SHLIBSDIR}" - </literallayout> - </para></listitem> - <listitem><para> - These methods also include the ability to take a - lockfile when manipulating shared state directory - structures, for cases where file additions or removals - are sensitive: - <literallayout class='monospaced'> - do_package[sstate-lockfile] = "${PACKAGELOCK}" - </literallayout> - </para></listitem> - </itemizedlist> - </para> - - <para> - Behind the scenes, the shared state code works by looking in - <ulink url='&YOCTO_DOCS_REF_URL;#var-SSTATE_DIR'><filename>SSTATE_DIR</filename></ulink> - and - <ulink url='&YOCTO_DOCS_REF_URL;#var-SSTATE_MIRRORS'><filename>SSTATE_MIRRORS</filename></ulink> - for shared state files. - Here is an example: - <literallayout class='monospaced'> - SSTATE_MIRRORS ?= "\ - file://.* http://someserver.tld/share/sstate/PATH;downloadfilename=PATH \n \ - file://.* file:///some/local/dir/sstate/PATH" - </literallayout> - <note> - The shared state directory - (<filename>SSTATE_DIR</filename>) is organized into - two-character subdirectories, where the subdirectory - names are based on the first two characters of the hash. - If the shared state directory structure for a mirror has the - same structure as <filename>SSTATE_DIR</filename>, you must - specify "PATH" as part of the URI to enable the build system - to map to the appropriate subdirectory. - </note> - </para> - - <para> - The shared state package validity can be detected just by - looking at the filename since the filename contains the task - checksum (or signature) as described earlier in this section. - If a valid shared state package is found, the build process - downloads it and uses it to accelerate the task. - </para> - - <para> - The build processes use the <filename>*_setscene</filename> - tasks for the task acceleration phase. - BitBake goes through this phase before the main execution - code and tries to accelerate any tasks for which it can find - shared state packages. - If a shared state package for a task is available, the - shared state package is used. - This means the task and any tasks on which it is dependent - are not executed. - </para> - - <para> - As a real world example, the aim is when building an IPK-based - image, only the - <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package_write_ipk'><filename>do_package_write_ipk</filename></ulink> - tasks would have their shared state packages fetched and - extracted. - Since the sysroot is not used, it would never get extracted. - This is another reason why a task-based approach is preferred - over a recipe-based approach, which would have to install the - output from every task. - </para> - </section> - </section> - - <section id='automatically-added-runtime-dependencies'> - <title>Automatically Added Runtime Dependencies</title> - - <para> - The OpenEmbedded build system automatically adds common types of - runtime dependencies between packages, which means that you do not - need to explicitly declare the packages using - <ulink url='&YOCTO_DOCS_REF_URL;#var-RDEPENDS'><filename>RDEPENDS</filename></ulink>. - Three automatic mechanisms exist (<filename>shlibdeps</filename>, - <filename>pcdeps</filename>, and <filename>depchains</filename>) - that handle shared libraries, package configuration (pkg-config) - modules, and <filename>-dev</filename> and - <filename>-dbg</filename> packages, respectively. - For other types of runtime dependencies, you must manually declare - the dependencies. - <itemizedlist> - <listitem><para> - <filename>shlibdeps</filename>: - During the - <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package'><filename>do_package</filename></ulink> - task of each recipe, all shared libraries installed by the - recipe are located. - For each shared library, the package that contains the - shared library is registered as providing the shared - library. - More specifically, the package is registered as providing - the - <ulink url='https://en.wikipedia.org/wiki/Soname'>soname</ulink> - of the library. - The resulting shared-library-to-package mapping - is saved globally in - <ulink url='&YOCTO_DOCS_REF_URL;#var-PKGDATA_DIR'><filename>PKGDATA_DIR</filename></ulink> - by the - <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-packagedata'><filename>do_packagedata</filename></ulink> - task.</para> - - <para>Simultaneously, all executables and shared libraries - installed by the recipe are inspected to see what shared - libraries they link against. - For each shared library dependency that is found, - <filename>PKGDATA_DIR</filename> is queried to - see if some package (likely from a different recipe) - contains the shared library. - If such a package is found, a runtime dependency is added - from the package that depends on the shared library to the - package that contains the library.</para> - - <para>The automatically added runtime dependency also - includes a version restriction. - This version restriction specifies that at least the - current version of the package that provides the shared - library must be used, as if - "<replaceable>package</replaceable> (>= <replaceable>version</replaceable>)" - had been added to <filename>RDEPENDS</filename>. - This forces an upgrade of the package containing the shared - library when installing the package that depends on the - library, if needed.</para> - - <para>If you want to avoid a package being registered as - providing a particular shared library (e.g. because the library - is for internal use only), then add the library to - <ulink url='&YOCTO_DOCS_REF_URL;#var-PRIVATE_LIBS'><filename>PRIVATE_LIBS</filename></ulink> - inside the package's recipe. - </para></listitem> - <listitem><para> - <filename>pcdeps</filename>: - During the <filename>do_package</filename> task of each - recipe, all pkg-config modules - (<filename>*.pc</filename> files) installed by the recipe - are located. - For each module, the package that contains the module is - registered as providing the module. - The resulting module-to-package mapping is saved globally in - <filename>PKGDATA_DIR</filename> by the - <filename>do_packagedata</filename> task.</para> - - <para>Simultaneously, all pkg-config modules installed by - the recipe are inspected to see what other pkg-config - modules they depend on. - A module is seen as depending on another module if it - contains a "Requires:" line that specifies the other module. - For each module dependency, - <filename>PKGDATA_DIR</filename> is queried to see if some - package contains the module. - If such a package is found, a runtime dependency is added - from the package that depends on the module to the package - that contains the module. - <note> - The <filename>pcdeps</filename> mechanism most often - infers dependencies between <filename>-dev</filename> - packages. - </note> - </para></listitem> - <listitem><para> - <filename>depchains</filename>: - If a package <filename>foo</filename> depends on a package - <filename>bar</filename>, then <filename>foo-dev</filename> - and <filename>foo-dbg</filename> are also made to depend on - <filename>bar-dev</filename> and - <filename>bar-dbg</filename>, respectively. - Taking the <filename>-dev</filename> packages as an - example, the <filename>bar-dev</filename> package might - provide headers and shared library symlinks needed by - <filename>foo-dev</filename>, which shows the need - for a dependency between the packages.</para> - - <para>The dependencies added by - <filename>depchains</filename> are in the form of - <ulink url='&YOCTO_DOCS_REF_URL;#var-RRECOMMENDS'><filename>RRECOMMENDS</filename></ulink>. - <note> - By default, <filename>foo-dev</filename> also has an - <filename>RDEPENDS</filename>-style dependency on - <filename>foo</filename>, because the default value of - <filename>RDEPENDS_${PN}-dev</filename> (set in - <filename>bitbake.conf</filename>) includes - "${PN}". - </note></para> - - <para>To ensure that the dependency chain is never broken, - <filename>-dev</filename> and <filename>-dbg</filename> - packages are always generated by default, even if the - packages turn out to be empty. - See the - <ulink url='&YOCTO_DOCS_REF_URL;#var-ALLOW_EMPTY'><filename>ALLOW_EMPTY</filename></ulink> - variable for more information. - </para></listitem> - </itemizedlist> - </para> - - <para> - The <filename>do_package</filename> task depends on the - <filename>do_packagedata</filename> task of each recipe in - <ulink url='&YOCTO_DOCS_REF_URL;#var-DEPENDS'><filename>DEPENDS</filename></ulink> - through use of a - <filename>[</filename><ulink url='&YOCTO_DOCS_BB_URL;#variable-flags'><filename>deptask</filename></ulink><filename>]</filename> - declaration, which guarantees that the required - shared-library/module-to-package mapping information will be available - when needed as long as <filename>DEPENDS</filename> has been - correctly set. - </para> - </section> - - <section id='fakeroot-and-pseudo'> - <title>Fakeroot and Pseudo</title> - - <para> - Some tasks are easier to implement when allowed to perform certain - operations that are normally reserved for the root user (e.g. - <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-install'><filename>do_install</filename></ulink>, - <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package_write_deb'><filename>do_package_write*</filename></ulink>, - <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-rootfs'><filename>do_rootfs</filename></ulink>, - and - <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-image'><filename>do_image*</filename></ulink>). - For example, the <filename>do_install</filename> task benefits - from being able to set the UID and GID of installed files to - arbitrary values. - </para> - - <para> - One approach to allowing tasks to perform root-only operations - would be to require - <ulink url='&YOCTO_DOCS_REF_URL;#bitbake-term'>BitBake</ulink> - to run as root. - However, this method is cumbersome and has security issues. - The approach that is actually used is to run tasks that benefit - from root privileges in a "fake" root environment. - Within this environment, the task and its child processes believe - that they are running as the root user, and see an internally - consistent view of the filesystem. - As long as generating the final output (e.g. a package or an image) - does not require root privileges, the fact that some earlier - steps ran in a fake root environment does not cause problems. - </para> - - <para> - The capability to run tasks in a fake root environment is known as - "<ulink url='http://man.he.net/man1/fakeroot'>fakeroot</ulink>", - which is derived from the BitBake keyword/variable - flag that requests a fake root environment for a task. - </para> - - <para> - In the - <ulink url='&YOCTO_DOCS_REF_URL;#build-system-term'>OpenEmbedded build system</ulink>, - the program that implements fakeroot is known as - <ulink url='https://www.yoctoproject.org/software-item/pseudo/'>Pseudo</ulink>. - Pseudo overrides system calls by using the environment variable - <filename>LD_PRELOAD</filename>, which results in the illusion - of running as root. - To keep track of "fake" file ownership and permissions resulting - from operations that require root permissions, Pseudo uses - an SQLite 3 database. - This database is stored in - <filename>${</filename><ulink url='&YOCTO_DOCS_REF_URL;#var-WORKDIR'><filename>WORKDIR</filename></ulink><filename>}/pseudo/files.db</filename> - for individual recipes. - Storing the database in a file as opposed to in memory - gives persistence between tasks and builds, which is not - accomplished using fakeroot. - <note><title>Caution</title> - If you add your own task that manipulates the same files or - directories as a fakeroot task, then that task also needs to - run under fakeroot. - Otherwise, the task cannot run root-only operations, and - cannot see the fake file ownership and permissions set by the - other task. - You need to also add a dependency on - <filename>virtual/fakeroot-native:do_populate_sysroot</filename>, - giving the following: - <literallayout class='monospaced'> - fakeroot do_mytask () { - ... - } - do_mytask[depends] += "virtual/fakeroot-native:do_populate_sysroot" - </literallayout> - </note> - For more information, see the - <ulink url='&YOCTO_DOCS_BB_URL;#var-FAKEROOT'><filename>FAKEROOT*</filename></ulink> - variables in the BitBake User Manual. - You can also reference the - "<ulink url='https://github.com/wrpseudo/pseudo/wiki/WhyNotFakeroot'>Why Not Fakeroot?</ulink>" - article for background information on Fakeroot and Pseudo. - </para> - </section> -</chapter> -<!-- -vim: expandtab tw=80 ts=4 ---> |