Bake 64-bit raspberryPI3 images with Yocto/OpenEmbedded

Overview

RaspberryPI3 SBC and CM come with cortex-a53 based SOC which is 64-bit capable and uses ARM-v8 architecture. Originally RaspberryPI 3 was released with 32bit support alone which is backward compatible with RaspberryPI 2. However, over period of last couple of years a lot of development has gone into enabling 64-bit on ARM in community,  Eric Anholt’s VC4 drivers have matured on 64bit kernels, various userspace packages have been ported by Linaro and other communities.  We have put efforts in integrating all these efforts for full system using OpenEmbedded/Yocto framework. The support is fairly new and therefore there might be some issues still lurking, Here we will go through the steps for building full Graphical images.

Build Setup

Since the support is new, its recommended to use Rocko 2.4 release or newer

mkdir rpi-yocto; cd rpi-yocto
git clone git://git.yoctoproject.org/poky
git clone git://git.openembedded.org/meta-openembedded
git clone git://git.yoctoproject.org/meta-raspberrypi
git clone git://github.com/96boards/meta-96boards

source poky/oe-init-build-env rpi64-build

bitbake-layers add-layer ../meta-raspberrypi
bitbake-layers add-layer ../meta-96boards
bitbake-layers add-layer ../meta-openembedded/meta-oe
bitbake-layers add-layer ../meta-openembedded/meta-python
bitbake-layers add-layer ../meta-openembedded/meta-perl
bitbake-layers add-layer ../meta-openembedded/meta-multimedia 
bitbake-layers add-layer ../meta-openembedded/meta-networking
bitbake-layers add-layer ../meta-openembedded/meta-gnome
bitbake-layers add-layer ../meta-openembedded/meta-xfce

Edit conf/local.conf  to set machine

MACHINE = "raspberrypi3-64"

Build Sato Image ( Yocto Project Reference GUI image )

bitbake core-image-sato

Build XFCE image

If you intend to build a full desktop system like XFCE then build core-image-minimal-xfce

bitbake core-image-minimal-xfce

Prepare SD-Card

sudo dd if=tmp/deploy/images/raspberrypi3-64/core-image-sato-raspberrypi3.rpi-sdimg of=/dev/sdX

where X is the letter a,b,c which your build machine would have mounted the Micro-SD card on you can check that with

dmesg | tail -10

Resize SD-Card

Add in local.conf

CORE_IMAGE_EXTRA_INSTALL_append = " 96boards-tools "

build the image again and flash it to SD-Card, this will expand the rootfs to span over complete SD card.

Using MUSL C library

Musl is a light weight alternative to glibc which is fully supported in OpenEmbedded-core and Yocto project. In order to use musl instead of glibc add ( in local.conf )

TCLIBC = "musl"

Using Clang Compiler

Clang is supported via a layer of its own, if you intend to use clang to compile the packages, you have to add meta-clang to layer mix.

cd rpi-yocto; git clone git://github.com/kraj/meta-clang

bitbake-layers add-layer ../meta-clang

If clang should be used as default compiler then add following in local.conf

TOOLCHAIN ?= "clang"

Please note that there is a small number of packages which still would use gcc to build. e.g. for core-image-sato we still build following packages with gcc

  • pixman
  • linux-kernel
  • glibc( if you use it)
  • elfutils
  • tcp-wrappers
  • busybox

Status

What works

  • Ethernet
  • Graphics
  • USB
  • Keyboard/mouse
  • HDMI

 

TO-DO

  • Validate bluetooth
  • Validate on board WiFi
  • Video playback
  • Sound

Need Help ?

Please ask questions on IRC or mailing lists

irc.freenode.net#yocto
irc.freenode.net#oe

Mailing lists

yocto@yoctoproject.org
openembedded-devel@lists.openembedded.org

We need Open Source ISA and methods for Chip design and Hardware

The open source phenomenon has swept the software industry in past few decades, and Linux based Operating Systems have taken the software world by storm. The Linux foundation has published studies where the worth of Linux is estimated at billions of dollars, but to me as an individual, it is priceless. High quality open source Compilers and tools for various language runtimes are readily available with source code. Ready to install OS distributions are a common thing. We have learnt that if we want to build something sustainable and scalable, open source development is a proven methodology and has become the primary choice.

Large organizations have embraced open source software tools and technologies, and have started open source software offices for helping the organization with adopting open source. Research and development budget is shifting an increasing portion to do open source software development. If we look into pie-charts of software composition in products, the open source share is even-increasing piece year after year. We have become more comfortable in collaborating in open ways where it is serving the common good, and keep the competition to where value is added.

Open source has certainly helped software in so many ways. Open collaboration alone has reduced the cost, and time to market. There has never been so much knowledge shared freely available to community. We have arrived at golden era of software development and it’s not limited to only a privileged few. The Internet has enabled us to share technology across the world.  The next whiz-kid sitting in remote corner of the world is not at disadvantage anymore. We should all be proud of this achievement. Recently I met with youngsters from the Caribbean, who were learning ins and out of full software stack on the Raspberry PI.  The real software revolution has just begun.

Can we apply Open Source techniques to chip & hardware design

My interaction with kids from the Caribbean turned into an interesting discussion when we started to talk about graphics stacks and how the low level software was coming in the way of their learning since they did not have source code for it.

Hardware is viewed as a brick, cast in stone, because it comes from a chosen few and they get to decide what to make. Secondly, customization costs are enormous, design cycles are looong. Today, all someone needs is a computer to unleash the software design ingenuity. This is not the case for hardware. Someone who has a great idea to solve a problem for the next Internet of things (IoT) wave does not have tools and techniques available as of today to implement it in hardware. Today, different kinds of demands for computing is taking us beyond general purpose computing, to application specific computer design.  We will need different hardware to solve new problems where existing system architectures are proving to be inefficient. There is a need for tools and techniques to do hardware design on par with the cost and pace of software. Software has blazed the trail using Open Source techniques, and hardware design will follow these footsteps.

As an example, IoT will require more chips and different kind of chips than ever at a scale that has never been seen before. The sensors will be required to be doing more computing yet get cheaper in price. ESP8266 drastically low priced WiFi chip with full TCP/IP stack has been adopted widely and has changed the low-cost WiFi space. Bluetooth LE mesh is another promising connectivity technology for IoT, and I imagine we are not far from designing a 25 cents BLE mesh chip low cost solution and transform the BLE markets forever. However, this will require a completely different way of thinking about design and manufacturing, since incremental design and manufacturing  process will not cut the bill.

Architectures (like RISC-V) which are completely open source from the ground up, and available to academia and industry, are the first steps in realizing this dream for hardware design.  These technologies are  accessible to everyone from schools to the semiconductor industry. In the future I hope a whiz-kid from remote corners of the world will be able to design the hardware with same access and tooling we have for software design today. That is going to unleash innovation like never before. When an optimized solution consists of efficient software running on efficient hardware, yet sharing the common good across hardware and software, we can really present technology as a compelling solution to fix enormous problems that humanity is facing. There is tremendous interest developing around Open Instruction Set Architectures ( ISA).  An openly licenced ISA would enable anyone to design a custom CPU yet be able to reuse the software ecosystem since the ISA would remain common. This would also make the instruction set architecture robust as it will be developed with constraints from different applications. This added excitement  is a good sign, back when Linux was in infancy, things were exactly same.

                            Hardware designers, Welcome On-board !!

PS: A big Thanks to Cliff Brake for adding data-points and reviews

Running LTP on musl based systems

Introduction

LTP is freely available testsuite used for validating the reliability and stability of Linux systems. It has functionality, stress as well as regression tests. Thus far it has been mainly run on Linux systems running glibc for C library, and its natural that there will be areas in the project which would have assumptions on glibc coded in. We have ported LTP to run on MUSL based systems. The patches are still being upstreamed. A recent series is ported to LTP mailing list

http://lists.linux.it/pipermail/ltp/2016-July/002259.html

Patches

LTP patches are currently residing on a branch ‘oe/master’ on git://github.com/kraj/ltp repository, There still are few more patches in addition to above patchset ported to mailing list.

Building and Running LTP

Prerequisites

LTP depends on ftw library and we need to install external ftw on musl based systems

On OpenEmbedded based systems just add “libfts-dev” to IMAGE_INSTALL

Building

git clone git://github.com/kraj/ltp -b oe/master
cd ltp
make autotools
./configure --without-numa --without-tirpc --with-power-management-testsuite --with-realtime-testsuite

make LIBC=musl -j8 all
make LIBC=musl install
rm /opt/ltp/testcases/bin/getdents01

getdents01 runs into infinite loop and fills the disk. This would install the ltp test suite under /opt/ltp

Running

hst=`hostname`
/opt/ltp/runltp -p -l "${hst}-$(date +%FT%T).log" -o "${hst}-$(date +%FT%T).out"

Conclusion

We can build and execute the testsuite on raspberryPI3, however the results dont look good. Many tests fail. Here are results of a run on raspberrypi3 running OpenEmbedded/Yocto

https://uclibc.org/~kraj/LTP_RUN_ON-2016_07_22-04h_19m_30s.log

 

 

 

Multilib GCC on ArchLinux

Some packages like LuaJIT uses native compiler to generate source files which are then used during cross compiling it for another architecture. Now if the target architecture has 32bit word length ( which most of them in embedded space are ) then it expects a matching bitness in compiler on build host. So we need to to use HOST_CC = “gcc -m32” on 64-bit ArchLinux build hosts. This requires multilib to be enabled on ArchLinux

 sudo vim /etc/pacman.conf

add or uncomment

[multilib]
Include = /etc/pacman.d/mirrorlist

Now update and install multilib gcc

sudo pacman -Syu gcc-multilib gcc-libs-multilib

To check the install is done correctly create hello.c with following content

 

#include <stdio.h>
int main() {
  printf("Voila!!\n");
  return 0;
}
$ gcc -m32 ~/hello.c

$ ./a.out

should say

“Voila!!”

Clang based Cross SDK with OpenEmbedded Framework

Introduction

Clang based toolchain can be used to compile large portions of packages in OpenEmbedded Framework, in this article we will cover the generating and using OpenEmbedded SDKs based on Clang, This SDK will also have the original cross gcc based compilers cohabiting with clang

Generating SDK

Building image base SDK in OpenEmbedded Framework is a simple process like its building any other image or component. One can generate a SDK corresponding to final image that will be shipped on target. This gets development headers and libraries from all the dependencies packaged into the SDK installer, this SDK then can be installed and used for stand alone application development.

Preparing OpenEmbedded Workspace

$ git clone https://github.com/openembedded/openembedded-core.git
$ cd openembedded-core
$ git clone https://github.com/openembedded/bitbake.git
$ git clone https://github.com/kraj/meta-clang.git
$ . ./oe-init-build-env

Add meta-clang to conf/bblayers.conf e.g.

# LAYER_CONF_VERSION is increased each time build/conf/bblayers.conf
# changes incompatibly
LCONF_VERSION = "6"
BBPATH = "${TOPDIR}"
BBFILES ?= ""
BBLAYERS ?= " \
 /home/kraj/openembedded-core/meta-clang \
 /home/kraj/openembedded-core/meta \
 "
BBLAYERS_NON_REMOVABLE ?= " \
 /home/kraj/openembedded-core/meta \
 "

Select qemuarm64 for MACHINE in conf/local.conf, you need to uncomment the line marked in red

#
# Machine Selection
#
# You need to select a specific machine to target the build with. There are a selection
# of emulated machines available which can boot and run in the QEMU emulator:
#
#MACHINE ?= "qemuarm"
MACHINE ?= "qemuarm64"
#MACHINE ?= "qemumips"
#MACHINE ?= "qemuppc"
#MACHINE ?= "qemux86"
#MACHINE ?= "qemux86-64"
#

Build SDK

$ bitbake -cpopulate_sdk core-image-minimal

Install SDK

SDK self-installer will be in deploy are under e.g. tmp-glibc/deploy/sdk/

$ ./tmp-glibc/deploy/sdk/oecore-x86_64-aarch64-toolchain-nodistro.0.sh

OpenEmbedded SDK installer version nodistro.0
===========================================================
Enter target directory for SDK (default: /usr/local/oecore-x86_64):
You are about to install the SDK to “/usr/local/oecore-x86_64”. Proceed[Y/n]?
Extracting SDK…………………………….done
Setting it up…done
SDK has been successfully set up and is ready to be used.
Each time you wish to use the SDK in a new shell session, you need to source the environment setup script e.g.
$ . /usr/local/oecore-x86_64/environment-setup-aarch64-oe-linux

Using SDK

The environment setup needs to be done in every shell where the SDK is to be used

$ . /usr/local/oecore-x86_64/environment-setup-aarch64-oe-linux

At this point the SDK is ready to be used for compiling applications as well as kernel. Since the SDK has both gcc and clang in it a new set of variables is introduced to make it easy to use clang

CLANGCC – C cross compiler

CLANGCXX – C++ cross compiler

Example 1 – Build musl C library

The example lists compiling musl C library using the cross SDK but this can be used for anything including kernel and application

$ git clone git://git.musl-libc.org/musl
$ cd musl
$ ./configure --host=aarch64-oe-linux CC=${CLANGCC}
$ make -j

Example 2 – Building llvmlinux kernel

$ git clone git://git.linuxfoundation.org/llvmlinux/kernel.git llvmlinux
$ cd llvmlinux
$ make ARCH=arm64 CC=${CLANGCC} LDFLAGS="" defconfig
$ make ARCH=arm64 CC=${CLANGCC} LDFLAGS="" -j vmlinux

If the kernel for aarch64 won’t build, then hey what have you been waiting for ? you have everything go fix it !!!

Notes

I have not provided prebuilt SDKs, if there is enough intrest for such SDKs, I would consider building the SDK self-installs and host them somewhere, contact me if you are interested.

Example is covering aarch64 but the SDKs are not limited to aarch64, one can build for x86,x86_64, ppc, mips, arm in same fashion. All one would need is to changes is MACHINE selection in conf/local.conf

Using Clang with OpenEmbedded/Yocto Project

Introduction

Clang is C/C++/ObjC  frontend for LLVM compiler suite,  supporting several architectures now. We have added support for clang into OpenEmbedded using a standalone layer

https://github.com/kraj/meta-clang

This layer requires OpenEmbedded-core layer as dependency

Configuring meta-clang

git clone git://github.com/openembedded/openembedded-core
cd openembedded-core
git clone git://github.com/openembedded/bitbake
git clone git://github.com/kraj/meta-clang

Setup

. ./oe-init-build-env

Edit conf/bblayer.conf to add meta-clang

# LAYER_CONF_VERSION is increased each time build/conf/bblayers.conf
# changes incompatibly
LCONF_VERSION = "6"

BBPATH = "${TOPDIR}"
BBFILES ?= ""

BBLAYERS ?= " \
 /a/wheezy/home/kraj/work/openembedded-core/meta-clang \
 /a/wheezy/home/kraj/work/openembedded-core/meta \
 "

BBLAYERS_NON_REMOVABLE ?= " \
 /a/wheezy/home/kraj/work/openembedded-core/meta \
 "

 

Selecting Default system Compiler

When meta-clang is added to layers it switches the default cross compiler for the system to be clang naturally. This selection is made by a variable which is assigned to choose clang if nothing else is selected

TOOLCHAIN ??= "clang"

One could start using GCC as default system compiler by setting TOOLCHAIN variable to ‘gcc’

TOOLCHAIN = "gcc"

This selection is done in configuration metadata files like conf/local.conf

Per-recipe ( package ) Compiler selection

When using clang, not all components are compilable yet, hence we have to sometimes choose gcc for certain package recipe. In order to do it, you have to write a .bbappend in meta-clang, as an example below bbappend shows how we select gcc for building glibc

$ cat recipes-excluded/nonclangable/glibc_%.bbappend
TOOLCHAIN = "gcc"

Current Status

We can build/boot core-image-minimal for qemux86 and raspberryPi machines, of course that includes few components which are still compiled using gcc. But at this point, we are ready to port the packages to compile with clang

Compiling Kernel with Clang

For kernel we still use gcc, to start using clang for kernel we would require to override KERNEL_CC variable

kernel-arch.bbclass:KERNEL_CC = "${CCACHE}${HOST_PREFIX}gcc ${HOST_CC_KERNEL_ARCH} -fuse-ld=bfd"

To use clang

kernel-arch.bbclass:KERNEL_CC = "${CCACHE}${HOST_PREFIX}clang ${HOST_CC_KERNEL_ARCH} -fuse-ld=bfd"

This can also be done in kernel recipe too.

KERNEL_CC_forcevariable = "${CCACHE}${HOST_PREFIX}clang ${HOST_CC_KERNEL_ARCH} -fuse-ld=bfd"

Compiler runtime

We currently use gcc runtime by default with clang as well, meta-clang does have recipes for compiler-rt and libc++ which can be used to replace libgcc and libstdc++, at application level this selection can be made to depend on compiler-rt and libc++

SDK

We can generate OpenEmbedded SDK which will have both gcc and clang based cross compilers

bitbake -cpopulate_sdk core-image-minimal

Installing the SDK should be done something like this Installing the Toolchain¶

To use clang in SDK one should use CLANGCC and CLANGCXX environment variables to reference the clang compilers