HERO Documentation

This page explains how to build and configure the host Linux system for HERO using PetaLinux/Yocto. This HOWTO is for HERO based on the Xilinx Zynq UltraScale+ MPSoC platform.

Overview

The specific board used for this HOWTO is the UltraSOM+ TE0808 module (using a XCZU9EG-1FFVC900E chip, with 2 GiB DDR4 RAM) on the UltraITX+ Baseboard TEBF0808 from Trenz Electronic. However, it should also work for similar boards as the ZCU102 from Xilinx. In fact, it should be even simpler to get it running on the Xilinx board as all the required board files come with the PetaLinux installation.

Build system

The following build system was used for this HOWTO:

CentOS 7 workstation
Xilinx Vivado 2017.2 with SDK and PetaLinux
Bash shell

Required input files

The following files are required for this HOWTO:

board support files
bigpulp-zux.hdf hardware definition file exported from Vivado
bigpulp_zux_top.bit bitstream file generated by Vivado
the helper scripts provided in the hero-support GIT repository

Generated output files

The following two output files will be generated in this HOWTO:

image.ub FIT image consisting of
- Linux kernel image
- device-tree blob (DTB)
- Root filesystem image
BOOT.BIN consisting of
- First-stage boot loader (FSBL) image
- U-Boot boot loader
- FPGA bitstream
- Power management unit (PMU) firmware
- ARM Trusted Firmware image

To boot the platform, these files need to be copied to the boot partition of the SD card.

NOTE: PetaLinux automatically regenerates the individual components from the sources when generating the two output files. Consequently, any changes manually applied to any of these components, for example to tweak a specific node in the device tree, are not propagated to the final images. The purpose of this tutorial is to show how and where to add properly add such customizations in the PetaLinux flow.

NOTE: With the default PetaLinux configuration used also by this tutorial, U-Boot loads the root filesystem image from the SD card into memory at startup. The filesystem is then read- and writeable only in memory (RAMDISK). No changes are written back to the image on the SD card. From a point of usability, this is not ideal. However, there are several important advantages:

No filesystem inconsistency and corruption can occur, for example due to system crashes, SD card wearout.
Very fast startup
Reduced SD card wearout

It is however recommended to setup either a second data partition on the SD card or and network file system (NFS) share and mount this for permanent storage. In the case of the latter, filesystem consistency is handled by the NFS server, which helps to avoid data corruption. This tutorial shows how to include the required scripts to (automatically) mount an NFS share at startup.

PetaLinux Project Setup

Change to a scratch directory on your workstation, source your preferred Vivado environment, and create a new PetaLinux project by entering the following command:

petalinux-create -t project -n $TARGET_NAME --template zynqMP

where $TARGET_NAME is the host name the target FPGA board will have. The newly created $TARGET_NAME directory will be your workspace directory.

Change to the workspace directory and copy the sourceme.sh file from the linux/petalinux/scripts/ directory in the hero-support repository into that directory. Edit sourceme.sh such that

BOARD is the same FPGA board type designator you have used to build the bigPULP bitstream,
TARGET_NAME is the host name defined above,
WORKSPACE_PATH is the absolute path of your workspace directory,
VIVADO_EXPORT_PATH is the path of the directory containing the hardware definition file and the bitstream, and
SD_BOOT_PARTITION is the mount point (on your workstation) of the SD card partition that will contain the boot files for the target system.
If you will be using a shared NFS directory, set SCP_TARGET_MACHINE and SCP_TARGET_PATH accordingly.

Then source sourceme.sh.

Configure the PetaLinux project by importing the bigPULP hardware definition file:

petalinux-config --get-hw-description=${VIVADO_EXPORT_PATH}/

This command will open PetaLinux top-level menuconfig. Navigate to the Subsystem AUTO Hardware Settings ---> Ethernet Settings ---> menu and configure the MAC address of your board. Save and exit menuconfig.

Finally, create a new directory images/linux/ in the workspace directory and copy all files in linux/petalinux/scripts/ of hero-support except sourceme.sh into images/linux/. These files will be used during the later steps.

To re-open menuconfig at a later point, simply type

petalinux-config

Xilinx Linux Kernel

By default, PetaLinux compiles the Linux kernel from sources delivered as part of the PetaLinux installation, which cannot be modified.

To switch to a different kernel version or modify the sources, PetaLinux must be configured to use external kernel sources. How to achieve this is shown in the following. If you do not need to modify the kernel sources, you can skip this step and directly jump to Device Tree Generation.

To configure PetaLinux for using external kernel sources, navigate to the components folder inside your workspace directory

cd $WORKSPACE_PATH/components

and create a new folder

mkdir ext_sources
cd ext_sources

and enter it.

Clone the kernel sources for Xilinx Linux

git clone git://github.com/Xilinx/linux-xlnx.git

Rename this newly created folder and enter it

mv linux-xlnx linux-xlnx-plnx
cd linux-xlnx-plnx

Checkout the proper version of the repository

git checkout tags/xilinx-v2017.2

Now, you can navigate back to your workspace directory. Type

petalinux-config

to open menuconfig. Go to the Linux Components Selection ---> linux-kernel ---> and then select ext-local-src. Select External linux-kernel local source settings ---> and type

${TOPDIR}/../components/ext_sources/linux-xlnx-plnx

for External linux-kernel local source path. Save and exit menuconfig.

Then, open menuconfig for the kernel configuration

petalinux-config -c kernel

and navigate to General setup ---> and enter

-xilinx-v2017.2

for Local version - append to kernel release. Save and exit menuconfig.

To enable shared virtual memory (SVM) between bigPULP and the ARM host using the ARM SMMU, the driver of this device needs to be patched. To this end, type

cd components/ext_sources/linux-xlnx-plnx/drivers/iommu

and apply the patches found in linux/petalinux/kernel inside the hero-support repository:

patch < arm-smmu.patch
patch < io-pgtable-arm.patch

The kernel will automatically be compiled when building the image.ub later. To manually re-compile the kernel, you can execute

petalinux-build -c kernel

Device Tree Generation

By default, PetaLinux generates the device tree based on sources provided with the installation (deployed to components/plnx_workspace/device-tree-generation) as well as information provided in the hardware defintion file (bigpulp-zux.hdf in our case).

For HERO, we need to provide a custom device node for PULP as well as modifications to other nodes, e.g., to reserve some DRAM as contiguous memory, to allocate system DMA channels and an interrupt line, and to enable the SMMU for SVM. The proper way to do this is through the file system-user.dtsi located in project-spec/meta-user/recipes-bsp/device-tree/files/. If you do not need to modify the device tree, you can skip this step and directly jump to Yocto Root Filesystem.

Copy the provided system-user.dtsi from linux/petalinux/device-tree inside the hero-support repository to project-spec/meta-user/recipes-bsp/device-tree/files/.

The device tree will automatically be compiled when building the image.ub later. To manually re-compile the device tree, you can execute

 petalinux-build -c device-tree

The compiled device tree blob system.dtb can be found in images/linux in the workspace directory. To convert back and forth between the blob and the source, you can use the provided scripts dtb_to_dts.sh and dts_to_dtb.sh, respectively.

Yocto Root Filesystem

For HERO, we make the following adjustments to the Yocto root filesystem.

Enable the NTPD

To enable the synchronization of the system time (useful for generating logs), we install the network time protocol deamon (NTPD). By default, the NTPD is not enabled by PetaLinux but it is part of BusyBox, which itself is part of Yocto.

To enable the NTPD, type

cd project-spec/meta-plnx-generated/recipes-core/busybox

to enter the BusyBox recipe folder. From the folder linux/petalinux/ntpd/ of the hero-support GIT repository, copy

the Yocto recipe busybox_1.%.bbappend to the current folder, and
the BusyBox config snippet ntpd.cfg to the files folder inside the current folder.

Include Custom Files

To include a set of custom files, such as SSH keys, startup scripts etc., in the root filesystem image (RAMDISK), a new PetaLinux app is created. We call this app “custom-files”.

In your workspace directory, execute

petalinux-create -t apps -n custom-files --enable

to create this app and enable its build as part of the root filesystem.

Then, type

cd project-spec/meta-user/recipes-apps/custom-files

to enter the app folder. From the folder linux/petalinux/custom-files/ of the hero-support GIT repository, copy

the Yocto recipe custom-files.bb to the app folder, and
the actual scripts and custom files from inside files to the files folder inside the app folder.

The custom-files.bb recipe tells PetaLinux to copy the stuff inside files to the root filesystem image. One of these scripts (/etc/rc5.d/S25time) is used to synchronize the system time using NTPD. Modify this script synchronize with a time server of your choice.

The host keys of the SSH server are not part of the Git repository. You need to generate them on your workstation and add them to the files directory of the custom-files application. Change to the files directory and execute

mkdir -p etc/ssh
ssh-keygen -t rsa -b 4096 -N '' -C $TARGET_NAME -f etc/ssh/ssh_host_rsa_key
ssh-keygen -t rsa -b 4096 -N '' -C $TARGET_NAME -f etc/ssh/ssh_host_key
ssh-keygen -t dsa -N '' -C $TARGET_NAME -f etc/ssh/ssh_host_dsa_key
ssh-keygen -t ecdsa -b 521 -N '' -C $TARGET_NAME -f etc/ssh/ssh_host_ecdsa_key
ssh-keygen -t ed25519 -N '' -C $TARGET_NAME -f etc/ssh/ssh_host_ed25519_key

NOTE: Sometimes, PetaLinux accidentally removes the actual custom files located inside files when generating the root filesystem image. It then complains about not finding them anymore. Therefore, it is worth to create a backup of the files including the SSH keys to copy them back if needed.

The root filesystem will automatically be generated when building the image.ub later. To manually trigger the build of the root filesystem execute

 petalinux-build -c rootfs

The generated root filesystem rootfs.cpio.gz can be found in images/linux in the workspace directory. Use the script unpack_rootfs.sh to unpack and inspect the image. This way, you can check whether the custom files are indeed included in the image.

Generating the Boot Files

In your workspace directory, execute

petalinux-build

to let PetaLinux build the boot loaders, firmware, kernel, device tree and root filesystem. Some of the generated files will be copied automatically to images/linux. In case the PetaLinux claims that system.hdf cannot be found, execute

cp project-spec/hw-description/system.hdf build/tmp/work/aarch64-xilinx-linux/external-hdf/1.0-r0/git/plnx_aarch64/.

To generate the actual images to be deployed on the SD card, three more steps are required.

FIT Image

To let PetaLinux build the FIT image image.ub (out of the kernel image, the device tree blob system.dtb and the root filesystem image rootfs.cpio), execute

petalinux-package --image

ARM Trusted Firmware

For some reason, the Zynq MPSoC does by default not allow accelerators placed in the PL to access memory coherent with the caches of the ARM host, despite being connected through the coherent ACE-Lite ports. In particular the cache coherent interconnect (CCI-400) rejects any transcations originating from the PL which are marked as shareable. To work around this issue, the ARM Trusted Firmware needs to be patched. More precisely, we need to enable access to the CCI-400 configuration registers by non-secure software, such that we can later reconfigure the CCI-400 using a driver module.

To this end, change to

cd ${WORKSPACE_PATH}/components/ext_sources

and execute

git clone https://github.com/Xilinx/arm-trusted-firmware.git

Then, change into arm-trusted-firmware and checkout the proper version of the repository

cd arm-trusted-firmware
git checkout tags/xilinx-v2017.2

navigate to the folder containing the CCI driver

cd drivers/arm/cci

and apply the patch found in linux/petalinux/arm-trusted-firmware inside the hero-support repository:

patch < cci.patch

Change to ${WORKSPACE_PATH}/images/linux and run

./compile_arm-trusted_firmware.sh

to build the patched firmware.

BOOT.BIN

To generate the BOOT.BIN containing FSBL, U-Boot image, PMU firmware, FPGA bitstream, and ARM Trusted Firmware, go to ${WORKSPACE_PATH}/images/linux and execute the script

./boot_gen.sh

Finally, the images image.ub and BOOT.BIN can be deployed to the SD card by executing

./copy_to_sd_card.sh

Dynamic FPGA Reconfiguration

To deploy a different FPGA bitstream without having to fiddle around with the SD card, PetaLinux allows to reconfigure the FPGA while the system is running. By default, the corresponding drivers are already installed on the system.

To generate the .bin file required by the dynamic reconfiguration infrastructure out of the bitstream and copy this file to the running target system, execute the script

./bit_to_bin.sh

located in images/linux.

Then, connect to the running system and execute the script

./update_bitstream.sh

NOTE: It is not safe to load a different bitstream at runtime if this changes the interfaces between the ARM host and the FPGA.

PetaLinux