.. _docs/totalcompute/tc2/user-guide: User Guide ========== .. contents:: Notice ------ The Total Compute 2022 (TC2) software stack uses bash scripts to build a Board Support Package (BSP) and a choice of Buildroot Linux distribution or Android. Prerequisites ------------- These instructions assume that: * Your host PC is running Ubuntu Linux 20.04; * You are running the provided scripts in a ``bash`` shell environment. To get the latest repo tool from google, run the following commands: :: mkdir -p ~/bin curl https://storage.googleapis.com/git-repo-downloads/repo > ~/bin/repo chmod a+x ~/bin/repo export PATH=~/bin:$PATH To build and run Android, the minimum requirements for the host machine can be found at https://source.android.com/setup/build/requirements, These include: * At least 250GB of free disk space to check out the code and an extra 150 GB to build it. If you conduct multiple builds, you need additional space. * At least 32 GB of available RAM/swap. * Git configured properly using "git config" otherwise it may throw error while fetching the code. To install and allow access to docker, run the following command: :: sudo apt install docker.io # ensure docker service is properly started and running sudo systemctl restart docker sudo chmod 777 /var/run/docker.sock NOTE: to manage Docker as a non-root user, run the following command: :: sudo groupadd docker sudo usermod -aG docker $USER newgrp docker Download the source code and build ------------------------------------ There are two distros supported in the TC2 software stack: * Buildroot (a minimal distro containing Busybox); * Android. Download the source code ######################## Create a new folder that will be your workspace, which will henceforth be referred to as ```` in these instructions. :: mkdir cd export TC2_RELEASE=refs/tags/TC2-2023.04.21 To sync Buildroot source code, run the following repo command: :: repo init -u https://gitlab.arm.com/arm-reference-solutions/arm-reference-solutions-manifest -m tc2.xml -b ${TC2_RELEASE} -g bsp repo sync -j `nproc` --fetch-submodules To sync Android source code, run the following repo command: :: repo init -u https://gitlab.arm.com/arm-reference-solutions/arm-reference-solutions-manifest -m tc2.xml -b ${TC2_RELEASE} -g android repo sync -j `nproc` --fetch-submodules NOTE: synchronization of the Android code from Google servers may fail due to connection problems and/or to an enforced rate limit related with the maximum number of concurrent fetching jobs. The previous commands assume that the maximum number of jobs concurrently fetching code will be a perfect match of the number of CPU cores available, which should work fine most of the times. If experiencing constant errors on consecutive fetch code attempts, please do consider deleting your entire workspace (which will ensure a clean of the support ``.repo`` folder containing the previously partial fetched files), by running the command ``cd .. ; rm -rf `` and repeat the previous commands listed in this section to recreate the workspace (optionally, also reducing the number of jobs, for example to a maximum of 4, by adopting the following command ``repo sync -j 4 --fetch-submodules``). Once the previous process finishes, the current ```` should have the following structure: * ``build-scripts/``: the components build scripts * ``run-scripts/``: scripts to run the FVP * ``src/``: each component's git repository Initial Setup ############# The setup includes two parts: 1. setup a docker image; 2. setup the environmet to build TC images. Setting up a docker image involves pulling the prebuilt docker image from a docker registry. If that fails, it will build a local docker image. To setup a docker image, patch the components, install the toolchains and build tools, run the following listed commands. For the Buildroot build: :: export PLATFORM=tc2 export FILESYSTEM=buildroot ./setup.sh For the Android build with hardware rendering: :: export PLATFORM=tc2 export FILESYSTEM=android-fvp export TC_GPU=true export TC_TARGET_FLAVOR=fvp export GPU_DDK_REPO= export GPU_DDK_VERSION=r40p0_01eac0 export LM_LICENSE_FILE= export ARM_PRODUCT_DEF= export ARMLMD_LICENSE_FILE= export ANDROID_TEST_EXAMPLES= export ARMCLANG_TOOL= ./setup.sh For the Android build with software rendering: :: export PLATFORM=tc2 export TC_GPU=false export TC_TARGET_FLAVOR=fvp export FILESYSTEM=android-fvp ./setup.sh The various tools will be installed in the ``tools/`` directory at the root of the workspace. To build Android with Android Verified Boot (AVB) enabled, run the following command to enable the corresponding flag in addition to any of the two previous Android command variants (please note that this needs to be run before running `./setup.sh`): :: export AVB=true NOTES: * If building the TC2 software stack for more than one target, please ensure you run a clean build between each different build to avoid setup/building errors (refer to the next section "*More about the build system*" for command usage examples on how to do this). * If running ``repo sync`` again is needed at some point, then the ``setup.sh`` script also needs to be run again, as repo sync can discard the patches. * Most builds will be done in parallel using all the available cores by default. To change this number, run ``export PARALLELISM=`` Build options ############# Android OS build **************** * tc2_fvp with ``TC_GPU=false`` : this supports Android display with swiftshader (software rendering); * tc2_fvp with ``TC_GPU=true`` : this supports Android display with Mali GPU (hardware rendering). GPU DDK source code is available only to licensee partners (please contact support@arm.com). The Android images can be built with or without authentication enabled using Android Verified Boot (AVB) through the use of the ``-a`` option. AVB build is done in userdebug mode and takes a longer time to boot as the images are verified. This option does not influence the way the system boots, rather it adds an optional sanity check on the prerequisite images. Android based stack takes considerable time to build, so start the build and go grab a cup of coffee! Build command ############# To build the whole TC2 software stack, simply run: :: ./run_docker.sh ./build-all.sh build Once the previous process finishes, the current ```` should have the following structure: * build files are stored in ``/build-scripts/output/tmp_build/``; * final images will be placed in ``/build-script/output/deploy/``. More about the build system ########################### The ``build-all.sh`` script will build all the components, but each component has its own script, allowing it to be built, cleaned and deployed separately. All scripts support the ``build``, ``clean``, ``deploy``, ``patch`` commands. ``build-all.sh`` also supports ``all``, which performs a clean followed by a rebuild of all the stack. For example, to build, deploy, and clean SCP, run: :: ./run_docker.sh ./build-scp.sh build ./run_docker.sh ./build-scp.sh deploy ./run_docker.sh ./build-scp.sh clean The platform and filesystem used should be defined as described previously, but they can also be specified as the following example: :: ./run_docker.sh ./build-all.sh -p $PLATFORM -f $FILESYSTEM -t $TC_TARGET_FLAVOR -g $TC_GPU build Build Components and its dependencies ##################################### A new dependency to a component can be added in the form of ``$component=$dependency`` in the ``dependencies.txt`` file To build a component and rebuild those components that depend on it, run: :: ./run_docker.sh ./$filename build with_reqs Those options work for all the ``build-*.sh`` scripts. Provided components ------------------- Firmware Components ################### Trusted Firmware-A ****************** Based on `Trusted Firmware-A `__ +--------+------------------------------------------------------------------------------------------------------------+ | Script | /build-scripts/build-tfa.sh | +--------+------------------------------------------------------------------------------------------------------------+ | Files | * /build-scripts/output/deploy/tc2/bl1-tc.bin | | | * /build-scripts/output/deploy/tc2/fip-tc.bin | +--------+------------------------------------------------------------------------------------------------------------+ System Control Processor (SCP) ****************************** Based on `SCP Firmware `__ +--------+------------------------------------------------------------------------------------------------+ | Script | /build-scripts/build-scp.sh | +--------+------------------------------------------------------------------------------------------------+ | Files | * /build-scripts/output/deploy/tc2/scp_ramfw.bin | | | * /build-scripts/output/deploy/tc2/scp_romfw.bin | +--------+------------------------------------------------------------------------------------------------+ U-Boot ****** Based on `U-Boot gitlab `__ +--------+---------------------------------------------------------------------------------------+ | Script | /build-scripts/build-u-boot.sh | +--------+---------------------------------------------------------------------------------------+ | Files | * /build-scripts/output/deploy/tc2/u-boot.bin | +--------+---------------------------------------------------------------------------------------+ Hafnium ******* Based on `Hafnium `__ +--------+--------------------------------------------------------------------------------------+ | Script | /build-scripts/build-hafnium.sh | +--------+--------------------------------------------------------------------------------------+ | Files | * /build-scripts/output/deploy/tc2/hafnium.bin | +--------+--------------------------------------------------------------------------------------+ OP-TEE ****** Based on `OP-TEE `__ +--------+------------------------------------------------------------------------------------------+ | Script | /build-scripts/build-optee-os.sh | +--------+------------------------------------------------------------------------------------------+ | Files | * /build-scripts/output/tmp_build/tfa_sp/tee-pager_v2.bin | +--------+------------------------------------------------------------------------------------------+ S-EL0 trusted-services ********************** Based on `Trusted Services `__ +--------+-----------------------------------------------------------------------------------------------+ | Script | /build-scripts/build-trusted-services.sh | +--------+-----------------------------------------------------------------------------------------------+ | Files | * /build-scripts/output/tmp_build/tfa_sp/crypto-sp.bin | | | * /build-scripts/output/tmp_build/tfa_sp/internal-trusted-storage.bin | +--------+-----------------------------------------------------------------------------------------------+ Linux ***** The component responsible for building a 5.15 version of the Android Common kernel (`ACK `__). +--------+-----------------------------------------------------------------------------------------------+ | Script | /build-scripts/build-linux.sh | +--------+-----------------------------------------------------------------------------------------------+ | Files | * /build-scripts/output/deploy/tc2/Image | +--------+-----------------------------------------------------------------------------------------------+ Trusty ****** Based on `Trusty `__ +--------+---------------------------------------------------------------------------+ | Script | /build-scripts/build-trusty.sh | +--------+---------------------------------------------------------------------------+ | Files | * /build-scripts/output/deploy/tc2/lk.bin | +--------+---------------------------------------------------------------------------+ Distributions ############# Buildroot Linux distro ********************** The layer is based on the `buildroot `__ Linux distribution. The provided distribution is based on BusyBox and built using glibc. +--------+-------------------------------------------------------------------------------------------------+ | Script | /build-scripts/build-buildroot.sh | +--------+-------------------------------------------------------------------------------------------------+ | Files | * /build-scripts/output/deploy/tc2/tc-fitImage.bin | +--------+-------------------------------------------------------------------------------------------------+ Android ******* +--------+-------------------------------------------------------------------------+ | Script | /build-scripts/build-android.sh | +--------+-------------------------------------------------------------------------+ | Files | * /build-scripts/output/deploy/tc2/android.img | | | * /build-scripts/output/deploy/tc2/ramdisk_uboot.img | | | * /build-scripts/output/deploy/tc2/system.img | | | * /build-scripts/output/deploy/tc2/userdata.img | | | * /build-scripts/output/deploy/tc2/boot.img (AVB only) | | | * /build-scripts/output/deploy/tc2/vbmeta.img (AVB only) | +--------+-------------------------------------------------------------------------+ Run scripts ########### Within the ``/run-scripts/`` there are several convenience functions for testing the software stack. Usage descriptions for the various scripts are provided in the following sections. Obtaining the TC2 FVP --------------------- The TC2 FVP is available to partners for build and run on Linux host environments. Please contact Arm to have access (support@arm.com). Running the software on FVP --------------------------- A Fixed Virtual Platform (FVP) of the TC2 platform must be available to run the included run scripts. The run-scripts structure is as follows: :: run-scripts |--tc2 |--run_model.sh |-- ... Ensure that all dependencies are met by running the FVP: ``./path/to/FVP_TC2``. You should see the FVP launch, presenting a graphical interface showing information about the current state of the FVP. The ``run_model.sh`` script in ``/run-scripts/tc2`` will launch the FVP, providing the previously built images as arguments. Run the ``./run_model.sh`` script: :: ./run_model.sh Incorrect script use, call script as: [OPTIONS] OPTIONS: -m, --model path to model -d, --distro distro version, values supported [buildroot, android-fvp] -a, --avb [OPTIONAL] avb boot, values supported [true, false], DEFAULT: false -t, --tap-interface [OPTIONAL] enable TAP interface -n, --networking [OPTIONAL] networking, values supported [user, tap, none] DEFAULT: tap if tap interface provided, otherwise user -e, --extra-model-params [OPTIONAL] extra model parameters Running Buildroot ################# :: ./run-scripts/tc2/run_model.sh -m -d buildroot Running Android ############### For running Android with AVB disabled: :: ./run-scripts/tc2/run_model.sh -m -d android-fvp For running Android with AVB enabled: :: ./run-scripts/tc2/run_model.sh -m -d android-fvp -a true For running Android with hardware rendering enabled: :: ./run-scripts/tc2/run_model.sh -m -d android-fvp -e '--plugin=' NOTE: ``crypto.so`` is part of your FVP bundle. Expected behaviour ################## When the script is run, four terminal instances will be launched: * ``terminal_uart_ap`` used by the non-secure world components U-boot, Linux Kernel and filesystem (Buildroot/Android); * ``terminal_uart1_ap`` used by the secure world components TF-A, Hafnium, Trusty and OP-TEE; * ``terminal_s0`` used for the SCP logs; * ``terminal_s1`` used by RSS logs (no output by default). Once the FVP is running, hardware Root of Trust will verify AP and SCP images, initialize various crypto services and then handover execution to the SCP. SCP will bring the AP out of reset. The AP will start booting from its ROM and then proceed to boot Trusted Firmware-A, Hafnium, Secure Partitions (OP-TEE, Trusted Services in Buildroot and Trusty in Android) then U-Boot, and then Linux and Buildroot/Android. When booting Buildroot, the model will boot Linux and present a login prompt on terminal_uart_ap. Login using the username ``root``. You may need to hit Enter for the prompt to appear. When booting Android, the GUI window ``Fast Models - Total Compute 2 DP0`` shows the Android logo and on boot completion, the window will show the Android home screen. When booting Android with Android Verified Boot (``AVB=true``), the GUI window will display an error, as illustrated in the related :ref:`Total Compute Platform Expected Test Results ` document section. This is expected with the current TC release. Running sanity tests ----------------------------------- OP-TEE ############### For OP-TEE, the TEE sanity test suite can be run using command ``xtest`` on the ``terminal_uart_ap``. Please be aware that this test suite will take some time to run all its related tests. NOTE: This test is specific to Buildroot only. An example of the expected test result for this test is ilustrated in the related :ref:`Total Compute Platform Expected Test Results ` document section. Trusted Services and Client application ######################################## For Trusted Services, run the command ``ts-service-test -sg ItsServiceTests -sg PsaCryptoApiTests -sg CryptoServicePackedcTests -sg CryptoServiceProtobufTests -sg CryptoServiceLimitTests -v`` for Service API level tests, and run ``ts-demo`` for the demonstration of the client application. NOTE: This test is specific to Buildroot only. An example of the expected test result for this test is ilustrated in the related :ref:`Total Compute Platform Expected Results ` document section. Trusty ############### On the Android distribution, Trusty provides a Trusted Execution Environment (TEE). The functionality of Trusty IPC can be tested using the command ``tipc-test -t ta2ta-ipc`` with root privilege (once Android boots to prompt, run ``su 0`` for root access). NOTE: This test is specific to Android only. An example of the expected test result for this test is ilustrated in the :ref:`Total Compute Platform Expected Test Results ` document section. Microdroid demo ############### On the Android distribution, Virtualization service provides support to run Microdroid based pVM (Protected VM). For running a demo Microdroid, boot TC FVP with Android distribution. Once the Android is completely up, run the following command: :: export ANDROID_PRODUCT_OUT=/src/android/out/target/product/tc_fvp/ ./run-scripts/tc2/run_microdroid_demo.sh NOTE: This test is specific to Android only. An example of the expected test result for this test is ilustrated in the related :ref:`Total Compute Platform Expected Test Results ` document section. Kernel Selftest ############### Tests are located at ``/usr/bin/selftest`` on the device. To run all the tests in one go, use ``./run_kselftest.sh`` script. Tests can be run individually also. :: ./run_kselftest.sh --summary NOTE 1: KSM driver is not a part of the TC2 kernel. Hence, one of the MTE Kselftests will fail for ``check_ksm_options`` test. NOTE 2: This test is specific to Buildroot only. An example of the expected test result for this test is ilustrated in the related :ref:`Total Compute Platform Expected Test Results ` document section. MPAM #### The hardware and the software requirements required for the MPAM feature can be verified by running the command ``testing_mpam.sh`` on ``terminal_uart_ap`` (this script is located inside the `/bin` folder, which is part of the default `$PATH` environment variable, allowing this command to be executed from any location in the device filesystem). NOTE: This test is specific to Buildroot only. An example of the expected test result for this test is ilustrated in the related :ref:`Total Compute Platform Expected Test Results ` document section. BTI ### To run the BTI unit test, navigate to ```` and run: :: adb connect 127.0.0.1:5555 cd /src/android/out/target/product/tc_fvp/testcases/bti-unit-tests/arm64 adb push bti-unit-tests /data/local/tmp cd /src/android/out/target adb push ./product/tc_fvp/obj/SHARED_LIBRARIES/libbti_basic_function_intermediates/libbti_basic_function.so /data/local/tmp On the ``terminal_uart_ap`` run: :: cd /data/local/tmp ./bti-unit-tests NOTE: This test is specific to Android builds with hardware rendering configuration enabled (i.e. `TC_GPU=true`). An example of the expected test result for this test is ilustrated in the related :ref:`Total Compute Platform Expected Test Results ` document section. MTE ### To run the MTE unit test, navigate to ```` and run: :: adb connect 127.0.0.1:5555 cd /src/android/out/target/product/tc_fvp/testcases/mte-unit-tests/arm64 adb push mte-unit-tests /data/local/tmp On the ``terminal_uart_ap`` run: :: cd /data/local/tmp ./mte-unit-tests NOTE: This test is specific to Android builds with hardware rendering configuration enabled (i.e. `TC_GPU=true`). An example of the expected test result for this test is ilustrated in the related :ref:`Total Compute Platform Expected Test Results ` document section. PAUTH ##### To run the PAUTH unit test, navigate to ```` and run: :: adb connect 127.0.0.1:5555 cd /src/android/out/target/product/tc_fvp/testcases/pauth-unit-tests/arm64 adb push pauth-unit-tests /data/local/tmp On the ``terminal_uart_ap`` run: :: cd /data/local/tmp ./pauth-unit-tests NOTE: This test is specific to Android builds with hardware rendering configuration enabled (i.e. `TC_GPU=true`). An example of the expected test result for this test is ilustrated in the related :ref:`Total Compute Platform Expected Test Results ` document section. EAS with LISA ############# This test requires Lisa to be installed. Please refer to the `LISA documentation `_ to get more information about the requirements, dependencies and installation process of LISA on your system. To setup Lisa, please run the following commands: :: git clone https://github.com/ARM-software/lisa.git cd lisa sudo ./install_base.sh --install-all The following commands should be run each time LISA is run: :: source init_env export TC_WORKSPACE= For FVP with buildroot, boot the FVP model to buildroot as you normally would making user user networking is enabled. :: exekall run lisa.tests.scheduler.eas_behaviour --conf The following excerpt illustrates the contents of the ``target_conf_buildroot.yml`` file: :: target-conf: kind: linux name: tc host: localhost port: 8022 username: root password: "" strict-host-check: false kernel: src: ${TC_WORKSPACE}/build-scripts/output/tmp_build/linux modules: make-variables: CC: clang build-env: alpine wait-boot: enable: false devlib: file-xfer: scp max-async: 1 NOTE: This test is specific to Buildroot only. An example of the expected test result for this test is ilustrated in the related :ref:`Total Compute Platform Expected Test Results ` document section. Debugging on Arm Development Studio ----------------------------------- This section describes the steps to debug the TC software stack using `Arm Development Studio `_. Creating a new connection ######################### To create a new connection, please follow the next steps: #. Select ``File->New->Model Connection``; #. Provide the name for the new ``Debug Connection`` and click the ``next`` button; #. Click on the ``Add a new model...`` button; #. Select ``CADI`` as the model interface and click the ``next`` button; #. Select ``Launch and connect to specific model``; #. Select the location on your system containing the TC2 FVP model path and click the ``Finish`` button; #. Once the import process of the model finishes, you can close the ``Model Connection`` window (used to add the new model). Attach and Debug ################ #. Build the target with debug enabled (the file ``/build-scripts/config`` can be configured to enable debug); #. Run Buildroot/Android as described in the section ``Running the software on FVP`` with the extra parameters ``-e -S`` to attach to the debugger. The full command should look like the following: :: ./run-scripts/tc2/run_model.sh -m -d -e -S #. Select the target created as mentioned in ``Creating a new connection`` and ``connect to target`` from debug control console. #. After connection, use options in debug control console (highlighted in the below diagram) or the keyboard shortcuts to ``step``, ``run`` or ``halt``. #. To add debug symbols, right click on target -> ``Debug configurations`` and under ``files`` tab add path to ``elf`` files. #. Debug options such as ``break points``, ``variable watch``, ``memory view`` and so on can be used. .. figure:: Debug_control_console.png NOTE: There is a known issue in connecting all AP cores together. The Hunter ELP core is missing from the cluster view. As a workaround, you can create two target connections as described in the ``Creating a new connection`` section: one for ELP core alone and the other one for the rest of AP cores. Switch between SCP and AP ######################### #. Right click on target and select ``Debug Configurations``; #. Under ``Connection``, select ``Cortex-M3`` for SCP and ``Arm-Hayes_x/Arm-Hunter_x`` for AP core x and then debug. .. figure:: switch_cores.png Enable LLVM parser (for Dwarf5 support) ####################################### To enable LLVM parser (with Dwarf5 support), please follow the next steps: #. Select ``Window->Preferences->Arm DS->Debugger->Dwarf Parser``; #. Tick the ``Use LLVM DWARF parser`` option; #. Click the ``Apply and Close`` button. .. figure:: enable_llvm.png Arm DS version ############## The previous steps apply to the following Arm DS Platinum version/build: .. figure:: arm_ds_version.png NOTE: Arm DS Platinum is only available to licensee partners. Please contact Arm to have access (support@arm.com). Firmware Update --------------- Currently, the firmware update functionality is only supported with the buildroot distro. Creating Capsule ################ Firmware Update in the total compute platform uses the capsule update mechanism. Hence, the Firmware Image Package (FIP) binary has to be converted to a capsule. This can be done with ``GenerateCapsule`` which is present in ``BaseTools/BinWrappers/PosixLike`` of the `edk2 project `__. To generate the capsule from the fip binary, run the following command: :: ./GenerateCapsule -e -o efi_capsule --fw-version 1 --lsv 0 --guid 0d5c011f-0776-5b38-8e81-36fbdf6743e2 --update-image-index 0 --verbose fip-tc.bin Command arguments explanation: * ``fip-tc.bin`` is the input fip file that has the firmware binaries of the total compute platform; * ``efi_capsule`` is the name of capsule to be generated; * ``0d5c011f-0776-5b38-8e81-36fbdf6743e2`` is the image type UUID for the FIP image. Loading Capsule ############### The capsule generated using the above steps has to be loaded into memory during the execution of the model by providing the below FVP arguments: :: --data board.dram=/efi_capsule@0x2000000 This will load the capsule to be updated at address 0x82000000. The final command to run the model for buildroot should look like the following: :: ./run-scripts/tc2/run_model.sh -m -d buildroot -e "--data board.dram=/efi_capsule@0x2000000" Updating Firmware ################# During the normal boot of the platform, stop at the U-Boot prompt and execute the following command: :: TOTAL_COMPUTE# efidebug capsule update -v 0x82000000 This will update the firmware. After it is completed, reboot the platform using the FVP GUI. *Copyright (c) 2022-2023, Arm Limited. All rights reserved.*