Xilinx Virtex-7 VC707 Evaluation Kit (Source: www.xilinx.com)

Debugging Programs on a Bare-metal SiFive U500 RISC-V System on Top of Xilinx VC707 FPGA

With the new changes in the hardware space, such as the acquisition of ARM by Nvidia, it is likely that RISC-V will become more popular. So it is timely to play with some RISC-V platforms, such as SiFive’s U500 which is part of SiFive’s Freedom Unleashed family. However, it is not straight-forward to run a bare-metal program on the U500 platform as compared to Freedom E310 platform. There is not documentation, no support in the freedom-u-sdk, and no Board Support Package (BSP). There is, indeed, support for running a Linux-based OS, but in this post we are interested in bare-metal programs.

Setup the Hardware

We are going to implement the U500 platform on a Xilinx VC707 evaluation kit. Fortunately, there is support in the freedom repository for building the U500 platform on VC707. By default, the system relies on an external clock source, but there is support for an on-board clock as well. You just need to use the following command to build it:

$ make MODEL=VC707BaseShell -f Makefile.vc707-u500devkit verilog
$ make MODEL=VC707BaseShell -f Makefile.vc707-u500devkit mcs

Also, the default JTAG of U500 is based on an external board, but you can also use the LCD pins to connect an Olimex ARM TINY-USB. And who wants to buy yet another board after spending almost $3.5k on the VC707 FPGA?

To make the JTAG work on the LCD pins of the VC707, you first need to remove the LCD, then wire the Olimex ARM TINY-USB to the VC707 as shown below.

Wiring of Olimex ARM TINY-USB JTAG to VC707 LCD pins

Then, you need to modify src/main/scala/shell/xilinx/VC707NewShell.scala by commenting the code in class JTAGDebugVC707Overlay (line 102) and adding the code below with the new pin assignments:

shell { InModuleBody {
    shell.sdc.addClock("JTCK", IOPin(io.jtag_TCK), 10)
    shell.sdc.addGroup(clocks = Seq("JTCK"))
    shell.xdc.clockDedicatedRouteFalse(IOPin(io.jtag_TCK))
    val packagePinsWithPackageIOs = Seq(("AT42", IOPin(io.jtag_TCK)),
      ("AR38", IOPin(io.jtag_TMS)),
      ("AR39", IOPin(io.jtag_TDI)),
      ("AR42", IOPin(io.srst_n)),
      ("AT40", IOPin(io.jtag_TDO)))
packagePinsWithPackageIOs foreach { case (pin, io) => {
      shell.xdc.addPackagePin(io, pin)
      shell.xdc.addIOStandard(io, "LVCMOS18")
      shell.xdc.addPullup(io)
    } }
  } }

Then, you need to add the srst_n in src/main/scala/shell/JTAGDebugOverlay.scala, line 21:

val srst_n = Input(Bool())

Another change is to use a single core instead of 4 in the U500 platform. This is because using more cores may result in an infinite loop during the debugging, in freedom-metal/src/synchronize_harts.c, line 44:

 while (METAL_MSIP(msip_base, i) == 0) ;

This is because the main core is waiting for the other cores to set the MSIP bit, but they are not doing it. I will leave this issue for future investigation. Meanwhile, we modify src/main/scala/unleashed/DevKitConfigs.scala, line 20 to:

new WithNBigCores(1) ++

At the end, run the build process:

$ make MODEL=VC707BaseShell -f Makefile.vc707-u500devkit verilog
$ make MODEL=VC707BaseShell -f Makefile.vc707-u500devkit mcs

If you encounter an error related to generating the ROM during the build, you need to replace the raise with return in rocket-chip-fp32/scripts/vlsi_rom_gen, line 97.

The .mcs and .prm files are in builds/vc707-u500devkit/obj. Open Vivado Hardware Manager, connect to the VC707 board and press Add Configuration Memory Device. Search for mt28gu01gaax1e-bpi-x16 and programm it with the generated .mcs and .prm files. After programming, click Boot from Configuration Memory Device.

You can skip the building process by using my pre-built images found in my fork of the freedom repository.

Note: You need to use Vivado v2016.4 because newer versions are know to have issues when building U500.

Build BSP Support

freedom-e-sdk does not have support for U500 on VC707. Fortunately, there is a tool to generate a BSP using the Device Tree of the generated U500 platform as input. The Device Tree file can be found in the freedom folder (the one used in the previous step to build the U500), under builds/vc707-u500devkit/sifive.freedom.unleashed.DevKitU500FPGADesign_WithDevKit50MHz.dts.

Go to the bsp subfolder of the freedom-e-sdk folder and run:

$ mkdir vc707-u500devkit
$ ./update-targets.sh --target-name vc707-u500devkit --target-type arty --sdk-path=.. --target-dts=sifive.freedom.unleashed.DevKitU500FPGADesign_WithDevKit50MHz.dts

All the files are generated in the vc707-u500devkit folder.

U500 does not have a flash memory. So we need to comment or delete the lines related to flash in openocd.cfg:

...
#flash bank spi0 fespi 0x0 0 0 0 $_TARGETNAME.0 0x64001000
...
#flash protect 0 64 last off

Next, adjust the vid and pid of you Olimex JTAG device. In my case, I had to replace the initial pid 0x002a with 0x0004.

If you encounter error such as “Error: unable to open ftdi device with vid 15ba, pid 0004, description ‘Olimex Ltd. OpenOCD JTAG TINY’, serial ‘*’ at bus location ‘*’”, you also need to comment the ftdi_device_desc line:

#ftdi_device_desc “Olimex Ltd. OpenOCD JTAG TINY”

U500 does not have a flash memory. So we need to modify the lds script file, metal.default.lds. I observed that the BSP generator generated some garbage value for the flash memory address and size:

flash (rxai!w) : ORIGIN = 0x7ffda0cffac0, LENGTH = 0x55e92ef9a9d0

My work-around is to map the flash into the RAM memory space. There is 1GB of RAM on the U500/VC707 setup. I reserved 128MB for the flash, and 896MB for RAM. The memory section of my metal.default.lds look like:

MEMORY
{
 flash (rxai!w) : ORIGIN = 0xB8000000, LENGTH = 0x8000000
 ram (wxa!ri) : ORIGIN = 0x80000000, LENGTH = 0x38000000
}

I also increased the stack and heap size to 1MB and 768MB:

...
SECTIONS
{
__stack_size = DEFINED(__stack_size) ? __stack_size : 0x100000;
PROVIDE(__stack_size = __stack_size);
__heap_size = DEFINED(__heap_size) ? __heap_size : 0x30000000;
...

Running the Debug Flow

Let’s test the platform and the metal BSP! We need a GCC toolchain and OpenOCD for this. We can use the riscv64-unknown-elf-gcc-8.3.0–2020.04.0-x86_64-linux-ubuntu14 and riscv-openocd-0.10.0–2020.04.6-x86_64-linux-ubuntu14 provided by SiFive. Set your PATH to point to the bin subfolders of the above tools.

Next, let’s write a simple program that initializes a big array and computes the sum of its elements. Add this program in the freedom-e-sdk folder, under software/test-mem/test-mem.c:

#include <stdint.h>
#include <stdlib.h>
#define DIM 100600000
void init(uint64_t* a, uint64_t size, uint64_t val) {
 uint64_t i;
 for (i = 0; i < size; i++)
  a[i] = val;
}
uint64_t sum(uint64_t* a, uint64_t size) {
 uint64_t sum = 0;
 uint64_t i;
 for (i = 0; i < size; i++)
  sum += a[i];
 return sum;
}
int main() { 
 uint64_t* a = (uint64_t*)malloc(DIM * sizeof(uint64_t));
 init(a, DIM, 1);
 uint64_t s1 = sum(a, DIM);
 init(a, DIM, 2);
 uint64_t s2 = sum(a, DIM);
 free(a);
 return (int)(s1+s2);
}

Add the following Makefile in the same folder:

PROGRAM ?= test-mem
$(PROGRAM): $(wildcard *.c) $(wildcard *.h) $(wildcard *.S)
clean:
 rm -f $(PROGRAM) $(PROGRAM).hex

In the root of the Freedom-E-SDK, run:

$ make PROGRAM=test-mem TARGET=vc707-u500devkit CONFIGURATION=debug clean
$ make PROGRAM=test-mem TARGET=vc707-u500devkit CONFIGURATION=debug software
$ make PROGRAM=test-mem TARGET=vc707-u500devkit CONFIGURATION=debug upload

Before running the upload command, I recommend commenting the gdb line in scripts/upload, that is:

# $gdb $elf — batch -ex “set remotetimeout 240” -ex “target extended-remote localhost:${GDB_PORT}” -ex “monitor reset halt” -ex “monitor flash protect 0 64 last off” -ex “load” -ex “monitor resume” -ex “monitor shutdown” -ex “quit”
# kill %1

Open a new terminal and run gdb:

$ riscv64-unknown-elf-gdb software/test-mem/debug/test-mem.elf
...
(gdb) set remotetimeout 300
(gdb) target remote localhost:3333
...
(gdb) load
...
(gdb) break test-mem.c:27
...
(gdb) cont
Continuing.
Breakpoint 1, main () at test-mem.c:27
27  return (int)(s1+s2);
(gdb) p s1
$1 = 100600000
(gdb) p s2
$2 = 201200000
(gdb)

Enjoy!

If you like this story, please star my freedom and freedom-e-sdk repositories on GitHub. Thank you.