Debugging Guide¶
Attention
This document is not updated to SDK 0.3.0. A few options and flags described here are outdated.
This section describes how to debug your kernel code. See Working With Code Samples for how to compile and simulate (run the program).
To debug, you use the sdk_debug_shell script. This script supports the following command groups:
Run the CSL compiler.
Instruction trace and statistics.
Wavelet trace.
PE symbol information.
Attention
The visualize option is deprecated for the debug tool sdk_debug_shell. Use the new SDK GUI tool to invoke the visualizer GUI.
$ sdk_debug_shell --help
Usage: sdk_debug_shell [OPTIONS] COMMAND [ARGS]
Debugger tool for the Cerebras WSE kernel code.
Options:
--help Show this message and exit.
Commands:
compile Command to run the CSLang compiler.
instr-trace Command to view instructions, tasks and uthreads
(microthreads).
symbol-info Command to view information for symbols, routes and
symbolic colors.
wavelet-trace Command to view wavelets received at a specified PE.
Compile¶
The compile option invokes the CSLang compiler.
$ sdk_debug_shell compile --help
Usage: sdk_debug_shell compile [OPTIONS]
Run the CSLang compiler.
Options:
--help Show this message and exit.
Instruction trace¶
The instr-trace option, shown below, can be used to view instructions, tasks and uthreads that were executed at a specified PE for a given simulation run. The instruction statistics are also attached at the end of the output.
Note
The term “uthread” is synonymous with microthread.
$ sdk_debug_shell instr-trace --help
Usage: sdk_debug_shell instr-trace [OPTIONS] COMMAND [ARGS]
Command to view instructions, tasks and uthreads (microthreads).
Options:
--artifact_dir PATH Required. Directory for ELF files
and simfab JSON logs.
--x INTEGER Required. X coordinate of PE.
--y INTEGER Required. Y coordinate of PE.
--help Show this message and exit.
Commands:
tasks Show a list of tasks for a PE.
trace Show instruction trace for a PE.
uthreads Show a list of uthreads for a PE.
The input option --artifact_dir is required to all three commands: tasks, trace, and uthread.
The --artifact_dir is the directory that holds all the ELF files and the simulation JSON log file for the simulation run.
Tasks in a PE¶
You can display all the tasks running in a specified PE.
$ sdk_debug_shell instr-trace tasks --help
Usage: sdk_debug_shell instr-trace tasks [OPTIONS]
Show a list of tasks for a PE.
Options:
--help Show this message and exit.
Example¶
sdk_debug_shell instr-trace --artifact_dir my_artifact_dir --x 0 --y 0 tasks
Inputs
--artifact_dir my_artifact_dir: All ELF files and the simulation JSON log file for the simulation run must be located in themy_artifact_dirdirectory.--x 0 --y 0: The x,y coordinate of PE (P0.0).
Outputs
==================================================
Showing list of tasks on P0.0
==================================================
[0]
There is one task (task 0) in PE (P0.0) in this simulation run.
Uthreads in a PE¶
This command displays all uthreads running in a specified PE.
$ sdk_debug_shell instr-trace uthreads --help
Usage: sdk_debug_shell instr-trace uthreads [OPTIONS]
Show a list of uthreads for a PE.
Options:
--help Show this message and exit.
Example¶
sdk_debug_shell instr-trace --artifact_dir my_artifact_dir --x 0 --y 0 uthreads
Inputs
--artifact_dir my_artifact_dir: All ELF files and the simulation JSON log file for the simulation run must be located in themy_artifact_dirdirectory.--x 0 --y 0: The x,y coordinate of PE (P0.0).
Outputs
==================================================
Showing list of uthreads on P0.0
==================================================
[]
There are no uthreads in PE (P0.0) in this simulation run.
Trace in a specified PE¶
This command displays instructions executed in a specified PE. The instruction stats are also displayed by this command.
$ sdk_debug_shell instr-trace trace --help
Usage: sdk_debug_shell instr-trace trace [OPTIONS]
Show instruction trace for a PE.
Options:
--uthread INTEGER Microthread to filter by.
--task INTEGER Task to filter by.
--help Show this message and exit.
Example¶
sdk_debug_shell instr-trace --artifact_dir my_artifact_dir --x 0 --y 0 trace
Inputs
--artifact_dir my_artifact_dir: All ELF files and the simulation JSON log file for the simulation run must be located in themy_artifact_dirdirectory.--x 0 --y 0: The x,y coordinate of PE (P0.0).
Outputs
==================================================
Showing instruction trace on P0.0
==================================================
cycle opaddr opname dest src0 src1 src2 // source
------- -------- -------- ---------- ---------- ------ ------ ---------------------
8 0x204 MOVRI 0x0 (0x2a) // my_source.csl:8
9 0x205 ST16 0x0 (0x2a) 0x0 (0x2a) // my_source.csl:9
Summary:
cycles (%) opaddr instruction
------------ -------- -------------
2 (100%) ALL
1 (50.00%) 0x205 ST16
1 (50.00%) 0x204 MOVRI
Each instruction is displayed with the cycle at which it occurs, opaddr (instruction opcode address), opname (instruction opcode), operands (dest, src0, src1, src2), and source.
For example, in Line 6 above, this instruction occurred at cycle 8. Its opaddr is 0x204. The opname is MOVRI. The instruction’s dest operand is at address 0x0 and has value 0x2a. The source for this instruction is at my_source.csl:8. The Summary section displays the statistics for all instructions. In this example, both instructions ST16 and MOVRI take 50% cycles each.
See also
See the section Builtins for DSD operations in Builtins for a list of operations (on a set of elements described by DSDs) that make use of native hardware instructions.
By default, the instructions for all tasks and uthreads will be displayed. You can further restrict the instructions by specifying either --task or --uthread.
Wavelet trace¶
Use the following wavelet-trace option to view wavelets received at a specified PE (Processor Element) for a given simulation run.
$ sdk_debug_shell wavelet-trace --help
Usage: sdk_debug_shell wavelet-trace [OPTIONS] COMMAND [ARGS]...
Command to view wavelets received at a specified PE.
Options:
--artifact_dir PATH Required. Artifact directory name where
ELF files and the simulation JSON log
file for the simulation run are located.
--x INTEGER Required. X coordinate of PE.
--y INTEGER Required. Y coordinate of PE.
--color INTEGER RANGE Color to filter by.
--fmt [i16|f16|f32|i16,f16|hex] Display format [Default: i16].
--help Show this message and exit.
Commands:
trace Show wavelet trace for a PE.
Example¶
sdk_debug_shell wavelet-trace --x 0 --y 0 trace
Inputs
--x 0 --y 0: The x,y coordinate of PE (P0.0).
Output
Showing wavelets received at router on P0.0 for all colors
----------------------------------------------------------
cycle color link ctrl wvlt(i16)
4 23 W 0 0,100
5 23 W 0 1,176
6 23 W 0 2,159
7 23 W 0 3,29
166 23 W 0 4,30
231 23 W 0 5,227
296 23 W 0 6,205
361 23 W 0 7,92
426 23 W 0 8,230
...
8579 1 E 0 0,100
8618 1 E 0 1,176
8657 1 E 0 2,159
8696 1 E 0 3,29
8735 1 E 0 4,30
8774 1 E 0 5,227
8813 1 E 0 6,205
8852 1 E 0 7,92
8891 1 E 0 8,230
8930 1 E 0 9,151
...
Each row in the wavelet trace table describes: the cycle, color, link, ctrl, wvlt (idx, data fields of the wavelet).
For example, the first data point in the table describes: On cycle 4, the processor element at X=0, Y=0 received a wavelet of color 23 on its West-facing link. The wavelet was a data wavelet and not a control wavelet (ctrl =0). The 16-bit index and data fields are 0 and 100 respectively in (i16 format).
Showing wavelets received at router on P0.0 for all colors
----------------------------------------------------------
cycle color link ctrl wvlt(i16)
4 23 W 0 0,100
By default, all colors for the processor element will be displayed. To view wavelets for a specified color, use --color <color> option.
For example, to see wavelets only at color 1 for the same PE, use the following command:
$ sdk_debug_shell wavelet-trace --x 0 --y 0 --color 1 trace
-------------------------------------------------------------
Showing wavelets received at router on P0.0 for color(s) (1,)
-------------------------------------------------------------
cycle color link ctrl wvlt(i16)
8579 1 E 0 0,100
8618 1 E 0 1,176
8657 1 E 0 2,159
...
PE symbol information¶
This command group is used to show information about the symbols, routes, symbolic colors information on the compiled ELF. It supports the following commands:
$ sdk_debug_shell symbol-info --help
Usage: sdk_debug_shell symbol-info [OPTIONS] COMMAND [ARGS]
Command to view information for symbols, routes and symbolic colors.
Options:
--artifact_dir PATH Required. Artifacts directory.
--x INTEGER X coordinate of PE.
--y INTEGER Y coordinate of PE.
--help Show this message and exit.
Commands:
show Show routes, colors and symbols for a PE.
show-colors Show colors.
show-routes Show routes.
show-symbols Show symbols.
Important
The artifact_dir is the required parameter and should refer to the directory where the compiled ELF files are present. An example output is given below:
PE4.4 </net/user1-dev/srv/nfs/user1-data/ws/cslang/test-e2e/gemv.csl, /cb/home/user1/ws/cslang/build/test-e2e/teste2e/Output/gemv.csl.tmp_3_3.elf>
# routes
route 1: ...N. -> ....R
route 3: ....R -> .E...
route 4: W.... -> ....R
# colors
x_in: 1
Ax_out: 3
y_out: 3
b_in: 4
sentinel: 43
# symbols
LLVMDialectModule
A$local
Ax_temp$local
memcpy$local
memset$local
memcpy
memset
rx_x_task
sentinel_task
PE0.0 </net/user1/srv/nfs/user1-data/ws/cslang/test-e2e/switch_basic.csl, /cb/home/user1/ws/cslang/build/test-e2e/teste2e/Output/switch_basic.csl.tmp_0_0.elf>
# routes
route 5: W.... -> ..S..
# colors
in: 5
# symbols
LLVMDialectModule
memcpy$local
memset$local
memcpy
memset
The output presents the information per ELF file. Each ELF corresponds to a single tile.
The routes section shows the route from ingress to egress: N for North, S for South, E for East, W for West and O for on-ramp or off-ramp.
The colors section shows the symbolic color in the source CSLang program to the route index mapping.
The symbols section shows the symbol names.
symbol-info¶
The symbol-info command optionally takes the tile coordinate as a x, y values. When given, only the symbol information on that tile is shown:
$ sdk_debug_shell symbol-info --artifact_dir /cb/home/user1/ws/cslang/build/test-e2e/teste2e/Output --x 4 --y 4 show
PE4.4 </home/tests/test-e2e/gemv.csl, /home/tests/test-e2e/teste2e/Output/gemv.csl.tmp_3_3.elf>
# routes
route 1: ...N. -> ....R
route 3: ....R -> .E...
route 4: W.... -> ....R
# colors
x_in: 1
Ax_out: 3
y_out: 3
b_in: 4
sentinel: 43
# symbols
LLVMDialectModule
A$local
Ax_temp$local
memcpy$local
memset$local
memcpy
memset
rx_x_task
sentinel_task
show-routes¶
The show-routes command can be used to show only the routes section. For example:
$ sdk_debug_shell symbol-info --artifact_dir /cb/home/user1/ws/cslang/build/test-e2e/teste2e/Output --x 4 --y 4 show-routes
PE4.4 </home/tests/test-e2e/gemv.csl, /home/tests/test-e2e/teste2e/Output/gemv.csl.tmp_3_3.elf>
# routes
route 1: ...N. -> ....R
route 3: ....R -> .E...
route 4: W.... -> ....R
show-colors¶
The show-colors command can be used to show only the colors section. For example:
$ sdk_debug_shell symbol-info --artifact_dir /cb/home/user1/ws/cslang/build/test-e2e/teste2e/Output --x 4 --y 4 show-colors
PE4.4 </home/tests/test-e2e/gemv.csl, /home/tests/test-e2e/teste2e/Output/gemv.csl.tmp_3_3.elf>
# colors
x_in: 1
Ax_out: 3
y_out: 3
b_in: 4
sentinel: 43
show-symbols¶
The show-symbols command can be used to show only the symbols section. For example:
$ sdk_debug_shell symbol-info --artifact_dir /cb/home/user1/ws/cslang/build/test-e2e/teste2e/Output --x 4 --y 4 show-symbols
PE4.4 </home/tests/test-e2e/gemv.csl, /home/tests/test-e2e/teste2e/Output/gemv.csl.tmp_3_3.elf>
# symbols
LLVMDialectModule
A$local
Ax_temp$local
memcpy$local
memset$local
memcpy
memset
rx_x_task
sentinel_task