.. _debugging-guide: Debugging Guide =============== .. attention:: This document is not updated to SDK 0.3.0. A few options and flags described here are outdated. This section describes how to debug your kernel code. See :ref:`working-with-code-samples` for how to compile and simulate (run the program). To debug, you use the ``sdk_debug_shell`` script. This script supports the following command groups: - Run the CSL compiler. - Instruction trace and statistics. - Wavelet trace. - PE symbol information. .. attention:: The ``visualize`` option is deprecated for the debug tool ``sdk_debug_shell``. Use the new :ref:`sdk-gui` tool to invoke the visualizer GUI. .. code-block:: bash $ sdk_debug_shell --help Usage: sdk_debug_shell [OPTIONS] COMMAND [ARGS] Debugger tool for the Cerebras WSE kernel code. Options: --help Show this message and exit. Commands: compile Command to run the CSLang compiler. instr-trace Command to view instructions, tasks and uthreads (microthreads). symbol-info Command to view information for symbols, routes and symbolic colors. wavelet-trace Command to view wavelets received at a specified PE. Compile ------- The ``compile`` option invokes the CSLang compiler. .. code-block:: bash $ sdk_debug_shell compile --help Usage: sdk_debug_shell compile [OPTIONS] Run the CSLang compiler. Options: --help Show this message and exit. Instruction trace ----------------- The ``instr-trace`` option, shown below, can be used to view instructions, tasks and uthreads that were executed at a specified PE for a given simulation run. The instruction statistics are also attached at the end of the output. .. note:: The term "uthread" is synonymous with microthread. .. code-block:: bash $ sdk_debug_shell instr-trace --help Usage: sdk_debug_shell instr-trace [OPTIONS] COMMAND [ARGS] Command to view instructions, tasks and uthreads (microthreads). Options: --artifact_dir PATH Required. Directory for ELF files and simfab JSON logs. --x INTEGER Required. X coordinate of PE. --y INTEGER Required. Y coordinate of PE. --help Show this message and exit. Commands: tasks Show a list of tasks for a PE. trace Show instruction trace for a PE. uthreads Show a list of uthreads for a PE. The input option ``--artifact_dir`` is required to all three commands: ``tasks``, ``trace``, and ``uthread``. The ``--artifact_dir`` is the directory that holds all the ELF files and the simulation JSON log file for the simulation run. Tasks in a PE ~~~~~~~~~~~~~ You can display all the tasks running in a specified PE. .. code-block:: bash $ sdk_debug_shell instr-trace tasks --help Usage: sdk_debug_shell instr-trace tasks [OPTIONS] Show a list of tasks for a PE. Options: --help Show this message and exit. Example """"""" .. code-block:: bash sdk_debug_shell instr-trace --artifact_dir my_artifact_dir --x 0 --y 0 tasks **Inputs** - ``--artifact_dir my_artifact_dir``: All ELF files and the simulation JSON log file for the simulation run must be located in the ``my_artifact_dir`` directory. - ``--x 0 --y 0``: The x,y coordinate of PE (P0.0). **Outputs** .. code-block:: bash ================================================== Showing list of tasks on P0.0 ================================================== [0] There is one task (task 0) in PE (P0.0) in this simulation run. Uthreads in a PE ~~~~~~~~~~~~~~~~ This command displays all uthreads running in a specified PE. .. code-block:: bash $ sdk_debug_shell instr-trace uthreads --help Usage: sdk_debug_shell instr-trace uthreads [OPTIONS] Show a list of uthreads for a PE. Options: --help Show this message and exit. Example """"""" .. code-block:: bash sdk_debug_shell instr-trace --artifact_dir my_artifact_dir --x 0 --y 0 uthreads **Inputs** - ``--artifact_dir my_artifact_dir``: All ELF files and the simulation JSON log file for the simulation run must be located in the ``my_artifact_dir`` directory. - ``--x 0 --y 0``: The x,y coordinate of PE (P0.0). **Outputs** .. code-block:: bash ================================================== Showing list of uthreads on P0.0 ================================================== [] There are no uthreads in PE (P0.0) in this simulation run. Trace in a specified PE ~~~~~~~~~~~~~~~~~~~~~~~ This command displays instructions executed in a specified PE. The instruction stats are also displayed by this command. .. code-block:: bash $ sdk_debug_shell instr-trace trace --help Usage: sdk_debug_shell instr-trace trace [OPTIONS] Show instruction trace for a PE. Options: --uthread INTEGER Microthread to filter by. --task INTEGER Task to filter by. --help Show this message and exit. Example """"""" .. code-block:: bash sdk_debug_shell instr-trace --artifact_dir my_artifact_dir --x 0 --y 0 trace **Inputs** - ``--artifact_dir my_artifact_dir``: All ELF files and the simulation JSON log file for the simulation run must be located in the ``my_artifact_dir`` directory. - ``--x 0 --y 0``: The x,y coordinate of PE (P0.0). **Outputs** .. code-block:: bash ================================================== Showing instruction trace on P0.0 ================================================== cycle opaddr opname dest src0 src1 src2 // source ------- -------- -------- ---------- ---------- ------ ------ --------------------- 8 0x204 MOVRI 0x0 (0x2a) // my_source.csl:8 9 0x205 ST16 0x0 (0x2a) 0x0 (0x2a) // my_source.csl:9 Summary: cycles (%) opaddr instruction ------------ -------- ------------- 2 (100%) ALL 1 (50.00%) 0x205 ST16 1 (50.00%) 0x204 MOVRI Each instruction is displayed with the ``cycle`` at which it occurs, ``opaddr`` (instruction opcode address), ``opname`` (instruction opcode), operands (``dest, src0, src1, src2``), and ``source``. For example, in Line 6 above, this instruction occurred at ``cycle`` 8. Its ``opaddr`` is 0x204. The ``opname`` is ``MOVRI``. The instruction’s ``dest`` operand is at address 0x0 and has value 0x2a. The ``source`` for this instruction is at ``my_source.csl:8``. The Summary section displays the statistics for all instructions. In this example, both instructions ``ST16`` and ``MOVRI`` take 50% cycles each. .. seealso:: See the section **Builtins for DSD operations** in :ref:`builtins` for a list of operations (on a set of elements described by DSDs) that make use of native hardware instructions. By default, the instructions for all tasks and uthreads will be displayed. You can further restrict the instructions by specifying either ``--task`` or ``--uthread``. Wavelet trace ------------- Use the following ``wavelet-trace`` option to view wavelets received at a specified PE (Processor Element) for a given simulation run. .. code-block:: bash $ sdk_debug_shell wavelet-trace --help Usage: sdk_debug_shell wavelet-trace [OPTIONS] COMMAND [ARGS]... Command to view wavelets received at a specified PE. Options: --artifact_dir PATH Required. Artifact directory name where ELF files and the simulation JSON log file for the simulation run are located. --x INTEGER Required. X coordinate of PE. --y INTEGER Required. Y coordinate of PE. --color INTEGER RANGE Color to filter by. --fmt [i16|f16|f32|i16,f16|hex] Display format [Default: i16]. --help Show this message and exit. Commands: trace Show wavelet trace for a PE. Example ~~~~~~~ .. code-block:: bash sdk_debug_shell wavelet-trace --x 0 --y 0 trace **Inputs** - ``--x 0 --y 0``: The x,y coordinate of PE (P0.0). **Output** .. code-block:: bash Showing wavelets received at router on P0.0 for all colors ---------------------------------------------------------- cycle color link ctrl wvlt(i16) 4 23 W 0 0,100 5 23 W 0 1,176 6 23 W 0 2,159 7 23 W 0 3,29 166 23 W 0 4,30 231 23 W 0 5,227 296 23 W 0 6,205 361 23 W 0 7,92 426 23 W 0 8,230 ... 8579 1 E 0 0,100 8618 1 E 0 1,176 8657 1 E 0 2,159 8696 1 E 0 3,29 8735 1 E 0 4,30 8774 1 E 0 5,227 8813 1 E 0 6,205 8852 1 E 0 7,92 8891 1 E 0 8,230 8930 1 E 0 9,151 ... Each row in the wavelet trace table describes: the ``cycle``, ``color``, ``link``, ``ctrl``, ``wvlt`` (``idx``, ``data`` fields of the wavelet). For example, the first data point in the table describes: On cycle 4, the processor element at X=0, Y=0 received a wavelet of ``color`` 23 on its West-facing link. The wavelet was a data wavelet and not a control wavelet (``ctrl`` =0). The 16-bit index and data fields are 0 and 100 respectively in (i16 format). .. code-block:: bash Showing wavelets received at router on P0.0 for all colors ---------------------------------------------------------- cycle color link ctrl wvlt(i16) 4 23 W 0 0,100 By default, all colors for the processor element will be displayed. To view wavelets for a specified color, use ``--color `` option. For example, to see wavelets only at ``color`` 1 for the same PE, use the following command: .. code-block:: bash $ sdk_debug_shell wavelet-trace --x 0 --y 0 --color 1 trace ------------------------------------------------------------- Showing wavelets received at router on P0.0 for color(s) (1,) ------------------------------------------------------------- cycle color link ctrl wvlt(i16) 8579 1 E 0 0,100 8618 1 E 0 1,176 8657 1 E 0 2,159 ... PE symbol information --------------------- This command group is used to show information about the symbols, routes, symbolic colors information on the compiled ELF. It supports the following commands: .. code-block:: bash $ sdk_debug_shell symbol-info --help Usage: sdk_debug_shell symbol-info [OPTIONS] COMMAND [ARGS] Command to view information for symbols, routes and symbolic colors. Options: --artifact_dir PATH Required. Artifacts directory. --x INTEGER X coordinate of PE. --y INTEGER Y coordinate of PE. --help Show this message and exit. Commands: show Show routes, colors and symbols for a PE. show-colors Show colors. show-routes Show routes. show-symbols Show symbols. .. important:: The ``artifact_dir`` is the required parameter and should refer to the directory where the compiled ELF files are present. An example output is given below: .. code-block:: bash PE4.4 # routes route 1: ...N. -> ....R route 3: ....R -> .E... route 4: W.... -> ....R # colors x_in: 1 Ax_out: 3 y_out: 3 b_in: 4 sentinel: 43 # symbols LLVMDialectModule A$local Ax_temp$local memcpy$local memset$local memcpy memset rx_x_task sentinel_task PE0.0 # routes route 5: W.... -> ..S.. # colors in: 5 # symbols LLVMDialectModule memcpy$local memset$local memcpy memset The output presents the information per ELF file. Each ELF corresponds to a single tile. The routes section shows the route from ingress to egress: N for North, S for South, E for East, W for West and O for on-ramp or off-ramp. The colors section shows the symbolic color in the source CSLang program to the route index mapping. The symbols section shows the symbol names. ``symbol-info`` ~~~~~~~~~~~~~~~ The ``symbol-info`` command optionally takes the tile coordinate as a x, y values. When given, only the symbol information on that tile is shown: .. code-block:: bash $ sdk_debug_shell symbol-info --artifact_dir /cb/home/user1/ws/cslang/build/test-e2e/teste2e/Output --x 4 --y 4 show PE4.4 # routes route 1: ...N. -> ....R route 3: ....R -> .E... route 4: W.... -> ....R # colors x_in: 1 Ax_out: 3 y_out: 3 b_in: 4 sentinel: 43 # symbols LLVMDialectModule A$local Ax_temp$local memcpy$local memset$local memcpy memset rx_x_task sentinel_task ``show-routes`` ~~~~~~~~~~~~~~~ The ``show-routes`` command can be used to show only the routes section. For example: .. code-block:: bash $ sdk_debug_shell symbol-info --artifact_dir /cb/home/user1/ws/cslang/build/test-e2e/teste2e/Output --x 4 --y 4 show-routes PE4.4 # routes route 1: ...N. -> ....R route 3: ....R -> .E... route 4: W.... -> ....R ``show-colors`` ~~~~~~~~~~~~~~~ The ``show-colors`` command can be used to show only the colors section. For example: .. code-block:: bash $ sdk_debug_shell symbol-info --artifact_dir /cb/home/user1/ws/cslang/build/test-e2e/teste2e/Output --x 4 --y 4 show-colors PE4.4 # colors x_in: 1 Ax_out: 3 y_out: 3 b_in: 4 sentinel: 43 ``show-symbols`` ~~~~~~~~~~~~~~~~ The ``show-symbols`` command can be used to show only the symbols section. For example: .. code-block:: bash $ sdk_debug_shell symbol-info --artifact_dir /cb/home/user1/ws/cslang/build/test-e2e/teste2e/Output --x 4 --y 4 show-symbols PE4.4 # symbols LLVMDialectModule A$local Ax_temp$local memcpy$local memset$local memcpy memset rx_x_task sentinel_task