IQ.Pilot Release Commit @ f2a861c

artifacts/package_sources/tinygrad/docs/developer/am.md

@@ -0,0 +1,39 @@
+# AM Driver
+
+AM driver is a userspace driver targeting AMD's RDNA3/RDNA4. You only need tinygrad to send compute tasks to your GPU!
+
+## How to run?
+Make sure that amdgpu module is unloaded and just run tinygrad with `DEV=AMD`!
+
+Optional requirements:
+
+* System without IOMMU for P2P / SDMA support
+* vfio-pci module for IRQ handling
+
+## Environment Variables
+
+| Variable | Possible Value(s) | Description |
+|----------|------------------|-------------|
+| AM_RESET | [1] | Performs a full GPU reset (reloading all firmware and IP blocks) |
+| AM_DEBUG | [0-4] | Sets the level of additional debugging information |
+
+## AM Driver Details
+
+### Compute & SDMA Queues
+
+AM binds compute queues directly to MEC (bypassing MES). Tinygrad uses only one compute queue, which is bound at `pipe=0 queue=0`. Similarly, the single SDMA queue is bound at `engine=0 queue=0`.
+
+### Boot
+
+The GPU being passed can be in one of several states:
+1. Not initialized
+2. Initialized by amdgpu
+3. Initialized by AM
+
+The first and second states require a full GPU setup since their states are unknown. The second state also requires a mode1 reset to reinitialize all components.
+
+The third state can be set up partially to optimize boot time. In this case, only the GFX and SDMA IPs need to be initialized. To enable this, AM uses a separate boot memory that is guaranteed not to be overwritten. This physical memory is utilized for all blocks that are initialized only during the initial AM boot. To determine if the GPU is in the third state, AM uses `regSCRATCH_REG7` as a flag.
+
+### VM Management
+
+Each AM device sets up only a single `VMID=0` and one page directory. The page directory used is 3-level and thus supports up to 512GB of virtual addresses. All AM devices are located in one virtual address space.

artifacts/package_sources/tinygrad/docs/developer/developer.md

@@ -0,0 +1,46 @@
+The tinygrad framework has four pieces
+
+* a PyTorch like <b>frontend</b>.
+* a <b>scheduler</b> which breaks the compute into kernels.
+* a <b>lowering</b> engine which converts ASTs into code that can run on the accelerator.
+* an <b>execution</b> engine which can run that code.
+
+There is a good [bunch of tutorials](https://mesozoic-egg.github.io/tinygrad-notes/) by Di Zhu that go over tinygrad internals.
+
+There's also a [doc describing speed](../developer/speed.md)
+
+## Frontend
+
+Everything in [Tensor](../tensor/index.md) is syntactic sugar around constructing a graph of [UOps](../developer/uop.md).
+
+The `UOp` graph specifies the compute in terms of low level tinygrad ops. Not all UOps will actually become realized. There's two types of UOps, base and view. base contains compute into a contiguous buffer, and view is a view. Inputs to a base can be either base or view, inputs to a view can only be a single base.
+
+## Scheduling
+
+The [scheduler](https://github.com/tinygrad/tinygrad/tree/master/tinygrad/schedule/__init__.py) converts the graph of UOps into a `LINEAR` UOp whose `src` is a list of `CALL` UOps. One `CALL` is one kernel on the GPU, and the scheduler is responsible for breaking the large compute graph into subgraphs that can fit in a kernel. The `CALL`'s `src[0]` (a `SINK` ast) specifies what compute to run, and the remaining `src` are the buffers to run it on.
+
+## Lowering
+
+The code in [realize](https://github.com/tinygrad/tinygrad/tree/master/tinygrad/engine/realize.py) lowers each `CALL` by compiling its ast into a `PROGRAM` and running it.
+
+::: tinygrad.engine.realize.run_linear
+
+There's a ton of complexity hidden behind this, see the `codegen/` directory.
+
+First we lower the AST to UOps, which is a linear list of the compute to be run. This is where the BEAM search happens.
+
+Then we render the UOps into code with a `Renderer`, then we compile the code to binary with a `Compiler`.
+
+## Execution
+
+`run_linear` walks the `LINEAR` UOp, dispatching each `CALL` to a runner (kernel, copy, view, encdec, or graph).
+
+## Runtime
+
+Runtimes are responsible for device-specific interactions. They handle tasks such as initializing devices, allocating memory, loading/launching programs, and more. You can find more information about the runtimes API on the [runtime overview page](runtime.md).
+
+All runtime implementations can be found in the [runtime directory](https://github.com/tinygrad/tinygrad/tree/master/tinygrad/runtime).
+
+### HCQ Compatible Runtimes
+
+HCQ API is a lower-level API for defining runtimes. Interaction with HCQ-compatible devices occurs at a lower level, with commands issued directly to hardware queues. Some examples of such backends are [NV](https://github.com/tinygrad/tinygrad/tree/master/tinygrad/runtime/ops_nv.py) and [AMD](https://github.com/tinygrad/tinygrad/tree/master/tinygrad/runtime/ops_amd.py), which are userspace drivers for NVIDIA and AMD devices respectively. You can find more information about the API on [HCQ overview page](hcq.md)

artifacts/package_sources/tinygrad/docs/developer/hcq.md

@@ -0,0 +1,128 @@
+# HCQ Compatible Runtime
+
+## Overview
+
+The main aspect of HCQ-compatible runtimes is how they interact with devices. In HCQ, all interactions with devices occur in a hardware-friendly manner using [command queues](#command-queues). This approach allows commands to be issued directly to devices, bypassing runtime overhead such as HIP or CUDA. Additionally, by using the HCQ API, these runtimes can benefit from various optimizations and features, including [HCQGraph](#hcqgraph) and built-in profiling capabilities.
+
+### Command Queues
+
+To interact with devices you create a `HWQueue`. Some methods are required, like timestamp and synchronization methods like [signal](#tinygrad.runtime.support.hcq.HWQueue.signal) and [wait](#tinygrad.runtime.support.hcq.HWQueue.wait), while others are dependent on it being a compute or copy queue.
+
+For example, the following Python code enqueues a wait, execute, and signal command on the HCQ-compatible device:
+```python
+HWQueue().wait(signal_to_wait, value_to_wait) \
+         .exec(program, args_state, global_dims, local_dims) \
+         .signal(signal_to_fire, value_to_fire) \
+         .submit(your_device)
+```
+
+Each runtime should implement the required functions that are defined in the `HWQueue` classes.
+
+::: tinygrad.runtime.support.hcq.HWQueue
+    options:
+        members: [
+            "signal",
+            "wait",
+            "timestamp",
+            "bind",
+            "submit",
+            "memory_barrier",
+            "exec",
+            "copy",
+        ]
+        show_source: false
+
+### HCQ Compatible Device
+
+The `HCQCompiled` class defines the API for HCQ-compatible devices. This class serves as an abstract base class that device-specific implementations should inherit from and implement.
+
+::: tinygrad.runtime.support.hcq.HCQCompiled
+    options:
+        show_source: false
+
+#### Signals
+
+Signals are device-dependent structures used for synchronization and timing in HCQ-compatible devices. They should be designed to record both a `value` and a `timestamp` within the same signal. HCQ-compatible backend implementations should use `HCQSignal` as a base class.
+
+::: tinygrad.runtime.support.hcq.HCQSignal
+    options:
+        members: [value, timestamp, wait]
+        show_source: false
+
+The following Python code demonstrates the usage of signals:
+
+```python
+signal = your_device.new_signal(value=0)
+
+HWQueue().timestamp(signal) \
+         .signal(signal, value_to_fire) \
+         .submit(your_device)
+
+signal.wait(value_to_fire)
+signaled_value = signal.value # should be the same as `value_to_fire`
+timestamp = signal.timestamp
+```
+
+##### Synchronization signals
+
+Each HCQ-compatible device must allocate two signals for global synchronization purposes. These signals are passed to the `HCQCompiled` base class during initialization: an active timeline signal `self.timeline_signal` and a shadow timeline signal `self._shadow_timeline_signal` which helps to handle signal value overflow issues. You can find more about synchronization in the [synchronization section](#synchronization)
+
+### HCQ Compatible Allocator
+
+The `HCQAllocator` base class simplifies allocator logic by leveraging [command queues](#command-queues) abstractions. This class efficiently handles copy and transfer operations, leaving only the alloc and free functions to be implemented by individual backends.
+
+::: tinygrad.runtime.support.hcq.HCQAllocator
+    options:
+        members: [
+            "_alloc",
+            "_free",
+        ]
+        show_source: false
+
+#### HCQ Allocator Result Protocol
+
+Backends must adhere to the `HCQBuffer` protocol when returning allocation results.
+
+::: tinygrad.runtime.support.hcq.HCQBuffer
+    options:
+        members: true
+        show_source: false
+
+### HCQ Compatible Program
+
+`HCQProgram` is a base class for defining programs compatible with HCQ-enabled devices. It provides a flexible framework for handling different argument layouts (see `HCQArgsState`).
+
+::: tinygrad.runtime.support.hcq.HCQProgram
+    options:
+        members: true
+        show_source: false
+
+#### Arguments State
+
+`HCQArgsState` is a base class for managing the argument state for HCQ programs. Backend implementations should create a subclass of `HCQArgsState` to manage arguments for the given program.
+
+::: tinygrad.runtime.support.hcq.HCQArgsState
+    options:
+        members: true
+        show_source: false
+
+**Lifetime**: The `HCQArgsState` is passed to `HWQueue.exec` and is guaranteed not to be freed until `HWQueue.submit` for the same queue is called.
+
+### Synchronization
+
+HCQ-compatible devices use a global timeline signal for synchronizing all operations. This mechanism ensures proper ordering and completion of tasks across the device. By convention, `self.timeline_value` points to the next value to signal. So, to wait for all previous operations on the device to complete, wait for `self.timeline_value - 1` value. The following Python code demonstrates the typical usage of signals to synchronize execution to other operations on the device:
+
+```python
+HWQueue().wait(your_device.timeline_signal, your_device.timeline_value - 1) \
+         .exec(...)
+         .signal(your_device.timeline_signal, your_device.next_timeline()) \
+         .submit(your_device)
+
+# Optionally wait for execution
+your_device.timeline_signal.wait(your_device.timeline_value - 1)
+```
+
+## HCQGraph
+
+[HCQGraph](https://github.com/tinygrad/tinygrad/tree/master/tinygrad/runtime/graph/hcq.py) is a core feature that implements `GraphRunner` for HCQ-compatible devices. `HCQGraph` builds static `HWQueue` for all operations per device. To optimize enqueue time, only the necessary parts of the queues are updated for each run using the symbolic variables, avoiding a complete rebuild.
+Optionally, queues can implement a `bind` API, which allows further optimization by eliminating the need to copy the queues into the device ring.

artifacts/package_sources/tinygrad/docs/developer/layout.md

@@ -0,0 +1,60 @@
+# tinygrad directory layout
+
+This explains the flow of a big graph down to programs.
+
+Directories are listed in order of how they are processed.
+
+---
+
+## tinygrad/schedule
+
+Group UOps into kernels.
+
+::: tinygrad.schedule.rangeify.get_kernel_graph
+    options:
+        members: false
+        show_labels: false
+        show_source: false
+
+---
+
+## tinygrad/codegen/opt
+
+Transforms the ast into an optimized ast. This is where BEAM search and heuristics live.
+
+---
+
+## tinygrad/codegen
+
+Transform the optimized ast into a linearized and rendered program.
+
+::: tinygrad.codegen.to_program
+    options:
+        members: false
+        show_labels: false
+        show_source: false
+
+---
+
+## tinygrad/renderer
+
+Transform the linearized list of UOps into a program, represented as a string.
+
+::: tinygrad.renderer.Renderer
+    options:
+        members:
+            - render
+        show_labels: false
+        show_source: false
+
+---
+
+## tinygrad/engine
+
+Abstracted high level interface to the runtimes.
+
+::: tinygrad.engine.realize.to_program
+    options:
+        members: false
+        show_labels: false
+        show_source: false

artifacts/package_sources/tinygrad/docs/developer/runtime.md

@@ -0,0 +1,51 @@
+# Runtime Overview
+
+## Overview
+
+A typical runtime consists of the following parts:
+
+- [Compiled](#compiled)
+- [Allocator](#allocator)
+- [Program](#program)
+- [Compiler](#compiler)
+
+### Compiled
+
+The `Compiled` class is responsible for initializing and managing a device.
+
+::: tinygrad.device.Compiled
+    options:
+        members: [
+            "synchronize"
+        ]
+        show_source: false
+
+### Allocator
+
+The `Allocator` class is responsible for managing memory on the device. There is also a version called the `LRUAllocator`, which caches allocated buffers to optimize performance.
+
+::: tinygrad.device.Allocator
+    options:
+        members: true
+        show_source: false
+
+::: tinygrad.device.LRUAllocator
+    options:
+        members: true
+        show_source: false
+
+### Program
+
+The `Program` class is created for each loaded program. It is responsible for executing the program on the device. As an example, here is a `CPUProgram` implementation which loads program and runs it.
+
+::: tinygrad.runtime.ops_cpu.CPUProgram
+    options:
+        members: true
+
+### Compiler
+
+The `Compiler` class compiles the output from the `Renderer` and produces it in a device-specific format.
+
+::: tinygrad.device.Compiler
+    options:
+        members: true

artifacts/package_sources/tinygrad/docs/developer/speed.md

@@ -0,0 +1,71 @@
+# speed in tinygrad
+
+## Overview
+
+Speed refers to many different things. To break it down to four, there's:
+
+- Compile Speed (Python)
+- Execution Speed (driver)
+- Model Speed (scheduler)
+- Kernel Speed (codegen)
+
+## Compile Speed (Python)
+
+This is how long the first run of your model takes. It's limited largely by the runtime of the Python doing UOp rewrites. Currently it's a bit slow, but on par with torch.compile. It gets even slower if you are using BEAM, since that's compiling many variants of each kernel.
+
+This will be improved by writing faster graph_rewrite, doing less graph_rewrite, and better parallelization.
+
+## Execution Speed (driver)
+
+After your model is compiled, you are often using the `TinyJIT`. tinygrad has the best execution speed of any framework because it usually bypasses the GPU driver and prebuilds the command queue. It's tons faster than normal CUDA, and often even faster than CUDA Graph.
+
+There's very little to improve here, as this is almost never the bottleneck.
+
+## Model Speed (scheduler)
+
+The scheduler determines how operations are grouped into kernels and which Tensors are written to memory. This is currently a big bottleneck of training speed.
+
+The decisions are often not obvious. For example, when is it worth recomputing an arithmetic operation instead of storing and loading from memory? Example:
+
+```python
+from tinygrad import Tensor
+a = Tensor.rand(100)
+b = Tensor.rand(100)
+c = Tensor.rand(100)
+d = Tensor.rand(100)
+out1 = a+b+c
+out2 = a+b+d
+Tensor.realize(out1, out2)
+```
+
+The real answer is obvious, compute both `out1` and `out2` in the same kernel. But you can't always do that. If you can't, should `a+b` first be saved to a subbuffer? Or should both the `out1` and `out2` kernels recompute `a+b`?
+
+In this case: with recompute (6 reads + 2 writes), no recompute (6 reads + 3 writes), so we should probably recompute. However, once you add movement ops and casts this is even harder to figure out. tinygrad doesn't yet have a systematic way to do it.
+
+## Kernel Speed (codegen)
+
+Given that you have decided how the model ops will be grouped and what will be written to memory, kernel speed determines how fast that operation is done. This is what BEAM changes, it searches over a set of equivalent kernels which all perform the same operation and finds the one which performs the task the fastest.
+
+In `kernel.py` we have a set of `OptOps`, these control the parameters of the speed optimizations applied to the kernel.
+
+### Memory
+
+The main bottleneck in most kernels is accessing memory. In a freshman algorithms class, you'll learn about cache aware matrix multiplication, and this is all forms of that. While the same math is run, the order in which you run it can have large impacts on the speed depending on if the data you are loading. OptOps will change this order.
+
+Memory, even cache, is often much slower than accessing the register file. The amount of times data is used in math is called the "arithmetic intensity". For operations like BS=1 GEMV, the arithmetic intensity is 1, but for GEMMs and convs it can be much higher. OptOps like UPCAST and UNROLL can increase this, but be careful of making them too large, as if there's too much register pressure on the GPU the warp scheduler may not be able to fit many warps, or even worse, it could be spilling to local memory.
+
+4090s have 1 TB/s of ram bandwidth and ~160 TFLOPS of compute, so you need to use each loaded value ~100 times. The L1 cache has around 40 TB/s of bandwidth, so in order to get full compute utilization you need to use each value ~4 times.
+
+A lot of work can still be done here. For example, we never copy the inputs to on chip SRAM, but this is often quite helpful for kernel speed. Also, we aren't doing a good job with L2 cache awareness (the locals handle L1 quite well)
+
+### Tensor Cores
+
+Many accelerators have Tensor Cores / MAC arrays / systolic arrays. The main value of these is that, since they are 2-D, they create an n^2 ratio between the compute and the input data.
+
+GPUs use Tensor Cores instead of MAC arrays to fit better in the GPU warp paradigm. This is because the output of Tensor Cores is O(n) wrt the input, while the output of MAC arrays is O(n^2)
+
+We have a simple framework in tinygrad for adding these ALU blocks and achieving good performance from them.
+
+### Indexing
+
+Indexing determines the address of the memory we need to load. GPUs often have less integer math resources than floating point math, so this can sometimes be the bottleneck. We have a symbolic math engine in our rewrite rules to simplify indexing before it's emitted to the kernel. Newer NVIDIA GPUs have a "Tensor Memory Accelerator" to assist with fast indexing, however, this is not supported in tinygrad yet.

artifacts/package_sources/tinygrad/docs/developer/uop.md

@@ -0,0 +1,11 @@
+::: tinygrad.uop.ops.UOp
+    options:
+        members: false
+        members_order: source
+        show_labels: false
+
+::: tinygrad.uop.ops.Ops
+    options:
+        members: true
+        members_order: source
+        show_labels: false