> ## Documentation Index
> Fetch the complete documentation index at: https://docs.alginte.com/llms.txt
> Use this file to discover all available pages before exploring further.

# SpEL reference & cookbook

> What every expression field can see, operator by operator; what each serde hands you for key and value, including the missing-field behaviour; windowed keys; and a cookbook of recipes, each one a row in the fidelity corpus that checks the editor against the deployed runtime.

This page is the scan-first companion to [SpEL expressions](https://docs.alginte.com/streams/spel.md), which explains the
ideas. Everything here is pinned: every scope rule and every recipe is a row in the fidelity
corpus, a test suite that runs the same expression through the editor's validator, the sample
preview and a deployed topology and fails when they disagree.

> [!NOTE]
>
> The scope rules below have always been what the runtime does. Since Alginte 0.12.0 the editor
> enforces two of them it used to let through: an aggregate **initializer** has no record in
> scope, and **Map Values** sees only the value.

## What each expression can see

`key` and `value` are the record at the operator's input. `#`-variables are extra context some
roles carry. A name that is not in scope is rejected by the editor with a hint, and would fail
deployed.

| Operator, field | `key` | `value` | Variables | Result |
| - | :-: | :-: | - | - |
| **Filter** / **Filter Not**, Inclusion Predicate | yes | yes | | `true` keeps the record |
| **Split**, each branch predicate | yes | yes | | first `true` branch wins; unmatched records take the default branch |
| **Map**, Map Key Expression and Map Value Expression | yes | yes | | the new key and the new value |
| **Map Values** | | yes | | the new value; the key is not in scope |
| **FlatMap**, key and value expressions | yes | yes | | lists, zipped into records |
| **FlatMap Values** | yes | yes | | a list, one record per element, same key |
| **Select Key** | yes | yes | | the new key |
| **Group By** | yes | yes | | the grouping key |
| **Group Table By**, key and value expressions | yes | yes | | the grouping key and value |
| **Foreach** / **Peek**, Action Expression | yes | yes | | discarded; a throw is logged and the record flows on |
| **Enrich from a service**, Request | yes | yes | | what the service is sent, as a JSON body |
| **Enrich from a service**, Answer Mapping | yes | yes | `answer`, the service's answer, a root like `value` | what is added under the answer field |
| **Aggregate** / **Cogroup**, Initializer | | | | the empty accumulator: a constant, a literal such as `{'total': 0}`, `T(...)` or a `#function` |
| **Aggregate** / **Cogroup**, Adder and Subtractor | grouping key | the new record | `#aggValue`, the accumulator | the new accumulator |
| **Reduce**, Adder and Subtractor | | the new record | `#aggValue`, the accumulated value | the new accumulated value |
| **Join** (every kind), Value Joiner | | | `#leftValue`, `#rightValue` | the joined value |
| **Join** with a global table, Key Value Mapper | yes | yes | | the key to look up |
| **Foreign-Key Join**, Foreign Key Extractor | | the left table's value | | the referenced key |

Two of these bite people:

* **Map Values has no `key`.** The runtime evaluates it against the value alone. To read the key
  while changing the value, use **Map** with the key expression set to `key`.
* **The initializer has no record.** Kafka calls it once per new key with no arguments, so
  `value.get('quantity')` there has nothing to read. Put the first record's contribution in the
  adder, which runs for it like for every other record.

## What each serde hands you

The deserializer's own object reaches your expression, for `key` and `value` alike. There is no
conversion layer, so what the preview shows is what runs.

| Serde | `key` / `value` is | Reach into it with | A field that is not there |
| - | - | - | - |
| String | `String` | String methods: `value.toUpperCase()`, `value.contains('Ket')` | `value.get('x')` fails: a String has no fields |
| Long, Integer, Short, Double, Float | the boxed number | arithmetic and comparisons | |
| Bytes | Kafka `Bytes` | `T(...)` on `Bytes` | |
| UUID | `UUID` | `.toString()` | |
| Void | `null` | | |
| JSON, JSON Schema | a `Map` | `get('field')`, `['field']`, `containsKey(...)`, `keySet()` | **`null`** |
| Avro | a `GenericRecord` | `get('field')`; nested records chain `get(...).get(...)`; arrays are `List`s | **throws** `Not a valid schema field` |
| Protobuf | a `DynamicMessage` | `get('field')`; nested messages chain; repeated fields are `List`s; unset proto3 scalars give their defaults | **throws** `Not a valid schema field` |

**Strings inside Avro** are `org.apache.avro.util.Utf8`, not `String`: `== 'Kettle'` compares
content and works, `.contains('ttl')` does not exist on it and fails, `.toString()` first makes
every String method available. Protobuf and JSON Schema hand you real `String`s. An Avro enum
symbol compares through `.toString()`; a `timestamp-millis` field is a `long` and does
arithmetic; a `decimal` logical type is bytes to the expression and does not compare as a number.

**A missing field** follows one rule: a name the schema can reject fails at access, and a Map
answers `null`. On Avro and Protobuf an unknown name is always a typo or schema drift, because
optionality lives inside the schema, so `get('nope')` fails with `Not a valid schema field:
nope`, in the editor and deployed alike. On JSON Schema an absent key may be an optional
property, and the record is a `Map` that cannot tell the two apart, which is why the quiet
`null` is the honest answer there. The practical consequence: a filter on a misspelt field fails
visibly on an Avro or Protobuf topic and quietly drops every record on a JSON Schema one. The
editor covers that case at authoring time: a literal `get('name')` whose name is not among the
subject's declared properties gets a warning on the literal, with the declared names and the
closest one. A warning, not an error, because a schema with `additionalProperties` may carry
names the subject does not list. Before Alginte 0.13.0, Protobuf answered `null` too.

## Windowed keys

After a **Tumbling**, **Hopping**, **Sliding** or **Session Window** and its aggregate, `key` is
Kafka's `Windowed` object all the way through **To Stream** and beyond, not the plain key. It
has to be unwrapped before a sink, because no key serde here can serialize a `Windowed`: deployed
as it is, the stream would die on the first record. Alginte refuses that graph, twice: the sink
node grows an error badge while you draw, and the deploy is rejected with the same message, both
naming the fix. Read the key with `key.key()`, `key.window().start()` and `key.window().end()` in
any expression that has the key in scope:

```text theme={null}
// Select Key: the plain key and the window start, as one String key
key.key() + '@' + key.window().start()

// Map: the plain key back as the key, the window into the value
key.key()                                                   // Map Key Expression
key.window().start() + '-' + key.window().end() + '=' + value   // Map Value Expression
```

Map Values cannot do this, since it does not see the key. The
[tumbling-window example](https://docs.alginte.com/streams/examples.md#3-—-tumbling-window-count-windowed) uses the Select Key
form.

## Cookbook

Each recipe shows the expression, the record before and the record after, as its corpus row runs
it. `orders` is an Avro topic (`orderId`, `customerId`, `item`, `quantity`, `priceEur`);
`shipments` is JSON Schema (`carrier`, `weightKg`); `payments` is Protobuf (`method`).

### Filter on a field

Filter, Inclusion Predicate, on `orders`:

```text theme={null}
value.get('quantity') > 1
```

`{quantity: 3, ...}` passes; `{quantity: 1, ...}` is dropped. The same predicate is what a
**Split** branch evaluates.

### Reach a nested field

Map Values, on an Avro record whose `customer` is itself a record:

```text theme={null}
{'cid': value.get('customer').get('id')}
```

`{customer: {id: 'c-1', ...}, ...}` becomes `{"cid": "c-1"}` at a JSON sink. A nested record
placed into the output map, `{'c': value.get('customer')}`, lands as a JSON object.

### Rekey by a field

Select Key, then a downstream filter on the new key:

```text theme={null}
value.get('customerId').toString()     // Select Key
key == 'c-6650'                        // a Filter downstream
```

The `.toString()` turns Avro's `Utf8` into a `String` key; without it the key still matches
`'c-6650'` and still reaches the sink, but as a `Utf8`.

### Map to a smaller record

Map Values, on `orders`, keeping three fields and computing one:

```text theme={null}
{'customer': value.get('customerId'), 'product': value.get('item'),
 'total': value.get('quantity') * value.get('priceEur')}
```

`{orderId: 'o-1', customerId: 'c-6650', item: 'Kettle', quantity: 3, priceEur: 17.0}` becomes
`{"customer": "c-6650", "product": "Kettle", "total": 51.0}`. Downstream, `value.get('total') > 50`
passes it and `value.get('total') > 100` drops it; the editor knows the computed shape and
completes `get('total')` for you, marked *(inferred)*.

### Several records from one

FlatMap Values, an inline list:

```text theme={null}
{value.get('item'), value.get('orderId')}
```

One order becomes two records, `"Kettle"` and `"o-1"`, with the same key. The word-count
example's `value.toLowerCase().split('\\W+')` is the same recipe over a String value.

### Count or sum per key

Aggregate over `orders`, grouped by key. A numeric accumulator:

```text theme={null}
0                                      // Initializer
#aggValue + value.get('quantity')      // Adder
```

Records with quantities 3 and 4 leave `7` in the store. A map-shaped accumulator, which is what
you want when the aggregate carries more than one figure:

```text theme={null}
{'total': 0}                                                  // Initializer
{'total': #aggValue.get('total') + value.get('quantity')}     // Adder
```

The initializer builds the shape; the adder reads it back with `get`. A String method on the
accumulator, `#aggValue.substring(0, 3)`, fails deployed on a map-shaped aggregate, and the
editor says so because it evaluates the adder against what your initializer really built.

### Concatenate with reduce

Reduce over a String topic:

```text theme={null}
value + '|' + #aggValue
```

`a`, `b`, `c` for one key reduce to `c|b|a`. Reduce keeps the value's type: it cannot change a
String into a map, which is what Aggregate is for.

### Join two records

Value Joiner, both sides Avro:

```text theme={null}
#leftValue.get('item') + '/' + #rightValue.get('quantity')
```

gives `Kettle/4`. Mixed serdes read each side its own way: `#leftValue.get('item') + ':' + #rightValue` with a String right side gives `Kettle:warehouse-7`, and `#rightValue.get('item')`
on that String right side fails, deployed and in the editor alike, once both join lanes are
wired. Arithmetic between two String sides, `#leftValue - #rightValue`, fails.

### Foreign-key join

Foreign Key Extractor, on the left table's value, then the joiner:

```text theme={null}
value.substring(value.indexOf(':') + 1)     // extractor on a String value like "3:Kettle"
value.get('item').toString()                // extractor on an Avro left table
#leftValue + '@' + #rightValue              // Value Joiner
```

`3:Kettle` joined to the right table's `Kettle` row gives `3:Kettle@warehouse-7`. The extractor
sees only `value`: `key` there fails.

### Unwrap a windowed key

Select Key after a windowed count, as the tumbling-window example does:

```text theme={null}
key.key() + '@' + key.window().start()
```

Seven records of key `k1` over seven seconds in 5-second windows produce counts keyed `k1@0` and
`k1@5000`, with the `Long` count as the value.

### Call your own Java

Any expression, once the function is registered ([Custom functions](https://docs.alginte.com/streams/custom-functions.md)):

```text theme={null}
{'masked': #maskEmail(value.get('customerId') + '@example.com')}
```

gives `{"masked": "c***@example.com"}`. A function that throws fails the expression deployed and
in the editor; an unregistered name fails everywhere too.

### Peek without risk

Peek, Action Expression: the result is discarded, and a throw is logged and swallowed, so a peek
can never stop the stream. That makes it a place to call a registered function for its side
effect, `#audit(key, value)`, and not a place for a guard: an expression that fails in a peek
does not hold the record back.

## The forms that do not work

Pinned as failures on every surface, so the editor reports them before you deploy:

* `value['item']` on Avro or Protobuf: indexing is a `Map` operation and those are not maps.
  It works on JSON and JSON Schema.
* `value.get('item')` on a String topic: a String has no fields.
* `value.get('priceEur').asDouble()` on Avro: the field is already a number; there is no
  `asDouble()`.
* `value.get('amount') > 10` on an Avro `decimal`: bytes do not compare as a number.
* `#newValue` in an adder: the new record is `value`; `#newValue` is not bound.
* `key` in a Map Values, `value` in an initializer, `key` in a foreign-key extractor.

## From the blog

* [The compiler was never what you
  wanted](https://www.alginte.com/blog/kafka-streams-without-codegen/): reading an Avro topic
  from Kafka Streams takes generated classes, and what they buy is a compiler, which answers a
  narrower question than the one you actually have.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.