SpEL reference & cookbook

This page is the scan-first companion to SpEL expressions, which explains the ideas. Everything here is pinned: every scope rule and every recipe is a row in the fidelity corpus, a test suite that runs the same expression through the editor’s validator, the sample preview and a deployed topology and fails when they disagree.

The scope rules below have always been what the runtime does. Since Alginte 0.12.0 the editor enforces two of them it used to let through: an aggregate initializer has no record in scope, and Map Values sees only the value.

What each expression can see

key and value are the record at the operator’s input. #-variables are extra context some roles carry. A name that is not in scope is rejected by the editor with a hint, and would fail deployed.

Operator, field	`key`	`value`	Variables	Result
Filter / Filter Not, Inclusion Predicate	yes	yes		`true` keeps the record
Split, each branch predicate	yes	yes		first `true` branch wins; unmatched records take the default branch
Map, Map Key Expression and Map Value Expression	yes	yes		the new key and the new value
Map Values		yes		the new value; the key is not in scope
FlatMap, key and value expressions	yes	yes		lists, zipped into records
FlatMap Values	yes	yes		a list, one record per element, same key
Select Key	yes	yes		the new key
Group By	yes	yes		the grouping key
Group Table By, key and value expressions	yes	yes		the grouping key and value
Foreach / Peek, Action Expression	yes	yes		discarded; a throw is logged and the record flows on
Aggregate / Cogroup, Initializer				the empty accumulator: a constant, a literal such as `{'total': 0}`, `T(...)` or a `#function`
Aggregate / Cogroup, Adder and Subtractor	grouping key	the new record	`#aggValue`, the accumulator	the new accumulator
Reduce, Adder and Subtractor		the new record	`#aggValue`, the accumulated value	the new accumulated value
Join (every kind), Value Joiner			`#leftValue`, `#rightValue`	the joined value
Join with a global table, Key Value Mapper	yes	yes		the key to look up
Foreign-Key Join, Foreign Key Extractor		the left table’s value		the referenced key

Two of these bite people:

Map Values has no key. The runtime evaluates it against the value alone. To read the key while changing the value, use Map with the key expression set to key.
The initializer has no record. Kafka calls it once per new key with no arguments, so value.get('quantity') there has nothing to read. Put the first record’s contribution in the adder, which runs for it like for every other record.

What each serde hands you

The deserializer’s own object reaches your expression, for key and value alike. There is no conversion layer, so what the preview shows is what runs.

Serde	`key` / `value` is	Reach into it with	A field that is not there
String	`String`	String methods: `value.toUpperCase()`, `value.contains('Ket')`	`value.get('x')` fails: a String has no fields
Long, Integer, Short, Double, Float	the boxed number	arithmetic and comparisons
Bytes	Kafka `Bytes`	`T(...)` on `Bytes`
UUID	`UUID`	`.toString()`
Void	`null`
JSON, JSON Schema	a `Map`	`get('field')`, `['field']`, `containsKey(...)`, `keySet()`	`null`
Avro	a `GenericRecord`	`get('field')`; nested records chain `get(...).get(...)`; arrays are `List`s	throws `Not a valid schema field`
Protobuf	a `DynamicMessage`	`get('field')`; nested messages chain; repeated fields are `List`s; unset proto3 scalars give their defaults	`null`

Strings inside Avro are org.apache.avro.util.Utf8, not String: == 'Kettle' compares content and works, .contains('ttl') does not exist on it and fails, .toString() first makes every String method available. Protobuf and JSON Schema hand you real Strings. An Avro enum symbol compares through .toString(); a timestamp-millis field is a long and does arithmetic; a decimal logical type is bytes to the expression and does not compare as a number. A missing field is loud on Avro and silent on the other two. On Avro and Protobuf an unknown name is always a typo or schema drift; on JSON Schema an absent key may be an optional property, which is why the quiet null is the honest answer there. The practical consequence: a filter on a misspelt field fails visibly on an Avro topic and quietly drops every record on a Protobuf or JSON Schema one. Whether to make the three agree is an open decision; until it is taken, this table is the behaviour.

Windowed keys

After a Tumbling, Hopping, Sliding or Session Window and its aggregate, key is Kafka’s Windowed object all the way through To Stream and beyond, not the plain key. It has to be unwrapped before a sink, because no key serde here can serialize a Windowed: deployed as it is, the stream would die on the first record. Alginte refuses that graph, twice: the sink node grows an error badge while you draw, and the deploy is rejected with the same message, both naming the fix. Read the key with key.key(), key.window().start() and key.window().end() in any expression that has the key in scope:

// Select Key: the plain key and the window start, as one String key
key.key() + '@' + key.window().start()

// Map: the plain key back as the key, the window into the value
key.key()                                                   // Map Key Expression
key.window().start() + '-' + key.window().end() + '=' + value   // Map Value Expression

Map Values cannot do this, since it does not see the key. The tumbling-window example uses the Select Key form.

Cookbook

Each recipe shows the expression, the record before and the record after, as its corpus row runs it. orders is an Avro topic (orderId, customerId, item, quantity, priceEur); shipments is JSON Schema (carrier, weightKg); payments is Protobuf (method).

Filter on a field

Filter, Inclusion Predicate, on orders:

value.get('quantity') > 1

{quantity: 3, ...} passes; {quantity: 1, ...} is dropped. The same predicate is what a Split branch evaluates.

Reach a nested field

Map Values, on an Avro record whose customer is itself a record:

{'cid': value.get('customer').get('id')}

{customer: {id: 'c-1', ...}, ...} becomes {"cid": "c-1"} at a JSON sink. A nested record placed into the output map, {'c': value.get('customer')}, lands as a JSON object.

Rekey by a field

Select Key, then a downstream filter on the new key:

value.get('customerId').toString()     // Select Key
key == 'c-6650'                        // a Filter downstream

The .toString() turns Avro’s Utf8 into a String key; without it the key still matches 'c-6650' and still reaches the sink, but as a Utf8.

Map to a smaller record

Map Values, on orders, keeping three fields and computing one:

{'customer': value.get('customerId'), 'product': value.get('item'),
 'total': value.get('quantity') * value.get('priceEur')}

{orderId: 'o-1', customerId: 'c-6650', item: 'Kettle', quantity: 3, priceEur: 17.0} becomes {"customer": "c-6650", "product": "Kettle", "total": 51.0}. Downstream, value.get('total') > 50 passes it and value.get('total') > 100 drops it; the editor knows the computed shape and completes get('total') for you, marked (inferred).

Several records from one

FlatMap Values, an inline list:

{value.get('item'), value.get('orderId')}

One order becomes two records, "Kettle" and "o-1", with the same key. The word-count example’s value.toLowerCase().split('\\W+') is the same recipe over a String value.

Count or sum per key

Aggregate over orders, grouped by key. A numeric accumulator:

0                                      // Initializer
#aggValue + value.get('quantity')      // Adder

Records with quantities 3 and 4 leave 7 in the store. A map-shaped accumulator, which is what you want when the aggregate carries more than one figure:

{'total': 0}                                                  // Initializer
{'total': #aggValue.get('total') + value.get('quantity')}     // Adder

The initializer builds the shape; the adder reads it back with get. A String method on the accumulator, #aggValue.substring(0, 3), fails deployed on a map-shaped aggregate, and the editor says so because it evaluates the adder against what your initializer really built.

Concatenate with reduce

Reduce over a String topic:

value + '|' + #aggValue

a, b, c for one key reduce to c|b|a. Reduce keeps the value’s type: it cannot change a String into a map, which is what Aggregate is for.

Join two records

Value Joiner, both sides Avro:

#leftValue.get('item') + '/' + #rightValue.get('quantity')

gives Kettle/4. Mixed serdes read each side its own way: #leftValue.get('item') + ':' + #rightValue with a String right side gives Kettle:warehouse-7, and #rightValue.get('item') on that String right side fails, deployed and in the editor alike, once both join lanes are wired. Arithmetic between two String sides, #leftValue - #rightValue, fails.

Foreign-key join

Foreign Key Extractor, on the left table’s value, then the joiner:

value.substring(value.indexOf(':') + 1)     // extractor on a String value like "3:Kettle"
value.get('item').toString()                // extractor on an Avro left table
#leftValue + '@' + #rightValue              // Value Joiner

3:Kettle joined to the right table’s Kettle row gives 3:Kettle@warehouse-7. The extractor sees only value: key there fails.

Unwrap a windowed key

Select Key after a windowed count, as the tumbling-window example does:

key.key() + '@' + key.window().start()

Seven records of key k1 over seven seconds in 5-second windows produce counts keyed k1@0 and k1@5000, with the Long count as the value.

Call your own Java

Any expression, once the function is registered (Custom functions):

{'masked': #maskEmail(value.get('customerId') + '@example.com')}

gives {"masked": "c***@example.com"}. A function that throws fails the expression deployed and in the editor; an unregistered name fails everywhere too.

Peek without risk

Peek, Action Expression: the result is discarded, and a throw is logged and swallowed, so a peek can never stop the stream. That makes it a place to call a registered function for its side effect, #audit(key, value), and not a place for a guard: an expression that fails in a peek does not hold the record back.

The forms that do not work

Pinned as failures on every surface, so the editor reports them before you deploy:

value['item'] on Avro or Protobuf: indexing is a Map operation and those are not maps. It works on JSON and JSON Schema.
value.get('item') on a String topic: a String has no fields.
value.get('priceEur').asDouble() on Avro: the field is already a number; there is no asDouble().
value.get('amount') > 10 on an Avro decimal: bytes do not compare as a number.
#newValue in an adder: the new record is value; #newValue is not bound.
key in a Map Values, value in an initializer, key in a foreign-key extractor.

Get Started

Installation

Streams builder

Guides

Operate

Editions

SpEL reference & cookbook

What each expression can see

What each serde hands you

Windowed keys

Cookbook

Filter on a field

Reach a nested field

Rekey by a field

Map to a smaller record

Several records from one

Count or sum per key

Concatenate with reduce

Join two records

Foreign-key join

Unwrap a windowed key

Call your own Java

Peek without risk

The forms that do not work

​What each expression can see

​What each serde hands you

​Windowed keys

​Cookbook

​Filter on a field

​Reach a nested field

​Rekey by a field

​Map to a smaller record

​Several records from one

​Count or sum per key

​Concatenate with reduce

​Join two records

​Foreign-key join

​Unwrap a windowed key

​Call your own Java

​Peek without risk

​The forms that do not work

What each expression can see

What each serde hands you

Windowed keys

Cookbook

Filter on a field

Reach a nested field

Rekey by a field

Map to a smaller record

Several records from one

Count or sum per key

Concatenate with reduce

Join two records

Foreign-key join

Unwrap a windowed key

Call your own Java

Peek without risk

The forms that do not work