Today, we are excited to announce Smithy IDL 2.1. With Smithy 2.1, you can use three new syntax features: inline collection declarations for lists and maps, tagged string literals for values that carry their own escaping rules, and a shorthand for declaring the wire order of a structure’s members.

Smithy is an open source Interface Definition Language (IDL) and set of tools for defining services. You describe a service once, and then generate clients, servers, and documentation for it in several programming languages.

In this post, we will walk through each feature with before-and-after examples, starting with inline collections.

Inline collection declarations

Consider a service that manages playlists. In IDL 2.0, every list and map is a top-level shape. The following IDL 2.0 example shows a Playlist structure where reading the member tells you its name and nothing about its contents:

$version: "2.0" namespace com.example structure Playlist { songs: SongList } list SongList { member: Song } structure Song { id: String title: String durationSeconds: Integer } 

SongList exists so that songs has something to point at. In IDL 2.1, you can write the collection in the member target position instead. In IDL 2.1, you write lists as [Target] and maps as {KeyTarget: ValueTarget}. The following IDL 2.1 example shows the updated Playlist structure:

$version: "2.1" namespace com.example structure Playlist { songs: [Song] tags: {String: String} } structure Song { id: String title: String durationSeconds: Integer } 

songs now declares its list where it is used, and tags does the same for a map without any top-level shape at all. Both forms work anywhere a member target works, including union members and the members of other inline collections, up to three levels of nesting.

Inline collections are syntax sugar, not a new kind of shape. When the assembler loads the model, each distinct inline collection becomes an ordinary list or map marked with the new smithy.api#synthetic trait. Code generators, selectors, and consumers of the JSON abstract syntax tree (AST) see nothing new.

Members share these generate shapes, so two members in the same namespace that both write [String] target the same one. The generated shape names start with a reserved _Synthetic prefix, and the rest of the name comes from the collection and its resolved element types. This naming scheme may change, so do not rely on it.

You apply traits to the member instead, and most of the traits you would commonly use here can apply to members as well, such as @length. The following IDL 2.1 example demonstrates this:

$version: "2.1" namespace com.example structure Playlist { songs: [Song] @length(max: 20) tags: {String: String} } structure Song { id: String title: String durationSeconds: Integer } 

Here @length constrains the number of entries in this member’s map, and only this member’s map. A @length on a named map shape would constrain every member that targets it, which is the shared-shape problem the member form avoids.

Inline collections do not support the @sparse and @uniqueItems traits. Declare a named shape to use these traits.

You can read the full rules in inline collection declarations in the Smithy specification.

Tagged string literals

Consider a service that has complex string validation requirements. In IDL 2.0, expressing these constraints is challenging because the @pattern trait accepts a regex string that you must escape. When the regex includes escape sequences, you must escape them twice: once fort he regex and once for the Smithy string. The following IDL 2.0 example demonstrates this:

$version: "2.0" namespace com.example @pattern("^\\d{1,2}:\\d{2}$") string TrackLength 

TrackLength writes four backslashes to express two. IDL 2.1 adds tagged string literals, where a tag prefix changes how the parser reads the string’s contents, so you can write the value the way its own notation defines it. The #re tag passes every character through with no escape evaluation. The following IDL 2.1 example shows the same TrackLength pattern:

$version: "2.1" namespace com.example @pattern(#re "^\d{1,2}:\d{2}$") string TrackLength 

The two @pattern values are identical. #re also strips newlines, so a text block lets you break a long pattern across lines instead of packing it onto one. The following IDL 2.1 model shows how you can split a pattern representing a Windows path:

$version: "2.1" namespace com.example @pattern(#re """ ^[A-Z]:\\ ([\w\s]+\\)* ([\w\s]+(\.[\w\d]+)?)?$ """) string WindowsPath 

Compare this to the following IDL 2.0 model, where every backslash in WindowsPath has to be doubled, on one line:

$version: "2.0" namespace com.example @pattern("^[A-Z]:\\\\([\\w\\s]+\\\\)*([\\w\\s]+(\\.[\\w\\d]+)?)?$") string WindowsPath 

You can use four tags in IDL 2.1:

  • #re is a regular expression literal. Every character passes through literally with no escape evaluation, and newlines are stripped.
  • #b is a binary literal. It reads the string as a sequence of bytes using hex escapes (\xHH), octal escapes (\OOO), and named escapes, then produces the base64 string that Smithy requires for binary values. #b "foo" produces "Zm9v".
  • #hex is a hex dump literal. It reads the string as hexadecimal byte values, ignores whitespace, and treats # as a comment to the end of the line. It also produces a base64 string.
  • #timestamp converts an RFC 3339 timestamp into its epoch-seconds representation, with optional millisecond precision. It is the one tag that produces a number instead of a string, so a model can say 2024-01-01T00:00:00Z while the tooling still receives 1704067200.

The following structure uses #hex and #timestamp in its @default values:

$version: "2.1" namespace com.example structure AudioFile { @default(#hex """ 66 4c 61 43 # fLaC stream marker """) header: Blob @default(#timestamp "2024-01-01T00:00:00Z") createdAt: Timestamp } 

header carries the four bytes of a FLAC stream marker with a comment explaining them, and createdAt carries an instant the loader converts to 1704067200.

Tags exist only in the IDL, so they do not nest or compose and they have no JSON AST representation. The parser resolves a tag to a plain value. If you serialize a model from the AST back to IDL, it emits the decoded string rather than the tag you wrote.

You can read the complete tag reference in the tagged string literals section of the Smithy specification.

Member index shorthand

Some binary protocols serialize members in a predetermined order. When the protocol knows a member’s position, it can identify the member with a compact integer or leave the identifier out of the payload entirely. Either way, the protocol needs a member numbering that does not shift when you reorder or transform the model.

You now model that numbering with two new traits. The smithy.protocols#idx trait assigns an index to a member. smithy.protocols#indexed trait marks a service whose every non-mixin structure and union in its closure must have indexed members.

As this IDL 2.0 example shows, applying the trait to every member by hand is verbose:

$version: "2.0" namespace com.example use smithy.protocols#idx structure Song { @idx(1) id: String @idx(2) title: String @idx(3) durationSeconds: Integer } 

You can use an ordinal-style shorthand in IDL 2.1, as this example shows:

$version: "2.1" namespace com.example structure Song { 1. id: String 2. title: String 3. durationSeconds: Integer } 

The two models are equivalent. The shorthand needs no use statement, but either form needs the smithy-protocol-traits model on the build path. With the Smithy CLI, add it to smithy-build.json:

{ "version": "1.0", "maven": { "dependencies": ["software.amazon.smithy:smithy-protocol-traits:<RELEASE_VERSION>"] } } 

With the Gradle plugin, add the same coordinate to your build script’s dependencies.

Declaration order has no effect on the indexes, so you can add a member wherever it reads best and give it the next index. If any member of a structure or union has an index, every member must have one.

A mixin does not index its own members. The shape that consumes it does, using the elision syntax, as in 1. $id. Members that have the @eventHeader or @eventPayload trait, and members that target a streaming blob or streaming union, don’t require an index.

Member indexes define stable ordering metadata but do not define a wire encoding, and each protocol decides what to do with them. No generally available protocol reads idx today, so this matters to you only if you are targeting a protocol that gives indexes a meaning. The traits were first published alongside Sparrowhawk, a compact binary format with no published release yet. IDL 2.1 defines them in Smithy itself, in the smithy-protocol-traits package.

Both traits are new in this release, but neither depends on IDL 2.1. An IDL 2.0 model can use smithy.protocols#idx and apply it explicitly, and only the shorthand requires 2.1. The Smithy CLI command, smithy format rewrites explicit @idx traits in 2.1 models into the shorthand, and the model serializer emits the shorthand by default. The formatter never changes a file’s declared version, so you must change the version first.

You can read the full rules in the indexed shapes section of the Smithy specification.

Adopting IDL 2.1

Adoption is per file. To activate the new syntax, change the version declaration at the top of each file to "2.1". This IDL snippet shows the required change:

$version: "2.1" namespace com.example 

You only need to change the version declaration to use the new syntax. Models that load under 1.0 or 2.0 continue to work. Code that you generate from unchanged models stays the same, because inline collections resolve to ordinary shapes. If you are still on IDL 1.0, the Smithy CLI smithy migrate can move the file to 2.1 for you.

Use caution when you convert an existing named collection to inline syntax. The member now targets a generated shape, so the original shape ID disappears. Anything outside the file that referenced it by name needs to be updated first. This includes selectors, trait applications, and code generators that derive names for lists and maps from shape names.

The JSON AST format has no changes. All three features resolve into constructs the AST already had.

Smithy IDL 2.1 does not introduce any breaking changes. A model that works with IDL 2.0 also works with 2.1. None of the new syntax changes the shapes a model produces. Code generators require no changes unless they validate the version string or derive type names from shape names.

Recap

In this post, we covered what IDL 2.1 adds:

  • Inline list and map declarations in the member target position.
  • Four tagged string literals for regular expressions, binary values, hex dumps, and timestamps.
  • A shorthand for the idx trait.

The JSON AST is unchanged, and every existing model keeps working. To see the complete list of changes in this release, review the changelog.

Next steps

  • Already on Smithy IDL 2.0 and want to adopt 2.1? Change $version to "2.1" in the files you want to convert, then run the Smithy CLI smithy format command. You can convert files one at a time, and you can stop at any point.
  • If you’re new to Smithy, start with the quick start guide, then read the IDL specification.
  • Let us know how you use IDL 2.1. For feature requests, bug reports, contributions, or feedback of any kind, head over to the Smithy repository on GitHub, or leave a comment below.

Much of this release grew from feedback on GitHub. Inline collections and readable binary literals were both asked for in issues. Your feedback in those threads directly shaped the final syntax. Thanks to all of the contributors who file those issues, review the changes, and help make Smithy better.