Disclaimer
In this post, I assume:
- You know a little bit about Rust.
- You know a little bit about Zig.
I'll be using my own syntax to express ideas from both of these languages.
Why "types are just values" doesn't work
As programmers, we like it when our programs run well.
Type systems are there to help us with that. They track the types of values in our programs before we run them, which not only saves us from runtime crashes but enforces guidelines for writing good code.
Normally, type syntax (and logic) is distinct from the rest of a programming language. A rare exception to this rule is Zig, where types are just treated as values, and compile-time (comptime) operations on those values are allowed. That's all the type system is.
In a normal language, the List[Int]
type is a very abstract idea. But in Zig, List(Int)
is just a regular function call where you're inputting a Type
and getting a Type
back. Simple, right?
Well, if we can make a List
function, why not a ListIfOddLength
function that's a bit more powerful?
ListIfOddLength = (T: Type) => Type: (
if (T.to_string.length % 2 == 1) then List(T) else T
)
Okay, now let's make a generic function using it. (Square brackets introduce a type parameter.)
weird_function = [T](value: T) => ListIfOddLength(T): (
...
)
The function above takes a value with some type (T
). If that type has an odd length, it'll return a List(T)
, and otherwise it'll just return a T
.
While it may not be useful, Zig lets us express this sort of idea. The problem is that now you're putting arbitrary logic into a type signature. To understand what type weird_function
returns, you'll need to understand exactly how ListIfOddLength
works. You'd have to know what %
means and what to_string
does.
Let's do more silly examples. Imagine that ListIfBHash(T)
turns T
into a string, then hashes it. The function usually returns T
, but it'll return List(T)
if the hash starts with a B. Or what about ListIfAmerica
, which returns a List(T)
if you're compiling from the United States?
With silly type-to-type functions like this, you lose the ability to reason about your program. This could be a line in your code:
y = weird_function(x)
And it could take some time to figure out the type of y
, even if you knew what the type of x
was. This is bad, because type systems have to work for humans.
We see values and variables as logical things. We want to be able to express what they are in a simple way, so that we can explain them to others and understand them ourselves. Letting any operation be done on any type makes this impossible and makes a type system human-unfriendly by allowing stuff like weird_function
.
Luckily for us, human-friendly type systems have already been invented. The most popular one is called Hindley-Milner (HM), and it's used by many big languages, including Swift, Rust, Scala, Haskell, OCaml, and more. (You don't need to be able to read type theory to know what HM means; just imagine the types you could write in your favorite language from that list.)
Hindley-Milner doesn't treat types as values; it treats them as abstract things that are limited to being (possibly generic) structs or enums.
By giving up that flexibility:
- No more
weird_function
orListIfBHash
allowed. - All legal type signatures can be expressed without arbitrary logic.
- Very strong type inference is supported, to the point where you almost never have to specify types.
However, Zig's comptime and type system allow for some capabilities that most HM languages can't replicate. And the ones that can (like Rust) use macros to do it, which introduces a second syntax and a whole lot of confusion.
I've seen people say Zig's features don't mix with a human-friendly type system, but I just don't think that's true. My goal in this post is to show that it's possible for a language to adopt many of them while keeping the comfort of Hindley-Milner.
Running code at compile-time
Principles:
- Certain variables are known at compile-time, and the compiler should keep track of which ones are.
- Most functions can be run at compile-time.
- A few functions can only be run at compile-time.
- Programmers should have to explicitly run things at compile-time. The language shouldn't do it for them.
(Keep in mind that when I call things "variables" in this blog post, I don't mean they can change. This stuff works best for a language with mostly immutable variables.)
In my opinion, the compiler should keep track of comptime-known variables, but not as part of the type system. The basic rules are simple: number/string literals are comptime-known, but function results are not.
// known
eight = 8
// not known
eight_fac = factorial(8)
An expression can be forced with @
to make its result known at compile-time.
// known
eight_fac = @(factorial(8))
// shorter syntax
eight_fac = @factorial(8)
// another short syntax
eight_fac @= factorial(8)
These would be the same as typing eight_fac = 40320
(assuming the factorial function works correctly).
A function can only be run at compile-time if all of its arguments are comptime-known.
The nice part of this is that none of the types changed: factorial(8)
is still an Int
whether you run it at compile-time or runtime. This means you can add compile-time execution to a Hindley-Milner language without messing the type system up.
Of course, there are still details to be considered. Reading files should probably be allowed at compile-time, but writing to them shouldn't be. Maybe printing should go to a separate output so it doesn't interfere with compiler messages. Should generating random numbers be allowed? These are all valid questions worth exploring. In any case, most functions could be run at compile-time.
So why would you ever want to call a function at compile-time if it'll return the same thing at runtime anyway? Let's do some examples. Imagine this function exists:
Regex::parse: (String) -> Option[Regex]
Imagine you're hardcoding some regexes in your program, and you want to figure out whether they're valid before you run the part of the program that parses and uses them. Well, for that, you'd just run parse
at compile-time. Simple.
// crashes at runtime
regex = Regex::parse("hi[").or_crash
// crashes at compile-time
regex @= Regex::parse("hi[").or_crash
Or imagine there's a Timestamp
type. It has a format function which turns it into a string using some formatter like %Y-%m-%d
.
Timestamp::format: (Timestamp, String) -> String
You want to know whether your formatter is valid at compile-time, not when it fails to parse during a daily loop and brings the program down with it (true story).
But if the library is modeled like this, it's impossible, because there's no good way to check the formatter (the second argument) at compile-time. You can run the whole function at compile-time, but that requires you to know the timestamp too.
To make this sort of thing possible, library developers need to split logic up:
Formatter::new: (String) -> Option[Formatter]
Timestamp::format: (Timestamp, Formatter) -> String
This is basically the Rust newtype idiom, but on steroids, because now you don't even have to wait until runtime to know whether your wrapped type is valid.
get_date = () => String: (
formatter @= Formatter::new("%Y-%m-%d").or_crash
Timestamp::now().format(formatter)
)
If you wrote an invalid formatter, it would have failed to compile. The fact that the program compiles means get_date
should run properly, which is a pretty nice guarantee.
We also need the ability to declare a function that can only be called at compile-time. This will especially come in handy later. Let's add an @
in front of the parameter list for this.
Within a comptime-only function, the arguments are comptime-known, and comptime-only actions are allowed. It's like the whole body is wrapped in an implicit @()
too.
normal_add = (a: Int, b: Int) => a + b
comptime_only_add = @(a: Int, b: Int) => a + b
// these are fine:
@normal_add(1, 2)
@comptime_only_add(1, 2)
normal_add(1, 2)
// this is not allowed:
comptime_only_add(1, 2)
I think normal_add
and comptime_only_add
should have the same type signature. This is because comptime-only functions are very rare and treating them differently isn't worth the hassle. The compiler just needs to make sure that comptime-only functions can't make it to runtime.
If you're designing a library, should you make Regex::parse
and Formatter::new
comptime-only functions? No, because there are valid reasons for them to be called at runtime, like when an app has a search bar that supports regex. You should only declare comptime-only functions when you absolutely need to.
In the next section, we'll see some reasons why we might want to.
Types and type objects
Principles:
- Both the compiler and the programmer should be able to reason about types.
- Every location in the source code should have a logical type.
- You should be able to reference a type from anywhere in the program.
- Types shouldn't exist at runtime.
When you really think about it, a type has two main parts: structure and identity.
struct Person (
name: String,
age: Int
)
Here, Person
is a type. We know from its structure that it has a string name and an int age. We know from its identity that it's a Person
and therefore not equal to a Dog
, which could also have a string name and int age.
There are type systems (structural) where the identity of the type doesn't matter at all, but they're a little confusing in my opinion. So we'll stick with a nominal type system like Rust's.
Now let's back up and think about what types really are.
The truth is that certain things in a language are just not values:
- Types
- Traits/interfaces
- Modules
Yes, you heard it here first: types are not values. Especially if you want the compiler to do logic and proofs with them, like Hindley-Milner requires. Zig tried having types be values, and it led to stuff like weird_function
where you don't know what type things are until you compile your program, which defeats the whole purpose.
We can still gain a little flexibility by decoupling the structure of these non-values from their identity. We'll write structure separately:
struct (
name: String,
age: Int
)
enum (
North, South, East, West
)
trait (
to_string: (Self) -> String
)
module (
pi = 3.141,
e = 2.718
)
Every single one of the things above is actually a value, for now. Each has type Abstract
, which has no methods or fields or anything; it's completely opaque, tracked by the compiler only.
Think of an Abstract
value as an elaborate, colorful machine, but one that doesn't work until you plug it into an outlet.
struct[T] (
value: T,
next: Option[Self],
)
So how do you plug it into an outlet?
LinkedListNode <- struct[T] (
value: T,
next: Option[Self],
)
We'll introduce a new "plug-in" operator, <-
, and only allow it to be used at the top level of modules. By doing this, we ensure that every plugged-in thing has only one path in the source code. Now we have both pieces of the puzzle: structure and unique identity.
With this, types and modules become valid. Abstract
values ascend to the kingdom of the compiler to become ideas that can be used and have proofs done on them.
A language should also give the ability to do the reverse: create values from non-values. We can introduce a comptime-only type object (of type TypeInfo
) that gives information about a type. A type object can have some variants, including:
Struct
Enum
Function
Generic
(For stuff like empty lists and generic functions. Not common, I'd imagine.)
You can get a TypeInfo
by using a type somewhere that expects a value, like in a function call. Using a generic type as a value makes it into a (TypeInfo) -> TypeInfo
or similar function (a comptime-only one, of course).
Now let's imagine the following functions exist in the language:
TypeInfo::fields: (TypeInfo) -> List[Field]
Field::new: (String, TypeInfo) -> Field
Field::name: (Field) -> String
Field::type: (Field) -> TypeInfo
// we will talk about how this could be implemented later
Abstract::struct_from_fields: (List[Field]) -> Abstract
With this, we can write some interesting stuff. For example, we can recreate Typescript's Partial
utility type.
Partial = @(T: TypeInfo) => Abstract: (
new_fields = T.fields.map((f) =>
Field::new(f.name, @Option(f.type))
)
Abstract::struct_from_fields(new_fields)
)
You can make a PartialPerson
by plugging it in at the top level.
PartialPerson <- @Partial(Person)
And it's as if you wrote this:
PartialPerson <- struct (
name: Option[String],
age: Option[Int]
)
What you cannot do is write a function that takes some value: T
and returns a Partial(T)
. Because how would you write code for that anyway? Hindley-Milner just doesn't support it.
Some other stuff you could write:
IntTuple = @(size: Int) => Abstract: (
if (size < 0) then crash("size must be >= 0")
fields = range(0, size).map((n) =>
Field::new("_" ++ n.to_string, Int)
)
Abstract::struct_from_fields(fields)
)
Omit = @(T: TypeInfo, to_omit: List[String]) => Abstract: (
good_fields = T.fields.filter((f) => !to_omit.contains(f.name))
Abstract::struct_from_fields(good_fields)
)
Or you could make a function that reads a Protobuf file and turns it into a real type, or a function that does the struct-of-arrays transformation to save memory, much like Zig's MultiArrayList
. And I'm sure there are many more practical uses that I'm just not creative enough to think of.
Person <- @ProtobufType("./types/person.proto")
EfficientPersonArray <- @MultiArrayList(Person)
The possibilities are pretty much endless.
(Note: TraitInfo
and ModuleInfo
could exist too. I just won't talk about them in this post.)
Code and code objects
Principles:
- Sometimes you need to manipulate source code at compile-time.
- Code doesn't have to make sense until it's parsed.
- It shouldn't matter where in the program a piece of source code is parsed.
- When you need an escape hatch for 1% of cases, you want it to be powerful enough to work in all 1% of them.
There are still a couple of Zig features we can't emulate yet, like turning a struct into JSON. To get there, we'll add code objects (Code
), which are comptime-only strings that describe source code. You're free to model them as ASTs, but I think that encourages devs to overuse them.
We'll use backticks to create code objects.
my_code: Code = `{1, 2, 3}`
Then we'll provide a (comptime-only) parse
function that parses code.
parse: [T](Code) -> T
Note how the return type of parse
is generic, Hindley-Milner style. That means the compiler needs to be able to infer the return type of parse
before ever running it. The value of the code object never affects the type, which makes things easy to reason about for the developer. If the parsed type doesn't match the expected type, that's a compile-time error.
Since code objects must be parseable anywhere, they can only reference external values that are global (exist everywhere in the program).
However, keep in mind that the validity of code is only checked when parsing it.
valid_code = `(x) => x + x`
// parse the function, then call it
// hindley-milner can infer the output of parse to be a () -> Int
result = @parse(code)(8)
expect(result == 16)
invalid_code = `() => foo`
invalid_code2 = `!@#$&!@#&*($#@!`
// no crash, because the invalid code objects were never parsed
Much like to_string
, (most) types can provide a to_code
method, which turns an object into code that, when parsed, returns an identical object.
expect(1.to_code == `1`)
expect("hello".to_code == `"hello"`)
expect(Person.to_code == `::module::submodule::Person`)
Look at the last line there. Type objects always come from types, and all types have a specific location, so a TypeInfo
can store information about the type's location and use that to generate an absolute path for it, which is the same anywhere in the program.
Therefore, type objects can be inserted into code objects, where they can be real types again. Let's use this to write Abstract::struct_from_fields
. We'll let \()
be the interpolation syntax.
struct_from_fields = @(fields: List[Field]) => Abstract: (
field_lines = fields.map((f) =>
`\(f.name): \(f.type.to_code),`
)
code = `struct (
\(field_lines.join(`\n`))
)`
parse(code)
)
Why no type annotation on parse
? Well, the compiler can figure out that you expect to parse an Abstract
by looking at the return type.
(Note: String-style code generation is significantly easier when a language has insignificant whitespace and indentation.)
Doing cool stuff
First: a function that sums up all the Int
fields in a struct. We do this by making a list of functions that get the Int
fields.
// standard library could provide this
getter = @(f: Field) => Code: (
`(x) => x.\(f.name)`
)
sum_int_fields = [T](object: T) => Int: (
getters @= T.fields
.filter((f) => f.type == Int)
.map(getter)
.map(parse[(T) -> Int])
getters.map((fn) => fn(object)).sum
)
You can imagine that something like this is generated for sum_int_fields[Person]
:
sum_int_fields = (object: Person) => Int: (
getters = {(x) => x.age}
getters.map((fn) => fn(object)).sum
)
A little brain-breaking? Yes, and I apologize. But good luck writing something like that with Rust macros.
Let's do another cool thing: say we have an enum and we want to make a function that turns it into a super short string. We want our generated function to look something like this:
Color = enum (
Red,
Green,
Blue,
Yellow,
Cyan,
Magenta,
White,
Orange,
Purple
)
to_short_string = (self: Color) => String: (
match self (
Red => "R",
Green => "G",
Blue => "B",
Yellow => "Y",
Cyan => "C",
Magenta => "M",
White => "W",
Orange => "O",
Purple => "P"
)
)
Even writing that match statement out as an example was exhausting. Why do that when this code could do the same thing for any enum?
to_short_string = [T](object: T) => String: (
branches @= T.variants.map((v) =>
short = v.name.slice(0, 1).uppercase
`\(v.name) => \(short.to_code),`
)
match_code @= `(x) => match x (
\(branches.join(`\n`))
)`
@parse(match_code)(object)
)
The real power of this version is its flexibility. If you wanted, you could add complex logic to detect when two variants start with the same letter, and extend the string if so. So for example, Brown
would become BR
while Blue
would become BL
. You wouldn't have to remember to update them manually.
(Note: A good compiler for a language like this would have a feature where you could see what to_short_string[Color]
becomes.)
Adding traits too
There's no reason you couldn't add Rust-style traits to this language.
HasArea = trait (
area: (Self) -> Int
)
double_area = [T: HasArea](shape: T) => Int: (
shape.area * 2
)
Without the T: HasArea
bound, shape.area
would fail to compile. This makes function signatures more descriptive. However, things in code objects shouldn't be restricted by traits. So the following must be allowed:
double_area = [T](shape: T) => Int: (
// will probably fail to parse for many types
@parse(`(x) => x.area * 2`)(shape)
)
This is unfortunate, but otherwise it would just be impossible to make many of our code-generating functions.
I think a good way to end this post is by putting it all together to make a trait called Stringable
where the default implementation of to_string
is automatically generated for structs.
entry_code = @(f: Field) => Code: (
`(x) => "\(f.name): " ++ x.\(f.name).to_string`
)
Stringable = trait (
to_string = (object: Self) => String: (
// this is just a default implementation
getters @= Self.fields.map(entry_code)
entries = getters.map((fn) => fn(object)).join(", ")
"{" ++ entries ++ "}"
)
)
As usual, we can show an example of what this would expand to. For our Person
struct, all we'd have to write is this:
Person = struct (
name: String,
age: Int,
impl Stringable
)
And it would be as if we implemented this manually:
Person = struct (
name: String,
age: Int,
impl Stringable (
to_string = (object: Person) => String: (
getters = {
(x) => "name: " ++ x.name.to_string,
(x) => "age: " ++ x.age.to_string
}
entries = getters.map((fn) => fn(object)).join(", ")
"{" ++ entries ++ "}"
)
)
)
// example of using it
// this will print {name: "Derin", age: 17}
author = Person::{
name = "Derin",
age = 17
}
print(author.to_string)
The nice thing about using a trait rather than a standalone function is it allows types to override the default behavior. Also, the language could implement certain traits by default, if they're really useful (like equality or stringification, if that's even a word).
Recap
Basically, we introduced four big features:
@
Abstract
TypeInfo
andField
Code
andparse
With these, we replaced pretty much all Rust macros, including derive
ones, while keeping Rust's type system. We also got (most of) Zig's comptime features and made metaprogramming look somewhat like normal programming. I'd call that a big win.
I'm not claiming this is some perfect solution that fixes everything; it's just food for thought. But I hope someone out there can take these ideas and make something cool out of them.