This month and a half in Rune

A bit more than one month ago I announced Rune 🥳. And the response so far has been amazing.

A lot of stuff has happened since then, so in this post I'll detail some of the developments since the initial announcement.

This is also an announcement for Rune 0.7. But the exact number shouldn't be taken too seriously given the experimental state of the project right now. In the future we'll be trying to bake a release roughly once every month which might or might not include breaking changes.

For those of you new to the project, Rune is an open source embeddable dynamic programming language that compiles and runs on a virtual machine called Runestick. It is designed to feel like Rust without types, and be about as fast and convenient to use as Lua. You can read about it in the foreword to the book which also explains why I started making a new programming language.

Feel free to Discuss this on Reddit.

Welcome to the playground

You might have noticed that this blog post contains runnable sections of code, like this:

const NAME = "Friend"; pub fn main() { println!("Hello, {}", NAME); }
A code section you can edit

This is used to power the Rune playground. A place that allow you to fiddle with Rune online and share code snippets with others.

In order to accomplish this, we made sure Rune could compile and run on WebAssembly. And introduced a module which provides an interface to the compiler.

The content of these snippets are currently stored in the URL, so try to keep them small for others' sake!

Modules and visibility

We've taught rune to expand modules and respect visibility rules. This is a nice feature that enabled better encapsulation and it brings the capabilities of Rune more in line with Rust.

You can see basic modules in action with the following test case borrowed from the Rust reference book (ignore the unused warnings 😉):

mod crate_helper_module { pub fn crate_helper() {} fn implementation_detail() {} } pub fn public_api() {} pub mod submodule { use crate::crate_helper_module; pub fn my_method() { crate_helper_module::crate_helper(); } fn my_implementation() {} mod test { fn test_my_implementation() { super::my_implementation(); } } } pub fn main() { submodule::my_method(); }
Basic modules and visibility smoke test

This means that we've had to extend the existing bare bones system so that it understands how to perform, and cache recursive imports and their visibility. At this point it doesn't behave exactly like Rust. One example of this is that conflicting wildcard imports simply override each other instead of being marked as ambiguous:

mod a { pub struct Foo; } mod b { pub struct Foo; } use {a::*, b::*}; pub fn main() { Foo is b::Foo }
The last wildcard import wins

But we do have the basic rules down as outlined in the reference.

  1. If an item is public, then it can be accessed externally from some module m if you can access all the item's ancestor modules from m. You can also potentially be able to name the item through re-exports. See below.
  2. If an item is private, it may be accessed by the current module and its descendants.

What hasn't been implemented yet is the separation between between the extern prelude and crate root, as described in this excellent post by Koxiaet. This would require refactoring some tricky parts of the module system, but it is on the roadmap.

Macros

We've taught Rune how to parse and execute macros. Macros are compile-time procedures which consumes one token stream and transforms it to produce another in its place.

This is of course a bit complicated. You have to take care that macro expansion happens in a manner that is well integrated into the rest of the compilation. Too early, and the items produces in the macro for example can't see imports. Too late, and it can't produce them. There are still a few things that need to be figured out. But it's shaping up pretty well.

Rune support for macros is currently experimental. Macros are restricted to native modules. This makes matters easier, because native modules are written in Rust and are therefore compiled before any Rune program using them. Neatly getting around the issue that you have to compile the macro before it can be run.

We've tried to learn about macros from Rust. Parsing in Rune has been designed in a way so that it can be re-used directly within macros, similarly to what you get with the syn crate. We also provide our own version of the quote! macro to ergonomically produce token streams.

The following is an example macro that comes with the std::experiments crate. It translates "stringy math" into rune expressions:

use rune::ast;
use rune::macros;
use rune::{quote, Parser, Spanned, TokenStream};
use runestick::SpannedError;

/// Implementation for the `stringy_math!` macro.
pub(crate) fn stringy_math(stream: &TokenStream) -> runestick::Result<TokenStream> {
    let mut parser = Parser::from_token_stream(stream);

    let mut output = quote!(0);

    while !parser.is_eof()? {
        let op = parser.parse::<ast::Ident>()?;
        let arg = parser.parse::<ast::Expr>()?;

        output = match macros::resolve(op)?.as_ref() {
            "add" => quote!((#output) + #arg),
            "sub" => quote!((#output) - #arg),
            "div" => quote!((#output) / #arg),
            "mul" => quote!((#output) * #arg),
            _ => {
                return Err(SpannedError::msg(
                    op.span(),
                    "unsupported operation",
                ).into())
            }
        }
    }

    parser.eof()?;
    Ok(output.into_token_stream())
}

Macros are intended to be a stable bedrock for language extension. To support use-cases which can use a bit of custom syntax or behavior glued into a project. A nice example of this in Rust is Rocket, which uses macros to great effect to improve the ergonomics of writing web services. The hopes are that macros can be used to provide similar experiences where appropriate in Rune.

The current macro system is also being dogfooded to provide a couple of utility macros that Rust developers would expect like println!, which will be covered in the next section.

println! and FormatArgs

In Rust, when you want to print something to stdout you can reach for the println! macro.

println!("Hello {:>12}", "World");

The first argument in this macro is called a format string. And combined it provides a convenient way for performing common text formatting operations in Rust. Now Rune can also use a limited form of println!, and format arguments in general.

pub fn main() { println!("Hello {:>12}", "World"); }
Formatting with println!

To implement string formatting like this we've added FormatArgs. A type that implements Parse that can be used to add message formatting to any macro. The full implementation of the println! is simply using it to format a string which is passed to std::io::println.

fn println_macro(stream: &TokenStream) -> Result<TokenStream> {
    let mut p = Parser::from_token_stream(stream);
    let args = p.parse_all::<macros::FormatArgs>()?;

    let expanded = args.expand()?;
    Ok(quote!(std::io::println(#expanded)).into_token_stream())
}

To accomplish this, FormatArgs is actually expanded into two internal macros:

Strictly speaking, these expansions result in valid Rune. The #[builtin] attribute modifies how the macros are looked up so that they are solely expanded at compile time into the appropriate instructions. They are intended for internal use only, so we'll probably restrict their use in the future. But for now you can simply type out the equivalent code that is being generated to get a better understanding for how they work 🙃.

use std::io; pub fn main() { io::println(#[builtin] template! { "Hello ", #[builtin] format! { "World", width = 12, align = right } }); }
Using the built-in template! and format! macros directly

This also means that the following macros now also support formatting:

constant evaluation

Work has been started to support constant evaluation. Usually all code is compiled to target the runestick virtual machine, but constant evaluation introduces a separate interpreted mode that the compiler can run directly.

A limited subset of the language is currently available in constant contexts, this includes functions and const items. Which can do the following:

Native functions are currently not visible during constant evaluation. This could be enabled, but we still need to decide which scope to limit constant evaluation to. I.e. do we want to be able to perform database requests during constant evaluation? In practice this will probably be determined selectively. Constant values are aggressively cached, so we should probably require a proof obligation that they have no side effects and leave more complex uses with potential side effects to macros.

Here's an example of what you can do today with constant evaluation:

const fn greeting(name) { `Hello {name}` } /// Define a collection of predefined greetings. const GREETINGS = [ greeting("Stranger"), greeting("Jane"), greeting("John"), greeting("Mio"), ]; pub fn main() { let rng = rand::Pcg64::new(); let greetings = GREETINGS; println(greetings[rng.int_range(0, greetings.len())]); }
Asynchronous programming using select

As a bonus, here's the Fibonacci example used in the playground as a constant function. We only need to introduce const to the fn item for it to work.

const fn fib(n) { if n <= 1 { n } else { fib(n - 1) + fib(n - 2) } } pub fn main() { fib(15) }
Asynchronous programming using select

Better iterator support

Iterators have gotten a bit of love in that they are now represented as a single consistent type called Iterator. This holds all the iterators transformation methods like map, filter, and rev. Any function producing an iterator should produce an instance of Iterator.

struct Foo { value, } pub fn main() { let values = [1, "foo", Foo { value: 42 }]; for v in values.iter().rev() { println!("{:?}", v); } }
Reversing an iterator

We've also added two collect functions: collect::<Vec> and collect::<Object>.

struct Foo { value, } pub fn main() { let values = [1, "foo", Foo { value: 42 }]; values.iter().filter(|v| v is Foo).collect::<Vec>() }
Apply filter to an iterator and collecting the result

Why two functions? Well, Rune doesn't have type annotations to select the desired output type. This solution should be considered preliminary, because this might be a good case where gradual typing might be used in the future.

IDE Support

Work has been started to provide editor support through rune-vscode. A Visual Studio Code extension that adds syntax highlighting and integration with the Rune language server.

Showcasing go to definitions

While it works pretty well for now, don't get too excited. There's still a lot of work to do. Next I'll detail the most important tasks yet to be done.

We absolutely need to support external definition files for this to work well. Right now the language server simply includes all the native modules that happen to be bundled with Rune. We don't see custom modules included elsewhere. The solution is to have support for dynamically loading separate declaration files which defines the content of native modules. Like with .d.ts files in TypeScript.

The language server is using the same compiler as Rune. Our hope is to be able to maintain the same internals and avoid as much duplication as possible. To accomplish this, we're looking closely at rust-analyzer. Specifically their approach to parsing which provides better error resilience in order to give the user a better experience when challenged with incomplete code. Something we hopefully can work incrementally towards.

Fixing a common miscompilation in Rune

The compiler has been redesigned to avoid a common root cause of miscompilations. This is such an extensive topic that it deserves its own post. But the cliff note is that the compiler has been redesigned in a way to make a class of bugs harder to introduce.

Relevant issues: #118, #127.

Contributors

A number of people have provided code, time, and extensive feedback to Rune over the last development period.

Thank you all very much for helping make Rune better!

Full Changelog

Added

Changed

Fixed