MWE2 Workflows using Xtend

If you've ever created an Xtext language you know that the Xtext generator is configured and launched using an MWE2 workflow. The MWE2 language is a simple external DSL to define a workflow by building a graph of JavaBeans objects representing the "workflow components". In addition to the ability to very concisely create and string together JavaBeans, the language also comes with support for:

• defining properties (of type String or boolean),
• Ant style string interpolation (which can reference properties as in e.g. "../${projectName}"),
• a means to inject the properties into matching properties of the created JavaBeans,
• and a way to call other workflows (with values to use for that workflow's properties).

This pretty much sums up what you can do with the MWE2 language. This is a very limited feature set (which generally is a good thing with DSLs) and has been used with great success in the Xtext project.

Recently I thought to myself: What would an MWE2 workflow, as used in an Xtext project, look like if it were written in Xtend? Xtend is also very well suited for concisely creating object graphs and also has support for string interpolation using rich strings. As you can see in the following Xtend defined workflow it really looks quite similar to the original MWE2 file (for brevity I have skipped the import list and also many of the generator fragments):

Some explanations and comments:

• The @Binding annotation was introduced here to denote that the particular field should be injected into JavaBeans when passed to autoInject() or when obtained from getInstance(). There would certainly be other (maybe more reasonable) ways to do this, but this best resembles what an MWE2 workflow looks like.
• As you can see the += operator is used quite extensively. This actually required a bit of a hack since it only works seamlessly with EMF-style JavaBeans, which have a getter for collection valued properties (returning a mutable collection) rather than an add-method as in standard JavaBeans. To solve this I introduced an Xtend extension method returning a Collection object on which only add(Object) can be invoked and which will call the corresponding JavaBeans add-method.
• The JavaBeans are created and injected in different ways (using the normal constructor which can be injected explicitly using autoInject() or using the getInstance() method returning an injected instance). There is no particular reason for not doing it the same way everywhere.
• A static main() method is required in order to be able to launch the workflow.

Now, why would you want to do this? Actually, I don't think there is any really good reasons, this was mostly a fun toy project. MWE2 works great for most people. It can certainly be harder to debug, takes longer to launch, and has worse tooling, but it works very well and there is a smaller risk of getting something wrong. If however Xtend had existed back in the days when MWE2 was "invented" (to replace the cumbersome XML based MWE1), I think it may very well have ended up as an internal Xtend DSL instead.

I think there is actually some people who might still be interested in writing MWE2 workflows in Xtend: If you set out to implement a whole family of Xtext based DSLs you will very likely want the workflows to be synchronized between them. And you will also quite likely have some custom generator fragments you want to use. This can be done with MWE2, but with Xtend you will certainly have more options. Let's say you for instance have some languages for which you want to generate the full IDE (with editor, etc.), but also some languages for which you basically only need the parser and serializer. How would you handle this with a single MWE2 workflow? In Xtend this would basically only be a matter of declaring a property and then some if-statements around the statements where UI-only generator fragments are added. I am sure more use cases can be concieved.

Dealing with the "could not even do k=1 for decision x" error in Xtext

If you have complex Xtext grammars, not the very latest and fastest computer, and are running other CPU intensive applications while kicking off the Xtext generator, you may have come across the following error message:

    error(10): internal error: org.antlr.tool.Grammar.createLookaheadDFA(Grammar.java:864): could not even do k=1 for decision 135

Increasing the amount of memory available to the JVM is the first thing you will have to do when dealing with complex grammars. That however doesn't help in this case. That's because this error is due to a timeout inside the ANTLR generator which by default kicks in after 1000ms. After that ANTLR gives up on creating a DFA for your grammar and you may end up with the error message above.

Fortunately this timeout can be configured by passing ANTLR the -Xconversiontimeout option. In Xtext you do this by adding two antlrParam elements to the ANTLR generator fragment as in the example below (here I use a timeout of 10000ms):

    // The antlr parser generator fragment.

    fragment = parser.antlr.XtextAntlrGeneratorFragment {

        antlrParam = "-Xconversiontimeout" antlrParam = "10000"

    }

Now don't forget to add the same option to the parser.antlr.XtextAntlrUiGeneratorFragment fragment!

Working with EMaps in Xtext

This post describes how you can support EMF EMaps in your Xtext grammar.

Xtext does not provide any direct support for EMaps, but since EMaps actually are little more than ELists of Map.Entry objects (see also EMF FAQ), they are quite easy to work with.

Consider the example where we create a simple grammar to parse a properties file with sections (as supported by the Python ConfigParser) as:

[test]

database=localhost

user=myapp

password=mysecret

The grammar (deliberately kept simple) may look something like this:

grammar org.xtext.example.config.Config with org.eclipse.xtext.common.Terminals

generate config "http://www.xtext.org/example/config/Config"

Config:

    sections+=Section*;

Section:

    '[' name=ID ']' properties+=Property*;

Property:

    key=ID '=' value=(ID|STRING);

As a result we will end up with a Section class with a getProperties() method with the return type EList<Property>. But it would actually be more convenient if the return type were EMap<String, String>, as that would allow us to directly access specific properties.

It turns out to be very easy to achieve this in this case. All we have to do is add an import for the Ecore metamodel and set the return type of the Property rule to EStringToStringMapEntry which is defined in Ecore. It is also essential that the assigned features are called key and value:

grammar org.xtext.example.config.Config with org.eclipse.xtext.common.Terminals

import "http://www.eclipse.org/emf/2002/Ecore"

generate config "http://www.xtext.org/example/config/Config"

Config:

    sections+=Section*;

Section:

    '[' name=ID ']' properties+=Property*;

Property returns EStringToStringMapEntry:

    key=ID '=' value=(ID|STRING);

We were able to reuse the EStringToStringMapEntry EClass of the Ecore metamodel. Of course this only works when both the key and the value are of type EString. If either of them is an attribute of another type (e.g. EInt) or a reference we need to create our own Map.Entry class. We can either do this by not generating the config metamodel (i.e. creating it by hand and importing it) or we can add a very simple post processor.

Let's try the latter. We change the Property rule in the grammar as follows (note the new return type):

Property returns StringToTypedValueMapEntry:

    key=ID '=' value=TypedValue;

TypedValue:

    text=(ID|STRING) | number=INT;

We have now defined our own EClass StringToTypedValueMapEntry. In order for the Ecore generator to properly generate the EMap accessor we must set the EClass' instance class name to java.util.Map$Entry. So let's do that by creating a file ConfigPostProcessor.ext next to Config.xtext with the following contents:

process(xtext::GeneratedMetamodel this):

    ePackage.getEClassifier('StringToTypedValueMapEntry').setInstanceClassName('java.util.Map$Entry')

;

That's it! Note that we could just as well change the grammar so that either of the key and value features are assigned as cross references to other objects (as opposed to containment references and attributes).

Note: There is currently a bug in Xtext 1.0.1 which prevents EMaps from being parsed correctly. It works correctly in earlier versions and upcoming versions of Xtext.

Extending models in Xtext projects

If you've been working with EMF for a while you've probably come across EAdapters before and might even had an occasion to roll your own. Working with models in Xtext projects I have grown fond of working with Google Guice and having all required services injected automatically.

In this post I briefly describe how you can easily create Guice injected EAdapters for your model. These can then be used in client code which itself doesn't even have to be Guice aware.

Let's assume we'd like to define an adapter Foo in which we'd like to use the Xtext scope provider service:

public class Foo extends AdapterImpl {

    @Inject

    private IScopeProvider scopeProvider;

    public void foo() {

        // ...

    }

}

We start by defining an accompanying adapter factory which in turn also must use a Guice provider to create the adapter instances:

public class FooAdapterFactory extends AdapterFactoryImpl {

    @Inject

    private Provider<Foo> adapterProvider;

    @Override

    public boolean isFactoryForType(Object type) {

        return Foo.class == type;

    }

    @Override

    protected Adapter createAdapter(Notifier target) {

        return adapterProvider.get();

    }

    

}

Next we register this adapter factory with the model's resource set. By doing this the client code can obtain an adapter without using the adapter factory directly and also without requiring to be created by the Guice container. E.g.

        Model model = ...;

        Foo foo = (Foo) EcoreUtil.getRegisteredAdapter(model, Foo.class);

        foo.foo();

The most natural place to add the adapter factory to the resource set is in the Xtext linker, where we have  the afterModelLinked() hook:

public class MyDslLinker extends LazyLinker {

    @Override

    protected void afterModelLinked(EObject model, IDiagnosticConsumer diagnosticsConsumer) {

        registerFooAdapterFactory(model.eResource().getResourceSet());

    }

    @Inject

    private Provider<FooAdapterFactory> factoryProvider;

    private void registerFooAdapterFactory(ResourceSet resourceSet) {

        EList<AdapterFactory> adapterFactories = resourceSet.getAdapterFactories();

        if (Iterables.isEmpty((Iterables.filter(adapterFactories, FooAdapterFactory.class)))) {

            adapterFactories.add(factoryProvider.get());

        }

    }

}

As a final step we must add a binding to MyDslRuntimeModule:

public class MyDslRuntimeModule extends AbstractCodeTabRuntimeModule {

    @Override

    public Class<? extends org.eclipse.xtext.linking.ILinker> bindILinker() {

        return MyDslLinker.class;

    }

}

Now we're all set! All the client code has to do is to obtain the adapter using the EcoreUtil.getRegisteredAdapter() method as described earlier.