Dealing with the "could not even do k=1 for decision x" error in Xtext

If you have complex Xtext grammars, not the very latest and fastest computer, and are running other CPU intensive applications while kicking off the Xtext generator, you may have come across the following error message:

    error(10): internal error: org.antlr.tool.Grammar.createLookaheadDFA(Grammar.java:864): could not even do k=1 for decision 135

Increasing the amount of memory available to the JVM is the first thing you will have to do when dealing with complex grammars. That however doesn't help in this case. That's because this error is due to a timeout inside the ANTLR generator which by default kicks in after 1000ms. After that ANTLR gives up on creating a DFA for your grammar and you may end up with the error message above.

Fortunately this timeout can be configured by passing ANTLR the -Xconversiontimeout option. In Xtext you do this by adding two antlrParam elements to the ANTLR generator fragment as in the example below (here I use a timeout of 10000ms):

    // The antlr parser generator fragment.

    fragment = parser.antlr.XtextAntlrGeneratorFragment {

        antlrParam = "-Xconversiontimeout" antlrParam = "10000"

    }

Now don't forget to add the same option to the parser.antlr.XtextAntlrUiGeneratorFragment fragment!

Working with EMaps in Xtext

This post describes how you can support EMF EMaps in your Xtext grammar.

Xtext does not provide any direct support for EMaps, but since EMaps actually are little more than ELists of Map.Entry objects (see also EMF FAQ), they are quite easy to work with.

Consider the example where we create a simple grammar to parse a properties file with sections (as supported by the Python ConfigParser) as:

[test]

database=localhost

user=myapp

password=mysecret

The grammar (deliberately kept simple) may look something like this:

grammar org.xtext.example.config.Config with org.eclipse.xtext.common.Terminals

generate config "http://www.xtext.org/example/config/Config"

Config:

    sections+=Section*;

Section:

    '[' name=ID ']' properties+=Property*;

Property:

    key=ID '=' value=(ID|STRING);

As a result we will end up with a Section class with a getProperties() method with the return type EList<Property>. But it would actually be more convenient if the return type were EMap<String, String>, as that would allow us to directly access specific properties.

It turns out to be very easy to achieve this in this case. All we have to do is add an import for the Ecore metamodel and set the return type of the Property rule to EStringToStringMapEntry which is defined in Ecore. It is also essential that the assigned features are called key and value:

grammar org.xtext.example.config.Config with org.eclipse.xtext.common.Terminals

import "http://www.eclipse.org/emf/2002/Ecore"

generate config "http://www.xtext.org/example/config/Config"

Config:

    sections+=Section*;

Section:

    '[' name=ID ']' properties+=Property*;

Property returns EStringToStringMapEntry:

    key=ID '=' value=(ID|STRING);

We were able to reuse the EStringToStringMapEntry EClass of the Ecore metamodel. Of course this only works when both the key and the value are of type EString. If either of them is an attribute of another type (e.g. EInt) or a reference we need to create our own Map.Entry class. We can either do this by not generating the config metamodel (i.e. creating it by hand and importing it) or we can add a very simple post processor.

Let's try the latter. We change the Property rule in the grammar as follows (note the new return type):

Property returns StringToTypedValueMapEntry:

    key=ID '=' value=TypedValue;

TypedValue:

    text=(ID|STRING) | number=INT;

We have now defined our own EClass StringToTypedValueMapEntry. In order for the Ecore generator to properly generate the EMap accessor we must set the EClass' instance class name to java.util.Map$Entry. So let's do that by creating a file ConfigPostProcessor.ext next to Config.xtext with the following contents:

process(xtext::GeneratedMetamodel this):

    ePackage.getEClassifier('StringToTypedValueMapEntry').setInstanceClassName('java.util.Map$Entry')

;

That's it! Note that we could just as well change the grammar so that either of the key and value features are assigned as cross references to other objects (as opposed to containment references and attributes).

Note: There is currently a bug in Xtext 1.0.1 which prevents EMaps from being parsed correctly. It works correctly in earlier versions and upcoming versions of Xtext.

Extending models in Xtext projects

If you've been working with EMF for a while you've probably come across EAdapters before and might even had an occasion to roll your own. Working with models in Xtext projects I have grown fond of working with Google Guice and having all required services injected automatically.

In this post I briefly describe how you can easily create Guice injected EAdapters for your model. These can then be used in client code which itself doesn't even have to be Guice aware.

Let's assume we'd like to define an adapter Foo in which we'd like to use the Xtext scope provider service:

public class Foo extends AdapterImpl {

    @Inject

    private IScopeProvider scopeProvider;

    public void foo() {

        // ...

    }

}

We start by defining an accompanying adapter factory which in turn also must use a Guice provider to create the adapter instances:

public class FooAdapterFactory extends AdapterFactoryImpl {

    @Inject

    private Provider<Foo> adapterProvider;

    @Override

    public boolean isFactoryForType(Object type) {

        return Foo.class == type;

    }

    @Override

    protected Adapter createAdapter(Notifier target) {

        return adapterProvider.get();

    }

    

}

Next we register this adapter factory with the model's resource set. By doing this the client code can obtain an adapter without using the adapter factory directly and also without requiring to be created by the Guice container. E.g.

        Model model = ...;

        Foo foo = (Foo) EcoreUtil.getRegisteredAdapter(model, Foo.class);

        foo.foo();

The most natural place to add the adapter factory to the resource set is in the Xtext linker, where we have  the afterModelLinked() hook:

public class MyDslLinker extends LazyLinker {

    @Override

    protected void afterModelLinked(EObject model, IDiagnosticConsumer diagnosticsConsumer) {

        registerFooAdapterFactory(model.eResource().getResourceSet());

    }

    @Inject

    private Provider<FooAdapterFactory> factoryProvider;

    private void registerFooAdapterFactory(ResourceSet resourceSet) {

        EList<AdapterFactory> adapterFactories = resourceSet.getAdapterFactories();

        if (Iterables.isEmpty((Iterables.filter(adapterFactories, FooAdapterFactory.class)))) {

            adapterFactories.add(factoryProvider.get());

        }

    }

}

As a final step we must add a binding to MyDslRuntimeModule:

public class MyDslRuntimeModule extends AbstractCodeTabRuntimeModule {

    @Override

    public Class<? extends org.eclipse.xtext.linking.ILinker> bindILinker() {

        return MyDslLinker.class;

    }

}

Now we're all set! All the client code has to do is to obtain the adapter using the EcoreUtil.getRegisteredAdapter() method as described earlier.