Language Workbenches Evaluation And Comparison

Published Date: 02 Nov 2017

In this section we will give an evaluation of the DSL workbenches we have tested in in this work. The evaluation is based on the experience gained during the implementation of the already presented and discussed DSL prototypes at earlier stages of this thesis.

Xtext, as an Eclipse plug-in, is based on a widespread application and can benefit from many of its functions and frameworks. Thus, the ability to extend the development environment leaves enough scope for the requirements of complex DSLs. Xtext is the only representative in this comparison that relies on the concept of programming in free text and on the resulting use of lexer and parser. However, thanks to this use of free text, the DSL development in Xtext is easy to learn for programmers. On the other hand, it remains questionable whether domain specialist can get used to such a development environment.

Strengths

Xtext benefits from its relatively fast learning curve: the DSL designer simply needs to get familiar with the EBNF-based grammar language and DSL definition becomes an easy task. This is partly due to the fact that Xtext is completely based on coding in free text and secondly to the fact that a very efficient language for programming code generators is available, namely Xtend.

Another advantage is the use of Eclipse. This latter IDE is widely used and therefore no new development environments have to be learned for creating DSLs. Positive is also the fact that the project wizard immediately supplies the DSL project with all the necessary configurations for the development and the creation of the language.

The Xtend language used for model-to-code transformation makes as well a very good impression especially by the deployment of rich strings. These enable simple commands to be integrated inside strings and to produce a clean as well as readable code.

Weaknesses

The flexibility that comes with the pure string processing by Xtend when generating code from the model brings also disadvantages with it. Since Xtend is not aware of the objective of the code generation, no considerable support is provided: It is for example necessary for the DSL designers to manage all the required imports of the generated class by themselves as well as to determine the necessity of such imports, as Xtend has no knowledge about the nature of the generated files. Those are simply added as rich strings in the Xtend file and are then copied into the generated file. Moreover, it is for example the task of the developer of the generator to ensure that variable names are unique and that the same object within the AST is always referenced using the same variable name.

Furthermore, Xtext cannot be exclusively considered as a language workbench, due to its strong reuse of different existing frameworks; it does therefore not appear as the ‘ultimate’ representative of this application group. The application does not give developers the feeling of developing a domain-specific language straightly, as they are always forced to alter between the various tool elements and between the related languages.

Encountered issues

One of the major problems in Xtext is that we cannot add general properties to upper-classes when defining the grammar of our language. This means that we have to provide all sub-classes with the same properties (same name and same type) in order to make Xtext understand that we are creating a taxonomy that has classes with generalized attributes.

Let’s have a look at the following example:

X: A | B;

A: name = ID;

B: name = ID;

We can see above that the attribute ‘name’ is declared in both A and B and this is redundant. Assigning the attribute to the X concept is not possible in Xtext and this can engender serious issues when dealing with large taxonomies.

Microsoft Modeling & Visualization SDK

Strengths

The main advantages of MVSDK can be seen in the level of abstraction, the flexibility and the consistency of the framework. This allows domain-specific languages to be easily implemented and extend as well. The flexibility of the framework is mainly due to the automatic generation of partial classes. These latter make almost any application-specific adaptation and language customization possible. In addition, many parts of the framework might be overridden if needed. This applies mostly to existing methods and properties. The process of the DSL creation is accompanied with a checking functionality as well. It uses constraint rules very similar to the validations we have defined for the generated editors for our DSLs. Quite understandable error messages and bug-fixing suggestions are provided and thus allow simple and more robust creation of DSLs.

The MVSDK is integrated in Microsoft Visual Studio and is therefore based on the .NET Framework. This brings with it a fairly powerful development environment on the one hand and object-oriented languages ​​as well as the ability to adapt automatically generated source code on the other hand. Furthermore, the generated editor (the DSL IDE) also runs in Visual Studio and this leads to a high increase in productivity.

Weaknesses

One of the main disadvantages of DSL Tools is its pretty steep learning curve. Getting started is relatively easy, but as soon as we want to take advantage of the advanced concepts of the tool and provide the model with extra properties that are not explicitly supported, the complexity increases rapidly. This was for example the case when we had to add fully customized validation rules for our ‘Form Creation’ DSL.

Encountered issues

It is true that the MVSDK offers a strong support to the DSL developer when assigning symbols to the language concepts but this also brings with it some uncomfortable restrictions: For instance, shapes cannot be put in each other; this means, in other words, that we cannot model nested objects unless we decide to use swimlanes as the containing object, obviously the only graphical representation that allows the containment of other objects.

Another major drawback of the MVSDK is the lack in the support of common data-types. It is true that generic lists, for example, are not primary types but they are common types, which we always need to have. The use of lists as a data-type is not supported by the MVSDK and it has to be implemented manually.

Therefore, an ‘external type’ has to be added to the data-types set, an editor that enables adding entries to the list has to be implemented as well as a methods for storing the list inputs and serializing them.

MetaEdit+

MetaEdit+ provides an optimal environment for the development of DSLs, thanks to its nature as a standalone application. Another very important feature is the fact that this tool is targeted for the design of domain specific modeling languages. Thereby, MetaEdit+ provides a metamodeling language, which was itself designed using this tool (a bootstrapped language); this leads to the fact that the language definition in MetaEdit+ is done itself in a graphical way using a DSML. So the tool is just a form of a run-time environment for modeling languages ​​and can dynamically get adjusted if new requirements have to be fulfilled.

Also for the domain specialists, who are most probably the DSL end-users, MetaEdit+ is the IDE used for modeling desired instances of the target domain. The advantage here is that the environment is already custom-tailored to the DSML, as this latter has been defined within it. Thus, the end-users are almost not confronted with domain-foreign elements and can better concentrate on the definition of their model.

Strengths

One of the strongest features of MetaEdit+ is the easy communicability of DSMLs. This does not only apply for the eventual end-users, but even also for the language developers, as the definition is carried out using modeling as well. The UML-like metamodeling language GOPRR provides here a widespread scheme which ensures a quick introduction into the task. The ability to define the structural elements of a modeling language in the form of classes, relationships and corresponding roles guarantees a quick and efficient mapping of complex constructs into the language. Possible inconsistencies, which may occur through a language misuse, can as well get fixed by defining needed validations.

For end-users, no great effort is required in order to get used to the IDE and to the language, as the former is precisely configured to the specific needs of the latter. In this way, incorrect operations in the modeling are almost impossible, since, for example for each language, only the elements for modeling are available, which may be instantiated in that language and no other foreign elements can get accessed. The end-users thus do not get confronted with generic constructs within the application and can purely model in the target domain. For an increased recognition of the model elements, symbols and icons maybe assigned to the language constructs.

Weaknesses

Being a standalone application does also bring drawbacks with it. This fact means that the developers of a DSL cannot accomplish the modeling task within their familiar development environments or within the preferred ones but must adapt to a new application;

Another major issue in MetaEdit+ is the back and forth switching between the different tools (windows) in order to accomplish a single task, namely the metamodel definition. As we will see next, this combination of different modeling methods leads to various inconsistencies.

One additional weakness of MetaEdit+ is restricting the validations that might be added to the DSL. The tool only allows special validation types such as uniqueness and number of occurrences check.

A last constraint of MetaEdit+ is the general problem that modeling does not always provide all the facilities which are needed for mapping a domain into a model.

Encountered issues

As we have seen earlier, the metamodel definition in MetaEdit+ is done in two steps, namely starting by graphically modeling the concepts and the in-between relationships and then building the MXT file and importing it for further refinements.

This brings with it a quite serious problem: later possible changes to the existing metamodel may induce the loss of the data added at the refinement stage. Consider the following example:

We model an object that has a property, whose type is ‘fixed list’. As mentioned earlier, this type represents a static list with predefined entries. Those latter however, cannot be defined at the modeling stage; we have to first build the metamodel and then import it into the tool for a further form-based processing. The ‘fixed list’ entries are added at this stage; so far, so good. Though, if we want later to make some changes to our metamodel (e.g. extending it), the list entries get lost just after building the MXT file from the modified metamodel and we have to add them again from scratch.

When defining the code generator for our DSLs, we have encountered some ambiguities, namely: quotation marks (single quotes ‘ ‘ ) have more than one role. They mark in the one hand the code or the text that is going to be directly pasted into the generated file. On the other hand they are in the conditions check context: if type = ‘ ‘. Another confusing fact is that the syntax coloring in both cases is the same.

Comparison Of The Tools

In the previous chapters, we have described in detail the properties of each language workbench. Those properties will make the basis of the following comparison. In this chapter the essential functionalities of these workbenches are summarized and presented in a direct comparison. First of all, the general technical realization is compared; then the specific concepts that are used in the definition of a DSL as well as the editors that are provided for the end-users to model their DSLs in. Next, a comparative analysis of the possibilities offered by the model-to-code transformations of the applications is presented. Finally we discuss and compare the different application fields of the tools.

Technical characteristics

The technical implementations of the language workbenches have no direct impact on the efficiency that can be achieved when designing DSLs; however the environments in which they are deployed still play a very important role in the design process. While Xtext and MVSDK are embedded respectively in the Eclipse IDE and in Visual Studio, MetaEdit+ is a standalone application.

The standalone application, namely MetaEdit+ has here the advantage that it can entirely focus on the implementation of DSLs. This means that no ‘external’ functions can be found that do not have a particular role in the DSL development. In general, the realization as a standalone application provides often the most advantages when it comes to offer a workbench with a total concentration on the language design; there is thus no need to make any compromises with any existing tool.

This entails also the issue that the workbench user (the DSL developer) has to get acquainted with a new development environment, which may not provide any integration capability with the existing IDE. This may engender a serious problem if the DSL user is strongly connected to a specific IDE, due to the amount of work that has been done in it and especially stored in it and if no migration strategies and techniques to the new IDE are provided.

The nature of Xtext, namely its integration in Eclipse, gives rise to the disadvantage that there is an excess of irrelevant (for the DSL development) functionality; this integration however brings with it many considerable advantages. On the one hand, Eclipse is a widely used development environment and many developers are already familiar with it. On the other hand, the ability to integrate and benefit from other existing applications within Eclipse is a quite strong feature. Even if this is not the case, many other integration options are available; these arise from the fact that Xtext is based on free text. Any program that is written in an Xtext-based DSL is stored as a text-like file in the file system and can eventually be processed by any other applications outside the Eclipse IDE. This is an advantage that other representatives in this comparison forfeit.

The DSL Definition

In the process of the ​​language definition, the general procedures of MetaEdit+ and of the MVSDK do not exceptionally differ from each other, while Xtext provides a totally different approach.

In MetaEdit+, DSLs are defined using a metamodeling language (GOPRR), which itself was designed within the same workbench. Through this definition, the semantic model is created. In the MVSDK the DSL definition is also carried out in a graphical way. Here, the semantic model is generated as well as the partial classes by transforming the T4 templates. Both semantic models can get refined in a later stage by adding validation rules. Defining such validations in MetaEdit+, in contrast to MVSDK, cannot be totally customized (by adding code for example). This is due to the fact that MetaEdit+ only allows special types of validations and this represents a major drawback of this workbench. After the generation and validation steps, the language can be released. Thereby the domain experts use symbols that were defined by the DSL developer. While the MVSDK offers predefined shapes such as geometry shapes and compartment ones, MetaEdit+ allows, thanks to the provide Symbol Editor, fully customized icon drawing. Both tools allow the import of Bitmap files.

Contrary to MetaEdit+ and to MVSDK, the DSL definition in Xtext is processed in a quite different style, as no graphical interfaces are provided: the DSL definition is conducted using free text. Accordingly, the grammar of the future DSL is ‘programmed’ within a designated language, namely the ‘Grammar Language’ that was itself defined in its own. In addition to outlining how the lexer and the parser are going to interpret the language, the semantic model is created as part of a required code generation step.

Validations can also be added to the language. Those are coded in Java and can access all language concepts found in the AST. Thereby, fully customized constraints can be implemented and no restrictions are given as it is the case in MetaEdit+.

Extensions of existing languages ​​are also possible within Xtext; this is done via grammar mixing or by referencing existing Ecore models. However, the goal here is rather the re-use of the grammar concepts, than the extension of an entire language. Even here, a full reusability is not as achieved as in MetaEdit+, since this latter allows even the re-use of code generators defined for other DSLs.

Overall, each application pursues its own approach for the language definition; all of them however, use a DSL definition language that was itself designed with the same tool. This can be seen as a testimony of the maturity of these workbenches.

DSL User-Interface (The Editor)

In this section we do not intend to compare the interfaces provided by the tool to support the DSL specification; it is more about the partially-generated editors that the DSL designer provides to the end-users in order to enable them model within the created DSL.

In Xtext, merely pure free-text editors are used. The grammar of the language specifies how the syntax of the DSL looks like. The typed code has to follow all the grammar syntax rules; only then, it can be transformed into a semantic model through the parser and the lexer. Thus, the capabilities of these latter limit the language instantiating possibilities. Beyond this, Xtext helps the developer provide several features to the end-users such as syntax highlighting, code completion and quick-fixes. It is however still questionable whether the Xtext editor brings with it a suitable user-interface for the domain experts, as this former is a free text editor and is often seen as a tool for ‘programmers’. Moreover, this editor is integrated in a development environment intended for programming.

In MetaEdit+ and MVSDK editors are defined by assigning graphical representations (symbols) to the domain concepts. While toolbox icons in MVSDK have to be individually designed and provided by the DSL developer, MetaEdit+ resizes the symbols given to the concepts and appends them to the toolbox. It is true that MetaEdit+ symbols are created using vector graphics, but this technique is not always the best way to create the toolbox, as DSL designers often import predefined bitmaps into their symbols; those do not always look fine if rescaled.

At last, it is important to mention that MetaEdit+ comes with the benefit of being a standalone tool and therefore has no other superfluous (regarding the DSL use) functions. Xtext and MVSDK users however, have to cope with the other existing functions that Eclipse and Visual studio respectively offer.

Model-To-Code Transformation

One of the most important advantages of Xtext and MVSDK is that the generated code from the DSL instance can directly get tested and refined if needed, since both workbenches are integrated in IDE that were already intended for programming purposes. Code generation in both tools is carried out using template languages: Xtend in Xtext and Text Templates in MVSDK.

MetaEdit+, on the other side, comes with the MERL language, which helps iterating through the model elements. The drawback thereby, is that this programming language does not use conventional programming methods and has thus to be learned from scratch. A significant advantage in MetaEdit+ is the ability to apply existing code generators on new DSLs.

Conclusion & Future Work

As we have mentioned earlier in the course of this thesis, the observation and analysis of three representatives of the available language workbenches allow no general conclusions to be drawn regarding the entire set. Nevertheless, based on this analysis, it can be predicted that those tools will gain significant potency and relevance in the future. This is very probably associated to the fact that domain-specific languages are getting more popular from day to day.

Language workbenches open, with respect to the DSLs developed using them, great possibilities for the reuse of languages ​​and their constructs. However, it is certain that the approaches followed by DSLs are not applicable to all areas of the software development and thus, they will not make the traditional programming obsolete. But still, a targeted and focused deployment of DSLs may lead to a substantial increase in the efficiency, especially in recurrent development activities. Furthermore, DSLs form the bridge between the software developers and the domain-experts; they represent the key to the communication with those latter, as well as to their further integration into the development process of an application. Precisely this last fact, namely the narrow cooperation between the developers and the domain-experts, forms an indispensable success factor for software projects; after all, the domain experts are the users the software was developed for.

Due to the increasing use of DSLs, language workbenches are gaining importance from day to another; they currently even form a requisite for the development of such languages. Within the compared workbenches in this thesis, it is certainly unfeasible to define an ‘overall winner’, based on the evaluation and comparison we made. This was also not the objective of this thesis. We have rather checked which of the workbenches constitutes the right tool for which aspect and how DSLs are actually developed with them.

Within the presented tools, only MetaEdit+ can certainly be seen as a pure and standalone language workbench, since both Xtext and MVSDK are respectively integrated in Eclipse and in Visual Studio. Thanks to their graphical orientation, MetaEdit+ and MVSDK provide the easiest access for the actual usage by domain experts. Their deployment, however, is restricted to the domains, for which graphical modeling is appropriate. Xtext appears to be the suitable representative of DSLs that are designed by ‘developers’ for ‘developers’; its integration into the Eclipse development environment allows many programmers an easy entry into the DSL development.

Further research work can be directed towards involving domain experts in the DSL development process and experimenting with them in order to check if it really fulfills their requirements and if its user-interface is suitable for their needs. The integration of human-computer interaction and information visualization approaches, with the intention of offering a more understandable and efficient design of the concrete syntax, could also build an important research field. It would also be fairly interesting to check if a DSL can be designed, which enables its users to customize it by adding new concepts for example.