MDR - Architecture

Contents:
1. Structure of MDR Project
1.1 JMI API
1.2 MOF API
1.3 MDR API
1.4 JMI Utilities
1.5 MDR Library
1.6 MDR Module
2. Architecture of MDR Core
2.1 Handlers & Generated Bytecode
2.2 Events
2.3 Additional Indexing
2.4 Persistence SPI
2.5 B-tree Storage
2.5.1 Indexing
2.5.2 Storage
2.6 Memory Storage

1. Structure of MDR Project

MDR project is partitioned into several modules and standalone libraries. The following picture shows, how the whole structure of MDR looks like.
The dark-blue boxes in a picture represent libraries (jar files), light-blue boxes represent NetBeans modules (each module consists of one module jar file and zero or more external libraries - in the picture depicted in a grey "ext" box). Arrows between modules show module dependencies (each arrow points from a dependent module to a module that it depends on).

The following sections individually describe all the MDR components. These sections also explain component dependencies on a library level rather than on a module level as in the picture.

1.1 JMI API

JMI API contains JMI reflective interfaces specified by JSR-40 (JMI). These interfaces can be used for accessing the metadata managed by MDR without an explicit knowledge of a metamodel the metadata conform to.

DEPENDENCIES: none

1.2 MOF API

Library containing generated JMI interfaces for MOF 1.4 metamodel. These can be used for accessing specifically MOF metadata (i.e. metadata conforming to MOF 1.4 metamodel).

DEPENDENCIES: JMI API (any generated JMI interfaces extend the JMI reflective interfaces and thus are dependent on them)

1.3 MDR API

MDR API library defines a generic API (additional to the standard API defined in JMI) for metadata repositories. This API is an extension to JMI API (adding API for events, facility, etc.) and it is not specific to NetBeans implementation of MDR. Each implementation of MDR API can define an additional "implementation specific" API that can be used by plug-ins specific to that particular implementation.

DEPENDENCIES: JMI API, part of openide.jar from NetBeans (Lookup API)

1.4 JMI Utilities

The library containing generic JMI utilities that operate on JMI reflective interfaces. These include implementation of XMI reader, XMI writer, JMI mapping utility and others. All the utilities should be able to run on any JMI compliant repository - i.e. they are not JMI implementation specific.

DEPENDENCIES: MDR API (some of the utilities implement interfaces included in MDR API), JMI API, MOF API, part of openide.jar from NetBeans (Lookup API)

1.5 MDR Engine

This library contains the core of MDR implementation - a standalone MOF 1.4 and pre-JMI compliant repository which can be used by any Java application.

DEPENDENCIES: MDR API, JMI API, MOF API, any implementation of JAXP API (this is because MDR is able to rebuild itself - bootstrap - from the MOF.xml definition file - to do this, it uses an XML parser), part of openide.jar from NetBeans (Lookup API)

1.6 MDR Module

NetBeans module that contains code for integration of MDR into NetBeans. This integration code includes implementation of MDRManager that resolves repositories using settings defined in NetBeans layered filesystem, registration of XMI reader, XMI writer and implementations of other MDR API interfaces into the NetBeans lookup, etc.

DEPENDENCIES: NetBeans (core, openide), MDR Library, JMI Utilities and everything else that these libraries depend on transitively (see above).

2. Architecture of MDR Core

Note: This section is not an API specification. The layers described herein should not be considered as a stable API. Any dependencies on the things described in this section are on clients' own risk.

There are several ways, how the MOF repositories can work. The most straightforward way is to create a repository source code generator, which is able to take any MOF metamodel and generate a repository implementation code from it. Such a code - after it is compiled - implements a MOF repository for that particular metamodel.

With MDR we took a different approach as it better fits our use cases. The MDR implementation itself is implementation of a MOF repository for and kind of metadata. This is possible via dynamical generation of bytecode that implements the metamodel specific JMI interfaces during the MDR runtime. So users can load any metamodel to the MDR, and supposing that the JMI interfaces for this specific metamodel are on the classpath, MDR can immediately instantiate the metamodel and serve as a repository for this particular kind of metadata simultaneously with supporting other known kinds of metadata.

In the following picture you can see all the architectural blocks of the MDR core (MDR Library together with the MDR and JMI API).
As indicated by the green box in a picture, all clients of MDR have access to the MDR functionality via API consisting of two parts - MDR API and JMI API. The orange boxes below the API layer represent implementation. As you can see, the orange implementation block is split by a light blue layer representing the Persistence SPI.

See the following sections for a more detailed description for the implementation blocks and the Persistence SPI. The orange boxes with italic font (JDBC, others) are not implemented - they are listed only to indicate that it is possible to implement them - and thus will not be described.

2.1 Handlers & Generated Bytecode

All the JMI generated interfaces are implemented during the runtime using a dynamicaly generated bytecode. The generated bytecode transfers all the JMI calls to Handlers, which handle the method calls by accessing the persistence SPI.

2.2 Events

MDR API includes API for MDR events, which is a very important extension to the standard JMI API. The events API allows MDR clients to be notified of any changes in the MDR made via JMI interfaces. The events functionality is implemented by the Handlers with use of a specialized helper classes. To learn more about events, see the MDR API javadoc.

2.3 Additional Indexing

MDR provides a feature of additional indexing. This means that using a MDR specific tag, any attribute in a metamodel can be marked as indexed, which will result in creating a separate index for this attribute in the MDR when the metamodel is instantiated. All the instances of classes containing this attribute are then indexed by the values of this attribute and implementations of derived operations and attributes can make use of these indexes in order to boost the performance.

2.4 Persistence SPI

This SPI is small set of interfaces which provide methods for basic indexing. By implementing this SPI one can provide an alternate storage for MDR which can be used instead of the default b-tree storage. This SPI is easy to implement by data structures like hashtable, binary tree or database tables/indexes.

Currently two implementations of this SPI are present in the MDR Library. See the following two sections for more details.

2.5 B-tree Storage

Default implementation of Persistence SPI. Implements the SPI using a b-tree database.

Btree is a subsystem for storing keyed records (usually containing serialized objects) persistently in a filesystem. Each Btree database consists of two files:

The data file contains all the information needed to rebuild the index file, should it become lost or damaged. Changes to the btree database are made via atomic transactions.
This design is based on that of the Btree repository from the Forte 4GL product and has been in production use for about four years. It has shown itself to combine great efficiency and reliability compared to other schemes used previously, and has completely eliminated low-level corruption as a source of repository problems.

Btree consists of two main subsystems: Indexing and storage

2.5.1 Indexing

The Btree indexing subsystem began as public domain code from BSD. As modified and implemented in Java for NetBeans, it supports a variety of types of b-tree indexes. Structures include: Supported key and value types include: These indexes can exist either as objects in memory or as disk files.

2.5.2 Storage

The Btree storage subsystem stores object persistently on disk. It is made up of several layers:

  • Transactional file cache - The lowest level of Btree is a file cache. This cache opens a set of files and presents the contents of the files to client code as a set of pages; where each page is identified by which file it comes from and which page it is within that file. Clients can read and modify existing pages and add new pages, i.e. extend the file.
    Changes to the files are made persistent via atomic transactions; that is, either all changes to all files are written, or none are. The file cache uses before-image logging to record the original sate of the files while the transaction is being committed. If the commit fails or is interrupted for any reason, the next time the cache opens the files it will roll them back to their pre-transaction state. Only one concurrent transaction is supported.

  • Data file - The data file is a record manager. It stores records, each with their own key, in a single file, allowing records to be added, deleted, and modified. These records are stored in extents of up to 32K, but extents can be chained together to form records of any desired length. The space from deleted records will be reused.
    Records are identified by their key and offset within the file. Translation of key to offset is not done by the data files itself; a separate indexing facility must be provided.

  • Index file - The Btree index file is a single-valued Btree index, stored in a file, whose keys are objects, and whose values are integers. It is used as the indexing facility for the data file, in that for each record in the data file:
    • The key for the index file entry is the key for the data file record
    • The data for the index file entry is the offset in the data file of the data file record
  • Btree database - A Btree database is the combination of a data file and an index file, stored together in the same file cache, that is, atomically updated together. The records in the data file contain serialized objects, so the basic operations on the Btree database are: fetch object, store object, and delete object. A cache of objects stored in the database is used both for efficiency and to ensure object identity.
    Note that btree indexes stored in memory are among the objects that can be serialized and stored in the Btree database. The MDR uses this sort of Btree index directly. It also uses the Btree Database layer directly. The other layers are available to any code that needs transactional persistent storage.

  • 2.6 Memory Storage

    A simple implementation of Persistence SPI using hash tables. This implementation stores all the repository content in memory only. It was used for testing purposes during the MDR developement, before the b-tree storage implementation was finished. Now it can be used for providing transient repositories.