Skip to main content

Software Structure

RoStore provides different components, that include:

  • rostore-media - a core data-access layer to store, retrieve the data in a secured and controlled way;
  • rostore-async - a set of functionality to facilitate a high-volume and high-frequency operations;
  • rostore-service - a stand-alone server on quarkus 3.x basis;
  • rostore-cli - a command line interface that allows to manage the standalone from the CLI interface;
  • rostore-client - a java client library that allows accessing the standalone server in the client applications.

Each component is released under Apache 2.0 License and can be used independently.

rostore-media​

This module contains some back-bone classes, like Media or Container and ContainerShard and can be used independently to implement some basic storage applications, that allow to write keys, data, serialize and deserialize it, contains operations on the block level. This module do not contain classes for multi-threaded access to the storage, which is covered in the rostore-async

Media​

The initialization of the storage can be executed by the method create of Media class:


class Media {

...

public static <T> Media create(final File file, final MediaProperties mediaProperties, final Function<Media, T> headerFactory) {
    ...
}

}

headerFactory is used for initialization of the RoStorage with some bootstrap information. The object that this factory creates will be written as a part of the storage header and it will be available when the storage will be opened again.

Block Allocators​

Among other important classes let's mention BlockAllocator, which can be obtained over the Media object, like this:


BlockAllocator blockAllocator = media.getBlockAllocator();

This class allows to allocate new blocks. After the block is allocated, it can be exclusively used. The calling process is responsible to free any allocated block after its ussage, otherwise it will be blocked forever.

Additionally Media supports the secondary allocator. This one can count all allocated blocks and allows to free all allocated blocks from this secondary allocator. This allocator has to first be created and persisted:


BlockAllocator secondaryBlockAllocator = media.createSecondaryBlockAllocator("central-one", 0);

long startIndex = secondaryBlockAllocator.getStartIndex();

// store startIndex in the storage

....

// later:

final long startIndex = ... // load the startIndex from the store

BlockAllocator secondaryBlockAllocator = loadSecondaryBlockAllocator("central-one", startIndex, 0)

Block Containers​

Another important class in BlockContainer. This class defines the blocks that the process uses for some local operation. Before process can read or write block data it should first create a new block container, after the task is over, it should be freed. The free operation would mark all the blocks to be released from the memory:


try (Media m = Media.open(mediaFile)) {
    try (BlockContainer bc = m.newBlockContainer()) {
        Block block = bc.getBlock(0L, BlockType.CATALOG);
        byte bt = block.getByte();
        ....
        // do something with the first byte
    }
}

In the example before both Media and BlockContainer are freed after the usage.

Containers​

Containers in this module can be created, but are not the integrative part of the storage. Once created they should be explicitly persisted on the media.

    try (Media media = Media.create(file, MediaProperties.from(mediaPropertiesBuilder))) {
        ContainerListProperties containerListProperties = new ContainerListProperties();
        // the following command will create a new container list
        ContainerListOperations containerListOperations = new ContainerListOperations(media, containerListProperties);
        // to open it afterwards, one should store the containerListOperations.getContainerListHeader() - not part of this example
        ContainerMeta cm = new ContainerMeta();
        cm.setShardNumber(1);
        // create a container
        try (Container c = containerListOperations.create("my-container", cm)) {
            // use the container, by writing to its shards
            c.getShard(0).keyFunction((ko)-> {
                    final Record r = new Record().id(123);
                    ko.putKey("my-key".getBytes(StandardCharsets.UTF_8), r);
                    return r;
                   }
                );
        }
        ....

As seen the containers are already part of the media bundle, but their usage is somehow cumbersome. It is to open different use cases to be implemented on top of media. The next bundle rostore-async adds more direct interface for the containers and implement a clean and easy to use interface to work with containers.

At the same time it makes the use cases available for the implementation more specific and narrow.

rostore-async​

Rostore-async module adds asynchronous access to the key-values on the containers with their shards, as well as all the functionality to execute many parallel requests on the media on the container basis.

The center piece here is AsyncContainerMedia:


File storageFile = new File(/** path is here**/);
AsyncContainerMedia asyncContainerMedia = AsyncContainerMedia.create(storageFile, executorService, asyncContainerMediaProperties);

This class replaces the Media from the [rostore-media] and offers all the functionality to create, list, open the containers:

final AsyncContainers asyncContainers = asyncContainerMedia.getAsyncContainers();
final ContainerMeta containerMeta = new ContainerMeta();
// configure containerMeta here
final AsyncContainer asyncContainer = asyncContainers.create("my-container", containerMeta);
asyncContainer.put(0, "my-key", "my-value");

There are a variety of put, get, delete commands that allows to manipulate or read the content of the container.

In the background the AsyncContainer uses the core Container class from the [rostore-media] bundle, but make it more easy to use, especially of the use cases when the containers has to be an integral part of the storage.

rostore-service​

This module adds the REST service on top of the rostore-async media and exposes it over the openapi. The service maintaines the API-keys based security that is stored in the special container.

The service is implemented as a quarkus-based application and exposes a simple command, that allows to start and stop the rostore.

rostore-cli​

This is a module that allows to execute the majority of the rostore functionality over simple unix-like command line interface. See details here

rostore-client​

It is the java client that is documented here.