Storage
We offer a comprehensive full-text search engine and storage within our blueprints, designed for user convenience. This storage is dynamically scalable, adjusting according to the size and field requirements specified by the user.
Storing Data
Our blueprints offer enterprise-scale storage solutions, easily accessible through blocks such as search and create document. This data store is JSON-based and schema-less, which means there's no need to predefine the data structure. However, it also implies that creating relationships between different storage entities is not possible.
As a result, when storing data, it's important to consider that elements typically combined from other sources, like a username, should be included within the document itself. This approach is particularly crucial for search-related activities, ensuring all necessary data is readily available within each individual document.
Meta Data
Each document generated includes a _meta key, commonly referred to as Meta Data. The following document illustrates the JSON content of a single document.
The Meta Data encompasses the following details:
| Key | Description |
|---|---|
guid |
This is the unique identifier of the document. The GUID is a generated hash and is immutable. |
expiry |
A Unix timestamp indicating the expiration date of the document. By default, this value is set to -1. |
entity |
A single cluster can host multiple entities, which are physically separated. However, the node requires specific information to locate and access these entities. |
removed |
A Unix timestamp showing if the document has been marked as removed. The default value is 0, indicating the document is active. |
created |
A Unix timestamp denoting the creation date of the document. |
updated |
A Unix timestamp marking the last update time of the document. |
ums |
The timestamp in milliseconds representing the creation time of the document. |
cms |
The timestamp in milliseconds representing the last update time of the document. |
update_hash |
A unique hash generated for the most recent modification, reflecting our storage's transaction-based nature. |
Document Get
For optimal performance when accessing documents from storages, it is strongly advised to utilize the document get method extensively. It's best to refrain from using custom unique identifiers for documents, such as code, or other methods. Searching for documents can be resource-intensive and challenging to cache effectively. Using the _meta.guid ensures the quickest retrieval of the complete document contents. This approach is second only to managing a redis cache on your own.
Document Updates
Our storage operates on a transaction-based system, ensuring that each update is sequentially processed, following the completion of the previous one, without exceptions. This approach allows for the use of counters within your documents or array updates, which would typically risk being overwritten in high-traffic non-locking storages. To facilitate this, we employ mutations for our updates. While this might require some initial adjustment, its benefits become apparent as you become more accustomed to the system.
When working with our documents, there are a few key points to keep in mind:
- Fields cannot include dots [
.]. If they do, we automatically convert them into an object structure. - Field names are always stored in lowercase. For example, if you use
UserName, it will be stored asusername. - Search queries are case-sensitive. For instance, if you store a value as
"Joe"and search for"joe", the document will not be found. It's a good practice to store searchable values in lowercase to ensure accurate search results.
The execution result will include additional keys, as it operates on an existing document (a release).
Document Removal
Documents can be removed and restored at any time. The remove document or remove multiple documents blocks facilitate the easy removal of one or multiple documents. However, removing a document is different from permanently deleting it. Removed documents are tagged as such, and a revision is created if revisions are enabled. By default, search queries exclude removed documents, but they can be included by using query include removed in your query.
Restoring removed documents is straightforward with the restore document block, as the data is not actually deleted from storage. A good practice is to initially remove documents and then, after a certain period, permanently purge them when you are certain they are no longer needed.
Document Expiry
By default, documents are set not to expire, with the _meta.expiry value at -1. However, if an expiry timestamp is specified, the document will expire accordingly. This expiry can be set or updated using the create document and update document blocks.
An existing expiry can be modified or removed by inputting a new expiry time. To reset a document's expiry to never expire, we offer a date_never block. While there is a slight delay before a document is permanently deleted post-expiry, the get document block will treat it as already deleted once the expiry time has passed.