Storage

RUAL includes a built-in, JSON-based storage system with full-text search. Learn how to create, update, search, and manage documents.

Storing Data

Blueprints include enterprise-scale storage, accessible through blocks like search and create document. The storage is JSON-based and schema-less — you don't need to predefine any data structure. The tradeoff is that there are no relationships between storage entities, so you should include all necessary data within each document (denormalization).

Meta Data

Every document includes a _meta key with the following fields:

Creating Documents

To create a document, you need a storage reference and a set of mutations that define the fields:

1. Add a storage block and select the storage you want to write to.
2. Add a mutations_set_bp_field_multiple block to define the fields and their values.
3. Add a function_create_document block, connect the storage and mutations to its in-pins, and connect the flow pin from your flow.

The function_create_document block returns the created document including the generated _meta with its unique guid.

Updating Documents

Storage uses a transaction-based update system through mutations. Each update is processed sequentially, so concurrent updates won't overwrite each other. This makes counters and array operations safe to use.

To update a document, use the function_update_document block with the document's guid and the mutations you want to apply.

Common mutation operations:

The execution result includes additional keys since it operates on an existing document:

Getting a Document by GUID

The fastest way to retrieve a document is by its _meta.guid using the function_get_document block. This is faster than searching and easier to cache. Avoid creating custom unique identifiers like code — use _meta.guid instead.

To retrieve multiple documents at once, use the function_get_documents block with an array of GUIDs.

Building Search Queries

Search queries are built by connecting query blocks together and feeding them into a function_search block. A typical search flow looks like this:

1. Create the query conditions using blocks like query_bool_term_fields (exact match on a field).
2. Wrap them in a boolean query using query_bool_filter, query_bool_must, or query_bool_should.
3. Optionally add sorting with query_sort_field and a limit with query_limit or a number block connected to the limit pin.
4. Combine all query parts using query_and.
5. Connect the final query output to the query in-pin of function_search.
6. Select the storage on the function_search block.

Search Query Example

This example finds documents in a messages storage where user_guid matches a specific value, sorted by creation date descending, limited to 10 results:

The function_search block returns an array of matching documents. For queries that should return at most one result, use function_search_single_result instead.

Query Types Overview

Search Tips

• Use get document when possible: Retrieving by _meta.guid is the fastest method and easy to cache. Avoid custom unique identifiers.
• Use query_bool_filter for most queries: Filter context is more efficient than must because it skips relevance scoring.
• Limit your results: Always set a reasonable limit. Smaller result sets are faster to return and process.
• Select specific fields: Use query_source_by_field to return only the fields you need.
• Stream large result sets: Use function_search_stream to process results as they arrive rather than loading everything into memory.

Real-Time Search Results

By default, search queries do not wait for the most recent inserts to be indexed. If you create a document and immediately search for it, it might not appear yet. In most cases this slight delay is acceptable.

If you need the absolute latest data, use the disable cache query block. This forces the system to wait until all pending inserts are indexed before running the search. In high-insert environments, this can introduce noticeable delays.

Full-Text Search

The storage includes a built-in full-text search engine. The main blocks for text searching are:

• query_bool_simple_query_string_field — for user-facing search inputs. Handles common search syntax.
• query_bool_wildcard_string_field — for pattern matching with wildcards.
• query_bool_prefix_string_field — for autocomplete-style "starts with" searches.
• query_bool_term_fuzzy — for fuzzy matching that tolerates typos.

Remember that searches are case-sensitive. Store searchable text in lowercase and convert user input to lowercase before searching.

Data Modelling Without Relations

RUAL storage does not support relationships between storage entities — there are no SQL-style joins. Instead, denormalize your data: include all the information you need within each document.

Example: Orders and Customers

In a relational database, you'd have a separate customers table and reference the customer ID from orders. In RUAL, include the relevant customer information directly in the order document:

This way, searching for orders gives you the customer name and email without a second query. If a customer changes their name, update all their order documents using a storage event on the customers storage.

Denormalization Strategies

• Embed frequently accessed data: Store customer_name on every order instead of just customer_guid.
• Keep a reference GUID: Always store the guid of related documents so you can get document when you need the full data.
• Use storage events for sync: When a source document changes (e.g., customer name), use a storage event to update all documents that embed that data.
• Store counters separately: Instead of aggregation queries, maintain a counter field on a parent document and increment it when child documents are created.

Document Lifecycle

Document Removal

Documents can be removed and restored at any time. Use function_remove_document or function_remove_multiple_documents to soft-remove documents. This is different from permanent deletion with function_delete_document.

Removed documents are tagged as such, and a revision is created if revisions are enabled. Search queries exclude removed documents by default, but you can include them with the query_include_removed block.

Restoring is straightforward with the function_restore_document block — the data is never actually deleted from storage. A good practice is to remove documents first, then permanently purge them after a retention period.

Document Expiry

By default, documents don't expire (_meta.expiry is -1). You can set an expiry timestamp when creating or updating a document using the function_create_document and function_update_document blocks.

To reset a document's expiry back to "never", use the date_never block, which sets _meta.expiry to -1. There is a slight delay before an expired document is physically deleted, but the get document block treats it as deleted once the expiry time has passed.

Caching with Redis

For high-traffic applications, cache search results in Redis instead of querying storage on every request. Common patterns:

• Cache frequently accessed search results with a TTL using redis cache blocks.
• Invalidate or update the cache when documents change, using storage events.
• Store aggregated statistics in Redis and update them periodically with repeating events.

RUAL Docs is an integral component of the RUAL ecosystem