Storage

A storage holds schema-less JSON documents with full-text search and no relations between them. Blocks create, update, search and remove documents, and every document carries a generated _meta with its unique guid.

Storage is JSON with no schema to define

Blueprints include enterprise-scale storage, reached through blocks like search and create document. You do not need to predefine any data structure. The tradeoff is that there are no relationships between storage entities, so include all necessary data within each document (denormalization).

Every document carries a _meta key

{
  "username": "Joe",
  "_meta": {
    "guid": "a23575f49d0af385314c1f02280163374297e018a692e5e4ab85eb307ebf6ebc",
    "expiry": -1,
    "entity": 1,
    "removed": 0,
    "created": 1706396400,
    "updated": 1706396400,
    "ums": 1706396400803,
    "cms": 1706396400362,
    "update_hash": "60e27209fa93063b5605e41605c2722ed428cae0"
  }
}

Create a document from a storage and mutations

1. Pick the storage

   Add a storage block and select the storage you want to write to.

2. Define the fields

   Add a mutations_set_bp_field_multiple block with the fields and their values.

3. Write it

   Add a function_create_document_from_mutations block, connect the storage and mutations to its in-pins, and connect the flow pin from your flow.

The block returns the created document, including the generated _meta with its unique guid.

{
  "username": "joe",
  "email": "joe@example.com",
  "role": "editor",
  "_meta": {
    "guid": "a23575f49d0af385314c1f02280163374297e018a692e5e4ab85eb307ebf6ebc",
    "expiry": -1,
    "entity": 1,
    "removed": 0,
    "created": 1706396400,
    "updated": 1706396400,
    "ums": 1706396400803,
    "cms": 1706396400362,
    "update_hash": "60e27209fa93063b5605e41605c2722ed428cae0"
  }
}

Updates are mutations, applied one at a time

Storage uses a transaction-based update system through mutations. Each update is processed sequentially, so concurrent updates do not overwrite each other. This makes counters and array operations safe to use.

To update a document, use the function_update_document_mutations block with the document's guid and the mutations you want to apply.

storages mutations

The result shows the new values on the existing document:

{
  "version": 130005,
  "title": "v13.0.5",
  "counter": 2,
  "_meta": {
    "guid": "723575f49d0af385314d1f02280163374297e018a692e5e4ab85eb907ebf6eb5",
    "expiry": -1,
    "entity": 1,
    "removed": 0,
    "created": 1706396400,
    "updated": 1708393100,
    "ums": 1708393100132,
    "cms": 1706396400803,
    "update_hash": "30e27209fa93073b5605e41605c2722ed448cae1"
  }
}

Getting a document by guid is the fastest read

Use the function_get_document block with the document's _meta.guid. It is faster than searching and easier to cache. Avoid creating custom unique identifiers like code; use _meta.guid instead.

To retrieve several documents at once, use function_get_documents with an array of guids.

Build a search by connecting query blocks

1. Create the conditions

   Use blocks like query_bool_term_fields (exact match on a field).

2. Wrap them in a boolean query

   Use query_bool_filter, query_bool_must or query_bool_should.

3. Sort and limit

   Optionally add sorting with query_sort_field, and a limit with query_limit or a number block on the limit pin.

4. Combine the parts

   Join all query parts with query_and.

5. Connect the search

   Wire the final query to the query in-pin of function_search and select the storage on that block.

Example: a user's last 10 messages

This finds documents in a messages storage where user_guid matches a value, sorted by creation date descending, limited to 10 results:

function_search returns an array of matching documents. For queries that should return at most one result, use function_search_single_result.

The query blocks at a glance

Searches are case-sensitive: store values in lowercase

• Use get document when possible. Retrieving by _meta.guid is the fastest method and easy to cache. Avoid custom unique identifiers.
• Use query_bool_filter for most queries. Filter context is more efficient than must because it skips relevance scoring.
• Limit your results. Always set a reasonable limit. Smaller result sets are faster to return and process.
• Select specific fields. Use query_source_by_field to return only the fields you need.
• Stream large result sets. Use function_search_stream to process results as they arrive rather than loading everything into memory.

A search can lag the latest write

By default, search queries do not wait for the most recent inserts to be indexed. If you create a document and immediately search for it, it might not appear yet. In most cases this slight delay is acceptable. Getting a document by guid does not have this delay.

If you need the absolute latest data, use the disable cache query block. It makes the system wait until all pending inserts are indexed before running the search. In high-insert environments, this can introduce noticeable delays.

Four blocks cover full-text search

• query_bool_simple_query_string_field: for user-facing search inputs. Handles common search syntax.
• query_bool_wildcard_string_field: for pattern matching with wildcards.
• query_bool_prefix_string_field: for autocomplete-style "starts with" searches.
• query_bool_term_fuzzy: for fuzzy matching that tolerates typos.

Searches are case-sensitive here too. Store searchable text in lowercase and convert user input with value_to_lowercase before searching.

Model data without relations: embed what you read

RUAL storage does not support relationships between storage entities, and there are no SQL-style joins. Include the information you need within each document.

Example: orders and customers

In a relational database, you would have a separate customers table and reference the customer ID from orders. In RUAL, include the relevant customer information directly in the order document:

{
  "order_number": "ORD-2024-0042",
  "status": "shipped",
  "total": 15990,
  "customer_name": "joe doe",
  "customer_email": "joe@example.com",
  "customer_guid": "a23575f49d0af385314c1f02280163374297e018a692e5e4ab85eb307ebf6ebc",
  "items": [
    { "name": "widget", "quantity": 3, "price": 5330 }
  ],
  "_meta": {
    "guid": "d56808172a3d26b86472153355496fa752ae341bc915e80b19de63a920e89203",
    "expiry": -1,
    "entity": 1,
    "removed": 0,
    "created": 1706396400,
    "updated": 1708393100,
    "ums": 1708393100132,
    "cms": 1706396400362,
    "update_hash": "3d82f70b5ce1946a208bd4f6019e7c5a86bb42d1"
  }
}

Searching for orders now gives you the customer name and email without a second query. If a customer changes their name, update all their order documents using a storage event on the customers storage.

Denormalization strategies

• Embed frequently accessed data. Store customer_name on every order instead of just customer_guid.
• Keep a reference guid. Always store the guid of related documents so you can get document when you need the full data.
• Use storage events for sync. When a source document changes (a customer's name, say), a storage event updates all documents that embed that data.
• Store counters separately. Instead of aggregation queries, maintain a counter field on a parent document and increment it when child documents are created.

Storage Examples works this through for profiles, catalogs, orders and logs.

Which block does what in a document's life

Removed documents can be restored

Use function_remove_document or function_remove_documents to soft-remove documents. This is different from permanent deletion with function_delete_document.

• Removed documents are tagged as such, and a revision is created if revisions are enabled.
• Search queries exclude removed documents by default. Include them with the query_include_removed block.
• Restore with function_restore_document. The data is never actually deleted from storage until you purge it.

Documents can expire on a date you set

By default, documents do not expire (_meta.expiry is -1). Set an expiry timestamp when creating or updating a document with function_create_document_from_mutations or function_update_document_mutations.

To reset a document's expiry back to "never", use the date_never block, which sets _meta.expiry to -1. There is a slight delay before an expired document is physically deleted, but the get document block treats it as deleted once the expiry time has passed.

Cache busy searches in Redis

For high-traffic applications, cache search results in Redis instead of querying storage on every request:

• Cache frequently accessed search results with a TTL using redis cache blocks.
• Invalidate or update the cache when documents change, using storage events.
• Store aggregated statistics in Redis and update them periodically with repeating events.

Next steps