Storage Guide

A hands-on guide to working with RUAL storage: creating your first document, building search queries, modelling data without relations, and using full-text search effectively.

This guide builds on the Storage documentation and walks you through practical examples of working with RUAL's built-in storage system. If you haven't read the storage docs yet, we recommend starting there to understand the fundamentals of metadata, document updates, and expiry.

Creating Your First Document

To create a document, you need two things: a storage reference and a set of mutations that define the fields. Here is the block chain for creating a document:

1. Add a storage block and select the storage you want to write to.
2. Add a mutations_set_bp_field_multiple block to define the fields and their values.
3. Add a function_create_document block, connect the storage and mutations to its in-pins, and connect the flow pin from your flow.

The function_create_document block returns the created document including the generated _meta data with its unique guid.

Updating Documents

RUAL uses a transaction-based update system through mutations. Each update is processed sequentially, ensuring that concurrent updates don't overwrite each other. This makes it safe to use counters and array operations.

To update a document, use the function_update_document block with the document's guid and the mutations you want to apply.

Common mutation operations include:

Getting a Document by GUID

The fastest way to retrieve a document is by its _meta.guid using the function_get_document block. This method is faster than searching and is easier for the system to cache. Whenever possible, prefer get document over search queries for individual document retrieval.

To retrieve multiple documents at once, use the function_get_documents block with an array of GUIDs.

Building Search Queries

Search queries are built by connecting query blocks together and feeding them into a function_search block. A typical search flow looks like this:

1. Create the query conditions using blocks like query_bool_term_fields (exact match on a field).
2. Wrap them in a boolean query using query_bool_filter, query_bool_must, or query_bool_should.
3. Optionally add sorting with query_sort_field and a limit with query_limit or a number block connected to the limit pin.
4. Combine all query parts using query_and (multiple queries).
5. Connect the final query output to the query in-pin of function_search.
6. Select the storage on the function_search block.

Search Query Example

Below is an example of a search that finds documents in a messages storage where user_guid matches a specific value, sorted by creation date descending, limited to 10 results:

The function_search block returns an array of matching documents. For queries that should return at most one result, use function_search_single_result instead.

Query Types Overview

Search Tips

• Use get document when possible: Retrieving a document by _meta.guid is the fastest method and is easy to cache. Avoid creating custom unique identifiers and searching for them.
• Use query_bool_filter for most queries: Filter context is more efficient than must because it does not calculate relevance scores.
• Limit your results: Always set a reasonable limit on search results. Smaller result sets are faster to return and process.
• Select specific fields: Use the query_source_by_field block to return only the fields you need, reducing data transfer.
• Stream large result sets: For queries that return many documents, use function_search_stream to process results as they arrive rather than loading everything into memory.

Real-Time Search Results

By default, search queries do not wait for the most recent inserts to be indexed. If you create a document and immediately search for it, it might not appear in the results yet. In most cases this slight delay is acceptable.

If your use case requires the absolute latest data, you can use the disable cache query block. This forces the system to wait until all pending inserts are indexed before executing the search. However, in high-insert environments, this can introduce significant delays.

Data Modelling Without Relations

RUAL storage is schema-less and does not support relationships between storage entities. This means you cannot do traditional SQL-style joins. Instead, you should denormalize your data: include all the information you need within each document.

Example: Orders and Customers

In a traditional relational database, you would have a separate customers table and reference the customer ID from the orders table. In RUAL, you should include the relevant customer information directly in the order document:

This way, when you search for orders, you immediately have the customer name and email available without needing a second query. If the customer changes their name, you can update all their order documents using a storage event on the customers storage.

Denormalization Strategies

• Embed frequently accessed data: Include the data you display most often directly in the document. For example, store customer_name on every order instead of just customer_guid.
• Keep a reference GUID: Always store the guid of related documents so you can use get document when you need the full, up-to-date data.
• Use storage events for sync: When a source document changes (e.g., customer name), use a storage event to update all documents that embed that data.
• Store counters separately: Instead of using aggregation queries to count documents, maintain a counter field on a parent document and increment it when child documents are created. Alternatively, store aggregated stats in Redis cache.

Full-Text Search

RUAL's storage includes a built-in full-text search engine. To take advantage of it:

• Use query_bool_simple_query_string_field for user-facing search inputs. This block handles common search syntax that users expect.
• Use query_bool_wildcard_string_field for pattern matching with wildcards.
• Use query_bool_prefix_string_field for autocomplete-style "starts with" searches.
• Use query_bool_term_fuzzy for fuzzy matching that tolerates typos.

Remember that search queries are case-sensitive. Store searchable text in lowercase and convert user input to lowercase before searching to ensure consistent results.

Document Lifecycle

Understanding the full lifecycle of a document helps you design robust data flows:

Caching with Redis

For high-traffic applications, caching search results in Redis is critical to maintain performance. Instead of hitting storage for every page visit, check Redis first and only query storage when the cache is empty or expired.

Common caching patterns include:

• Cache frequently accessed search results with a TTL (time to live) using redis cache blocks.
• Invalidate or update the cache when documents change, using storage events.
• Store aggregated statistics in Redis and update them periodically with repeating events.

RUAL Docs is an integral component of the RUAL ecosystem