For both types of persistence, you can specify that the persistence is maintained synchronously or asynchronously.
Shared-Nothing Persistence
With shared-nothing persistence, each node that joins a space as a seeder maintains a copy of the space data on disk.
Persisted Data Store
When you configure shared-nothing persistence, you must use a unique name for each member joining the spaces, and you must specify an existing directory path for which ActiveSpaces has read and write access.
You can specify the directory path for data storage as follows:
● Through calls to the API operations.
● For as-agents, by using the command line arguments for as-agent.
● Through an environment variable.
The directory you specify is used as the root path under which ActiveSpaces creates its own subdirectory structure, using the format metaspace/space/member.
ActiveSpaces creates and manages persistence files automatically. You do not have to provide a filename for the stored data—the data store directory is used as the location to create and use the file.
For detailed information on implementing shared-nothing persistence, see Persistence.
For detailed information on implementing shared-all persistence, see Persistence.
Terms and Concepts for Persistence
The following terms and concepts are useful for understanding persistence:
Space State
Indicates whether the space can accept regular space operations or not. This happens only when the space is in READY state.
Persistence Type
Defines what type of persistence ActiveSpaces uses. Shared-all and shared-nothing are the supported types. Only one type of persistence can be configured on the same space at the same time.
Persistence Policy
Defines how the changes to space will be persisted—synchronously or asynchronously.
Member Name
A unique name to identify each node/seeder/member. Recommended if using shared-nothing persistence.
Data Store
The file system/directory location where ActiveSpaces stores the persistence files.
Data Loss
Data loss is detected by ActiveSpaces when the number of nodes (seeders) that either leave or fail (due to a crash) exceeds the count set for the space. In this situation, the space is marked as FAILED.
Space Recovery
ActiveSpaces recovers a space (based on user intervention) when the space state is FAILED either due to data loss or cluster startup.
Space Resume
ActiveSpaces resumes a space (based on user intervention) when the space goes into a SUSPENDED mode due to loss of a persister.
TIBCO ActiveSpaces Cluster Startup with Persistence
With shared-nothing persistence, when ActiveSpaces nodes are started for the first time and join the metaspace (and subsequently the defined space), ActiveSpaces creates new data store files, and since there are no old files to recover from, the space automatically enters the READY state and is available for space operations.
If any ActiveSpaces node is restarted either after a failure or as a new member, the space is available for space operations if none of the nodes in the space find files to load data from. If any node has an old file, the space state is set to RECOVER, and your application must initiate a load action.
Space Recovery with Persistence
When you configure persistence, you have the option of configuring space recovery.
Space recovery has two options that you can specify through the API functions or the recover command.
Recovery with Data
Use this option if data loss is not acceptable and you want to reload the data from persistence files into the space.
Recovery Without Data
If data loss is acceptable, then use recovery without data. This specifies that ActiveSpaces does not load data back into the space from persistence files.
You can perform recovery by using:
● API operations
● The Admin CLI
For detailed information on setting up recovery, see Recovery with Persistence. For Space recovery options, refer to the recover space command listed in the TIBCO ActiveSpaces Administration guide.
Space Resume with Shared-All Persistence
When the space loses all of its persisters, the space is set to a SUSPENDED state, which means that no writes to persister can happen. In this case, to resume writing to the persister you can use the resume
space command from as-admin. Implementing Persistence
For details on how to set up persistence, see Persistence.
Space Life Cycle and Persistence
The space life cycle starts when the space is first defined in the metaspace and ends when the space definition is dropped from the metaspace.
A space can be in one of the following states:
● INITIAL - The space has been defined and is waiting for the minimum number of seeders required by the space's definition to be reached, and for at least one persister to be registered if it is a
persisted space.
● LOADING - The space is a persisted space that has reached the required minimum number of seeders and has at least one registered persister. One of the persister's onLoad methods is being invoked and the space data is being loaded from the persistence layer.
● READY - The space has the required minimum number of seeders and if persisted, data has been loaded and has at least one registered persister.
● RECOVER - The space has lost some seeders; all data might not be available to the application. In this case, no operation on the space is possible until a recovery is initiated (generally from
as-admin). Use the recover space command to resume normal operations. This is applicable for shared-nothing mode of persistence only.
● SUSPENDED - Applicable for shared-all when all persisters are lost.
● INVALID - The space should, ideally, not go to this state. In most of the cases, this state is a result of a bug in the internal implementation.
Space operations that read or write data in the space are only allowed when the space is in the READY state. The only exception to this rule is that the space's load method can be invoked when the space is in the LOADING state (typically by the registered persister onLoad method).
Your application can check that the space is in the READY state before attempting to use it by using the space's isReady() method.
Your application can also synchronize itself with the space's state by using the space's waitForReady method. This method takes a timeout that is the number of milliseconds for which it will block while waiting for the space to reach the READY state, and returns a boolean value indicating whether the timeout was reached or not (Java also has a convenient version of the method that does not take a timeout and just blocks until the space is ready).
Another way to synchronize an application with the space's state is to rely on the space definition's SpaceWait attribute: a configurable timeout that is used to block space operations when the space is not in the READY state until either the space becomes ready (at which point the operation is executed) or the SpaceWait timeout expires (at which point the operation will fail).
Persistence and Space Life Cycle
When a space needs to be persistent so that the data that is stored in it does not disappear after a disaster (all seeders have crashed) or a maintenance shutdown, you should define it as a persisted space.
Two choices are available for persistence: built-in shared-nothing persistence or external shared-all persistence.
At a high level, persistence is invoked at various steps in the life-cycle of a space:
● When a space is first reinstantiated after a complete shutdown or crash, the persistence “loading”
phase is first invoked to “re-hydrate” the data (or rebuild indexes) from the persistent storage medium:
— With built-in shared-nothing persistence, loading occurs in parallel on all the space's seeders at the same time.
— With external shared-all persistence, the onLoad method of one of the registered persistence implementations is invoked.
● When a change is made to the data in the space (because of a put, a take or any other space action that modifies the data), this change needs to be reflected to the persistent storage medium.
This is done either synchronously or asynchronously in a distributed manner by each seeder (including those that replicate the data). Data is persisted to it's designated local storage file folder in shared-nothing persistence, or by the persistence implementation's onWrite method in external shared-all persistence mode.
Because in shared-nothing mode writes are automatically distributed between the seeders (taking into account the degree of the space) and are done to local disk on each seeder, write performance scales along with the number of seeders (just as for a non-persistent space). However, when you use shared-all external persistence is used, because the persistence layer is shared (is a centralized RDBMS, for example) the number of writes per second is ultimately limited by what the external persistence layer can handle and does not scale when more seeders are added to the space.
● When memory is used as a transparent in-line cache (rather than to store the entire data set), if there is a request to read an entry (as a result of a get, take or lock operation, for example) that is not currently in the part of the data cached in memory, then the entry is automatically loaded from the persistence layer either automatically by the seeders from their local persistent storage in shared-nothing mode or by the persistence's onRead method.
● When a query (rather than a single entry read) is issued on the space, then when external shared-all persistence is used, the query will only return the matching records that are in the in-memory cache at the time, while with shared-nothing persistence, when indexes are used, the query will return ALL matching records, including those that may have been evicted at the time the query was issued.
Write-Behind Caching
ActiveSpaces supports write-behind caching in addition to write-through caching to a back-end database. Write-behind caching provides asynchronous writes to the database for faster performance, and allows writes to be buffered in cache and written to the database later in case the database is down.
The writes to the database are asynchronous to the cache operation. The process is fault-tolerant of Persister failures and seeder failures of the space up to its replication degree.
To enable the write-behind feature, you must set the shared-all persistence policy to ASYNC.
To indicate that an update failed to persist to the back-end data store due to the database connection being down, your persister code must return a PERSISTER_OFFLINE error within the onWrite callback.
The PERSISTER_OFFLINE error indicates that the update is to be retried later. ActiveSpaces does not handle any other persister-related failures, continues with a warning, and removes the specific update that failed.
The following example shows how your code can return the PERSISTER_OFFLINE error message.
ActionResult.create().setFailed(new ASException(ASStatus.PERSISTER_OFFLINE, sqlException))
The as-admin utility can display statistics indicating the pending update count on each node. If a system is not able to catch up with incoming user updates, the ToPersist count shows the status of pending updates.
Because writes are now asynchronous, changes done in cache are not reflected immediately in the database. ActiveSpaces conflates multiple updates on the same key to a single update to reflect the last update. If the database cannot keep up with cache updates, then updates on different keys are
aggregated in memory and potentially use more memory than expected. Running more Persisters might help scale out the workload.
If there are pending updates, caching with a defined limit cannot guarantee that the limit is respected.
Once all updates have been flushed, the limit is applied and eviction takes place. Until then, the cache might grow and necessary tuning might be required.