This package contains internal GemStone classes for implementating caching on top of the GemFire Distributed System.

# Local Regions

LocalRegion implements the basic caching mechanism and allows for subclasses to perform message distribution and other specialization of LocalRegion functionality. A LocalRegion is an implementation of the jre Map interface that supports expiration, callbacks, server cache communication, and so on.

LocalRegion has a RegionMap that holds the actual data for the region.

Most changes to an entry in a LocalRegion are performed in three steps:

    * The entry is modified under synchronization using an instance of
    EntryEventImpl object.  The event
    is also queued for later callback invokation under this synchronization.

    * Distribution is allowed to occur outside of synchronization

    * Synchronization is again obtained on the entry and callbacks are invoked

A LocalRegion may also have a DiskRegion associated with it for persistence or overflow to disk.

# Distributed Regions

Distributed Regions are a subclass of LocalRegion that interact with locking and the DistributedSystem to implement distributed caching. Most DistributedRegion operations are carried out using subclasses of DistributedCacheOperation.

# Partitioned Regions

The contents of a partitioned region is spread evenly across multiple members of a distributed system. From the user's standpoint, each member hosts a partition of the region and data is moved from partition to partition in order to provide scalability and high availability. The actual implementation of partitioned regions divides each partition into sub-partitions named "buckets". A bucket may be moved from one partition to another partition in a process called "migration" when GemFire determines that the partitioned region's data is not spread evenly across all members. When a bucket reaches a maximum size, it is split in two and may be migrated to a different partition.

Data is split among buckets using the Extensible Hashing algorithm that hashes data based upon the lower-order bits ("mask") of the data's (the `Region` entry's key in the case of GemFire) value. All partitions of a given region share a directory that maintains a mapping between a mask and information about the bucket that holds data that applies to the mask. When an entry is placed into a partitioned region, the bucket directory is consulted to determine which member(s) of the distributed system should be updated. The Extensible Hashing algorithm is useful when a bucket fills us with data and needs to be split. Other hashing algorithm require a complete rebalancing of the partitioned region when a bucket is full. Extensible Hashing, however, only requires that the full bucket be split into two, thus allowing the other buckets to be accessed without delay. The below diagram demonstrates bucket splitting with extensible hashing.

A `BucketInfo` contains metadata about a bucket (most importantly the locations of all copies of the bucket) that is distributed to members that access the partitioned region. Changes to the `BucketDirectory` metadata are coordinate through GemFire's distributed lock service. Inside of a region partition are a number of `Bucket`s that hold the values for keys that match the bucket's mask as shown in the below diagram.

The total size (in bytes) of a bucket is maintained as key/value pairs are added. It is not necessary for the bucket to store the value of a region entry as an actual object. So, the bucket stores the value in its serialized `byte` form. This takes up less space in the VM's heap and allows us to accurately calculate its size. The entry's key, however, is used when looking up data in the bucket and must be deserialized. As an estimate, the size of the key object is assumed to the size of object's serialized `byte`s. When a entry's value is replaced via an update operation, the size of the old value is subtracted from the total size before the size of the new value is added in. It is assumed that the key does not change size.

When a bucket's size exceeds the "maximum bucket size", it is split in two based on the extensible hashing algorithm: a new `Bucket` is created and is populated with the key/value pairs that match its mask, the `Bucket`'s local depth is incremented by 1, update the global depth if the new local depth exceeds the current global depth. The splitting process is repeated while all of the following conditions are met: the size of either bucket continues to exceed the "maximum bucket size", the full bucket has more than 1 element, the global depth is less than the "maximum global depth".

### Primary Bucket

One bucket instance is selected as the primary. All bucket operations target the primary and are passed on to the backups from the primary.

Identification of the primary is tracked using metadata in `BucketAdvisor`. The following diagram shows the standard state transitions of the `BucketAdvisor`:

### Partitioned Region Cache Listeners

User CacheListeners are registered on the PartitionedRegion. Activity in the Buckets may fire callbacks on the PartitionedRegion's CacheListeners. The following tabled figures attempt to demonstrate the logic and sequence involved.

Definition of Participants
pr_A1 | a pure accessor
pr_B1_pri | a datastore which hosts primary for bucket B1
pr_B1_c1 | a datastore which hosts copy 1 for Bucket B1
pr_B1_c2 | a datastore which hosts copy 2 for Bucket B1
pr_A2_listener pr_A3_bridge pr_A4_gateway | pure accessors with CacheListener, Bridge or Gateway

Fig. 1 (Flow of Put to CacheListeners)
pr_A1 | pr_B1_pri | pr_B1_c1 | pr_B1_c2 | pr_A2_listener pr_A3_bridge pr_A4_gateway
putMessage1 --> | operateOnPartitionRegion() |  |  |
| sync entry |  |  |
| update entry |  |  |
| UpdateOperation.distribute |  |  |
| if bucket add adjunct.recips |  |  |
| send ----------------------> | --> see Fig. 2 <------- reply |  |
| send ----------------------> | ---------> | --> see Fig. 2 <------- reply |
| if adjunct.recips > 0 |  |  |
| PutMessage2 (notificationOnly == true) |  |  |
| send --------------------------> | ---------> | ---------> | --> CacheListener fires on pr iif InterestPolicy != CACHE_CONTENT <------- reply
| waitForReplies (from all above msgs) |  |  |
| fire local CacheListener on pr |  |  |
| release entry sync |  |  |

Fig. 2 (Processing of UpdateOperation by non-Primary Bucket Host)
sync entry
update entry
CacheListener fires on pr iif InterestPolicy != CACHE_CONTENT
release entry sync
reply

### Migration

Buckets are "migrated" to other members of the distributed system to ensure that the contents of a partitioned region are evenly spread across all members of a distributed system that wish to host partitioned data. After a bucket is split, a migration operation is triggered. Migration may also occur when a `Cache` exceeds its `maxParitionedData` threshold and when a new member that can host partitioned data joins the distributed system. Each member is consulted to determine how much partitioned region data it is currently hosting and the org.apache.geode.cache.Cache#getMaxPartitionedData maximum amount of partitioned region data it can host. The largest bucket hosted by the VM is migrated to the member with the large percentage of space available for partitioned data. This ensures that data is spread evenly among members of the distributed system and that their space available partitioned region data fills consistently. Migration will continue until the amount of partitioned data hosted by the member initiating the migration falls below the average for all members. When a member that hosts partitions {@linkplain org.apache.geode.cache.Cache#close closes} its `Cache`, the partitions are migrated to other hosts.

### High Availability

The high availability ( org.apache.geode.cache.PartitionAttributes#getRedundancy redundancy) feature of partitioned regions effects the implementation in a number of ways. When a bucket is created, the implementation uses the migration algorithm to determine the location(s) of any redundant copies of the buckets. A warning is logged if there is not enough room (or not enough members) to guarantee the redundancy of the partitioned region. When an entry is `put` into a redundant partitioned region, the key/value is distributed to each bucket according to the consistency specified by the region's scope. That is, is the region is `DISTRIBUTED_ACK`, the `put` operation will not return until it has received an acknowledgment from each bucket. When a `get` is performed on a partitioned region and the value is not already in the partitioned region's local cache, a targeted `netSearch` is performed. When there are redundant copies of the region's buckets, the `netSearch` chooses one bucket at random from which to fetch the value. If the bucket does not respond within a given timeout, then the process is repeated on another randomly chosen redundant bucket. If the bucket has been migrated to another member, then the member operating on the region will re-consult its metadata and retry the operation. When redundant buckets are migrated from one machine to another, the implementation is careful to ensure that multiple copies of a bucket are not hosted by the same member.

# System Properties

All of the system properties used by GemFire are discussed here.