MACRIM.Core 6.8.1

MACRIM.Caching Project

Purpose

The MACRIM.Caching project extends the MACRIM framework’s caching capabilities to support table-based caching in Redis, replacing the in-memory LookupService with a scalable, persistent solution. It caches IEntity instances for SQL tables (or other data sources), enabling fast lookups for batch processing (5-10K rows) with dynamic indexing and segmented caching. The rebuilt RedisService addresses compile errors from the original CacheService<T>, integrating with IConfigService for configuration-driven caching.

Phase 1: Table-Based Caching

Phase 1 implements caching in Redis, focusing on:

• Caching: Stores IEntity instances as Redis Hashes (segment:cacheName:segment:catalog) with TTL from expires.
• Indexing: Indexes jsondata via keypattern expressions (e.g., {catalog},{name},{catalog}:{name}) using Redis Sets (segment:cacheName:index:value).
• Loading: Retrieves data via IRepositoryBypass.SearchAsync, using LoadJSON for IEntity creation. Synchronized to ensure only one load per cache, with concurrent requests waiting.
• Upsert: Updates cache with IEntity instances (SQL sync deferred to Phase 2).
• Configuration: Uses IConfigService (_configs[segment]["commands"]["caching"][cacheName]) for table, profile, keypattern, expires, segment.
• Synchronization: Syncs on initial load, TTL expiration, and cache misses. No ongoing sync within TTL.
• Lookups: Supports lookups by any keypattern index entry (e.g., svc123) using GetRecord/GetRecordAsync, with expressions evaluated via ApplyLookups.

Usage Example

To perform a lookup like {?:all-services(svc123,(catalog))} (retrieve a service by svc123 and return its catalog):

1. Configure Cache:

   <command name="all-services" table="services" profile="sql" expires="600" segment="true" keypattern="{catalog},{name},{servicename},{catalog}:{name}">
       <methods>
           <method name="retrieve" type="MACRIM.Data.Extensions.Base.Search">
               <parameters>
                   <parameter name="model" value="services" />
                   <parameter name="profile" value="sql" />
                   <parameter name="segment" value="{var:tenant}" />
               </parameters>
           </method>
       </methods>
   

2. Set Up DI:

   services.AddSingleton<ITableCachingService, RedisService>();
   

3. Update ApplyLookups (as static extension method):

   case "lookup":
   case "?":
       var cacheSvc = me.Context.Services.GetRequiredService<ITableCachingService>();
       if (cacheSvc == null) break;
       var argStart = what.IndexOf('(');
       var instruct = argStart > -1 ? what.Substring(++argStart) : string.Empty;
       instruct = instruct.Length > 0 ? instruct.Remove(instruct.Length - 1) : string.Empty;
       what = argStart > -1 ? what.Substring(0, argStart - 1) : what;
   
       if (what == ".")
       {
           var pstart = instruct.IndexOf(',');
           var lkey = instruct.Split(',').FirstOrDefault();
           var lexp = pstart > -1 ? instruct.Substring(++pstart) : string.Empty;
           if (lkey.StartsWith('(')) lkey = (string)me.ApplyLookups(lkey.Replace('(', '{').Replace(')', '}'), returnType, subReturnType, command, defaults);
           var el = me.LookupCache.ContainsKey(lkey) ? me.LookupCache[lkey] : me.Spawn();
           if (el != null && el.Found)
           {
               oFound = el.ApplyLookups(lexp.Replace('(', '{').Replace(')', '}'), returnType, subReturnType, command, defaults);
           }
       }
       else
       {
           var pstart = instruct.IndexOf(',');
           var lkey = instruct.Split(',').FirstOrDefault();
           var lexp = pstart > -1 ? instruct.Substring(++pstart) : string.Empty;
           if (string.IsNullOrEmpty(lkey) || string.IsNullOrEmpty(lexp))
           {
               oFound = string.Empty;
               break;
           }
   
           var keyvalue = me.Attributes.GetValueOrDefault(lkey, "");
           if (lkey.StartsWith('(')) keyvalue = (string)me.ApplyLookups(lkey.Replace('(', '{').Replace(')', '}'), returnType, subReturnType, command, defaults);
           if (string.IsNullOrEmpty(keyvalue))
           {
               oFound = string.Empty;
               break;
           }
   
           var record = cacheSvc.GetRecord(
               keyvalue: keyvalue,
               cacheName: what,
               segment: me.Context?.Segment ?? "default"
           );
   
           if (record != null && record.Found)
           {
               oFound = record.ApplyLookups(lexp.Replace('(', '{').Replace(')', '}'), returnType, subReturnType, command, defaults);
           }
           else
           {
               oFound = string.Empty;
           }
       }
       break;
   

4. Test Lookup:

   • Input: {?:all-services(svc123,(catalog))} with me.Attributes["servicename"] = "svc123".
   • Output: oFound = "cat1" (from record.ApplyLookups("{catalog}", ...)).
   • Complex Input: {?:all-services(svc123,(user:{id}:name:upper))} → oFound = "JOHN".

Phase 2: Queuing (Deferred)

Phase 2 will implement async SQL sync using Redis Lists, leveraging StartWatchingList, Polly retries, and a poison queue.

Key Components

• Interfaces (in MACRIM.Core.Abstractions):
  • ITableCachingService: LoadCacheAsync, GetRecordAsync, GetRecord, UpsertRecordAsync (version: a37df99a-233d-424d-9c2d-fd5b56875548).
  • IQueueService, QueueTask (Phase 2).
• RedisService (in MACRIM.Caching):
  • Implements ICachingService and ITableCachingService (version: b9f8c7b5-4d2e-4f9b-9e2c-6f7a8b9c0f4e).
  • Uses ConnectionHelper for Redis.
  • Resolves IRepositoryBypass via IContextService.Services.
• Configuration:
  • IEntity with attributes (table, profile, keypattern, expires, segment).

IEntity Mapping

• segment: Attributes["segment"], partition key.
• catalog: Attributes["catalog"], typically Key.
• jsondata: Attributes (e.g., {"segment": "seg1", "catalog": "cat1", "name": "John", "age": "30"}).
• Key: Attributes["catalog"].
• Element: Table name (e.g., contacts).
• Value: Null.
• Index: keypattern expressions (e.g., Index["cat1"], Index["cat1:John"]).

Implementation Status

• Phase 1 (Complete):
  • RedisService.cs rebuilt for caching, supporting lookups by any keypattern index entry using keyvalue (e.g., svc123).
  • Removed profile parameter, deriving it from configuration.
  • Fixed batch execution error (ITransaction.ExecuteAsync).
  • Added synchronous GetRecord for ApplyLookups, optimized for high-frequency calls without logging.
  • Synchronized LoadCacheAsync to ensure one load per cache, with concurrent requests waiting.
• Phase 2 (Deferred): Queuing pending prioritization.

Next Steps

1. Test Phase 1:

   • Test RedisService with sample data (e.g., contacts, keypattern="{catalog},{name},{ssn},{catalog}:{name}").
   • Verify concurrent lookups (e.g., 20 queued processes) to ensure single cache load.
   • Validate {?:cacheName(keyvalue,(expression))} in ApplyLookups (e.g., {?:all-services(svc123,(catalog))}).

2. Sync Enhancements:

   • Decide if sync_version checks are needed for Phase 1 to detect table changes within TTL.
   • Plan Phase 2 for cache-to-SQL sync.

3. DI Confirmation:

   • Confirm Microsoft.Extensions.DependencyInjection and singleton lifetime.

4. T Handling:

   • Confirm BaseEntityConvertible for IRepository<T>, or specify models (e.g., Documents).

5. Configuration:

   • Provide example IEntity output for _configs[segment]["commands"]["caching"][cacheName].

6. Phase 2:

   • Define queuing requirements when ready.