Caching Best Practices

The NetForum cache-update mechanism depends on proper configuration of the system. Use the steps in the sections below to configure your NetForum system to maintain cache synchronization across all three applications (iWeb, eWeb, and xWeb).

Make sure to update the sites to their latest version and configure properly for your environment to run smoothly.

Important! The EnableCachingOfMetaData system option should always be turned on for any NetForum site (whether production or test). The only reason to turn it off would be for some debugging/troubleshooting activities.

 

Cache Update Notification Overview

A Cache update clears the existing cached data that is affected. When that data is next requested, a fresh copy will be retrieved and cached.

You can trigger a cache update two ways:

  1. Using the Clear Cache function in the Admin module. Doing this deletes all existing cache for all three applications and all active users.
  2. Changing metadata, e.g. editing a form. Doing this deletes any cached data involving the updated metadata table(s).

How does the command to clear cache get communicated from iWeb to another NetForum application, such as eWeb or xWeb? The process is as follows:

You must have Administrator privileges in NetForum to perform the following steps.

  1. Go to the Admin module and choose the Clear cache group item link. The Initiate All Clear Cache page will appear. Press the OK button. This adds a row to the md_cache_clear_flag table indicating that all cache should be cleared.
    OR
    Make a change to metadata. Some ways you can change metadata include editing a dynamic form, or editing a child form. This will add a row to the md_cache_clear_flag table for each of the affected metadata tables, instructing NetForum to clear any cached data involving that table.

  2. SQLDependency starts to work. Using SQL Dependency functionality, NetForum has instructed SQL Server to notify it of any changes to results of the query:: “SELECT [caf_key],[caf_table_name] FROM [dbo].[md_cache_clear_flag]” and notifies all running NetForum apps (eWeb, iWeb, or xWeb) that something has changed in md_cache_clear_flag.

  3. The individual app (eWeb, iWeb, or xWeb) then responds to this notification by deleting its cache entries that involve the tables indicated in the recent md_cache_clear_flag records.

In order for this process to run smoothly, you must use the following best practices.

Setting up SQLDependency

  • In each (iWeb, eWeb, and xWeb) web.config file, make sure the following statement exists in the <appSettings> section and that the value is set to true:
    <add key="EnableSQLDependency" value="true" />.
  • Check the SQL Server for the NetForum database and ensure that the Service Broker is enabled.

    Note: Notification requires SQL Broker to be turned ON. Sites using Row-Level-Security might experience problems with this configuration.

  • Check the AppPool Identity(s) associated with each app (iWeb, eWeb, and xWeb). Make sure each AppPool Identity has sufficient privileges in SQL Server to turn on SQL Notification.

Setting up Disk Cache Folders

The following System Options control the location of the cache folders for the three NetForum apps (iWeb, eWeb, and xWeb). The following list shows the best practice for each setting:

  • CacheRootFolder: blank
  • CacheRootFolder_eWeb: blank
  • CacheRootFolder_xWeb: blank

Using this configuration causes NetForum to use the default configuration of a single, shared CacheRoot\Cache\ folder located in the same directory as the iWeb\, eWeb\ and xWeb\ application folders. Having all three cache folders in the same place and at the same level as the (iWeb, eWeb, or xWeb) application folders is a best practice.

Troubleshooting Cache not Updating in eWeb/xWeb

Step 1: Make sure that the EnableCachingOfMetaData system option is turned on. If not, turn it on and restart the application pool.

Step 2: Turn on enhanced log4net logging of cache processes in the iWeb/eWeb/xWeb web.config.

Inside the <log4net> section of web.config (usually at near the bottom of the file), add the following block to define the “logger” that logs the CacheProvider activity and its “appender” which defines the “CacheProvider-Log.txt” file that the log entries get written to.

Copy 
<log4net>

    … existing content

    <!-- The name=”CacheProvider” controls all Cache-related log4net logging since this logging is tagged in the source code with the name “CacheProvider”-->
    <logger additivity="false" name="CacheProvider">
        <level value="DEBUG" />
        <appender-ref ref="CacheLogFileAppender" />
    </logger>
    <appender name="CacheLogFileAppender" 
    type="log4net.Appender.RollingFileAppender">
        <param name="File" value="CacheProvider-Log.txt" />
        <param name="AppendToFile" value="true" />
        <rollingStyle value="Size" />
        <maxSizeRollBackups value="10" />
        <maximumFileSize value="10MB" />
        <staticLogFileName value="true" />
        <lockingModel type="log4net.Appender.FileAppender+MinimalLock" />
        <layout type="log4net.Layout.PatternLayout">
        <conversionPattern value="%-date %message%newline" />
        </layout>
    </appender>
</log4net>

 

This CacheProvider-Log.txt file will appear in the eWeb (or xWeb) root folder.

Saving this change to the web.config will cause the eWeb (or xWeb) app pool to recycle.

 

Step 3: Verify that the CacheProvider-Log.txt file is created

If you then go to eWeb and update anything, you should see this CacheProvider-Log.txt file in its root folder. The top of the file should contain the start-up steps of the CacheProvider.

2021-10-08 15:25:09,742 Application: 'eWeb', CacheProvider: 'Avectra.netForum.Data.CacheProvider+MemoryCacheProvider'. Created new instance of MemoryCacheProvider in MemoryCacheProvider.CacheProviderInstance.Get. Items in Memory Cache: 0.

2021-10-08 15:25:09,745 Application: 'eWeb', CacheProvider: 'Avectra.netForum.Data.CacheProvider+DiskCacheProvider'. Created new instance of DiskCacheProvider in DiskCacheProvider.CacheProviderInstance.Get. Disk Cache folder: C:\Work_NetForum\GIT\AltRepos\NFEAlt2\eWeb\..\CacheRoot\Cache

2021-10-08 15:25:09,747 Application: 'eWeb', CacheProvider: 'Avectra.netForum.Data.CacheProvider+MemoryAndDiskCacheProvider'. Created new instance of MemoryAndDiskCacheProvider in MemoryAndDiskCacheProvider.CacheProviderInstance.Get. Items in Memory Cache: 0. Disk Cache folder: C:\Work_NetForum\GIT\AltRepos\NFEAlt2\eWeb\..\CacheRoot\Cache

2021-10-08 15:25:09,749 Application: 'eWeb', CacheProvider: 'Avectra.netForum.Data.CacheProvider'. ============ In CanRequestNotifications, called from Application_Start ============

2021-10-08 15:25:09,811 Application: 'eWeb', CacheProvider: 'Avectra.netForum.Data.CacheProvider'. In CanRequestNotifications, called from Application_Start. SQL Broker is enabled.

2021-10-08 15:25:09,812 Application: 'eWeb', CacheProvider: 'Avectra.netForum.Data.CacheProvider'. In CanRequestNotifications, called from Application_Start. Returned TRUE

2021-10-08 15:25:09,814 Application: 'eWeb', CacheProvider: 'Avectra.netForum.Data.CacheProvider'. In StopSqlDependency, called from Application_Start

2021-10-08 15:25:09,816 Application: 'eWeb', CacheProvider: 'Avectra.netForum.Data.CacheProvider'. StopSqlDependency ran successfully, called from Application_Start

2021-10-08 15:25:09,818 Application: 'eWeb', CacheProvider: 'Avectra.netForum.Data.CacheProvider'. In StartSqlDependency, called from Application_Start

2021-10-08 15:25:09,963 Application: 'eWeb', CacheProvider: 'Avectra.netForum.Data.CacheProvider'. StartSqlDependency ran successfully, called from Application_Start

2021-10-08 15:25:10,417 Application: 'eWeb', CacheProvider: 'Avectra.netForum.Data.CacheProvider'. In StartCacheSyncMonitor

2021-10-08 15:25:10,447 Application: 'eWeb', CacheProvider: 'Avectra.netForum.Data.CacheProvider'. StartCacheSyncMonitor ran successfully.

2021-10-08 15:25:10,448 Application: 'eWeb', CacheProvider: 'Avectra.netForum.Data.CacheProvider+MemoryAndDiskCacheProvider'. Called StartCacheSyncMonitor and assigned CacheEventDispatch.ChangeEvent from MemoryAndDiskCacheProvider.CacheProviderInstance.Get

2021-10-08 15:25:10,594 Application: 'netFORUM eWeb', Browser: (Chrome). User '' added first new entry to their empty CachedTablePermissions and set Config.CurrentUserCachedTablePermissionsTimeStamp = 19:25:10.586 (637693179105865481 Ticks)

 

If something was mis-configured (SQL Broker off on the database, etc.) you should see error messages here. Resolve those configuration issues.

 

Step 4: Verify that eWeb reacts to a simple System Option value change

Now try changing some innocuous System Option value in iWeb and verify that eWeb got notified and cleared its cache in response.

For example, change the System Option CommandTimeout setting by 1 second. (e.g., if it is currently set to 240, change it to 241).

Then review the CacheProvider-Log.txt file, you should see that

  • eWeb was notified via the SQL Dependency on the md_cache_clear_flag table

  • the Memory cache and Disk cache were flushed of entries in regions containing “_fws_”

  • clearSystemOptions was called

    •  

2021-10-08 15:29:17,246 Application: 'eWeb', CacheProvider: 'Avectra.netForum.Data.CacheProvider'. -----------> In DependencyOnChange triggered by SQLDependency on changes to md_cache_clear_flag table.

2021-10-08 15:29:17,247 Application: 'eWeb', CacheProvider: 'Avectra.netForum.Data.CacheProvider'. Calling OnMdCacheClearFlagChange

2021-10-08 15:29:17,251 Application: 'eWeb', CacheProvider: 'Avectra.netForum.Data.CacheProvider'. In OnMdCacheClearFlagChange

2021-10-08 15:29:17,269 Application: 'eWeb', CacheProvider: 'Avectra.netForum.Data.CacheProvider+MemoryCacheProvider'. MemoryCacheProvider.FlushCache(region='_fws_')

2021-10-08 15:29:17,269 Application: 'eWeb', CacheProvider: 'Avectra.netForum.Data.CacheProvider+MemoryCacheProvider'. Before Flush Cache Memory Content Count = 102')

2021-10-08 15:29:17,270 Application: 'eWeb', CacheProvider: 'Avectra.netForum.Data.CacheProvider+MemoryCacheProvider'. After Flush Cache Memory Content Count = 102'

2021-10-08 15:29:17,271 Application: 'eWeb', CacheProvider: 'Avectra.netForum.Data.CacheProvider+DiskCacheProvider'. DiskCacheProvider.FlushCache(region='_fws_'), CacheFolder=C:\Work_NetForum\GIT\AltRepos\NFEAlt2\eWeb\..\CacheRoot\Cache

2021-10-08 15:29:17,274 Application: 'eWeb', CacheProvider: 'Avectra.netForum.Data.CacheProvider'. In OnMdCacheClearFlagChange - clearSystemOptions

2021-10-08 15:29:17,283 Application: 'eWeb', CacheProvider: 'Avectra.netForum.Data.CacheProvider'. In StartCacheSyncMonitor

2021-10-08 15:29:17,294 Application: 'eWeb', CacheProvider: 'Avectra.netForum.Data.CacheProvider'. StartCacheSyncMonitor ran successfully.

If something went wrong, it should be logged here.

 

Step 5: Verify changing a System Option that has a visible effect in eWeb

Here you want to change a System Option that affects something you can see in eWeb. If you have Require Strong Passwords enabled in Admin > eWeb Login Security, you should be able to use the eWebEnableStoredPayments System Option to enable/disable the users’ ability to use Saved payment info in the eWeb Checkout.

If the eWebEnableStoredPayments system option is turned on, the Saved payment: dropdown will appear in the Payment Information section of the Checkout page as shown below:

So, note the current value of the eWebEnableStoredPayments system option.

Login to eWeb. Add something to your cart and go to checkout.

You should see the Saved payment dropdown or not, based on the current value of that system option.

Leave the checkout page (e.g., click on the shopping cart icon, or something else).

Go to iWeb and flip the setting of eWebEnableStoredPayments system option.

Check the CacheProvider-Log.txt file, you should see the same lines as in step 4 (verify from the timestamps that these lines were written just now in response to you changing the System Option.)

Go back to eWeb and go back to the Shopping Cart checkout page and verify that the Save payment dropdown is present or missing as expected based on what you just changed the eWebEnableStoredPayments setting to. Restore the System Option to its original value.

If iWeb cache updates seem to work and are adding the corresponding record(s) into md_cache_clear_flag, but eWeb/xWeb is not getting the update, check the cache logs in eWeb/xWeb.

Check for the following entries (in this example with region=’_fws_’) if you changed a system option. If you had changed something in md_column, for example, it would have been with region=’_mdc_’

2021-10-08 15:29:17,246 Application: 'eWeb', CacheProvider: 'Avectra.netForum.Data.CacheProvider'. -----------> In DependencyOnChange triggered by SQLDependency on changes to md_cache_clear_flag table.

2021-10-08 15:29:17,247 Application: 'eWeb', CacheProvider: 'Avectra.netForum.Data.CacheProvider'. Calling OnMdCacheClearFlagChange

2021-10-08 15:29:17,251 Application: 'eWeb', CacheProvider: 'Avectra.netForum.Data.CacheProvider'. In OnMdCacheClearFlagChange

2021-10-08 15:29:17,269 Application: 'eWeb', CacheProvider: 'Avectra.netForum.Data.CacheProvider+MemoryCacheProvider'. MemoryCacheProvider.FlushCache(region='_fws_')

 

If you do not see those, the SQLDependency part is not working in eWeb/xWeb.

SQL Dependency not working – reason 1

SQL Broker not enabled on the database. (This can be broken when restoring a database backup to a new database. Your new database will be pointing to the SQL Broker of the original database that was backed-up – and that does not work.) You need to create a new broker.

 

SQL Dependency not working – reason 2

eWeb/xWeb web.config <appSettings> is missing the <add key="EnableSQLDependency" value="true" />

 

SQL Dependency not working – reason 3

The database owner might be an undefined user. This can happen when creating the database from a backup of a database that existed on another server.

If this is the problem, inserting the records into md_cache_clear_flag will try to trigger the notifications, but they will fail. You can check with:

    SELECT enqueue_time,transmission_status from sys.transmission_queue

If you see one or more rows with a transmission_status containing:

    An exception occurred while enqueueing a message in the target queue. Error: 15517, State: 1. Cannot execute as the database principal because the principal "dbo" does not exist, this type of principal cannot be impersonated, or you do not have permission.

Then it could be this database owner problem.

The fix is to set the database owner to a valid database principal

To investigate further:

SELECT owner_sid, SUSER_SNAME(owner_sid) AS dbowner

FROM sys.databases

WHERE name = '<databasename>’

SELECT name,sid, SUSER_SNAME(sid) AS dbo

FROM <databasename>.sys.database_principals

WHERE name = 'dbo';

If the dbo column in the second query is null, that specified database owner is not properly defined.

You fixed this by re-setting the database owner to a known good user with a command like

    ALTER AUTHORIZATION ON DATABASE::NFEP2017PACK TO [CB\Netforumapp]

(Where NFEP2017PACK was the name of the database you were fixing and [CB\Netforumapp] is the user you were making the new database owner).

If the above ALTER AUTHORIZATION ON DATABASE… command gives error “The proposed new database owner is already a user or aliased in the database.”, you can look at this suggestion from https://subhrosaha.wordpress.com/2012/04/26/sql-server-error-the-proposed-new-database-owner-is-already-a-user-or-aliased-in-the-database/

After fixing the owner issue, the notifications started to work:

  • Seeing the relevant updated values reflected in eWeb/xWeb behavior

  • Seeing expected content in Cache log files

  • Running “SELECT enqueue_time,transmission_status from sys.transmission_queue” on the database a few moments after making the metadata change should return no records since the queue entries have already been added/read/discarded