Upgrade and data migration of user trail logging

During the upgrade to Airlock IAM 8.0, scripts attempt to configure and migrate outdated user trail logging settings to the new defaults (logging into a database). Although fresh installations of IAM 8.0 and later will solely use logging into the database by default, upgraded installations will keep on logging into the medusa-usertrail.log file in parallel until configured otherwise (see Step 4).

• The upgrade migration does not automatically import old log messages from the medusa-usertrail.log into the database. After successful automatic migration, proceed with Step 3 to check the logging functionality before importing old log messages into the database as required (see Step 5).
• When the automatic migration fails, the log list in Adminapp >> Users >> ​Activities tab remains empty because the logging database is not configured as required. In this case, user trail logs are still stored in the outdated medusa-usertrail.log file, and user activities can be looked up in the Logs menu for each IAM module.

0. Requirements for automatic migration:
1. A single SQL Data Source is configured under MAIN SETTINGS >> Database User Persister.
2. No LDAP or Active Directory Database User Persister is configured. If this is the case, the user trail logging target database must be configured manually.
3. A custom plugin is available in the plugin tree. If this is the case, the migration must be done manually.

This article describes how to set up and configure the default logging into a database for Airlock IAM 8.0 and later. Depending on the actual IAM configurational state before the upgrade to 8.0, some of the steps may already be prepared and ready to use (e.g., a global data source is already configured), but the migration failed anyway.

If the migration fails partly, needed changes are displayed. Proceed with the matching configurational steps of the instructions in this article.

1. In the Config Editor, go to:
   MAIN SETTINGS >> Data Sources >> User Trail Data Source
2. On the User Trail Log Database Repository page:
• For property SQL Data Source, select the upgraded IAM database.
3. Open the SQL Data Source plugin and test the database connection.
4. The global database for user trail logging has been successfully configured. User trails can now be written into the database and the medusa-usertrail.log file in parallel.

1. In the Config Editor, go to:
   Loginapp >> Advanced Settings
2. Set property Log User Trail To Database to point to the User Trail Log Database Repository data source.
3. Repeat the above steps for all other IAM modules, i.e. Adminapp, Transaction Approval, etc.
4. The configured data source has been set as the database for user trail logging for each module.

0. After IAM upgrade and a successful automatic configuration migration:
1. User trails are shown in the Adminapp UI under:
   Users >> tab Activities
   Here, all user activities (logged to a corresponding user account) are accumulated in a list. For example, both Loginapp and Adminapp log entries to the user account will be displayed.

See also Clean-up task for user trail logs in database.

2. User trails can also be looked up in the Adminapp UI under Logs >> Application >> module name for each IAM module individually.

• User trail logging into the medusa-usertrail.log file has to be commented out in the following files:
• /log4j/all-modules.xml, central configuration for Loginapp and Adminapp logging.
• /log4j/loginapp.xml, individually logging configuration for the Loginapp.
• log4j/adminapp.xml, individually logging configuration for the Adminapp.

Example configuration of an /log4j/all-modules.xml file with disabled logging to file:

<?xml version="1.0" encoding="UTF-8"?> 
 
<Configuration name="Loginapp Log4j 2 Configuration"> 
	<Loggers> 
		<Logger name="com.airlock.iam.log.audit"> 
			<!-- <AppenderRef ref="IAM-AUDIT"/> --> 
		</Logger> 
 
		<!-- Re-enable file-based user trail log by uncommenting this block 
		<Logger name="com.airlock.iam.log.usertrail"> 
		        <AppenderRef ref="IAM-USER-TRAIL"/> 
		</Logger> 
		--> 
 
		<Root level="${sys:iam.log.level}"> 
			<!-- <AppenderRef ref="PARSABLE"/> --> 
		</Root> 
	</Loggers> 
</Configuration>

After disabling the logging into the medusa-usertrail.log file, restart the IAM instance and consider a log message import from the old logging file into the database.

When the logging into file configuration has been set to inactive, the log viewers entry for the user trail logs in the Adminapp UI might be removed. To do so, go to:

1. To do so, go to:
   Adminapp >> Log Viewer
2. In the property Logfiles, remove the plugin entry for the user trail log file.

A user trail log import task can import logs from the medusa-usertrail.log file into the database.

1. In the Config Editor, go to:
   Service Container >> Services >> Task Scheduler Config >> Service
2. On the Task Scheduler Service plugin:
   Add and configure a new log importer Task Schedule plugin in the property Tasks.
• The import should be configured as a one-shot task. Set property Interval to never.
3. In the Task Schedule plugin of the new log importer:
   Configure the property Task with the Plugin class User Trail Log Import Task and click the Create & Edit button.
4. On the User Trail Log Import Task plugin:
   Add the storage path of the medusa-usertrail.log file(s) to the property field Import Directory.

• The import task allows configuring a regex Filename Filter (e.g., medusa-usertrail\.log(\.\d+)?) to import only the required log files. Import from sub-directories is supported as well.
• Importing existing log files into the database multiple times will duplicate the user trail logs. The property Delete Files After Import can be used to remove imported log files permanently to avoid duplication.


Known Issue with large log files

The User Trail Log Import Task has a bug that prevents loading large log files into databases. Depending on the database type, it can only handle log files up to several thousand lines.

It is recommended to split large log files into small ones. This can be, for example, done using the following Linux command:

split -d -l 1000 orig-user-trail.log splitted-user-trail-

Note that the Filename Filter regular expression must be adapted to match the split files.

5. Activate the new configuration.
6. Click on the Adminapp UI menu Service Container to open the Service Container page.
7. To start the import task, go to:
   Adminapp >> Service Container.
   This opens up the new browser window/tab of the Service Container application, where the new task can be started in the Task Scheduler Service menu. Alternatively, start the new task using the CLI.


Tenant IDs are added to the imported log messages during import to accomplish strong data segregation.

8. Wait for the old log messages to be transferred into the database. The messages appear in the Adminapp UI under Users >> tab Activities. For logged details about the import, see servicecontainerapp.log.