PeopleSync V2 - Akumina Community

PeopleSync V2

Estimated reading time: 9 min

IMPORTANT: You must be running Akumina AppManager version 4.0.1802.x or later and PeopleSync V2 version 2.1.0 or greater to use the PeopleSync V2

The PeopleSyncV2 application is used to synchronize data from Azure Active Directory to the storage location in Azure which provides for faster data access. PeopleSyncV2 is required if you are using user or group information for one or more of the following Akumina functionalities:

  • People Directory
  • Personalization and/or targeting
  • Akumina Workspaces
  • Leveraging AD Groups to provide permissions in AppManager

PeopleSyncV2 leverages the Azure Search Service to provide even faster data retrieval for optimum performance and scalability. Following are the steps to configuring PeopleSyncV2 with Azure Search Service.

 

Configure PeopleSync V2

  1. Download PeopleSync V2 latest from https://akumina.azureedge.net/interchange/peoplesyncv2/peoplesyncv2.zip.
  2. Unzip the downloaded PeopleSyncV2.zip file
  3. Open the config.json file for edit.
  4. Open the FieldsMappingsource.json file for edit.
  5. You must set all of the highlighted values in the following table. The other values may be left as is or updated depending on your requirements.
Key Descriptions Default
AzureStorageConnectionString From Azure Portal – This is the Storage Account > Access Keys>ConnectionString. If you stored your connection string in Key Vault you can specify the Key Vault URI
AzureSearchServiceName API Key generated during the Azure Search Service setup
AzureSearchServiceApiKey API Key generated during the Azure Search Service setup
EntityPrefix

 

EntityPrefix – can be change HOWEVER if you change this then you need to change this key in the interchange.settings.config

<add key=”akumina:PeopleDataEntityPrefix” value=”akad”/>

This is the prefix of the peoplesync items you will see in “Windows Azure Storage Explorer”.

On your storage container you will find new entries under “Queues and Tables” that are prefixed akad

akad

TenantId

 

Domain name used to login to your account.

For Azure Active Directory – TenantID or DirectoryID (in 4.0 Site Creator we call this the Subscription ID)

For Okta, set your subdomain. Example: For example.okta.com, the value is example.

ApplicationSecret

 

API Token generated in website.

For Azure Active Directory – AD APP Secret – (in 4.0 Site Creator we call the AAD Secret)

For Okta, refer to “Configuring Okta application secret” section.

FieldsMappingSource

 

JSON format.

Refer to steps in configuring field mapping section

AppManagerUrl AppManager Redirect URL
AppManagerQueryKey BackgroundProcessorKey
ResumeOnError

 

Leave this set as default – if there is an error it will retry the value set for “MaxRetryAttempt”: “3” number of times before it exists the process..

true

ScheduleIntervalInSeconds If you want this process to continue to run tand pickup differential changes in Azure AD such as creating new user or groups, then change this to false. If you are running on a server, you may want to leave this value as “true” for the initial sync, while you run many processes in parallel, then set this to false when you are running only differentials. The Schedule is set by key ScheduleIntervalInSeconds (described below).

false

ExitConsoleWhenDone If set to true then the Console will close upon completion. true
ScheduleIntervalInSeconds If “ContinueOnSchedule” is set to true then a differential sync will occur every 60 seconds or whatever value desired and set for this key in seconds

60

WindowsGraphResourceUri

 

MAKE NO CHANGE HERE

https://graph.windows.net

MicrosoftGraphResourceUri, MAKE NO CHANGE HERE

https://graph.microsoft.com

ApiVersion MAKE NO CHANGE HERE

1.5

LoginUri MAKE NO CHANGE HERE

https://login.microsoftonline.com/{0}”,

QueueItemVisibilityTimeoutInMin If there are more than 5,000 users in Azure AD then this should be set to 5

2 minutes

MaxRetryAttempt Number of times the Process will retire before erroring out

3

RetryDelayInMilliseconds If throttling issue in the cloud, then the system will retry after given interval

1000

CustomFilterAssemblyType This allows you to execute c# code logic to manipulate the data or remove the data on certain condition. The value should be in “AssemblyName,FullName of the class”. For the complete implementation refer tohttps://github.com/akumina/AkuminaSamples
SyncSourceAssemblyType This allows you to execute c# code to extaract data from any datasource for example Xls file, SQL server or another IDP. The value should be in “AssemblyName,FullName of the class”. For the complete implementation refer to

https://github.com/akumina/AkuminaSamples

FetchOptions MAKE NO CHANGE HERE
SyncUsers

 

If SyncUsers = true, you can configure the required properties from “UserProperties” and “UserExtendedProperties”. For better performance always review the fields from the properties

true

SyncGroups If true, groups will be synced along with the users

If SyncGroups = false, then all the following keys will be ignored:

SyncTags

SyncTypes

TagExtension

TypeExtension

true

SyncTags ONLY IF WORKPLACES IS REGISTERED

If SyncGroups= true and SyncTags is true then the group tags will be synced along with the groups.

 

This requires TagExtention to be set.

It indicates whether group tags will be synced along with the groups.

 

false

SyncTypes ONLY IF WORKPLACES IS REGISTERED

If SyncGroups= true and SyncTypes = true

This requires TypeExtension to be set

 

false

 TagExtension ONLY IF WORKPLACES IS REGISTEREDThis setting is used if SyncGroups=true and SyncTags is true.

It specifies the name of the schema extension property that is added to Microsoft Graph Groups to store the Tag values. The value for this property can be found in your site’s “DigispaceConfigurationIDS_AK” list in SharePoint. You will see an item named “WorkspaceSchemaExtension”. Edit that item and copy the following highlighted value that is specific to your tenant:

“tags”:{“id”:”extxxxx_grouptags”

 

TypeExtension

 

ONLY IF WORKPLACES IS REGISTERED

This setting is used if SyncGroups=true and SyncTags is true.

 

It specifies the name of the schema extension property that is added to Microsoft Graph Groups to store the Type value. The value for this property can be found in your site’s “DigispaceConfigurationIDS_AK” list in SharePoint. You will see an item named “WorkspaceSchemaExtension”. Edit that item and copy the following highlighted value that is specific to your tenant:

“type”:{“id”:”extxxxxxxx_grouptype”

 

UserProperties User Properties that are pulled in from Azure AD for each user. This is configurable to match the properties you have/want to pull in from Azure AD for each user. Properties can be added or removed.

 

If you remove or add a property, it will be reflected on any new or changed user only. To update for all users a full sync is required.

The following fields are applicable to AAD (other providers check the fields mapping file):

AccountEnabled, City, CompanyName, Country, DeletionTimestamp, Department, DirSyncEnabled, DisplayName, FacsimileTelephoneNumber, GivenName, JobTitle, Mail, MailNickName, Mobile, ObjectId, ObjectType, PhysicalDeliveryOfficeName,PostalCode, PreferredLanguage, SipProxyAddress, State, StreetAddress, Surname, TelephoneNumber, UsageLocation, UserPrincipalName, UserType, OtherMails, ProvisionedPlans, ProvisioningErrors, ProxyAddresses

UserExtendedProperties Refer to the User Extended Properties table below.

If you remove or add a property, it will be reflected on any new or changed user only. To update for all users a full sync is required.

The following fields are applicable to AAD (other providers check the fields mapping file):

Manager, AssignedLicenses, AppRoleAssignments, DirectReports, OwnedDevices, RegisteredDevices, OtherProperties

GroupProperties Group Properties that are pulled in from Azure AD for each group. This is configurable to the customers desire. Properties can be added or removed..

If you remove or add a property, it will be reflected on any new or changed user only. To update for all users a full sync is required.

Description, DisplayName, ObjectId

MergeGroupFieldsToUserGroup Set this value to true if you want to extract group values to usergroup mapping. This helps index all group details that are part of usergroup entries.

false

 

UserExtendedProperties Fields retrieved
Manager

Manager

AssignedLicenses

All Licenses and Count

AppRoleAssignments

Id, ObjectId, PrincipalDisplayName, PrincipalId, PrincipalType, ResourceDisplayName, ResourceId

DirectReports

DisplayName, GivenName, Surname, Mobile, City, CompanyName, Country, Department, JobTitle, Mail, MailNickname, FacsimileTelephoneNumber, ObjectId

OwnedDevices

DeviceId, DeviceMetadata, DeviceObjectVersion, DeviceOSType, DeviceOSVersion, DeviceTrustType, DisplayName

RegisteredDevices

DeviceId, DeviceMetadata, DeviceObjectVersion, DeviceOSType, DeviceOSVersion, DeviceTrustType, DisplayName

OtherProperties

All the other external properties configured in the AAD

 

Configuring Personas with PeopleSync V2

  1. Open commands folder, open config.json set the value for RootBlobContainer this value should be same as AppManager “akumina:SettingsBlobContainerName”
  2. Open commands folder then run the files in the following sequence:a. SyncPersonaSchema.cmdb. AddSearchIndex.cmdc. Persona.Cmd
  3. For incremental sync schedule, use the Persona.cmd using task scheduler

Configuring keyvault to store sensitive config elements

If you need to secure connectionstring, tenantid, clientid and secrets then please follow the following steps.

  1. Create a JSON file in this format, please note only the following fields supported at this time thru keyvault:

    {

        “AzureStorageConnectionString”“<YOUR CONNECTION STRING>”,

        “AzureSearchServiceName”“<SEARCH INSTANCE NAME>”,

        “AzureSearchServiceApiKey”“<SEARCH INSTANCE KEY>”,

        “TenantId”“<TENANT ID>”,

        “ApplicationId”“<AAD APPLICATION ID>”,

        “ApplicationSecret”“<AAD APPLICATION SECRET>”

      }

  2. Create a keyvault secret then set the value to above JSON file.
  3. Open PeopleSync config.json and update the AzureStorageConntetionString to URI from the above step.

Fields Mapping Source

For any customization of the filed, for example in your AD you may have a field called _firstname but in the Akumina System you want to map to FirstName, then the filed mapping will become “FirstName”:”_firstname” for complete details refer the fieldmapping.*.json

 

Any value that starts with “~” is considered to be a constant. For example: “ObjectType” : “~Group” will be stored in our system as ObjectType=”Group”.

 

Here’s an example of AAD fields mapping:

Prefix: Set unique value to this field if you need to initialize prefix to fields, this is required to uniquify the fields if you sync cross multiple data sources.

 

Generating the Azure Tables

It is important that the following steps are run COMPLETELY before continuing with the configuration.

Access the PeopleSyncV2 unzipped directory. Run the following commands by double clicking on them in the exact order:

UserAndGroup.cmd

wait until that command has started processing data, then click on…

Queue.com

When both commands have completed, close the associated command prompts.

 

Creating the Indexes of the Azure PeopleSyncV2 data

When the Azure tables have been completely generated (both commands above have completed), execute the following command to generate the azure search indexes:

addSearchIndex.cmd

When this command has completed close the command prompt. This initial generation of the indexes takes some time.

 

Configure and Start the Differential Sync

After the createSearchIndex.cmd has completed, edit the config.json file again and set the following keys:

ContinueOnSchedule“: true,

Verify the following key is set:

ExitConsoleWhenDone“: false,

Set the Differential time as desired:

ScheduleIntervalInSeconds“: 60,

Start the differential sync by executing the command:

UserGroupAndQueue.cmd

 

Configuring AppManager

Do not configure AppManager to use PeopleSearchQuery until AFTER the search service has been generated and the indexes are available.
  • Update the following key in the unity.config file to set the value to “PeopleSearchQuery”

<register type=”IPeopleQuery” mapTo=”PeopleSearchQuery” />

  • Update the following keys in the AppManager interchange.settings.config file. Note these values must match the values set in the PeopleSyncV2 config.json file.

<!–EntityPrefix used for PeopleDirectory and indirect AAD Group permissions–>

<add key=”akumina:PeopleDataEntityPrefix” value=””/>

<!–Azure Search Service Name–>

<add key=”akumina:AzureSearchServiceName” value=””/>

<!–Azure Search Key–>

<add key=”akumina:AzureSearchServiceApiKey” value=””/>

  • Restart your AppManager in IIS (or update the web.config with a “space”) so the changes will take effect.

 

Running PeopleSync as a WebJob

If you are hosting the AppManager as webapp, then you can run PeopleSync as a webjob that is part of the webapp, or create a dedicated webapp just for PeopleSync. The PeopleSync can run as a WebJob that is running “Continuously” or “Triggered”.

Continuously Running WebJob

In Azure portal select yourWebApp, on 2nd blade scroll down and select “Diagnostic logs”.

Set:

  • “Web server logging” = File System
  • Quota = 35 MB  (this is default and should be enough)
  • Retention Period (Days) = 7
  • SAVE

Add the Continuous WebJob to your WebApp

In Azure portal select your WebApp.

  1. On 2nd blade scroll down and select “WebJobs”
  2. Select “+ADD”
  3. Enter Name of WebJob
  4. Click on folder icon and select the zipped-up peoplesync from your local computer
  5. Set Type = Continuous
  6. Select Scale = Multi Instance
  7. Click “OK”

Triggered WebJob

Configure the PeopleSync for Triggered Run

 

Add the Triggered WebJob to your WebApp

In Azure portal select your WebApp.

  1. On 2nd blade scroll down and select “WebJobs”
  2. Select “+ADD”
  3. Enter Name of WebJob
  4. Click on folder icon and select the zipped-up people sync from your local computer
  5. Set Type = Triggered
  6. Set Tiggers = Scheduled
  7. Set CRON Expressing = 0 */1 * * * *
  8. Click OK

 

Running PeopleSync on a Virtual Machine

  1. Install .NET Core 2.1.x or greater on your VMhttps://dotnet.microsoft.com/download/dotnet-core/2.1

NOTE:  The more Processors you have the faster the people Sync will run. It is multi-threaded and will generate a thread for each of your cores to process.

 

Start the Initial People sync

For the initial run of PeopleSync

  • Right click and “Run as Administrator” command: .
  • Right click and “Run as Administrator” command: (Note: several command prompt widows will open running the process queue in parallel threads depending on the number of processors you have on your machine)

 

Creating the Azure Search Service

Setting up the Azure Search Service requires administrator permissions to the Azure Portal

Access the Azure Portal. Click on “All Services” (left rail) and search for “Search Services” and select.

  • On the Search Service Blade click on “+Add
  • URL – enter a URL (the value entered is your AzureSearchServiceName) – all lower case, example clientnameprodpeoplesearch
  • Subscription – Select the appropriate Subscription.
  • Resource group – Use existing – and select the appropriate resource group.
  • Location – Set as the client preferred location, in my example East US 2
  • Pricing tier – based on number of users and indexes customer can choose anywhere from basic to standard. You can start with basic and scale up as needed.

Acquiring the AzureSearchServiceApiKey

On the Search service page, select the AzureSearchServiceName, in this example clientnameprodpeoplesearch.

A set of blades will open. Select the navigation blade item “Keys” and copy the “PRIMARY ADMIN KEY” – this is the value need for the AzureSearchServiceApiKey.

NOTE: If you have Multiple AppManagers and they are all interfacing to the same O365 tenant, then you need only one PeopleSyncV2 to run. AppManagers will use the same PeopleSyncEnityPrefix, AzureSearchServiceName and AzureSearchServiceApiKey Keys. If your AppManager’s websites are interfaced to different Azure Tenants, then you will need to generate a Search service for each AppManager pointing to a different Tenant.

 

Best Practices for running PeopleSyncV2

  1. For better performance remove the properties that are not required by your implementation.  For example, only enable the User Extended Properties if you need the information in the User Context to support site functionality.
  2. The more processors that your VM has the faster it will process the queue items.  For example, if you have 16 processors, then it will generate 16 threads of the process queue and process the items fast. Azure WebJobs will scale automatically to utilize all the available processes.
  3. If you are running on a VM and you have multiple VM’s you can copy the entire PeopleSync Directory to another VM and  run startQueueSync.cmd at the same time.

 

Configuring Okta application secret

This is here for sample purposes only to show you how to configure Okta:

  • Login as an Administrator for Okta Account using domain name.
  • From Application home page, use “Admin” button to navigate to Admin Dashboard page
  • Navigate “API” link in “Security” menu to land into “Token” page
  • Click on “Create Token” button to initiate the token generation screen
  • Enter any token name in the field and click “Create Token” button to generate token.
  • New token will be generated and displayed in the screen. Copy this token for future reference.
  • This token will expire in 30 days if it’s not used. Expiration will extend for next 30 days whenever we use it. Please make sure to configure the valid Token to run the sync.

Here’s an example of Okta fields mapping:

Was this article helpful?
Dislike 0
Views: 187
//]]>