PeopleSync V2.5 - Akumina Community

PeopleSync V2.5

IMPORTANT: For the latest PeopleSync file version, please reference PeopleSync V5 for clients running Akumina AppManager version 4.8 and later.  This version, PeopleSync V2.5, is ONLY for clients running Akumina AppManager version 4.0 or 4.5 and PeopleSync V2.

The PeopleSyncV2 application is used to synchronize data from Azure Active Directory to the storage location in Azure which provides for faster data access. PeopleSyncV2 is required if you are using user or group information for one or more of the following Akumina functionalities:

  • People Directory
  • Personalization and/or targeting
  • Akumina Workspaces
  • Leveraging AD Groups to provide permissions in AppManager

PeopleSyncV2 leverages the Azure Search Service to provide even faster data retrieval for optimum performance and scalability. The following are steps to configure PeopleSyncV2 with Azure Search Service.

 

Configure PeopleSync V2.5

  1. Download PeopleSync V2 latest preview version from https://akumina.azureedge.net/interchange/peoplesyncv2/peoplesyncv2.5.zip.  The prior version of the file can be downloaded from https://akumina.azureedge.net/interchange/peoplesyncv2/peoplesyncv2.zip.
  2. Unzip the downloaded PeopleSync file
  3. Open the config.json file for edit.
  4. Open the FieldsMappingsource.json file for edit.

Please note the zip file will contain individual configuration files for the configuration of Okta data source, Sharepoint ON-PREM data source and CosmosDB Storage configuration.

 

Key Descriptions Default
AzureStorageConnectionString From Azure Portal – This is the Storage Account > Access Keys>ConnectionString. If you stored your connection string in Key Vault you can specify the Key Vault URI
AzureSearchServiceName API Key generated during the Azure Search Service setup
AzureSearchServiceApiKey API Key generated during the Azure Search Service setup
RootBlobContainer The value set for the RootBlobContainer name must match what is set in the interchange.settings.config file, value for key “akumina:SettingsBlobContainerName”. If the key “akumina:SettingsBlobContainerName”. is not set in the interchange.settings.config then aksettings is the appropriate value. RootBlobContainer is the location where the persona data is stored. aksettings
EntityPrefix

 

EntityPrefix – can be changed, HOWEVER if you change this then you need to change this key in the interchange.settings.config

<add key=”akumina:PeopleDataEntityPrefix” value=”akad”/>

This is the prefix of the peoplesync items you will see in “Windows Azure Storage Explorer”.

On your storage container you will find new entries under “Queues and Tables” that are prefixed akad

akad

TenantId

 

Domain name used to login to your account.

For Azure Active Directory – TenantID or DirectoryID (in 4.0 Site Creator we call this the Subscription ID)

For Okta, set your subdomain. Example: For example.okta.com, the value is example.

ApplicationSecret

 

API Token generated in website.

For Azure Active Directory – AD APP Secret – (in 4.0 Site Creator we call the AAD Secret)

For Okta, refer to “Configuring Okta application secret” section.

FieldsMappingSource

 

JSON format.

Refer to steps in configuring field mapping section

AppManagerUrl AppManager Redirect URL
AppManagerQueryKey BackgroundProcessorKey
ResumeOnError

 

Leave this set as default. If there is an error it will retry the value set for “MaxRetryAttempt”: “3” number of times before it exists the process.

true

ContinueOnSchedule If you want this process to continue to run and pickup differential changes in Azure AD such as creating new user or groups, then change this to  false. If you are running on a server, you may want to leave this value as “true” for the initial sync, while you run many processes in parallel, then set this to false when you are running only differentials. The Schedule is set by key ScheduleIntervalInSeconds (described below).

false

ExitConsoleWhenDone If set to true then the console will close upon completion. true
ScheduleIntervalInSeconds If “ContinueOnSchedule” is set to true then a differential sync will occur every 60 seconds or whatever value desired and set for this key in seconds

60

WindowsGraphResourceUri

 

MAKE NO CHANGE HERE

https://graph.windows.net

MicrosoftGraphResourceUri, MAKE NO CHANGE HERE

https://graph.microsoft.com

ApiVersion MAKE NO CHANGE HERE

1.5

LoginUri MAKE NO CHANGE HERE

https://login.microsoftonline.com/{0}”,

QueueItemVisibilityTimeoutInMin If there are more than 5,000 users in Azure AD then this should be set to 5.

2 minutes

MaxRetryAttempt The number of times the process will retire before erroring out.

3

RetryDelayInMilliseconds If throttling issue in the cloud, then the system will retry after given interval.

1000

StorageAssemblyType This allows to execute c# code to store data to any storage. PeopleSync supports Azure Storage, FlatFile and CosmosDb.

The value should be specified in the format of “AssemblyName, FullName of the class.”

For AzureStorage:
Akumina.PeopleSync.Storage.AzureStorage, Akumina.PeopleSync.Storage.AzureStorage.CloudTableStorage

For FlatFile:
Akumina.PeopleSync.Sharepoint, Akumina.PeopleSync.Sharepoint.FlatFileStorage

For CosmosDb:
Akumina.PeopleSync.CosmosDb.SqlApi, Akumina.PeopleSync.CosmosDb.SqlApi.Storage

QueueAssemblyType This allows to execute c# code to execute the queue process for Azure Storage. The value should be in the format of “AssemblyName, FullName of the class.”

For AzureStorage:
Akumina.PeopleSync.Storage.AzureStorage, Akumina.PeopleSync.Storage.AzureStorage.CloudQueueAction

PersonaAssemblyType This allows to execute c# code to process the persona details and store data to respective storage. PeopleSync supports Azure Storage, FlatFile and CosmosDb. Same medium should be used in AppManager as well for Persona Configuration.

The value should be in the format of “AssemblyName, FullName of the class.”

For AzureStorage:
Akumina.PeopleSync.Storage.AzureStorage, Akumina.PeopleSync.Storage.AzureStorage.PersonaStorageClient

For FlatFile:
Akumina.PeopleSync.Sharepoint, Akumina.PeopleSync.Sharepoint.FlatFilePersonaClient

For CosmosDb:
Akumina.PeopleSync.Storage.CosmosDb.SqlApi, Akumina.PeopleSync.Storage.CosmosDb.SqlApi.PersonaClient

SearchAssemblyType This allows to execute c# code to execute the Azure Search Service. The value should be in the format of “AssemblyName, FullName of the class.”

For Azure Search Service:
Akumina.PeopleSync.Storage.AzureStorage, Akumina.PeopleSync.Storage.AzureStorage.SearchAdmin

For CosmosDb configuration, Azure Search Service is supported.

For Sharepoint ON-PREM configuration, Search Service is not required. We can keep the assembly value and Search Service configuration values as empty.

SyncSourceAssemblyType This allows you to execute c# code to extract data from any datasource for example Xls file, SQL server or another IDP. The value should be in the format of “AssemblyName,FullName of the class.” For the complete implementation refer to

https://github.com/akumina/AkuminaSamples

For Azure Active Directory:
Akumina.PeopleSync.AzureActiveDirectory, Akumina.PeopleSync.AzureActiveDirectory.SyncSource

For Okta Source:
Akumina.PeopleSync.Okta, Akumina.PeopleSync.Okta.OktaSource

For Sharepoint ON-PREM (FlatFile):
Akumina.PeopleSync.Sharepoint, Akumina.PeopleSync.Sharepoint.SharepointSource

CustomFilterAssemblyType This allows you to execute c# code logic to manipulate the data or remove the data on certain condition. The value should be in “AssemblyName,FullName of the class.” For the complete implementation refer to https://github.com/akumina/AkuminaSamples
SyncSourceAssemblyType This allows you to execute c# code to extract data from any datasource for example Xls file, SQL server or another IDP. The value should be in the format of “AssemblyName,FullName of the class.” For the complete implementation refer to

https://github.com/akumina/AkuminaSamples

FetchOptions MAKE NO CHANGE HERE
SyncUsers

 

If SyncUsers = true, you can configure the required properties from “UserProperties” and “UserExtendedProperties.” For better performance always review the fields from the properties.

true

SyncGroups If true, groups will be synced along with the users.

If SyncGroups = false, then all the following keys will be ignored:

SyncTags

SyncTypes

TagExtension

TypeExtension

true

SyncTags ONLY IF WORKPLACES IS REGISTERED

If SyncGroups= true and SyncTags is true then the group tags will be synced along with the groups.

 

This requires TagExtention to be set.

It indicates whether group tags will be synced along with the groups.

 

false

SyncTypes ONLY IF WORKPLACES IS REGISTERED

If SyncGroups= true and SyncTypes = true

This requires TypeExtension to be set

 

false

 TagExtension ONLY IF WORKPLACES IS REGISTERED This setting is used if SyncGroups=true and SyncTags is true.

It specifies the name of the schema extension property that is added to Microsoft Graph Groups to store the tag values. The value for this property can be found in your site’s “DigispaceConfigurationIDS_AK” list in SharePoint. You will see an item named “WorkspaceSchemaExtension.” Edit that item and copy the following highlighted value that is specific to your tenant:

“tags”:{“id”:”extxxxx_grouptags”

 

TypeExtension

 

ONLY IF WORKPLACES IS REGISTERED

This setting is used if SyncGroups=true and SyncTags is true.

 

It specifies the name of the schema extension property that is added to Microsoft Graph Groups to store the Type value. The value for this property can be found in your site’s “DigispaceConfigurationIDS_AK” list in SharePoint. You will see an item named “WorkspaceSchemaExtension.” Edit that item and copy the following highlighted value that is specific to your tenant:

“type”:{“id”:”extxxxxxxx_grouptype”

 

UserProperties User Properties that are pulled in from Azure AD for each user. This is configurable to match the properties you have/want to pull in from Azure AD for each user. Properties can be added or removed.

 

If you remove or add a property, it will be reflected on any new or changed user only. To update for all users a full sync is required.

The following fields are applicable to AAD (other providers check the fields mapping file):

AccountEnabled, City, CompanyName, Country, DeletionTimestamp, Department, DirSyncEnabled, DisplayName, FacsimileTelephoneNumber, GivenName, JobTitle, Mail, MailNickName, Mobile, ObjectId, ObjectType, PhysicalDeliveryOfficeName,PostalCode, PreferredLanguage, SipProxyAddress, State, StreetAddress, Surname, TelephoneNumber, UsageLocation, UserPrincipalName, UserType, OtherMails, ProvisionedPlans, ProvisioningErrors, ProxyAddresses

UserExtendedProperties Refer to the User Extended Properties table below.

If you remove or add a property, it will be reflected on any new or changed user only. To update for all users a full sync is required.

The following fields are applicable to AAD (other providers check the fields mapping file):

Manager, AssignedLicenses, AppRoleAssignments, DirectReports, OwnedDevices, RegisteredDevices, OtherProperties

GroupProperties Group Properties that are pulled in from Azure AD for each group. This is configurable to the customers desire. Properties can be added or removed..

If you remove or add a property, it will be reflected on any new or changed user only. To update for all users a full sync is required.

Description, DisplayName, ObjectId

MergeGroupFieldsToUserGroup Set this value to true if you want to extract group values to usergroup mapping. This helps index all group details that are part of usergroup entries.

false

PersonaUserProperties User properties that are already available in User Table will be merged to Persona mapping. Properties can be added or removed. Newly added property will be included after the persona sync command. But if you remove a property, it will be reflected only after the Persona re-creation or full sync process. Properties should be added as comma separated values.

UserGroupUserProperties User properties that are already available in User Table will be merged to UserGroup mapping. Properties can be added or removed. If you remove or add a property, it will be reflected on any new or changed user only. To update for all users a full sync is required. Properties should be added as comma separated values.

ExpandProperties User properties that are already available in user table will be expanded. The output of the new expanded properties in the format of _. Only object value will be expanded and this won’t support any array elements. Properties in the object should be unique.

ExternalConnection This is the External Connection we specify to sync the data. Current External Connection will support to fetch the data from Sharepoint ON-PREM and setting CosmosDb and Flat file Storages.
If you stored your connection string in Key Vault, then this external connection attributes can be stored in key vault as well.

Sample configuration template:

“ExternalConnection”: {
“EndPointUrl”: “”,
“Domain”: “”,
“Username”: “”,
“Password”: “”,
“UploadPath”: “”
“Database”: “”
“Throughput”: “”
},

EndPointUrl This is the External Connection we specify to sync the data. Current External Connection will support to fetch the data from Sharepoint ON-PREM and setting CEndpoint URL used to define the External Connection endpoint detail. PeopleSync uses this property to determine the endpoint for CosmosDb and SharepointUrl. One endpoint must be mapped.

For Sharepoint ON-PREM, the EndPointUrl should be Sharepoint URL path.

For CosmosDb Storage, the EndPointUrl should be Connection URI of CosmosDb, which contains the AccountEndPoint and AccountKey values.

Domain Used to specify the domain for the Sharpoint Admin user account.

Used for Sharepoint ON-PREM

Username Used to specify the username of the Sharpoint Admin user account.

Used for Sharepoint ON-PREM

Pasword Used to specify the password of the Sharpoint Admin user account.

Used for Sharepoint ON-PREM

UploadPath Local path to store the File

Used for FileStorage option used in Sharepoint ON-PREM

Database Database Id/Name used for CosmosDb Storage

Throughput Database Throughput used for CosmosDb Storage

StoreInSingleContainer If the peoplesync data should be stored in single container for CosmosDb then we need to set this flag as true. Default setting value is false, which will create separate collection for User, Group, Persona and UserGroup.

SamplingCount Total number of rows require to get the fields for generating the PersonaQueryFields.json and Generating the IndexColumns in Azure Search Service. Default value is 1000. If any missing column in Azure Search Service, increase the sampling count based on the requirement.

1000

 

UserExtendedProperties Fields retrieved
Manager

Manager

AssignedLicenses

All Licenses and Count

AppRoleAssignments

Id, ObjectId, PrincipalDisplayName, PrincipalId, PrincipalType, ResourceDisplayName, ResourceId

DirectReports

DisplayName, GivenName, Surname, Mobile, City, CompanyName, Country, Department, JobTitle, Mail, MailNickname, FacsimileTelephoneNumber, ObjectId

OwnedDevices

DeviceId, DeviceMetadata, DeviceObjectVersion, DeviceOSType, DeviceOSVersion, DeviceTrustType, DisplayName

RegisteredDevices

DeviceId, DeviceMetadata, DeviceObjectVersion, DeviceOSType, DeviceOSVersion, DeviceTrustType, DisplayName

OtherProperties

All the other external properties configured in the AAD

 

Configuring keyvault to store sensitive config elements

If you need to secure connectionstring, tenantid, clientid and secrets then please follow the following steps.

  1. Create a JSON file in this format, please note only the following fields supported at this time thru keyvault:

    {

        “AzureStorageConnectionString”“<YOUR CONNECTION STRING>”,

        “AzureSearchServiceName”“<SEARCH INSTANCE NAME>”,

        “AzureSearchServiceApiKey”“<SEARCH INSTANCE KEY>”,

        “TenantId”“<TENANT ID>”,

        “ApplicationId”“<AAD APPLICATION ID>”,

        “ApplicationSecret”“<AAD APPLICATION SECRET>”,

        “ExternalConnection”“{\“EndPointUrl”:\ “<SHAREPOINT URL PATH / COSMOS DB CONNECTION URI>\”,
    \“Domain\”:\“<DOMAIN FOR THE ADMIN USER ACCOUNT>\”,
    \“Username\”:\“<USERNAME FOR THE ADMIN ACCOUNT>\”,
    \“Password\”:\“<PASSWORD FOR THE ADMIN ACCOUNT>\”,
    \“UploadPath\”:\“<LOCAL PATH TO STORE THE FILES>\”,
    \“Database\”:\“<COSMOS DB NAME>\”,
    \“Throughput\”:\“<COSMOS DB THROUGHPUT\”,
    }”

      }

  2. Create a keyvault secret then set the value to above JSON file.
  3. Open PeopleSync config.json and update the AzureStorageConntetionString to URI from the above step.

Fields Mapping Source

For any customization of the filed, for example in your AD you may have a field called _firstname but in the Akumina System you want to map to FirstName, then the filed mapping will become “FirstName”:”_firstname” for complete details refer the fieldmapping.*.json

 

Any value that starts with “~” is considered to be a constant. For example: “ObjectType” : “~Group” will be stored in our system as ObjectType=”Group”.

 

Here’s an example of AAD fields mapping:

Prefix: Set unique value to this field if you need to initialize prefix to fields, this is required to uniquify the fields if you sync cross multiple data sources.

 

Generating the Azure Tables

It is important that the following steps are run COMPLETELY before continuing with the configuration.

Access the PeopleSyncV2 unzipped directory. Run the following commands by double clicking on them in the exact order:

UserAndGroup.cmd

Wait until that command has started processing data, then click on…

Queue.cmd

When both commands have completed, close the associated command prompts.

 

Creating the Indexes of the Azure PeopleSyncV2.5 data

When the Azure tables have been completely generated (both commands above have completed), execute the following command to generate the azure search indexes:

addSearchIndex.cmd

When this command has completed close the command prompt. This initial generation of the indexes takes some time.

Configuring Personas with PeopleSync V2.5

  1. Open commands folder then run the files in the following sequence:
    a. SyncPersonaSchema.cmd
    b. AddSearchIndex.cmd
    c. Persona.Cmd
  2. For incremental sync schedule, use the Persona.cmd using task scheduler

Configure and Start the Differential Sync

After the createSearchIndex.cmd has completed, edit the config.json file again and set the following keys:

ContinueOnSchedule“: true,

Verify the following key is set:

ExitConsoleWhenDone“: false,

Set the Differential time as desired:

ScheduleIntervalInSeconds“: 60,

Start the differential sync by executing the command:

UserGroupAndQueue.cmd

 

Configuring AppManager

Do not configure AppManager to use PeopleSearchQuery until AFTER the search service has been generated and the indexes are available.
  • Update the following key in the unity.config file to set the value to “PeopleSearchQuery”

<register type=”IPeopleQuery” mapTo=”PeopleSearchQuery” />

  • Update the following keys in the AppManager interchange.settings.config file. Note these values must match the values set in the PeopleSyncV2 config.json file.

<!–EntityPrefix used for PeopleDirectory and indirect AAD Group permissions–>

<add key=”akumina:PeopleDataEntityPrefix” value=””/>

<!–Azure Search Service Name–>

<add key=”akumina:AzureSearchServiceName” value=””/>

<!–Azure Search Key–>

<add key=”akumina:AzureSearchServiceApiKey” value=””/>

  • Recycle the app pool.

 

Running PeopleSync as a WebJob

If you are hosting the AppManager as webapp, then you can run PeopleSync as a webjob that is part of the webapp, or create a dedicated webapp just for PeopleSync. The PeopleSync can run as a WebJob that is running “Continuously” or “Triggered”.

Continuously Running WebJob

In Azure portal select yourWebApp, on 2nd blade scroll down and select “Diagnostic logs”.

Set:

  • “Web server logging” = File System
  • Quota = 35 MB  (this is default and should be enough)
  • Retention Period (Days) = 7
  • SAVE

Add the Continuous WebJob to your WebApp

In Azure portal select your WebApp.

  1. On 2nd blade scroll down and select “WebJobs”
  2. Select “+ADD”
  3. Enter Name of WebJob
  4. Click on folder icon and select the zipped-up peoplesync from your local computer
  5. Set Type = Continuous
  6. Select Scale = Multi Instance
  7. Click “OK”

Triggered WebJob

Configure the PeopleSync for Triggered Run

 

Add the Triggered WebJob to your WebApp

In Azure portal select your WebApp.

  1. On 2nd blade scroll down and select “WebJobs”
  2. Select “+ADD”
  3. Enter Name of WebJob
  4. Click on folder icon and select the zipped-up people sync from your local computer
  5. Set Type = Triggered
  6. Set Tiggers = Scheduled
  7. Set CRON Expressing = 0 */1 * * * *
  8. Click OK

 

Running PeopleSync on a Virtual Machine

  1. Install .NET Core 2.1.x or greater on your VM https://dotnet.microsoft.com/download/dotnet-core/2.1

NOTE:  The more Processors you have the faster the people Sync will run. It is multi-threaded and will generate a thread for each of your cores to process.

 

Start the Initial People sync

For the initial run of PeopleSync

  • Right click and “Run as Administrator” command: .
  • Right click and “Run as Administrator” command: (Note: several command prompt widows will open running the process queue in parallel threads depending on the number of processors you have on your machine)

 

Creating the Azure Search Service

Setting up the Azure Search Service requires administrator permissions to the Azure Portal

Access the Azure Portal. Click on “All Services” (left rail) and search for “Search Services” and select.

  • On the Search Service Blade click on “+Add
  • URL – enter a URL (the value entered is your AzureSearchServiceName) – all lower case, example clientnameprodpeoplesearch
  • Subscription – Select the appropriate Subscription.
  • Resource group – Use existing – and select the appropriate resource group.
  • Location – Set as the client preferred location, in my example East US 2
  • Pricing tier – based on number of users and indexes customer can choose anywhere from basic to standard. You can start with basic and scale up as needed.

Acquiring the AzureSearchServiceApiKey

On the Search service page, select the AzureSearchServiceName, in this example clientnameprodpeoplesearch.

A set of blades will open. Select the navigation blade item “Keys” and copy the “PRIMARY ADMIN KEY” – this is the value need for the AzureSearchServiceApiKey.

NOTE: If you have Multiple AppManagers and they are all interfacing to the same O365 tenant, then you need only one PeopleSyncV2 to run. AppManagers will use the same PeopleSyncEnityPrefix, AzureSearchServiceName and AzureSearchServiceApiKey Keys. If your AppManager’s websites are interfaced to different Azure Tenants, then you will need to generate a Search service for each AppManager pointing to a different Tenant.

 

Best Practices for running PeopleSyncV2

  1. For better performance remove the properties that are not required by your implementation.  For example, only enable the User Extended Properties if you need the information in the User Context to support site functionality.
  2. The more processors that your VM has the faster it will process the queue items.  For example, if you have 16 processors, then it will generate 16 threads of the process queue and process the items fast. Azure WebJobs will scale automatically to utilize all the available processes.
  3. If you are running on a VM and you have multiple VM’s you can copy the entire PeopleSync Directory to another VM and  run startQueueSync.cmd at the same time.

 

Configuring Okta application secret

This is here for sample purposes only to show you how to configure Okta:

  • Login as an Administrator for Okta Account using domain name.
  • From Application home page, use “Admin” button to navigate to Admin Dashboard page
  • Navigate “API” link in “Security” menu to land into “Token” page
  • Click on “Create Token” button to initiate the token generation screen
  • Enter any token name in the field and click “Create Token” button to generate token.
  • New token will be generated and displayed in the screen. Copy this token for future reference.
  • This token will expire in 30 days if it’s not used. Expiration will extend for next 30 days whenever we use it. Please make sure to configure the valid Token to run the sync.

Here’s an example of Okta fields mapping:

Views: 2553
//]]>