Chat Operations Test

A chat is a collection of chat messages between one or more participants. Participants can be users or apps.

Microsoft Teams enables users to quickly and securely communicate with their peers and also helps to securely retrieve/update the properties of the chat, list/retrieve one/more members, and update the role of a member in a chat through any desktop/device. However, if Microsoft Teams is unable to deliver the high-quality experience it promises while users communicate through chat messages, IT administrators will be forced to hunt for alternatives. To avoid such an outcome, administrators should make sure that the guaranteed Microsoft Teams service levels are delivered at all times. For this, administrators will have to periodically check the time it takes to update the chat, retrieve the chat properties, list and update the role of the members in the chat used for communication using Microsoft Teams, isolate bottlenecks proactively, and plug the holes before users complain. The Chat Operations test helps administrators in this exercise.

At configured intervals, this test emulates an update to the properties of the chat, retrieve the member collection of the chat and update the role of a member in the chat and reports the time it takes to perform each of these operations. This provides administrators with early indicators of problem conditions, thus enabling them to resolve the problem before any irrepairable damage is done.

Note:

This test uses the Microsoft Graph API. Tests that use Microsoft Graph API may not start reporting metrics right away. Sometimes, they may go without reporting metrics for over 48 hours. This is normal behavior, and it occurs because, Microsoft does not collect/refresh the metrics as frequently as the test executes.

Target of the test : Microsoft Teams

Agent deploying the test : A remote agent

Outputs of the test : One set of results for the monitored Microsoft Teams

Configurable parameters for the test
Parameters Description

Test period

How often should the test be executed

Host

The host for which the test is to be configured. By default, this is portal.office.com

Tenant Name

Azure AD certificate-based authentication (CBA) enables customers to allow or require users to authenticate with X.509 certificates against their Azure Active Directory (Azure AD) for applications and browser sign-in.

This test requires a valid X.509 certificate, so that it can securely access the target tenant and its resources. For this purpose, you should do the following:

  1. Enable Azure AD Certificate-based authentication for the target O365 tenant; this can be achieved manually, via the Office 365 portal, or automatically, using Powershell scripts we provide. For the manual procedure, refer to Manually Enabling Certificate-based Authentication For an Office 365 Tenantunder Microsoft Office 365. For the automatic procedure, refer to Automatically Fulfilling Pre-requisites in a Modern Authentication-Enabled Environmentunder Microsoft Office 365.

    When enabling certificate-based authentication, an X.509 certificate will be generated for the target tenant.

  2. Configure the Tenant Name parameter with the name of the tenant for which certificate-based authentication is enabled. Using the tenant name, the test will be able to read the details of the X.509 certificate that is generated for that tenant, and use that certificate to access that tenant's resources. To determine the tenant name, do the following:

    • Log in to the Microsoft 365 Admin Center as an administrator.

    • Under Setup, click on Domains.

    • Find a domain that ends with .onmicrosoft.com - this is your Microsoft O365 tenant name.

O365 User Name, O365 Password, and Confirm Password

For execution, this test requires the privileges of an O365 user who has been assigned the Service support admin role and is vested with the View-Only Audit Logs and Team administrator permission. Configure the credentials of such a user against O365 User Name and O365 Password text boxes. Confirm the password by retyping it in the Confirm Password text box.

Domain Name, Domain User Name, Domain Password, and Confirm Password

These parameters are applicable only if the eG agent needs to communicate with the Office 365 portal via a Proxy server.

In this case, in the Domain text box, specify the name of the Windows domain to which the eG agent host belongs. In the Domain User Name text box, mention the name of a valid domain user with login rights to the eG agent host. Provide the password of that user in the Domain Password text box and confirm that password by retyping it in the Confirm Password text box.

On the other hand, if the eG agent is not behind a Proxy server, then you need not disturb the default setting of these parameters. By default, these parameters are set to none.

Proxy Host, Proxy Port, Proxy User Name, Proxy Password and Confirm Password

These parameters are applicable only if the eG agent needs to communicate with the Office 365 portal via a Proxy server.

In this case, provide the IP/host name and port number of the Proxy server that the eG agent should use in the Proxy Host and Proxy Port parameters, respectively.

If the Proxy server requires authentication, then specify the credentials of a valid Proxy user against the Proxy User Name and Proxy Password text boxes. Confirm that password by retyping it in the Confirm Password text box. If the Proxy server does not require authentication, then specify none against the Proxy User Name, Proxy Password, and Confirm Password text boxes.

On the other hand, if the eG agent is not behind a Proxy server, then you need not disturb the default setting of any of the Proxy-related parameters. By default, these parameters are set to none.

Team Members

By default, this test emulates the chat operations performed on the target Microsoft Teams. For this, this test should create a chat message communication between two Office 365 users. When the test runs for the first time, this test will create a chat message involving these two users. The eG agent will then collect the required metrics based on the chat operations involving these two Office 365 users. Specify the name of such users separated by a comma against this text box. Note that the Office 365 users used for this purpose need not be licensed users.

Measurements made by the test
Measurement Description Measurement Unit Interpretation

Get chat status

Indicates whether/not retrieval of the properties and relationships of a chat was successful.

 

The values that this measure can report and their corresponding numeric values are discussed in the table below:

Measure Value

Numeric Value

Success

1

Failed

0

Note:

By default, this measure reports the Measure Values discussed in the table above. However, in the graph of this measure, the retrieval status of the properties and relationships of a chat is indicated using the numeric equivalents only.

Time taken to get chat

Indicates the time taken by this test to retrieve the properties and relationships of the chat.

Seconds

Ideally, the value of this measure should be low. A high value is indicative of a poor user experience while sending/receiving chat messages. The reasons for the same should be investigated.

Update chat status

IIndicates whether/not the update to the properties of the chat was successful.

 

The values that this measure can report and their corresponding numeric values are discussed in the table below:

Measure Value

Numeric Value

Success

1

Failed

0

Note:

By default, this measure reports the Measure Values discussed in the table above. However, in the graph of this measure, the update status of the properties of the chat is indicated using the numeric equivalents only.

Time taken to update chat

Indicates the time taken by this test to update the properties of the chat.

Seconds

Ideally, the value of this measure should be low. A high value is indicative of a poor update experience with the properties of the chat. The reasons for the same should be investigated.

Get member status

Indicates whether/not the retrieval of a single user in the chat was successful.

 

The values that this measure can report and their corresponding numeric values are discussed in the table below:

Measure Value

Numeric Value

Success

1

Failed

0

Note:

By default, this measure reports the Measure Values discussed in the table above. However, in the graph of this measure, the retrieval status of a single user in the chat is indicated using the numeric equivalents only.

Time taken to get member

Indicates the time taken by this test to retrieve a single user in the chat.

Seconds

Ideally, the value of this measure should be low. A high value is indicative of a poor user experience while sending/receiving chat messages. The reasons for the same should be investigated.

List members status

Indicates whether/not the retrieval / listing all users in the chat was successful.

 

The values that this measure can report and their corresponding numeric values are discussed in the table below:

Measure Value

Numeric Value

Success

1

Failed

0

Note:

By default, this measure reports the Measure Values discussed in the table above. However, in the graph of this measure, the retrieval status of all the users in the chat is indicated using the numeric equivalents only.

Time taken to list members

ndicates the time taken by this test to retrieve/list all the users in the chat.

Seconds

Ideally, the value of this measure should be low. A high value is indicative of a poor user experience listing the users in the chat. The reasons for the same should be investigated.

Send message status

Indicates whether/not sending a new chat message in the chat was successful.

 

The values that this measure can report and their corresponding numeric values are discussed in the table below:

Measure Value

Numeric Value

Success

1

Failed

0

Note:

By default, this measure reports the Measure Values discussed in the table above. However, in the graph of this measure, the status of sending a new chat message in the chat is indicated using the numeric equivalents only.

Time taken to send message

Indicates the time taken by this test to send a new chat message in the chat.

Seconds

Ideally, the value of this measure should be low. A high value is indicative of a poor user experience while sending/receiving chat messages in the chat. The reasons for the same should be investigated.

Get message status

Indicates whether/not retrieving a single message or a message reply in the chat was successful.

 

The values that this measure can report and their corresponding numeric values are discussed in the table below:

Measure Value

Numeric Value

Success

1

Failed

0

Note:

By default, this measure reports the Measure Values discussed in the table above. However, in the graph of this measure, the status of retrieving a single message or a message reply in the chat is indicated using the numeric equivalents only.

Time taken to get message

Indicates the time taken by this test to retrieve a single message or a message reply in the chat.

Seconds

Ideally, the value of this measure should be low. A high value is indicative of a poor user experience while sending/receiving chat messages in the chat. The reasons for the same should be investigated.

List message status

Indicates whether/not retrieving the list of messages in the chat was successful.

 

The values that this measure can report and their corresponding numeric values are discussed in the table below:

Measure Value

Numeric Value

Success

1

Failed

0

Note:

By default, this measure reports the Measure Values discussed in the table above. However, in the graph of this measure, the status of retrieving the list of messages in the chat is indicated using the numeric equivalents only.

Time taken to list message

Indicates the time taken by this test to retrieve the list of messages in the chat.

Seconds

Ideally, the value of this measure should be low. A high value is indicative of a poor user experience while sending/receiving chat messages in the chat. The reasons for the same should be investigated.