Azure Non Interactive Sign-ins Test

The Sign-in logs provided by the Azure Active Directory (AD) portal is a treasure-chest of information about user sign-ins to the Azure organization and how signed-in users use the organization's resources.

One of the four types of sign-in logs offered by Azure AD is the Non Interactive Sign-in log. Non-interactive user sign-ins are sign-ins that were performed by a client app or OS components on behalf of a user. Like interactive user sign-ins, these sign-ins are done on behalf of a user. Unlike interactive user sign-ins, these sign-ins do not require the user to provide an authentication factor. Instead, the device or client app uses a token or code to authenticate or access a resource on behalf of a user. In general, the user will perceive these sign-ins as happening in the background of the user’s activity.

Frequent non-interactive sign-in failures can degrade the overall Azure sign-in experience. To assure users of an above-par experience, administrators should be able to spot sign-in failures rapidly, find out the reasons for the same, and fix them.

Besides sign-in failures, administrators should also watch out for suspicious actions related to user accounts. Additionally, administrators should also keep an eye out for disturbing sign-in patterns. For instance, a specific user/IP address/location may be attempting to sign-in more often than normal. Likewise, a unusually large number of sign-in attempts may be made for a specific application/service principal/resource. Such abnormal activities could hint at an attempted security breach. To avert the breach, administrators should be able to spot such deviant sign-in patterns quickly and investigate them.

It is also important for administrators to know which authentication method was popularly used at sign-in - legacy authentication or modern authentication. Less secure authentication methods need to be identified, so that administrators can disable them for a tenant.

With the help of the Azure Non Interactive Sign-ins test, an administrator can achieve all of the above! 

This test monitors Azure non-interactive sign-in logs for failed sign-ins and reports their count and details. With the help of these details, administrators can effectively troubleshoot the failures. The test also promptly captures and reports 'risky sign-ins', so that dubious sign-in attempts can be investigated and prevented. Additionally, the test reveals whether any sign-ins were made using unsecure legacy authentication protocols. Since such authentication protocols are a security threat, administrators may want to disable them. The test also helps administrators closely scrutinize the sign-ins to isolate abnormal patterns, such as the following:

  • Are there an unusually high number of sign-ins coming from specific IP addresses/locations/users/client applications?

  • Are any applications/resources making a suspicious number of sign-in attempts?

  • Are sign-in attempts from specific users/IP addresses/locations/client applications failing often?

  • Are specific applications/resources seeing more sign-in failures than others?

This way, the test sheds light on sign-in attempts that are 'suspect', so their authenticity can be verified, and any potential security risks pre-empted.

Target of the Test: A Microsoft Azure Active Directory

Agent deploying the test: A remote agent

Output of the test: One set of results for the Azure Active Directory tenant being monitored

Configurable parameters for the test
Parameters Description

Test Period

How often should the test be executed.

Host

The host for which the test is to be configured.

Tenant ID

Specify the Directory ID of the Azure AD tenant to which the target subscription belongs. To know how to determine the Directory ID, refer to Configuring the eG Agent to Monitor Microsoft Azure Active Directory Using Microsoft Graph API

Client ID, Client Password, and Confirm Password

To connect to Azure AD, the eG agent requires an Access token in the form of an Application ID and the client secret value. If a Microsoft Azure Subscription component is already monitored in your environment, then you would have already created an Application for monitoring purposes. You can provide the Application ID and Client Secret value of that application here. However, if no such application pre-exists, you will have to create one for monitoring Azure AD. To know how to create such an application and determine its Application ID and Client Secret, refer to Configuring the eG Agent to Monitor Microsoft Azure Active Directory Using Microsoft Graph API. Specify the Application ID of the Application in the Client ID text box and the client secret value in the Client Password text box. Confirm the Client Password by retyping it in the Confirm Password text box.

Proxy Host and Proxy Port

In some environments, all communication with the Azure cloud be routed through a proxy server. In such environments, you should make sure that the eG agent connects to the cloud via the proxy server and collects metrics. To enable metrics collection via a proxy, specify the IP address of the proxy server and the port at which the server listens against the Proxy Host and Proxy Port parameters. By default, these parameters are set to none, indicating that the eG agent is not configured to communicate via a proxy, by default.

Proxy Username, Proxy Password and Confirm Password

If the proxy server requires authentication, then, specify a valid proxy user name and password in the Proxy Username and Proxy Password parameters, respectively. Then, confirm the password by retyping it in the Confirm Password text box.

Non Interactive Sign-in Workspace Name

Typically, the Non Interactive Sign-in logs are sent to a Log Analytics Workspace. By default, the Log Analytics Workspace Name parameter is set to All. This indicates that the test reads sign-in data from all Log Analytics Workspaces configured for the target tenant, by default. However, if you want the test to use only specific Log Analytics Workspaces for metrics collection, then provide the names of these workspaces here as a comma-separated list. To determine the names of the workspaces, do the following:

  1. Login to the Microsoft Azure Portal and select the Sign-in Logs option (see Figure 1).

    Figure 1 : Selecting the Sign-in Logs option

  1. Figure 2 will then appear listing the log entries. Next, click on the Export Data Settings button indicated by Figure 2.

    Figure 2 : Clicking on the Export Data Settings button

  2. The diagnostic settings that pre-exist for the sign-in logs will then appear. If any of the existing diagnostic settings have already been configured with Log Analytics Workspaces, then the Log Analytics workspace column highlighted in Figure 3 will display these workspace names. You can configure the LOG ANALYTICS WORKSPACE NAME parameter of this test with any of these workspace names. If required, you can even configure this parameter with two/more workspaces listed in Figure 3, as a comma-separated list.

    Figure 3 : List of Log analytics workspaces

However, If the Log Analytics workspace column in Figure 3 is blank for all the existing diagnostic settings, it is a clear indication that the sign-in logs are not yet configured to be sent to any Log Analytics Workspace. In this case therefore, you should create a new diagnostic setting, where a Log Analytics Workspace is configured as the destination for the Sign-in logs. To achieve this, follow the procedure detailed in Configuring the Sign-in Logs to be Sent to a Log Analytics Workspace.

Successful Signin DD

By default, this flag is set to No. This means that, by default, this test will not report detailed diagnostics for the 'Success' metrics (eg., Successful sign-ins, Success IP addresses etc.). This is because, in a typical Azure cloud organization, there may be numerous successful sign-in attempts. A well-tuned, well-sized eG database is required for storing these detailed metrics. Without it,over a period of time, the detailed statistics of 'Successful' sign-in attempts may end up choking the eG database. To avoid it, this parameter is set to No by default. Set this flag to Yes only if your eG database has sufficient space to store detailed diagnostics for the 'Success' metrics.

No of Attempts

By default, this flag is set to 5. This means that, by default, the test will count a sign-in attempt from a specific user / IP address / location or for a specific application / service principal / resource as a success/failure, only if 5 or more consecutive sign-in attempts from/for that entity succeed/fail (as the case may be). For instance, if 5 or more sign-in attempts from a specific IP address fail, then the test will count that as one failure - i.e., the Failure IP addresses measure will report the value 1. Similarly, if 5 or more sign-in attempts from a specific IP address succeed, then the test will count that as a single sign-in success - i.e., the Success IP addresses measure will report the value 1. Needless to say, by default, the values of these measures will also be incremented only with every 5 or more consecutive sign-in successes/failures. This is done so that administrators are alerted only to those sign-in attempts that are 'suspect' - for example, repeated sign-in failures from the same IP address. You can change the value of this measure based on what is normal sign-in activity and what is not in your environment.

This parameter applies to the following measures reported by the test:

  • Successful IP addresses
  • Successful locations
  • Successful applications
  • Successful resources
  • Successful users
  • Failure IP addresses
  • Failure locations
  • Failure applications
  • Failure service principals
  • Failure resources
  • Failure users
  • Failure client apps

Detailed Diagnosis

To make diagnosis more efficient and accurate, the eG Enterprise embeds an optional detailed diagnostic capability. With this capability, the eG agents can be configured to run detailed, more elaborate tests as and when specific problems are detected. To enable the detailed diagnosis capability of this test for a particular server, choose the On option. To disable the capability, click on the Off option.

The option to selectively enable/disable the detailed diagnosis capability will be available only if the following conditions are fulfilled:

  • The eG manager license should allow the detailed diagnosis capability
  • Both the normal and abnormal frequencies configured for the detailed diagnosis measures should not be 0.
Measures made by the test:
Measurement Description Measurement Unit Interpretation

Total non- interactive sign-ins

Indicates the total number of non-interactive sign-ins attempted.

Number

 

Successful sign-ins

Indicates the number of sign-in attempts that were successful.

Number

Use the detailed diagnosis of this measure to know which sign-in attempts succeeded.

Note that this measure will report detailed diagnostics only if the Successful Signin DD parameter of this test is set to Yes.

Failed sign-ins

Indicates the number of sign-in attempts that failed.

Number

Ideally, the value of this measure should be 0.

Use the detailed diagnosis of this measure to know which sign-in attempts failed.

Risky sign-ins

Indicates the number of risky non-interactive sign-in attempts.

Number

In Azure AD Identity Protection, risk detections include any identified suspicious actions related to user accounts in Azure AD.

Ideally, the value of this measure should be 0. If a non-zero value is reported, then use the detailed diagnosis of this measure to know the risky sign-in attempts.

Successful IP addresses

Indicates the number of IP addresses from which successful sign-in attempts were made.

Number

Use the detailed diagnosis of this measure to know from which IP addresses successful sign-in attempts were made.

Note that this measure will report detailed diagnostics only if the Successful Signin DD parameter of this test is set to Yes.

Successful locations

Indicates the number of locations from which successful sign-in attempts were made.

Number

Use the detailed diagnosis of this measure to know from which locations successful sign-in attempts were made.

Note that this measure will report detailed diagnostics only if the Successful Signin DD parameter of this test is set to Yes.

Successful applications

Indicates the number of applications that successfully used managed identities to sign into Azure.

Number

Use the detailed diagnosis of this measure to know which applications signed into Azure successfully.

Note that this measure will report detailed diagnostics only if the Successful Signin DD parameter of this test is set to Yes.

Successful resources

Indicates the number of services that were used in successful sign-ins.

Number

Use the detailed diagnosis of this measure to know which services were used in successful sign-ins.

Note that this measure will report detailed diagnostics only if the Successful Signin DD parameter of this test is set to Yes.

Successful users

Indicates the number of users who successfully signed in.

Number

Use the detailed diagnosis of this measure to know which users' sign-in attempts succeeded.

Note that this measure will report detailed diagnostics only if the Successful Signin DD parameter of this test is set to Yes.

Success client apps

Indicates the number of client apps that successfully signed into Azure using the non-interactive sign-in method.

Number

Use the detailed diagnosis of this measure to know which client apps signed in successfully.

Note that this measure will report detailed diagnostics only if the Successful Signin DD parameter of this test is set to Yes.

Failed sign-ins percentage

Indicates the percentage of sign-in attempts that failed.

Number

Ideally, the value of this measure should be low.

Use the detailed diagnosis of this measure to know which sign-in attempts failed.

Failure IP addresses

Indicates the number of IP addresses from which sign-in attempts failed.

Number

Use the detailed diagnosis of this measure to know from which IP addresses the maximum number of failed sign-in attempts were made. You may want to investigate these attempts to figure out if they were geniuine attempts or malicious attacks.

Failure locations

Indicates the number of locations from which sign-in attempts failed.

Number

Use the detailed diagnosis of this measure to know from which locations the maximum number of failed sign-in attempts were made. You may want to investigate these attempts to figure out if they were geniuine attempts or malicious attacks.

Failure applications

Indicates the number of applications that were unable to sign-into Azure.

Number

Use the detailed diagnosis of this measure to know which applications failed to sign into Azure the maximum number of times. You may want to investigate these attempts to figure out if they were geniuine attempts or malicious attacks.

Failure resources

Indicates the number of services that were used in failed sign-ins.

Number

Use the detailed diagnosis of this measure to know which services were used in the maximum number of failed sign-ins. You may want to investigate these attempts to figure out if they were geniuine attempts or malicious attacks.

Failure users

Indicates the number of users whose sign-in attempts failed.

Number

Use the detailed diagnosis of this measure to know which users experienced the maximum number of failed sign-ins. You may want to investigate these attempts to figure out if they were geniuine attempts or malicious attacks.

Failure client apps

Indicates the number of client apps that could not sign-in using the non-interactive sign-in method.

Number

Use the detailed diagnosis of this measure to know which client apps failed to sign in. You may want to investigate these attempts to figure out if they were geniuine attempts or malicious attacks.

Single-factor authentications

Indicates the number of sign-ins made using the single-factor authentication method.

Number

Use the detailed diagnosis of this measure to view the sign-ins made using single-factor authentication.

Multi-factor authentications

Indicates the number of sign-ins made using the multi-factor authentication method.

Number

Use the detailed diagnosis of this measure to view the sign-ins made using multi-factor authentication.

Modern client authentications

Indicates the number of sign-ins that used modern client authentication techniques.

Number

Modern authentication is a method of identity management that offers more secure user authentication and authorization. Modern authentication is an umbrella term for a combination of authentication and authorization methods between a client (for example, your laptop or your phone) and a server, as well as some security measures that rely on access policies that you may already be familiar with.

Use the detailed diagnosis of this measure to view the sign-ins made using modern client authentication.

Legacy client authentications

Indicates the number of sign-ins that used legacy client authentication techniques.

Number

Legacy authentication refers to basic authentication, which was once a widely used industry-standard method for passing user name and password information through a client to an identity provider.

Use the detailed diagnosis of this measure to view the sign-ins made using legacy client authentication.

Not applied conditional accesses

Indicates the number of sign-ins during which no conditional access policy applied to the user and application.

Number

Conditional Access policies at their simplest are if-then statements, if a user wants to access a resource, then they must complete an action. Example: A payroll manager wants to access the payroll application and is required to do multi-factor authentication to access it.

Successful conditional accesses

Indicates the number of sign-ins during which one or more conditional access policies applied to the user and application.

Number

 

Failed conditional accesses

Indicates the number of sign-ins that satisfied the user and application condition of at least one Conditional Access policy and grant controls are either not satisfied or set to block access.

Number

Use the detailed diagnosis of this measure to know which conditional access policies failed.

Use the detailed diagnosis of the Successful applications measure to know which applications signed into Azure successfully.

Figure 4 : The detailed diagnosis of the Successful applications measure of the Azure Non Interactive Sign-ins Test

Use the detailed diagnosis of the Successful IP addresses measure to know from which IP addresses successful sign-in attempts were made.

Figure 5 : The detailed diagnosis of the Successful IP addresses measure reported by the Azure Non Interactive Sign-ins Test

Use the detailed diagnosis of the Successful locations measure to know from which locations successful sign-in attempts were made.

Figure 6 : The detailed diagnosis of the Successful locations measure reported by the Azure Non Interactive Sign-ins Test

Use the detailed diagnosis of the Successful resources measure to know which services were used in successful sign-ins.

Figure 7 : The detailed diagnosis of the Successful resources measure reported by the Azure Non Interactive Sign-ins Test

Use the detailed diagnosis of the Successful users measure to know which users' sign-in attempts succeeded.

Figure 8 : The detailed diagnosis of the Successful users measure reported by the Azure Non Interactive Sign-ins Test

Use the detailed diagnosis of the Failed sign-ins percentage and Failed sign-ins measures to know which sign-in attempts failed.

Figure 9 : The detailed diagnosis of the Failed sign-ins percentage measure reported by the Azure Non-Interactive Sign-ins Test

Use the detailed diagnosis of the Failure applications measure to know which applications failed to sign into Azure the maximum number of times.

Figure 10 : The detailed diagnosis of the Failure applications measure reported by the Azure Non Interactive Sign-ins Test

Use the detailed diagnosis of the Failure client apps measure to know which client apps failed to sign in.

Figure 11 : The detailed diagnosis of the Failure client apps measure reported by the Azure Non Interactive Sign-ins Test

Use the detailed diagnosis of the Failure IP addresses measure to know from which IP addresses the maximum number of failed sign-in attempts were made.

Figure 12 : The detailed diagnosis of the Failure IP adddresses measure of the Azure Non Interactive Sign-ins test

Use the detailed diagnosis of the Failure locations measure to know from which locations the maximum number of failed sign-in attempts were made.

Figure 13 : The detailed diagnosis of the Failure locations measure of the Azure Non Interactive Sign-ins test

Use the detailed diagnosis of the Failure resources measure to know which services were used in the maximum number of failed sign-ins.

Figure 14 : The detailed diagnosis of the Failure resources measure of the Azure Non Interactive Sign-ins Test

Use the detailed diagnosis of the Failure users measure to know which users experienced the maximum number of failed sign-ins.

Figure 15 : The detailed diagnosis of the Failure users measure of the Azure Non Interactive Sign-ins test