Conditional Access Report-Only Mode: Test Policies Before You Enforce Them

Report-only mode is a Conditional Access policy state that evaluates the policy at every matching sign-in and logs what would have happened, without blocking anyone or prompting for multifactor authentication. You create the policy, set Enable policy to Report-only, wait for real sign-ins, review the results in the sign-in logs or the insights workbook, and only then switch it to On.

This guide is for Microsoft Entra administrators who are about to roll out a new Conditional Access policy, such as requiring MFA for all users, blocking legacy authentication or requiring a compliant device. By the end you will know how to create a report-only policy, read each result type, query results at scale, handle the known limits and promote the policy safely. It builds on the ideas in troubleshooting Conditional Access with the What If tool and sign-in logs. Facts here come from Microsoft Learn pages checked on 31 October 2026. Portal layouts change, so use the names below as search terms if a menu has moved.

Who this is for and what you will have at the end

You hold at least the Conditional Access Administrator role and have a policy idea. After following this guide you will have a policy running in report-only mode, a repeatable way to check its impact, a list of the sign-ins that would have failed, and a promotion checklist.

Prerequisites

• Microsoft Entra ID P1 licenses, which Microsoft lists as required to use Conditional Access.
• The Conditional Access Administrator role to create the policy. To read sign-in logs and the workbook, Microsoft lists at least Security Reader, plus Log Analytics workspace permissions for the workbook.
• For the workbook and KQL queries, a Log Analytics workspace receiving your Entra sign-in logs. If not set up yet, follow Microsoft's guide to integrate Entra logs with Azure Monitor logs. Only sign-ins after the integration appear.
• Break-glass accounts excluded from the policy. See Conditional Access break-glass accounts.

How report-only fits into policy evaluation

Microsoft describes policy enforcement in two phases. In phase 1 the system collects session details such as network location and device identity; this happens for enabled policies and report-only policies. In phase 2 it enforces requirements, and phase 2 occurs only for enabled policies. That is why a report-only policy can tell you what its conditions matched without changing the outcome.

Two cautions from Microsoft's documentation:

• You cannot evaluate policies in report-only mode for items in the User Actions scope.
• Report-only evaluates but does not enforce grant or session controls. Users are not prompted for MFA or blocked by report-only policies.

Step 1: Create the policy in report-only mode

1. Sign in to the Microsoft Entra admin center as at least a Conditional Access Administrator.
2. Browse to Entra ID > Conditional Access > Policies.
3. Select an existing policy or create a new policy.
4. Under Enable policy, set the toggle to Report-only.
5. Select Save.

Give the policy a name that says it is a test, for example "CA-TEST Require MFA all users (report-only)". Define users, target resources and grant controls as you would for the enforced version.

Do not convert a live policy to report-only to test it

Microsoft's tip is that editing the Enable policy state of an existing policy from On to Report-only disables existing policy enforcement. If you want to test a change to an enforced policy, follow Microsoft's guidance instead: create a copy in report-only mode with your proposed changes, compare its results with the original enforced policy, and when satisfied update the enforced policy and remove the copy.

Exclude mobile platforms from compliance checks

Microsoft warns that report-only policies requiring a compliant device can prompt users on macOS, iOS and Android to select a device certificate during policy evaluation, even though compliance is not enforced. The prompts can repeat until the device is compliant. To avoid this, exclude the Mac, iOS and Android device platforms from report-only policies that check compliance. This means your test will not show impact for those platforms, so plan a separate check for them.

Step 2: Wait for representative traffic

Report-only data only reflects sign-ins that actually happen. A policy that targets all users should run for long enough to include typical patterns: weekday and weekend, month-end, travelling users, service accounts and shared mailboxes. Microsoft's workbook supports time ranges from 4 hours to 90 days, and the policy impact view covers the past 24 hours, 7 days or 1 month. Decide in advance how long you will observe, for example two weeks.

Step 3: Read the results

Microsoft lists three places to review results: the sign-in logs, the insights and reporting workbook, and the policy impact view.

Understand the result values

Treat Failure as "this user or sign-in would be blocked or denied if you enforced the policy," and investigate each cause. Treat User action required as unknown, because the sign-in might pass once the user completes the prompt.

Sign-in logs

For a single sign-in, open the event and use the Conditional Access and Report-only tabs. Each event shows which policies were enabled, in report-only mode, applied or not applied. In the Entra admin center the sign-in logs are under Entra ID > Monitoring & health > Sign-in logs. When you download sign-in logs, Microsoft notes you should choose JSON format to include Conditional Access report-only result data.

Policy impact view

Microsoft describes a policy impact view where admins with at least the Security Reader role see a snapshot of the potential or existing impact of policies on interactive sign-ins over the past 24 hours, 7 days or 1 month, with a sampling of sign-in events to link to. It is labelled as a preview, so its location and name may change.

Insights and reporting workbook

Open Entra ID > Conditional Access > Insights and reporting. Set the parameters at the top:

• Conditional Access policy: policies are split into Enabled and Report-only groups. By default all Enabled policies are selected, so also select your report-only policy.
• Time range: 4 hours up to 90 days, but only sign-ins after you integrated logs with Azure Monitor appear.
• User and App: leave empty for all, or type a name.
• Data view: users or sign-ins. In the user view a person can appear under both Success and Failure.

The Impact summary shows Total, Success, Failure, User action required and Not applied. Select a tile to filter, and use the breakdown by device state, device platform, client app, location, application and sign-in risk to see which condition drives failures. Microsoft recommends comparing report-only and enforced results side by side in this workbook to see where a new policy would change access decisions.

Because multiple policies apply at once, remember that the combined result matters. Microsoft states that when more than one policy applies, all must be satisfied.

Step 4: Query results with KQL

If your sign-in logs are in Log Analytics, you can count outcomes for one policy. Microsoft documents that the policy results are stored in the ConditionalAccessPolicies column of SignInLogs, with result values including success, failure, notApplied, reportOnlySuccess, reportOnlyFailure and reportOnlyNotApplied.

SigninLogs | where TimeGenerated > ago(14d) | extend caps = parse_json(tostring(ConditionalAccessPolicies)) | mv-expand cap = caps | where tostring(cap.displayName) == "CA-TEST Require MFA all users (report-only)" | summarize signIns = count() by result = tostring(cap.result) | order by signIns desc 

To list who would be affected:

SigninLogs | where TimeGenerated > ago(14d) | extend caps = parse_json(tostring(ConditionalAccessPolicies)) | mv-expand cap = caps | where tostring(cap.displayName) == "CA-TEST Require MFA all users (report-only)" | where tostring(cap.result) == "reportOnlyFailure" | summarize failures = count() by UserPrincipalName, AppDisplayName, ClientAppUsed | order by failures desc 

Check the table and column names in your workspace. Microsoft's workbook page uses SignInLogs in one query and SigninLogs in the troubleshooting steps, and table names in KQL are normally case-sensitive, so use the name your workspace shows. If the workbook is slow, Microsoft suggests shortening the time range or creating an Azure Monitor transformation, with cost considerations to review first. Excluding certain results from a transformation means you will not see those results in the workbook afterward.

You can also confirm which policies are in report-only through Microsoft Graph: the policy state property accepts enabled, disabled and enabledForReportingButNotEnforced. In sign-in data, the applied policy result can be reportOnlySuccess, reportOnlyFailure, reportOnlyNotApplied or reportOnlyInterrupted.

Step 5: Fix what the data shows

For each group of failures decide whether the policy or the user is wrong.

• Legitimate users failing a control: adjust the policy, for example add a trusted named location, exclude a service account or add an alternative grant control.
• Unexpected apps or client types: confirm whether they should be in scope. Microsoft notes that by default all newly created policies apply to all client app types even if the client apps condition is not configured.
• Device platform surprises: Microsoft notes device platform information comes from unverified sources such as user agent strings, which can be changed.
• Users who would need to act: pilot with a small group in enforced mode, or communicate before enforcement.

After each change, let a new window of traffic accumulate and compare again. Keep a note of date, change and result.

Step 6: Promote to On

Microsoft's instruction is to move the Enable policy toggle from Report-only to On after confirming your settings using policy impact or report-only mode. Before you do:

1. Confirm break-glass accounts are excluded.
2. Confirm there are no unexplained Failure results, and that User action required cases are acceptable.
3. Tell affected users what will change and when.
4. Enable during a period when you can watch the sign-in logs and respond.
5. Keep the sign-in log filter for the policy ready to catch fresh failures.

If a policy causes trouble after enabling, see AADSTS53003 blocked by Conditional Access for tracing the blocking policy.

Microsoft also notes that Conditional Access does not support classic A/B testing that splits live traffic between two enforced variants. Where the Conditional Access Optimization Agent is available (it requires Microsoft Security Copilot with provisioned security compute units and Entra ID P1), it creates suggested policies in report-only mode and offers phased rollout, but it is not enabled in every tenant.

Troubleshooting

The workbook policy list is empty

Microsoft says the list is generated from policies evaluated for the most recent sign-in, so wait a few minutes after configuring Log Analytics or if there is little recent activity.

Queries fail with a permissions error

You need both Entra permissions and Log Analytics workspace permissions. Microsoft's test: as at least Security Reader, go to Entra ID > Monitoring & health > Log Analytics, run SigninLogs, and if no results return the workspace may not be configured correctly.

Workbook queries fail or time out

Make sure only one workspace is associated with the workbook (Edit, then the Settings gear, remove extra workspaces). For large tenants, shorten the time range to 4 hours.

No report-only results appear

Check the policy is saved with Report-only state, that the users and apps in scope actually signed in, and that the policy does not rely on the User Actions scope, which report-only cannot evaluate.

Users get certificate prompts

That is the compliant-device report-only behavior on macOS, iOS and Android. Exclude those platforms from the test policy.

Checklist

• Create the policy with Enable policy set to Report-only.
• Exclude break-glass accounts, and mobile platforms for compliance tests.
• Observe real traffic for a defined period.
• Review the Report-only tab, workbook or policy impact view.
• Query failures with KQL and fix the policy or the user cause.
• Test changes to live policies in a report-only copy, not by switching the live policy.
• Switch to On, and watch the sign-in logs afterwards.

References

• Conditional Access policy insights: monitoring and evaluation (report-only mode)
• Conditional Access insights and reporting workbook
• Build Conditional Access policies in Microsoft Entra
• conditionalAccessPolicy resource type (Microsoft Graph)
• appliedConditionalAccessPolicy resource type (Microsoft Graph)

This guide was researched and drafted with AI assistance and checked against the vendor documentation listed in its References. Read the maintained version and related guides on EpiFive.