Feedback loop and tuning best practices

The most useful labels are the ones tied directly to the business decision you want to optimize:

• If the priority is detecting account takeover earlier, focus on login actions
• If the priority is transaction fraud, focus on transaction actions
• If the priority is new account opening protection, focus on registration actions

If your team resources are limited, begin with the confirmed fraud since labeling such events typically provides the highest value for tuning, especially during the early stages of deployment. As the program matures, expand to include high-confidence suspected fraud, confirmed legitimate outcomes, and broader coverage of deny and challenge decisions.

An effective feedback loop should improve both fraud prevention and customer experience. That means tracking not only what fraud was caught, but also which legitimate users were challenged or denied, whether trusted users are being recognized effectively, and whether investigation teams are reviewing the right cases. Confirmed legitimate labels are especially useful for improving trust recognition and reducing false positives.

A fraud outcome can be valid even if no monetary loss occurred—for example, fraud may have been prevented before funds were lost, or abuse may have been confirmed through investigation before a transaction completed. This is especially important in prevention-focused flows such as login, new account opening, verification, and early journey enforcement. Base fraud confirmation on confidence in the conclusion, not only on whether a realized loss occurred.

When Fraud Prevention denies an action early on, the fraud may never fully unfold, making direct validation more difficult. In these cases, organizations may need to rely on shadow mode or non-blocking evaluation during early rollout, retrospective analyst review, related downstream evidence, or portfolio-level performance analysis.

Fraud Prevention supports multiple label types, each serving a different purpose. Use them intentionally:

• Confirmed fraud for high-confidence fraud outcomes
• Suspected fraud when fraud is likely but not yet fully confirmed
• Confirmed legit when the case has been investigated and cleared
• Undetermined when the outcome remains inconclusive

Consistent usage improves the quality and interpretability of the feedback loop. Define a clear internal labeling policy so teams apply labels consistently over time.

Fraud Prevention supports labeling several types of subjects, including individual actions, correlated journeys, users, verification sessions, campaigns, and fraud rings. In many cases, the best starting points are:

• Action-level labeling when feedback applies to a specific event
• Correlation-level labeling when fraud spans multiple linked actions in the same journey or session

Use the narrowest subject level that accurately represents the fraud outcome while preserving the necessary context.

Fraud often spans more than one event. A single fraud case may involve multiple connected actions, devices, or sessions. Where possible, connect feedback to the broader journey using relevant identifiers such as correlation ID, claimed user ID, etc. When fraud affects multiple related actions, use correlated labeling instead of only labeling the final event.

The source of the label helps explain how the outcome was determined. Common sources include manual review, customer complaints, chargebacks, and other vendors or fraud systems. Tracking the source improves transparency and supports future analysis.

Each factor can be set to one of five levels:

• Ignore: Removes the factor from score calculation entirely.
• Low: Reduces the factor's contribution to the score.
• Default: Uses the system's baseline weighting.
• High: Amplifies the factor's contribution, making it more likely to trigger a Challenge or Deny.
• Deny: Automatically blocks any action where this factor is detected, regardless of other signals.

Sensitivity adjustments are most effective when:

• A specific risk factor generates too many false positives (e.g., VPN usage is common among your legitimate users—consider lowering the is VPN weight).
• A new attack pattern emerges that requires immediate response (e.g., a surge in bot activity—set the Bot factor to Deny to block all bot-flagged actions until a proper strategy is defined).
• Certain factors are not relevant to your environment or business (e.g., corporate users are using virtual machines by policy—set the Virtual machine factor to Ignore).

• Use the Preview mode first: Save sensitivity configurations as a preview policy and evaluate their impact on the Recommendations dashboard before deploying to production. This avoids unintended disruption.
• Adjust incrementally: Change one or two factors at a time so you can isolate the effect of each adjustment on your recommendation distribution.
• Review after labeling milestones: When a significant batch of labels has been submitted, revisit sensitivity settings to see whether the system's baseline has shifted enough to warrant recalibration.
• Combine with rules for layered control: Sensitivity adjusts the score broadly; rules override the recommendation for specific conditions. Use both together—for example, increase sensitivity for public Wi-Fi signals generally, but also create a rule that denies transactions from public Wi-Fi combined with a new device.

Rules are best suited for two scenarios:

• Mitigating immediate threats: When a new attack vector is identified, a rule can be deployed immediately to block it (e.g., deny all actions from IP ranges confirmed as malicious). Once the threat is contained and enough labels have been submitted, the ML models adapt on their own and the rule can be retired.
• Permanent business policies: For strict, well-defined conditions that should always produce the same outcome regardless of risk score—such as allowlisting corporate devices or always challenging transactions above a certain threshold.

• Start in the Preview mode: Every new rule should be tested in Preview mode first. Preview mode simulates the rule's effect against live traffic without impacting production decisions, so you can evaluate its reach before going live.
• Keep rules focused and minimal: Each rule should address a single, well-defined scenario. Overly broad rules can override the system's ML intelligence in ways that increase false positives or reduce fraud capture.
• Manage rule priority carefully: Rules are evaluated top to bottom, and only the first match applies. Place the most critical rules (e.g., deny rules for confirmed threats) higher in the priority order, and more permissive rules (e.g., trust rules for known-good devices) lower.
• Audit rules regularly: As fraud patterns evolve and your label history grows, rules created during early deployment may become redundant or counterproductive. Review active rules during your periodic tuning reviews and disable or remove rules that the ML models now handle effectively on their own.
• Coordinate rules with sensitivity settings: If you increase the sensitivity of a factor to High, you may not also need a rule that denies actions based on the same factor. Avoid duplicating logic across both controls.

Initial tuning is the early phase of aligning Fraud Prevention to a new environment, channel, or use case. Typical goals include:

• Establishing a performance baseline
• Understanding the organization's fraud patterns
• Reviewing early recommendation quality
• Aligning on operational processes
• Calibrating detection sensitivity and creating initial rules

This phase may take place during a proof of concept, proof of value, or early production rollout. During initial tuning, lean on rules and sensitivity adjustments for quick wins while you build up the label volume that improves ML accuracy over time.

Ongoing optimization is the continuous improvement phase once a baseline is in place. Typical goals include:

• Adapting to new fraud patterns
• Improving precision and fraud capture over time
• Monitoring performance trends
• Revisiting strategy when business conditions change
• Retiring rules that are no longer needed as the ML models mature

A good starting model includes:

• Defining the primary fraud use case
• Labeling confirmed fraud first
• Using action-level or correlation-level feedback
• Using manual review as the main source
• Labeling a manageable sample if full coverage is not feasible
• Adjusting detection sensitivity for the most impactful risk factors
• Creating a small set of rules for known threat patterns, tested in Preview mode first
• Establishing a recurring review cadence
• Aligning with Mosaic on early tuning decisions

A more advanced model includes:

• Automating label submission via the Send label API
• Correlating feedback across journeys and sessions
• Including confirmed legitimate outcomes
• Classifying by fraud scenario and source
• Maintaining a curated rule set that complements ML-driven recommendations, with regular audits to retire redundant rules
• Coordinating sensitivity settings with rules and label trends
• Integrating with existing fraud operations
• Revisiting tuning based on KPI movement