Using the Duplicate Assistant

Overview

The Duplicate Assistant feature replaces the legacy Deduplication tool and removes many of the challenges associated with it by utilizing:

• Semantic Similarity Detection + large language model (LLM) review to recognize near duplicates.

• Responsive AI-recommended content merges to reduce the manual effort required to merge content.

• The ability to mark something as Not a Duplicate, allowing you to dictate when records are not duplicates of each other and should never appear in the list together again.

Once duplicates are identified, you can manually update the primary Q&A pair, use AI Edit for full changes, or apply suggestions on a field by field basis.

How it works

There are two parts to the duplicate detection process:

• First, semantic similarity is used to identify Q&A pairs that appear to be related based on meaning, not just exact wording. This helps surface records that may be duplicates even when the questions are phrased differently (dog and canine, for example).

  • The assistant currently evaluates content with a semantic similarity score between 0.85–1.0 (85%–100% similar) as potential duplicate candidates.

  • A higher similarity score means the records are more likely to discuss the same topic or requirement, but similarity alone doesn’t determine whether content should be merged.

• Then, any duplicate candidates are passed to an LLM for further analysis. Responsive AI evaluates whether the records should actually be merged or kept separate using rules designed to protect content quality and prevent incorrect merges. Responsive AI checks whether:

  • one answer can fully and correctly answer all questions.

  • the questions have the same requirement or intent.

  • one question is broader or narrower than the other.

  • the questions introduce different sub-questions or control requirements.

  • merging would create a broader, blurrier, or multi-topic answer.

  • the answers conflict or require different conclusions.

Note that the assistant is intentionally conservative. It recommends a merge only when the content can be combined without losing meaning or creating confusion. When records are related, but not truly interchangeable, the assistant recommends keeping them separate. For example, two questions about the same security policy may still be kept separate if one asks about the policy itself and the other asks about scan cadence, monitoring, or enforcement. 

Permissions

You must be an Admin or Super Admin to set up the Duplicate Assistant.

Setting up the Duplicate Assistant

Once enabled:

1. Go to the Q&A pair in the Content Library and click Tools, then select Duplicate Assistant (Q&A Pairs) Beta.
   

2. Click Get Started. You’ll be asked a series of questions to determine what content should be shown in the Duplicate queue.
   

   • Exact Duplicates page: Responsive can automatically merge exact duplicates for you. Exact duplicates are records whose questions, answers and metadata are identical.
     

     • Click Yes if you want Responsive to merge all duplicates into the one Q&A pair that was most recently edited. This option also combines the use counts of all records. Click Continue.

     • Click No if you want exact duplicates to appear in your Duplicate list for manual review.  Click Continue.

   • Intentional Duplicates page: You can exclude intentional duplicates, which are records that are typically classified differently and intentionally duplicated in the library.
     

     • Click Yes if you want to exclude intentional duplicates, then select how to organize them (by Business Unit [BU], tag, collections and/or custom fields), then click Finish.

     • Click No if you want all intentional duplicates to be reviewed, then click Finish.

3. The Duplicate Results page opens with all the duplicate clusters that were identified and their primary information (the Q&A pair itself, the number of duplicates and their similarity score, and metadata such as tags and collections).
   Note: It can take a few minutes to load your duplicates. Meanwhile, you can continue with other work. Your results will automatically appear here once the process is complete.
   

You can go to the Settings page within the feature to change instructions on whether to archive or delete merged content, or include/exclude exact and intentional duplicates.

Using the Duplicate Assistant

1. Go to the Q&A pair in the Content Library and click Tools, then select Duplicate Assistant (Q&A Pairs) Beta.
   

2. The Duplicate Results page opens with all the duplicate clusters that were identified and their primary information (the Q&A pair itself, the number of duplicates and their similarity score, and metadata such as tags and collections). Select the filter and sorting options you want to use:
   Note: It can take a few minutes to load your duplicates. Meanwhile, you can continue with other work. Your results will automatically appear here once the process is complete.

   • Sort by Near duplicates and Exact duplicates.
     Note: These options are available only if your admin chose to leave exact duplicates in your queue during the setup process.
     

   • Sort by Most similar, Least similar, Highest duplicate count and Lowest duplicate count.
     

   • Filter the results using the normal options.
     

   • Click the three-dot menu and select either Not a duplicate, Archive or Delete to remove or delete a cluster.

   • Click the three-dot menu and select Hide for now to hide a cluster.

3. Click on a cluster to open it in the Duplicate Comparison page. The primary content is displayed in the left-hand pane and its duplicates are displayed on the right-hand pane.

4. Review the duplicate Q&A pairs and select which questions, responses, and metadata you want to merge into the primary record. Use the following tools to compare them:

   • If there is more than one duplicate, use Jump tags to easily move between them and view each duplicate Q&A pair in the result.
     

   • Turn the Show Difference toggle on to highlight only the fields and metadata that are different from each other.
     

   • Use the Compare by Field drop-down list to compare specific fields (such as Alternate Questions or Metadata) at a time.
     

5. Once you've review each duplicate record to determine if it’s actually a duplicate of the primary Q&A pair, do one of the following:

   • If the record isn’t an actual duplicate, click Not a duplicate.
     

   • If you want to delete or archive a duplicate result, click the three-dot menu and select Delete or Archive.

   • If you think the duplicate would be a better fit as the primary, click the three-dot menu and select Set as Primary.

   • If you want to merge one or more results, go to Step 6.

6. Do one of the following:

   • If the duplicate is an exact match, go to Step 7.

   • If the duplicate is a near duplicate, you can do one of the following to make changes if needed:
     Note: When changes are made, the original primary Q&A pair moves to the right side of the screen and the updated version displays on the left for easy comparison. You can click Revert to Original to undo the changes and revert back to the original primary.

     • Edit the results manually, then click the Checkmark to save your changes, then go to Step 7.

     • Use Responsive AI to assist you with the edits:

       1. Click AI Edit and do one of the following:
          

          • Click Write my own prompt and tell Responsive AI how you’d like to merge your results. For example, Keep the response to 500 characters or less or Only merge the question from duplicate number two with the primary.

          • Click Suggest merge for me to have Responsive AI suggest how to merge your content.

       2. Review the results, then choose to Apply, Discard, or Write Prompt. The results are generated and a summary of the changes is provided based on the prompt.
          Note: You can revert back to the original primary at any time.
          

       3. Go to Step 7.

7. When you’re ready to merge, click Merge Duplicates.

   • Merged content is archived by default.

   • Go to the Activity log within the feature to view historical merge details. Activities are logged by cluster.
     Note: By default, merge data is retained for 90 days. It will be configurable to retain anywhere between 30-90 days in upcoming releases.

FAQs

Q1: Will I still be able to access the old duplicate tool?

A: For Beta customers, the Duplicate Assistant replaces the legacy tool. However, we can revert it back for you at any time.

Q2: What types of instructions can I give to Responsive AI when I write my own merge prompt? 

A: Detailed instructions that explain how you want to merge the results, such as Only merge the responses from Duplicates 2 & 4 with the primary response or Keep the response to 500 characters or less.
Note: As a best practice, we recommend being as specific as possible about the field name so that the LLM can precisely identify the exact field name to make the changes.

Q3: Can I change the primary Q&A pair?

A: Yes. Click the three-dot menu of the duplicate you want to use as the primary and select Set as primary.

Q4: Does Responsive AI merge the metadata as well?

A: No. Currently only question, alternate question, standard response, and response and fields are merged.

Q5: Does the Duplicate Assistant merge documents?

A: Not at this time, but we plan to offer this in the future.

Q6: What subscription package do I need to access this feature?

A: You must have Responsive AI in order to access the Duplicate Assistant. The Agentic package is not required.