Testing for duplicates

You can test one or more fields in the active table to detect whether duplicate values or entire duplicate records exist.

To test for duplicates:

  1. Select Analyze > Look for Duplicates.
  2. If you want to detect duplicates in one or more fields, in the Main tab do the following:
    1. Select the field(s) to test from the Duplicates On list, or click Duplicates On to select the field(s), or to create an expression.

      The order in which you select the fields is the order in which the columns appear in the results. If you are appending results to an existing ACL table, the column selection and order must be identical to the column selection and order in the existing table.

    2. If you clicked Duplicates On, you can optionally specify a descending sort order in the output results for one or more selected fields by clicking the sort arrow (the default is ascending).
    3. Select one or more List Fields to include any additional field(s) in the output results, or click List Fields to select the field(s), to Add All fields, or to create an expression.

      Additional fields can provide useful context for the results. Fields selected for duplicates testing are displayed automatically at the beginning of any result records and do not need to be specifically selected under List Fields.

  3. If you want to detect entire duplicate records, in the Main tab do the following:
    1. Click Duplicates On.
    2. Click Add All to add all fields to Selected Fields.
    3. Optionally specify a descending sort order in the output results for one or more fields by clicking the sort arrow (the default is ascending).
    4. Click OK.

      You do not need to select any fields from the List Fields list because all fields in the table are displayed automatically in the result records.

  4. If there are records in the current view that you want to exclude from processing, enter a condition in the If text box, or click If to create an IF statement using the Expression Builder.

    The IF statement considers all records in the view and filters out those that do not meet the specified condition.

  5. If the field or fields you are testing are already sorted based on a prior operation, you can optionally deselect Presort to save time when testing large tables for duplicates.
    Note

    If you deselect Presort, the field or fields you select for duplicates testing must match the field or fields that were previously sorted for results to be valid. The message Warning: File out of sequence accompanies results if there is a mismatch between selected and sorted fields. If you output results to an ACL table, the warning message appears in the command log.

    If data in field(s) is not sorted you must leave Presort selected to ensure that all duplicates are found.

  6. Click the Output tab.
  7. Select the appropriate output option in the To panel:
    • Screen – Select this option to display the results in the ACL display area. If the output table contains a large number of records, it is faster and more useful to save the results to a file than to display the results on the screen.

    • Print – Select this option to send the results to the default printer.

    • Graph – Select this option to create a graph of the results and display it in the ACL display area.

    • File – Select this option to save or append the results to an ACL table or a text file. If you save or append to an ACL table, the table is added to the open project if it is not already in the project. If you save or append to a text file, the file is saved outside ACL.

    Note

    Output options that do not apply to a particular analytical operation are disabled.

  8. If you selected File as the output type, specify the following information in the As panel:
    • File Type – Select ACL Table to save the results to a new ACL table, or append the results to an existing ACL table. Select ASCII Text File or Unicode Text file (depending on which edition of ACL you are using) to save or append the results to a text file.

    • Name – Enter a file name in the Name text box. Or click Name and enter the file name, or select an existing file in the Save or Save File As dialog box to overwrite or append to the file. If ACL prefills a file name, you can accept the prefilled name, or change it.

      You can also specify an absolute or relative file path, or navigate to a different folder, to save or append the file in a location other than the project location. For example: C:\Results\Output.fil or Results\Output.fil.

    • Local – Only enabled when connected to a server table and saving or appending the results to an ACL table. Select Local to save the file to the same location as the project, or to specify a path or navigate to a different local folder. Leave Local deselected to save the file to the Prefix folder on the ACL Server.

      Note

      For output results produced from analysis or processing of ACL Analytics Exchange server tables, select Local. You cannot use the Local setting to import results tables to ACL Analytics Exchange Server.

  9. Depending on the output type, you can optionally specify a Header and/or a Footer in the text box(es).

    Headers and footers are centered by default. Type a left angle bracket (<) before the header or footer text to left align the text. Click Header or Footer to enter a header or footer of more than one line. Alternatively, you can enter a semi-colon (;) as a line-break character in the header or footer text box. Left aligning multiple lines requires a left angle bracket at the beginning of each line.

  10. Click the More tab.
  11. Select the appropriate option in the Scope panel.
    • All – This option is selected by default. Leave it selected to specify that all records in the view will be processed.

    • First – Select this option and enter a number in the text box to start processing at the first record in the view and include only the specified number of records.

    • Next – Select this option and enter a number in the text box to start processing at the currently selected record in the view and include only the specified number of records. The actual record number in the leftmost column must be selected, not data in the row.

    • While – Select this option to use a WHILE statement to limit the processing of records in the view based on a particular criterion or set of criteria. You can enter a condition in the While text box, or click While to create a WHILE statement using the Expression Builder.

      A WHILE statement allows records in the view to be processed only while the specified condition evaluates to true. As soon as the condition evaluates to false, the processing terminates, and no further records are considered. You can use the While option in conjunction with the All, First, or Next options.

    Note

    The number of records specified in the First or Next options references either the physical or the indexed order of records in a table, and disregards any filtering or quick sorting applied to the view. However, results of analytical operations respect any filtering.

    If a view is quick sorted, Next behaves like First.

  12. If you selected File as the output type, and want to append the output results to the end of an existing file, do one of the following:
    • Select Append To Existing File if you are appending to a text file, or to an ACL table that you are certain is identical in structure to the output results.

    • Leave Append To Existing File deselected if you are appending to an ACL table and you want ACL to compare the record lengths of the output results and the existing table. If the record lengths are not identical, the data structure is not identical.

    Note

    Leaving Append To Existing File deselected is recommended if you are uncertain about the data structure. For more information about appending and data structure, see Appending results to ACL tables and text files.

  13. If you selected File (ACL Table) as the output type, select or deselect Use Output Table depending on whether or not you want the ACL table containing the output results to open automatically upon completion of the operation.
  14. Click OK.
    Note

    Only the duplicate values or records are displayed, and not the initial occurrence of a value or record, if you  a) output the results to screen or a text file, and  b) include only tested fields in the output results and do not select any additional field(s). If you output to screen, you can click any value to see the initial occurrence of a value or record along with the duplicates.

  15. If the overwrite prompt appears, select the appropriate option.

    If you are expecting the Append option to appear and it does not, click No to cancel the operation and see Appending results to ACL tables and text files.

Related concepts
Saving results and specifying results output folders
About duplicates
Related tasks
Removing duplicate records
Testing for fuzzy duplicates


(C) 2013 ACL Services Ltd. All Rights Reserved. | Send feedback