Downloads:

1,269

Downloads of v 1.102:

180

Last Update:

28 Jan 2019

Package Maintainer(s):

Software Author(s):

  • Hermann Schinagl

Tags:

cli ntfs data-integrity checksum dedupe freeware

Dupemerge

1.102 | Updated: 28 Jan 2019

Downloads:

1,269

Downloads of v 1.102:

180

Maintainer(s):

Software Author(s):

  • Hermann Schinagl

Dupemerge 1.102

Some Checks Have Failed or Are Not Yet Complete

1 Test Passing and 1 Failing Test


Validation Testing Passed


Verification Testing Failed

Details

To install Dupemerge, run the following command from the command line or from PowerShell:

>

To upgrade Dupemerge, run the following command from the command line or from PowerShell:

>

To uninstall Dupemerge, run the following command from the command line or from PowerShell:

>

NOTE: This applies to both open source and commercial editions of Chocolatey.

1. Ensure you are set for organizational deployment

Please see the organizational deployment guide

  • Open Source or Commercial:
    • Proxy Repository - Create a proxy nuget repository on Nexus, Artifactory Pro, or a proxy Chocolatey repository on ProGet. Point your upstream to https://chocolatey.org/api/v2. Packages cache on first access automatically. Make sure your choco clients are using your proxy repository as a source and NOT the default community repository. See source command for more information.
    • You can also just download the package and push it to a repository Download

3. Enter your internal repository url

(this should look similar to https://chocolatey.org/api/v2)

4. Choose your deployment method:


choco upgrade dupemerge -y --source="'STEP 3 URL'" [other options]

See options you can pass to upgrade.

See best practices for scripting.

Add this to a PowerShell script or use a Batch script with tools and in places where you are calling directly to Chocolatey. If you are integrating, keep in mind enhanced exit codes.

If you do use a PowerShell script, use the following to ensure bad exit codes are shown as failures:


choco upgrade dupemerge -y --source="'STEP 3 URL'" 
$exitCode = $LASTEXITCODE

Write-Verbose "Exit code was $exitCode"
$validExitCodes = @(0, 1605, 1614, 1641, 3010)
if ($validExitCodes -contains $exitCode) {
  Exit 0
}

Exit $exitCode

- name: Ensure dupemerge installed
  win_chocolatey:
    name: dupemerge
    state: present
    version: 1.102
    source: STEP 3 URL

See docs at https://docs.ansible.com/ansible/latest/modules/win_chocolatey_module.html.

Coming early 2020! Central Managment Reporting available now! More information...


chocolatey_package 'dupemerge' do
  action    :install
  version  '1.102'
  source   'STEP 3 URL'
end

See docs at https://docs.chef.io/resource_chocolatey_package.html.


Chocolatey::Ensure-Package
(
    Name: dupemerge,
    Version: 1.102,
    Source: STEP 3 URL
);

Requires Otter Chocolatey Extension. See docs at https://inedo.com/den/otter/chocolatey.


cChocoPackageInstaller dupemerge
{
   Name     = 'dupemerge'
   Ensure   = 'Present'
   Version  = '1.102'
   Source   = 'STEP 3 URL'
}

Requires cChoco DSC Resource. See docs at https://github.com/chocolatey/cChoco.


package { 'dupemerge':
  provider => 'chocolatey',
  ensure   => '1.102',
  source   => 'STEP 3 URL',
}

Requires Puppet Chocolatey Provider module. See docs at https://forge.puppet.com/puppetlabs/chocolatey.


salt '*' chocolatey.install dupemerge version="1.102" source="STEP 3 URL"

See docs at https://docs.saltstack.com/en/latest/ref/modules/all/salt.modules.chocolatey.html.

5. If applicable - Chocolatey configuration/installation

See infrastructure management matrix for Chocolatey configuration elements and examples.

Private CDN cached downloads available for licensed customers. Never experience 404 breakages again! Learn more...

There are versions of this package awaiting moderation . See the Version History section below.

This package was approved as a trusted package on 09 Sep 2019.

Description

Overview

Dupemerge searches for identical files on a logical drive and creates hardlinks among those files, thus saving lots of hard disk space. Most hard disks contain quite a lot of completely identical files, which consume a lot of disk space. This waste of space can be drastically reduced by using the NTFS file system hardlink functionality to link the identical files ("dupes") together.

Technical

Dupemerge creates a cryptological hashsum for each file found below the given paths and compares those hashes to each other to find the dupes. There is no file date comparison involved in detecting dupes, only the size and content of the files.

To speed up comparison, only files with the same size get compared to each other. Furthermore the hashsums for equal sized files get calculated incrementally, which means that during the first pass only the first 4 kilobyte are hashed and compared, and during the next rounds more and more data are hashed and compared.

Due to long run time on large disks, a file which has already been hashsummed might change before all dupes to that file are found. To prevent false hardlink creation due to intermediate changes, dupemerge saves the file write time of a file when it hashsums the file and checks back if this time changed when it tries to hardlink dupes.

Multiple Runs

If dupemerge is run once, hardlinks among identical files are created. To save time during a second run on the same locations, dupemerge checks if a file is already a hardlink, and tries to find the other hardlinks by comparing the unique NTFS file-id. This saves a lot of time, because checksums for large files need not be created twice.

Transaction based Hardlinking

Before DupeMerge hardlinks file together it renames the file to a temporary name, then creates the hardlink, and afterwards deletes the temporary file. All that is done to be able to roll-back the operation if e.g the hardlinking failes.

TimeStamp Handling

A tupel of hardlinks for one file has always the one timestamp. This is by design of NTFS. But things are a bit confusing sometimes, because after hardlinking the same timestamp is only shown, after the hardlink was once e.g. opened/accessed. So it may happen, that immediatley after dupemerge one observes different timestamps within a tupel of hardlinks, but after such a hardlink has been opened for e.g. read, the timestamp changes to the timestamp of the whole tupel. That's also by design of NTFS.

Dupemerge has a dupe-find algorithm which is tuned to perform especially well on large server disks, where it has been tested in depth to guarantee data integrity.

Documentation


tools\chocolateyInstall.ps1
$packageName = 'dupemerge'
$url = 'http://schinagl.priv.at/nt/dupemerge/dupemerge.zip'
$checksum = '2818fd202048beec296d0c79c33ccdb2d3124aa3d85f37a5de49736e394f1819'
$checksumType = 'sha256'
$url64 = 'http://schinagl.priv.at/nt/dupemerge/dupemerge64.zip'
$checksum64 = 'd15236956de6a555606324560735edd7d143fd7b00582ec428ddd44cc5524731'
$checksumType64 = 'sha256'
$toolsDir = "$(Split-Path -parent $MyInvocation.MyCommand.Definition)"

Install-ChocolateyZipPackage -PackageName "$packageName" `
                             -Url "$url" `
                             -UnzipLocation "$toolsDir" `
                             -Url64bit "$url64" `
                             -Checksum "$checksum" `
                             -ChecksumType "$checksumType" `
                             -Checksum64 "$checksum64" `
                             -ChecksumType64 "$checksumType64"

In cases where actual malware is found, the packages are subject to removal. Software sometimes has false positives. Moderators do not necessarily validate the safety of the underlying software, only that a package retrieves software from the official distribution point and/or validate embedded software against official distribution point (where distribution rights allow redistribution).

Chocolatey Pro provides runtime protection from possible malware.

Version Downloads Last Updated Status
Dupemerge 1.103 2 Saturday, November 30, 2019 Ready
Dupemerge 1.080.0.20170625 332 Thursday, September 21, 2017 Approved
Dupemerge 1.080 364 Sunday, June 18, 2017 Approved
Dupemerge 1.07 378 Sunday, April 19, 2015 Approved

Program

Package

  • Automatically built and uploaded by dtgm

Discussion for the Dupemerge Package

Ground Rules:

  • This discussion is only about Dupemerge and the Dupemerge package. If you have feedback for Chocolatey, please contact the Google Group.
  • This discussion will carry over multiple versions. If you have a comment about a particular version, please note that in your comments.
  • The maintainers of this Chocolatey Package will be notified about new comments that are posted to this Disqus thread, however, it is NOT a guarantee that you will get a response. If you do not hear back from the maintainers after posting a message below, please follow up by using the link on the left side of this page or follow this link to contact maintainers. If you still hear nothing back, please follow the package triage process.
  • Tell us what you love about the package or Dupemerge, or tell us what needs improvement.
  • Share your experiences with the package, or extra configuration or gotchas that you've found.
  • If you use a url, the comment will be flagged for moderation until you've been whitelisted. Disqus moderated comments are approved on a weekly schedule if not sooner. It could take between 1-5 days for your comment to show up.
comments powered by Disqus