Support streaming hashes #747

ruckc · 2016-05-25T14:26:02Z

What You Are Seeing?

Installing large packages fail when file size is larger than 2GB.

When looking at CryptoHashProvider, it appears it reads the entire file into memory and then compute the hash, and then throw the byte[] away.

It would make sense to change the IHashAlgorithm/HashAlgorithm to handle IO.Stream.

var hash = _hashAlgorithm.ComputeHash(_fileSystem.read_file_bytes(filePath));

What is Expected?

The ability to install large packages.

How Did You Get This To Happen? (Steps to Reproduce)

Build a large nupkg and install via chocolatey.

Output Log

Error computing hash for 'C:\ProgramData\chocolatey\lib\BigPackage\BigPackage.nupkg'
 Hash will be special code for locked file or file too big instead.
 Captured error:
  The file is too long. This operation is currently limited to supporting files less than 2 gigabytes in size.

Additional Information

Zendesk Ticket 1
Zendesk Ticket 2

The text was updated successfully, but these errors were encountered:

ferventcoder · 2016-05-25T14:58:48Z

Please submit the gist of the chocolatey log output. Then we can look from there.

ferventcoder · 2016-05-25T14:59:29Z

Trying to determine where the actual issue occurs - more explanation: We only warn if we can't compute a hash. If you are seeing an error that is something else entirely.

ruckc · 2016-05-25T15:29:15Z

Roger... looking further in the long and we found the reason the install failed. It would be nice if streaming hashes were supported though...

ferventcoder · 2016-05-25T16:56:41Z

Was it a reason we could correct or is it something else?

MichaelGerhart · 2017-07-13T18:26:30Z

@ferventcoder was there ever any progress made on this? I am also having trouble deploying a package that is larger than 2GB with the same hash error, see the attached log file.
log.txt

ferventcoder · 2017-07-13T18:28:17Z

Error or Warning?

MichaelGerhart · 2017-07-13T18:30:15Z

Its listed as a warning, but causes the package installation to fail.

ferventcoder · 2017-07-13T18:32:06Z

Looks like warning

 Hash will be special code for locked file or file too big instead.
 Captured error:
  The file is too long. This operation is currently limited to supporting files less than 2 gigabytes in size.
2017-07-13 14:18:27,536 5380 [WARN ] - Error computing hash for 'C:\ProgramData\chocolatey\lib\MSSQL2014STD\tools\Sources\SQLServer2014x64Standard\STDISO.ISO'
 Hash will be special code for locked file or file too big instead.
 Captured error:
  The file is too long. This operation is currently limited to supporting files less than 2 gigabytes in size.sh

And then I see the package fails. I see the package fails afterwards.

Can you open up the chocolatey.log instead of the chocolatey.summary.log and include that output instead? We'd need all of that to determine whether that is really causing the issue or not. Thanks!

ferventcoder · 2017-07-13T18:32:35Z

Debug output is going to be much more helpful.

KurtTheBerner · 2017-08-11T13:41:08Z

I have exactly the same problem with a package larger than 2GB.
Gist Link with Debug Output:
https://gist.github.com/KurtTheBerner/6efdf08cbcac24a1f4ee2d28323383c0

Console Output:

Chocolatey v0.10.7
Installing the following packages:
MyBigPackage
By installing you accept licenses for the packages.

mybigpackage v1.2.3
mybigpackage package files install completed. Performing other installation steps.
Error computing hash for 'C:\ProgramData\chocolatey\lib\mybigpackage\mybigpackage.nupkg
 Hash will be special code for locked file or file too big instead.
 Captured error:
  The file is too long. This operation is currently limited to supporting files less than 2 gigabytes in size.
 The install of mybigpackage was successful.
  Software install location not explicitly set, could be in package or
  default install location if installer.
Chocolatey installed 1/1 packages.
 See the log for details (C:\ProgramData\chocolatey\logs\chocolatey.log).

Although chocolatey says that The install of mybigpackage was successful, the package was not installed.
For us this is an important point for the use of Chocolatey, we would be glad if the problem were solved soon.

KurtTheBerner · 2017-09-06T13:59:14Z

@ferventcoder any news about this issue?

ferventcoder · 2017-09-06T15:57:41Z

@KurtTheBerner I just reviewed your log, and the installation was successful for the package, but maybe not the software you were attempting to install. Perhaps there was some issue in the install-cmd for the software, but it did not report the error correctly back to to the powershell script which did not fail.

Looking at https://gist.githubusercontent.com/KurtTheBerner/6efdf08cbcac24a1f4ee2d28323383c0/raw/2522ba1cadfe8f6c11afd1f0e731661d21411ee1/chocolatey.log, you will see it reports the package install as successful, and it does see all of the files when it goes through and runs the checksums for the files (after the powerShell script runs). Finding the files and reporting success means the package installation was successful.

I would spend a closer look at the actual thing the installer is calling to see how it is not installing the software itself.

Here is the script you are running:

<#
==============================================================================
 FILE     : $chocolateyInstall.ps1 $
 PROJECT  : Install Script Library
 COPYRIGHT: Copyright (c) myCompany
 PURPOSE  : 
 COMMENT  : 
------------------------------------------------------------------------------
 HISTORY (latest entry first):
 2012-03-22  Version 1.0
==============================================================================
#>

$ErrorActionPreference = 'Stop'

$packageName = 'My big Package'
$toolsDir    = "$(Split-Path -parent $MyInvocation.MyCommand.Definition)"
$logDir      = $env:ALLUSERSPROFILE + "\MyCompany\InstallLogs\"
$logFileName = $logDir + "$packageName.log"

& $toolsDir\SQL-Install.cmd | Out-Null

Copy-Item -Path "$env:ProgramFiles\Microsoft SQL Server\120\Setup Bootstrap\Log\summary.txt" -Destination "$logDir\$packageName.log"

As you can see from the log, it finishes running that without any reported issues:

2017-08-11 14:36:26,081 7020 [DEBUG] - Built-in PowerShell host called with ['[System.Threading.Thread]::CurrentThread.CurrentCulture = '';[System.Threading.Thread]::CurrentThread.CurrentUICulture = ''; & import-module -name 'C:\ProgramData\chocolatey\helpers\chocolateyInstaller.psm1'; & 'C:\ProgramData\chocolatey\helpers\chocolateyScriptRunner.ps1' -packageScript 'C:\ProgramData\chocolatey\lib\mybigpackage\tools\chocolateyInstall.ps1' -installArguments '' -packageParameters '''] exited with '0'.

Since you are copying the log from the software install, I would take a closer look at that to see what errors it ran into. That's where you are going to likely find the issue. HTH

ferventcoder · 2017-09-06T15:58:55Z

Calling a .cmd file may not produce an exit code like you want, but you can always check LastExitCode to see if it errored on install.

KurtTheBerner · 2017-09-08T13:16:51Z

@ferventcoder I found the issue, the installation was cancelled and the ExitCode was not reported correctly.
I fixed it and now the Package has been successfully installed, but still the following Warning occurs:

Error computing hash for 'C:\ProgramData\chocolatey\lib\mybigpackage\mybigpackage.nupkg'
 Hash will be special code for locked file or file too big instead.
 Captured error:
  The file is too long. This operation is currently limited to supporting files less than 2 gigabytes in size.

This warning has irritated me and I thought the problem is caused by this Warning and prevents me from looking deeper in my Code.
Thanks for your Help.

Previously, when getting the checksum of a file, this read the file into a byte array to pass to ComputeHash. However, this was limited to files of 2gb or less, and was not efficent memory wise. This changes the method to using a filestream which is passed to ComputeHash, which should allow arbitrary file sizes with better memory usage. The unit test for this method had to be removed because the mock filesystem does not allow opening a file stream.

…files Previously, when getting the checksum of a file, this read the file into a byte array to pass to ComputeHash. However, this was limited to files of 2gb or less, and was not efficent memory wise. This changes the method to using a filestream which is passed to ComputeHash, which should allow arbitrary file sizes with better memory usage. The unit test for this method had to be removed because the mock filesystem does not allow opening a file stream.

ferventcoder added the 0 - _Triaging label May 25, 2016

ferventcoder changed the title ~~Error with packages larger than 2GB.~~ Support streaming hashes May 25, 2016

ferventcoder added the Improvement label May 25, 2016

TheCakeIsNaOH mentioned this issue Jul 7, 2021

Higher limit for package size chocolatey/home#59

Closed

TheCakeIsNaOH linked a pull request May 7, 2022 that will close this issue

(#747) Switch CryptoHashProvider to filestream for hashing files #2714

Open

8 tasks

TheCakeIsNaOH added 3 - Review and removed 0 - _Triaging labels Jul 30, 2022

TheCakeIsNaOH self-assigned this Jul 30, 2022

TheCakeIsNaOH mentioned this issue Jun 10, 2024

Chocolatey is using checksum.exe which is slow and can't cope with larger than 2GB files chocolatey/checksum#23

Open

6 tasks

imm0rtalsupp0rt added the Customer label Aug 23, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support streaming hashes #747

Support streaming hashes #747

ruckc commented May 25, 2016 •

edited by imm0rtalsupp0rt

Loading

ferventcoder commented May 25, 2016

ferventcoder commented May 25, 2016

ruckc commented May 25, 2016

ferventcoder commented May 25, 2016

MichaelGerhart commented Jul 13, 2017

ferventcoder commented Jul 13, 2017

MichaelGerhart commented Jul 13, 2017

ferventcoder commented Jul 13, 2017

ferventcoder commented Jul 13, 2017

KurtTheBerner commented Aug 11, 2017

KurtTheBerner commented Sep 6, 2017

ferventcoder commented Sep 6, 2017

ferventcoder commented Sep 6, 2017

KurtTheBerner commented Sep 8, 2017

Support streaming hashes #747

Support streaming hashes #747

Comments

ruckc commented May 25, 2016 • edited by imm0rtalsupp0rt Loading

What You Are Seeing?

What is Expected?

How Did You Get This To Happen? (Steps to Reproduce)

Output Log

Additional Information

ferventcoder commented May 25, 2016

ferventcoder commented May 25, 2016

ruckc commented May 25, 2016

ferventcoder commented May 25, 2016

MichaelGerhart commented Jul 13, 2017

ferventcoder commented Jul 13, 2017

MichaelGerhart commented Jul 13, 2017

ferventcoder commented Jul 13, 2017

ferventcoder commented Jul 13, 2017

KurtTheBerner commented Aug 11, 2017

KurtTheBerner commented Sep 6, 2017

ferventcoder commented Sep 6, 2017

ferventcoder commented Sep 6, 2017

KurtTheBerner commented Sep 8, 2017

ruckc commented May 25, 2016 •

edited by imm0rtalsupp0rt

Loading