summaryrefslogtreecommitdiff
path: root/tool/downloader.rb
AgeCommit message (Collapse)Author
2021-10-28Downloader: retry when RFC 2616 noncompliant dates [ci skip]Nobuyoshi Nakada
zlib.net rarely returns the current time in RFC 2616 noncompliant format in the response header, and the checksum does not match in that case (maybe creating the tarball on the fly?).
2021-09-13Downloader: accept some RFC 2616 noncompliant datesNobuyoshi Nakada
2020-11-23Increase the # of downloader.rb's retry attemptsTakashi Kokubun
Apparently 9 was not enough either. https://rubyci.org/logs/rubyci.s3.amazonaws.com/rhel_zlinux/ruby-master/log/20201123T063303Z.fail.html.gz
2020-11-20Try to fix download errorKazuhiro NISHIYAMA
https://github.com/ruby/ruby/runs/1428320660?check_suite_focus=true#step:9:10 ``` tool/downloader.rb:243:in `rescue in download': failed to download config.guess (RuntimeError) OpenURI::HTTPError: 403 Forbidden: https://cdn.jsdelivr.net/gh/gcc-mirror/gcc@master/config.guess ``` Notes: Merged: https://github.com/ruby/ruby/pull/3794
2020-03-09Use https for downloading Unicode data.Hiroshi SHIBATA
Retry with 90de118a75921a08497b41de2950971073b3c924 [Bug #13962][ruby-core:83074]
2019-11-22Use jsDelivr instead of raw.githubusercontent.comKazuhiro NISHIYAMA
Try to fix download error on Solaris CI https://rubyci.org/logs/rubyci.s3.amazonaws.com/unstable11x/ruby-master/log/20191121T162422Z.fail.html.gz ``` tool/downloader.rb:243:in `rescue in download': failed to download config.guess (RuntimeError) Net::HTTPFatalError: 503 "Service Unavailable": https://raw.githubusercontent.com/gcc-mirror/gcc/master/config.guess ```
2019-05-29Revert "Fix building with 1.8 BASERUBY"Jeremy Evans
This reverts commit 05bc14d81a1d7f6af826a92371aeff0c3fb2a67e. We have decided that the cost of reintroducing support for 1.8 BASERUBY outweighs the benefit. If you are still using 1.8 and want to build master/trunk, build and install the latest release, and use that as BASERUBY.
2019-05-28Clean a garbage [ci skip]Nobuyoshi Nakada
2019-05-27Fix building with 1.8 BASERUBYJeremy Evans
2019-05-24Escape dots in regexpKazuhiro NISHIYAMA
2019-05-24switch UNICODE_BETA to NO (one more try, first try didn't work)Martin Dürst
Unicode version 12.1.0 was officially released on May 7th, 2019. There were no changes at all from the "real" beta published shortly after the new era name "Reiwa" was announced. So we can switch UNICODE_BETA back to NO. common.mk: switch UNICODE_BETA back to NO tool/downloader.rb: add additional conditions to avoid an error when moving from beta to final
2019-04-30Change Accept-Encoding from `*` to `identity`Kazuhiro NISHIYAMA
When `Accept-Encoding` is `*`, http://www.unicode.org/Public/12.1.0/ucd/ returns gzipped content now. So set `identity`.
2019-04-27Retry downloads more for unicode outageTakashi Kokubun
like https://ci.appveyor.com/project/ruby/ruby/builds/24142523/job/v6aq4srj7c3hgt86
2019-04-17downloader.rb: enable cache filesnobu
* tool/downloader.rb (Downloader::Unicode.download): enable cache files when index.html has not been modified since the previous download, even if beta version. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@67594 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2019-04-17downloader.rb: use File.read instead of IO.readnobu
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@67591 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2019-04-06appveyor.yml: Use pre-generated headers and tablesnobu
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@67454 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2019-04-05revert r67445, (r67446,) r67447duerst
Debugging output is no longer needed because the problem has been fixed with r67449. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@67451 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2019-04-05downloader.rb: fix typo, extra %nobu
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@67450 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2019-04-05downloader.rb: quote base namenobu
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@67448 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2019-04-05output more debug information in downloader.rb (temporary)duerst
Unicode file download doesn't work with Visual Studio, we need more debug output. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@67447 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2019-04-05* remove trailing spaces.svn
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@67446 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2019-04-05add puts statements to debug Unicode file download (temporary)duerst
Unicode file download doesn't work with Visual Studio, see e.g. https://ci.appveyor.com/project/ruby/ruby/builds/23614399/job/f8vya2l7fjdfcye4 We temporarily produce more output for debugging. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@67445 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2019-04-02downloader.rb: keep linked file newer than cached filenobu
* tool/downloader.rb (Downloader.save_cache): keep linked file newer than cached file, so that GNU make triggers when the content is updated. it uses the timestamp of symlink itself instead of the target. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@67413 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2019-01-06simplify filename-related codeduerst
(Thanks to Nobuyoshi Nakada for the hint!) git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@66738 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2019-01-06add logic to handle Unicode beta period file namesduerst
In downloader.rb, add logic to handle file names of the form UnicodeData-12.0.0d6.txt. To find the right file, we download the index of the directory. Then we download the files by finding the file names from the index. Files are always checked for changes, because changes might be frequent during the beta period. We also check whether any index.html files are left when we are not in the beta period. This would indicate that we might have stale data from the beta period rather than the actual release data. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@66737 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-12-19avoid passing unnecessary options to download methodduerst
The option --unicode-beta for tool/downloader.rb introduced in r66448 must not be passed as an option to actual download machinery. Thanks to MSP-Greg (Greg L.) for bug report and patch. This closes issue #15434. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@66449 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-12-19add option to deal with Unicode beta data filesduerst
Unicode uses file names with explicit versions for beta publication. This commit introduces a variable on the makefile level to distinguish between beta and regular versions of file names. common.mk: Define new variable UNICODE_BETA, usually set to NO, but would be YES during tests with beta data files. Pass the value of this variable to tool/downloader.rb with option --unicode-beta. tool/downloader.rb: Receive and store value of --unicode-beta. Raise an exception if value is YES, because we don't yet actually deal with this case. Continue as usual if value is not YES. This completes the changes needed in the makefile. Upcomming changes will only affect tool/downloader.rb. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@66448 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-12-01tool/downloader.rb: retry Net::OpenTimeoutk0kubun
to prevent random failure https://travis-ci.org/ruby/ruby/jobs/461667474 git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@66110 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-10-16tool/downloader.rb: retry Net::ReadTimeoutk0kubun
as failed in https://ci.appveyor.com/project/ruby/ruby/builds/19540849/job/9p9pq0b7hfrb2r8q git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@65097 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-08-18tool/downloader.rb: retry 500 from GitHubk0kubun
Unfortunately, GitHub may return 500 for temporary failure: https://travis-ci.org/ruby/ruby/builds/417246523 git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64425 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-08-12tool/downloader.rb: increase retriesk0kubun
GitHub download failed on 13:50:36 https://ci.appveyor.com/project/ruby/ruby/build/1.0.9221 and it also failed on 13:51:35 (all builds between them failed too). It means that we need to expect GitHub 502 that continues 1 minute. So I configured 6 retries, that will sleep at most 91s in total. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64317 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-08-11tool/downloader.rb: retry on 502 and 503 errork0kubun
rubyci was failed by download 503 https://rubyci.org/logs/rubyci.s3.amazonaws.com/scw-9d6766/ruby-trunk/log/20180811T021706Z.fail.html.gz git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64282 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-07-16tool/downloader.rb:duerst
* remove initial "./" from destdir to make downloading ./enc/unicode/data/11.0.0/ucd/auxiliary/GraphemeBreakProperty.txt work properly * tweak a comment git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@63978 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-02-05tool/downloader.rb: retry SocketErrork0kubun
I would like to stabilize RubyCI. https://rubyci.org/logs/rubyci.s3.amazonaws.com/fedora26/ruby-trunk/log/20180205T123003Z.fail.html.gz downloading did_you_mean-1.2.0.gem ... /home/hsbt/chkbuild/tmp/build/20180205T123003Z/tmp/ruby-snapshot20180205-23226-3201ha/ruby-2.6.0-r62226/tool/downloader.rb:212:in `rescue in download': failed to download did_you_mean-1.2.0.gem (RuntimeError) SocketError: Failed to open TCP connection to rubygems.org:443 (getaddrinfo: Name or service not known): https://rubygems.org/downloads/did_you_mean-1.2.0.gem from /home/hsbt/chkbuild/tmp/build/20180205T123003Z/tmp/ruby-snapshot20180205-23226-3201ha/ruby-2.6.0-r62226/tool/downloader.rb:123:in `download' from /home/hsbt/chkbuild/tmp/build/20180205T123003Z/tmp/ruby-snapshot20180205-23226-3201ha/ruby-2.6.0-r62226/tool/downloader.rb:66:in `download' from -e:4:in `<main>' make[1]: *** [Makefile:1685: update-gems] Error 1 git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@62231 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-01-10tool/downloader.rb: retry against Errno::ETIMEDOUTk0kubun
instead of Errno::ECONNREFUSED. As I commented in r61498, at that moment I was not sure if the actual exception is really `Errno::ECONNREFUSED` or not. In https://ci.appveyor.com/project/ruby/ruby/build/1.0.6974, I could confirm that the download can fail with `Errno::ETIMEDOUT`. So I want to retry that. Let's add `Errno::ECONNREFUSED` too if it fails with the exception. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@61762 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2017-12-27tool/downloader.rb: don't suppress exceptionk0kubun
on the end of retry failure. Fixed mistake in r61498 git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@61499 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2017-12-27tool/downloader.rb: retry downloadsk0kubun
because it's randomly failing on CI like https://ci.appveyor.com/project/ruby/ruby/build/1.0.6724 Actually I'm not sure whether the exception class is Errno::ECONNREFUSED or not. Please change the rescued exception to the correct one if it's wrong. I changed to log exception class too in this commit. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@61498 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2017-12-12Revert "switch from http to https for Unicode data file downloads"hsbt
This reverts commit 90de118a75921a08497b41de2950971073b3c924. The environment of mswinci couldn't handle https download. (It's certificates problem, We should upgrade its environment) git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@61169 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2017-12-12switch from http to https for Unicode data file downloadsduerst
(patch from MSP-Greg (Greg L), this closes issue #13962) git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@61145 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2017-07-09downloader.rb: get rid of symlinks in dist filesnobu
* tool/downloader.rb: download to the target file directly if CACHE_SAVE=no, to get rid of symlinks in dist files. [ruby-core:81887] [Bug #13709] git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@59292 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2017-06-27downloader.rb: get rid of symlinksnobu
* tool/downloader.rb: do not save downloaded files to cache. [ruby-core:81780] [Bug #13684] git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@59185 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2017-05-22downloader.rb: true symlink on cygwinnobu
* tool/downloader.rb (Downloader.link_cache): on Cygwin, make true symlink only, which is provided by the OS. as mingw/mswin ruby can't follow cygwin's pseudo symlink, it will fail on the same source tree. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@58843 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2017-05-22fix r58833nobu
* tool/downloader.rb (Downloader.download): just link to the cache when downloaded to the cache but not the target file. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@58841 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2017-05-21downloader cachenobu
* tool/downloader.rb (Downloader.download): manage download cache. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@58833 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2017-05-21remove verify optionnobu
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@58832 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2017-05-21downloader.rb: utilize dir argumentnobu
* tool/downloader.rb (Downloader::RubyGems.download): utilize `dir` argument of Downloader.download. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@58831 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2017-03-23fix GraphemeBreakProperty.txtnobu
* tool/downloader.rb: download to the file given in ARGV. * tool/enc-unicode.rb (parse_GraphemeBreakProperty): fix data file path as $(UNICODE_PROPERTY_FILES) in common.mk. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@58069 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2017-03-21tool/downloader.rb: dryrun option [ci skip]nobu
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@58043 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2016-11-30Regexp supports Unicoe 9.0.0's \Xnaruse
* meta character \X matches Unicode 9.0.0 characters with some workarounds for UTR #51 Unicode Emoji, Version 4.0 emoji zwj sequences. [Feature #12831] [ruby-core:77586] The term "character" can have many meanings bytes, codepoints, combined characters, and so on. "grapheme cluster" is highest one of such words, which means user-perceived characters. Unicode Standard Annex #29 UNICODE TEXT SEGMENTATION specifies how to handle grapheme clusters (extended grapheme cluster). But some specs aren't updated to current situation because Unicode Emoji is rapidly extended without well definition. It breaks the precondition of UTR#29 "Grapheme cluster boundaries can be easily tested by looking at immediately adjacent characters". (the sentence will be removed in the next version) Though some of its detail are described in Unicode Technical Report #51 UNICODE EMOJI but it is not merged into UTR#29 yet. http://unicode.org/reports/tr29/ http://unicode.org/reports/tr51/ http://unicode.org/Public/emoji/4.0/ git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@56949 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2016-10-12* tool/downloader.rb: Removed verification of gem certification.hsbt
Because signed gem is not working on rubygems ecosystem. * tool/gem-unpack.rb: ditto. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@56399 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
generated by cgit v1.2.3 (git 2.25.1) at 2024-05-17 04:04:25 +0000