Validate chunk download and fall back to release-asset mirror
archive.nyu.edu rate-limits by IP (mod_qos) and serves an HTML
restrictions page to CI runners instead of the archive.
2 changed files+31−5
README.mdmodified+7−0View file
@@ -45,6 +45,13 @@ extracts the first 1000 model directories, gzips each `.step` file, generates
4545 `index.json`, and deploys the result to GitHub Pages. The content is a fixed
4646 slice of the dataset, so the workflow runs on manual dispatch only.
4747
48+The canonical host (archive.nyu.edu) rate-limits downloads per IP and often
49+serves CI runners an HTML restrictions page instead of the archive, so the
50+build validates the download and falls back to a
51+[release asset](https://github.com/concept-collection/abc-step-1000/releases/tag/data-v00)
52+on this repository that repacks just the first 1000 model directories of the
53+canonical chunk (byte-identical files).
54+
4855 ## Source and acknowledgments
4956
5057 All CAD models come from the **ABC dataset**:
scripts/build.shmodified+24−5View file
@@ -16,15 +16,34 @@ N=${N:-1000}
1616 mkdir -p "$WORK"
1717 cd "$WORK"
1818
19+# The name recorded in index.json always refers to the canonical source chunk.
20+CHUNK_NAME=abc_0000_step_v00.7z
21+
22+# Mirror holding a repack of just the first 1000 model directories of the
23+# canonical chunk (byte-identical files). Used because archive.nyu.edu
24+# rate-limits by IP (mod_qos) and often redirects CI runners to an HTML
25+# restrictions page instead of serving the file.
26+MIRROR_URL=https://github.com/concept-collection/abc-step-1000/releases/download/data-v00/abc_0000_step_v00_first1000.7z
27+
28+is_7z() { [ "$(head -c 2 "$1" 2> /dev/null)" = "7z" ]; }
29+
1930 if [ -z "${CHUNK_ARCHIVE:-}" ]; then
2031 wget -q https://deep-geometry.github.io/abc-dataset/data/step_v00.txt
2132 CHUNK_URL=$(sed '1q;d' step_v00.txt | awk '{print $1}')
22- CHUNK_NAME=$(sed '1q;d' step_v00.txt | awk '{print $2}')
23- echo "Downloading $CHUNK_NAME from $CHUNK_URL ..."
24- wget -q --no-check-certificate "$CHUNK_URL" -O "$CHUNK_NAME"
25- CHUNK_ARCHIVE="$WORK/$CHUNK_NAME"
33+ for attempt in 1 2 3; do
34+ echo "Downloading $CHUNK_NAME from $CHUNK_URL (attempt $attempt) ..."
35+ wget -q --no-check-certificate "$CHUNK_URL" -O chunk.7z || true
36+ is_7z chunk.7z && break
37+ echo "Response is not a 7z archive (rate-limited?); retrying in 30 s"
38+ sleep 30
39+ done
40+ if ! is_7z chunk.7z; then
41+ echo "Falling back to mirror: $MIRROR_URL"
42+ wget -q "$MIRROR_URL" -O chunk.7z
43+ is_7z chunk.7z
44+ fi
45+ CHUNK_ARCHIVE="$WORK/chunk.7z"
2646 fi
27-CHUNK_NAME=$(basename "$CHUNK_ARCHIVE")
2847
2948 # Extract only the first N model directories; the full chunk holds 10000
3049 # models (~15 GB uncompressed), far more than we need or than CI disk allows.