Host first 1000 ABC-dataset STEP files on GitHub Pages
6 changed files+342−0
.github/workflows/deploy.ymladded+42−0View file
@@ -0,0 +1,42 @@
1+name: Build and deploy to GitHub Pages
2+
3+# The hosted content is static (a fixed slice of the ABC dataset), so the
4+# workflow only runs on manual dispatch — no need to re-download the ~1.6 GB
5+# chunk on every push.
6+on:
7+ workflow_dispatch:
8+
9+permissions:
10+ contents: read
11+ pages: write
12+ id-token: write
13+
14+concurrency:
15+ group: pages
16+ cancel-in-progress: false
17+
18+jobs:
19+ build:
20+ runs-on: ubuntu-latest
21+ steps:
22+ - uses: actions/checkout@v4
23+ - name: Ensure 7z is available
24+ run: |
25+ if ! command -v 7z > /dev/null; then
26+ sudo apt-get update && sudo apt-get install -y p7zip-full
27+ fi
28+ - name: Build site
29+ run: scripts/build.sh
30+ - uses: actions/upload-pages-artifact@v3
31+ with:
32+ path: build/site
33+
34+ deploy:
35+ needs: build
36+ runs-on: ubuntu-latest
37+ environment:
38+ name: github-pages
39+ url: ${{ steps.deployment.outputs.page_url }}
40+ steps:
41+ - id: deployment
42+ uses: actions/deploy-pages@v4
.gitignoreadded+1−0View file
@@ -0,0 +1 @@
1+build/
README.mdadded+74−0View file
@@ -0,0 +1,74 @@
1+# abc-step-1000
2+
3+The first 1000 STEP files from the [ABC dataset](https://deep-geometry.github.io/abc-dataset/),
4+hosted on GitHub Pages for convenient direct download.
5+
6+- **Browse:** https://concept-collection.github.io/abc-step-1000/
7+- **Manifest:** https://concept-collection.github.io/abc-step-1000/index.json
8+
9+The files are served gzip-compressed (`.step.gz`, ~300 MB total; ~1.6 GB
10+uncompressed). `index.json` lists every file with its download path, model ID,
11+and compressed/uncompressed sizes.
12+
13+## Downloading
14+
15+A single file:
16+
17+```sh
18+curl -sL https://concept-collection.github.io/abc-step-1000/step/00000002_1ffb81a71e5b402e966b9341_step_001.step.gz \
19+ | gunzip > model.step
20+```
21+
22+All files, using the manifest:
23+
24+```sh
25+BASE=https://concept-collection.github.io/abc-step-1000
26+curl -sL $BASE/index.json | jq -r '.files[].path' \
27+ | xargs -P 8 -I{} sh -c 'curl -sL "$1/$2" | gunzip > "$(basename "$2" .gz)"' _ $BASE {}
28+```
29+
30+In the browser, decompress with
31+[`DecompressionStream`](https://developer.mozilla.org/en-US/docs/Web/API/DecompressionStream):
32+
33+```js
34+const res = await fetch(url);
35+const step = await new Response(
36+ res.body.pipeThrough(new DecompressionStream('gzip'))
37+).text();
38+```
39+
40+## How it is built
41+
42+The GitHub Actions workflow ([deploy.yml](.github/workflows/deploy.yml))
43+downloads the first chunk of the STEP format (`abc_0000_step_v00.7z`, ~1.6 GB),
44+extracts the first 1000 model directories, gzips each `.step` file, generates
45+`index.json`, and deploys the result to GitHub Pages. The content is a fixed
46+slice of the dataset, so the workflow runs on manual dispatch only.
47+
48+## Source and acknowledgments
49+
50+All CAD models come from the **ABC dataset**:
51+
52+> Koch, Sebastian and Matveev, Albert and Jiang, Zhongshi and Williams, Francis
53+> and Artemov, Alexey and Burnaev, Evgeny and Alexa, Marc and Zorin, Denis and
54+> Panozzo, Daniele. *ABC: A Big CAD Model Dataset For Geometric Deep Learning.*
55+> CVPR 2019.
56+
57+```bibtex
58+@InProceedings{Koch_2019_CVPR,
59+ author = {Koch, Sebastian and Matveev, Albert and Jiang, Zhongshi and Williams, Francis and Artemov, Alexey and Burnaev, Evgeny and Alexa, Marc and Zorin, Denis and Panozzo, Daniele},
60+ title = {ABC: A Big CAD Model Dataset For Geometric Deep Learning},
61+ booktitle = {The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
62+ month = {June},
63+ year = {2019}
64+}
65+```
66+
67+Please cite the paper if you use these models. The ABC dataset authors are
68+grateful to [Onshape](https://www.onshape.com/) for providing the CAD models
69+and support.
70+
71+The copyright of the CAD models is owned by their creators; for licensing
72+details see the
73+[Onshape Terms of Use 1.g.ii](https://www.onshape.com/en/legal/terms-of-use#your_content).
74+The dataset authors give no warranties regarding the dataset.
scripts/build.shadded+47−0View file
@@ -0,0 +1,47 @@
1+#!/usr/bin/env bash
2+# Build the GitHub Pages site: download the first STEP chunk of the ABC
3+# dataset, extract the first N models, gzip each file, and write an index.
4+#
5+# Env:
6+# CHUNK_ARCHIVE path to an already-downloaded abc_0000_step_v00.7z
7+# (skips the ~1.6 GB download)
8+# WORK working directory (default: ./build)
9+# N number of models to host (default: 1000)
10+set -euo pipefail
11+
12+REPO_ROOT=$(cd "$(dirname "$0")/.." && pwd)
13+WORK=${WORK:-"$REPO_ROOT/build"}
14+N=${N:-1000}
15+
16+mkdir -p "$WORK"
17+cd "$WORK"
18+
19+if [ -z "${CHUNK_ARCHIVE:-}" ]; then
20+ wget -q https://deep-geometry.github.io/abc-dataset/data/step_v00.txt
21+ CHUNK_URL=$(sed '1q;d' step_v00.txt | awk '{print $1}')
22+ CHUNK_NAME=$(sed '1q;d' step_v00.txt | awk '{print $2}')
23+ echo "Downloading $CHUNK_NAME from $CHUNK_URL ..."
24+ wget -q --no-check-certificate "$CHUNK_URL" -O "$CHUNK_NAME"
25+ CHUNK_ARCHIVE="$WORK/$CHUNK_NAME"
26+fi
27+CHUNK_NAME=$(basename "$CHUNK_ARCHIVE")
28+
29+# Extract only the first N model directories; the full chunk holds 10000
30+# models (~15 GB uncompressed), far more than we need or than CI disk allows.
31+echo "Extracting first $N model directories from $CHUNK_NAME ..."
32+seq -f '%08g/*' 0 $((N - 1)) > include.txt
33+rm -rf extracted
34+7z x -y "$CHUNK_ARCHIVE" -i@include.txt -oextracted > /dev/null
35+
36+rm -rf site
37+mkdir -p site/step
38+find extracted -name '*.step' | sort | head -n "$N" > files.txt
39+echo "Compressing $(wc -l < files.txt) STEP files ..."
40+xargs -a files.txt -P "$(nproc)" -I{} \
41+ sh -c 'gzip -9 -c "$1" > "site/step/$(basename "$1").gz"' _ {}
42+
43+python3 "$REPO_ROOT/scripts/make_index.py" files.txt site "$CHUNK_NAME"
44+cp "$REPO_ROOT/site/index.html" site/
45+
46+echo "Done. Site size:"
47+du -sh site
scripts/make_index.pyadded+38−0View file
@@ -0,0 +1,38 @@
1+#!/usr/bin/env python3
2+"""Write site/index.json listing every hosted file with its download path."""
3+import json
4+import sys
5+from datetime import datetime, timezone
6+from pathlib import Path
7+
8+files_txt, site_dir, chunk_name = sys.argv[1], Path(sys.argv[2]), sys.argv[3]
9+
10+files = []
11+for line in Path(files_txt).read_text().splitlines():
12+ src = Path(line)
13+ gz = site_dir / "step" / (src.name + ".gz")
14+ files.append({
15+ "id": src.name[:8],
16+ "name": src.name,
17+ "path": f"step/{gz.name}",
18+ "stepBytes": src.stat().st_size,
19+ "gzBytes": gz.stat().st_size,
20+ })
21+
22+index = {
23+ "name": "abc-step-1000",
24+ "description": "First 1000 STEP files from the ABC dataset, "
25+ "served gzip-compressed",
26+ "source": "https://deep-geometry.github.io/abc-dataset/",
27+ "chunk": chunk_name,
28+ "generated": datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ"),
29+ "count": len(files),
30+ "totalStepBytes": sum(f["stepBytes"] for f in files),
31+ "totalGzBytes": sum(f["gzBytes"] for f in files),
32+ "files": files,
33+}
34+
35+(site_dir / "index.json").write_text(json.dumps(index, indent=1))
36+print(f"index.json: {len(files)} files, "
37+ f"{index['totalStepBytes'] / 1e9:.2f} GB uncompressed, "
38+ f"{index['totalGzBytes'] / 1e9:.2f} GB compressed")
site/index.htmladded+140−0View file
@@ -0,0 +1,140 @@
1+<!DOCTYPE html>
2+<html lang="en">
3+<head>
4+<meta charset="utf-8">
5+<meta name="viewport" content="width=device-width, initial-scale=1">
6+<title>abc-step-1000 — first 1000 STEP files from the ABC dataset</title>
7+<style>
8+ :root { color-scheme: light dark; }
9+ body {
10+ font-family: system-ui, sans-serif;
11+ max-width: 60rem;
12+ margin: 0 auto;
13+ padding: 1.5rem;
14+ line-height: 1.5;
15+ }
16+ h1 { margin-bottom: 0.2rem; }
17+ .muted { opacity: 0.7; }
18+ code, pre {
19+ font-family: ui-monospace, monospace;
20+ background: rgba(128, 128, 128, 0.12);
21+ border-radius: 4px;
22+ }
23+ code { padding: 0.1em 0.3em; }
24+ pre { padding: 0.7em 1em; overflow-x: auto; }
25+ input#filter {
26+ width: 100%;
27+ box-sizing: border-box;
28+ padding: 0.5em 0.7em;
29+ margin: 1rem 0 0.5rem;
30+ font: inherit;
31+ border: 1px solid rgba(128, 128, 128, 0.5);
32+ border-radius: 6px;
33+ background: transparent;
34+ color: inherit;
35+ }
36+ table { border-collapse: collapse; width: 100%; font-size: 0.9rem; }
37+ th, td {
38+ text-align: left;
39+ padding: 0.3em 0.8em 0.3em 0;
40+ border-bottom: 1px solid rgba(128, 128, 128, 0.25);
41+ white-space: nowrap;
42+ }
43+ td.num, th.num { text-align: right; }
44+ .tablewrap { overflow-x: auto; }
45+ footer { margin-top: 2rem; font-size: 0.85rem; opacity: 0.7; }
46+</style>
47+</head>
48+<body>
49+<h1>abc-step-1000</h1>
50+<p class="muted" id="subtitle">Loading index…</p>
51+
52+<p>The first 1000 STEP files from the
53+<a href="https://deep-geometry.github.io/abc-dataset/">ABC dataset</a>
54+(Koch et al., CVPR 2019), a collection of one million CAD models for
55+geometric deep learning. Files are served gzip-compressed; decompress after
56+downloading:</p>
57+
58+<pre>curl -sL <file-url> | gunzip > model.step</pre>
59+
60+<p>A machine-readable manifest with paths and sizes is at
61+<a href="index.json"><code>index.json</code></a>. See the
62+<a href="https://github.com/concept-collection/abc-step-1000">repository</a>
63+for details, bulk-download recipes, and attribution. The copyright of the CAD
64+models is owned by their creators
65+(<a href="https://www.onshape.com/en/legal/terms-of-use#your_content">Onshape
66+Terms of Use 1.g.ii</a>).</p>
67+
68+<input id="filter" type="search" placeholder="Filter by file name…" hidden>
69+<div class="tablewrap">
70+<table id="files" hidden>
71+ <thead>
72+ <tr>
73+ <th>File</th>
74+ <th class="num">STEP size</th>
75+ <th class="num">Download (.gz)</th>
76+ </tr>
77+ </thead>
78+ <tbody></tbody>
79+</table>
80+</div>
81+
82+<footer>
83+Source: ABC dataset — <em>ABC: A Big CAD Model Dataset For Geometric Deep
84+Learning</em>, Koch, Matveev, Jiang, Williams, Artemov, Burnaev, Alexa,
85+Zorin, Panozzo. CVPR 2019.
86+</footer>
87+
88+<script>
89+function fmtBytes(n) {
90+ if (n >= 1e9) return (n / 1e9).toFixed(2) + ' GB';
91+ if (n >= 1e6) return (n / 1e6).toFixed(1) + ' MB';
92+ return (n / 1e3).toFixed(1) + ' kB';
93+}
94+
95+fetch('index.json').then(r => r.json()).then(index => {
96+ document.getElementById('subtitle').textContent =
97+ `${index.count} files · ${fmtBytes(index.totalGzBytes)} compressed · ` +
98+ `${fmtBytes(index.totalStepBytes)} uncompressed · from ${index.chunk}`;
99+
100+ const tbody = document.querySelector('#files tbody');
101+ for (const f of index.files) {
102+ const tr = document.createElement('tr');
103+
104+ const tdName = document.createElement('td');
105+ tdName.textContent = f.name;
106+ tr.appendChild(tdName);
107+
108+ const tdSize = document.createElement('td');
109+ tdSize.className = 'num';
110+ tdSize.textContent = fmtBytes(f.stepBytes);
111+ tr.appendChild(tdSize);
112+
113+ const tdGz = document.createElement('td');
114+ tdGz.className = 'num';
115+ const a = document.createElement('a');
116+ a.href = f.path;
117+ a.textContent = fmtBytes(f.gzBytes);
118+ tdGz.appendChild(a);
119+ tr.appendChild(tdGz);
120+
121+ tbody.appendChild(tr);
122+ }
123+
124+ const filter = document.getElementById('filter');
125+ filter.addEventListener('input', () => {
126+ const q = filter.value.toLowerCase();
127+ for (const tr of tbody.rows) {
128+ tr.hidden = !tr.cells[0].textContent.toLowerCase().includes(q);
129+ }
130+ });
131+
132+ filter.hidden = false;
133+ document.getElementById('files').hidden = false;
134+}).catch(err => {
135+ document.getElementById('subtitle').textContent =
136+ 'Failed to load index.json: ' + err;
137+});
138+</script>
139+</body>
140+</html>