-
Notifications
You must be signed in to change notification settings - Fork 1
Expand file tree
/
Copy pathindex.html
More file actions
500 lines (468 loc) · 19.9 KB
/
index.html
File metadata and controls
500 lines (468 loc) · 19.9 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
<!DOCTYPE html>
<html>
<head lang="en">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
<meta http-equiv="x-ua-compatible" content="ie=edge" />
<title>DART</title>
<meta name="description" content="" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<meta
property="og:image"
content="./img/teaser.png"
/>
<meta property="og:image:type" content="image/png" />
<meta property="og:image:width" content="1200" />
<meta property="og:image:height" content="630" />
<meta property="og:type" content="website" />
<meta property="og:url" content="https://dart2022.github.io/" />
<meta
property="og:title"
content="DART: Articulated Hand Model with Diverse Accessories and Rich
Textures (NeurIPS 2022)"
/>
<meta
property="og:description"
content="We extend MANO with more Diverse Accessories and Rich Textures,
namely DART. DART is comprised of 325 exquisite hand-crafted texture maps
which vary in appearance and cover different kinds of blemishes, make-ups,
and accessories. We also generate large-scale (800K), diverse, and
high-fidelity hand images, paired with perfect-aligned 3D labels, called
DARTset."
/>
<meta name="twitter:card" content="summary_large_image" />
<meta
name="twitter:title"
content="DART: Articulated Hand Model with Diverse Accessories and Rich
Textures (NeurIPS 2022)"
/>
<meta
name="twitter:description"
content="We extend MANO with more Diverse Accessories and Rich Textures,
namely DART. DART is comprised of 325 exquisite hand-crafted texture maps
which vary in appearance and cover different kinds of blemishes, make-ups,
and accessories. We also generate large-scale (800K), diverse, and
high-fidelity hand images, paired with perfect-aligned 3D labels, called
DARTset."
/>
<meta
name="twitter:image"
content="./img/teaser.png"
/>
<!-- mirror: F0%9F%AA%9E< -->
<link
rel="icon"
href="data:image/svg+xml,<svg xmlns=%22http://www.w3.org/2000/svg%22
viewBox=%220 0 100 100%22><text y=%22.9em%22
font-size=%2290%22>%E2%9C%A8</text></svg>"
/>
<link rel="stylesheet" href="css/bootstrap.min.css" />
<link rel="stylesheet" href="css/font-awesome.min.css" />
<link rel="stylesheet" href="css/codemirror.min.css" />
<link rel="stylesheet" href="css/app.css" />
<script src="js/jquery.min.js"></script>
<script src="js/bootstrap.min.js"></script>
<script src="js/codemirror.min.js"></script>
<script src="js/clipboard.min.js"></script>
<script src="js/video_comparison.js"></script>
<script src="js/app.js"></script>
</head>
<body>
<div class="container" id="header" style="text-align: center; margin: auto">
<div
class="row"
id="title-row"
style="max-width: 100%; margin: 0 auto; display: inline-block">
<h2 class="col-md-12 text-center" id="title">
<b>DART</b>: Articulated Hand Model with <br />
<b>D</b>iverse <b>A</b>ccessories and <b>R</b>ich
<b>T</b>extures<br />
<small> NeurIPS 2022 - Datasets and Benchmarks Track </small>
<br>
</h2>
</div>
<div class="row">
<div class="col-sm-8 col-sm-offset-2 text-center">
<ul class="nav nav-pills nav-justified author">
<li>
<a
style="text-decoration: none;"
href="https://github.com/tomguluson92">Daiheng Gao <sup>*</sup></a>
<br />Alibaba XR Lab<br />
</li>
<li>
<a style="text-decoration: none" href="https://xiuyuliang.cn/">Yuliang
Xiu <sup>*</sup></a> <br />MPI-IS<br />
</li>
<li>
<a style="text-decoration: none" href="https://kailinli.top/#">
Kailin Li <sup>*</sup></a> <br />SJTU MVIG
</li>
<li>
<a style="text-decoration: none" href="https://lixiny.github.io/">
Lixin Yang <sup>*</sup></a> <br />SJTU MVIG
</li>
<br>
<li>
<a style="text-decoration: none"> Feng Wang </a>
<br />Alibaba XR Lab
</li>
<li>
<a style="text-decoration: none"> Peng Zhang </a>
<br />Alibaba XR Lab
</li>
<li>
<a style="text-decoration: none"> Bang Zhang </a>
<br />Alibaba XR Lab
</li>
<li>
<a style="text-decoration: none" href="https://www.mvig.org/">
Cewu Lu
</a>
<br />SJTU MVIG
</li>
<li>
<a
style="text-decoration: none"
href="https://www.cs.sfu.ca/~pingtan/">
Ping Tan
</a>
<br />Simon Fraser University
</li>
</ul>
</div>
</div>
</div>
<br /><br>
<script>
document.getElementById("author-row").style.maxWidth =
document.getElementById("title-row").clientWidth + "px";
</script>
<div class="container" id="main">
<div class="row">
<div class="col-sm-8 col-sm-offset-2 text-center">
<ul class="nav nav-pills nav-justified">
<li>
<a href="https://arxiv.org/abs/2210.07650">
<img src="./img/paper_image.jpg" height="60px" />
<h4><strong>Paper</strong></h4></a>
</li>
<li>
<a href="https://www.youtube.com/embed/kvWqtdLf6hs">
<img src="./img/youtube_icon.png" height="60px" />
<h4><strong>Video</strong></h4></a>
</li>
<li>
<a href="https://github.com/DART2022/DARTset" target="_blank">
<img src="img/github.png" height="60px" />
<h4><strong>Dataloader</strong></h4></a>
</li>
<li>
<a
href="https://huggingface.co/datasets/Yuliang/DART"
target="_blank">
<img src="img/database_icon.png" height="60px" />
<h4><strong>DARTset</strong></h4></a>
</li>
<li>
<a
href="https://drive.google.com/file/d/1HGfTZwwEm-rBeaYDB4d3nntZHIvhaL88/view"
target="_blank">
<img src="img/unity.png" height="60px" />
<h4><strong>Unity GUI</strong></h4></a>
</li>
</ul>
</div>
</div>
<div class="row">
<div class="col-md-8 col-md-offset-2">
<h2>UPDATE</h2>
<ul>
<li>
2024.04.01: DARTset could be easily cloned from Huggingface/Dataset at <a href="https://huggingface.co/datasets/Yuliang/DART">DARTset</a>
</li>
<li>
2022.09.16: DART got accepted by NeurIPS 2022 - Datasets and
Benchmarks Track!
</li>
<li>
2022.09.29: DART's GUI source code publicly available at <a href="https://drive.google.com/file/d/1xtfc-fMHR5ax-e5S5Drx53Rm2ddL5mHs/view?usp=sharing"><strong>Unity GUI source code</strong></a>!
</li>
</ul>
</div>
<div class="row">
<div class="col-md-8 col-md-offset-2">
<h3>Abstract</h3>
<div class="text-justify">
Hand, the bearer of human productivity and intelligence, is
receiving much attention due to the recent fever of 3D digital
avatars. Among different hand morphable models, MANO has been
widely
used in various vision & graphics tasks. However, MANO disregards
textures and accessories, which largely limits its power to
synthesize photorealistic & lifestyle hand data. In this paper, we
extend MANO with more Diverse Accessories and Rich Textures,
namely
<b>DART</b>. <b>DART</b> is comprised of 325 exquisite
hand-crafted
texture maps which varies in appearance, and covers different
kinds
of blemishes, make-ups and accessories. We also provide the Unity
GUI which allows people to render hands with user-specific
settings,
e.g pose, camera, background, lighting, and <b>DART</b>'s
textures.
In this way, we generate large-scale (800K), diverse, and
high-fidelity hand images, paired with perfect-aligned 3D labels,
called <b>DARTset</b>. Experiments demonstrate its superiority in
generalization and diversity. As a great complement for existing
datasets, <b>DARTset</b> could boost hand pose estimation &
surface
reconstruction tasks. <b>DART</b> and Unity software is publicly
available for research purpose.<br /><br /><br />
</div>
</div>
<img
src="img/teaser.png"
class="img-responsive"
alt="overview"
width="64%"
style="max-height: 450px; margin: 30px auto"
/>
<div class="row">
<div class="col-md-8 col-md-offset-2">
<h3>Video</h3>
<div class="text-center">
<div style="position: relative; padding-top: 56.25%">
<iframe
src="https://www.youtube.com/embed/kvWqtdLf6hs"
title="[Neurips 2022] DART demo"
frameborder="0"
allow="accelerometer; autoplay; clipboard-write;
encrypted-media; gyroscope; picture-in-picture"
allowfullscreen
style="
position: absolute;
top: 0;
left: 0;
width: 100%;
height: 100%;
"></iframe>
</div>
</div>
</div>
</div>
<br /><br />
<div class="row">
<div class="col-md-8 col-md-offset-2">
<h3>Comparison between DART and MANO basic topology.</h3>
<div class="text-justify">
In DART, we remould the standard template hand mesh of MANO,
which
has 778 vertices and 1,538 faces, to a wrist-enhanced template
mesh of 842 vertices and 1,666 faces. The pose parameter that
drive the template hand mesh in MANO can be used as a direct
placement for DART without any modifications.
<br /><br />
</div>
<div class="text-center">
<img src="./img/dart_vs_mano.png" width="50%" />
</div>
</div>
</div>
<div class="row">
<div class="col-md-8 col-md-offset-2">
<h3>How to use DART tool in generating your own data?</h3>
<h4>
Step1: click
<a
href="https://drive.google.com/file/d/1iqymPPPSF_rlKbHRvvgaVHlcmPsoEx25/view?usp=sharing">DART
GUI & Code</a>
and download <font color="red">Build_Hand.zip</font>, unzip and
execute <front color="red">Hand.exe</front>.
</h4>
<h4>
Step2: Pose Editing: allow arbitrarily, illumination, accessory,
skin color and other terms.
<br><br>
<td align="left" valign="top" width="50%">
<video id="v2" width="100%" playsinline autoplay loop muted>
<source src="video/step2.mp4" type="video/mp4" />
</video>
</td>
</h4>
<h4>
Step3: Exporting: rendered image with GT(mano pose, 2d/3d joint
are put into the <font color="brown">output.pkl</font>)
<br><br>
<video
id="v2" width="100%" playsinline autoplay loop muted>
<source src="video/step3.mp4" type="video/mp4" />
</video>
</td>
</h4>
<div class="text-justify">
<b>2022.09.22</b> We put the <b>postprocess</b> folder into <a href="https://github.com/DART2022/DARTset/tree/master/postprocess">link</a>.
Please use that code for <b>postprocessing the intermediate output yielded by DART's GUI!</b>
<br /><br />
</div>
</div>
</div>
<div class="row">
<div class="col-md-8 col-md-offset-2">
<h3>DARTset Datasheet & Explanation</h3>
<h4>
Dataset Structure
<ul>
<li>
<b>Rendered Image</b>: with background (<font color="red">384x384</font>),
without background RGBA (<font color="red">512x512</font>).
</li>
<li>
<b>Annotation</b>: 2D/3D positions for 21 keypoints of the
hand, MANO pose, vertex locations.
</li>
<li>
<b>Visualization & Code</b>:
<a>https://github.com/DART2022/DARTset</a>
</li>
<li>
<b>DARTset Download</b>:
<a
href="https://huggingface.co/datasets/Yuliang/DART"
target="_blank">HuggingFace Page (Train & Test)</a>
</li>
</ul>
<p>
<code class="language-plaintext highlighter-rouge">DARTset</code>
is composed of <b>train</b> and <b>test</b>. The folder of each
is described as below.
</p>
<div class="language-plaintext highlighter-rouge">
<div class="highlight text-left">
<pre class="highlight"><code>Train set
* Train: 729,189 single hand frames, 25% of which are wearing accessories.
Test set
* Test (H): 288,77 single hand frames, 25% of which are wearing accessories.
Total set
* DARTset: 758,066 single hand frames. Noteworthy here, we conduct experiments on full 800K DARTset and filter out ~42,000 of images which left wrist unsealed on the final version.
</code></pre>
</div>
</div>
<p>
<code class="language-plaintext highlighter-rouge">Pose sampling</code>
we use spherical linear interpolation (Slerp) in pose and root
rotation sampling. Among these hands, ~25% are assigned an
accessory. In DARTset, basic UV map (skin tones, scars, moles,
tattoos) and accessories are all uniformly sampled, the number
of their corresponding renders are roughly equal.
</p>
</h4>
<h4>
Dataset Creation
<p class="text-justify">
Texture map (4096 x 4096) are all created manually by 3D
artists. GUI and batch data generator is programmed by DART's
authors.
</p>
<ul>
<li>
<b>(a) Pose Sampling</b>: In DART, we totally sampled 800K
pose, 758K of them are valid. For each pose, we adopt
synthetic pose <b>θs</b> from A-MANO and randomply picked
2,000 pose from FreiHand <b>θr</b>, through calculate the
difference between <b>θs</b> and <b>θr</b>, we select the one
<b>θr_max</b> that differs most from <b>θs</b>. Then we
interpolate 8 rotations from <b>θr_max</b> to <b>θs</b> by
spherical linear interpolation (Slerp). For more detail,
please check the paper.
</li>
<li>
<b>(b) Mesh Reconstruction & Rendering</b>: After part (a), we
have <font color="blue">N x (16,3)</font> MANO poses (N=758K).
We input those poses into our revised MANO layer directly.
Through which we can easily get DART mesh (MANO mesh plus
extra wrist) with 842 vertices and 1,666 faces.
</li>
<li>
<b>(c) Rendering</b>: We adopt
<font color="blue">Unity</font> and our revised
<font color="red">SSS skin shader</font> for rendering. During
the rendering pipeline, we randomly select the illumination
(intensity, color and position), texture (including
accessory). After rendering, we have the final image and 2D
reprojected landmarks.
</li>
<li>
<b>(d) Ground Truth Data</b>: At part (a) and (b), we already
gather all 3D related ground truth for monocular 3D pose
estimation & hand tracking yet leave 2D landmark untouched.
After the rendering process (part (c)), we obtain the 2D
landmark and inject it to the ground truth data file. Since
our method in creating DARTset is build on top of the ground
truth annotations, so the ground truth annotations are born
with renderings.
</li>
</ul>
</h4>
<h4>
Considerations for Using the Data
<ul>
<li>
<b>Social Impact</b>: No ethical issues because DART do not
involve any human biological information.
</li>
<li>
<b>Gender Biases</b>: Since our basic mesh topology is FIXED
(we focus more on pose rather than shape), which means the
mesh itself is gender-agnostic. In our data generation
pipeline, both texture maps and accessories in DARTset are
randomly sampled. Considering this, we do not think it may
involve extreme gender bias in DARTset.
</li>
</ul>
</h4>
<h4>
Additional Information
<ul>
<li>
<b>Dataset Curator</b>:
<a>Daiheng Gao, daiheng.gdh@alibaba-inc.com</a>
</li>
<li>
<b>Licensing Information</b>: Codes are MIT license.
GUI(Unity) tools are CC BY-NC 4.0 license. Dataset is CC
BY-NC-ND 4.0 license.
</li>
</ul>
</h4>
</div>
</div>
<div class="row">
<div class="col-md-8 col-md-offset-2">
<h3>Acknowledgements</h3>
<div class="text-justify">
If you find our work useful in your research, please cite:
<div class="language-plaintext highlighter-rouge">
<div class="highlight text-left">
<pre class="highlight"><code>
@inproceedings{gao2022dart,
title={{DART: Articulated Hand Model with Diverse Accessories and Rich Textures}},
author={Daiheng Gao and Yuliang Xiu and Kailin Li and Lixin Yang and Feng Wang and Peng Zhang and Bang Zhang and Cewu Lu and Ping Tan},
booktitle={Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track},
year={2022},
}
</code></pre>
</div>
</div>
</div>
<p class="text-justify">
<br />
The website template was borrowed from
<a href="http://mgharbi.com/">Michaël Gharbi</a>.
</p>
</div>
</div>
</div>
</div>
</body>
</html>