forked from cis-ds/course-site
-
Notifications
You must be signed in to change notification settings - Fork 0
Expand file tree
/
Copy pathblock003_transform-data.html
More file actions
1097 lines (1023 loc) · 58.3 KB
/
block003_transform-data.html
File metadata and controls
1097 lines (1023 loc) · 58.3 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
<!DOCTYPE html>
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta charset="utf-8">
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<meta name="generator" content="pandoc" />
<title>Data transformation</title>
<script src="site_libs/jquery-1.11.3/jquery.min.js"></script>
<meta name="viewport" content="width=device-width, initial-scale=1" />
<link href="site_libs/bootstrap-3.3.5/css/readable.min.css" rel="stylesheet" />
<script src="site_libs/bootstrap-3.3.5/js/bootstrap.min.js"></script>
<script src="site_libs/bootstrap-3.3.5/shim/html5shiv.min.js"></script>
<script src="site_libs/bootstrap-3.3.5/shim/respond.min.js"></script>
<script src="site_libs/jqueryui-1.11.4/jquery-ui.min.js"></script>
<link href="site_libs/tocify-1.9.1/jquery.tocify.css" rel="stylesheet" />
<script src="site_libs/tocify-1.9.1/jquery.tocify.js"></script>
<script src="site_libs/navigation-1.1/tabsets.js"></script>
<script>
(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
})(window,document,'script','https://www.google-analytics.com/analytics.js','ga');
ga('create', 'UA-45631879-2', 'auto');
ga('send', 'pageview');
</script>
<style type="text/css">code{white-space: pre;}</style>
<style type="text/css">
div.sourceCode { overflow-x: auto; }
table.sourceCode, tr.sourceCode, td.lineNumbers, td.sourceCode {
margin: 0; padding: 0; vertical-align: baseline; border: none; }
table.sourceCode { width: 100%; line-height: 100%; }
td.lineNumbers { text-align: right; padding-right: 4px; padding-left: 4px; color: #aaaaaa; border-right: 1px solid #aaaaaa; }
td.sourceCode { padding-left: 5px; }
code > span.kw { color: #007020; font-weight: bold; } /* Keyword */
code > span.dt { color: #902000; } /* DataType */
code > span.dv { color: #40a070; } /* DecVal */
code > span.bn { color: #40a070; } /* BaseN */
code > span.fl { color: #40a070; } /* Float */
code > span.ch { color: #4070a0; } /* Char */
code > span.st { color: #4070a0; } /* String */
code > span.co { color: #60a0b0; font-style: italic; } /* Comment */
code > span.ot { color: #007020; } /* Other */
code > span.al { color: #ff0000; font-weight: bold; } /* Alert */
code > span.fu { color: #06287e; } /* Function */
code > span.er { color: #ff0000; font-weight: bold; } /* Error */
code > span.wa { color: #60a0b0; font-weight: bold; font-style: italic; } /* Warning */
code > span.cn { color: #880000; } /* Constant */
code > span.sc { color: #4070a0; } /* SpecialChar */
code > span.vs { color: #4070a0; } /* VerbatimString */
code > span.ss { color: #bb6688; } /* SpecialString */
code > span.im { } /* Import */
code > span.va { color: #19177c; } /* Variable */
code > span.cf { color: #007020; font-weight: bold; } /* ControlFlow */
code > span.op { color: #666666; } /* Operator */
code > span.bu { } /* BuiltIn */
code > span.ex { } /* Extension */
code > span.pp { color: #bc7a00; } /* Preprocessor */
code > span.at { color: #7d9029; } /* Attribute */
code > span.do { color: #ba2121; font-style: italic; } /* Documentation */
code > span.an { color: #60a0b0; font-weight: bold; font-style: italic; } /* Annotation */
code > span.cv { color: #60a0b0; font-weight: bold; font-style: italic; } /* CommentVar */
code > span.in { color: #60a0b0; font-weight: bold; font-style: italic; } /* Information */
</style>
<style type="text/css">
pre:not([class]) {
background-color: white;
}
</style>
<style type="text/css">
h1 {
font-size: 34px;
}
h1.title {
font-size: 38px;
}
h2 {
font-size: 30px;
}
h3 {
font-size: 24px;
}
h4 {
font-size: 18px;
}
h5 {
font-size: 16px;
}
h6 {
font-size: 12px;
}
.table th:not([align]) {
text-align: left;
}
</style>
</head>
<body>
<style type = "text/css">
.main-container {
max-width: 940px;
margin-left: auto;
margin-right: auto;
}
code {
color: inherit;
background-color: rgba(0, 0, 0, 0.04);
}
img {
max-width:100%;
height: auto;
}
.tabbed-pane {
padding-top: 12px;
}
button.code-folding-btn:focus {
outline: none;
}
</style>
<style type="text/css">
/* padding for bootstrap navbar */
body {
padding-top: 66px;
padding-bottom: 40px;
}
/* offset scroll position for anchor links (for fixed navbar) */
.section h1 {
padding-top: 71px;
margin-top: -71px;
}
.section h2 {
padding-top: 71px;
margin-top: -71px;
}
.section h3 {
padding-top: 71px;
margin-top: -71px;
}
.section h4 {
padding-top: 71px;
margin-top: -71px;
}
.section h5 {
padding-top: 71px;
margin-top: -71px;
}
.section h6 {
padding-top: 71px;
margin-top: -71px;
}
</style>
<script>
// manage active state of menu based on current page
$(document).ready(function () {
// active menu anchor
href = window.location.pathname
href = href.substr(href.lastIndexOf('/') + 1)
if (href === "")
href = "index.html";
var menuAnchor = $('a[href="' + href + '"]');
// mark it active
menuAnchor.parent().addClass('active');
// if it's got a parent navbar menu mark it active as well
menuAnchor.closest('li.dropdown').addClass('active');
});
</script>
<div class="container-fluid main-container">
<!-- tabsets -->
<script>
$(document).ready(function () {
window.buildTabsets("TOC");
});
</script>
<!-- code folding -->
<script>
$(document).ready(function () {
// move toc-ignore selectors from section div to header
$('div.section.toc-ignore')
.removeClass('toc-ignore')
.children('h1,h2,h3,h4,h5').addClass('toc-ignore');
// establish options
var options = {
selectors: "h1,h2,h3",
theme: "bootstrap3",
context: '.toc-content',
hashGenerator: function (text) {
return text.replace(/[.\\/?&!#<>]/g, '').replace(/\s/g, '_').toLowerCase();
},
ignoreSelector: ".toc-ignore",
scrollTo: 0
};
options.showAndHide = true;
options.smoothScroll = true;
// tocify
var toc = $("#TOC").tocify(options).data("toc-tocify");
});
</script>
<style type="text/css">
#TOC {
margin: 25px 0px 20px 0px;
}
@media (max-width: 768px) {
#TOC {
position: relative;
width: 100%;
}
}
.toc-content {
padding-left: 30px;
padding-right: 40px;
}
div.main-container {
max-width: 1200px;
}
div.tocify {
width: 20%;
max-width: 260px;
max-height: 85%;
}
@media (min-width: 768px) and (max-width: 991px) {
div.tocify {
width: 25%;
}
}
@media (max-width: 767px) {
div.tocify {
width: 100%;
max-width: none;
}
}
.tocify ul, .tocify li {
line-height: 20px;
}
.tocify-subheader .tocify-item {
font-size: 0.90em;
padding-left: 25px;
text-indent: 0;
}
.tocify .list-group-item {
border-radius: 0px;
}
</style>
<!-- setup 3col/9col grid for toc_float and main content -->
<div class="row-fluid">
<div class="col-xs-12 col-sm-4 col-md-3">
<div id="TOC" class="tocify">
</div>
</div>
<div class="toc-content col-xs-12 col-sm-8 col-md-9">
<div class="navbar navbar-default navbar-fixed-top" role="navigation">
<div class="container">
<div class="navbar-header">
<button type="button" class="navbar-toggle collapsed" data-toggle="collapse" data-target="#navbar">
<span class="icon-bar"></span>
<span class="icon-bar"></span>
<span class="icon-bar"></span>
</button>
<a class="navbar-brand" href="index.html">Computing for the Social Sciences</a>
</div>
<div id="navbar" class="navbar-collapse collapse">
<ul class="nav navbar-nav">
<li>
<a href="index.html">Home</a>
</li>
<li>
<a href="faq.html">FAQ</a>
</li>
<li>
<a href="syllabus.html">Syllabus</a>
</li>
</ul>
<ul class="nav navbar-nav navbar-right">
</ul>
</div><!--/.nav-collapse -->
</div><!--/.container -->
</div><!--/.navbar -->
<div class="fluid-row" id="header">
<h1 class="title toc-ignore">Data transformation</h1>
</div>
<div id="objectives" class="section level1">
<h1>Objectives</h1>
<ul>
<li>Identify computer programming as a form of problem solving</li>
<li>Practice decomposing an analytical goal into a set of discrete, computational tasks</li>
<li>Identify the verbs for a language of data manipulation</li>
<li>Clarify confusing aspects of data transformation from <a href="http://r4ds.had.co.nz/transform.html"><em>R for Data Science</em></a></li>
<li>Practice implementing <code>dplyr</code> data transformation functions</li>
</ul>
</div>
<div id="computer-programming-as-a-form-of-problem-solving" class="section level1">
<h1>Computer programming as a form of problem solving</h1>
<div class="figure">
<img src="images/xmen_xavier.jpg" alt="Professor X from X-Men (the Patrick Stewart version, not James Mcavoy)" />
<p class="caption">Professor X from <em>X-Men</em> (the Patrick Stewart version, not James Mcavoy)</p>
</div>
<p><a href="https://xkcd.com/722/"><img src="images/xkcd_computer_problems.png" alt="Computer Problems. XKCD." /></a></p>
<p>Computers are not mind-reading machines. They are very efficient at certain tasks, and can perform calculations thousands of times faster than any human. But they are also very dumb: they can only do what you tell them to do. If you are not explicit about what you want the computer to do, or you misspeak and tell the computer to do the wrong thing, it will not correct you.</p>
<p>In order to translate your goal for the program into clear instructions for the computer, you need to break the problem down into a set of smaller, discrete chunks that can be followed by the computer (and also by yourself/other humans).</p>
<div id="decomposing-problems-using-diamonds" class="section level2">
<h2>Decomposing problems using <code>diamonds</code></h2>
<div class="sourceCode"><pre class="sourceCode r"><code class="sourceCode r"><span class="kw">library</span>(tidyverse)</code></pre></div>
<pre><code>## Loading tidyverse: ggplot2
## Loading tidyverse: tibble
## Loading tidyverse: tidyr
## Loading tidyverse: readr
## Loading tidyverse: purrr
## Loading tidyverse: dplyr</code></pre>
<pre><code>## Conflicts with tidy packages ----------------------------------------------</code></pre>
<pre><code>## filter(): dplyr, stats
## lag(): dplyr, stats</code></pre>
<div class="sourceCode"><pre class="sourceCode r"><code class="sourceCode r"><span class="kw">str</span>(diamonds)</code></pre></div>
<pre><code>## Classes 'tbl_df', 'tbl' and 'data.frame': 53940 obs. of 10 variables:
## $ carat : num 0.23 0.21 0.23 0.29 0.31 0.24 0.24 0.26 0.22 0.23 ...
## $ cut : Ord.factor w/ 5 levels "Fair"<"Good"<..: 5 4 2 4 2 3 3 3 1 3 ...
## $ color : Ord.factor w/ 7 levels "D"<"E"<"F"<"G"<..: 2 2 2 6 7 7 6 5 2 5 ...
## $ clarity: Ord.factor w/ 8 levels "I1"<"SI2"<"SI1"<..: 2 3 5 4 2 6 7 3 4 5 ...
## $ depth : num 61.5 59.8 56.9 62.4 63.3 62.8 62.3 61.9 65.1 59.4 ...
## $ table : num 55 61 65 58 58 57 57 55 61 61 ...
## $ price : int 326 326 327 334 335 336 336 337 337 338 ...
## $ x : num 3.95 3.89 4.05 4.2 4.34 3.94 3.95 4.07 3.87 4 ...
## $ y : num 3.98 3.84 4.07 4.23 4.35 3.96 3.98 4.11 3.78 4.05 ...
## $ z : num 2.43 2.31 2.31 2.63 2.75 2.48 2.47 2.53 2.49 2.39 ...</code></pre>
<p>The <code>diamonds</code> dataset contains prices and other attributes of almost 54,000 diamonds. Let’s answer the following questions by <strong>decomposing</strong> the problem into a series of discrete steps we can tell R to follow.</p>
</div>
<div id="what-is-the-average-price-of-an-ideal-cut-diamond" class="section level2">
<h2>What is the average price of an ideal cut diamond?</h2>
<p>Let’s think about what we need to have the computer do to answer this question:</p>
<ol style="list-style-type: decimal">
<li>First we need to identify the <strong>input</strong>, or the data we’re going to analyze.</li>
<li>Next we need to select only the observations which are ideal cut diamonds.</li>
<li>Finally we need to calculate the average value, or <strong>mean</strong>, of price.</li>
</ol>
<p>Here’s how we tell the computer to do this:</p>
<div class="sourceCode"><pre class="sourceCode r"><code class="sourceCode r"><span class="kw">data</span>(<span class="st">"diamonds"</span>)
diamonds_ideal <-<span class="st"> </span><span class="kw">filter</span>(diamonds, cut ==<span class="st"> "Ideal"</span>)
<span class="kw">summarize</span>(diamonds_ideal, <span class="dt">avg_price =</span> <span class="kw">mean</span>(price))</code></pre></div>
<pre><code>## # A tibble: 1 × 1
## avg_price
## <dbl>
## 1 3457.542</code></pre>
<p>The first line of code copies the <code>diamonds</code> data frame from the hard drive into memory so we can actively work with it. The second line creates a new data frame called <code>diamonds_ideal</code> that only contains the observations in <code>diamonds</code> which are ideal cut diamonds. The third line summarizes the new data frame and calculates the mean value for the <code>price</code> variable.</p>
</div>
<div id="what-is-the-average-price-of-a-diamond-for-each-cut" class="section level2">
<h2>What is the average price of a diamond for each cut?</h2>
<p><strong>Exercise: decompose the question into a discrete set of tasks to complete using R.</strong></p>
<details> <summary>Click for the solution</summary>
<p>
<ol style="list-style-type: decimal">
<li>First we need to identify the <strong>input</strong>, or the data we’re going to analyze.</li>
<li>Next we need to group the observations together by their value for <code>cut</code>, so we can make separate calculations for each category.</li>
<li>Finally we need to calculate the average value, or <strong>mean</strong>, of price for each cut of diamond.</li>
</ol>
<p>Here’s how we tell the computer to do this:</p>
<div class="sourceCode"><pre class="sourceCode r"><code class="sourceCode r"><span class="kw">data</span>(<span class="st">"diamonds"</span>)
diamonds_cut <-<span class="st"> </span><span class="kw">group_by</span>(diamonds, cut)
<span class="kw">summarize</span>(diamonds_cut, <span class="dt">avg_price =</span> <span class="kw">mean</span>(price))</code></pre></div>
<pre><code>## # A tibble: 5 × 2
## cut avg_price
## <ord> <dbl>
## 1 Fair 4358.758
## 2 Good 3928.864
## 3 Very Good 3981.760
## 4 Premium 4584.258
## 5 Ideal 3457.542</code></pre>
</p>
<p></details></p>
</div>
<div id="what-is-the-average-carat-size-and-price-for-each-cut-of-i-colored-diamonds" class="section level2">
<h2>What is the average carat size and price for each cut of “I” colored diamonds?</h2>
<p><strong>Exercise: decompose the question into a discrete set of tasks to complete using R.</strong></p>
<details> <summary>Click for the solution</summary>
<p>
<ol style="list-style-type: decimal">
<li>Use <code>diamonds</code> as the input</li>
<li>Filter <code>diamonds</code> to only keep observations where the color is rated as “I”</li>
<li>Group the filtered <code>diamonds</code> data frame by cut</li>
<li>Summarize the grouped and filtered <code>diamonds</code> data frame by calculating the average carat size and price</li>
</ol>
<div class="sourceCode"><pre class="sourceCode r"><code class="sourceCode r"><span class="kw">data</span>(<span class="st">"diamonds"</span>)
diamonds_i <-<span class="st"> </span><span class="kw">filter</span>(diamonds, color ==<span class="st"> "I"</span>)
diamonds_i_group <-<span class="st"> </span><span class="kw">group_by</span>(diamonds_i, cut)
<span class="kw">summarize</span>(
diamonds_i_group,
<span class="dt">carat =</span> <span class="kw">mean</span>(carat),
<span class="dt">price =</span> <span class="kw">mean</span>(price)
)</code></pre></div>
<pre><code>## # A tibble: 5 × 3
## cut carat price
## <ord> <dbl> <dbl>
## 1 Fair 1.1980571 4685.446
## 2 Good 1.0572222 5078.533
## 3 Very Good 1.0469518 5255.880
## 4 Premium 1.1449370 5946.181
## 5 Ideal 0.9130291 4451.970</code></pre>
</p>
<p></details></p>
</div>
</div>
<div id="verbiage-for-data-transformation" class="section level1">
<h1>Verbiage for data transformation</h1>
<div class="figure">
<img src="images/data-science.png" alt="Data science workflow" />
<p class="caption">Data science workflow</p>
</div>
<p>Above we practiced decomposing problems that required data transformation steps. Rarely will your data arrive in exactly the form you require in order to analyze it appropriately. As part of the data science workflow you will need to <strong>transform</strong> your data in order to analyze it. Just as we established a syntax for generating graphics (the <strong>layered grammar of graphics</strong>), so to will we have a syntax for data transformation.</p>
<p>From the same author of <code>ggplot2</code>, I give you <code>dplyr</code>! This package contains useful functions for transforming and manipulating data frames, the bread-and-butter format for data in R. These functions can be thought of as <strong>verbs</strong>. The noun is the data, and the verb is acting on the noun. All of the <code>dplyr</code> verbs (and in fact all the verbs in the wider <code>tidyverse</code>) work similarly:</p>
<ol style="list-style-type: decimal">
<li>The first argument is a data frame</li>
<li>Subsequent arguments describe what to do with the data frame</li>
<li>The result is a new data frame</li>
</ol>
</div>
<div id="key-functions-in-dplyr" class="section level1">
<h1>Key functions in <code>dplyr</code></h1>
<table>
<colgroup>
<col width="20%" />
<col width="79%" />
</colgroup>
<thead>
<tr class="header">
<th><code>function()</code></th>
<th>Action performed</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td><code>filter()</code></td>
<td>Subsets observations based on their values</td>
</tr>
<tr class="even">
<td><code>arrange()</code></td>
<td>Changes the order of observations based on their values</td>
</tr>
<tr class="odd">
<td><code>select()</code></td>
<td>Selects a subset of columns from the data frame</td>
</tr>
<tr class="even">
<td><code>rename()</code></td>
<td>Changes the name of columns in the data frame</td>
</tr>
<tr class="odd">
<td><code>mutate()</code></td>
<td>Creates new columns (or variables)</td>
</tr>
<tr class="even">
<td><code>group_by()</code></td>
<td>Changes the unit of analysis from the complete dataset to individual groups</td>
</tr>
<tr class="odd">
<td><code>summarize()</code></td>
<td>Collapses the data frame to a smaller number of rows which summarize the larger data</td>
</tr>
</tbody>
</table>
<p>These are the basic verbs you will use to transform your data. By combining them together, you can perform powerful data manipulation tasks.</p>
<div id="american-vs.british-english" class="section level2">
<h2>American vs. British English</h2>
<p>Hadley Wickham is from New Zealand. As such he (and base R) favours British spellings.</p>
<blockquote class="twitter-tweet" data-lang="en">
<p lang="en" dir="ltr">
The holy grail: “For consistency, aim to use British (rather than American) spelling.” <a href="https://twitter.com/hashtag/rstats?src=hash">#rstats</a> <a href="http://t.co/7qQSWIowcl">http://t.co/7qQSWIowcl</a>. Colour is right!
</p>
— Hadley Wickham (<span class="citation">@hadleywickham</span>) <a href="https://twitter.com/hadleywickham/status/405707093770244097">November 27, 2013</a>
</blockquote>
<script async src="http://platform.twitter.com/widgets.js" charset="utf-8"></script>
<p>While British spelling is preferred, let us rally around our new leader:</p>
<blockquote class="twitter-tweet" data-lang="en">
<p lang="en" dir="ltr">
We have to make America great again!
</p>
— Donald J. Trump (<span class="citation">@realDonaldTrump</span>) <a href="https://twitter.com/realDonaldTrump/status/266254611919282177">November 7, 2012</a>
</blockquote>
<script async src="http://platform.twitter.com/widgets.js" charset="utf-8"></script>
<p>Therefore many R functions can be written using American or British variants:</p>
<ul>
<li><code>summarize()</code> = <code>summarise()</code></li>
<li><code>color()</code> = <code>colour()</code></li>
</ul>
</div>
<div id="saving-transformed-data" class="section level2">
<h2>Saving transformed data</h2>
<p><code>dplyr</code> never overwrites existing data. If you want a copy of the transformed data for later use in the program, you need to explicitly save it. You can do this by using the assignment operator <code><-</code>:</p>
<div class="sourceCode"><pre class="sourceCode r"><code class="sourceCode r"><span class="kw">filter</span>(diamonds, cut ==<span class="st"> "Ideal"</span>) <span class="co"># printed, but not saved</span></code></pre></div>
<pre><code>## # A tibble: 21,551 × 10
## carat cut color clarity depth table price x y z
## <dbl> <ord> <ord> <ord> <dbl> <dbl> <int> <dbl> <dbl> <dbl>
## 1 0.23 Ideal E SI2 61.5 55 326 3.95 3.98 2.43
## 2 0.23 Ideal J VS1 62.8 56 340 3.93 3.90 2.46
## 3 0.31 Ideal J SI2 62.2 54 344 4.35 4.37 2.71
## 4 0.30 Ideal I SI2 62.0 54 348 4.31 4.34 2.68
## 5 0.33 Ideal I SI2 61.8 55 403 4.49 4.51 2.78
## 6 0.33 Ideal I SI2 61.2 56 403 4.49 4.50 2.75
## 7 0.33 Ideal J SI1 61.1 56 403 4.49 4.55 2.76
## 8 0.23 Ideal G VS1 61.9 54 404 3.93 3.95 2.44
## 9 0.32 Ideal I SI1 60.9 55 404 4.45 4.48 2.72
## 10 0.30 Ideal I SI2 61.0 59 405 4.30 4.33 2.63
## # ... with 21,541 more rows</code></pre>
<div class="sourceCode"><pre class="sourceCode r"><code class="sourceCode r">diamonds_ideal <-<span class="st"> </span><span class="kw">filter</span>(diamonds, cut ==<span class="st"> "Ideal"</span>) <span class="co"># saved, but not printed</span>
(diamonds_ideal <-<span class="st"> </span><span class="kw">filter</span>(diamonds, cut ==<span class="st"> "Ideal"</span>)) <span class="co"># saved and printed</span></code></pre></div>
<pre><code>## # A tibble: 21,551 × 10
## carat cut color clarity depth table price x y z
## <dbl> <ord> <ord> <ord> <dbl> <dbl> <int> <dbl> <dbl> <dbl>
## 1 0.23 Ideal E SI2 61.5 55 326 3.95 3.98 2.43
## 2 0.23 Ideal J VS1 62.8 56 340 3.93 3.90 2.46
## 3 0.31 Ideal J SI2 62.2 54 344 4.35 4.37 2.71
## 4 0.30 Ideal I SI2 62.0 54 348 4.31 4.34 2.68
## 5 0.33 Ideal I SI2 61.8 55 403 4.49 4.51 2.78
## 6 0.33 Ideal I SI2 61.2 56 403 4.49 4.50 2.75
## 7 0.33 Ideal J SI1 61.1 56 403 4.49 4.55 2.76
## 8 0.23 Ideal G VS1 61.9 54 404 3.93 3.95 2.44
## 9 0.32 Ideal I SI1 60.9 55 404 4.45 4.48 2.72
## 10 0.30 Ideal I SI2 61.0 59 405 4.30 4.33 2.63
## # ... with 21,541 more rows</code></pre>
<blockquote>
<p>Do not use <code>=</code> to assign objects. <a href="http://stackoverflow.com/a/1742550">Read this for more information on the difference between <code><-</code> and <code>=</code>.</a></p>
</blockquote>
</div>
<div id="comparisons" class="section level2">
<h2>Comparisons</h2>
<table>
<thead>
<tr class="header">
<th>Symbol</th>
<th>Action</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td><code>></code></td>
<td>greater than</td>
</tr>
<tr class="even">
<td><code>>=</code></td>
<td>greater than or equal to</td>
</tr>
<tr class="odd">
<td><code><</code></td>
<td>less than</td>
</tr>
<tr class="even">
<td><code><=</code></td>
<td>less than or equal to</td>
</tr>
<tr class="odd">
<td><code>!=</code></td>
<td>not equal</td>
</tr>
<tr class="even">
<td><strong><code>==</code></strong></td>
<td><strong>equal</strong></td>
</tr>
</tbody>
</table>
<p>Don’t confuse <code>=</code> and <code>==</code>. <code>=</code> is used within function calls, whereas <code>==</code> is used to compare two objects or values.</p>
</div>
<div id="logical-operators" class="section level2">
<h2>Logical operators</h2>
<p>Multiple arguments to <code>filter()</code> are combined with “and”: the function checks for every condition to be true in order to be included in the output. For other types, we use Boolean notation.</p>
<table>
<thead>
<tr class="header">
<th>Symbol</th>
<th>Action</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td><code>&</code></td>
<td>and</td>
</tr>
<tr class="even">
<td><code>|</code></td>
<td>or</td>
</tr>
<tr class="odd">
<td><code>!</code></td>
<td>not</td>
</tr>
</tbody>
</table>
<div class="figure">
<img src="http://r4ds.had.co.nz/diagrams/transform-logical.png" alt="Complete set of boolean operations [@hadley2016]" />
<p class="caption">Complete set of boolean operations <span class="citation">[@hadley2016]</span></p>
</div>
<p>For example, using the <code>flights</code> data demonstrated in <a href="http://r4ds.had.co.nz/transform.html">Chapter 5 of R for Data Science</a>, to find all flights that departed in January or February:</p>
<div class="sourceCode"><pre class="sourceCode r"><code class="sourceCode r"><span class="kw">library</span>(nycflights13)
<span class="kw">filter</span>(flights, month ==<span class="st"> </span><span class="dv">1</span> |<span class="st"> </span>month ==<span class="st"> </span><span class="dv">2</span>)</code></pre></div>
<pre><code>## # A tibble: 51,955 × 19
## year month day dep_time sched_dep_time dep_delay arr_time
## <int> <int> <int> <int> <int> <dbl> <int>
## 1 2013 1 1 517 515 2 830
## 2 2013 1 1 533 529 4 850
## 3 2013 1 1 542 540 2 923
## 4 2013 1 1 544 545 -1 1004
## 5 2013 1 1 554 600 -6 812
## 6 2013 1 1 554 558 -4 740
## 7 2013 1 1 555 600 -5 913
## 8 2013 1 1 557 600 -3 709
## 9 2013 1 1 557 600 -3 838
## 10 2013 1 1 558 600 -2 753
## # ... with 51,945 more rows, and 12 more variables: sched_arr_time <int>,
## # arr_delay <dbl>, carrier <chr>, flight <int>, tailnum <chr>,
## # origin <chr>, dest <chr>, air_time <dbl>, distance <dbl>, hour <dbl>,
## # minute <dbl>, time_hour <dttm></code></pre>
<p>Do not try:</p>
<div class="sourceCode"><pre class="sourceCode r"><code class="sourceCode r"><span class="kw">filter</span>(flights, month ==<span class="st"> </span><span class="dv">1</span> |<span class="st"> </span><span class="dv">2</span>)</code></pre></div>
<p><a href="http://r4ds.had.co.nz/transform.html#logical-operators">This will not work.</a></p>
</div>
<div id="missing-values" class="section level2">
<h2>Missing values</h2>
<p><code>NA</code> represents an unknown value. Missing values are contagious, in that their properties will transfer to any operation performed on it.</p>
<div class="sourceCode"><pre class="sourceCode r"><code class="sourceCode r"><span class="ot">NA</span> ><span class="st"> </span><span class="dv">5</span></code></pre></div>
<pre><code>## [1] NA</code></pre>
<div class="sourceCode"><pre class="sourceCode r"><code class="sourceCode r"><span class="dv">10</span> ==<span class="st"> </span><span class="ot">NA</span></code></pre></div>
<pre><code>## [1] NA</code></pre>
<div class="sourceCode"><pre class="sourceCode r"><code class="sourceCode r"><span class="ot">NA</span> +<span class="st"> </span><span class="dv">10</span></code></pre></div>
<pre><code>## [1] NA</code></pre>
<p>To determine if a value is missing, use the <code>is.na()</code> function.</p>
<p>When filtering, you must explicitly call for missing values to be returned.</p>
<div class="sourceCode"><pre class="sourceCode r"><code class="sourceCode r">df <-<span class="st"> </span><span class="kw">tibble</span>(<span class="dt">x =</span> <span class="kw">c</span>(<span class="dv">1</span>, <span class="ot">NA</span>, <span class="dv">3</span>))
<span class="kw">filter</span>(df, x ><span class="st"> </span><span class="dv">1</span>)</code></pre></div>
<pre><code>## # A tibble: 1 × 1
## x
## <dbl>
## 1 3</code></pre>
<div class="sourceCode"><pre class="sourceCode r"><code class="sourceCode r"><span class="kw">filter</span>(df, <span class="kw">is.na</span>(x) |<span class="st"> </span>x ><span class="st"> </span><span class="dv">1</span>)</code></pre></div>
<pre><code>## # A tibble: 2 × 1
## x
## <dbl>
## 1 NA
## 2 3</code></pre>
<p>Or when calculating summary statistics, you need to explicitly ignore missing values.</p>
<div class="sourceCode"><pre class="sourceCode r"><code class="sourceCode r">df <-<span class="st"> </span><span class="kw">tibble</span>(
<span class="dt">x =</span> <span class="kw">c</span>(<span class="dv">1</span>, <span class="dv">2</span>, <span class="dv">3</span>, <span class="dv">5</span>, <span class="ot">NA</span>)
)
<span class="kw">summarize</span>(df, <span class="dt">meanx =</span> <span class="kw">mean</span>(x))</code></pre></div>
<pre><code>## # A tibble: 1 × 1
## meanx
## <dbl>
## 1 NA</code></pre>
<div class="sourceCode"><pre class="sourceCode r"><code class="sourceCode r"><span class="kw">summarize</span>(df, <span class="dt">meanx =</span> <span class="kw">mean</span>(x, <span class="dt">na.rm =</span> <span class="ot">TRUE</span>))</code></pre></div>
<pre><code>## # A tibble: 1 × 1
## meanx
## <dbl>
## 1 2.75</code></pre>
</div>
<div id="piping" class="section level2">
<h2>Piping</h2>
<p>As we discussed, frequently you need to perform a series of intermediate steps to transform data for analysis. If we write each step as a discrete command and store their contents as new objects, it can be convoluted.</p>
<p>Drawing on <a href="http://r4ds.had.co.nz/transform.html#combining-multiple-operations-with-the-pipe">this example from <em>R for Data Science</em></a>, let’s explore the relationship between the distance and average delay for each location. At this point, we would write it something like this:</p>
<div class="sourceCode"><pre class="sourceCode r"><code class="sourceCode r">by_dest <-<span class="st"> </span><span class="kw">group_by</span>(flights, dest)
delay <-<span class="st"> </span><span class="kw">summarise</span>(by_dest,
<span class="dt">count =</span> <span class="kw">n</span>(),
<span class="dt">dist =</span> <span class="kw">mean</span>(distance, <span class="dt">na.rm =</span> <span class="ot">TRUE</span>),
<span class="dt">delay =</span> <span class="kw">mean</span>(arr_delay, <span class="dt">na.rm =</span> <span class="ot">TRUE</span>)
)
delay <-<span class="st"> </span><span class="kw">filter</span>(delay, count ><span class="st"> </span><span class="dv">20</span>, dest !=<span class="st"> "HNL"</span>)
<span class="kw">ggplot</span>(<span class="dt">data =</span> delay, <span class="dt">mapping =</span> <span class="kw">aes</span>(<span class="dt">x =</span> dist, <span class="dt">y =</span> delay)) +
<span class="st"> </span><span class="kw">geom_point</span>(<span class="kw">aes</span>(<span class="dt">size =</span> count), <span class="dt">alpha =</span> <span class="dv">1</span>/<span class="dv">3</span>) +
<span class="st"> </span><span class="kw">geom_smooth</span>(<span class="dt">se =</span> <span class="ot">FALSE</span>)</code></pre></div>
<pre><code>## `geom_smooth()` using method = 'loess'</code></pre>
<p><img src="block003_transform-data_files/figure-html/unnamed-chunk-11-1.png" width="672" /></p>
<p>Decomposing the problem, there are three basic steps:</p>
<ol style="list-style-type: decimal">
<li>Group <code>flights</code> by destination.</li>
<li>Summarize to compute distance, average delay, and number of flights.</li>
<li>Filter to remove noisy points and the Honolulu airport, which is almost twice as far away as the next closest airport.</li>
</ol>
<p>The code as written is inefficient because we have to name each intermediate data frame, even though we don’t care about them. It also provides more opportunities for typos and errors.</p>
<p>Because all <code>dplyr</code> verbs follow the same syntax (data first, then options for the function), we can use the pipe operator <code>%>%</code> to chain a series of functions together in one command:</p>
<div class="sourceCode"><pre class="sourceCode r"><code class="sourceCode r">delays <-<span class="st"> </span>flights %>%<span class="st"> </span>
<span class="st"> </span><span class="kw">group_by</span>(dest) %>%<span class="st"> </span>
<span class="st"> </span><span class="kw">summarize</span>(
<span class="dt">count =</span> <span class="kw">n</span>(),
<span class="dt">dist =</span> <span class="kw">mean</span>(distance, <span class="dt">na.rm =</span> <span class="ot">TRUE</span>),
<span class="dt">delay =</span> <span class="kw">mean</span>(arr_delay, <span class="dt">na.rm =</span> <span class="ot">TRUE</span>)
) %>%<span class="st"> </span>
<span class="st"> </span><span class="kw">filter</span>(count ><span class="st"> </span><span class="dv">20</span>, dest !=<span class="st"> "HNL"</span>)</code></pre></div>
<p>Now, we don’t have to name each intermediate step and store them as data frames. We only store a single data frame (<code>delays</code>) which contains the final version of the transformed data frame. We could read this code as use the <code>flights</code> data, then group by destination, then summarize for each destination the number of flights, the average disance, and the average delay, then subset only the destinations with at least 20 flights and exclude Honolulu.</p>
<div id="things-not-to-do-with-piping" class="section level3">
<h3>Things not to do with piping</h3>
<p>Remember that with pipes, we don’t have to save all of our intermediate steps. We only use one assignment, like this:</p>
<div class="sourceCode"><pre class="sourceCode r"><code class="sourceCode r">delays <-<span class="st"> </span>flights %>%<span class="st"> </span>
<span class="st"> </span><span class="kw">group_by</span>(dest) %>%<span class="st"> </span>
<span class="st"> </span><span class="kw">summarize</span>(
<span class="dt">count =</span> <span class="kw">n</span>(),
<span class="dt">dist =</span> <span class="kw">mean</span>(distance, <span class="dt">na.rm =</span> <span class="ot">TRUE</span>),
<span class="dt">delay =</span> <span class="kw">mean</span>(arr_delay, <span class="dt">na.rm =</span> <span class="ot">TRUE</span>)
) %>%<span class="st"> </span>
<span class="st"> </span><span class="kw">filter</span>(count ><span class="st"> </span><span class="dv">20</span>, dest !=<span class="st"> "HNL"</span>)</code></pre></div>
<p>Do not do this:</p>
<div class="sourceCode"><pre class="sourceCode r"><code class="sourceCode r">delays <-<span class="st"> </span>flights %>%<span class="st"> </span>
<span class="st"> </span>by_dest <-<span class="st"> </span><span class="kw">group_by</span>(dest) %>%<span class="st"> </span>
<span class="st"> </span>delay <-<span class="st"> </span><span class="kw">summarize</span>(
<span class="dt">count =</span> <span class="kw">n</span>(),
<span class="dt">dist =</span> <span class="kw">mean</span>(distance, <span class="dt">na.rm =</span> <span class="ot">TRUE</span>),
<span class="dt">delay =</span> <span class="kw">mean</span>(arr_delay, <span class="dt">na.rm =</span> <span class="ot">TRUE</span>)
) %>%<span class="st"> </span>
<span class="st"> </span>delay <-<span class="st"> </span><span class="kw">filter</span>(count ><span class="st"> </span><span class="dv">20</span>, dest !=<span class="st"> "HNL"</span>)</code></pre></div>
<pre><code>Error: bad assignment:
summarize(count = n(), dist = mean(distance, na.rm = TRUE), delay = mean(arr_delay,
na.rm = TRUE)) %>% delay <- filter(count > 20, dest != "HNL")</code></pre>
<p>Or this:</p>
<div class="sourceCode"><pre class="sourceCode r"><code class="sourceCode r">delays <-<span class="st"> </span>flights %>%<span class="st"> </span>
<span class="st"> </span><span class="kw">group_by</span>(flights, dest) %>%<span class="st"> </span>
<span class="st"> </span><span class="kw">summarize</span>(flights,
<span class="dt">count =</span> <span class="kw">n</span>(),
<span class="dt">dist =</span> <span class="kw">mean</span>(distance, <span class="dt">na.rm =</span> <span class="ot">TRUE</span>),
<span class="dt">delay =</span> <span class="kw">mean</span>(arr_delay, <span class="dt">na.rm =</span> <span class="ot">TRUE</span>)
) %>%<span class="st"> </span>
<span class="st"> </span><span class="kw">filter</span>(flights, count ><span class="st"> </span><span class="dv">20</span>, dest !=<span class="st"> "HNL"</span>)</code></pre></div>
<pre><code>## Error in eval(expr, envir, enclos): unknown variable to group by : flights</code></pre>
<p>If you use pipes, you don’t have to reference the data frame with each function - just the first time at the beginning of the pipe sequence.</p>
</div>
</div>
</div>
<div id="resources-for-transforming-data" class="section level1">
<h1>Resources for transforming data</h1>
<p><a href="https://www.rstudio.com/wp-content/uploads/2015/02/data-wrangling-cheatsheet.pdf"><strong>Data Wrangling with <code>dplyr</code> and <code>tidyr</code> Cheat Sheet</strong></a> - print this cheatsheet off! It is a great guide for implementing functions from <code>dplyr</code> (and <code>tidyr</code>, to be introduced next week).</p>
</div>
<div id="transforming-college-education" class="section level1">
<h1>Transforming college education</h1>
<p>The Department of Education collects <a href="https://collegescorecard.ed.gov/">annual statistics on colleges and universities in the United States</a>. I have included a subset of this data from 2013 in the <a href="https://github.com/uc-cfss/rcfss"><code>rcfss</code></a> library from GitHub. To install the package, run the command <code>devtools::install_github("uc-cfss/rcfss")</code> in the console.</p>
<blockquote>
<p>If you don’t already have the <code>devtools</code> library installed, you will get an error. Go back and install this first using <code>install.packages("devtools")</code>, then run <code>devtools::install_github("uc-cfss/rcfss")</code>.</p>
</blockquote>
<div class="sourceCode"><pre class="sourceCode r"><code class="sourceCode r"><span class="kw">library</span>(rcfss)
<span class="kw">data</span>(<span class="st">"scorecard"</span>)
scorecard</code></pre></div>
<pre><code>## # A tibble: 1,849 × 12
## unitid name state type
## <int> <chr> <chr> <chr>
## 1 450234 ITT Technical Institute-Wichita KS Private, for-profit
## 2 448479 ITT Technical Institute-Swartz Creek MI Private, for-profit
## 3 456427 ITT Technical Institute-Concord CA Private, for-profit
## 4 459596 ITT Technical Institute-Tallahassee FL Private, for-profit
## 5 459851 Herzing University-Brookfield WI Private, for-profit
## 6 482477 DeVry University-Illinois IL Private, for-profit
## 7 482547 DeVry University-Nevada NV Private, for-profit
## 8 482592 DeVry University-Oregon OR Private, for-profit
## 9 482617 DeVry University-Tennessee TN Private, for-profit
## 10 482662 DeVry University-Washington WA Private, for-profit
## # ... with 1,839 more rows, and 8 more variables: cost <int>,
## # admrate <dbl>, satavg <dbl>, avgfacsal <dbl>, pctpell <dbl>,
## # comprate <dbl>, firstgen <dbl>, debt <dbl></code></pre>
<p>Type <code>?scorecard</code> in the console to open up the help file for this data set. This includes the documentation for all the variables. Use your knowledge of the <code>dplyr</code> functions to answer the following questions.</p>
<div id="which-schools-have-a-greater-than-40-share-of-first-generation-students" class="section level3">
<h3>Which schools have a greater than 40% share of first-generation students?</h3>
<details> <summary>Click for the solution</summary>
<p>
<div class="sourceCode"><pre class="sourceCode r"><code class="sourceCode r"><span class="kw">filter</span>(scorecard, firstgen ><span class="st"> </span>.<span class="dv">40</span>)</code></pre></div>
<pre><code>## # A tibble: 578 × 12
## unitid name state type
## <int> <chr> <chr> <chr>
## 1 450234 ITT Technical Institute-Wichita KS Private, for-profit
## 2 448479 ITT Technical Institute-Swartz Creek MI Private, for-profit
## 3 456427 ITT Technical Institute-Concord CA Private, for-profit
## 4 459596 ITT Technical Institute-Tallahassee FL Private, for-profit
## 5 459851 Herzing University-Brookfield WI Private, for-profit
## 6 482477 DeVry University-Illinois IL Private, for-profit
## 7 482547 DeVry University-Nevada NV Private, for-profit
## 8 482592 DeVry University-Oregon OR Private, for-profit
## 9 482617 DeVry University-Tennessee TN Private, for-profit
## 10 482662 DeVry University-Washington WA Private, for-profit
## # ... with 568 more rows, and 8 more variables: cost <int>, admrate <dbl>,
## # satavg <dbl>, avgfacsal <dbl>, pctpell <dbl>, comprate <dbl>,
## # firstgen <dbl>, debt <dbl></code></pre>
</p>
<p></details></p>
</div>
<div id="which-were-the-10-most-expensive-colleges-in-2013" class="section level3">
<h3>Which were the 10 most expensive colleges in 2013?</h3>
<details> <summary>Click for the solution</summary>
<p>
<div class="sourceCode"><pre class="sourceCode r"><code class="sourceCode r"><span class="kw">arrange</span>(scorecard, <span class="kw">desc</span>(cost))</code></pre></div>
<pre><code>## # A tibble: 1,849 × 12
## unitid name state
## <int> <chr> <chr>
## 1 195304 Sarah Lawrence College NY
## 2 179867 Washington University in St Louis MO
## 3 144050 University of Chicago IL
## 4 190150 Columbia University in the City of New York NY
## 5 182670 Dartmouth College NH
## 6 130697 Wesleyan University CT
## 7 147767 Northwestern University IL
## 8 120254 Occidental College CA
## 9 115409 Harvey Mudd College CA
## 10 230816 Bennington College VT
## # ... with 1,839 more rows, and 9 more variables: type <chr>, cost <int>,
## # admrate <dbl>, satavg <dbl>, avgfacsal <dbl>, pctpell <dbl>,
## # comprate <dbl>, firstgen <dbl>, debt <dbl></code></pre>
<div class="sourceCode"><pre class="sourceCode r"><code class="sourceCode r"><span class="kw">top_n</span>(scorecard, <span class="dv">10</span>, <span class="dt">wt =</span> cost)</code></pre></div>
<pre><code>## # A tibble: 10 × 12
## unitid name state
## <int> <chr> <chr>
## 1 120254 Occidental College CA
## 2 195304 Sarah Lawrence College NY
## 3 115409 Harvey Mudd College CA
## 4 130697 Wesleyan University CT
## 5 147767 Northwestern University IL
## 6 144050 University of Chicago IL
## 7 230816 Bennington College VT
## 8 182670 Dartmouth College NH
## 9 179867 Washington University in St Louis MO
## 10 190150 Columbia University in the City of New York NY
## # ... with 9 more variables: type <chr>, cost <int>, admrate <dbl>,
## # satavg <dbl>, avgfacsal <dbl>, pctpell <dbl>, comprate <dbl>,
## # firstgen <dbl>, debt <dbl></code></pre>
</p>
<p></details></p>
</div>
<div id="which-type-of-college-has-the-highest-average-sat-score" class="section level3">
<h3>Which type of college has the highest average SAT score?</h3>
<details> <summary>Click for the solution</summary>
<p>
<div class="sourceCode"><pre class="sourceCode r"><code class="sourceCode r">scorecard %>%
<span class="st"> </span><span class="kw">group_by</span>(type) %>%
<span class="st"> </span><span class="kw">summarize</span>(<span class="dt">mean_sat =</span> <span class="kw">mean</span>(satavg, <span class="dt">na.rm =</span> <span class="ot">TRUE</span>))</code></pre></div>
<pre><code>## # A tibble: 3 × 2
## type mean_sat
## <chr> <dbl>
## 1 Private, for-profit 1002.500
## 2 Private, nonprofit 1075.287
## 3 Public 1037.410</code></pre>
</p>
<p></details></p>
</div>
<div id="how-many-private-nonprofit-schools-are-cheaper-than-the-university-of-chicago" class="section level3">
<h3>How many private, nonprofit schools are cheaper than the University of Chicago?</h3>
<details> <summary>Click for the solution</summary>
<p>
<div class="sourceCode"><pre class="sourceCode r"><code class="sourceCode r"><span class="co"># number of schools</span>
scorecard %>%
<span class="st"> </span><span class="kw">filter</span>(type ==<span class="st"> "Private, nonprofit"</span>) %>%
<span class="st"> </span><span class="kw">arrange</span>(cost) %>%
<span class="st"> </span><span class="kw">mutate</span>(<span class="dt">row =</span> <span class="kw">row_number</span>()) %>%
<span class="st"> </span><span class="kw">filter</span>(name ==<span class="st"> "University of Chicago"</span>)</code></pre></div>
<pre><code>## # A tibble: 1 × 13
## unitid name state type cost admrate
## <int> <chr> <chr> <chr> <int> <dbl>
## 1 144050 University of Chicago IL Private, nonprofit 62425 0.0881
## # ... with 7 more variables: satavg <dbl>, avgfacsal <dbl>, pctpell <dbl>,
## # comprate <dbl>, firstgen <dbl>, debt <dbl>, row <int></code></pre>
<div class="sourceCode"><pre class="sourceCode r"><code class="sourceCode r"><span class="co"># percentage of schools</span>
scorecard %>%
<span class="st"> </span><span class="kw">filter</span>(type ==<span class="st"> "Private, nonprofit"</span>) %>%
<span class="st"> </span><span class="kw">mutate</span>(<span class="dt">cost_rank =</span> <span class="kw">percent_rank</span>(cost)) %>%
<span class="st"> </span><span class="kw">filter</span>(name ==<span class="st"> "University of Chicago"</span>)</code></pre></div>
<pre><code>## # A tibble: 1 × 13
## unitid name state type cost admrate
## <int> <chr> <chr> <chr> <int> <dbl>
## 1 144050 University of Chicago IL Private, nonprofit 62425 0.0881
## # ... with 7 more variables: satavg <dbl>, avgfacsal <dbl>, pctpell <dbl>,
## # comprate <dbl>, firstgen <dbl>, debt <dbl>, cost_rank <dbl></code></pre>
</p>
<p></details></p>
</div>
</div>
<div id="session-info" class="section level1 toc-ignore">
<h1>Session Info</h1>
<div class="sourceCode"><pre class="sourceCode r"><code class="sourceCode r">devtools::<span class="kw">session_info</span>()</code></pre></div>
<pre><code>## Session info --------------------------------------------------------------</code></pre>
<pre><code>## setting value
## version R version 3.3.2 (2016-10-31)
## system x86_64, darwin13.4.0
## ui RStudio (1.0.136)
## language (EN)
## collate en_US.UTF-8
## tz America/Chicago
## date 2017-03-06</code></pre>
<pre><code>## Packages ------------------------------------------------------------------</code></pre>
<pre><code>## package * version date source
## assertthat 0.1 2013-12-06 CRAN (R 3.3.0)
## backports 1.0.5 2017-01-18 CRAN (R 3.3.2)
## base64enc 0.1-3 2015-07-28 CRAN (R 3.3.0)
## bigrquery * 0.3.0 2016-06-28 CRAN (R 3.3.0)
## bitops 1.0-6 2013-08-17 CRAN (R 3.3.0)
## boot * 1.3-18 2016-02-23 CRAN (R 3.3.2)
## broom * 0.4.2 2017-02-13 CRAN (R 3.3.2)
## car 2.1-4 2016-12-02 CRAN (R 3.3.2)
## caret * 6.0-73 2016-11-10 CRAN (R 3.3.2)
## class 7.3-14 2015-08-30 CRAN (R 3.3.2)
## codetools 0.2-15 2016-10-05 CRAN (R 3.3.2)
## colorspace 1.3-2 2016-12-14 CRAN (R 3.3.2)
## curl * 2.3 2016-11-24 CRAN (R 3.3.2)
## DBI 0.5-1 2016-09-10 CRAN (R 3.3.0)
## devtools 1.12.0 2016-06-24 CRAN (R 3.3.0)
## digest 0.6.12 2017-01-27 CRAN (R 3.3.2)
## dplyr * 0.5.0 2016-06-24 CRAN (R 3.3.0)
## e1071 * 1.6-8 2017-02-02 CRAN (R 3.3.2)
## evaluate 0.10 2016-10-11 CRAN (R 3.3.0)
## FNN * 1.1 2013-07-31 CRAN (R 3.3.0)
## forcats * 0.2.0 2017-01-23 CRAN (R 3.3.2)
## foreach * 1.4.3 2015-10-13 CRAN (R 3.3.0)
## foreign 0.8-67 2016-09-13 CRAN (R 3.3.2)
## gam * 1.14 2016-09-10 CRAN (R 3.3.0)
## gapminder * 0.2.0 2015-12-31 CRAN (R 3.3.0)
## gbm * 2.1.1 2015-03-11 CRAN (R 3.3.0)
## geosphere 1.5-5 2016-06-15 CRAN (R 3.3.0)
## ggmap * 2.7 2016-12-07 Github (dkahle/ggmap@c6b7579)
## ggplot2 * 2.2.1 2016-12-30 CRAN (R 3.3.2)
## ggrepel * 0.6.5 2016-11-24 CRAN (R 3.3.2)
## ggstance * 0.3 2016-11-16 CRAN (R 3.3.2)
## gridExtra * 2.2.1 2016-02-29 cran (@2.2.1)
## gtable 0.2.0 2016-02-26 CRAN (R 3.3.0)
## haven * 1.0.0 2016-09-23 cran (@1.0.0)
## here * 0.0-6 2017-02-04 Github (krlmlr/here@007bfd9)
## hexbin * 1.27.1 2015-08-19 CRAN (R 3.3.0)
## highr 0.6 2016-05-09 CRAN (R 3.3.0)
## hms 0.3 2016-11-22 CRAN (R 3.3.2)
## htmltools 0.3.5 2016-03-21 CRAN (R 3.3.0)
## htmlwidgets 0.8 2016-11-09 CRAN (R 3.3.1)
## httpuv 1.3.3 2015-08-04 CRAN (R 3.3.0)
## httr * 1.2.1 2016-07-03 CRAN (R 3.3.0)
## igraph 1.0.1 2015-06-26 CRAN (R 3.3.0)
## ISLR * 1.0 2013-06-11 CRAN (R 3.3.0)
## iterators 1.0.8 2015-10-13 CRAN (R 3.3.0)
## janeaustenr 0.1.4 2016-10-26 CRAN (R 3.3.0)
## jpeg 0.1-8 2014-01-23 cran (@0.1-8)
## jsonlite * 1.2 2016-12-31 CRAN (R 3.3.2)
## kknn * 1.3.1 2016-03-26 CRAN (R 3.3.0)
## knitr * 1.15.1 2016-11-22 cran (@1.15.1)
## labeling 0.3 2014-08-23 CRAN (R 3.3.0)
## lattice * 0.20-34 2016-09-06 CRAN (R 3.3.2)
## lazyeval 0.2.0 2016-06-12 CRAN (R 3.3.0)
## lme4 1.1-12 2016-04-16 cran (@1.1-12)
## lubridate * 1.6.0 2016-09-13 CRAN (R 3.3.0)
## lvplot * 0.2.0.9000 2017-01-06 Github (hadley/lvplot@8ce61c7)
## magrittr 1.5 2014-11-22 CRAN (R 3.3.0)
## mapproj 1.2-4 2015-08-03 CRAN (R 3.3.0)
## maps * 3.1.1 2016-07-27 cran (@3.1.1)
## MASS 7.3-45 2016-04-21 CRAN (R 3.3.2)
## Matrix 1.2-8 2017-01-20 CRAN (R 3.3.2)
## MatrixModels * 0.4-1 2015-08-22 CRAN (R 3.3.0)
## memoise 1.0.0 2016-01-29 CRAN (R 3.3.0)
## mgcv 1.8-17 2017-02-08 CRAN (R 3.3.2)