Skip to main content

Pretty Good Hat

Tag: data

A scatterplot showing five days of game history. The plots for June 3 and 4 are dense, with lots of varying size dots representing games. About two thirds of the dots on the plot show winning games.

Destiny 2 talk: I had a good time in Iron Banner games this weekend. Most of my matches were fun. I didn’t have any long losing streaks like last season, and really liked the build I settled on. Maybe next time I’ll play my other characters, but it’s all for fun now that I completed the gilded title one more time!

Had a pretty good hobby coding weekend learning how to use purrr::pluck() to get specific nested fields from piles of json data rather than blowing out the whole series of nested lists.

Super-pleased to see that a couple of small Shiny app improvements I made this weekend correctly picked up on some source data changes today and automatically handled them. Pretty cool!

Having seen how easy it was to convert a couple of hobby projects to use {pins}, I’m now daydreaming about putting it at the center of an S3 data lake operation.

screenshot showing a set of colorful scatterplots displaying a variety of armor types and their stats

I spent a bunch of the weekend beginning a redesign of my ArmoreR project, which aims to be a Destiny 2 armor stats profiler built in R and Shiny. A year on from when I began it, I have a much better understanding of how a Shiny app works, and have also incorporated a proper, working oauth2 workflow into this revision (incorporating the things I figured out for my power level tracker). It’s really, really satisfying to be rebuilding it with all the things I’ve learned. I think the application is going to be so much better and less complicated than the first iteration. I still have a ways to go, and am happy with just how much I have transformed it with a year of learning and practice on other things.

Today I made improvements to some R code in my Destiny 2 hobby-coding-verse after learning how to much more cleanly deal with nested lists. I had previously used a solution using map() to apply a selector to each item in the list, but this was clunky, hard to remember, and became really hard to read with several levels of a nested list.

The far better solution is the unnest_auto function from {tidyr}, which I came upon when tinkering with the API data recently. Once I understood how it works, it’s so easy and satisfying! The key is to first make a named tibble.

> tibble(my_tibble = instanced)
# A tibble: 906 × 1
   <named list>     
 1 <named list [10]>
 2 <named list [10]>
 3 <named list [10]>
 4 <named list [10]>
 5 <named list [12]>
 6 <named list [10]>
 7 <named list [10]>
 8 <named list [10]>
 9 <named list [9]> 
10 <named list [9]> 
# … with 896 more rows

That nice tibble can be operated on by unnest_auto():

> tibble(my_tibble = instanced) %>% unnest_auto(my_tibble) %>%
select(itemLevel, breakerType)
Using `unnest_wider(my_tibble)`; elements have 8 names in common
# A tibble: 906 × 2
   itemLevel breakerType
       <int>       <int>
 1       132          NA
 2       133          NA
 3       133          NA
 4       132          NA
 5       133           3
 6       132          NA
 7       133          NA
 8       133          NA
 9         0          NA
10         0          NA
# … with 896 more rows

(I selected just a couple of columns for readability there; if you don’t do that, you’ll receive all fields at the current list level, including additional nested lists if they exist). After figuring this out, I realized that I also needed to keep the names of each list element, because they constitute a unique ID for the element returned from the API query, and I banged my head a bit on trying to do that as a part of the unnest operation, before I backed up, recentered on the outcome I wanted to produce, and realized I could do it really cleanly using mutate()! The final code looks like this:

> tibble(my_tibble = instanced) %>% unnest_auto(my_tibble) %>% 
select(itemLevel, breakerType) %>% mutate(id = names(instanced))
Using `unnest_wider(my_tibble)`; elements have 8 names in common
# A tibble: 906 × 3
   itemLevel breakerType id                 
       <int>       <int> <chr>              
 1       132          NA 6917529338105913753
 2       133          NA 6917529550016812142
 3       133          NA 6917529281178546429
 4       132          NA 6917529231127188610
 5       133           3 6917529301504642848
 6       132          NA 6917529234625200021
 7       133          NA 6917529193313832065
 8       133          NA 6917529182955737017
 9         0          NA 6917529417667489181
10         0          NA 6917529490520758715
# … with 896 more rows

The D2 API returns a ton of nested lists, so this simplified, accessible and effective tool is 100% getting a featured spot in my toolbox.

Data storytelling with Ever since wondering if I could make something vaguely Spotify Wrapped-like with data from, I’ve dived deep into retrieving, slicing and making pictures of my music data over the past several years. Experimenting with a view of my most-played tracks over, it started to click for me just how many stories are found in this data, that this is much more than listening history – my own stories are embedded in this data, too.

In this plot of my most-played tracks for each year since 2005, you can see the huge spike in 2015, featuring huge numbers of plays of songs by Lord Huron, Frank Taylor, Hem, and Josh Ritter. I knew immediately what this was: The bedtime playlist for our kiddo, whom we would tuck in and leave to listen to songs they loved at the time. They’ve never, ever slept well without a lot of work, and for a while we had a beautiful routine of tuckins and music.

a parallel coordinates plot showing many colorful dots and a vertical spike of much larger dots in 2015

It’s a little bittersweet to see this, as it represents a really hard time in all our lives but also signifies a lot of love and, in the music itself, something that’s always been and continues to be really special.

(Methodological note: Because the spike in 2015 is so high, I had to re-do the entire plot on a log scale in order to see any variation at all in the lower-numbered tracks! I made up for that by scaling the size of the dots to the true total annual play count of each track.)

You can see the data entirely fall off a cliff after 2015. This same picture also tells the story of the death of Rdio, at one time my very favorite way to listen to music and which had great support for tools like; I replaced it with Apple Music, which has never natively supported that record-keeping, so 2015 is the last year until 2020 that I even have much of this information.

That, of course, is the first year of pandemic and also the first year I started using Roon, listening to music exclusively at home, and shifted all my workouts to home, too. So that big 2020-2021 spike of lots of colors? That’s my wife and me lifting weights and doing squat jumps in our living room!

a colorful stack of close-together dots in a scatterplot

In 2007 you can see the light blue dots appear: That’s when I started listening to Josh Ritter, while away from home for a few months. I was living alone in Bellevue, WA, on a pre-doctoral internship, and carrying my iPod in my pocket while jogging or walking in the park across the street from my long-term company housing complex. I can trace those light blue dots and lines all the way through to now (though the frequency of that workout playlist swamps even my favorite artists right out of the annual top twenty). And looking at this graph I can see the trails in that little park, remember the first time I heard Josh Ritter play “Wings” in my headphones; I can recall tucking the iPod into the console of my rental car, and then spiral into memories of the post-internship vacation that my wife and I took up to the San Juan Islands, where years before I spent a summer teaching climbing.

photo of a shiny silver car gear-shift, with a white third-generation iPod tucked beside it

I can click the 2011 dots to see the music I played for our kiddo in the car while driving to and from day-care. There are some silly kids songs that they used to love, and I was also on a pretty good OK Go kick at the time, so “This Too Shall Pass” was a common track in the car. God that song still gets me. I got into TV On the Radio around this time, too, and found it good car music. (Again: Thanks to Rdio, I was discovering and listening to a lot of music.)

Way back in 2005-2006 I was working at home on finishing my disseration. In iTunes I had albums from Richmond Fontaine and Tom Waits in heavy rotation – Ripped from actual CDs that I bought at Gopher Sounds in downtown Flagstaff!

I’m honestly surprised that is still operating. I’m not sure what it really does, anymore that would make revenue, but I’m glad it’s still there and that I have good tools to store my listening history once again (Oh – including now adopting Marvis as a great iOS app for scrobbling when I do listen to Apple Music again, thanks to some good discussion with music folks at!) When scrobbling stopped working reliably there after 2015, for a while I decided that I didn’t care: listening to music was an ephemeral experience, and so would be my interest in online web-2.0 tools, I reasoned, and I made peace with leaving behind that history.

Now, as long as it continues to work, I’m so happy to have this somewhat constant and almost invisible trace, this throughline of so many moments in my life.

screenshot of a parallel coordinates plot, which draws lines between ranked points over time. The Josh Ritter song 'Good Man' is highlighted as the top ranking track in 2013, and has lines connecting to its rank in several prior and subsequent years.

Today’s data time produced this plot of my top 20 tracks for each year since 2005. I’m going to write a (much?) longer post about this, because it has my brain really spinning on storytelling with data; for now, enough to say I’m excited and having a good time!

screenshot showing a plot of data bars

Continuing to tinker with data, I’ve built a tool to pull my per-track history, and I can do a bunch of fun things with that detail. This plot shows how much I listened to artists in their first year in my data set, compared with all listening in the same year.