Category Archives: Open Source

Change SP Object Polygon Rendering Order in R

The Problem

I have a geoJSON file that was made by combining many (as in ~250) geoJSON files each containing a single polygon in R.  The polygons are of varying size and can overlap in space.  When I open the file in a GIS, often the smaller polygons are hidden under the larger ones, making displaying the data challenging, if not impossible.

Map showing few polygons when there should be many

I know there are more polygons than are visible in this area, so they must be hiding under the larger polygons.

Map showing larger polygons obscuring smaller polygons

With transparency set to 60% for each polygon (see the Draw Effects dialog in QGIS for the fill symbol layer), you can see that smaller polygons are hiding under larger ones.

The Goal

I would prefer that the polygons stack up so that the largest is on the bottom and the smallest is on the top.  This requires that I change the rendering order based on the area of each polygon.

The Quick Fix

QGIS has an option to control the rendering order.  Open the Layer Properties; go to the Style tab; check the box for “Control feature rendering order”; click the button on the right with the A Z and an arrow to enter the expression you want (I would order by area for example).

Why isn’t this the right solution for me?  I’m sharing a user-contributed dataset.  One of the goals is that anyone can use it.  When polygons are obscured, it makes the dataset just a little harder to use and understand, which means people won’t like using it.  Another goal is that anyone with a reasonable understanding of GIS can contribute.  If I have to write a bunch of instructions on how to visualize the data before they can add to the dataset, people are less likely to contribute.

Map showing all the polygons expected.

Now I can see all of the polygons because the larger ones are on the bottom and the smaller ones are on top.

My Solution

Hat tip to Alex Mandel and Ani Ghosh for spending a couple of hours with me hammering out a solution.  We used R because I already had a script that takes all of the polygons made by contributors and combines them into one file.  It made sense in this case to add a few lines to this post-processing code to re-order the polygons before sending the results to the GitHub repository.

What you need to know about rendering order & SP Objects

The order in which items in an SP object are rendered is controlled by the object’s ID value.  The ID value is hidden in the ID slot nested inside the polygons slot.  If you change these values, you change the order items are rendered.  ID = 1 goes first, ID =2 goes on top of 1, 3 goes on top of 2, and so on.  So for my case, assigning the IDs based on the area will get me the solution.

How

This Stack Exchange Post on re-ording spatial data was a big help in the process.  Note that every SP object should have the slots and general structure I used here.  There’s nothing special about this dataset.  If you’d like the dataset and another copy of the R code, however, it is in the UC Davis Library’s AVA repository.

#load the libraries you'll need
library(raster)
library(geojsonio)
library(rgdal)

### FIRST: Some context about how I made my dataset in the first place

# search in my working directory for the files inside the folders 
# called "avas" and "tbd"
avas <- list.files(path="./avas", pattern = "*json$", full.names = "TRUE")
tbd <- list.files(path="./tbd", pattern = "*json$", full.names = "TRUE")

#put the two lists into one list
gj <- c(avas, tbd)

#read all the geojson files & create an SP object
vects <- lapply(gj, geojson_read, what="sp")

#combine all the vectors together. bind() is from the raster package.
#probably could just rbind geojson lists too, but thats harder to plot
all <- do.call(bind, vects)

#Change any "N/A" data to nulls
all@data[all@data=="N/A"]<- NA


### SECOND: How I did the sorting

#Calculate area of polygons - needed for sorting purposes
# the function returns the value in the area slot of each row
all@data$area<-sapply(slot(all, "polygons"), function(i){slot(i, "area")})

#add the row names in a column - needed for sorting purposes
all$rows<-row.names(all)

#Order by area - row names & area are needed here
# decreasing = TRUE means we list the large polygons first
all<-all[match(all[order(all$area, decreasing = TRUE),]$rows, row.names(all@data)),]

#add new ID column - essentially you are numbering the rows 
# from 1 to the number of rows you have but in the order of 
# largest to smallest area
all$newid<-1:nrow(all)

#assign the new id to the ID field of each polygon
for (i in 1:nrow(all@data)){
 all@polygons[[i]]@ID<-as.character(all$newid[i])}

#drop the 3 columns we added for sorting purposes (optional)
all@data<-all@data[,1:(ncol(all@data)-3)]

#write the data to a file in my working directory
geojson_write(all, file="avas.geojson", overwrite=TRUE, convert_wgs84 = TRUE)
Advertisements

A Visual Summary of FOSS4G 2017

FOSS4G_2_08212017.jpg

It’s the afternoon of Saturday, August 19th.  I’m sitting near the back of an airplane wondering how I’m going to keep from going stir crazy on this almost 6 hour flight back to California.  As the plane takes off, I’m thinking about the last week at FOSS4G 2017 and images are flashing through my brain.  Ok, I think, once I can take some stuff out of my bag (neatly stowed under the seat in front of me), I’ll doodle for a while.  That should keep me busy for an hour or so.  5 hours later, I’ve almost finished this whole page and it’s just about time to land.

What struck me at the conference was how important the giving and sharing culture of our community is.  The news from Charlottesville and the US President’s response seemed impossible. I caught up with people I hadn’t seen in a year, met people in person that I’d only known on Twitter, and found potential collaborators for a pet project that needs more people.  I also found inspiration in many of the talks and came home wanting to get started on a thousand new things (except that this cold someone shared with me is preventing me from getting too much done yet).  The best experience though was when I got to share my skills with the community.  I taught a workshop (at Harvard!!!) to 20 incredibly skilled people and gave a talk to about 80 – both about cartography.  I hope that what I shared will help them with some aspect of their work.

While I think it’s clear to everyone how the coders contribute, I think we need to do a better job acknowledging the contributions of users.  After hearing a few presenters say they didn’t feel like they belonged because they were “just users”, I started speaking up during the question time telling the speaker how important their role in the community is.  Making every member of our community feel welcome and valued is key to our continued success.

We also need to do a better job with diversity.  The breakdown of attendees neatly avoided discussing race and gender.  A look around the room, probably told you everything you needed to know about those topics though.  How do we fix it?  I’m not sure, but if we keep the discussion going rather than igoring it, we’ll find the solution faster.

So, thank you to everyone who made FOSS4G 2017 possible



My art process:
(pencil: Mirado Black Warrior HB 2) sketch in an image
(pen: Pilot Rolling Ball Precise V7 fine or Pentel Sign Pen ST150 felt tip) ink in the sketch
(pen: Pentel Sign Pen SES15N brush tip) fill in the spaces between the images
Erase pencil


Dealing with Factors in R

What is the deal with the data type “Factor” in R?  It has a purpose and I know that a number of packages use this format, however, I often find that (1) my data somehow ends up in the format and (2) it’s not what I want.

My goal for this post: to write down what I’ve learned (this time, again!) before I forget and have to learn it all over again next time (just like all the other times).  If you found this, I hope it’s helpful and that you came here before you started tearing your hair out, yelling at the computer, or banging your head on the desk.

So here we go.  Add your ways to deal with factors in the comments and I’ll update the page as needed.

Avoid Creating Factors

Number 1 best way to deal with factors (when you don’t need them) is to not create them in the first place!  When you import a csv or other similar data, use the option stringsAsFactors = FALSE (or similar… read the docs for find the options for the command you’re using) to make sure your string data isn’t converted automatically to a factor.  R will sometimes also convert what seems to clearly be numerical data to a factor as well, so even if you only have numbers, you may still need this option.

MyData<-read.csv(file="SomeData.csv", header=TRUE, stringsAsFactors = FALSE)

Convert Data

Ok, but what if creating a factor is unavoidable?  You can convert it.  It’s not intuitive so I keep forgetting.  Wrap your factor in an as.character() to just get the data.  It’s now in string format, so if you need numbers, wrap all of that in as.numeric().

#Convert from a factor to a list
CharacterData<-as.character(MyFactor)

#Convert from a factor to numerical data
NumericalData<-as.numeric(as.character(MyFactor))

 

What’s Missing?

Do you have any other tricks to working with data that ends up as a Factor?  Let me know in the comments!


Inkscape for Applique Sewing Patterns

Inkscape is a vector illustration program so most people think of it as an art program for producing slick graphics.  But it’s a really useful tool for planning an preparing for other art forms.  For example, I’ve been using it for sewing.  What?  Yes, sewing.  It’s incredibly useful for drawing patterns.  Recently I’ve been working on a needle turn applique quilt based on the work of Charley Harper, but for the past few years I’ve made felt Christmas ornaments for friends and family, for all of which I used Inkscape to draw the patterns.

If you’re familiar with Inkscape already, making applique patterns will be pretty straight forward. If you’re new to the program, I highly recommend working through a couple of tutorials.  Here’s my general workflow (yours may differ):

  1. Start with an image.  On Pinterest, great projects abound, but sometimes the post links to costly instructions, or no pattern at all.  I’ve also found things that I like the look of, but are a different scale – too big or too small.  Or, as with my latest project, I’m creating my own pattern pieces from an image.  Look for images with distinct polygons of colors.  Blended or faded areas are going to be harder to duplicate with applique unless you can find fabric with the right fade or you dye your own.
  2. Put the image into an Inkscape file and resize it to the size you want your final project to be.
  3. Draw polygons around each of the colors you see in your image.  You’ll want to think about how you’ll put the whole thing together as you trace, so think about how the layers will work together.  For example, if you have polka-dots, you’ll want to place the circles on top of a larger background color, not have a section of background color with holes cut out like Swiss cheese.

    Cat_PatternMaking

    Start by tracing out all of the sections you’ll need to cut from various colors of fabric.

  4. Start a new Inkscape file and make the size of the page whatever size you plan to print.  For those in the US, you’ll probably want US Letter Size.
  5. Copy your polygons from the first file and past them into the second.  (I find keeping both files is helpful later for placement of the pieces.)  Arrange all your polygons on the page so that none overlap.  For larger projects, I’ve made several files. If the same shape shows up multiple times in your pattern, for example maybe eyes or ears, you only need to include that shape once.

    Cat_PatternPieces

    One of 3 pages of pattern pieces for a larger work with many pieces.

  6. On each piece, I like to print the color of the fabric I plan to use and how many of this piece I need to cut out.
  7. If you have a really big pattern piece that’s bigger than your printable page size, there’s a solution.  Put the big pieces into one Inkscape file, then size the page to the content, giving it a reasonable margin for your printer.  Then save the file as a PDF.  Open the PDF and in the print options, pick Poster (or similar setting).  It will divide up the pattern into printable pages.  Then you can tape the pages together before you cut out the pattern.

    Cat_BigPieces

    My printer settings have an option to print large PDFs in pieces.  Yours probably has something similar.  Super useful for printing larger pattern pieces.

Bonus! Now when you’re placing your pieces, some stuff you can just eyeball and it will be fine.  In some situations though you might need to be more precise.  Because you have your original pattern tracing in Inkscape, you can go back to that file and measure the distance between items.  I set my units to inches and draw a line, then see how long my line is.  Super simple, but very effective.

Measure

The red line measures how long the vertical eye whisker is.

See the finished piece on a previous blog post.


Limp on a Limb: Another Charley Harper Inspired Quilt Block

This second block in my Charley Harper Quilt is inspired by the piece Limp on a Limb.  If you compare the original and the block, you’ll see that I’ve made some edits.  Most notably, I have decided (for now at least) to not include the leaf pattern in the background.  Repeated shapes are a hallmark of Harper’s work, so including the pattern would be more true to the work, but in reality, it would require extensive embroidery and I’m afraid that won’t hold up long-term, especially given the light weight of the fabric I’ve chosen for the background.  That being said, the fabric I chose is mottled green and I hope it at least gives the piece some more depth.

IMG_20160805_101552[1]

Example diagram from placing the cat’s eye wiskers.

For this block, I thought I would show some of the detail of how I transfer lines from the pattern to the piece.  All my patterns are digital svg files, which means I can measure the size of each object in Inkscape.  (I promise to write a post about this with more detail and hopefully convert some quilters to Inkscape quilt designers… but later.  Ok, it’s later. See the post here.)  I make measurements from a reference point, draw out a diagram, then transfer the measurements to the fabric using a chalk pencil (either white or blue depending on the color of the fabric).  Then I embroider.  It’s important to mark as little as possible on the fabric with the calk pencils, because the marks are hard to get out.

IMG_3862.JPG

Faint chalk pencil marks show where to embroider the eye whiskers.

When placing any object in a piece, whether it’s embroidery or a layer of fabric, I’ve found that it’s important to figure out what feature the new object needs to be inline with. For placing the eye whiskers, at first I was going to reference the corner of the eye. It seemed logical. Then I found that in the original piece, the left eye and whiskers don’t line up. What? But there’s always such precision in Harper’s work! But after some staring at the piece, I realized that the vertical line of both sets of eye whiskers intersects the point where the ear meets the head. Bingo! Now my whiskers are in the right spot.

img_3865

The finished piece.


Designing The Charley Harper Quilt

I’ve been a big fan of modern artists Charley Harper’s work since 2012 when I taught may Science to Art course at University of California Davis.  In that class, students were tasked with communicating science concepts through art.  At the suggestion of the Wildlife Museum staff who sponsored the course, we chose to emulate Charley Harper’s style of simplifying species down to their most simple yet still recognizable forms.  We learned how Harper repeated forms – leaves and mice in one image are the same teardrop shape, for example – and simplified bodies down to their most elemental form.  The work was all completed in Inkscape and eventually printed on large banners that hang in Academic Surge.  The results were magical.

Fast forward a few years and I saw a post on the Charley Harper Studios’ Facebook page asking people to post images of quilts they had made using Harper’s images or their line of fabrics.  The quilts were charming and it made me think about how those simplified forms would easily translate into quilt blocks.

So, I set out to figure out how I could make one.  I have made one traditional quilt and one comforter more than ten years ago, but I regularly sew and do felt applique for Christmas ornaments, so the skills are there.  I learned about applique for quilting (particularly with woven fabrics that can fray, unlike felt) and decided that needleturn applique sounded like the least fussy option.  I found this video on YouTube to be an excellent quick tutorial:

Next, I needed a plan so I found a bunch of images I liked and arranged them in an Inkscape file with a page size set to the size of the finished product that I could later turn into vector lines for pattern pieces.  More on that later.  This is my plan (please note that the artwork is copyrighted by the artist):

quiltplan.jpg

I’ll update my progress as I go.


Making of a Moon Tree Map

29_CleanUp.png

I’m presenting a workflow for finishing maps in Inkscape at FOSS4G North America this year (2016). To really show the process effectively, I made a map and took screenshots along the way.

The Data

I decided to work with Moon Tree location data.  It’s quirky and interesting… and given that this is a geek conference I figured the space reference would be appreciated.  A few months ago I learned about Moon Trees watching an episode of Huell Howser on KVIE Public Television and then visited the one on the California State Capitol grounds.  I later learned from my aunt that my grandfather was a part of the telemetry crew that retrieved the Apollo 14 mission that carried the seeds that would become the Moon Trees, so there’s something of a connection to this idea.  Followers of my research also know that I’m a plant person, particularly plant geography.  So this seemed like the perfect dataset.  I was fortunate to find that Heather Archuletta had already digitized the locations of public trees and made them available in KML format.

Data Processing

The KML format is great for some applications (particularly Google maps, for which it was designed) but it poses some challenges.  I spent several hours… maybe more than I want to admit… formatting the .dbf to make the shapefile more useful for my purposes.  I created columns and standardized the content.  The map does not present all the data available (um… duh.).  It was challenge enough getting all this onto one page.

Yes, Inkscape is Necessary

You can’t make this map in QGIS completely.  I mean, normally you can make some fantastic maps in QGIS, but this one is actually not possible.  Right now, QGIS can’t handle having map frames with different projections.  I tried, but I found that even when the map composer looked right, the export in all three export options changed the projection and center of each frame to match that of the last active frame.  So I ended up with a layout with three zoom levels centered on Brazil… interesting, but not what I had in mind.  So I exported an .svg file three times from the map composer – one for each map frame – and put them together in Inkscape.

Sneaky Cartography

One of the methods I often use in my maps is to create subtle blurred halos behind text or icons that might otherwise get lost on a busy background.  I don’t like when the viewer sees the halos (maybe it’s from teaching ArcMap far too many years at universities).  It’s not quite a pet peeve, but I think there’s often better ways to handle busy backgrounds and readability.  My blog, my soapbox.  Can you spot them?  There are a couple in the map and in the final slide of the pitch video.  It doesn’t look like much, but I promise the text is easier to read.

The texture on the continents is the moon.  I clipped a photo of the moon using the continent outlines.  I liked the idea of trees on the moon.

Icons

The icons are special to me.  I’ve been really wanting to make a map using images from Phylopic and I thought this was the perfect opportunity… but… but… no one had uploaded outlines for any of the species I needed.  So I made them and uploaded them.  So if you want an .svg of these, help yourself.  If, however, you need dinosaurs, they’ve got you covered.

Watch it happen:

My pitch video captures the process from start to finish:

 Want more open source cartography?

Come to FOSS4G North America and see my and several other talks focused on cartography.  I’ll cover methods and tools in Inkscape common for cartography.