Geocamp ES 2017


Quedan poco más de dos semanas para la GeocampES 2017, que este año se va a celebrar el sábado 16 de Septiembre en Almería, organizada por Geoinquietos AlmeríaHacklab Almería y UNIA. Como cada año la idea es organizar un evento sencillo, divertido y muy técnico y participativo. No hay agenda, siguiendo el modelo de desconferencia habitual, al llegar allí se espera que un buen porcentaje de los asistentes se postulen para salir a contar algo: puede ser una charla convencional, una demostracíon de algo en lo que estás trabajando, una dinámica de grupo que te parezca interesante, etc. Cualquier actividad con una componente geo bien definida (o no tanto) es bienvenida.

La idea de Geocamp ES se tomó siguiendo el liderazgo de los compañeros de Portugal. A diferencia sus geocamps en pequeños pueblos más o menos alejados de todo, las anteriores Geocamp españolas se han celebrado en ciudades. En cualquier caso este año nos vamos a Almería, que está un poco más difícil de acceder (sin exagerar) pero sigue siendo una ciudad que seguro va a ofrecernos un espacio agradable y totalmente adecuado para pasarlo bien aprendiendo.

Si lo que te he contado te parece mínimamente interesante, anímate y prepara un fin de semana por el sur de España con el resto de la comunidad geoespacial.

Apúntate aquí.

Advertisements

Aggregating points: JSON on SQL and loops on infowindows


NOTE: I’ll use CARTO but you can apply all this to any webmapping technology backed by a modern database.

Get all the data

So we start with the typical use case where we have a one to many relationship like this:

    select e.cartodb_id,
           e.displayname,
           e.division,
           e.photourl,
           l.cartodb_id as locaction_id,
           l.location,
           l.the_geom_webmercator
      from locations l
inner join employees e
        on e.location = l.location
  order by location

Easy peasy, we have a map with many stacked points. From here you can jump to this excellent post by James Milner about dense point maps. My example is not about having thousands of scattered points that at certain zoom levels overlap. Mine is a small set of locations but many points “stacking” on them. In this case you can do two things: aggregate or not. When you aggregate you pay a prize for readability: reducing all your data to those locations and maybe using visual variables to show counts or averages or any other aggregated value and finally try to use the interactivity of your map to complete the picture.

So at this point we have something like this map, no aggregation yet, but using transparency we can see where CARTO has many employees. We could also use a composite operation instead of transparency to modify the color of the stacked points.

Stacking points using transparency

Stacking points using transparency

Aggregate and count

OK, let’s do a GROUP BY the geometry and an aggregation like counting. At least now we know how many people are there but that’s all, we loose the rest of the details.

    select l.the_geom_webmercator,
           min(e.cartodb_id) as cartodb_id,
           count(1) as counts
      from locations l
inner join employees e
        on e.location = l.location
  group by l.the_geom_webmercator
Grouping by location and counting

Grouping by location and counting

Aggregate one field

But in my case, with CARTO we have PostgreSQL at hand so we can do way more than that. PostgreSQL has many many cool features, handling JSON types is one of them. Mix that with the fact that almost all template systems for front-end applications allow you to iterate over JavaScript Objects and you have a winner here.

So we can combine the json_agg function with MustacheJS iteration over objects to allow rendering the names of our employees.

    select l.the_geom_webmercator,
           min(e.cartodb_id) as cartodb_id,
           l.location,
           json_agg(e.firstname) as names, -- JSON aggregation
           count(1) as counts
      from locations l
inner join employees e
        on e.location = l.location
  group by l.the_geom_webmercator,l.location

And this bit of HTML and Mustache template to create a list of employees we can add to the infowindow template:

<ul style="margin:1em;list-style-type: disc;max-height:10em;">
{{#names}}<li class="CDB-infowindow-title">{{.}}</li>{{/names}}
</ul>
{{^names}}loading...{{/names}}

List of employees on the infowindow

We could do this without JSON types, composing all the markup in the SQL statement but that’s generating quite a lot of content to move to the frontend and of course making the whole thing way harder to maintain.

Aggregate several fields

At this point we can repeat the same function for the rest of the fields but we need to iterate them separatedly. It’d be way better if we could create JSON objects with all the content we want to maintain in a single output field we could iterate on our infowindow. With PostgreSQL we can do this with the row_to_json function and nesting an inner double SELECT to give the properties names. We can use directly row_to_json(row(field1,field2,..)) but then our output fields would have generic names.

    select l.the_geom_webmercator,
           min(e.cartodb_id) as cartodb_id,
           l.location,           
           count(1) as counts,
           json_agg(row_to_json((
             SELECT r
               FROM (
                 SELECT photourl as photo,
                        coalesce(preferredname,firstname,'') as name
             ) r
           ),true)) as data
      from solutions.bamboo_locations l
inner join solutions.bamboo_employees e
        on e.location = l.location
  group by l.the_geom_webmercator,l.location
  order by counts asc

With this query now we have a data field with an array of objects with the display name and web address for the employee picture. Easy now to compose this in a simple infowindow where you can see the faces and names of my colleagues.

<div style="column-count:3;">
{{#data}}
<span style="display:inline-block;margin-bottom:5px;">
  <img style="height:35px;" src="{{photo}}"/> 
  <br/>
  <span style="font-size:0.55em;">{{name}}</span>
</span>
{{/data}}
</div>

{{^data}}
loading...
{{/data}}

Adding pictures and names

That’s it. You can do even more if you retrieve all the data directly from your database and render on the frontend, for example if you use D3 you probably can do fancy symbolizations and interactions.

One final note is that if you use UTF grids (like in these maps with CARTO) you need to be conservative with the amount of content you put on your interactivity because with medium and big datasets this can make your maps slow and too heavy for the front-end. On those cases you may want to change to an interactivity that works like WMS GetFeatureInfo workflow, where you retrieve the information directly from the backend when the user clicks on the map, instead of retrieving everything when loading your tiles.

Check the map below and how the interactions show the aggregated contents. What do you think of this technique? Any other procedure to display aggregated data that you think is more effective?

How a daily digest of geospatial links is distributed


TL;DR If you are interested on getting a daily digest of geospatial links subscribe to this mailing list or this atom feed. Take «daily» with a grain of salt.


Over the last six years Raf Roset, one of my favourite geonerds out there, has been sending all the cool stuff he founds about our geospatial world to Barcelona mailing list at OSGeo mailman server. He started circa 2011 sending one link per mail, but in 2013-04-03 he started to make a daily digest. A gun burst in Spanish is called Ráfaga so the joke was really at hand when someone proposed to call those digests that way.

Time passes, September 2014 and I ask Raf to send them also to Valencia mailing list, since most people there understand Catalan and the content was too good to be enjoyed only by our loved neighbours. Finally in January 2015 I decide to start translating them into Spanish and send them also to Spanish and Seville mailing lists.

Then in May I join CARTO and @jatorre thinks is a good idea if I can send them to the whole company mailing list so after some weeks I stop translating them into Spanish. Since that day I only do it English, trying to follow Raf lead everyday translating his mails and forwarding them to CARTO internal mailing list and the rest of the OSGeo ones.

Also at June I decided to put those mails in a simple website so the Ráfagas would also be accessible on GitHub and a static jekyll website so anyone could use the Atom feed to reach them.

Final chapter, in July I also decide to create a dedicated mailing list just for those people who are only interested in receiving those digest mails, obviously thinking in a broader audience, not just my fellow friends from Spain. I think at some point I will stop sending them to the Spanish lists because normally Ráfagas don’t fire any discussion and I’m sending the same message to three lists. To be fair they sometimes provoke discussions at CARTO mailing list. By the way I’m almost certain the full team has a filter to move them to their archives and they think I’m just an annoying spammer (a couple of times I’ve changed the subject just to troll them xDDD).

To conclude I want to post here my daily Ráfagas experience:

  • Raf is an early bird and sends the digest in the morning, I copy the contents into a shared Google Doc where a group of collaborators help me on translating the content. It may seem not a lot of effort, but doing this every single day needs a team. Really.
  • I go to my favorite text editor, put the translated content into a new file and start a local server to check the website renders properly.
  • If everything is OK I copy the rendered content and send it to CARTO and OSGeo mailing lists
  • I commit and Push to the GitHub repo so the website is updated along with the feed.
  • I archive Raf’s mail from my inbox.

Creating a Ráfaga

That’s it. Raf you are a formidable example of perseverance and I hope you’ll have the energy to keep giving us all those contents for many years. Thanks mate!

Creating a collaborative photo map: From Flickr to CARTO with Amazon Lambda


Phew, it’s been almost two years since my last techie blog post. I know I know, blame on me, I should’ve been writing more here but at least I did some nice posts at CARTO blog. Anyway, It’s Christmas today and because Internet is my playground and any piece of data I can put on a map can be a toy I spent a few hours having some geeky fun.

A few days ago a friend asked I knew any service to create a map of pictures in a collaborative environment. I thought maybe a trendy photo service like Instagram would be a good fit but it happens it’s super restrictive for developers so I headed to the good old Flickr. Flickr is one of those services that are really developer friendly, has a ton of super cool features and a decent mobile application and still, for some reason, it’s loosing its traction. Sad.

Anyway, Flickr has groups so a number of individuals can share geolocated pictures and they can display it on a map but sincerely, it has a very bad interface so probably we can do something better with CARTO. The issue then is how can we maintain an updated map in CARTO from a Flickr Group?

I’m a big fan of unmanaged services. I know there are people that love to maintain their servers but I’m not one of them. If I have to publish a website I try to use something static like uploading the site to Amazon S3 (i.e. my own website) or even better, use Github Pages like the Geoinquietos website. In this case not so long ago the only option to build an application to solve this issue was going to a PaaS service like Heroku, Amazon Beanstalk or Google App Engine, but they are meant for big applications typically involving a database and in general an architecture prepared for bigger things than this simple requirement. Over the last two years a new approach has emerged, a type of service that provides an automatically managed infrastructure to define small functions where each one is aimed to do a single functionality. They only live while they are being executed and afterwards the server is shut down. Amazon Lambda was the first of it’s class but recently also appeared Google Cloud Functions. On both services you can write your function in different languages (Python, Java, NodeJS, even PHP) and they can be triggered from a HTTP call or schedule its execution periodically.

As everything with Amazon, configuration from their website can be difficult and using it from the command line can be heroic. But it was a matter of time that something like Zappa would appear. Zappa is an application that makes deploying Python functions to Lambda dead easy. You basically configure a few settings and code your function and it takes care of the full cycle of deploying, versioning and even you can tail the logs from the cloud into your console in real time.

So to make this as short as possible, I coded a Lambda function that is exposed as a url acting as a proxy to Flickr API. This proxy will take URL arguments (or use some defaults) to retrieve photos information and will output them as a valid GeoJSON file. This allows me to create a CARTO Synchronized Table that updates every hour for example and retrieves the last pictures sent to a group (up to 500, as a Flickr API limitation). This dataset can then be used to create a BUILDER dashboard to present the pictures as nicely as possible.

2016-12-25_225604-selection

Map for the last 500 pictures of the “Your Best Shot of 2016” group on Flickr

Additionally, on this map I kind of reverse geocoded image locations using a world borders dataset so I was able to add a country widget. Apart from that and a bit of CartoCSS to reproduce Flickr logo, the dashboard is quite simple. If you click on any of the images the pop up highlights the image (I love this feature) and you can go and visit the picture page in Flickr.

2016-12-25_225653-selection

Pop up with the picture

But there are other methods on Flickr that returns photos, you can create a map of an account public uploads, or a map of the most interesting photos of the day, by photoset, etc. etc. All using the same proxy!!

I’ve created a github repo with the source code of this proxy (just around 130 lines of code) and more detailed instructions on how to set up your environment to deploy your own version of it on your account and use it in your own integrations. I have more ideas that I want to explore and I’ll try to share it here when I do them.

  • Leverage the Foursquare real time API to create a dataset in CARTO that is automatically updated every time I do a check-in
  • How to configure a scheduled task using the CARTO Engine SQL API. This is a super common use case when you develop projects with CARTO.
  • Create a CARTO Engine proxy to allow anonymous users to perform some tasks only accessible by default to authenticated users.

What do you think of this approach? Have you used Lambda for any other interesting use cases? Do you want me to continue posting on this topic about the other ideas I have? Feel free to comment here or reach me on twitter.

III Edición Geocamp ES


A través de Geoinquietos Sevilla

El próximo 20 de junio se celebrará en Sevilla la tercera edición de la versión española de la Geocamp este año organizada por Geoinquietos Sevilla.

¿Qué es una Geocamp?

– Una desconferencia sobre temática GEO y software libre.
– Evento basado en el concepto de Barcamp y que se viene celebrando desde hace tiempo en Portugal.

El mundo de las tecnologías de la información geográfica (TIG) y el software libre está teniendo cada vez más peso en el mundo en que vivimos, tanto desde el punto de vista personal como profesional. Muestra de eso, en últimos años, se está generando un movimiento muy significativo formado por gente con grandes inquietudes sobre estos temas, a la par que están apareciendo cada vez más  profesionales del sector que participan en proyectos y jornadas de carácter nacional e internacional.

A partir de una de estas participaciones, concretamente en la Geocamp del año 2012 en Campo Maior (Portugal), nace la idea de crear un evento similar aquí en España, donde compartir experiencias y conocimientos sobre temas geo y software libre. Aquí está el germen de la Geocamp española.

Organizada por un grupo de Geoinquietos y apoyados por un conjunto de empresas e instituciones, la Geocamp es un evento con formato desconferenciaque intenta favorecer la participación y la colaboración, puesto que serán los propios participantes de la Geocamp los que conformen el programa de charlas el mismo día de su celebración, justo antes de comenzar. Por supuesto, está garantizada la participación de grandes nombres del mundo del software libre y de la geomática a nivel nacional e internacional.

La Geocamp 2015, que viene precedida de sus anteriores ediciones, realizadas en La Coruña y Vigo, respectivamente, se celebrará en el Parque del Alamillo, parque metropolitano que discurre entre el municipio de Santiponce y la propia ciudad de Sevilla y es de acceso libre (previo registro) y gratuito. Además, se ofrecerá a los asistentes coffe break y almuerzo para reponer fuerzas, regalos, etc., así como alguna sorpresa final. Todo esto gracias a los patrocinadores del evento.

Patrocinadores III Geocamp.es

Para estar atentos a todas las novedades sobre la Geocamp 2015 está disponible su portal web (http://geocamp.es/) y el blog de Geoinquietos (http://geoinquietos.blogspot.com.es/), así como sus perfiles de Twitter y Facebook.

Página de registro: http://geocampes2015.eventbrite.es/

¡Os esperamos en Sevilla!

Fuente de la foto original: https://flic.kr/p/ambxAZ

Dear OSM, just chill out.


This is kinda a reply to Gary Gale’s “Dear OSM, it’s time to get your finger out“. The more I read that, the less sense it makes to me.


Update 2015-04-30: Read also Richard Fairhurst’s Realpolitik and the OpenStreetMap License and Simon Poole’s Is OSM business unfriendly?

I think of myself as a Linux nerd. I consider myself a hacker. And I’ve spoken so many times about open/libre licensing in conferences the issue became boring.

A couple of years ago, a psychologist asked me a question as part of a job interview: What makes you angry?. And my response was clear: Things not working and logical fallacies. So my brain hurt a little when I read these particular sentences in Gary’s blog post:

There are really only three sources of global mapping […]: NAVTEQ, TeleAtlas , and OpenStreetMap. […]

Surely now is the moment for OpenStreetMap to accelerate adoption, usage and uptake? But why hasn’t this already happened?

See, a tiny part of my brain screams “fallacy“. «OpenStreetMap has things in common with NAVTEQ and TeleAtlas, ergo it has to play in the same field and by the same rules as NAVTEQ and TeleAtlas».

Maybe OSM was given birth by SteveC to cover the lack of affordable datasources, and then a way for him and his VC-fueled CloudMade to compete with them. But for me and a whole lot of nerds, OSM basically was, and still is, a playground where we can run database queries all night long. OSM is a wholly different animal.

In 2010 I argued that Geo businesses and Geo hackers are playing the same game, but with different goals, which makes for an interesting game; a game in which it would be foolish to think that the opponent will play for the same goal as you. You have to play this game to maximize your score, which is not a synonim of decreasing the opponent’s score.

In other words: when I put something into OSM, I don’t frakking care what happens to NAVTEQ or TeleAtlas. The same way when I cook something for friends I don’t care what happens to the local pizza joint.

See, maps are an infrastructure. In Spanish GIS circles, specially those around the Spanish NSDI, cartography is often called “the infrastructure of infrastructures” You need maps to plan roads, power lines, land zoning. You need stupidly accurate maps to collect taxes based on how many centimeters square your house has, or to give out grants based on exactly how many olive trees you own.

During the late 2000’s I heard a lot of criticism about OSM. But most of it didn’t come from data-collecting companies – it came from public servants. “Volunteers use cheap GPS with low accuracy, OSM will never be accurate”, they said. “The OSM data model cannot be loaded into ArcGIS and won’t be of any use”, they said. “OSM data will never be authoritative”, they said. And a few years later, this happened:

That, right there, is the Guardia Civil (who are public servants) in a freakin’ control room using OSM for freakin’ pan-european coastal border patrols. Which means OSM is a freakin’ de facto infrastructure for sovereignty.

Fact is, government agencies play a wholly different game than commercial providers. The goal of a govt’ agency (and specifically those which maintain infrastructures) is to provide a service to citizens, not to make money. As a thought exercise, think about the public servants who place the border milestones, or the ones checking road surface quality, and guess how many fucks they give about NAVTEQ or TeleAtlas.

OSM is about the ownership of infrastructure. It’s about the efficiency of copyright law. It’s all about the digital commons, dammit.

And this is where I kick a wasps’ nest by turning this post into a political one.

A capitalistic liberal will argue that infrastructure is better handled by competing private entities, renting it to citizens for a fee. But remember, I’m a Spaniard, and that means that I’ve seen infrastructures such as power lines, water companies, telcos, motorways, airports and banks privatized under the excuse of theoretically running more efficiently in private hands. We have a nice word for what happened in the real world: “expolio“, which english-speakers might translate as “plunder”.

Thanks but no, thanks. I want infrastructures to be as close to the commons as possible. Maybe it’s because I now live in the land of the dugnad, and my bias makes me see maintaining the commons as beneficial for the society as a whole.

So when you look at OSM (or at the Wikipedia, for that matter) you’re really looking at a digital common good. Which puts OSM in the same basket as other digital common goods, such as programming languages, the radioelectric spectrum, technical RFCs, or state-owned cartography. Now go read the tragedy of the digital commons.

It’s sad to live in a world where the money-making is held above many commons, even at the expense of the commons. Fortunately it’s difficult for a private entity to pollute air, water or the radioelectrical spectrum without going unnoticed, but unfortunately copyright law cares next to nothing about intellectual commons.

<rant>Please, someone explain to me how giving me intellectual ownership of something I thought about until 70 years after my death makes me think about better things; then explain to me how that reverts into the common good. </rant>


TL;DR: Dear OpenStreetMap: just chill out and don’t listen to what they say. Corporations may come and go, but a common infrastructure such as you is here to stay.

Mapping Party en Poio, Pontevedra


El sábado 11 de abril en el Centro Xaime Illa de Raxó (Poio) llega la fiesta de las fiestas, la Mapping Party Poio’15 con el objetivo de pasarlo bien mientras aumentamos los datos geográficos de la costa de Poio con una licencia libre.

http://www.openstreetmap.org/export/embed.html?bbox=-8.757382929325104%2C42.403051783861024%2C-8.752903640270233%2C42.40485013856018&layer=mapnik&marker=42.4039509676542%2C-8.755143284797668

Este taller está organizado por la asociación de software libre Xeopesca y cuenta con la colaboración del Concello de Poio y las asociaciones SCD Raxó y ACDeM Armadiña.

Horario

  • 10:00-11:00 Presentación de OSM, organización de equipos para la zona a cartografiar.
  • 11:00-14:00 Trabajo de campo con explicaciones de como emplear osmAndroid.
  • 14:00 -16:00 Comida.
  • 16:00-20:00 Trabajar con las computadoras para el volcado de los datos OSM
  • 20:00-20:30 Clausura del curso.

Asistentes

El número de asistentes será de 25. La selección de los candidatos se realizará por orden de inscripción. Se recomienda la disposición de cualquiera de los siguientes dispositivos:  GPS, teléfono con GPS y cámara digital.

Formulario de Inscripción

Para inscribirse a Mapping Party Poio’15 cubre el formulario. (ver aquí) (poio.xeopesca.gal) .

Material fungible

Se hará entrega a cada uno de los asistentes de un bloc de notas, un bolígrafo y un lápiz.

Redes SociaLes

Establecemos el hashtag #mappingpartypoio para seguir el evento a través de las redes sociales. Además también puedes seguir la  Mapping Party Poio’15 a través de twitter mediante el hashtag #mappingpartypoio o en la  página de facebook de XeoPesca.

 

SITIO WEB OFICIAL

http://poio.xeopesca.gal/