Category Archives: development

Aggregating points: JSON on SQL and loops on infowindows


NOTE: I’ll use CARTO but you can apply all this to any webmapping technology backed by a modern database.

Get all the data

So we start with the typical use case where we have a one to many relationship like this:

    select e.cartodb_id,
           e.displayname,
           e.division,
           e.photourl,
           l.cartodb_id as locaction_id,
           l.location,
           l.the_geom_webmercator
      from locations l
inner join employees e
        on e.location = l.location
  order by location

Easy peasy, we have a map with many stacked points. From here you can jump to this excellent post by James Milner about dense point maps. My example is not about having thousands of scattered points that at certain zoom levels overlap. Mine is a small set of locations but many points “stacking” on them. In this case you can do two things: aggregate or not. When you aggregate you pay a prize for readability: reducing all your data to those locations and maybe using visual variables to show counts or averages or any other aggregated value and finally try to use the interactivity of your map to complete the picture.

So at this point we have something like this map, no aggregation yet, but using transparency we can see where CARTO has many employees. We could also use a composite operation instead of transparency to modify the color of the stacked points.

Stacking points using transparency

Stacking points using transparency

Aggregate and count

OK, let’s do a GROUP BY the geometry and an aggregation like counting. At least now we know how many people are there but that’s all, we loose the rest of the details.

    select l.the_geom_webmercator,
           min(e.cartodb_id) as cartodb_id,
           count(1) as counts
      from locations l
inner join employees e
        on e.location = l.location
  group by l.the_geom_webmercator
Grouping by location and counting

Grouping by location and counting

Aggregate one field

But in my case, with CARTO we have PostgreSQL at hand so we can do way more than that. PostgreSQL has many many cool features, handling JSON types is one of them. Mix that with the fact that almost all template systems for front-end applications allow you to iterate over JavaScript Objects and you have a winner here.

So we can combine the json_agg function with MustacheJS iteration over objects to allow rendering the names of our employees.

    select l.the_geom_webmercator,
           min(e.cartodb_id) as cartodb_id,
           l.location,
           json_agg(e.firstname) as names, -- JSON aggregation
           count(1) as counts
      from locations l
inner join employees e
        on e.location = l.location
  group by l.the_geom_webmercator,l.location

And this bit of HTML and Mustache template to create a list of employees we can add to the infowindow template:

<ul style="margin:1em;list-style-type: disc;max-height:10em;">
{{#names}}<li class="CDB-infowindow-title">{{.}}</li>{{/names}}
</ul>
{{^names}}loading...{{/names}}

List of employees on the infowindow

We could do this without JSON types, composing all the markup in the SQL statement but that’s generating quite a lot of content to move to the frontend and of course making the whole thing way harder to maintain.

Aggregate several fields

At this point we can repeat the same function for the rest of the fields but we need to iterate them separatedly. It’d be way better if we could create JSON objects with all the content we want to maintain in a single output field we could iterate on our infowindow. With PostgreSQL we can do this with the row_to_json function and nesting an inner double SELECT to give the properties names. We can use directly row_to_json(row(field1,field2,..)) but then our output fields would have generic names.

    select l.the_geom_webmercator,
           min(e.cartodb_id) as cartodb_id,
           l.location,           
           count(1) as counts,
           json_agg(row_to_json((
             SELECT r
               FROM (
                 SELECT photourl as photo,
                        coalesce(preferredname,firstname,'') as name
             ) r
           ),true)) as data
      from solutions.bamboo_locations l
inner join solutions.bamboo_employees e
        on e.location = l.location
  group by l.the_geom_webmercator,l.location
  order by counts asc

With this query now we have a data field with an array of objects with the display name and web address for the employee picture. Easy now to compose this in a simple infowindow where you can see the faces and names of my colleagues.

<div style="column-count:3;">
{{#data}}
<span style="display:inline-block;margin-bottom:5px;">
  <img style="height:35px;" src="{{photo}}"/> 
  <br/>
  <span style="font-size:0.55em;">{{name}}</span>
</span>
{{/data}}
</div>

{{^data}}
loading...
{{/data}}

Adding pictures and names

That’s it. You can do even more if you retrieve all the data directly from your database and render on the frontend, for example if you use D3 you probably can do fancy symbolizations and interactions.

One final note is that if you use UTF grids (like in these maps with CARTO) you need to be conservative with the amount of content you put on your interactivity because with medium and big datasets this can make your maps slow and too heavy for the front-end. On those cases you may want to change to an interactivity that works like WMS GetFeatureInfo workflow, where you retrieve the information directly from the backend when the user clicks on the map, instead of retrieving everything when loading your tiles.

Check the map below and how the interactions show the aggregated contents. What do you think of this technique? Any other procedure to display aggregated data that you think is more effective?

Creating a collaborative photo map: From Flickr to CARTO with Amazon Lambda


Phew, it’s been almost two years since my last techie blog post. I know I know, blame on me, I should’ve been writing more here but at least I did some nice posts at CARTO blog. Anyway, It’s Christmas today and because Internet is my playground and any piece of data I can put on a map can be a toy I spent a few hours having some geeky fun.

A few days ago a friend asked I knew any service to create a map of pictures in a collaborative environment. I thought maybe a trendy photo service like Instagram would be a good fit but it happens it’s super restrictive for developers so I headed to the good old Flickr. Flickr is one of those services that are really developer friendly, has a ton of super cool features and a decent mobile application and still, for some reason, it’s loosing its traction. Sad.

Anyway, Flickr has groups so a number of individuals can share geolocated pictures and they can display it on a map but sincerely, it has a very bad interface so probably we can do something better with CARTO. The issue then is how can we maintain an updated map in CARTO from a Flickr Group?

I’m a big fan of unmanaged services. I know there are people that love to maintain their servers but I’m not one of them. If I have to publish a website I try to use something static like uploading the site to Amazon S3 (i.e. my own website) or even better, use Github Pages like the Geoinquietos website. In this case not so long ago the only option to build an application to solve this issue was going to a PaaS service like Heroku, Amazon Beanstalk or Google App Engine, but they are meant for big applications typically involving a database and in general an architecture prepared for bigger things than this simple requirement. Over the last two years a new approach has emerged, a type of service that provides an automatically managed infrastructure to define small functions where each one is aimed to do a single functionality. They only live while they are being executed and afterwards the server is shut down. Amazon Lambda was the first of it’s class but recently also appeared Google Cloud Functions. On both services you can write your function in different languages (Python, Java, NodeJS, even PHP) and they can be triggered from a HTTP call or schedule its execution periodically.

As everything with Amazon, configuration from their website can be difficult and using it from the command line can be heroic. But it was a matter of time that something like Zappa would appear. Zappa is an application that makes deploying Python functions to Lambda dead easy. You basically configure a few settings and code your function and it takes care of the full cycle of deploying, versioning and even you can tail the logs from the cloud into your console in real time.

So to make this as short as possible, I coded a Lambda function that is exposed as a url acting as a proxy to Flickr API. This proxy will take URL arguments (or use some defaults) to retrieve photos information and will output them as a valid GeoJSON file. This allows me to create a CARTO Synchronized Table that updates every hour for example and retrieves the last pictures sent to a group (up to 500, as a Flickr API limitation). This dataset can then be used to create a BUILDER dashboard to present the pictures as nicely as possible.

2016-12-25_225604-selection

Map for the last 500 pictures of the “Your Best Shot of 2016” group on Flickr

Additionally, on this map I kind of reverse geocoded image locations using a world borders dataset so I was able to add a country widget. Apart from that and a bit of CartoCSS to reproduce Flickr logo, the dashboard is quite simple. If you click on any of the images the pop up highlights the image (I love this feature) and you can go and visit the picture page in Flickr.

2016-12-25_225653-selection

Pop up with the picture

But there are other methods on Flickr that returns photos, you can create a map of an account public uploads, or a map of the most interesting photos of the day, by photoset, etc. etc. All using the same proxy!!

I’ve created a github repo with the source code of this proxy (just around 130 lines of code) and more detailed instructions on how to set up your environment to deploy your own version of it on your account and use it in your own integrations. I have more ideas that I want to explore and I’ll try to share it here when I do them.

  • Leverage the Foursquare real time API to create a dataset in CARTO that is automatically updated every time I do a check-in
  • How to configure a scheduled task using the CARTO Engine SQL API. This is a super common use case when you develop projects with CARTO.
  • Create a CARTO Engine proxy to allow anonymous users to perform some tasks only accessible by default to authenticated users.

What do you think of this approach? Have you used Lambda for any other interesting use cases? Do you want me to continue posting on this topic about the other ideas I have? Feel free to comment here or reach me on twitter.

About Antipodes Map


We’ve been pretty quiet over the last year but that doesn’t mean we’ve been unoccupied. Last summer we (Pedro and me) participated with some friends on a hackathon with a project to give to teachers from our region a tool to help them to relocate, precalculating travelling times with OSRM and some open datasets, one of them a database of schools that our government published as a spreadsheet. That gave us the chance to work and improve our knowledge on the CartoDB Platform, we used their JavaScript API to place a Leaflet map with a parametrized map where the SQL that defined the layer changed depending on user selections. The project is online with some slides with further information, all in Spanish.

De Casa al Cole map

De Casa al Cole Map

After that experience, and thanks to Pedro’s friendship with Carlos Galcerán, a Cuban GIS consultant working in New Zealand, we had the opportunity to put our brains working again for another pet project. The idea is easy, have you ever wondered who is on your antipodes? Yes, three quarters of our planet are oceans so the chance to have an inhabited antipodes is not high but here in Spain, it happens that half of the Iberian Peninsula is antipodal to New Zealand. Join that with the possibility to have data about schools on both countries and well, that’s reason enough for us to start playing. Imagine a geography class on primary school, say in the north coast of Galicia, where kids can contact their antipodal school in Christchurch and practice their English, or kiwis practicing their Spanish, both learning about our cultures, favorite sports, our differences.

We started with a dataset only for Galician schools and end up digging a national registry of schools to create a full dataset of schools for the country. That and the help of Carlos and some help from the Spanish Embassy in New Zealand, gave us all the data needed to set up two tables on CartoDB, so the last piece was just a web interface to develop. With the recent release of OpenLayers 3, and having played with it a bit before, I wanted to do something more complex. We’ll leave the technical details about data and software for another post or two. The application is available at http://antipodes.decasaalcole.com.

Antipodes Map

Antipodes Map

If you like the idea and know someone in New Zealand or Spain that could be interested, please spread the word. And of course, the data is available for reuse on CartoDB and the code is also on GitHub, ready to be reused on other lucky antipodal combinations, we’d love to see both data and software reused and improved!!

MOSKittGeo: from UML to Spatial Databases and back


Probably some of you do a real engineering process to build your GIS projects, I mean, the classical analyse-design-implement cycle. As a GIS is in fact an Information System with the (complex of course) spatial component, one should expect that the common tools and methodologies of «normal Information Systems» are available for GIS engineers but on the case of database modelling that’s not so common.

So, what happens when you want to design a GIS data model? Well, one usually had to use a “fake” geometry type on your CASE tool of choice and afterwards create the geometry columns by hand or using custom post-processes.

Until now, because my smart colleagues at Prodevelop have developed an extension to the MOSKitt software called MOSKitt Geo that adds the spatial data type to the UML and Entity-Relationship models so you can perform the regular design process on MOSKitt: you do your UML model, then you convert it to the Entity-Relationship model and then you select your target database and MOSKitt creates the DDL scripts automatically.

MOSKitt ER designer

Even more, you can also do reverse engineering of your spatial database and redesign or improve you data models using excellent MOSKitt graphical tools!

Take a look on these videos from the new Prodevelop YouTube channel to see how it works and go (and comment!) to the Carlos post about MOSKitt Geo at Prodevelop blog.

Probando SUSE Studio


SUSE studio

SUSE es una de las grandes distribuciones del Sistema Operativo GNU/LInux. Hace ya unos años que fue adquirida por Novell y que adoptó la solución de RedHat de ofrecer una versión para la comunidad y otra de pago. Bueno, el caso es que hace ya unos meses se anunció que Novell estaba preparando SUSE Studio, una aplicación web en la que se podrían generar appliances, máquinas virtuales basadas en SUSE (tanto la versión libre como la de pago). Estas versiones serían personalizadas, ya que se podrían configurar a través de una interfaz web y para más tarde descargarlas y usarlas donde quieras.

El servicio estaba en pruebas con acceso privado y solicité el alta. Pues nada, esta mañana me ha llegado el correo en el que me avisaban de ya podía darme de alta (usando OpenID por cierto) y he estado trasteado un poco.

Continue reading

Python, nubes de etiquetas y geoinformatics…


A lo largo de la semana pasada y la corriente he visto dosreferencias, en Microsiervosy en Genbeta,a dos servicios que  hacen cosas muy similares aunque noexactamente lo mismo: nubede etiquetas. Uno lo obtiene como resultado y el otrolo usa como herramienta.

El caso es que me he tomado como ejercicio de Python (de vezen cuando hay que hacer estas cosas o se te oxida la serpiente)elaborar una nube de etiqueta para un texto dado.

Continue reading

Instalando ka-Map en un XAMPP… pegándose con php_mapscript en un Apache que ya está en marcha.


Como ya os conté en su momento, tengo instalado un mapserver y un tilecache en el ordenador, lo que no os conté es que está encima de un XAMPP. Recientemente se me ha ocurrido probar ka-Map, a raíz del taller de las IIas Jornadas de SIG Libre,y cuando lo instalé, siguiendo rigurosamente las instrucciones, me encontré con  un problemilla que me ha llevado un poco resolver, sobretodo porque pese a que hay bastante gente que le ha pasado, no he sido capaz de encontrar la solución en San Google. Se trata de un problema para instalar MapScript sobre un Apache ya existente (o sobre un XAMPP).

Todo empieza con este error cuando pedimos la siguiente dirección: http://localhost/ka-map/init.php

 Warning: dl() [function.dl]: Not supported in multithreaded Web servers -  use extension=php_mapscript.dll in your php.ini in C:xampphtdocska-mapinit.php  on line 118

 Fatal error: Call to undefined function ms_newmapobj()   in C:xampphtdocska-mapinit.php on line 124

Para empezar la distribución estándar de xampp no tiene la dll php_mapscript así que vamos a la página de PHP MapScript decididísimo a descargarme lo que haga falta… te llegas a la página de PHP MapScript y descubres que, para MSW$, ahora solo se distribuye junto con ms4w,pero claro, yo ya tenía un Apache y un MapServer funcionando así que no quería instalar otro servicio Apache… cojo, la versión en zip de ms4w y rapiño la dll…

 Ojo, acordaros de modificar convenientemente el config.php de ka-Map 

 $szPHPMapScriptModule = 'php_mapscript.'.PHP_SHLIB_SUFFIX; $szPHPGDModule = 'php_gd2.'.PHP_SHLIB_SUFFIX;

… retoco el php.ini del xampp (usad el phpinfo() para saber dónde está) para que cargue php_mapscript como extensión,  reinicio Apache y …

 Warning: dl() [function.dl]: Not supported in multithreaded Web servers   - use extension=php_mapscript.dll in your php.ini in C:xampphtdocska-mapinit.php on line 118
 Fatal error: Call to undefined function ms_newmapobj()   in C:xampphtdocska-mapinit.php on line 124

vale, al parecer estoy haciendo algo mal… tras unas búsquedas en Google veo que es un mal bastante común y que no parece que nadie sepa cómo solucionarlo, así que me pongo a investigar y descubro que ms4w tiene montado PHP como FastCGI en vez de como módulo (opción por defecto en XAMPP), bueno, pues probemos si es eso…

Vamos a C:/xampp/apache/conf/extra/httpd-xampp.conf y modificamos las primeras líneas del archivo para que recen así:

 ScriptAlias /php/ "C:/xampp/php/" Action application/x-httpd-php "/php/php-cgi.exe" #LoadModule php5_module "C:/xampp/apache/bin/php5apache2.dll" AddType application/x-httpd-php-source .phps AddType application/x-httpd-php .php .php5 .php4 .php3 .phtml .phpt

si lo reiniciáis y ejecutáis “as is” lo más fácil es que os de un error de Acceso prohibido cuando intentéis ejecutar cualquier archivo php ya que por defecto C:/xampp/php/ NO tiene permisos asignados en Apache (ojo con esto que puede producir fallos GORDOS de seguridad)

 <Directory "C:/xampp/php/"> AllowOverride AuthConfig Order allow,deny Allow from all </Directory> 

reiniciamos Apache y “sorpresa”

 Warning: dl() [function.dl]: Unable to load dynamic library  'C:xamppphpextphp_mapscript.dll' - No se puede encontrar el módulo especificado.in C:xampphtdocska-mapinit.php on line 118

 Fatal error: Call to undefined function ms_newmapobj() in  C:xampphtdocska-mapinit.php on line 124

El hecho de que hayamos cambiado de error indica que estamos progresando 😀 aunque el error que indica es que no puede cargar php_mapscript.dll la realidad es un poco más compicada, lo que no puede cargar es alguna de sus dependencias pero ¿cuales? pues para contestar eso tendremos que ejecutar un programita que “escarba” las dependencias de las dll el Dependency Walker(esto lo averigüe leyendo la ayuda de Apache).

Cuando falta una dll, la podemos obtener de la versión en zip de ms4w, descomprimiendo lo que necesitemos, las dlls están todas en ms4w_2.2.7.zipms4wApachecgi-bin. Conforme vamos añadiendo dlls a C:/xampp/php/nos irá señalando nuevas faltas (refrescando Dependency walker con F5). Una forma de “acelerar” el proceso es descomprimir directamente todas las dlls que hay en el .zip en la carpeta (básicamente GDALes y PROJes)… Hay una dependencia que no es necesario resolver para que php_mapscript funcione “MSJAVA.dll“.

Fijaros que cómo PHP está montado como FastCGI no hace falta reiniciar Apache para que surtan efecto los cambios. Una vez hecho esto el mensaje de error de ka-Map es distinto, haciendo referencia a que no encuentra el mapa de ejemplo… señal de que funciona.

 Warning: [MapServer Error]: msLoadMap():  (C:xampphtdocska-map/../../gmap/htdocs/gmap75.map) in  C:xampphtdocska-mapinit.php on line 124
 Warning: Failed to open map file ../../gmap/htdocs/gmap75.map in  C:xampphtdocska-mapinit.php on line 124
 Fatal error: Call to a member function getMetaData() on a non-object in  C:xampphtdocska-mapinit.php on line 130

Bueno, pues con esto en un periquete tenéis ka-Map montado sobre un XAMPP ya existente (ojito,  again, con las configuraciones de seguridad, este es un ejemplo para un puesto de desarrollo, no de producción). Espero que os sirva la recetilla y si tenéis dudas… ya sabéis.

¡Un saludo!

PD. A mis fans, os juro que ahora mismo me pongo con la 2a parte de lo de Girona, pero como esto lo tenía a mano…