I presented the movie below as part of a presentation at the Safe Software FME User Conference. We had a great time and the Safe crew put on a marvelous show.
The scaling occurred automatically, spinning up FME Engine AMIs, and then shutting them down when the job queue was completed. This is one of our first examples of bringing scalable processing to difficult geospatial tasks.
Panel 1 (upper left hand corner) refers to the total number of engines in the constellation processing job.
Panel 2 (upper right hand corner) refers to the total constellation utilization percentage. The constellation is polled and when the utilization exceeds the pre-set threshold (50% in this example), it increases (doubles here) the number of engines until it reaches the pre-set maximum number of engines (64 here). The downward spikes occur when each new set of engines are added.
Panel 3 (lower left hand corner) is the average job processing time. There is an increase in velocity when the number of engines exceeds 16, which may be a function of increased overhead costs on the FME Core or bandwidth to the database.
Panel 4 (lower right hand corner) is the total number of jobs completed. 2000 jobs were submitted for this test. The job completion rate accelerates until the maximum number of engines are brought on-line.
Moreton Bay Research Station Computer Room trying to unlock the mysteries of physics-based inversion algorithms
I am still involved with various spectral mapping efforts from aircraft and space. I spent the past week at the Moreton Bay Research Station on North Stradbroke Island in Queensland, Australia with a bunch of people trying to determine sea grass distributions from imagery. Most of the time we were locked into this room –
But I was able to get out one day to look at the environment we were trying to map, with guys who actually know the environment and can tell one form of sea grass from another.
One of the sea grass species (Cymodocea serrulata) we were trying to identify from aircraft imagery.
Below we were engaged in putting the “map on the pixel” instead of the traditional other way around (I am the geek in the hat).
I rarely write about such endeavors, mostly because of time, but a thought crystallized in my mind during this trip that is relevant for what we are trying to do here. The mapping industry has created some expensive tools (consider that the LandSat follow-on is likely to be over $700 million). Yet the “last mile” of knowledge is always created by someone in the field, applying their skills and knowledge to the data coming from the survey instruments.
Our mission is to make the human part of this process more efficient and more rewarding. When we succeed in this mission, we will be creating a healthier place to live, with better funding for the experts to pursue their studies and acquisition of knowledge.
On the left, Chris Roelfsema is showing where we are on the map. On the right, we are "putting the map on the pixel".
He took the iFrame and embedded it on his site to directly brand and market the maps to his customers. He now has a fully functioning hosting and eCommerce site for a fraction of the cost and time of doing it himself.
With the release of our APIs, the Market experience can be further customized to a unique customer desired solution, generating many different ways to store, host, search, and deliver geo-enabled content. This is just one example of how we are trying to move towards platform-level services for our customers.
WeoGeo is continuously developing our scalable geo-content management technology. One of the most “visible” recent changes is the switch from ka-Map to Openlayers for displaying our pre-cached image tiles for preview and customization. The switch was driven by the need to enhance the performance and functionality of the site, as well as enhancing the ability of our developers to create a better product in the future.
Below are some of the benefits of our switch to Openlayers:
No more IFRAMEs = faster site loading, fewer HTTP requests, less memory usage (1 browser process instead of 4)
When zooming in and out, tiles from previous zoom-level are resized and will appear dithered, until new tiles load. This is an improvement over the old behavior, where zooming in would result in a blank canvas temporarily while the new zoom-level tiles loaded.
Rubber-band zoom – hold-down shift and draw a box around area of interest
Dataset selection is retained in the navigation maps grid when zooming in and out, as long as the selected dataset continues to appear in the result list.
Zooming in with the mouse wheel will keep the area under the mouse cursor at the same position, rather than zooming straight into the center.
The dataset map extents are limited to the bounds of the dataset.
We continue to focus on developing the best content library, hosting, and market place service for the mapping industry. Stay tuned for more exciting things to come.
Dave’s talk dealt with the technology required to help GIS programmers be more responsive to the needs of a larger, less GIS-focused, web community. It included the most slides that I have ever seen in 25-minute talk (140!). What struck me the most about his talk was the level of web expertise that had to be acquired before beginning to address the location or GIS issues that would best serve an end user of the web service. It makes me wonder if there might be a need for a middle-ware solution between the producers, who have high value geo-content but do not have Dave’s web expertise, and the consumers of that content. In some respect, I think that is part of what Dave’s organization provides.
My talk focused on a theme that I have been developing for the last couple of months, which is how should we index and search all the hidden geo-content that exists within the deep web storage networks of the world. Indexing and discovering this content would make the jobs of millions of professionals, who create GIS and CAD grade content a whole lot easier, as well as opening the information to a lot of other uses. My slides may be found below.
One of the take home messages that I got from this session is that the problems we see in sharing and accessing geo-content are felt by many others. I hope to contact a few of the participants to help refine the problems and hopefully “illuminate” some possible solutions.
We’re headed to WhereCampPDX on Saturday. It is an event that everyone in the office has been looking forward to attending. This is exactly the type of event that prompted our move across country.
We hope to lead a couple of sessions. The first is by Scott Becker entitled, “Intro to Open Layers, a Free Open Source JavaScript Mapping API”. Open Layers is a powerful JavaScript API for web-based mapping that provides a multitude of tools for the geo hacker, including support for many tile formats, vector data, a drawing API, as well as support for various open data standards like GeoRSS, GeoJSON, GML, KML and more. This session will attempt to introduce the participants to these capabilities and spark ideas for what they might be able to create using them. To view Open Layers in action, check out our WeoGeo Marketplace (www.WeoGeo.com), which has recently undergone an upgrade from KaMap to Open Layers.
Our other session will be led by me: “Illuminating the Dark GeoWeb”. Valuable geocontent is stored in databases, network storage drives, and desktops. These are not accessible to search engines spidering the web making the geocontent undiscoverable. This session will be an open-ended discussion on how to develop tools to index these data, as well as how to motivate the owners of the content to make these data available for discovery and sharing.
I am not sure how many people in our industry are paying attention to the meltdown of the global financial services industry, but I want to point out the following interview with Jack Dangermond:
In it, Jack describes that growth of our industry as well as his expectations of ESRI:
ESRI has been growing annually at about 10-15% worldwide for many years, and in 2008 the company is growing at a rate of 17%, said Dangermond, adding that this “is counter intuitive to the whole stock market and economic downturn.”
In addition, he gives some thought to the differences between money capital and resource capital:
In terms of the global financial turmoil, Dangermond said “people today are very nervous about what is happening on Wall Street and what I am more concerned about is ecological sustainability and global warming issues on the planet, because they are not something that you can go to the bank and borrow more money from. The real sustainability issue and the real economic foundation is nature’s capital, it is not artificial money capital.”
Now, I’ll be the first to admit that I don’t see eye-to-eye with Jack on a lot of things. Money and intellectual capital are important for transitioning goods and services towards a more sustainable path. But for the most part, I think he is dead-on about the opportunities for our industry and the importance of our field.
Hang in there people, we’re in for a wild ride for the next 18-24 months. Keep your heads down, work hard, and keep pushing the envelope, and we will change the world.
I am trying to synthesize my thoughts on the GeoWeb 2008 conference, as well as things that have been coming at us since the Safe Software press release. One of the thoughts that continues to rise to the surface is that the GeoWeb is not an independent section of the internet, but rather a niche location in the web; a niche section that perhaps is being rolled into its core.
I think I heard a similar statement from Ron Lake at the GeoWeb conference (”The GeoWeb is the Internet”). James Fee has picked up the theme in a recent post. The comments to that post seem to have echoes of the neogeography vs. GIS clashes. Yet, there does not appear to be any of the vitriol associated with this thread compared to past debates. Maybe we are moving past the divisions and into the realm of synergy between the different focuses of the groups.
And then bang, you see an announcement by ESRI and Microsoft Virtual Earth offering Virtual Earth products inside ArcGIS Server. Talk about a fusion of GIS and neogeography. You will have to purchase a Virtual Earth license, so it fits within Microsoft’s tradition licensing revenue model, but still, Virtual Earth is more about the future of yellow pages advertising than about GIS or GeoWeb services. At least according to Microsoft’s Vincent Tao at the Location Intelligence 2008 conference.
So where do we go from here? The internet is “evolving”. It started out as a robust means of communicating and sharing supercomputer resources (see ARPANET). It has become a ubiquitous communication, processing, and storage array that touches nearly every aspect of our daily lives. In its evolution to date, it has rolled through many businesses (e.g. telecommunication, financial, advertising, etc.), upending their business models, to create new services, products, and business opportunities. I think the spatial industry is just the next industry to overwhelmed (shaken? stirred? smashed?) by the evolution of this omnipresent technology in our lives.
Location awareness is just the next information stream to be absorbed by the massive data ingestion services that reside within the internet. If you own a mobile phone, your position on the planet is already trackable to within meters (see Enhanced 911). New business models are started daily on using these feeds as new Location-Based Services (LBS). As a consumer, geo-spam is a concern, but you can always turn-off the buzzer. But new models built on real-time crowd dynamics or “mobbing” offer a whole new approach to targeted marketing and advertising (see Peter Batty’s presentation at GeoWeb as an example).
Would you have seen this coming 10 years ago? It’s clear Microsoft did not see Google coming, and they had some very smart people trying to predict the future. I think we in the spatial data industry are facing the fertile grounds that lead to similar revolutionary technologies. Such environments lead to the classic innovator’s dilemma for business organizations – how do we maintain our current revenue stream, while creating new, faster growing opportunities so that we do not end up as the last maker of horse drawn carriages?
We have a packed week with presentations at GeoWeb, participation at OSCON, and a sponsorship at FOSCON. Our own Brian Artiaco will be competing in the FOSCON Live Coding Competition. I’ll try to keep blogging through GeoWeb, but if it happens it will be short and sweet.
