From big tape to Big data

From big tape to big data

Francesco Vuolo

The systematic acquisition of satellite images of land surface started in 1972 with the U.S. Landsat program. With this data-stream, it was finally possible to monitor and the study soils and vegetation. However, with limited on-board storage capacity (using video tape recorders), images could be collected and downlinked only when the satellites were in sight of ground receiving stations (initially only present on the U.S. territory). This enormously limited the spatial coverage. Starting from 1975, other countries had access to the technology to communicate with the Landsat mission satellites and they could autonomously collect data acquired over their respective regions. This allowed a broader land coverage but it also scattered the data over different archives. Only in 2010, the Landsat Global Archive Consolidation started to bring together as many scenes as possible under one global collection. The process is ongoing and more that 5 million Landsat scenes acquired over the last 50 years have been reconciled and made available.

In January 2003, I was working at my Master´s Thesis to complete the University studies in Agricultural Science and Technologies in Naples. My task was to analyze satellite image data to study the irrigation water use in orange orchards in Sicily. The Landsat receiving stations for my region were operated by the European Space Agency (ESA) and were located in Fucino and Matera in Italy and in Kiruna in Sweden.

To request new image data, we had to submit an order to ESA and then we could receive a CD-ROM with the image files some weeks later. The day the CD-ROM arrived at our laboratory, it was an exciting moment; I opened the postal parcel, took the CD-ROM and loaded the raw image files to inspect the quality and information content. Then I started the actual pre-processing and interpretation. I was a student assistant and those galvanizing early experiences gave a clear direction to my future carrier pathway as a remote sensing scientist today.

Sometime later, the data were delivered via FTP over the Internet. We could download the image files few days after the satellite acquisitions and create new information content nearly in real-time. At the end of 2003, one of the concluding remarks of my Master´s Thesis was that “remote sensing from satellite was a technology mature enough to provide operational services for water management”. We were already discussing about possibilities to use the results of our data analysis to support water managers and farmers in their decision-making process. I was very impatient and I wanted this to happen soon.

The technological gap was closing but it was still too big for applications. My supervisor reminded me of his experience: just 10 years earlier, with Internet and Web technologies in their infancy (e.g. no data preview was available) ordering was based on the ephemeris of Landsat, actual cloud cover was a “last minute surprise” before ordering the image. Once received the big magnetic tapes, he had to travel to a specialized computer center to read data and expensive printing equipment was necessary to print and visualize a color composite image.

In 2005 (using one of the first 3G smartphones - the NOKIA 6633), we developed a processing chain that would generate graphical report and text data that we were “shipping” to farmers via MMS and SMS on a weekly basis. Their feedback was very useful and it helped us to move forward in the development process.  

Since then, the availability of operational services that provide satellite-based information to support farmers has grown extensively and I am proud of having being part of this small transformation in the Digital Age.

I recognize at least three main paradigm shifts that made this possible. First, the availability of free and open-access satellite data starting from 2008. With the opening of the Landsat data archive, the data use has increased 20-fold and with it the development and availability of open-source software solutions. Secondly, the evolution of the Internet and of information technologies that made exploring and accessing data archives possible. Finally, the advent of the Sentinels, defined the “game changer”, the constellation of satellites of the European Commission’s Copernicus programme. A totally new concept in image data quality, in frequency of acquisition and in availability of toolsets that allows the community to focus on harnessing information rather than accessing data and pre-processing.

With most of the Sentinel satellites in operation, the volume of fresh image data acquired every day is growing at an impressive rate. For example, the optical Sentinel-2 mission alone brings over 4 Terabytes of new products every day. Fast Internet connectivity is not sufficient anymore to download the data and process them locally. At some point (late 2015), satellites were giving us more information than we could actually collect at our laboratory and analyze. A new shift was necessary and we moved to cloud processing: we brought our algorithms to the data with the development of the first data service platform for obtaining Sentinel-2 value-added products, specifically designed to monitor vegetation. This platform is running on the Earth Observation Data Center (EODC), funded in 2016 by several Austrian institutions and private companies, fully dedicated and specialized in harvesting, archiving and maintaining Big Data. Building on this, we created other datasets and services exploited in other projects such as LANDSUPPORT for land management in the European policy context (https://www.landsupport.eu/) or COALA for irrigation and fertilizer management in Australia (https://www.coalaproject.eu/).

We have been in this digital revolution over the last 18 years and we have contributed to create a number of new concepts and innovative products. The term digital farming is now entering into our daily vocabulary and there is a great deal of activity going on. We hope that farmers will benefit from the use of satellite technology for the management of their fields as much as they do with maps and GPS technologies on their smartphones. Therefore, I am positive, and I am really looking forward for this revolution to reach the final users.

Francesco Vuolo

_{I would like to thank Guido D´Urso, Werner Schneider, Laura Essl, Markus Immitzer and Clement Atzberger for providing valuable comments and information.} 

HP minicomputer system

The HP minicomputer (year 1984) that was used to read and process large data sets in the early 1980s. The HP minicomputer used at the Institute of Geomatics at BOKU was the machine model 1000 A900. It was based on the new "Magic" processor and it was used along with a Grinnell image processing computer (GMR 275). The “Magic” processor incorporated a pipeline data path (implementing instruction-level parallelism within a single processor) and performed at 3 Millions of instructions per second (MIPS). For comparison, a modern computer processor performs more than 2,000,000 MIPS. At that time the 1000 A900 was the fastest minicomputer in the world. According to the HP museum data, the computer was so fast that special US government export restrictions were imposed. Only 16 countries had direct access to this technology and other countries required an individual evaluation to obtain a license. The Institute of Geomatics had the first system of this type in Austria and it took one year to set it up.