We are often not fully aware of the significance, extent and potential of the data that is continuously generated and recorded by our services, from which we can learn a lot about the infrastructure, problems, needs and how and where our services and products are used. Machine-generated and user-provided data from perfSONAR Lookup Service and analytical visualizations of perfSONAR nodes and community are used to illustrate the possibilities. A data processing pipeline is built to extract, transform, validate, supplement and visualize daily snapshots, as it could be done for any service by tapping into available JSON or XML dumps or application and component logs. At the end of the pipeline, the tweaked data are loaded into a Splunk repository. This operational intelligence tool allows to create complex dashboards and to group, select and drill down data in real time and obtain comprehensible metrics, indicators and distributions on interactive charts and maps. Splunk perfSONAR dashboards give an insight into trends or state at a specific point in time. Like OLAP or business intelligence tools, Splunk allows to dice, drill down, summarise or pivot the data by picking values and attributes, but its main strengths are the out-of-the-box understanding of application, network and security related items and formats, behind the scenes data indexing, and web-based dashboard designer. Regardless of how powerful our visualisation platform is, it cannot do much with incomplete or imprecise input. By focusing on location data, we illustrate common problems and how limited and often inaccurate location clues can be turned into almost complete location descriptors. Starting from often misspelt place names, coarsely picked coordinates, DNS names and IP addresses, we infer the location and get the missing details from Google Maps, OpenStreetMap or GeoIP, producing beautiful heat maps and maps with clickable points and clusters.
Networks · Services · People | Code of Conduct | Privacy Policy
