Splunk

Finding Exoplanets with Splunk

Finding Exoplanets with Splunk

Splunk is a software platform designed to search, analyze and visualize machine-generated data, making sense of what, to most of us, looks like chaos.

Ordinarily, the machine data used by Splunk is gathered from websites, applications, servers, network equipment, sensors, IoT (internet-of-things) devices, etc, but there’s no limit to the complexity of data Splunk can consume.

Splunk specializes in Big Data, so why not use it to search the biggest data of all and find exoplanets?

What is an exoplanet?

An exoplanet is a planet in orbit around another star.

The first confirmed exoplanet was discovered in 1995 orbiting the star 51 Pegasi, which makes this an exciting new, emerging field of astronomy. Since then, Earth-based and space-based telescopes such as Kepler have been used to detect thousands of planets around other stars.

At first, the only planets we found were super-hot Jupiters, enormous gas giants orbiting close to their host stars. As techniques have been refined, thousands of exoplanets have been discovered at all sizes and out to distances comparable with planets in our own solar system. We have even discovered exomoons!

 

How do you find an exoplanet?

Imagine standing on stage at a rock concert, peering toward the back of the auditorium, staring straight at one of the spotlights. Now, try to figure out when a mosquito flies past that blinding light. In essence, that’s what telescopes like NASA’s TESS (Transiting Exoplanet Survey Satellite) are doing.

The dip in starlight intensity can be just a fraction of a percent, but it’s enough to signal that a planet is transiting the star.

Transits have been observed for hundreds of years in one form or another, but only recently has this idea been applied outside our solar system.

Australia has a long history of human exploration, starting some 60,000 years ago. In 1769 after (the then) Lieutenant James Cook sailed to Tahiti to observe the transit of Venus across the face of the our closest star, the Sun, he was ordered to begin a new search for the Great Southern Land which we know as Australia. Cook’s observation of the transit of Venus used largely the same technique as NASA’s Hubble, Kepler and TESS space telescopes but on a much simpler scale.

Our ability to monitor planetary transits has improved considerably since the 1700s.

NASA’s TESS orbiting telescope can cover an area 400 times as broad as NASA’s Kepler space telescope and is capable of monitoring a wider range of star types than Kepler, so we are on the verge of finding tens of thousands of exoplanets, some of which may contain life!

How can we use Splunk to find an exoplanet?

 Science thrives on open data.

All the raw information captured by both Earth-based and space-based telescopes like TESS are publicly available, but there’s a mountain of data to sift through and it’s difficult to spot needles in this celestial haystack, making this an ideal problem for Splunk to solve.

While playing with this over Christmas, I used the NASA Exoplanet Archive, and specifically the PhotoMetric data containing 642 light curves to look for exoplanets. I used wget in Linux to retrieve the raw data as text files, but it is possible to retrieve this data via web services.

MAST, the Mikulski Archive for Space Telescopes, has made available a web API that allows up to 500,000 records to be retrieved at a time using JSON format, making the data even more accessible to Splunk.

Some examples of API queries that can be run against the MAST are:

The raw data for a given observation appears as:

Information from the various telescopes does differ in format and structure, but it’s all stored in text files that can be interrogated by Splunk.

Values like the name of the star (in this case, Gliese 436) are identified in the header, while dates are stored either using HJD (Heliocentric Julian Dates) or BJD (Barycentric Julian Dates) centering on the Sun (with a difference of only 4 seconds between them).

Some observatories will use MJD which is the Modified Julian Date (being the Julian Date minus 2,400,000.5 which equates to November 17, 1858). Sounds complicated, but MJD is an attempt to simplify date calculations.

Think of HJD, BJD and MJD like UTC but for the entire solar system.

One of the challenges faced in gathering this data is that the column metadata is split over three lines, with the title, the data type and the measurement unit all appearing on separate lines.

The actual data captured by the telescope doesn’t start being displayed until line 138 (and this changes from file to file as various telescopes and observation sets have different amounts of associated metadata).

In this example, our columns are…

  • HJD - which is expressed as days, with the values beyond the decimal point being the fraction of that day when the observation occurred
  • Normalized Flux - which is the apparent brightness of the star
  • Normalized Flux Uncertainty - capturing any potential anomalies detected during the collection process that might cast doubt on the result (so long as this is insignificant it can be ignored).

Heliocentric Julian Dates (HJD) are measured from noon (instead of midnight) on 1 January 4713 BC and are represented by numbers into the millions, like 2,455,059.6261813 where the integer is the days elapsed since then, while the decimal fraction is the portion of the day. With a ratio of 0.00001 to 0.864 seconds, multiplying the fraction by 86400 will give us the seconds elapsed since noon on any given Julian Day. Confused? Well, your computer won’t be as it loves working in decimals and fractions, so although this system may seem counterintuitive, it makes date calculations simple math.

We can reverse engineer Epoch dates and regular dates from HJD/BJD, giving Splunk something to work with other than obscure heliocentric dates.

  • As Julian Dates start at noon rather than midnight, all our calculations are shifted by half a day to align with Epoch (Unix time)
  • The Julian date for the start of Epoch on CE 1970 January 1st 00:00:00.0 UT is 2440587.500000
  • Any-Julian-Date-minus-Epoch = 2455059.6261813 - 2440587.5 = 14472.12618
  • Epoch-Day = floor(Any-Julian-Date-minus-Epoch) * milliseconds-in-a-day = 14472 * 86400000 = 1250380800000
  • Epoch-Time = floor((Any-Julian-Date-minus-Epoch – floor(Any-Julian-Date-minus-Epoch)) * milliseconds-in-a-day = floor(0. 6261813 * 86400000) = 10902064
  • Observation-Epoch-Day-Time = Epoch-Day + Epoch-Time = 1250380800000 + 10902064 = 1250391702064

That might seem a little convoluted, but we now have a way of translating astronomical date/times into something Splunk can understand.

I added a bunch of date calculations like this to my props.conf file so dates would appear more naturally within Splunk.

[exoplanets]

SHOULD_LINEMERGE = false

LINE_BREAKER = ([\r\n]+)

EVAL-exo_observation_epoch = ((FLOOR(exo_HJD - 2440587.5) * 86400000) + FLOOR(((exo_HJD - 2440587.5) - FLOOR(exo_HJD - 2440587.5))  *  86400000))

EVAL-exo_observation_date = (strftime(((FLOOR(exo_HJD - 2440587.5) * 86400000) + FLOOR(((exo_HJD - 2440587.5) - FLOOR(exo_HJD - 2440587.5))  *  86400000)) / 1000,"%d/%m/%Y %H:%M:%S.%3N"))

EVAL-_time = strptime((strftime(((FLOOR(exo_HJD - 2440587.5) * 86400000) + FLOOR(((exo_HJD - 2440587.5) - FLOOR(exo_HJD - 2440587.5))  *  86400000)) / 1000,"%d/%m/%Y %H:%M:%S.%3N")),"%d/%m/%Y %H:%M:%S.%3N")

Once date conversions are in place, we can start crafting queries that map the relative flux of a star and allow us to observe exoplanets in another solar system.

Let’s look at a star with the unassuming ID 0300059.

sourcetype=exoplanets host="0300059"

| rex field=_raw "\s+(?P<exo_HJD>24\d+.\d+)\s+(?P<exo_flux>[-]?\d+.\d+)\s+(?P<exo_flux_uncertainty>[-]?\d+.\d+)" | timechart span=1s avg(exo_flux)

And there it is… an exoplanet blotting out a small fraction of starlight as it passes between us and its host star!

What about us?

While curating the Twitter account @RealScientists, Dr. Jessie Christiansen made the point that we only see planets transit stars like this if they’re orbiting on the same plane we’re observing. She also pointed out that “if you were an alien civilization looking at our solar system, and you were lined up just right, every 365 days you would see a (very tiny! 0.01%!!) dip in the brightness that would last for 10 hours or so. That would be Earth!”

There have even been direct observations of planets in orbit around stars, looking down from above (or up from beneath depending on your vantage point). With the next generation of space telescopes, like the James Webb, we’ll be able to see these in greater detail.

 

Image credit: NASA exoplanet exploration

Next steps

From here, the sky’s the limit—quite literally.

Now we’ve brought data into Splunk we can begin to examine trends over time.

Astronomy is BIG DATA in all caps. The Square Kilometer Array (SKA), which comes on line in 2020, will create more data each day than is produced on the Internet in a year!

Astronomical data is the biggest of the Big Data sets and that poses a problem for scientists. There’s so much data it is impossible to mine it all thoroughly. This has led to the emergence of citizen science, where regular people can contribute to scientific discoveries using tools like Splunk.

Most stars have multiple planets, so some complex math is required to distinguish between them, looking at the frequency, magnitude and duration of their transits to identify them individually. Over the course of billions of years, the motion of planets around a star fall into a pattern known as orbital resonance, which is something that can be predicted and tested by Splunk to distinguish between planets and even be used to predict undetected planets!

Then there’s the tantalizing possibility of exomoons orbiting exoplanets. These moons would appear as a slight dip in the transit line (similar to what’s seen above at the end of the exoplanet’s transit). But confirming the existence of an exomoon relies on repeated observations, clearly distinguished from the motion of other planets around that star. Once isolated, the transit lines should show a dip in different locations for different transits (revealing how the exomoon is swinging out to the side of the planet and increasing the amount of light being blocked at that point).

Given its strength with modelling data, predictive analytics and machine learning, Splunk is an ideal platform to support the search for exoplanets.

Find out more

If you’d like to learn more about how Splunk can help your organization reach for the stars, contact one of our account managers.

Our team on the case

Document as you go.

Peter Cawdron

Consultant

Length of Time at JDS

5 years

Skills

ServiceNow, Loadrunner, HP BSM, Splunk.

Workplace Passion

I enjoy working with the new AngularJS portal in ServiceNow.

Our Splunk stories

Posted by Jillian Hunter in Blog, Splunk, Tech Tips, 0 comments
 

 

Splunk .conf18 – Splunk Next: 10 Innovations

As part of .conf18 and in the balmy Florida weather surrounded by theme parks, JDS was keen to hear more about what’s coming next from Splunk – namely Splunk Next.

Splunk CEO Doug Merritt announced a lot of new features released with Splunk 7.2 which you can read about in our earlier post (Splunk .conf recap). He also talked about Splunk’s vision of creating products that reduce the barriers to getting the most out of your data. As part of that vision he revealed Splunk Next which comprises a series of innovations that are still in the beta phase.

Being in beta, these features haven’t been finalised yet, but they showcase some of the exciting things Splunk is working towards. Here are the Top 10 innovations that will help you get the most out of your data:

  1. SplunkDeveloper Cloud – develop data-driven apps in the cloud, using the power of Splunk to provide rich data and analytics.
  2. SplunkBusiness Flow – an analytics-driven approach to users’ interactions and identify ways to optimise and troubleshoot. This feature generates a process flow diagram based solely on your index data, and shows you what users are doing, and where you can optimise the system to make smarter decisions.
  3. SplunkData Fabric Search – with the addition of an Apache Spark cluster, you can now search over multiple disparate Splunk instances with surprising speed. This federated search will allow you to search trillions of events and metrics across all your Splunk environments.
  4. SplunkData Stream Processor - a GUI interface to allow you to test your data ingestion in real-time without relying on config files. You can mask data, send it to various indexes or even different Splunk instances, all from the GUI.
  5. SplunkCloud Gateway – a new gateway for the Splunk Mobile App, get Splunk delivered to your mobile device securely.
  6. SplunkMobile – a new mobile interface for Splunk, which shows dashboards in a mobile-friendly format. Plays nicely with the Cloud Gateway.
  7. SplunkAugmented Reality – if you have a VR headset, you can pin glass-table style KPI metrics onto real-world devices. It’s designed so you can walk around a factory floor and see IoT data metrics from the various sensors installed. Also works with QR codes and your smart phone. Think Terminator vision!
  8. SplunkNatural Language Processor – lets you integrate an AI assistant like Alexa and ask English-language based questions and get English-language responses – all from Splunk. e.g. “Alexa, what was the highest selling product last month?” It would be a great addition to your organisation’s ChatOps.
  9. SplunkInsights for Web and Mobile Apps – helps your developers and operators improve the quality of experience delivered by your applications.
  10. SplunkTV – an Apple TV app which rotates through Splunk You no longer need to have a full PC running next to your TV display – just Apple TV.

To participate in any of the above betas go here:

https://www.splunk.com/en_us/software/splunk-next.html

Find out more

Interested to know more about these new Splunk capabilities? We’d love to hear from you. Whether it’s ChatOps, driving operational insight with ITSI, or leveraging Machine Learning - our team can take you through new ways of getting the most out of your data.

Our team on the case

Work smarter, not harder. (I didn't even come up with that. That's smart.)

Daniel Spavin

Performance Test Lead

Length of Time at JDS

7 years

Skills

IT: HPE Load Runner, HPE Performance Center, HPE SiteScope, HPE BSM, Splunk

Personal: Problem solving, Analytical thinking

Workplace Solutions

I care about quality and helping organisations get the best performance out of their IT projects.

Organisations spend a great deal of time and resources developing IT solutions. You want IT speeding up the process, not holding it up. Ensuring performance is built in means you spend less time fixing your IT solutions, and more time on the problems they solve.

I solve problems in our customers’ solutions, so customers can use their solutions to solve problems.

Our Splunk stories

Posted by Jillian Hunter in Blog, Splunk, Tech Tips, 0 comments
Splunk .conf18

Splunk .conf18

Splunk’s annual conference took place in Orlando, Florida this year, and JDS was there to soak up sun and the tech on offer.

Three days went by quickly, with exciting announcements (dark mode anyone?), interesting discussion and the chance to mingle with customers and Splunkers alike. We also enjoyed the chance to meet up with the US distributors of PowerConnect, and time spent with the uberAgent team.

Splunk CEO Doug Merritt kicked off the keynote, announcing a raft of features to Splunk 7.2 along with advancements released in beta – dubbed Splunk Next (but more of that to come, so stay tuned). Here’s a rundown of what’s new to 7.2:

  • SmartStore– some smarts behind using S3 for storage, allowing you to scale your indexer compute and storage separately. Great news if you want to expand your indexers, but don’t want the associated costs of SSD storage. SmartStore also gives you access to the impressive durability and availability of S3, simplifying your backup requirements.
  • Metrics Workspace– a new GUI interface for exploring metrics. You can drag and drop both standard events and metrics to create graphs over time and easily save them directly to dashboards.
  • Dark Mode– as simple as it sounds, with the crowd going wild for this one. You can now have your NOC display dark themed dashboards at the click of a mouse.
  • Official Docker support– Splunk Enterprise 7.2 now officially supports Docker containers, letting you quickly scale up and down based on user demands.
  • Machine Learning Tool Kit 4.0– now easier to train, test and validate your Machine Learning use cases. Includes the announcement of GitHub based solutions to share with fellow Splunkers.
  • ITSI 4.0– this latest version includes predictive KPIs, so your glass tables can show the current state, and the predicted state 30 minutes in the future. There’s also predictive cause analysis – to drill down and find out what will likely cause issues in the future. Metrics can now also feed into KPIs, allowing for closer integration with Splunk Insights for Infrastructure.
  • ES 5.1.1– introduces event sequencing to help with investigations, a Use Case Library to help with adoption, and the Investigation Workbench for incident investigation.
  • Health Report– in addition to the monitoring console, the health report shows the health of the platform, including disk, CPU, memory, and Splunk specific checks. It’s accessible via a new icon next to your login name.
  • Guided Data Onboarding– guides now available to help you onboard data, like those you can find in Enterprise Security. They include diagrams, high-level steps, and documentation links to help set up and configure your data source.
  • Logs to Metrics– a new GUI feature to help configure and convert logs into metric indexes.
  • Workload Management– prioritise users’ searches based on your own criteria – like a QoS for Searching.

 

If you weren’t lucky enough to go in person, or want to catch up on a missed presentation, the sessions are now available online:

https://conf.splunk.com/conf-online.html

Find out more

Interested to know more about these new Splunk capabilities? We’d love to hear from you. Whether it’s ChatOps, driving operational insight with ITSI, or leveraging Machine Learning - our team can take you through new ways of getting the most out of your data.

Our team on the case

Work smarter, not harder. (I didn't even come up with that. That's smart.)

Daniel Spavin

Performance Test Lead

Length of Time at JDS

7 years

Skills

IT: HPE Load Runner, HPE Performance Center, HPE SiteScope, HPE BSM, Splunk

Personal: Problem solving, Analytical thinking

Workplace Solutions

I care about quality and helping organisations get the best performance out of their IT projects.

Organisations spend a great deal of time and resources developing IT solutions. You want IT speeding up the process, not holding it up. Ensuring performance is built in means you spend less time fixing your IT solutions, and more time on the problems they solve.

I solve problems in our customers’ solutions, so customers can use their solutions to solve problems.

Our Splunk stories

Posted by Jillian Hunter in Blog, Splunk, Tech Tips, 0 comments
Event: What will drive the next wave of business innovation?

Event: What will drive the next wave of business innovation?

It’s no secret that senior managers and C-level executives are constantly wading through the latest buzzwords and jargon as they try to determine the best strategies for their business. Disruption, digital transformation, robots are taking our jobs, AI, AIOps, DevSecOps… all of the “next big thing” headlines, terms, clickbait articles, and sensationalism paint a distorted picture of what the business technology landscape really is.

Understand the reality amongst the virtuality, and make sense of what technology will drive the next wave of business innovation.

Join Tim Dillon, founder  of Tech Research Asia (TRA), for a presentation that blends technology market research trends with examples from Australian businesses already deploying solutions in areas such as cloud computing, intelligent analytics, robotics, artificial intelligence, and “the realities” (mixed, virtual, and augmented). Tim will examine when these innovation technologies will genuinely transform Australian industry sectors as well as the adoption and deployment plans of your peers. Not just a theoretical view, the presentation will provide practical tips and learnings drawn from real-life use cases.

Hosted by JDS Australia and Splunk, this is an event not to be missed by any executive who wants an industry insider view of what’s happening in technology in 2018, and where we’re headed in the future.

When: Tuesday 1 May, 11.45am-2pm (includes welcome drinks and post-event networking)

Where: Hilton Brisbane, 190 Elizabeth St, Brisbane City, QLD 4000

Cost: Complimentary

Agenda

11.45-12.30 Registration, canapes and drinks

12.30-12.40 Opening: Gene Kaalsen, Splunk Practice Manager, JDS Australia

12.35-1.05 Presentation: Tim Dillon

1.05-1.20 Q and A

1.20-1.25 Closing: Amanda Lugton, Enterprise Sales Manager, Splunk

1.25- 2.00 Networking, drinks and canapes

Business Innovation Brisbane

By clicking this button, you submit your information to JDS Australia, who will use it to communicate with you about this enquiry and their other services.

Tim Dillon, Founder and Director, Tech Research Asia

Tim is passionate about the application of technology for business benefit. He has been involved in business and technology research and analysis since 1991. In July 2012 he established Tech Research Asia (www.techresearch.asia) to provide bespoke analyst services to vendors in the IT&T sector. From 2007 to late 2012, he held the role of IDC’s Associate Vice President Enterprise Mobility and End-User (Business) research, Asia Pacific. Prior to this he was Current Analysis’ (now Global Data) Director of Global Telecoms Research and European and Asia Pacific Research Director.

For a period of time he also worked with one of Europe’s leading competitive intelligence research houses as research director with a personal focus on the telecoms and IT sectors. He combines more than 20 years of business and technology research with a blend of professional, international experience in Australia, Asia Pacific, and Europe. Of late, his particular areas of interest have centred upon emerging innovation technologies such as AI, virtual and augmented realities, security and data management, and governance. Tim truly delights in presenting, facilitating, and communicating with organisations and audiences discussing how trends and development in technology will shape the future business environment. 

A strong communicator, he has presented to large (1500+) audiences through to small, intimate round table discussions. A high proportion of Tim’s roles have been client focused—leading and delivering consulting projects or presenting at client conferences and events and authoring advisory reports. A regular participant in industry judging panels, Tim also works with event companies in an advisory role to help create strong, relevant technology business driven agendas. He has also authored expert witness documents for cases relating to the Australian telecommunication markets. A Tasmanian by birth, Tim holds a Bachelor of Economics from the University of Tasmania.

Key risk management tools for IT managers in 2018

Key risk management tools for IT managers in 2018

It’s no secret that technology is advancing at a faster rate now than ever before. While technological advances often enable better functionality and more efficiency, they also bring with them a swathe of risks and challenges for businesses. Implementing good risk management tools in your IT operations is critical to protect your brand and your reputation.

No matter what your organisation does, chances are you rely on effective and functional IT systems to keep the doors open. If their IT systems go down or experience delays, businesses across all industries see significant impacts on their revenue and reputation. Many customers will swear off a business entirely after even one major issue if they know they can get a better service from their competitors.

So, how do organisations keep up with the constantly changing IT landscape? One of the most effective ways to manage risk in technology is by using other technology. To that end, here are some of the key risk management tools for IT managers in 2018.

ServiceNow IT Operations Management

If you have a particularly complex solution stack or a broad range of devices and components, ServiceNow ITOM is designed for your business. Seamlessly integrating with your current ServiceNow instance and configuration management database (CMDB), ITOM gives you a single pane of glass view of how your entire environment is impacted when one system or server has an error. There are three key components of this service that make it a good choice for risk management:

1. Event Management

What if you could see your entire IT environment on one easy-to-read dashboard? Event management cuts down on the noise from your monitoring tools, proactively informs your IT team of any issues with performance, and helps you virtually eliminate outages altogether.

2. Service Mapping

When one device or server goes down, it often impacts a variety of your IT services. With service mapping, you can see how each of your IT components impacts the others and easily identify the root cause of an issue if it occurs.

3. Orchestration

One of the most important things you can do to manage risk in your organisation is to ensure your skilled IT operators are focused on business-critical tasks. Using orchestration, you can automate repetitive, time-consuming processes to resolve common errors, alleviating the burden on your IT team’s time and resources. Find out more about this here.

PagerDuty

PagerDuty is an event notification solution designed for any type of incident response that involves people stepping in to execute a resolution. It integrates with the top development, deployment, monitoring, and ticketing tools to route all alerts to one location, and distribute those alerts to the right individuals—the first time.

PagerDuty uses sophisticated machine learning to orchestrate a coordinated response in real time, which leads to increased operational efficiencies, faster incident resolution, less employee burnout, and less downtime. This solution functions as the single source of truth for incidents and stakeholder management, with detailed reporting and automated post-mortem analysis, making sure you learn from any issues that arise. PagerDuty manages the on-call roster for all operational responses as well as planned maintenance.

Active Robot Monitoring (ARM) with Splunk

This custom-built synthetic monitoring solution from JDS employs scripts that emulate the steps taken by real users engaged with your business services, automatically and according to your preferred schedule. This significantly de-risks new projects, as you are able to accurately predict your user experience with an application or service by simulating any action you want from any location.

ARM allows you to test variables ahead of time and throughout the lifecycle of an IT service, meaning you can always add new scenarios and actions to your scripts to see how your service will perform. Plus, since it all takes place in Splunk and transactions are unlimited, the cost to your business is significantly reduced compared to other synthetic monitoring solutions.

Not sure what's right for you?

JDS specialises in helping organisations analyse their IT environments and determine what solutions will work best for them. We provide independent advice; training and ongoing support; and IT testing, monitoring, and management solutions to businesses across Australia and around the world. If you want to enhance your risk management tools in 2018, give JDS a call and we’ll discuss the best options for your business.

If you want to learn more about orchestration with ServiceNow ITOM, sign up for an on-site workshop in one of our four city locations, and we’ll show you how automation will cut down on outages and free up time for your IT staff.

Conclusion

To find out more about how JDS can help you with risk management in your IT services, contact our team today on 1300 780 432, or email contactus@jds.net.au.

Our team on the case

Every day, do something that people want.

Nick Wilton

Consultant

Length of Time at JDS

8.5 years

Skills

Primary: Software security, Performance optimisation

Secondary: DevOps, Software development, Technical sales

Workplace Solutions

I help clients to solve problems like:
  • Is my application secure?
  • How do I manage threats?
  • Will my application perform when I need it to?

Workplace Passion

It’s all about managing risk whilst driving business confidence in technology and software solutions. That’s what I’m passionate about.

Commas save lives.

Amy Clarke

Marketing Communications Manager

Length of Time at JDS

Since July 2017

Skills

Writing, communications, marketing, design, developing and maintaining a brand, social media, sales.

Workplace Solutions

Words matter, so make sure you get them right!

Workplace Passion

Helping a company develop its voice and present that to their clients with pride.

Our success stories

Posted by Amy Clarke in Blog, Splunk, Tech Tips, 0 comments
Event: What can Splunk do for you?

Event: What can Splunk do for you?

Registration Form

Splunk Event: 23 November

By clicking this button, you submit your information to JDS Australia, who will use it to communicate with you about this event and their other services.

Event Details

Splunk .conf2017 was one of the biggest events of the year, with thousands gathering in Washington D.C. to experience the latest Splunk has to offer. One of JDS' senior consultants and Splunk experts, Michael Clayfield, delivered two exceptional presentations highlighting specific Splunk capabilities and how JDS can work with businesses to make them happen.

We don't want our Australian clients to miss out on hearing these exciting presentations, which is why we are pleased to invite you to our .conf17 recap event in Melbourne. You'll get to hear both presentations, and will also have a chance to chat with account executives and discuss Splunk solutions for your business.

The presentations will cover:

  • Using Active Robot Monitoring with Splunk to Improve Application Performance
  • Running Splunk within Docker

When: Thursday 23 November, 5-8pm 
Where:
Splunk Melbourne Office, Level 16, North Tower, 525 Collins Street

Case Study: Netwealth bolster their security with Splunk

Case Study: Netwealth bolster their security with Splunk

Netwealth, an Australian superannuation and investment company, implemented Splunk as an SIEM-compliant monitoring tool to bolster their information security.

The project was certainly a success, and Splunk is working well in our environment. We see a number of opportunities to use Splunk in other use-cases across the business.

June 2017

The prompt and decision

"As a financial services organisation, information security and system availability are core to the success of our business. With the business growing, we needed a solution that was scalable and which allowed our team to focus on high-value management tasks rather than on data collection and review."

Information security is vital to modern organisations, and particularly for those that deal in sensitive data, such as Netwealth. It is essential to actively assess software applications for security weaknesses to prevent exploitation and access by third parties, who could otherwise extract confidential and proprietary information. Security monitoring looks for abnormal behaviours and trends that could indicate a security breach.

"The continued growth of the business and the increased sophistication of threats prompted us to look for a better way to bring together our security and IT operations information and events," explains Chris Foong, Technology Infrastructure Manager at Netwealth. "Advancements in the technology available in this space over the last few years meant that a number of attractive options were available."

The first stage in Netwealth’s project was to select the right tool for the job, with several options short-listed. Each of these options was pilot tested, to establish which was the best fit to the requirements—and Splunk, with its high versatility and ease of use, was the selected solution.

The power in the solution comes from Splunk’s ability to combine multiple real-time data flows with machine learning and analysis which prioritises threats and actions, and the use of dynamic visual correlations and on-demand custom queries to more easily triage threats. Together, this empowers IT to make informed decisions.

Objective

Netwealth’s business objective was to implement a security information and event management (‘SIEM’) compliant tool to enhance management of security vulnerabilities and reporting. Their existing tool no longer met the expanding needs of the business, and so they looked to Splunk and JDS to provide a solution.

Approach

Netwealth conducted a proof of concept with various tools, and Splunk was selected. JDS Australia, as Splunk Implementation Partner, provided licensing and expertise.

IT improvements

Implementing Splunk monitoring gave Netwealth enhanced visibility over their security environment, and the movement of sensitive data through the business. This enabled them to triage security events and vulnerabilities in real time.

About Netwealth

Founded in 1999, Netwealth was established to provide astute investors and wealth professionals with a better way to invest, protect and manage their current and future wealth. As a business, Netwealth seeks to enable, educate and inspire Australians to see wealth differently and to discover a brighter future.

Netwealth offers a range of innovative portfolio administration, superannuation, retirement, investment, and managed account solutions to investors and non-institutional intermediaries including financial advisers, private clients, and high net worth firms.

Industry

Financial Services

Primary applications

  • Office365
  • Fortigate
  • IIS
  • Juniper SRX
  • Microsoft DNS
  • Microsoft AD and ADFS (Active Directory Federation Services)
  • JBoss (Java EE Application Server)
  • Fortinet

Primary software

  • Splunk Enterprise
  • Splunk Enterprise Security (application add-on)

The process

Now that Splunk had been identified as the best tool for the job, it was time to find an Implementation Partner—and that’s where JDS came in. JDS, as the most-certified Australian Splunk partner, was the natural choice. "JDS provided Splunk licensing, expertise on integrating data sources, and knowledge transfer to our internal team," says Foong.  

An agile, project managed approach was taken.  

  1. Understand the business requirements and potential threats associated with Netwealth’s environment.
  2. Identify the various services that required security monitoring.
  3. Identify the data feed for those services.
  4. Deploy and configure core Splunk.
  5. Deploy the Enterprise Security application onto Splunk.
  6. Configure the Enterprise Security application to enable features. These features gave visibility into areas of particular concern.
JDS provided Splunk licensing, expertise on integrating data sources, and knowledge transfer to our internal team.
Chris FoongTechnology Infrastructure Manager

What is Splunk security monitoring/Splunk Enterprise Security?

Splunk Enterprise Security (‘ES’) is a security solution built on the powerful features of Splunk. It gives the ability to continuously monitor the security environment; optimise, centralise, and automate incident response using workflows and alerting; conduct rapid investigation of potential incidents; and trace activities associated with compromised systems, applying a ‘kill-chain’ methodology to see the attack lifecycle.

The JDS team worked well with our team, were knowledgeable about the product, and happy to share that knowledge with our team.
JDS are professional. They delivered what they said they would, and didn’t under- or over-sell themselves. They’ve provided ongoing support and advice beyond the end of the project. We would work with them again.

The JDS difference

For this project, JDS "assisted Netwealth in deploying and configuring Splunk, and gaining confidence in Splunk Enterprise Security," explains the JDS Consultant on the case. "We were engaged as a trusted partner with Splunk, and within hours of deployment, we had helped Netwealth to gain greater visibility of the environment."

JDS were able to leverage their Splunk expertise to give added value to the client, advising them on how to gain maximum value in terms of both project staging, and in the onboarding of new applications. "We advocated a services approach—start by designing the dashboard you want, and work backwards towards the data required to build that dashboard."

"The JDS team worked well with our team, were knowledgeable about the product, and happy to share that knowledge with our team," says Netwealth’s Chris Foong. "They delivered what they said they would, and didn’t under- or over-sell themselves. We would work with them again."

End results

Chris Foong says that Netwealth was looking for "improved visibility over security and IT operations information and events, to aid in faster response and recovery"—and the project was a success on all counts.

"The project was delivered on time and to budget, and Splunk is now capturing data from all the required sources," adds Foong. "We also identified a number of additional use cases, over and above the base Enterprise Security case, such as rapidly troubleshooting performance degradation."

Now that Netwealth has implemented Splunk, the software has further applicability across the business. The next step is continuing to leverage Splunk, and JDS will be there to help.

Business Benefits

  • Gave Netwealth better visibility into the organisation’s security posture
  • Presents the opportunity for leveraging of Splunk in other areas of the business; for example, marketing
  • Allows Netwealth to have greater visibility into application and business statistics, with the potential to overlay machine learning and advanced statistical analysis of this business information
The project was certainly a success, and Splunk is working well in our environment.
The Splunk Gardener

The Splunk Gardener

The Splunk wizards at JDS are a talented bunch, dedicated to finding solutions—including in unexpected places. So when Sydney-based consultant Michael Clayfield suffered the tragedy of some dead plants in his garden, he did what our team do best: ensure it works (or ‘lives’, in this case). Using Splunk’s flexible yet powerful capabilities, he implemented monitoring, automation, and custom reporting on his herb garden, to ensure that tragedy didn’t strike twice.

My herb garden consists of three roughly 30cm x 40cm pots, each containing a single plant—rosemary, basil, and chilli. The garden is located outside our upstairs window and receives mostly full sunlight. While that’s good for the plants, it makes it harder to keep them properly watered, particularly during the summer months. After losing my basil and chilli bush over Christmas break, I decided to automate the watering of my three pots, to minimise the chance of losing any more plants. So I went away and designed an auto-watering setup, using soil moisture sensors, relays, pumps, and an Arduino—an open-source electronic platform—to tie it all together.

Testing the setup by transferring water from one bottle to another.

Testing the setup by transferring water from one bottle to another.

I placed soil moisture sensors in the basil and the chilli pots—given how hardy the rosemary was, I figured I could just hook it up to be watered whenever the basil in the pot next to it was watered. I connected the pumps to the relays, and rigged up some hosing to connect the pumps with their water source (a 10L container) and the pots. When the moisture level of a pot got below a certain level, the Arduino would turn the equivalent pump on and water it for a few seconds. This setup worked well—the plants were still alive—except that I had no visibility over what was going on. All I could see was that the water level in the tank was decreasing. It was essential that the tank always had water in it, otherwise I'd ruin my pumps by pumping air.

To address this problem, I added a float switch to the tank, as I was aiming to set it up so I could stop pumping air if I forgot to fill up the tank. Using a WiFi adapter, I connected the Arduino to my home WiFi. Now that the Arduino was connected to the internet, I figured I should send the data into Splunk. That way I'd be able to set up an alert notifying me when the tank’s water level was low. I'd also be able to track each plant’s moisture levels.

The setup deployed: the water tank is on the left; the yellow cables coming from the tank are for the float switch; and the plastic container houses the pumps and the Arduino, with the red/blue/black wires going to the sensors planted in the soil of the middle (basil) and right (chilli) pots. Power is supplied via the two black cables, which venture back inside the house to a phone charger.

The setup deployed: the water tank is on the left; the yellow cables coming from the tank are for the float switch; and the plastic container houses the pumps and the Arduino, with the red/blue/black wires going to the sensors planted in the soil of the middle (basil) and right (chilli) pots. Power is supplied via the two black cables, which venture back inside the house to a phone charger.

Using the Arduino’s Wifi library, it’s easy to send data to a TCP port. This means that all I needed to do to start collecting data in Splunk was to set up a TCP data input. Pretty quickly I had sensor data from both my chilli and basil plants, along with the tank’s water status. Given how simple it was, I decided to add a few other sensors to the Arduino: temperature, humidity, and light level. With all this information nicely ingested into Splunk, I went about creating a dashboard to display the health of my now over-engineered garden.

The overview dashboard for my garden. The top left and centre show current temperature and humidity, including trend, while the top right shows the current light reading. The bottom left and centre show current moisture reading and the last time each plant was watered. The final panel in the bottom right gives the status of the tank's water level.

The overview dashboard for my garden. The top left and centre show current temperature and humidity, including trend, while the top right shows the current light reading. The bottom left and centre show current moisture reading and the last time each plant was watered. The final panel in the bottom right gives the status of the tank's water level.

With this data coming in, I was able to easily understand what was going on with my plants:

  1. I can easily see the effect watering has on my plants, via the moisture levels (lower numbers = more moisture). I generally aim to maintain the moisture level between 300 and 410. Over 410 and the soil starts getting quite dry, while putting the moisture probe in a glass of water reads 220—so it’s probably best to keep it well above that.
  2. My basil was much thirstier than my chilli bush, requiring about 50–75% more water.
  3. It can get quite hot in the sun on our windowsill. One fortnight in February recorded nine 37+ degree days, with the temperature hitting 47 degrees twice during that period.
  4. During the height of summer, the tank typically holds 7–10 days’ worth of water.

Having this data in Splunk also alerts me to when the system isn't working properly. On one occasion in February, I noticed that my dashboard was consistently displaying that the basil pot had been watered within the last 15 minutes. After a few minutes looking at the data, I was able to figure out what was going on.

Using the above graph from my garden’s Splunk dashboard, I could see that my setup had correctly identified that the basil pot needed to be watered and had watered it—but I wasn't seeing the expected change in the basil’s moisture level. So the next time the system checked the moisture level, it saw that the plant needed to be watered, watered it again, and the cycle continued. When I physically checked the system, I could see that the Arduino was correctly setting the relay and turning the pump on, but no water was flowing. After further investigation, I discovered that the pump had died. Once I had replaced the faulty pump, everything returned to normal.

Since my initial design, I have upgraded the system a few times. It now joins a number of other Arduinos I have around the house, sending data via cheap radio transmitters to a central Arduino that then forwards the data on to Splunk. Aside from the pump dying, the garden system has been functioning well for the past six months, providing me with data that I will use to continue making the system a bit smarter about how and when it waters my plants.

I've also 3D printed a nice case in UV-resistant plastic, so my gardening system no longer has to live in an old lunchbox.

Our team on the case

Just Splunk it.

Michael Clayfield

Splunk Consultant

Length of Time at JDS

2.5 years

Skills

Splunk, HP BSM, 3D Printing.

Workplace Passion

Improving monitoring visibility and saving people’s time.

Posted by Laura Skillen in Blog, Case Study, Monitor, Splunk, 1 comment
Using Splunk and Active Robot Monitoring to resolve website issues

Using Splunk and Active Robot Monitoring to resolve website issues

Recently, one of JDS’ clients reached out for assistance, as they were experiencing inconsistent website performance. They had just moved to a new platform, and were receiving alerts about unexpectedly slow response times, as well as intermittent logon errors. They were concerned that, were the reports accurate, this would have an adverse impact on customer retention, and potentially reduce their ability to attract new customers. When manual verification couldn’t reproduce the issues, they called in one of JDS’ sleuths to try to locate and fix the problem—if one existed at all.

The Plot Thickens

The client’s existing active robot monitoring solution using the HPE Business Process Monitor (BPM) suite showed that there were sporadic difficulties in loading pages on the new platform and in logging in, but the client was unable to replicate the issue manually. If there was an issue, where exactly did it lie?

Commencing the Investigation

The client had deployed Splunk and it was ingesting logs from the application in question—but its features were not being utilised to investigate the issue.

JDS consultant Danesen Narayanen entered the fray and was able to use Splunk to analyse the data received. He could therefore immediately understand the issue the client was experiencing. He confirmed that the existing monitoring solution was reporting the problem accurately, and that the issue had not been affecting the client’s website prior to the re-platform

Using the data collected by HPE BPM as a starting point, Danesen was able to drill down and compare what was happening with the current system on the new platform to what had been happening on the old one. He quickly made several discoveries:

1. There appeared to be some kind of server error.

Since the re-platform, there had been a spike in a particular server error. Our JDS consultant reviewed data from the previous year, to see whether the error had happened before. He noted that there had previously been similar issues, and validated them against BPM to determine that the past errors had not had a pronounced effect on BPM—the spike in server errors seemed to be a symptom, rather than a cause.

Database deadlocks were spiking.

Database deadlocks were spiking

It was apparent that the error had happened before

2. There seemed to be an issue with user-end response time.

Next, our consultant used Splunk to look at the response time by IP addresses over time, to see if there was a particular location being affected—was the problem at server end, or user end? He identified one particular IP address which had a very high response time. What’s more, this was a public IP address, rather than one internal to the client. It seemed like there was a end-user problem—but what was the IP address that was causing BPM to report an issue?

Daily response time for all IPs (left axis), and for the abnormal IP (right axis). All times are in seconds.

Daily response time for all IPs (left axis), and for the abnormal IP (right axis). All times are in seconds.

Tracking Down the Mystery IP Address

At this point our consultant called for the assistance of another JDS staff member, to track down who owned the problematic IP address. As it turned out, the IP address was owned by the client, and was being used by a security tool running vulnerability checks on the website. After the re-platform, the tool had gone rogue: rather than running for half an hour after the re-platform, it continued to open a number of new web sessions throughout the day for several days.

The Resolution

Now that the culprit had been identified, the team were quickly able to log in to the security tool to turn it off, and the problem disappeared. Performance and availability times returned to what they should be, BPM was no longer reporting issues, and the client’s website was running smoothly once more. Thanks to the combination of Splunk’s power, HPE's active monitoring tools, and JDS’ analytical and diagnostic experience, resolution was achieved in under a day.

Our team on the case

Technology is only one part of IT.

Shane Andrewartha

Consultant

Length of Time at JDS

Since early 2016

Skills

OMi, HP Operations Manager/Agents, SiteScope, xMatters, ITIL, Unix, Coding.

Workplace Passion

I enjoy working with monitoring systems. It is satisfying to solve technical issues that would otherwise keep people up at night.

Why choose JDS?

At JDS, our purpose is to ensure your IT systems work wherever, however, and whenever they are needed. Our expert consultants will help you identify current or potential business issues, and then develop customised solutions to suit you.

JDS is different from other providers in the market. We offer 24/7 monitoring capabilities and support throughout the entire application lifecycle. We give your IT Operations team visibility into the health of your IT systems, enabling them to identify and resolve issues quickly.

We are passionate about what we do, working seamlessly with you to ensure you are getting the best possible performance from your environment. All products sold by JDS are backed by our local Tier One support desk, ensuring a stress-free solution for the entire product lifecycle.

Posted by Laura Skillen in Case Study, Financial Services, Splunk, 1 comment
JDS is now a CAUDIT Splunk Provider

JDS is now a CAUDIT Splunk Provider

Splunk Enterprise provides universities with a fast, easy and resilient way to collect, analyse and secure the streams of machine data generated by their IT systems and infrastructure.  JDS, as one of Australia’s leading Splunk experts, has a tradition of excellence in ensuring higher education institutions have solutions that maximise the performance and availability of campus-critical IT systems and infrastructure.

The CAUDIT Splunk offering provides Council of Australian University Directors of Information Technology (CAUDIT) Member Universities with the opportunity to buy on-premise Splunk Enterprise on a discounted, 3-year basis.  In acknowledgement of JDS’ expertise and dedication to client solutions, Splunk Inc. has elevated JDS to a provider of this sector-specific offering, meaning we are now better placed than ever to help the higher education sector reach their data collection and analysis goals.

What does this mean for organisations?

Not-for-profit higher education institutions that are members of CAUDIT can now use JDS to access discounted prices for on-premises deployments of Splunk Enterprise.  JDS are able to leverage their expertise in Splunk and customised solutions built on the platform, in combination with their insight into the higher education sector, to ensure that organisations have the Splunk solution that meet their specific needs.

 

Secure organisational applications and data, gain visibility over service performance, and ensure your organisation has the information to inform better decision-making.  JDS and Splunk are here to help.

 

You can learn more about JDS’ custom Splunk solutions here: Active Robot Monitoring with Splunk.
Contact one of our Australia-based consultants today on 1300 780 432.

Posted by Laura Skillen in Higher Education, News, Splunk, 0 comments
Active Robot Monitoring with Splunk

Active Robot Monitoring with Splunk


Why synthetic monitoring?

Finding out about a performance issue with your application or website from a real user is the most expensive and risky way for a business to discover a problem. Fortunately, synthetic monitoring overcomes the need for real users to report errors or interruption in services.

It does so by employing scripts that emulate the steps taken by real users engaged with your business services, automatically and according to your preferred schedule. Synthetic monitoring is an important IT solution that adds dimension to the other information gathered from your servers and networks.

Immediate reporting means you have visibility over what systems are causing issues and where. Synthetic monitoring provides you with consistent and predictable measurement of real-time performance and availability, without having to rely on feedback from users.

 

Introducing Splunk ARM

Splunk Active Robot Monitoring (ARM) is a capability developed by JDS that enables synthetic performance monitoring for websites, mobile, cloud-based, onpremise, and SaaS apps. It provides IT staff and managers a global view of what's happening in your environment, as it's happening. Use the customisable results dashboard to easily consume performance data, and drill down to easily isolate issues by location or transaction layer. Get better data to measure and report on your Service Level Agreements.

Current Splunk users: This offers you the ability to leverage more of Splunk’s capabilities to get the best from your environment.

New to Splunk?  JDS will provide you with independent advice and assist with licensing and deployment, training, and support.

The benefits of Splunk ARM

JDS’ custom-built ARM is different from other synthetic monitoring solutions in several ways:

  • Splunk Points of Presence (PoPs) are flexible, meaning you can determine your own PoPs and establish them in the locations that matter to you.
  • The Splunk machine learning toolkit can be applied over the insights you gain from ARM to understand your data and identify trends/recurring issues.
  • There are a variety of robot scripting protocols developed by JDS that allow you to do complex transactions that emulate sophisticated tasks users do.
  • Dashboards and visibility are customisable, and you have access to the latest Splunk features, such as Event Annotation.
  • When performance changes, access to a rich breakdown of network/server time lets you answer the age-old question: is it the network or the application?

Active monitoring

Leverage Splunk PoPs to simulate user sessions, so that you can ensure application availability and performance. Have visibility over operations anywhere in the world, in or outside of your firewall—anywhere your Splunk PoPs are located.

Fast setup, easy interface

Create robust, browserbased monitoring scripts for distribution across multiple environments, using Splunk’s built-in features only. There’s no need to download or deploy further applications—everything you need is on the one system and the one dashboard.

Predictive analytics

Use historical performance data to predict future issues, and set dynamic thresholds to keep an eye on performance. Forget taking manual measurements against KPIs for Service Level Agreements: automatic alerts can notify you the moment you deviate from expectations.

Utilise the power of Splunk

One fast point of reference
View all your performance data in a single location, and drill down on areas of interest to gain deeper insight. Combine your current situation with historical data, and identify trends and scope for performance gain.

Compare locations
Are slowdowns and outages happening at a particular location, or at a particular time? View application availability and performance status over different locations, sort geographical data by issue severity, and establish how user experience could be affected by their physical location.

Track transactions
See which transactions are problematic, and in what locations. Isolate transaction layer response times to learn where time is being spent, whether at client, network, or server level. Keep your third-party services in check, and ensure your application is performing at the top of its game.

Pinpoint causes
Evaluate the root causes for performance issues. Leverage JDS’ Splunk solution to pinpoint problems at a granular level, see how historical code changes have affected user experience, and check for hardware overload.

Failure and alert logs
Easily view logs relating to application status changes, and ensure the continuing health of your application.

Spotlight on errors
See exactly what errors users are seeing: should an unexpected result be received, full screen-shots of the problem allow for faster diagnosis and better communication to users.

Why choose JDS?

JDS is a leading IT solutions provider and systems integrator, with expertise across industry-leading tools such as ServiceNow, Splunk, AppDynamics, Micro Focus, SAP, PagerDuty, and more. We provide local, skilled, and responsive services to support IT projects and operations. Bringing together expert services, the latest technology, and best practices, JDS achieves improved IT outcomes for businesses. We do this by giving independent advice, providing training and ongoing support, and implementing IT testing, monitoring, and management solutions.

We are one of the most-certified partners of Splunk in Australia, and we use our expertise to deliver projects in ITSI, Enterprise Security, AIOps, application monitoring, dashboard configuration, and more. Our consultants are skilled developers, capable of building custom applications within Splunk to help you maximise your investment, such as our Active Robot Monitoring solution. JDS is an SAP partner and the sole distribution and implementation partner in Australia for the SAP-Certified PowerConnect for Splunk solution.

Monitoring success stories

Posted by Laura Skillen in Blog, Splunk, Tech Tips, 0 comments
Machine Learning with Splunk

Machine Learning with Splunk

Learn about how to optimise machine learning using Splunk, with this video from JDS Consultant Danesen Narayanen.

Posted by Joseph Banks in News, Splunk, 0 comments