[Nagios-users] Splunk Integration Question...

[Sean Alderman]

Greetings,
I was hoping I might find someone who's got the splunk integration
actively working. I'm running Nagios Core (via EPEL) and Splunk 5.0.3 on
OracleLinux 6.4.

When I edit cgi.cfg and enable splunk integration, then set the splunk
URL to https://<mysplunkserver>:8000/en-US/app/search/flastimeline, I
notice the nagios URLs look like: https://
<mysplunkserver>:8000/en-US/app/flashtimeline?q=search%20test1.udayton.edu%20<nagios
plugin check>. I have two questions...

- Is there a way I can make nagios use the hostname only, not the FQDN?
We use short names in splunk so we don't a mix of fqdn and short names
since we use both forwarders and syslog as input.
- What data is this query looking for, is it expected that I should have
my nagios log in splunk? The <nagios plugin check> in the query doesn't
seem useful to me, unless there's splunk data specifically tied to that
check, and I'm hoping someone could provide an example.

Kind regards,

[Frost, Mark {BIS}]

Sean,

Can you describe what you're doing for Splunk integration with Nagios? I've used Splunk with Nagios in a couple different ways, but I'm not aware of any single standard for doing so.

Originally, I just had Splunk run a scheduled search, which would trigger a script which sent a passive check result back to a Nagios service via NSCA. That way - having Nagios process passive check results from Splunk - was the only way I could see to do that.

Recently, I played around a bit with writing scripts that made use of Splunk's REST API so the checks could be run as active checks from Nagios. (I always prefer active checks). I set this up for only one check, but once I got it working it worked pretty well.

As a side note, I'm still a little on the fence about whether or not I really want to have Nagios find problems through Splunk and then alert on them or have Splunk find an alert on them directly without using Nagios at all...

Are you referring to another way of making Splunk and Nagios talk together?

Mark

From: Sean Alderman [mailto:***@udayton.edu]
Sent: Monday, September 09, 2013 1:12 PM
To: nagios-***@lists.sourceforge.net
Subject: [Nagios-users] Splunk Integration Question...

Greetings,
I was hoping I might find someone who's got the splunk integration actively working. I'm running Nagios Core (via EPEL) and Splunk 5.0.3 on OracleLinux 6.4.
When I edit cgi.cfg and enable splunk integration, then set the splunk URL to https://<mysplunkserver>:8000/en-US/app/search/flastimeline<https://%3cmysplunkserver%3e:8000/en-US/app/search/flastimeline>, I notice the nagios URLs look like: https://<mysplunkserver>:8000/en-US/app/flashtimeline?q=search%20test1.udayton.edu<http://20test1.udayton.edu>%20<nagios plugin check>. I have two questions...
* Is there a way I can make nagios use the hostname only, not the FQDN? We use short names in splunk so we don't a mix of fqdn and short names since we use both forwarders and syslog as input.
* What data is this query looking for, is it expected that I should have my nagios log in splunk? The <nagios plugin check> in the query doesn't seem useful to me, unless there's splunk data specifically tied to that check, and I'm hoping someone could provide an example.
Kind regards,
--
Sean M. Alderman
Senior Engineer, UDit Systems Integration and Engineering
University of Dayton

[Sean Alderman]

Just what's in the nagios doc on CGI.cfg. The doc is lacking about what it
does, so I guess I'm a little curious what that config is about.

- Sean Alderman
Senior Engineer, UDit Systems Integration

This message had been brought to you by Android Bionic.

Sean,****
** **
Can you describe what you’re doing for Splunk integration with Nagios?
I’ve used Splunk with Nagios in a couple different ways, but I’m not aware
of any single standard for doing so.****
** **
Originally, I just had Splunk run a scheduled search, which would trigger
a script which sent a passive check result back to a Nagios service via
NSCA. That way – having Nagios process passive check results from Splunk
– was the only way I could see to do that.****
** **
Recently, I played around a bit with writing scripts that made use of
Splunk’s REST API so the checks could be run as active checks from Nagios.
(I always prefer active checks). I set this up for only one check, but
once I got it working it worked pretty well.****
** **
As a side note, I’m still a little on the fence about whether or not I
really want to have Nagios find problems through Splunk and then alert on
them or have Splunk find an alert on them directly without using Nagios at
all…****
** **
Are you referring to another way of making Splunk and Nagios talk together?
****
** **
Mark****
** **
*Sent:* Monday, September 09, 2013 1:12 PM
*Subject:* [Nagios-users] Splunk Integration Question...****
** **
Greetings,****
I was hoping I might find someone who's got the splunk integration
actively working. I'm running Nagios Core (via EPEL) and Splunk 5.0.3 on
OracleLinux 6.4.****
When I edit cgi.cfg and enable splunk integration, then set the splunk
URL to https://<mysplunkserver>:8000/en-US/app/search/flastimeline, I
notice the nagios URLs look like: https://
<mysplunkserver>:8000/en-US/app/flashtimeline?q=search%20test1.udayton.edu%20<nagios
plugin check>. I have two questions...****
**· **Is there a way I can make nagios use the hostname only, not
the FQDN? We use short names in splunk so we don't a mix of fqdn and short
names since we use both forwarders and syslog as input.****
**· **What data is this query looking for, is it expected that I
should have my nagios log in splunk? The <nagios plugin check> in the
query doesn't seem useful to me, unless there's splunk data specifically
tied to that check, and I'm hoping someone could provide an example.
****
Kind regards,****
-- ****
Sean M. Alderman
Senior Engineer, UDit Systems Integration and Engineering
University of Dayton****
------------------------------------------------------------------------------
1. Consolidate legacy IT systems to a single system of record for IT
2. Standardize and globalize service processes across IT
3. Implement zero-touch automation to replace manual, redundant tasks
http://pubads.g.doubleclick.net/gampad/clk?id=51271111&iu=/4140/ostg.clktrk
_______________________________________________
Nagios-users mailing list
https://lists.sourceforge.net/lists/listinfo/nagios-users
::: Please include Nagios version, plugin version (-v) and OS when
reporting any issue.
::: Messages without supporting info will risk being sent to /dev/null

[Frost, Mark {BIS}]

Huh. Where did those new options come from? They weren't in the cgi.cfg docs the last time I looked :).

I agree, it's not terribly clear to me what that option does, but it does reference "Splunk IT" which is a special Splunk package that you can use for Splunk benchmarking. That still doesn't make it clear what it's used for.

I see a second parameter, "splunk_url" that lets you specify the URL for your Splunk server.

Maybe it just somehow says to pepper the logs with your Splunk URL in appropriate places.

Mark

From: Sean Alderman [mailto:***@udayton.edu]
Sent: Tuesday, September 10, 2013 1:34 PM
To: Nagios Users List
Subject: Re: [Nagios-users] Splunk Integration Question...


Just what's in the nagios doc on CGI.cfg. The doc is lacking about what it does, so I guess I'm a little curious what that config is about.

- Sean Alderman
Senior Engineer, UDit Systems Integration

This message had been brought to you by Android Bionic.
On Sep 10, 2013 1:10 PM, "Frost, Mark {BIS}" <***@pepsico.com<mailto:***@pepsico.com>> wrote:
Sean,

Can you describe what you're doing for Splunk integration with Nagios? I've used Splunk with Nagios in a couple different ways, but I'm not aware of any single standard for doing so.

Originally, I just had Splunk run a scheduled search, which would trigger a script which sent a passive check result back to a Nagios service via NSCA. That way - having Nagios process passive check results from Splunk - was the only way I could see to do that.

Recently, I played around a bit with writing scripts that made use of Splunk's REST API so the checks could be run as active checks from Nagios. (I always prefer active checks). I set this up for only one check, but once I got it working it worked pretty well.

As a side note, I'm still a little on the fence about whether or not I really want to have Nagios find problems through Splunk and then alert on them or have Splunk find an alert on them directly without using Nagios at all...

Are you referring to another way of making Splunk and Nagios talk together?

Mark

From: Sean Alderman [mailto:***@udayton.edu<mailto:***@udayton.edu>]
Sent: Monday, September 09, 2013 1:12 PM
To: nagios-***@lists.sourceforge.net<mailto:nagios-***@lists.sourceforge.net>
Subject: [Nagios-users] Splunk Integration Question...

Greetings,
I was hoping I might find someone who's got the splunk integration actively working. I'm running Nagios Core (via EPEL) and Splunk 5.0.3 on OracleLinux 6.4.
When I edit cgi.cfg and enable splunk integration, then set the splunk URL to https://<mysplunkserver>:8000/en-US/app/search/flastimeline<https://%3cmysplunkserver%3e:8000/en-US/app/search/flastimeline>, I notice the nagios URLs look like: https://<mysplunkserver>:8000/en-US/app/flashtimeline?q=search%20test1.udayton.edu<http://20test1.udayton.edu>%20<nagios plugin check>. I have two questions...
* Is there a way I can make nagios use the hostname only, not the FQDN? We use short names in splunk so we don't a mix of fqdn and short names since we use both forwarders and syslog as input.
* What data is this query looking for, is it expected that I should have my nagios log in splunk? The <nagios plugin check> in the query doesn't seem useful to me, unless there's splunk data specifically tied to that check, and I'm hoping someone could provide an example.
Kind regards,
--
Sean M. Alderman
Senior Engineer, UDit Systems Integration and Engineering
University of Dayton

------------------------------------------------------------------------------
How ServiceNow helps IT people transform IT departments:
1. Consolidate legacy IT systems to a single system of record for IT
2. Standardize and globalize service processes across IT
3. Implement zero-touch automation to replace manual, redundant tasks
http://pubads.g.doubleclick.net/gampad/clk?id=51271111&iu=/4140/ostg.clktrk
_______________________________________________
Nagios-users mailing list
Nagios-***@lists.sourceforge.net<mailto:Nagios-***@lists.sourceforge.net>
https://lists.sourceforge.net/lists/listinfo/nagios-users
::: Please include Nagios version, plugin version (-v) and OS when reporting any issue.
::: Messages without supporting info will risk being sent to /dev/null

[Sean Alderman]