Versions Compared

Version Old Version 14 New Version Current
Changes made by

Former user

Former user

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: Migrated to Confluence 4.0

...

Overview
Architecture
Installation
Configuration

Service Now data from Service Desk Connector, stock

Code Block
titlehttp://yalesandbox.service-now.com/nav_to.do?uri=ecc_queue.do?sys_id=f89666ae71a42000a8d1496cf964454c

<notification><priority>1</priority><short_description>HTTPS - 443 - ADDR1 - Dumb is CRITICAL on host trapeze3.its.yale.edu</short_description><comments>Service: HTTPS - 443 - ADDR1 - Dumb
Host: trapeze3.its.yale.edu
Address: trapeze3.its.yale.edu
State: CRITICAL
Date/Time: Tue Mar 6 14:26:18 EST 2012

Additional Info:

CRITICAL - Socket timeout after 10 seconds</comments><category>Network</category><checktime>1331061975</checktime><correlation_id>trapeze3.its.yale.edu;HTTPS - 443 - ADDR1 - Dumb;1331061544</correlation_id><state>CRITICAL</state><servicename>HTTPS - 443 - ADDR1 - Dumb</servicename><hostname>trapeze3.its.yale.edu</hostname><contact_type>Opsview</contact_type></notification>
Code Block
titlehttp://yalesandbox.service-now.com/nav_to.do?uri=ecc_queue.do?sys_id=549626ae71a42000a8d1496cf96445d0

<notification><priority>1</priority><short_description>physical2.virtual.yale.edu is DOWN</short_description><comments>Host: physical2.virtual.yale.edu
Address: physical2.virtual.yale.edu
State: DOWN
Date/Time: Tue Mar 6 14:25:44 EST 2012

Additional Info:

CRITICAL - physical2.virtual.yale.edu: rta nan, lost 100%</comments><category>Network</category><checktime>1331061942</checktime><correlation_id>physical2.virtual.yale.edu;;1330856341</correlation_id><state>DOWN</state><hostname>physical2.virtual.yale.edu</hostname><contact_type>Opsview</contact_type></notification>
Code Block
titlehttp://yalesandbox.service-now.com/nav_to.do?uri=ecc_queue.do?sys_id=e8966eead4a0200054d3e703207fe9e4

<notification><priority>1</priority><short_description>Load Average is CRITICAL on host meg.its.yale.edu</short_description><comments>Service: Load Average
Host: meg.its.yale.edu
Address: meg.its.yale.edu
State: CRITICAL
Date/Time: Tue Mar 6 14:23:59 EST 2012

Additional Info:

CRITICAL - load average: 13.19, 12.05, 10.57</comments><category>Network</category><checktime>1331061769</checktime><correlation_id>meg.its.yale.edu;Load Average;1331061048</correlation_id><state>CRITICAL</state><servicename>Load Average</servicename><hostname>meg.its.yale.edu</hostname><contact_type>Opsview</contact_type></notification>

ServiceNow-Side Configuration

Concerns

There is a lot of state change within Opsview. Sometimes this state change is considered spiurious upon human inspection while its always deemed a genuine issue per Opsview. Yale should tread carefully regarding opening this Opsview dataflow up to Service Now so as to avoid a firehose condition.

Moreover this integration should be used as a lightning rod to initiate and push for amending the monitoring stack where appropriate to dial back the amount of state change.
-nick, 20120305

Next Steps

...

...

  • http://docs.opsview.com/doku.php?id=servicedesk-connector-latest:configuration#testing
  • No Format
    $ NAGIOS_LASTHOSTCHECK=1234567891 NAGIOS_LASTHOSTUP=1234567890 NAGIOS_HOSTOUTPUT="Test failure" NAGIOS_HOSTADDRESS=10.11.12.13 NAGIOS_LONGDATETIME="Dec 1 2009" NAGIOS_HOSTALIAS="temp host 1" NAGIOS_LASTHOSTDOWN=0 NAGIOS_LASTHOSTSTATECHANGE=0 NAGIOS_NOTIFICATIONTYPE=PROBLEM NAGIOS_HOSTSTATE=DOWN NAGIOS_HOSTNAME=host1 /opt/opsview/notifications/bin/opsview_notifications servicenow
Notification submitted
  • Egress packet filter in the way. Eval scalable means to open up.

...

  • Is 199.91.136.0/21 too much, too little, etc?

...

  1. We can work on dialing down yelling by tracking Top Talkers
  2. Need to confirm that renotifies dont generate dupe incidents
  3. Vet process of auto-acking of states during incident open @ service now
  4. Nail process on how to target top talkers + optimize minimizing false poz where possible (timeout increases, consecutive hit increases, etc)

Tickets

Opsview

Service Now

  • inquire about least privilege needed for bind account
  • inquire about addresses to permit egress to from monitoring stations

Next Steps

  • Consider keywords for service now only ; not all events would enter SN only those we say to per tags

Final considerations

  • per Lou: Critical Alerts from Opsview should be set to a priority level of "High" in SN and Warnings be set as "Average".
  • field mappings: "CLIENT=OPsView, "CONTACT=Operator" "NOTIFY=NONE", "CONTACT TYPE=Tier 2", "Incident Type=Service Event", and Priority=3-High" on the SN screen.
  • left-nav item for unassigned opsview tickets (or bookmark/OVS-3111view) for ops staff
  • Downtimed state change => handled => open ticket?
  • Yellows/warnings + tickets as they effect workflow
    • OVS-3236
    • Assign warns to proper group ; assign crit to DC Ops
  • Acknowledgements
    • auto-ack @ Opsview and how this impacts Yale workflow
    • this means that Unhandled column in Opsview goes away, becomes meaningless, etc. Important if thats a catalyst for starting Ops work (procedure, call, etc).
  • Getting keyword data transmitted
  • Oddballs (non-ITS like YUHS)
    • Oddballs can stay behind if needed; they dont block ITS from moving. This is all driven with notification profiles where we can poz match keywords.