As the Appliance collects log data from your log sources, you can search on those collected log messages. In addition to running various simple and complex searches, you can define search filters and run reports.
Pre-defining search filters lets you include specific search criteria in an Index Search, a Regular Expression Search, the Real Time Viewer, and All Saved Searches without having to re-enter the filtering criteria each time.
Viewing archived data files lets you reload and open older, compressed log data for viewing on an Appliance.
Topics
• Search Overview on page 46
• Using Index Search on page 48
• Tag-Based Searches Using the Tag Picker Interface on page 63
• Using Regular Expression Search on page 64
• Using Search Filters on page 71
• Viewing All Saved Index Searches on page 80
• Using and Creating All Index Reports on page 81
For details on Boolean expressions, Regular Expression usage, what gets indexed, and available delimiters, see the Search Strings topic in the Online Help.
46
|
Chapter 4 Searching Collected Log MessagesSearch Overview
LogLogic provides search and reporting tools for finding specific information in collected log message content. The tool you use varies depending on the task you want to perform.
• Index Search—Search on indexed log source messages using a Boolean expression and see the results immediately. Use Index Search when a simple, fast search can provide the information you need to analyze failures or other anomalies.
• Regular Expression (RegEx) Search—Search using a single regular expression or pre-defined search filter, either immediately or at a scheduled time.
• Real Time Viewer—The Real-Time Viewer shows an immediate scrolling display of real-time log messages as they are received by the Appliance. The options form allows for pre-filtering of these messages by log source or device group, message severity, and text matches. Only log messages meeting the filter settings are shown. See Viewing Log Messages in Real Time on page 41.
• Index Report—Generate a report based on indexed data using pre-defined Boolean search filters. Essentially, an Index Report is a compilation of multiple Index Searches run at once. You can specify one or more pre-defined filters to use, and add additional criteria to those filters.
Table 14 Search and Reporting Feature Comparison
Feature Index
Multiple filters in search Yes No No Yes
Boolean Expressions Yes Yes No No
Regular Expressions No No Yes Yes
Graphical Results Available Yes Yes No No
Graphically view trends over time or log sources
No Yes No No
Schedulable Search No No Yes No
Save customized search criteria for future use Yes Yes Yes Yes
View finished/past search results No No Yes Yes
Search Overview
|
47For a simple search to match a specific string, use Index Search. To search for strings that match more complex patterns, use RegEx Search.
48
|
Chapter 4 Searching Collected Log MessagesUsing Index Search
Use Index Search to perform targeted searches on log messages using keywords, Boolean expressions, and wildcards on the Appliance or log sources. Index Search lets you pinpoint problem areas on all log sources captured on the Appliance and then view the search results quickly.
Due to the dynamic nature of LogLogic reporting, when paging between the last page of search results and other pages, additional messages matching the search criteria might have been received since the initiation of the original search. As such, you might see additional messages included on subsequent visits to the last search results page.
Index Search works on indexed logs making it faster than a search using regular expressions (RegEx search). By default, the Appliance performs an Index Search on the Appliance itself and all log sources collected on the Appliance in the last hour.
Search Expression Rules
The following rules apply when you enter a search expression:
• Use Boolean operators, such as AND, OR, or NOT for your search expression (but do not begin the expression with leading NOT)
• Use wildcard characters, such as an asterisk (*) or question mark (?) to match strings (but do not begin the expression with the wildcard)
• Do not use < or > as these are not valid characters
• Use delimiters such as parentheses to tell Index Search what to evaluate first
• Enter up to 4096 characters for your search expression
• When using Index Search and Tag Based search, the system does not support the use of search patterns shorter than 3 characters
Index Searches are case insensitive, so you do not have to use all uppercase letters when using Boolean operators, although it helps readability. Some simple Index Search examples include:
Table 15 Index Search Examples
Index Search Example Rule
tcp Use search expressions containing at least three characters.
authenticate AND failed Use Boolean operators, such as AND, OR, or NOT.
Using Index Search
|
49For details on Boolean expressions, search strings, and available delimiters, see the Search Strings topic in the Online Help.
Running an Index Search
Index Search is available on all Appliances. By default, the Appliance performs an Index Search on the Appliance itself and all log sources from which logs were collected on the Appliance in the last hour. You can search using these defaults or change them.
To run an Index Search from the Index Search Interface 1. Access the Index Search page from home: Search > Index Search.
2. Enter your search expression in the search text box and click the Run button.
If you want, you can adjust the search scope and rerun the search by selecting specific log sources and/or a different timeframe.
admin*
10.*
Use wildcard characters such as an asterisk (*) or a question mark (?) as shortcuts to match strings.
Note: Wildcard character Index Search on IPv6 addresses will only work if the asterisk or question mark is at the end of the address. As shown below it will not work if the wildcards are used anywhere else in the address:
2001:db8::ff00:42:83??
2001:db8::ff00:*:8329 2001:db8::ff0?:42:8329 2001:db8::ff0*:42:8329 2001:db8::????:42:8329
(tcp and udp) and service Use a delimiter such as parentheses to specify what gets evaluated first. In this example, tcp and udp will be evaluated before the service keyword.
Table 15 Index Search Examples (Cont’d)
Index Search Example Rule
Do not use < or > in your search expression as these are not valid characters.
50
|
Chapter 4 Searching Collected Log MessagesSelecting Specific Log Sources
To perform a more targeted search, you can narrow the search scope to a group of log sources, such as all firewall interfaces, all routers, all General Syslog, Microsoft sources, other UNIX, or LogLogic Appliances.
The default rule is set as All Sources except LogLogic. This includes all logs except LogLogic Appliance logs. You can add any individual and/or group of non-LogLogic sources to this rule. However, if you specify any other log source, other than LogLogic source, the default rule will be removed from the filter list (from the left pane) and the new log source is added. This enhancement applies to only system-defined groups and not the user-defined groups. For example, if you select a user-defined group that only includes LogLogic source, then the default rule will be removed.
On the Management Station, you can select from one managed Appliance or all
Appliances, or particular groups of Appliances (for example, all LX Appliances or all ST Appliances) on which to run the search. The Choose Device pop-up automatically populates the log sources included on all defined groups.
To run a targeted Index Search
1. Click the All Sources except LogLogic button to open the Select Source(s) window.
2. Select log sources from the Add Log Sources pane. You can select sources by Appliance, and filter by Name, Collector Domain, IP Address, Group or Type.
a. If you picked “Name”, enter a Source Name, a specific Device Name or a Name Mask. Wild cards are accepted in this field.
b. If you picked "Collector Domain", enter the name of the Collector Domain. This is the name used to identify each message sent from a specific device.
c. If you picked “IP Address”, enter a Source IP Address, a specific IP Address or an IP Address Mask. Wild cards are accepted in this field.
d. If you picked “Group”, enter a Group Name, or click the down arrow to the right of the text field and select “All” or one of the other Group names displayed in the drop-down box.
e. If you picked “Type”, enter a Source Type (a specific device type), or click the down arrow to the right of the text field and select “All” or one of the other Device Types displayed in the drop-down box
When Appliance selection is “All”, “All LX/MX”, or “All ST”, only system defined groups (e.g. All Cisco PIX) and user defined global groups that reside on the management station will be displayed.
Using Index Search
|
513. Click << Add filters as a rule.
4. Enter a name for the dynamic rule in the pop-up window and click OK.
5. Click on the sources you want in your report and then click << Add selected log sources to add the selected devices and filters to the left-hand pane.
6. Click Set. The new Index Report search selection appears in the Sources row. The Index Search Sources field displays the newly added log sources.
Select Time Frame for an Index Search
To select time frame for an Index Search
1. Click the calendar icon (to the right of Last Hour) to launch the Date and Time Range Picker.
2. Select a preset time interval by clicking the down arrow to the right of Last Hour, or pick a timeframe from the pop-up calendar. Click Set.
3. Click Run.
4. At the Search pop-up, select whether you want to retrieve all messages. Click Yes.
After a few moments, the Index Search results will be displayed.
Using the Search Results Tab
Viewing Index Search Results
Index Search results are displayed in the Search Results tab and the keywords you entered are highlighted in different colors.
For example, when entering login AND user as your Boolean expression, the Search Results tab shows the first keyword “login” in yellow and second keyword “user” in turquoise.
When adding a large number of devices, create a dynamic rule which contains all listed devices. To create a rule, first filter by Name or Type to retrieve the list of devices. Then, click the << Add filters as a rule button that will create a dynamic rule which contains all listed devices on the right pane.
52
|
Chapter 4 Searching Collected Log MessagesFigure 3 Viewing Index Search Results
The UI uses several different colors to highlight search keywords after which it repeats the same color scheme.
In the results tab the Collector Domain will be displayed in one of two ways:
• For Collector Domains specified in a UC the following format; <collector domainid>_<device IP>_<devicetype> will be displayed in the Name field. For example a windows machine with an IP address of 10.10.10.10 and collector domain will be displayed as 1_10.10.10.10._windows.
• For Collector Domains specified in LMI (Managment>Devices>AddNew) the Collector Domain name will be displayed in the Collector Domain field.
To view search results using different view options
1. From the top right of the Index Search screen, click the View drop-down menu to open different view options. The options are: Reset to Default, Show Timeline, Hide Meta Header, View by, Chart Type.
2. The Search Results view options are:
Table 16 Index Report Search--View options
Element Description
Reset to Default Resets to default settings.
Using Index Search
|
53Configuring Search Results Settings
To configure Search Results settings
1. From the top right of the Index Search page, click the Options button. The Columns and Grouping window appears as shown below.
2. Optionally, enter a filter keyword in the Keyword field to narrow the displayed columns in your report.
3. Select the appropriate Column Name by clicking in the checkbox to include or exclude that column from your report. You can change the column name by clicking on the name. The column name field becomes an editable field allowing you to make the changes.
4. Click or to move the selected column.
5. Choose the Display options.
Hide Meta Header Select this checkbox to hide the metadata header information.
View By Select the option to view by Time or Device type.
Chart Type Select the type. The options are Bar chart or Line chart.
Table 16 Index Report Search--View options
Element Description
If you enter the same column name for two columns, the Index Search Results page displays the results for those two columns merged into one column.
Table 17 Display Options
Element Description
Raw Select this option to display Index Search Results in time-increasing order.
Grouped Select this option to display Index Search Results grouped by the selected column.
54
|
Chapter 4 Searching Collected Log Messages6. Click Apply to apply the new settings. The Index Search Results page displays the Group By Choose the appropriate column to display group search results from the
drop-down menu. The default options are:
• Time
• Device IP
• Device Source
• Facility
• Severity
You can add more columns by creating custom tags using Log Labels.
See Device Types online help video tutorial for instructions.
Time Interval This option is enabled when you select to Group By Time. The results are grouped based on the specified time interval. Select the Time Interval from the following options:
Sum By This optional setting allows you to add the numerical value of the selected column so that Search Results Summary displays the sum value of the grouped column instead of the count of message instances.
Aggregation Size Select the option from the drop-down menu. The results will be sorted based on the selected option. The options are:
• Top 1
• Top 5
• Top 50
• All Table 17 Display Options
Element Description
Using Index Search
|
55Managing Search Results
The Search Results tab provides a toolbar with several options for managing Search results.
Table 18 Search Results Tab Toolbar Elements
Element Description
Collapses and condenses the results display view.
Allows you to view selected message in relation to all others in your Index Search results. For details, see Viewing Index Search Results In Context on page 56
Create a new log message pattern with the selected message. Highlight a message in the Search Results and click the Create Message Pattern button. The Message Pattern Editor is displayed, which can be used to select a particular message from a particular device and then create a pattern based on the parameters of that message for use in further searches. For detailed instructions, see online help tutorial or Creating Message Signatures chapter in the LogLogic Administration Guide.
Clip Selected message(s)
From the drop-down menu use the default clipboard, a saved clipboard, or create a new clipboard to save results.
Saves the results. You can choose to Save or Save as from the
drop-down menu to save your results. You can update your saved results using the Save as option, see Saving Search Results on page 56.
Number of Indexed Pages
Get the total number of indexed messages on the indexed search results.
This is particularly useful for large volumes of log messages as it lets you go through matched messages one page at a time. To page through the results, click the next arrow; to return to the previous page click the previous page arrow. You can also return to the first page or go to the last page by clicking on the first and last page arrows accordingly. The total results number is automatically updated when you select the Show Timeline graphical view.
Displays context-sensitive help.
56
|
Chapter 4 Searching Collected Log MessagesViewing Index Search Results In Context
When analyzing log events, you can select a particular message and see the log messages that immediately preceded or followed the message from your search results.
To view a particular log message in context
1. On the Search Results tab, select the message that you want to view and then select the icon.
The In Context tab appears (next to the Clipboard tab) and the message you selected is immediately displayed in the Search Results tab.
2. By scrolling down on the page, the affected log message is highlighted in blue to show its relationship to the log messages that preceded this condition as well as those that occurred after this message.
3. Click the appropriate button to save the report. You can choose to save results in CSV, PDF, or HTML format.
Saving Search Results
You can download Index Search results to view immediately or save them in CSV, PDF, or HTML formats. These buttons are located on the left side of the Save button. After few moments, the report in your chosen format will appear.
The In Context tab appears only after the first time you click the icon in the search results toolbar.
Table 19 Save Search Results
Output Description
CSV Use Microsoft Excel or other spreadsheet program to display Index Search results in a spreadsheet. By default, search results are written to SearchExpressionHits.csv and saved on the desktop.
PDF Use Adobe Acrobat Reader to display the Index Search results. By default, search results are written to report.pdf and saved on the desktop. The first page incudes a table of contents with links to the query used for the Index Search and the results table.
HTML Opens a new tab in your Web browser and immediately displays HTML Index Search results as a LogLogic report. The HTML results include a table of contents with links to the query used for the Index Search and the results table. By default, the downloaded results are saved as LogLogicReport.zip in a temp folder on the local
Using Index Search
|
57To save search results report
1. Click Save As option from the icon drop-down menu to save the report. You can update the saved report by using the Save option. The Save As Report window appears.
2. Enter the name and description of the report in the Name and Description fields respectively. The Name field is a mandatory field.
3. Select the Suite option from the drop-down menu.
4. Select the Share? checkbox if you want to share the report.
5. Select the desired print option. For Grouped Search, the options are: Print Summary Report or Print Detailed Report.
6. Click Save to save the results.
Viewing Trends
After running Index Searches, you can use the View menu to view search results graphically using the timeline option. The trend output you see is based on your chosen time range and chosen devices referenced by the Index Search and always includes only the messages and devices for that distribution.
The trend feature can be a powerful tool during your analysis of certain events and lets you see trends for certain activities by Time and Device.
Each option lets you view timeline data in either bar chart or line chart format. These
Each option lets you view timeline data in either bar chart or line chart format. These