SPLK-2002 Splunk Enterprise Certified Architect Questions and Answers

 The tailing_processor.log file would be the best place to search if you suspect there is a problem interpreting a regular expression in a monitor stanza. This log file contains information about how Splunk monitors files and directories, including any errors or warnings related to parsing the monitor stanza. The splunkd.log file contains general information about the Splunk daemon, but it may not have the specific details about the monitor stanza. The btool.log file contains information about the configuration files, but it does not log the runtime behavior of the monitor stanza. The metrics.log file contains information about the performance metrics of Splunk, but it does not log the event breaking issues. For more information, see About Splunk Enterprise logging in the Splunk documentation.

The correct answer is B. Set the Replication Factor based on allowed indexer failure. This is a best practice for adding data resiliency to a single-site indexer cluster, as it ensures that there are enough copies of each bucket to survive the loss of one or more indexers without affecting the searchability of the data1. The Replication Factor is the number of copies of each bucket that the cluster maintains across the set of peer nodes2. The Replication Factor should be set according to the number of indexers that can fail without compromising the cluster’s ability to serve data1. For example, if the cluster can tolerate the loss of two indexers, the Replication Factor should be set to three1.

The other options are not best practices for adding data resiliency. Option A, setting the Replication Factor to 49, is not recommended, as it would create too many copies of each bucket and consume excessive disk space and network bandwidth1. Option C, always using the default Replication Factor of 3, is not optimal, as it may not match the customer’s requirements and expectations for data availability and performance1. Option D, setting the Replication Factor based on allowed search head failure, is not relevant, as the Replication Factor does not affect the search head availability, but the searchability of the data on the indexers1. Therefore, option B is the correct answer, and options A, C, and D are incorrect.

1: Configure the replication factor 2: About indexer clusters and index replication

Splunk Enterprise allows administrators to dynamically increase logging verbosity for a specific search by adding a | noop log_debug=* command immediately after the base search. This method provides temporary, search-specific debug logging without requiring global configuration changes or restarts.

The noop (no operation) command passes all results through unchanged but can trigger internal logging actions. When paired with the log_debug=* argument, it instructs Splunk to record detailed debug-level log messages for that specific search execution in search.log and the relevant internal logs.

This approach is officially documented for troubleshooting complex search issues such as:

Unexpected search behavior or slow performance.

Field extraction or command evaluation errors.

Debugging custom search commands or macros.

Using this method is safer and more efficient than modifying server-wide logging configurations (server.conf or limits.conf), which can affect all users and increase log noise. The “Server logging” page in Splunk Web (Option D) adjusts global logging levels, not per-search debugging.

References (Splunk Enterprise Documentation):

• Search Debugging Techniques and the noop Command

• Understanding search.log and Per-Search Logging Control

• Splunk Search Job Inspector and Debugging Workflow

• Troubleshooting SPL Performance and Field Extraction Issues

Splunk Enterprise Search Optimization documentation consistently emphasizes that filtering data as early as possible in the search pipeline is the most effective way to improve search performance. The base search (the part before the first pipe |) determines the volume of raw events Splunk retrieves from the indexers. Therefore, by applying restrictive conditions early—such as time ranges, indexed fields, and metadata filters—you can drastically reduce the number of events that need to be fetched and processed downstream.

The best practice is to use indexed field filters (e.g., index=security sourcetype=syslog host=server01) combined with search or where clauses at the start of the query. This minimizes unnecessary data movement between indexers and the search head, improving both search speed and system efficiency.

Using non-streaming commands early (Option C) can degrade performance because they require full result sets before producing output. Likewise, focusing solely on shortening queries (Option A) or excessive use of the not operator (Option D) does not guarantee efficiency, as both may still process large datasets.

Filtering early leverages Splunk’s distributed search architecture to limit data at the indexer level, reducing processing load and network transfer.

References (Splunk Enterprise Documentation):

• Search Performance Tuning and Optimization Guide

• Best Practices for Writing Efficient SPL Queries

• Understanding Streaming and Non-Streaming Commands

• Search Job Inspector: Analyzing Execution Costs

Increasing the number of parallel ingestion pipelines in server.conf is most likely to improve indexing performance when indexing is slow and real-time search results are delayed in a Splunk environment with two indexers and one search head. The parallel ingestion pipelines allow Splunk to process multiple data streams simultaneously, which increases the indexing throughput and reduces the indexing latency. Increasing the maximum number of hot buckets in indexes.conf will not improve indexing performance, but rather increase the disk space consumption and the bucket rolling time. Decreasing the maximum size of the search pipelines in limits.conf will not improve indexing performance, but rather reduce the search performance and the search concurrency. Decreasing the maximum concurrent scheduled searches in limits.conf will not improve indexing performance, but rather reduce the search capacity and the search availability. For more information, see Configure parallel ingestion pipelines in the Splunk documentation.

The splunk_instrumentation.log file is the least helpful in troubleshooting a crash, because it contains information about the Splunk Instrumentation feature, which collects and sends usage data to Splunk Inc. for product improvement purposes. This file does not contain any information about the Splunk processes, errors, or crashes. The other options are more helpful in troubleshooting a crash, because they contain relevant information about the Splunk daemon, the standard error output, and the crash report12

1: https://docs.splunk.com/Documentation/Splunk/9.1.2/Troubleshooting/WhatSplunklogsaboutitself#splunk_instrumentation.log 2: https://docs.splunk.com/Documentation/Splunk/9.1.2/Troubleshooting/WhatSplunklogsaboutitself#splunkd_stderr.log

According to the Splunk Enterprise Search Head Clustering (SHC) Deployer documentation, the “local_only” push mode is the correct option when deploying built-in apps. This mode ensures that the deployer only pushes configurations from the local directory of built-in Splunk apps (such as search, learned, or launcher) without overwriting or merging their default app configurations.

In an SHC environment, the deployer is responsible for distributing configuration bundles to all search head members. Each push can be executed in different modes depending on how the admin wants to handle the app directories:

full: Overwrites both default and local folders of all apps in the bundle.

merge_to_default: Merges configurations into the default folder (used primarily for custom apps).

local_only: Pushes only local configurations, preserving default settings of built-in apps (the safest method for core Splunk apps).

default only: Pushes only default folder configurations (rarely used and not ideal for built-in app updates).

Using the “local_only” mode ensures that default Splunk system apps are not modified, preventing corruption or overwriting of base configurations that are critical for Splunk operation. It is explicitly recommended for pushing Splunk-provided (built-in) apps like search, launcher, and user-prefs from the deployer to all SHC members.

References (Splunk Enterprise Documentation):

• Managing Configuration Bundles with the Deployer (Search Head Clustering)

• Deployer Push Modes and Their Use Cases

• Splunk Enterprise Admin Manual – SHC Deployment Management

• Best Practices for Maintaining Built-in Splunk Apps in SHC Environments

The correct action to take first if a deployment client is not updating apps is to choose a corrective action based on the splunkd.log of the deployment client. This log file contains information about the communication between the deployment server and the deployment client, and it can help identify the root cause of the problem1. The other actions may or may not help, depending on the situation, but they are not the first steps to take. Choosing a longer phone home interval may reduce the load on the deployment server, but it will also delay the updates for the deployment clients2. Increasing the number of CPU cores or the amount of memory for the deployment server may improve its performance, but it will not fix the issue if the problem is on the deployment client side3. Therefore, option C is the correct answer, and options A, B, and D are incorrect.

1: Troubleshoot deployment server issues 2: Configure deployment clients 3: Hardware and software requirements for the deployment server

As a best practice, the internal licensing logs should be stored on the license server. The license server is a Splunk instance that manages the distribution and enforcement of licenses in a Splunk deployment. The license server generates internal licensing logs that contain information about the license usage, violations, warnings, and pools. The internal licensing logs should be stored on the license server itself, because they are relevant to the license server’s role and function. Storing the internal licensing logs on the license server also simplifies the license monitoring and troubleshooting process. The internal licensing logs should not be stored on the indexing layer, the deployment layer, or the search head layer, because they are not related to the roles and functions of these layers. Storing the internal licensing logs on these layers would also increase the network traffic and disk space consumption

Per the Splunk Enterprise Licensing Documentation, a license peer (such as an indexer or search head) must regularly communicate with its license manager to report data usage and verify license validity. Splunk allows a 72-hour grace period during which the peer continues operating normally even if communication with the license manager fails.

If this communication is not re-established within 72 hours, the peer enters a “license violation” state. In this state, the system blocks all search activities, including ad-hoc and scheduled searches, but continues to ingest and index data. Administrative and licensing-related searches may still run for diagnostic purposes, but user searches are restricted.

The intent of this design is to prevent prolonged unlicensed data ingestion while ensuring the environment remains compliant. The 72-hour rule is hard-coded in Splunk Enterprise and applies uniformly across license types (Enterprise or Distributed). This ensures consistent licensing enforcement across distributed deployments.

Warnings are generated during the grace period, but after 72 hours, searches are automatically blocked until the peer successfully reconnects to its license manager.

References (Splunk Enterprise Documentation):

• Managing Licenses in a Distributed Environment

• License Manager and Peer Communication Workflow

• Splunk License Enforcement and Violation Behavior

• Splunk Enterprise Admin Manual – License Usage and Reporting Policies

 The following is a good practice for a search head cluster deployer: The deployer must be used to distribute non-replicable configurations to search head cluster members. Non-replicable configurations are the configurations that are not replicated by the search factor, such as the apps and the server.conf settings. The deployer is the Splunk server role that distributes these configurations to the search head cluster members, ensuring that they have the same configuration. The deployer does not only distribute configurations to search head cluster members when they “phone home”, as this would cause configuration inconsistencies and delays. The deployer does not distribute configurations to search head cluster members to be valid configurations, as this implies that the configurations are invalid without the deployer. The deployer does not only distribute configurations to search head cluster members with splunk apply shcluster-bundle, as this would require manual intervention by the administrator. For more information, see Use the deployer to distribute apps and configuration updates in the Splunk documentation.

The default interval at which metrics.log generates a periodic report regarding license utilization is 60 seconds. This report contains information about the license usage and quota for each Splunk instance, as well as the license pool and stack. The report is generated every 60 seconds by default, but this interval can be changed by modifying the license_usage stanza in the metrics.conf file. The other intervals (10 seconds, 30 seconds, and 300 seconds) are not the default values, but they can be set by the administrator if needed. For more information, see About metrics.log and Configure metrics.log in the Splunk documentation.

The curl https://serverhost:8089/services/admin/inputstatus/TailingProcessor:FileStatus command is used to check the status of the tailed files when troubleshooting monitor inputs. Monitor inputs are inputs that monitor files or directories for new data and send the data to Splunk for indexing. The TailingProcessor:FileStatus endpoint returns information about the files that are being monitored by the Tailing Processor, such as the file name, path, size, position, and status. The splunk cmd btool inputs list | tail command is used to list the inputs configurations from the inputs.conf file and pipe the output to the tail command. The splunk cmd btool check inputs layer command is used to check the inputs configurations for syntax errors and layering. The curl https://serverhost:8089/services/admin/inputstatus/TailingProcessor:Tailstatus command does not exist, and it is not a valid endpoint.

When converting from a single-site to a multi-site cluster, existing single-site clustered buckets will maintain replication as required according to the single-site policies, but never age out. Single-site clustered buckets are buckets that were created before the conversion to a multi-site cluster. These buckets will continue to follow the single-site replication and search factors, meaning that they will have the same number of copies and searchable copies across the cluster, regardless of the site. These buckets will never age out, meaning that they will never be frozen or deleted, unless they are manually converted to multi-site buckets. Single-site clustered buckets will not continue to replicate within the origin site, because they will be distributed across the cluster according to the single-site policies. Single-site clustered buckets will not be replicated across all peers in the multi-site cluster, because they will follow the single-site replication factor, which may be lower than the multi-site total replication factor. Single-site clustered buckets will not stop replicating within the single-site and remain on the indexer they reside on, because they will still be subject to the replication and availability rules of the cluster

Comprehensive and Detailed Explanation (From Splunk Enterprise Documentation)Splunk’s official planning guidance explains that the deployment cycle begins with defining the system architecture, capacity planning, platform design, and topologies. Splunk states that before any component is installed, administrators must complete “infrastructure planning and buildout,” which includes determining indexer capacity, search head roles, clustering strategy, storage layout, and performance requirements. This foundational step ensures that all Splunk components have the proper hardware, network design, and scaling expectations.

After the environment is built, Splunk documentation states that the next stage is “deployment and data onboarding,” which includes configuring indexers, forwarders, parsing rules, event processing pipelines, data source validation, and enrichment steps such as field extractions, tagging, event types, and CIM alignment. Splunk describes this as the phase where you bring in data and confirm correctness, completeness, and normalization.

Only after the system is stable and populated with data does Splunk recommend “user planning and rollout”, which includes developing dashboards, roles, knowledge objects, search best practices, and enabling user access. Splunk emphasizes that user onboarding should occur last, once infrastructure and data pipelines are fully validated.

Comprehensive and Detailed Explanation (From Splunk Enterprise Documentation)Splunk’s Search Head Clustering documentation explains that the cluster uses a majority-based election system. A captain is elected only when a node sees more than half of the cluster. In a two-site design where site1 has the majority of members, Splunk states that the majority site continues normal operation during a network partition. The minority site (site2) is not allowed to elect a captain and should not promote itself.

Splunk specifically warns administrators not to enable static captain on a minority site during a network split. Doing so creates two independent clusters, leading to configuration divergence and severe data-consistency issues. The documentation emphasizes that static captain should only be used for a complete loss of majority, not for a site partition.

Because Site1 maintains majority, it remains the active cluster and site2 does not perform any actions. Splunk states that minority-site members should simply wait until network communication is restored.

Thus the correct answer is B: No action is required.

The Indexer Discovery feature enables forwarders to dynamically connect to the available peer nodes in an indexer cluster. To use this feature, the manager node must be configured with the [indexer_discovery] stanza and a pass4SymmKey value. The forwarders must also be configured with the same pass4SymmKey value and the master_uri of the manager node. The pass4SymmKey value must be encrypted using the splunk _encrypt command. Therefore, option A indicates that the Indexer Discovery feature has not been fully configured on the manager node, because the pass4SymmKey value is not encrypted. The other options are not related to the Indexer Discovery feature. Option B shows the configuration of a forwarder that is part of an indexer cluster. Option C shows the configuration of a manager node that is part of an indexer cluster. Option D shows an invalid configuration of the [indexer_discovery] stanza, because the pass4SymmKey value is not encrypted and does not match the forwarders’ pass4SymmKey value12

1: https://docs.splunk.com/Documentation/Splunk/9.1.2/Indexer/indexerdiscovery 2: https://docs.splunk.com/Documentation/Splunk/9.1.2/Security/Secureyourconfigurationfiles#Encrypt_the_pass4SymmKey_setting_in_server.conf

Two indexers clustered is the recommended deployment for a customer who plans to ingest 600 GB of data per day into Splunk, has six concurrent users, and wants high data availability and high search performance. This deployment will provide enough indexing capacity and search concurrency for the customer’s needs, while also ensuring data replication and searchability across the cluster. The customer can also save on the hardware cost by using only two indexers. Two indexers not in a cluster will not provide high data availability, as there is no data replication or failover. Three indexers not in a cluster will provide more indexing capacity and search concurrency, but also more hardware cost and no data availability. The customer’s data retention period, number of long searches, or volume of saved/scheduled searches are not relevant for determining the number of indexers. For more information, see [Reference hardware] and [About indexer clusters and index replication] in the Splunk documentation.

The best practice for ingesting syslog data from network devices on port 514 into Splunk is to configure syslog to write logs and use a Splunk forwarder to collect the logs. This practice will ensure that the data is reliably collected and forwarded to Splunk, without losing any data or overloading the Splunk indexer. Configuring syslog to send the data to multiple Splunk indexers will not guarantee data reliability, as syslog is a UDP protocol that does not provide acknowledgment or delivery confirmation. Using a Splunk indexer to collect a network input on port 514 directly will not provide data reliability or load balancing, as the indexer may not be able to handle the incoming data volume or distribute it to other indexers. Using a Splunk forwarder to collect the input on port 514 and forward the data will not provide data reliability, as the forwarder may not be able to receive the data from syslog or buffer it in case of network issues. For more information, see [Get data from TCP and UDP ports] and [Best practices for syslog data] in the Splunk documentation.

 The following statements are true about Splunk indexer clustering:

All peer nodes must run exactly the same Splunk version. This is a requirement for indexer clustering, as different Splunk versions may have different data formats or features that are incompatible with each other. All peer nodes must run the same Splunk version as the master node and the search heads that connect to the cluster.

The search head must run the same or a later Splunk version than the peer nodes. This is a recommendation for indexer clustering, as a newer Splunk version may have new features or bug fixes that improve the search functionality or performance. The search head should not run an older Splunk version than the peer nodes, as this may cause search errors or failures. The following statements are false about Splunk indexer clustering:

The master node must run the same or a later Splunk version than the search heads. This is not a requirement or a recommendation for indexer clustering, as the master node does not participate in the search process. The master node should run the same Splunk version as the peer nodes, as this ensures the cluster compatibility and functionality.

The peer nodes must run the same or a later Splunk version than the master node. This is not a requirement or a recommendation for indexer clustering, as the peer nodes do not coordinate the cluster activities. The peer nodes should run the same Splunk version as the master node, as this ensures the cluster compatibility and functionality. For more information, see [About indexer clusters and index replication] and [Upgrade an indexer cluster] in the Splunk documentation.

According to Splunk Enterprise administrative documentation, the btool utility is used to troubleshoot configuration settings by merging and displaying effective configurations from all configuration files (system, app, and local levels). When diagnosing license master configuration issues on a search peer or any cluster node, Splunk recommends using the command that specifically lists the license-related stanzas from server.conf.

The correct command is:

splunk cmd btool server list license --debug

This command reveals all configuration parameters under the [license] stanza, including those related to license master connections such as master_uri, pass4SymmKey, and disabled flags. The --debug flag ensures full path tracing of each configuration file, making it easy to identify conflicting or missing parameters that cause communication or validation errors between a search peer and the license master.

Other commands listed, such as btool server list clustering, are meant for diagnosing cluster configuration issues (like replication or search factors), not licensing. The check --debug command only validates syntax and structure, not specific configuration errors tied to licensing. Therefore, the correct and Splunk-documented method for diagnosing license configuration problems on a search peer is to inspect the license stanza using the btool server list license --debug command.

References (Splunk Enterprise Documentation):

• btool Command Reference – Troubleshooting Configuration Issues

• server.conf – License Configuration Parameters

• Managing Distributed License Configurations in Search Peer Clusters

• Troubleshooting License Master and Peer Connectivity

 A Technical Add-On (TA) is a Splunk app that contains configurations for data collection, parsing, and enrichment. It can also enable event data for a data model, which is useful for creating dashboards and reports. Therefore, before installing a TA, it is important to identify the number of scheduled or real-time searches that will use the data model, and to validate if the TA enables event data for a data model. The number of forwarders that the TA can support is not relevant, as the TA is installed on the indexer or search head, not on the forwarder. The installation location of the TA depends on the type of data and the use case, so it is not a fixed requirement

The following security option must be explicitly configured, as it is not enabled by default:

Certificate authentication between forwarders and indexers. This option allows the forwarders and indexers to verify each other’s identity using SSL certificates, which prevents unauthorized data transmission or spoofing attacks. This option is not enabled by default, as it requires the administrator to generate and distribute the certificates for the forwarders and indexers. For more information, see [Secure the communication between forwarders and indexers] in the Splunk documentation. The following security options are enabled by default:

Data encryption between Splunk Web and splunkd. This option encrypts the communication between the Splunk Web interface and the splunkd daemon using SSL, which prevents data interception or tampering. This option is enabled by default, as Splunk provides a self-signed certificate for this purpose. For more information, see [About securing Splunk Enterprise with SSL] in the Splunk documentation.

Certificate authentication between Splunk Web and search head. This option allows the Splunk Web interface and the search head to verify each other’s identity using SSL certificates, which prevents unauthorized access or spoofing attacks. This option is enabled by default, as Splunk provides a self-signed certificate for this purpose. For more information, see [About securing Splunk Enterprise with SSL] in the Splunk documentation.

Data encryption for distributed search between search heads and indexers. This option encrypts the communication between the search heads and the indexers using SSL, which prevents data interception or tampering. This option is enabled by default, as Splunk provides a self-signed certificate for this purpose. For more information, see [Secure your distributed search environment] in the Splunk documentation.

The minimum reference server specification for a Splunk indexer is 12 CPU cores, 12GB RAM, and 800 IOPS. This specification is based on the assumption that the indexer will handle an average indexing volume of 100GB per day, with a peak of 300GB per day, and a typical search load of 1 concurrent search per 1GB of indexing volume. The other specifications are either higher or lower than the minimum requirement. For more information, see [Reference hardware] in the Splunk documentation.

The following statements describe licensing in a clustered Splunk deployment: Free licenses do not support clustering, and replicated data does not count against licensing. Free licenses are limited to 500 MB of daily indexing volume and do not allow distributed searching or clustering. To enable clustering, a license with a higher volume limit and distributed features is required. Replicated data is data that is copied from one peer node to another for the purpose of high availability and load balancing. Replicated data does not count against licensing, because it is not new data that is ingested by Splunk. Only the original data that is indexed by the peer nodes counts against licensing. Each cluster member does not require its own clustering license, because clustering licenses are shared among the cluster members. Cluster members must share the same license pool and license master, because the license master is responsible for distributing licenses to the cluster members and enforcing the license limits

To resync a stale KV Store member in a search head cluster, you need to stop the search head that has the stale KV Store member, run the command splunk clean kvstore --local, and then restart the search head. This triggers the initial synchronization from other KV Store members12.

The command splunk resync kvstore [-source sourceId] is used to resync the entire KV Store cluster from one of the members, not a single member. This command can only be invoked from the node that is operating as search head cluster captain2.

The command splunk clean eventdata -local is used to delete all indexed data from a standalone indexer or a cluster peer node, not to resync the KV Store3.

 An indexer clustering requirement is that the cluster members must share the same license pool and license master. A license pool is a group of licenses that are assigned to a set of Splunk instances. A license master is a Splunk instance that manages the distribution and enforcement of licenses in a pool. In an indexer cluster, all cluster members must belong to the same license pool and report to the same license master, to ensure that the cluster does not exceed the license limit and that the license violations are handled consistently. An indexer cluster does not require shared storage, because each cluster member has its own local storage for the index data. An indexer cluster does not have to reside on a dedicated rack, because the cluster members can be located on different physical or virtual machines, as long as they can communicate with each other. An indexer cluster does not have to have at least three members, because a cluster can have as few as two members, although this is not recommended for high availability

When building a deployment plan, the architect should perform the following tasks:

Use case checklist. A use case checklist is a document that lists the use cases that the deployment will support, along with the data sources, the data volume, the data retention, the data model, the dashboards, the reports, the alerts, and the roles and permissions for each use case. A use case checklist helps to define the scope and the functionality of the deployment, and to identify the dependencies and the requirements for each use case1

Inventory data sources. An inventory of data sources is a document that lists the data sources that the deployment will ingest, along with the data type, the data format, the data location, the data collection method, the data volume, the data frequency, and the data owner for each data source. An inventory of data sources helps to determine the data ingestion strategy, the data parsing and enrichment, the data storage and retention, and the data security and compliance for the deployment1

Review network topology. A review of network topology is a process that examines the network infrastructure and the network connectivity of the deployment, along with the network bandwidth, the network latency, the network security, and the network monitoring for the deployment. A review of network topology helps to optimize the network performance and reliability, and to identify the network risks and mitigations for the deployment1

Installing Splunk apps is not a task that the architect should perform when building a deployment plan, as it is a task that the administrator should perform when implementing the deployment plan. Installing Splunk apps is a technical activity that requires access to the Splunk instances and the Splunk configurations, which are not available at the planning stage

According to the Splunk documentation1, increasing the current deployment client phone home interval can minimize performance issues by reducing the frequency of communication between the clients and the deployment server. This can also reduce the network traffic and the load on the deployment server. The other options are false because:

Modifying deploymentclient.conf to change from a Pull to Push mechanism is not possible, as Splunk does not support a Push mechanism for deployment server2.

Reducing the number of apps in the Manager Node repository will not affect the performance of the deployment server, as the apps are only downloaded when there is a change in the configuration or a new app is added3.

Decreasing the current deployment client phone home interval will increase the performance issues, as it will increase the frequency of communication between the clients and the deployment server, resulting in more network traffic and load on the deployment server1.

This is the fall-back method for Splunk if .delta replication fails during knowledge bundle replication. Knowledge bundle replication is the process of distributing the knowledge objects, such as lookups, macros, and field extractions, from the search head cluster to the indexer cluster1. Splunk uses two methods of knowledge bundle replication: .delta replication and .bundle replication1. .Delta replication is the default and preferred method, as it only replicates the changes or updates to the knowledge objects, which reduces the network traffic and disk space usage1. However, if .delta replication fails for some reason, such as corrupted files or network errors, Splunk automatically switches to .bundle replication, which replicates the entire knowledge bundle, regardless of the changes or updates1. This ensures that the knowledge objects are always synchronized between the search head cluster and the indexer cluster, but it also consumes more network bandwidth and disk space1. The other options are not valid fall-back methods for Splunk. Option A, restarting splunkd, is not a method of knowledge bundle replication, but a way to restart the Splunk daemon on a node2. This may or may not fix the .delta replication failure, but it does not guarantee the synchronization of the knowledge objects. Option B, .delta replication, is not a fall-back method, but the primary method of knowledge bundle replication, which is assumed to have failed in the question1. Option D, restarting mongod, is not a method of knowledge bundle replication, but a way to restart the MongoDB daemon on a node3. This is not related to the knowledge bundle replication, but to the KV store replication, which is a different process3. Therefore, option C is the correct answer, and options A, B, and D are incorrect.

1: How knowledge bundle replication works 2: Start and stop Splunk Enterprise 3: Restart the KV store

 When planning a search head cluster, the following statement is true: All indexers must belong to the underlying indexer cluster (no standalone indexers). A search head cluster is a group of search heads that share configurations, apps, and search jobs. A search head cluster requires an indexer cluster as its data source, meaning that all indexers that provide data to the search head cluster must be members of the same indexer cluster. Standalone indexers, or indexers that are not part of an indexer cluster, cannot be used as data sources for a search head cluster. All search heads do not have to use the same operating system, as long as they are compatible with the Splunk version and the indexer cluster. All search heads do not have to be members of the cluster, as standalone search heads can also search the indexer cluster, but they will not have the benefits of configuration replication and load balancing. The search head captain does not have to be assigned to the largest search head in the cluster, as the captain is dynamically elected from among the cluster members based on various criteria, such as CPU load, network latency, and search load.

Splunk Enterprise performs a CRC check against the first and last 256 bytes of a file by default, as stated in the inputs.conf specification. This is controlled by the initCrcLength parameter, which can be changed if needed. The CRC check helps Splunk Enterprise to avoid re-indexing the same file twice, even if it is renamed or rotated, as long as the content does not change. However, this also means that Splunk Enterprise might miss some files that have the same CRC but different content, especially if they have identical headers. To avoid this, the crcSalt parameter can be used to add some extra information to the CRC calculation, such as the full file path or a custom string. This ensures that each file has a unique CRC and is indexed by Splunk Enterprise. You can read more about crcSalt and initCrcLength in the How log file rotation is handled documentation.

The splunk offline --enforce-counts command is the official and documented method used to gracefully decommission a search peer (indexer) from an indexer cluster in Splunk Enterprise. This command ensures that all replication and search factors are maintained before the peer is removed.

When executed, Splunk initiates a controlled shutdown process for the peer node. The Cluster Manager verifies that sufficient replicated copies of all bucket data exist across the remaining peers according to the configured replication_factor (RF) and search_factor (SF). The --enforce-counts flag specifically enforces that replication and search counts remain intact before the peer fully detaches from the cluster, ensuring no data loss or availability gap.

The sequence typically includes:

Validating cluster state and replication health.

Rolling off the peer’s data responsibilities to other peers.

Removing the peer from the active cluster membership list once replication is complete.

Other options like disablepeer, decommission, or remove cluster-peers are not valid Splunk commands. Therefore, the correct documented method is to use:

splunk offline --enforce-counts

References (Splunk Enterprise Documentation):

• Indexer Clustering: Decommissioning a Peer Node

• Managing Peer Nodes and Maintaining Data Availability

• Splunk CLI Command Reference – splunk offline

• Cluster Manager and Peer Maintenance Procedures

 To reduce the captain’s work load in a search head cluster, the setting that will prevent scheduled searches from running on the captain is captain_is_adhoc_searchhead = true (on the current captain). This setting will designate the current captain as an ad hoc search head, which means that it will not run any scheduled searches, but only ad hoc searches initiated by users. This will reduce the captain’s work load and improve the search head cluster performance. The adhoc_searchhead = true (on all members) setting will designate all search head cluster members as ad hoc search heads, which means that none of them will run any scheduled searches, which is not desirable. The adhoc_searchhead = true (on the current captain) setting will have no effect, as this setting is ignored by the captain. The captain_is_adhoc_searchhead = true (on all members) setting will have no effect, as this setting is only applied to the current captain. For more information, see Configure the captain as an ad hoc search head in the Splunk documentation.

The master node is the Splunk server role that regulates the functioning of the indexer cluster. The master node coordinates the activities of the peer nodes, such as data replication, data searchability, and data recovery. The master node also manages the cluster configuration bundle and distributes it to the peer nodes. The indexer is the Splunk server role that indexes the incoming data and makes it searchable. The deployer is the Splunk server role that distributes apps and configuration updates to the search head cluster members. The monitoring console is the Splunk server role that monitors the health and performance of the Splunk deployment. For more information, see About indexer clusters and index replication in the Splunk documentation.

 The algorithm used to determine captaincy in a Splunk search head cluster is Raft distributed consensus. Raft is a consensus algorithm that is used to elect a leader among a group of nodes in a distributed system. In a Splunk search head cluster, Raft is used to elect a captain among the cluster members. The captain is the cluster member that is responsible for coordinating the search activities, replicating the configurations and apps, and pushing the knowledge bundles to the search peers. The captain is dynamically elected based on various criteria, such as CPU load, network latency, and search load. The captain can change over time, depending on the availability and performance of the cluster members. Rapt, Rift, and Round-robin are not valid algorithms for determining captaincy in a Splunk search head cluster

As per Splunk Enterprise Search Performance documentation, the most significant factor affecting search performance when querying across a large number of buckets is disk I/O throughput. A search that spans “All Time” forces Splunk to inspect all historical buckets (hot, warm, cold, and potentially frozen if thawed), even if only a few events match the query. This dramatically increases the amount of data read from disk, making the search bound by I/O performance rather than CPU or memory.

Increasing the number of indexing pipelines (Option B) only benefits data ingestion, not search performance. Changing to a real-time search (Option D) does not help because real-time searches are optimized for streaming new data, not historical queries. The indexed_realtime_use_by_default setting (Option C) applies only to streaming indexed real-time searches, not historical “All Time” searches.

To improve performance for such searches, Splunk documentation recommends enhancing disk I/O capability — typically through SSD storage, increased disk bandwidth, or optimized storage tiers. Additionally, creating summary indexes or accelerated data models may help for repeated “All Time” queries, but the most direct improvement comes from faster disk performance since Splunk must scan large numbers of buckets for even small result sets.

References (Splunk Enterprise Documentation):

• Search Performance Tuning and Optimization

• Understanding Bucket Search Mechanics and Disk I/O Impact

• limits.conf Parameters for Search Performance

• Storage and Hardware Sizing Guidelines for Indexers and Search Heads

When installing Enterprise Security on a Search Head Cluster (SHC), the following steps should be done: Install Enterprise Security on the deployer, and use the deployer to deploy Enterprise Security to the cluster members. Enterprise Security is a premium app that provides security analytics and monitoring capabilities for Splunk. Enterprise Security can be installed on a SHC by using the deployer, which is a standalone instance that distributes apps and other configurations to the SHC members. Enterprise Security should be installed on the deployer first, and then deployed to the cluster members using the splunk apply shcluster-bundle command. Enterprise Security should not be installed on a staging instance, because a staging instance is not part of the SHC deployment process. Enterprise Security configurations should not be copied to the deployer, because they are already included in the Enterprise Security app package.

According to the Splunk documentation1, any search head in the cluster can update the KV store collection. The KV store collection is replicated across all the cluster members, and any write operation is delegated to the KV store captain, who then synchronizes the changes with the other members. The KV store primary search head is not a valid term, as there is no such role in a search head cluster. The other options are false because:

The search head cluster captain is not the only node that can update the KV store collection, as any member can initiate a write operation1.

Any search head except the captain can also update the KV store collection, as the write operation will be delegated to the captain1.

Splunk’s KV Store management documentation specifies that when converting a static CSV lookup to a KV Store lookup, the lookup data is stored in a MongoDB-based collection defined in collections.conf. To ensure that the KV Store lookup is replicated and available across all search head cluster members, administrators must include the attribute replicate=true within the collections.conf file.

This configuration instructs Splunk to replicate the KV Store collection’s data to all members in the Search Head Cluster (SHC), enabling consistent access and reliability across the cluster. Without this attribute, the KV Store collection would remain local to a single search head, making it unavailable for automatic lookups performed by other members.

Here’s an example configuration snippet from collections.conf:

[customer_lookup]

replicate = true

field.name = string

field.age = number

The attribute repFactor=true (mentioned in Options A and D) is unrelated to KV Store behavior—it applies to index replication, not KV Store replication. Similarly, replicate=true in lookups.conf (Option B) has no effect, as KV Store replication is controlled exclusively via collections.conf.

Once properly configured, the lookup can be defined in transforms.conf and referenced in props.conf for automatic lookup functionality.

References (Splunk Enterprise Documentation):

• KV Store Collections and Configuration – collections.conf Reference

• Managing KV Store Data in Search Head Clusters

• Automatic Lookup Configuration Using KV Store

• Splunk Enterprise Admin Manual – Distributed KV Store Replication Settings

Three peers can fail. This is the maximum number of search peers that can fail before data becomes unsearchable in the indexer cluster with the given characteristics. The searchability of the data depends on the Search Factor, which is the number of searchable copies of each bucket that the cluster maintains across the set of peer nodes1. In this case, the Search Factor is 3, which means that each bucket has three searchable copies distributed among the 10 search peers. If three or fewer search peers fail, the cluster can still serve the data from the remaining searchable copies. However, if four or more search peers fail, the cluster may lose some searchable copies and the data may become unsearchable. The other options are not correct, as they either underestimate or overestimate the number of search peers that can fail before data becomes unsearchable. Therefore, option C is the correct answer, and options A, B, and D are incorrect.

1: Configure the search factor

According to the Splunk documentation1, when migrating an indexer cluster from single-site to multi-site, you must remove the existing single-site attributes from the server.conf file of each peer node. These attributes include replication_factor, search_factor, and cluster_label. You must also restart each peer node after removing the attributes. The other options are false because:

Multi-site policies will apply only to the data created after migration, unless you configure the manager node to convert legacy buckets to multi-site1.

All peer nodes do not need to run the same version of Splunk, as long as they are compatible with the manager node2.

Single-site buckets can be converted to multi-site buckets by changing the constrain_singlesite_buckets setting in the manager node’s server.conf file to "false"1.

The Splunk Managing Indexes and Storage Documentation explains that when multiple data sources with significantly different ingestion rates share a single index, index bucket management is governed by volume-based rotation, not by source or time. This means that high-volume data causes buckets to fill and roll more quickly, which in turn causes low-volume data to age out prematurely, even if it is relatively recent — hence Option C is correct.

Additionally, because Splunk organizes data within index buckets based on event time and storage characteristics, low-volume data mixed with high-volume data results in inefficient searches for smaller datasets. Queries that target the low-volume source will have to scan through the same large number of buckets containing the high-volume data, leading to slower-than-necessary search performance — Option B.

Compression efficiency (Option A) and performance optimization through data mixing (Option D) are not influenced by mixing volume patterns; these are determined by the event structure and compression algorithm, not source diversity. Splunk best practices recommend separating data sources into different indexes based on usage, volume, and retention requirements to optimize both performance and lifecycle management.

References (Splunk Enterprise Documentation):

• Managing Indexes and Storage – How Splunk Manages Buckets and Data Aging

• Splunk Indexing Performance and Data Organization Best Practices

• Splunk Enterprise Architecture and Data Lifecycle Management

• Best Practices for Data Volume Segregation and Retention Policies

In the context of splunkd.log events written to the _internal index, the field that identifies the specific log channel is the "channel" field. This information is confirmed by the Splunk Common Information Model (CIM) documentation, where "channel" is listed as a field name associated with Splunk Audit Logs​​.

Search is not locked out when a customer has installed a 500GB Enterprise license and a 300GB, no enforcement license on the same license master. The no enforcement license allows the customer to exceed the license quota without locking search, but violations are still recorded. The customer can ingest up to 800GB of data per day without violating the license, but if they ingest more than that, they will incur a violation. However, the violation will not lock search, as the no enforcement license overrides the enforcement policy of the Enterprise license. For more information, see [No enforcement licenses] and [License violations] in the Splunk documentation.

 ITSI can impact the planning of a Splunk deployment depending on the Key Performance Indicators (KPIs) that are being tracked. KPIs are metrics that measure the health and performance of IT services and business processes. ITSI collects, analyzes, and displays KPI data from various data sources in Splunk. Depending on the number, frequency, and complexity of the KPIs, additional infrastructure may be needed to support the data ingestion, processing, and visualization. ITSI does not require a dedicated deployment server, nor does it affect the number of users using ITSI. ITSI in a Splunk deployment does require additional hardware resources, such as CPU, memory, and disk space, to run the ITSI components and apps

 The index-time props.conf attributes that impact indexing performance are LINE_BREAKER and SHOULD_LINEMERGE. These attributes determine how Splunk breaks the incoming data into events and whether it merges multiple events into one. These operations can affect the indexing speed and the disk space consumption. The REPORT attribute does not impact indexing performance, as it is used to apply transforms at search time. The ANNOTATE_PUNCT attribute does not impact indexing performance, as it is used to add punctuation metadata to events at search time. For more information, see [About props.conf and transforms.conf] in the Splunk documentation.

The site_mappings attribute should be added to the master node’s server.conf file when decommissioning a site in an indexer cluster. The site_mappings attribute is used to specify how the master node should reassign the buckets from the decommissioned site to the remaining sites. The site_mappings attribute is a comma-separated list of site pairs, where the first site is the decommissioned site and the second site is the destination site. For example, site_mappings = site1:site2,site3:site4 means that the buckets from site1 will be moved to site2, and the buckets from site3 will be moved to site4. The available_sites attribute is used to specify which sites are currently available in the cluster, and it is automatically updated by the master node. The site_search_factor and site_replication_factor attributes are used to specify the number of searchable and replicated copies of each bucket for each site, and they are not affected by the decommissioning process

Index files (. tsidx files) are the main components of an index that store the raw data and the inverted index of terms. They take the most space in an index, especially if the raw data has many unique terms that increase the size of the inverted index. Bloom filters, source metadata, and sourcetype metadata are much smaller in comparison and do not depend on the number of unique terms in the raw data.

The master node distributes configuration bundles to peer nodes in the slave-apps directory under $SPLUNK_HOME/etc. The configuration bundle method is the only supported method for managing common configurations and app deployment across the set of peers. It ensures that all peers use the same versions of these files1. Bundles typically contain a subset of files (configuration files and assets) from $SPLUNK_HOME/etc/system, $SPLUNK_HOME/etc/apps, and $SPLUNK_HOME/etc/users2. The process of distributing knowledge bundles means that peers by default receive nearly the entire contents of the search head’s apps3.

A dedicated deployment server is a Splunk instance that manages the distribution of configuration updates and apps to a set of deployment clients, such as forwarders, indexers, or search heads. A dedicated deployment server should be used when there are more than 50 deployment clients, because this number exceeds the recommended limit for a non-dedicated deployment server. A non-dedicated deployment server is a Splunk instance that also performs other roles, such as indexing or searching. Using a dedicated deployment server can improve the performance, scalability, and reliability of the deployment process. Option C is the correct answer. Option A is incorrect because the number of search peers does not affect the need for a dedicated deployment server. Search peers are indexers that participate in a distributed search. Option B is incorrect because the number of apps to deploy does not affect the need for a dedicated deployment server. Apps are packages of configurations and assets that provide specific functionality or views in Splunk. Option D is incorrect because the number of server classes does not affect the need for a dedicated deployment server. Server classes are logical groups of deployment clients that share the same configuration updates and apps12

1: https://docs.splunk.com/Documentation/Splunk/9.1.2/Updating/Aboutdeploymentserver 2: https://docs.splunk.com/Documentation/Splunk/9.1.2/Updating/Whentousedeploymentserver

Per the Search Head Clustering (SHC) Deployer Administration Guide, the deployer is responsible for distributing configuration bundles, apps, and baseline settings to all members of a Search Head Cluster (SHC). However, the replication of knowledge objects (Option A) is not handled by the deployer.

Knowledge object replication—covering items such as saved searches, dashboards, lookups, and alerts—is managed internally within the Search Head Cluster using the captain node. The captain coordinates replication among all SHC members using a mechanism called Knowledge Object Replication Framework, which ensures that user-created or runtime configuration changes (e.g., dashboards saved in Splunk Web) are automatically shared across members.

In contrast, the deployer’s primary responsibilities include:

Deploying and updating baseline app configurations (Option B).

Distributing non-replicated, non-runtime configuration updates like props, transforms, and inputs (Option C).

Assisting in the initial migration of apps and configurations into a cluster during setup (Option D).

Therefore, while the deployer handles static configuration management, knowledge object replication is performed dynamically by the SHC itself under captain control, making Option A the correct answer.

References (Splunk Enterprise Documentation):

• Search Head Clustering: How the Deployer Works

• Managing Knowledge Object Replication in Search Head Clusters

• Splunk Enterprise Admin Manual – Deployer vs. Captain Responsibilities

• Distributing Apps and Configurations with the Deployer

Splunk indexing is read/write intensive, as it involves reading data from various sources, writing data to disk, and reading data from disk for searching and reporting. Therefore, it is important to select the appropriate disk storage solution for each deployment, based on the performance, reliability, and cost requirements. The recommended RAID setup for Splunk indexers is RAID 10 (1 + 0), as it provides the best balance of performance and reliability. RAID 10 combines the advantages of RAID 1 (mirroring) and RAID 0 (striping), which means that it offers both data redundancy and data distribution. RAID 10 can tolerate multiple disk failures, as long as they are not in the same mirrored pair, and it can improve the read and write speed, as it can access multiple disks in parallel2

High performance SAN (Storage Area Network) can be used for Splunk indexers, but it is not recommended, as it is more expensive and complex than local disks. SAN also introduces additional network latency and dependency, which can affect the performance and availability of Splunk indexers. SAN is more suitable for Splunk search heads, as they are less read/write intensive and more CPU intensive2

NFS (Network File System) should not be used for storing hot and warm buckets, as it can cause data corruption, data loss, and performance degradation. NFS is a network-based file system that allows multiple clients to access the same files on a remote server. NFS is not compatible with Splunk index replication and search head clustering, as it can cause conflicts and inconsistencies among the Splunk instances. NFS is also slower and less reliable than local disks, as it depends on the network bandwidth and availability. NFS can be used for storing cold and frozen buckets, as they are less frequently accessed and less critical for Splunk operations2

Virtualized environments are not usually preferred over bare metal for Splunk indexers, as they can introduce additional overhead and complexity. Virtualized environments can affect the performance and reliability of Splunk indexers, as they share the physical resources and the network with other virtual machines. Virtualized environments can also complicate the monitoring and troubleshooting of Splunk indexers, as they add another layer of abstraction and configuration. Virtualized environments can be used for Splunk indexers, but they require careful planning and tuning to ensure optimal performance and availability2

Setting site=site0 on all Search Head Cluster members disables search site affinity. Search site affinity is a feature that allows search heads to preferentially search the peer nodes that are in the same site as the search head, to reduce network latency and bandwidth consumption. By setting site=site0, which is a special value that indicates no site, the search heads will search all peer nodes regardless of their site. Setting site=site0 does not set all members to dynamic captaincy, enable multisite search artifact replication, or enable automatic search site affinity discovery. Dynamic captaincy is a feature that allows any member to become the captain, and it is enabled by default. Multisite search artifact replication is a feature that allows search artifacts to be replicated across sites, and it is enabled by setting site_replication_factor to a value greater than 1. Automatic search site affinity discovery is a feature that allows search heads to automatically determine their site based on the network latency to the peer nodes, and it is enabled by setting site=auto

The correct answer is C. Increase the parallellngestionPipelines attribute. This is an option in limits.conf that may provide performance benefits at the forwarding tier, as it allows the forwarder to process multiple data inputs in parallel1. The parallellngestionPipelines attribute specifies the number of pipelines that the forwarder can use to ingest data from different sources1. By increasing this value, the forwarder can improve its throughput and reduce the latency of data delivery1. The other options are not effective options to provide performance benefits at the forwarding tier. Option A, enabling the indexed_realtime_use_by_default attribute, is not recommended, as it enables the forwarder to send data to the indexer as soon as it is received, which may increase the network and CPU load and degrade the performance2. Option B, increasing the maxKBps attribute, is not a good option, as it increases the maximum bandwidth, in kilobytes per second, that the forwarder can use to send data to the indexer3. This may improve the data transfer speed, but it may also saturate the network and cause congestion and packet loss3. Option D, increasing the max_searches_per_cpu attribute, is not relevant, as it only affects the search performance on the indexer or search head, not the forwarding performance on the forwarder4. Therefore, option C is the correct answer, and options A, B, and D are incorrect.

1: Configure parallel ingestion pipelines 2: Configure real-time forwarding 3: Configure forwarder output 4: Configure search performance

According to the Splunk documentation1, the Splunk Deployer deploys app content to the etc/slave-apps/ directory on the search head cluster members by default. This directory contains the apps that the deployer distributes to the members as part of the configuration bundle. The other options are false because:

The etc/apps/ directory contains the apps that are installed locally on each member, not the apps that are distributed by the deployer2.

The etc/shcluster/ directory contains the configuration files for the search head cluster, not the apps that are distributed by the deployer3.

The etc/deploy-apps/ directory is not a valid Splunk directory, as it does not exist in the Splunk file system structure4.

The splunk edit licenser-localslave command is used to convert a Splunk instance to a license slave. This command will configure the Splunk instance to contact a license master and receive a license from it. This command should be used when the Splunk instance is part of a distributed deployment and needs to share a license pool with other instances. The splunk add licenses command is used to add a license to a Splunk instance, not to convert it to a license slave. The splunk list licenser-slaves command is used to list the license slaves that are connected to a license master, not to convert a Splunk instance to a license slave. The splunk list licenser-localslave command is used to list the license master that a license slave is connected to, not to convert a Splunk instance to a license slave. For more information, see Configure license slaves in the Splunk documentation.