Executor

The Executor allows for VM modification and state checking after creation through the use of Feature, Inject and Condition type Deputy packages. Redis is used to keep track of the installed files and their metadata.

Execution of these deputy packages is done in several steps. First the package assets are copied to the VM, then the package action is executed. In case of the Condition services stream endpoint, the package assets are copied and later the action is executed continuously with respect to the defined interval seconds until the stream is closed by Ranger.

Queueing

Due to best practice and API limitations, operations on VMs are executed one at a time per VM until max_connections is reached. This means that if a VM is currently being modified, any other operations on that VM will be queued until the previous operation is finished. This includes actions such as file transfers and running commands. For this reason the Executor implements a queue system that aims to give fair access for all operations and all VMs. The queue is primarily leveraged by the Condition stream endpoint as it is the most resource intensive. For this reason it is discouraged to use quick intervals for Condition type packages as the Openstack API is relatively slow and cannot keep up with the amount of requests. In case a queue is created, it is only evident in Ranger in the scoring graph where the interval for individual Conditions is longer than the one specified in the Deputy package.

Operations that run once, such as Inject and Feature type packages are prioritized over Condition type packages as the queue priority is based on the amount of times an action has been run.

Services

The Executor provides several gRPC services dependant on the type of Deputy package that is to be executed.

The Executor assumes that the VM is already created and running and VM tools are installed on the target VM. During setup, restarting the VM is only permissible if it is defined in the package under the Deputy packages restarts field. After setup, the VM may be restarted at any time and the streaming conditions should be able to recover.

Feature

Feature type packages allow for installing additional services, configurations and more on the VM immediately after creation. There are two gRpc endpoints for feature type packages:

create

The create endpoint is used to install the listed assets on the VM and run its accompanying action. Feature create requests are sent by Ranger immediately after the VM is created.

Request

message Feature {
  string name = 1;
  string virtual_machine_id = 2;
  FeatureType featureType = 3;
  Source source = 4;
  Account account = 5;
  repeated string environment = 7;
}

For reference, the FeatureType, Source and Account messages are defined as follows:

enum FeatureType {
  service = 0;
  configuration = 1;
  artifact = 2;
}

message Source {
    string name = 1;
    string version = 2;
}

message Account {
    string username = 1;
    string password = 2;
    string private_key = 3;
}

Successful Response

message ExecutorResponse {
    Identifier identifier = 1;
    string stdout = 2;
    string stderr = 3;
}

delete

Deletes the listed assets from the VM. Currently, if the feature installs additional files or services on the VM, they are not removed.

Request

message Identifier {
    string value = 1;
}

Successful Response

message Empty {}

Inject

Injects are in essence the same as Feature type packages with the exception that they are run on the VM during the runtime of the exercise and not immediately after the VM is created.

create

Request

message Inject {
  string name = 1;
  string virtual_machine_id = 2;
  Source source = 3;
  Account account = 4;
  repeated string to_entities = 5;
  repeated string environment = 7;
}

Successful Response

message ExecutorResponse {
    Identifier identifier = 1;
    string stdout = 2;
    string stderr = 3;
}

delete

Request

message Identifier {
    string value = 1;
}

Successful Response

message Empty {}

Condition

Conditions are scripts that run every Interval seconds on the VM and return a value between 0 and 1.0 to indicate its current state. They are installed on the VM during its creation and then continuously run with their results streamed back to Ranger unless closed by Ranger.

The condition streams may be closed by Ranger ungracefully if the Condition has served its purpose, for example when a conditional Event has been triggered, said Condition will be closed.

create

The create endpoint is used to install the Condition on the VM.

Request

message Condition {
  string name = 1;
  string virtual_machine_id = 2;
  Account account = 3;
  Source source = 4;
  string command = 5;
  int32 interval = 6;
  repeated string environment = 7;
}

Successful Response

message ExecutorResponse {
    Identifier identifier = 1;
    string stdout = 2;
    string stderr = 3;
}

stream

The stream gRPC endpoint is used to continuously run the Condition on the VM and stream the results back to Ranger.

Request

message Identifier {
    string value = 1;
}

Successful Response

The Response field indicates the ID of the condition run.

stream message ConditionStreamResponse {
  string response = 1;
  float command_return_value = 2;
}

delete

The delete gRPC endpoint is used to uninstall the Condition from the VM.

Request

message Identifier {
    string value = 1;
}

Successful Response

message Empty {}

Configuration

The following environment variables are available for the Executor handler configuration:

Global Settings

Variable	Mandatory	Type	Description
GLOBAL_HOST	Yes	string	Address where the Executor handler listens
GLOBAL_INSECURE	Yes	bool	Enables or disables SSL certificate verification

Service Ports

Variable	Mandatory	Type	Description
EXECUTOR_PORT	Yes	integer	Port where the Executor handler listens

DevStack Credentials

Variable	Mandatory	Type	Description
AUTH_URL	Yes	string	OpenStack authentication endpoint
CREDENTIAL_ID	Yes	string	OpenStack credential username
CREDENTIAL_SECRET	Yes	string	OpenStack credential password
DOMAIN	Yes	string	OpenStack domain (default: Default)
PROJECT	Yes	string	OpenStack project name
GATEWAY_NETWORK	Yes	string	Gateway network for the OpenStack environment
REGION	Yes	string	Region name for the OpenStack environment
HYPERVISOR	Yes	string	OpenStack hypervisor connection URL
HYPERVISOR_USER	Yes	string	Username for the OpenStack hypervisor
HYPERVISOR_PASSWORD	Yes	string	Password for the OpenStack hypervisor

Logging Configuration

Variable	Mandatory	Type	Description
LOG_LEVEL	No	string	Logging level (e.g., INFO, DEBUG, WARN)
LOG_DIR	No	string	Directory where logs are stored
LOG_FILE_NAME	No	string	Name of the log file

Redis Configuration

Variable	Mandatory	Type	Description
REDIS_HOST	Yes	string	Redis server address
REDIS_PORT	Yes	integer	Port where the Redis server is running
REDIS_PASSWORD	Yes	string	Password for Redis authentication

Optional Environment Variables

Variable	Mandatory	Type	Description
HYPERVISOR_WAIT_TIMEOUT	No	integer	Timeout in seconds for the hypervisor to be ready
HYPERVISOR_WAIT_INTERVAL	No	integer	Interval in seconds to check if the hypervisor is ready

Example .env Configuration

# Global Settings
GLOBAL_HOST=0.0.0.0
GLOBAL_INSECURE=True

# Service Ports
EXECUTOR_PORT=7000

# DevStack Credentials
AUTH_URL=https://openstack-auth-url:5000/v3
CREDENTIAL_ID=openstack_client_id
CREDENTIAL_SECRET=openstack_application_secret
DOMAIN=openstack_domain
PROJECT=openstack_project
GATEWAY_NETWORK=openstack_gateway_network
REGION=openstack_region
HYPERVISOR=qemu+tcp://hostname/system
HYPERVISOR_USER=openstack_hypervisor_user
HYPERVISOR_PASSWORD=openstack_hypervisor_password

# Logging Configuration
LOG_LEVEL=INFO
LOG_DIR=/var/log
LOG_FILE_NAME=ocr_openstack_plugin

# Redis Configuration
REDIS_HOST=redis-host
REDIS_PORT=6379
REDIS_PASSWORD=redis_password

# Optional
HYPERVISOR_WAIT_TIMEOUT=300
HYPERVISOR_WAIT_INTERVAL=5