Endian banner

endian.job package

Submodules

endian.job.commons module

call_job_function(function, params=None, options=None)

Call a jobsengine function and returns the result or None in case of error

chgrp(glob_expression, group)
chown(glob_expression, username)
create_bypasses(f, sign, logger=None)
create_empty_file(filename)

Create a blank file if it doesn’t exist, delete the content if the file exist

download_file_to(url, target, webuser=None, webpass=None)

Downloads the file with the url url to the target place at the file system. If you need html authentication, you may use webuser and webpass.

This function will be take care of the upstream proxy settings.

Parameters:
  • url (String) – The url of the file to download.
  • target (String) – The location to save the file.
  • webuser (String) – The html authentication username.
  • webpass (String) – The html authentication password.
file_exists(file)

Returns true if the file exist.

get_bypasses(source, destination, logger=None)
get_config(filename, logger=None)

Reads a configuration file and returns a hash

get_free_memory(logger=None)

Return the ammount of free memory in Kb

get_proxy()

Returns a dict with the upstream proxy settings of the fw.

Returns:Dict with upstream proxy settings.
Return type:dict
get_total_memory(logger=None)

Return the total ammount of memory in Kb

getoutput(command_line)

Exec an external command and returns the output as a string

getstatusoutput(command_line)

Execute an external command and return a tuple (status, output). For compatibility with ‘commands.getstatusoutput’, a trailing newline is stripped from the output.

Parameters:command_line (string) – The command line to be executed
Returns:A tuple (status, output) The returned output will contain output and/or error messages. A trailing newline is stripped from the output.
Return type:tuple
is_ipaddress(addr)

Checks if addr is and IP address

Parameters:addr (string) – address to check
Returns:True if the string is valid IP address
Return type:boolean
is_macaddress(addr)

Checks if addr is a MAC address

Parameters:addr (string) – address to check
Returns:True if the string is valid MAC address
Return type:boolean
is_valid_domain(addr)

Checks if addr is a valid domain

Parameters:addr (string) – address to check
Returns:True if the string is valid domain
Return type:boolean
monit(args, logger=None, blocking=False, monitor=False)

Send a command to the monit daemon.

Parameters:
  • args (String) – The command string for monit.
  • logger (Object with the usual logging functions.) – The logger to use.
  • blocking (bool) – ignored
  • monitor (bool) – ignored
Returns:

The string return by monit or the exit status of the operation as number. For the ‘status’ command: 0 = running, 1 = not running

Return type:

String or Number

read_file_content(filename, logger=None, readlines=False)

Read the content of a file into a string

Parameters:
  • logger (logger) – A logger instance
  • readlines (boolean) – If true, return an list of string instead of a single string
Returns:

the file content as a single string or as a list of strings (one for each line)

Return type:

a string or a list of strings

reload_service(service, logger=None, changed=True, can_reload=True, blocking=False)
remove_from_path(path, exclude=[], remove_directories=False)

Removes all the files that does not match exclude from a path. If the remove_directories parameter is True, remove also the directories. Roughly equivalent to find . ! -type d -exec rm -f – {} or find . ! -exec rm -f – {}

remove_glob(glob_expression)

Equivalent to rm -f expression, e.g. rm -f /var/run/*.pid

remove_path(path)

Equivalent to rm -rf

require_enabled_service(service, callingservice, key='ENABLED')

Creates a settingsfile in /var/lib/efw/<service>/settings.<callingservice> with <key>=on whose purpose is to enable another service because callingservice needs it running.

Parameters:
  • service (String) – The service to be enabled
  • callingservice (String) – The service which needs it enabled
  • key (String) – Name of the key which tells the service being enabled or not.
run(command_line, async=False, suppress_output=False, logger=None, env=None)

Exec an external command.

save_old_config(filename, separator='.')

Create a backup of a configuration file (filename<separator>old).

send_cmd_to_engine(cmd, params=None, options=None)

Send an internal cmd to the engine and wait for the response

set_all_true(hash, exclude=None)

Set each key to true

set_config(filename, config, logger=None)

Writes down the SettingsFile object ‘config’ to the file ‘filename’ On success returns True otherwise False.

to_bool(value, default=False)

transforms ‘’‘value’‘’ to a boolean value.

In order to return True, ‘’‘value’‘’ may be one of (case insensitive) ‘on’, 1, ‘1’, True, ‘enabled’

If the value does not correspond to True or False, for example ‘’ or None, the value of ‘’‘default’‘’ will be returned, which is False if not specified.

Parameters:
  • value (any) – value to transform to boolean
  • default (boolean) – value to return when no corresponding boolean value for ‘’‘value’‘’ could be found. Defaults to False.
Returns:

corresponding boolean value of ‘’‘value’‘’

Return type:

boolean

touch(filename)

Sets the modification and access times of file to the current time of day or create a file with default permissions if it doesn’t exist.

unrequire_enabled_service(service, callingservice)

Removes settingsfile /var/lib/efw/<service>/settings.<callingservice> which causes that the service <service> is no longer required to be enabled because of service <callingservice>.

After calling this method it does not automatically mean that <service> is disabled!

Parameters:
  • service (String) – The service to be not enabled anymore
  • callingservice (String) – The service which doesn’t it need enabled anymore
write_config(template, config_file, config_values, skip_compare_old_config=False, alter_config=None, utf8=False, logger=None, separator='.')

Write the configuration with a cheetah template

Parameters:
  • template (string) – location of the config template
  • config_file (string) – where to save the config file
  • utf8 (boolean) – if true, save as utf-8
Params config_values:
 

list which contains dictionaries or one dictionary which holds the value which substitutes the keys of the template

Returns:

a boolean which informs if the config has changed

Return type:

boolean

write_file_content(filename, content, logger=None, append=False, newline=False)

Write a string to a file

Parameters:
  • content (string) – Content to be written on the file
  • logger (logger) – A logger instance
  • append (boolean) – If true, append to the file insted ad overwrite
  • newline (boolean) – If true, add a newline after the content

endian.job.engine module

class Engine(debug=False, test=False, foreground=False, use_syslog=True, log_filename=None, timing=False, timing_filename=None, interactive=False, startup=False, restore_status=False, reload_jobs=False, multithreads=False, force_modules_import=False, requests_filename=None, first_startup=False, restart=False)

Bases: object

start()
main()

endian.job.engine_actions module

class Action(job, name, module=None, class_name=None, on=None, depends=None, exit_status=None, priority=None, options=None, not_concurrent=False, use_notifications=False, jobs_engine=None)

Bases: object

action(options={})

This method is called by engine_job._fire_action to execute the action.

  • import the module if it is not already loaded
  • create an instance of the job class
  • call the correspondent class method
depends = None
depends_services = None
emit_dependencies(request_event)

Check the satisfaction of the dependencies and emit the dependencies requests

exit_status = None
fire(event)

Check if an action is fireable and fire the action or emit the dependencies requests

fireable(event)

Return True if the action is fireable

job = None
jobs_engine = None
name = None
not_concurrent = None
priority = None
start_on_startup = False
use_notifications = None
class Function(job, name, module=None, class_name=None, options=None, jobs_engine=None)

Bases: object

Functions are job method executed synchronously.

call(options={})

Execute the function an return the result

help()

Return an help message for this function

job = None
jobs_engine = None
name = None
class InternalAction(name, jobs_engine, temp=False)

Bases: object

fire(event)
jobs_engine = None
name = None
not_concurrent = False
priority = 1
class ErrorsHandlerAction(jobs_engine)

Bases: endian.job.engine_actions.InternalAction

This action is executed when a job failed (more precisely when ‘change_status(sub:err)’ is received)

fire(event)

Remove all the temporary handlers (installed for the dependencies) of the job This action is invoked when is received the following event:

e.g: change_status(status:, sub:err, name:)
Exampe:

job_a – depends on –> job_b job_b – depends on –> job_c

the job_c failed -> emit ‘change_status(status:start,sub:err,name:job_c)’ this action is executed:

we have to change status of job_b to ‘err’, and emit and event ‘change_status(status:start,sub:err,name: job_b)’

(and then it is executed again)

class ReloadJobsHandlerAction(jobs_engine)

Bases: endian.job.engine_actions.InternalAction

This action reload all the jobs and the loaded modules It is executed when the event “event(msg:reload_jobs)” is received

fire(event)
class RunTriggersHandlerAction(jobs_engine)

Bases: endian.job.engine_actions.InternalAction

This handler run the triggers in the /etc/rc.d/event_<event name>.d directory (e.g ‘/etc/rc.d/event_startup.d’) when the event <evnet name> is received

fire(event)

endian.job.engine_control module

main()
send_cmd_to_engine(cmd, params=None, options=None, prefix=None)

Send a cmd to the engine and wait for the response

send_query_to_engine(query, single_result=False)

Send a query to the engine and wait for the response. If ‘single_result’ is True, return a single Fact instance (if any) or None else return a FactBase instance.

1
2
3
4
5
6
7
facts = send_query_to_engine("action")
for fact in facts:
  job = fact.get_value('job')
  name = fact.get_value('name')
  print job, name
  for key, option in fact['options'].iteritems():
      print "   %s type: %s description: %s " % (key, option.get_value('type'), option.get_value('description'))

endian.job.engine_jobs module

class JobsEngine(packages_names=['endian.job.test'], debug=True, console=None, timing=False, timing_filename=None, interactive=False, startup=False, restore_status=False, multithreads=False, force_modules_import=False, requests_filename=None, first_startup=False)

Bases: object

add_job(job)

Add a new engine job

Parameters:job (EngineJob) – a job
autoreload(autoreload)

Start/stop jobs autoreloading

call_function(function, params, sync=None)
creation_time = None
critical(msg, *args, **kw)
debug(msg, *args, **kw)
decrement_fork_counter()
emit(event, first=True, delay=0)

Add a new event to the event queue

error(msg, *args, **kw)
fact_base = None
get_events()

Return the events queue as a list of strings.

get_job_by_name(name)

Return a job by its name

Parameters:name (string) – job’s name
Returns:returns a job or None
Return type:EngineJob
get_job_by_pid(pid)

Return a job by its pid

Parameters:pid (int) – job’s pid
Returns:returns a job or None
Return type:EngineJob
getkey(prompt=None)
handlers = None
increment_fork_counter()
info(msg, *args, **kw)
interactive = False
list_jobs(name=None, status=None, sub=None, schedule=None, filter=None, exact_match=True, is_group=False)

Return the jobs list as a list of string

Parameters:
  • filter (string) – display only the jobs with names matching with ‘filter’
  • exact_match (boolean) – if True display the jobs with names equal to filter, if False display the jobs with names starting with ‘filter’
load_jobs_from_packages(force_modules_import=False)

Load (or reload) engine jobs from the packages in self._packages_names

Parameters:force_modules_import (boolean) – if true, don’t use the the modules cache
log_message(msg)

Store a message in the jobsengine’s message list

Parameters:msg (string) – a messagems formatted as: JOBNAME-SEVERITY-Message text e.g. HOSTNAME-I-Start
log_request(cmd)

Store a request (i.e. an emit/start/stop/restart command sent to the jobsengine) on the job requests log file

Parameters:cmd – a command, e.g. ‘restart ssh’
logger = None
messages = None
reload_modules()

Reload all the reloadable modules

restore_status(restore_status)

Load the fact base and, if restore_status is True, the jobs status from a file

save_status()

Save the jobs status in a file

shutdown()
start()
stop_request = False
warning(msg, *args, **kw)

endian.job.engine_remote_control module

class EngineRemoteControl(host, username='root', password=None)

Bases: object

Use this class to remotely control jobsengine

check_if_connected(pid)

Return ture if connected, false otherwise (check if the ssh process is alive)

send_cmd_to_remote_engine(cmd)

Exec a single jobsengine remote command on a target host and returns the command response.

send_query_to_remote_engine(query, single_result=False)

Send a query to the engine and wait for the response. If ‘single_result’ is True, return a single Fact instance (if any) or None else return a FactBase instance.

terminate()

Force the child process to terminate

exception EngineRemoteException

Bases: exceptions.Exception

send_cmd_to_remote_engine(cmd, host, username='root', password=None)

Exec a single jobsengine remote command on a target host. Dont’ raise exception but returns the errors a string, eg: 592 Error sending cmd to jobsengine: [Errno 5] Input/output error

send_query_to_remote_engine(query, host, username='root', password=None, single_result=False)

Send a query to the engine and wait for the response. If ‘single_result’ is True, return a single Fact instance (if any) or None else return a FactBase instance.

endian.job.engine_shell module

engine_shell(say_hello=False)
engine_shell_check_connection(channel)
engine_shell_read_response(channel_file)
engine_shell_send_cmd(cmd, channel)
main()

endian.job.fact_base module

class Fact(s, dont_tokenize=False, **kargs)

Bases: dict

Prolog-like fact, i.e. a predicate followed by an optional list of attributes Example:

a_fact
emit(signal:startup)
service(job:ssh, status:start)

Fact arguments can be underscores, which will match anything. Example:

service(job:ssh, status: _) matches service(job:ssh, status:start)
arg(n)
get_value(n, default=None, get_type=None, include_predicate=True)

Return the value of an attribute as a ‘basic’ value, e.g. a string, int, float or a boolean If the get_type parameter is presente and the values is not an instance of get_type, try typecast and if fail return the default value If the value is a fact with attributes, return it as a dictionary If include_predicate is True, include the predicate into the dictionary (if the result is dictionary)

get_values(*args)
has_attributes

True if the fact as attributes

keys()
match(query)

Return True if query match self. Fact arguments can be underscores, which will match anything.

predicate

Predicate converted to string

to_dict(include_predicate=True)

Convert the fact into a dictionary. If include_predicate is True, include the predicate into the dictionary

class FactBase(facts=None, *facts_args)

Bases: object

Prolog-like fact base, more prosaically a set of facts

assert_fact(fact, **kargs)

Assert a new fact

Parameters:fact (string or Fact) – a fact
facts
inverse_in(fact, **kargs)

Returns if a fact a fact matching the parameter is ‘inverse in’ the fact base

Parameters:fact (string or Fact:) – a fact
Returns:True if the fact is ‘inverse in’ the fact base
Return type:boolean
inverse_matches(fact, **kargs)

Returns a list of facts matching the parameter

Parameters:fact (string or Fact) – a fact
inverse_matches_one(fact, **kargs)

Returns one of facts matching the parameter or None

Parameters:fact (string or Fact) – a fact
matches(fact, **kargs)

Returns a list of facts matching the parameter

Parameters:fact (string or Fact) – a fact
matches_one(fact, **kargs)

Returns one of facts matching the parameter or None

param fact:a fact
type fact:string or Fact
remove(fact, **kargs)

Retract (remove) a single fact that matches the argument, raise a KeyError if the fact is not present in the fact base

Parameters:fact (string or Fact) – a fact
retract(fact, **kargs)

Retract (remove) a single fact that matches the argument

Parameters:fact (string or Fact) – a fact
retractall(fact, **kargs)

Retract (remove) all facts that match the argument

Parameters:fact (string or Fact) – a fact
tables = None
class FactTable(facts=None, *facts_args)

Bases: object

A set of facts with the same predicate (i.e. of the same type)

assert_fact(fact, **kargs)

Assert a new fact

Parameters:fact (string or Fact) – a fact
facts = None
inverse_in(fact, **kargs)

Returns if the fact is ‘inverse in’ the table

Parameters:fact (string or Fact:) – a fact
Returns:True if the fact is ‘inverse in’ the table
Return type:boolean
inverse_matches(fact, **kargs)

Returns a list of facts matching the parameter

Parameters:fact (string or Fact) – a fact
inverse_matches_one(fact, **kargs)

Returns one of facts matching the parameter or None

Parameters:fact (string or Fact) – a fact
matches(fact, **kargs)

Returns a list of facts matching the parameter

Parameters:fact (string or Fact) – a fact
matches_one(fact, **kargs)

Returns one of facts matching the parameter or None

Parameters:fact (string or Fact) – a fact
remove(fact, **kargs)

Retract (remove) a single fact that matches the argument, raise a KeyError if the fact is not present in the fact base

Parameters:fact (string or Fact) – a fact
retract(fact, **kargs)

Retract (remove) a single fact that matches the argument

Parameters:fact (string or Fact) – a fact
retractall(fact, **kargs)

Retract (remove) all facts that match the argument

Parameters:fact (string or Fact) – a fact

endian.job.fake_logger module

class FakeLogger(name='', level=4)

Bases: object

CRITICAL = 1
DEBUG = 5
ERROR = 2
INFO = 4
WARNING = 3
critical(msg, *args, **kw)

Log ‘msg % args’ with severity ‘CRITICAL’.

debug(msg, *args, **kw)

Log ‘msg % args’ with severity ‘DEBUG’.

enable_notifications()

Send log messages to the jobsengine

end_notifications()

Disable sending log messages to the jobsengine

error(msg, *args, **kw)

Log ‘msg % args’ with severity ‘ERROR’.

info(msg, *args, **kw)

Log ‘msg % args’ with severity ‘INFO’.

is_notification_enabled

Return true if the notification is enabled

log_traceback(level, tb, log_name=None)
set_level(level)

Set the logging level.

warn(msg, *args, **kw)

Log ‘msg % args’ with severity ‘WARNING’.

warning(msg, *args, **kw)

Log ‘msg % args’ with severity ‘WARNING’.

log_to_syslog()

Redirect log to syslog

log_to_console()

Write log to console

set_process_name(new_process_name, set_only_thread_name=False, logger=None)

Change the process name

set_thread_name(new_thread_name, logger=None)

Change the current thread name

get_monotonic_time(second_only=False, logger=None)

Return the system monotonic time Monotonic clock cannot be set and represents monotonic time since some unspecified starting point.

get_thread_id()

Get the current thread ID

endian.job.handlers module

class Handlers(handelers=None, **handlers_args)

Bases: object

add_handler(event, **kargs)

Assert a new handler

handlers = None
has_match(event, **kargs)
Parameters:event (string or Fact) – a fact
has_temp_action_handlers(action)

Return true if there is a temp action handler for a given action

matches(event, remove_temp_handlers=True)

Returns a list of handler matching the event sorted by the handler priority or the action priority

Parameters:event (string or Fact) – a fact
remove_action_handlers(action)

Remove all the handlers for an action

remove_handler(event, **kargs)

Remove an handler

remove_temp_event_handlers(event)
remove_temp_job_handlers(job)

Remove all the temporary handlers for a job

endian.job.http_get module

http_get(host, port, path, timeout=None)

endian.job.job module

class Job

Bases: object

critical(msg, *args, **kw)
debug(msg, *args, **kw)
enable_debug()
error(msg, *args, **kw)
info(msg, *args, **kw)
job_groups = None
load_config()
logger = None
name = None
restart(options)
save_config()
start(options)
status = None
stop(options)
sub = None
warning(msg, *args, **kw)
use_notifications(service_name=None)

>>>>> OBSOLETE - use action annotation with argument ‘use_notifications’ ! <<<<

Use notification annotation

Enable the notification before the method execution and disable it after the execution. If the notification was enabled, does nothing. Use this annotation only on Job subclasses methods.

Usage example:

@use_notifications("serviceName")
def start(self):
    self.info("bla, bla, bla")
action(name, status_after_action=None, on=None, depends=None, exit_status=None, priority=50, options=None, not_concurrent=False, use_notifications=False)

Action’s annotation

Parameters:
  • name – action name
  • status_after_action – status of the job after the execution of the action
  • on – list of events that fire the action
  • depends – list of dependencies
  • exit_status – new status after the execution of the action (e.g. the exit status can be ‘start’ for both the actions ‘start’ and ‘restart’)
  • priority – action priority 1-100 (1 higher priority, 100 lowest priority)
  • options – action options as a dictionary mapping maps to type and (optional) description: (e.g. {‘debug’: (bool,”enable debug”), ‘force’: (bool, “force restart”)} )
  • not_concurrent – if true, no other action is executed in concurrence with this action. before starting the action, the job engine waits until all the other action are finished and anfter the execution of this action waits until the action terminated
  • use_notifications – Enable the notification before the method execution and disable it after the execution. If the notification is enable, the info and error log messages are sent to the jobsengine. If the notification was enabled, does nothing.

Usage example:

@action(name="start", on="startup", depends=["network"], options={'debug': (bool,"enable debug"), 'force': (bool, "force restart")})
def start(self, options):
    self.info("bla, bla, bla")

endian.job.local_execution module

exec_job_locally(cmd, use_syslog=False, debug=False, test=False)

Exec a job out of the jobsengine context.

Parameters:cmd (string) – A cmd to be executed, eg. ‘ssh.start’

endian.job.modules module

get_modules(package_name, logger=None, force_modules_import=False)

Return the list of modules in a given package. For each module, an Fact instance is returned.

Parameters:
  • logger (loger) – logger
  • force_modules_import (boolean) – if true, don’t use the the modules cache
Returns:

the list of modules

Return type:

list of Fact

get_compilation_date()

Get the JobsEngine compilation date

endian.job.socket_server module

class CommandsInterpreter(jobs_engine, logger)

Bases: object

action_actions(session, args)

actions Print the actions list

action_autoreload(session, args)

autoreload [on|off] Enable of disable jobs autoreload

action_call(session, args)

call [sync|async] job.func [args] Exec a function

action_config(session, args)

config var [value] Set (or display) the value of a config var

action_debug(session, args)

debug job [on|off] Enable of disable debug on a job

action_dump(session, args)

dump Dump the fact base

action_emit(session, args)

emit [delayed [sec]] event Emit an event

action_events(session, args)

events Print the events queue

action_functions(session, args)

functions Print the functions list

action_handlers(session, args)

handlers Print the list of the events handlers

action_hello(session, args)
action_help(session, args, errro_msg=None, code=214)
action_info(session, args)

info Print jobs engine info

action_list(session, args)

list [ok|err|start|stop|exe|group] Print jobs short status information

action_log(session, args)

log msg Log a message (JOBNAME-SEVERITY-Message text)

action_messages(session, args)

messages [raw] Print the last messages

action_query(session, args)

query fact... Query the fact base

action_quit(session, args)
action_request(session, args)

request [sync] [delayed [sec]] job.ac [args] Request job action execution

action_restart(session, args)

restart [sync] [delayed [sec]] job [args] Restart a job

action_start(session, args)

start [sync] [delayed [sec]] job [args] Start a job

action_status(session, args)

status job Print short status information about a job

action_stop(session, args)

stop [sync] [delayed [sec]] job [args] Stop a job

action_summary(session, args)

summary Print short status information for each job

action_threads(session, args)

threads Print jobsengine threads list

emit_and_lock(event, service, delay=0)
exec_command(session, cmd)

Exec a command and return the commands result as a string

is_demo_mode_enabled(session)

Return true if demo mode is enabled

class ManagerServerThread(jobs_engine)

Bases: threading.Thread

commands_interpreter = None
jobs_engine = None
logger = None
run()
server = None
shutdown()
stop_request = False
class ServerThread(channel, details, commands_interpreter)

Bases: threading.Thread

run()
send(args)
split_head_tail(l)