Configuration¶
The Dallinger configuration
module provides tools for reading and writing
configuration parameters that control the behavior of an experiment. To use the
configuration, first import the module and get the configuration object:
import dallinger
config = dallinger.config.get_config()
You can then get and set parameters:
config.get("duration")
config.set("duration", 0.50)
When retrieving a configuration parameter, Dallinger will look for the parameter
first among environment variables, then in a config.txt
in the experiment
directory, and then in the .dallingerconfig
file, using whichever value
is found first. If the parameter is not found, Dallinger will use the default.
If a value is extracted from the environment or a config file it will be converted
to the correct type. You can also specify a value of file:/path/to/file
to
use the contents of that file on your local computer.
Built-in configuration¶
Built-in configuration parameters, grouped into categories:
General¶
mode
unicodeRun the experiment in this mode. Options include
debug
(local testing),sandbox
(MTurk sandbox), andlive
(MTurk).logfile
unicodeWhere to write logs.
loglevel
unicodeA number between 0 and 4 that controls the verbosity of logs, from
debug
tocritical
. Note thatdallinger debug
ignores this setting and always runs at 0 (debug
).whimsical
booleanWhat’s life without whimsy? Controls whether email notifications sent regarding various experiment errors are whimsical in tone, or more matter-of-fact.
dashboard_password
unicodeAn optional password for accessing the Dallinger Dashboard interface. If not specified, a random password will be generated.
dashboard_user
unicodeAn optional login name for accessing the Dallinger Dashboard interface. If not specified
admin
will be used.enable_global_experiment_registry
booleanEnable a global experiment id registration. When enabled, the
collect
API check this registry to see if an experiment has already been run and reject re-running an experiment if it has been.
Recruitment (General)¶
auto_recruit
booleanA boolean on whether recruitment should be automatic.
browser_exclude_rule
unicode - comma separatedA set of rules you can apply to prevent participants with unsupported web browsers from participating in your experiment.
recruiter
unicodeThe recruiter class to use during the experiment run. While this can be a full class name, it is more common to use the class’s
nickname
property for this value; for examplemturk
,cli
,bots
, ormulti
. NOTE: when running in debug mode, the HotAir (hotair
) recruiter will always be used. The exception is if the--bots
option is passed todallinger debug
, in which case the BotRecruiter will be used instead.recruiters
unicode - custom formatWhen using multiple recruiters in a single experiment run via the
multi
setting for therecruiter
config key,recruiters
allows you to specify which recruiters you’d like to use, and how many participants to recruit from each. The special syntax for this value is:recruiters = [nickname 1]: [recruits], [nickname 2]: [recruits], etc.
For example, to recruit 5 human participants via MTurk, and 5 bot participants, the configuration would be:
recruiters = mturk: 5, bots: 5
Amazon Mechanical Turk Recruitment¶
aws_access_key_id
unicodeAWS access key ID.
aws_secret_access_key
unicodeAWS access key secret.
aws_region
unicodeAWS region to use. Defaults to
us-east-1
.ad_group
unicodeObsolete. See
group_name
.assign_qualifications
booleanA boolean which controls whether an experiment-specific qualification (based on the experiment ID), and a group qualification (based on the value of
group_name
) will be assigned to participants by the recruiter. This feature assumes a recruiter which supports qualifications, like theMTurkRecruiter
.group_name
unicodeAssign a named qualification to workers who complete a HIT.
mturk_qualification_blocklist
unicode - comma seperatedComma-separated list of qualification names. Workers with qualifications in this list will be prevented from viewing and accepting the HIT.
mturk_qualification_requirements
unicode - JSON formattedA JSON list of qualification documents to pass to Amazon Mechanical Turk.
title
unicodeThe title of the HIT on Amazon Mechanical Turk.
description
unicodeThe description of the HIT on Amazon Mechanical Turk.
keywords
unicodeA comma-separated list of keywords to use on Amazon Mechanical Turk.
lifetime
integerHow long in hours that your HIT remains visible to workers.
duration
floatHow long in hours participants have until the HIT will time out.
us_only
booleanControls whether this HIT is available only to MTurk workers in the U.S.
base_payment
floatBase payment in U.S. dollars. All workers who accept the HIT are guaranteed this much compensation.
approve_requirement
integerThe percentage of past MTurk HITs that must have been approved for a worker to qualify to participate in your experiment. 1-100.
organization_name
unicodeObsolete.
Preventing Repeat Participants¶
If you set a group_name
and assign_qualifications
is also set to
true
, workers who complete your HIT will be given an MTurk qualification for
your group_name
. In the future, you can prevent these workers from
participating in a HIT with the same group_name
by including that name in
the qualification_blacklist
configuration. These four configuration keys
work together to create a system to prevent recuiting workers who have already
completed a prior run of the same experiment.
Email Notifications¶
See Email Notification Setup for a much more detailed explanation of these values and their use.
contact_email_on_error
unicodeThe email address used as the recipient for error report emails, and the email displayed to workers when there is an error.
dallinger_email_address
unicodeAn email address for use by Dallinger to send status emails.
smtp_host
unicodeHostname and port of a mail server for outgoing mail. Defaults to
smtp.gmail.com:587
smtp_username
unicodeUsername for outgoing mail host.
smtp_password
unicodePassword for the outgoing mail host.
Deployment Configuration¶
database_url
unicodeURI of the Postgres database.
database_size
unicodeSize of the database on Heroku. See Heroku Postgres plans.
dyno_type
unicodeHeroku dyno type to use. See Heroku dynos types.
redis_size
unicodeSize of the redis server on Heroku. See Heroku Redis.
num_dynos_web
integerNumber of Heroku dynos to use for processing incoming HTTP requests. It is recommended that you use at least two.
num_dynos_worker
integerNumber of Heroku dynos to use for performing other computations.
host
unicodeIP address of the host.
port
unicodePort of the host.
clock_on
booleanIf the clock process is on, it will perform a series of checks that ensure the integrity of the database.
heroku_python_version
unicodeThe python version to be used on Heroku deployments. The version specification will be deployed to Heroku in a runtime.txt file in accordance with Heroku’s deployment API. Note that only the version number should be provided (eg: “2.7.14”) and not the “python-” prefix included in the final runtime.txt format. See Dallinger’s global_config_defaults.txt for the current default version. See Heroku supported runtimes.
heroku_team
unicodeThe name of the Heroku team to which all applications will be assigned. This is useful for centralized billing. Note, however, that it will prevent you from using free-tier dynos.
worker_multiplier
floatMultiplier used to determine the number of gunicorn web worker processes started per Heroku CPU count. Reduce this if you see Heroku warnings about memory limits for your experiment. Default is 1.5
Choosing configuration values¶
When running real experiments it is important to pick configuration variables that result in a deployment that performs appropriately.
The number of Heroku dynos that are required and their specifications can make a very large difference to how the application behaves.
num_dynos_web
This configuration variable determines how many dynos are run to deal with web traffic. They will be transparently load-balanced, so the more web dynos are started the more simultaneous HTTP requests the stack can handle. If an experiment defines the
channel
variable to subscribe to websocket events then all of these callbacks happen on the dyno that handles the initial/launch
POST, so experiments that use this functionality heavily receive significantly less benefit from increasingnum_dynos_web
. The optimum value differs between experiments, but a good rule of thumb is 1 web dyno for every 10-20 simultaneous human users.num_dynos_worker
Workers are dynos that pull tasks from a queue and execute them in the background. They are optimized for many short tasks, but they are also used to run bots which are very long-lived. Each worker can run up to 20 concurrent tasks, however they are co-operatively multitasked so a poorly behaving task can cause all others sharing its host to block. When running with bots, you should always pick a value of
num_dynos_worker` that is at least ``0.05*number_of_bots
, otherwise it is guaranteed to fail. In practice, there may well be experiment-specific tasks that also need to execute, and bots are more performant on underloaded dynos, so a better heuristic is0.25*number_of_bots
.dyno_type
This determines how powerful the heroku dynos started by Dallinger are. It is applied as the default for both web and worker dyno types. The minimum recommended is
standard-1x
, which should be sufficient for experiments that do not rely on real-time coordination, such as Bartlett (1932), stories. Experiments that require significant power to process websocket events should consider the higher levels,standard-2x
,performance-m
andperformance-l
. In all but the most intensive experiments, eitherdyno_type
ornum_dynos_web
should be increased, not both. Seedyno_type_web
anddyno_type_worker
below for information about more specific settings.dyno_type_web
This determines how powerful the heroku web dynos are. It applies only to web dynos and will override the default set in
dyno_type
. Seedyno_type
above for details on specific values.dyno_type_worker
This determines how powerful the heroku worker dynos are. It applies only to worker dynos and will override the default set in
dyno_type
.. Seedyno_type
above for details on specific values.redis_size
A larger value for this increases the number of connections available on the redis dyno. This should be increased for experiments that make substantial use of websockets. Values are
premium-0
topremium-14
. It is very unlikely that values higher thanpremium-5
are useful.duration
The duration parameter determines the number of hours that an MTurk worker has to complete the experiment. Choosing numbers that are too short can cause people to refuse to work on a HIT. A deadline that is too long may give people pause for thought as it may make the task seem underpaid. Set this to be significantly above the total time from start to finish that you’d expect a user to take in the worst case.
base_payment
The amount of US dollars to pay for completion of the experiment. The higher this is, the easier it will be to attract workers.