.. Licensed to the Apache Software Foundation (ASF) under one
   or more contributor license agreements.  See the NOTICE file
   distributed with this work for additional information
   regarding copyright ownership.  The ASF licenses this file
   to you under the Apache License, Version 2.0 (the
   "License"); you may not use this file except in compliance
   with the License.  You may obtain a copy of the License at

   http://www.apache.org/licenses/LICENSE-2.0

   Unless required by applicable law or agreed to in writing,
   software distributed under the License is distributed on an
   "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
   KIND, either express or implied.  See the License for the
   specific language governing permissions and limitations
   under the License.

.. include:: ../common.defs

.. default-domain:: cpp

.. highlight:: cpp

.. _threads-and-events:

Threads and Event Processing
****************************

Internally |TS| is a cooperative multi-threaded environment. There are a fixed number of threads for core operations, determined at process start time. All core operations take place on one of these existing threads. Plugins may spawn additional threads but these are outside the scope of this document.

Threads
=======

|TS| has a taxonomy of thread types which are layered to create the threading infrastructure. At the
most basic are threads as the operating system provides. Classes provide additional data and
operations on these threads to make them operate properly for |TS|.

Thread
------

The abstract :class:`Thread` is the base class for thread operations. It contains a mutex and a
thread identifier. The logic for starting the thread at the system level is embedded in this class.
All threads started by |TS| have an instance of this class (or subclass). Plugins can directly start
their own threads via system calls and those are not tracked. :class:`Thread` sets up thread local
storage via :code:`pthread_setspecific`. Threads can be started via an explicit function provided to
:func:`Thread::start` or by subclassing :class:`Thread` and overriding :func:`Thread::execute`.

:class:`Thread` also performs the basic time keeping for |TS|. The class contains a global static value which is treated as the current time for |TS|. Usually this class is accessed as a static but it can also be accessed in a way to update the current time. Because of the large number of threads the static use is generally sufficiently accurate because it contains the last time any thread updated.

EThread
-------

:class:`EThread` is a subclass of :class:`Thread` which provides support for |TS| core operations.
It is this class that provides support for using :class:`Continuation` instances. :class:`EThread`
overrides the :func:`Thread::execute` method to gain control after the underlying thread is started.
This method executes a single continuation at thread start. If the thread is :enumerator:

`ThreadType::DEDICATED` it returns after invoking the start continuation. No join is executed, the
presumption is the start continuation will run until process termination. This mechanism is used
because it is, from the |TS| point of view, the easiest to use because of the common support of
continuations.

A :enumerator:`ThreadType::REGULAR` thread will first execute its start continuation and then process its event queue until explicitly stopped after executing the start continuation.

Despite the name :class:`EventProcessor` is primarily a thread management class. It enables the
creation and management of thread groups which are then used by the |TS| core for different types of
computation. The set of groups is determined at run time via subsystems making calls to the
:func:`EventProcessor::register_event_type`. Threads managed by :class:`EventProcessor` have the
:class:`EThread` start continuation controlled by :class:`EventProcessor`. Each thread group (event
type) has a list of continuations to run when a thread of that type starts. Continuations are added
to the list with :func:`EventProcessor::schedule_spawn`. There are two variants of this method, one
for continuations and one for just a function. The latter creates a continuation to call the
function and then schedules that using the former. The :class:`EventProcessor` internal start
continuation for the :class:`EThread` executes the continuations on this list for the appropriate
thread group and then returns, after which :func:`EThread::execute` loops on processing its event
queue.

:class:`EventProcessor` is intended to be a singleton and the global instance is :var:`eventProcessor`.

In general if a subsystem in the |TS| core is setting up a thread group, it should use code of the
form

.. code-block:: cpp

   int ET_GROUP; // global variable, where "GROUP" is replaced by the actual group / type name.
   int n_group_threads = 3; // Want 3 of these threads by default, possibly changed by configuration options.
   constexpr size_t GROUP_STACK_SIZE = DEFAULT_STACK_SIZE; // stack size for each thread.
   void Group_Thread_Init(EThread*); // function to perform per thread local initialization.

   ET_GROUP = eventProcessor::registerEventType("Group");
   eventProcessor.schedule_spawn(&Group_Per_Thread_Init, ET_GROUP);
   eventProcessor.spawn_event_threads(ET_GROUP, n_group_threads, GROUP_STACK_SIZE);


The function :code:`Group_Thread_Init` can be replaced with a continuation if that's more
convenient. One advantage of a continuation is additional data (via :arg:`cookie`) can be provide
during thread initialization.

If there is no thread initialization needed, this can be compressed in to a single call

.. code-block:: cpp

   ET_GROUP = eventProcessor.spawn_event_threads("Group", n_group_threads, GROUP_STACK_SIZE);

This registers the group name and type, starts the threads, and returns the event type.

Types
=====

.. type:: EventType

   A thread classification value that represents the type of events the thread is expected to process.

.. var:: EventType ET_CALL

   A predefined :type:`EventType` which always exists. This is deprecated, use :var:`ET_NET` instead.

.. var:: EventType ET_NET

   A synonym for :var:`ET_CALL`.

.. var:: EventProcessor eventProcessor

   The global single instance of :class:`EventProcessor`.

.. type:: ThreadFunction

   The type of function invoked by :func:`Thread::start`. It is a function returning :code:`void*` and taking no arguments.

.. class:: Thread

   Wrapper for system level thread.

   .. function:: start(const char * name, void * stack, size_t stacksize, ThreadFunction const &f)

      Start the underlying thread. It is given the name :arg:`name`. If :arg:`stack` is
      :code:`nullptr` then a stack is allocated for it of size :arg:`stacksize`. Once the thread is
      started, :arg:`f` is invoked in the context of the thread if non :code:`nullptr`, otherwise
      the method :func:`Thread::execute` is called. The thread execution returns immediately after
      either of these, leaving a zombie thread. It is presumed both will execute until process
      termination.

   .. function:: void execute()

      A pure virtual method that must be overridden in a subclass.

.. class:: EThread

   Event processing thread.

   .. function:: EventType registerEventType(const char* name)

      Register an event type by name. This reserves an event type index which is returned as :type:`EventType`.

   .. function:: void execute()

      Call the start continuation, if any. If a regular (not dedicated) thread, continuously process the event queue.

.. enum:: ThreadType

   .. enumerator:: DEDICATED

      A thread which executes only the start continuation and then exits.

   .. enumerator:: REGULAR

      A thread which executes the start continuation and then processes its event queue.

.. class:: Continuation

   A future computation. A continuation has a handler which is a class method with a
   specific signature. A continuation is invoked by calling its handler. A future computation can be
   referenced by an :class:`Action` instance. This is used primarily to allow the future work to be
   canceled.

.. class:: Action

   Reference to a future computation for a :class:`Continuation`.

.. class:: Event : public Action

   Reference to code to dispatch. Note that an :class:`Event` is a type of :class:`Action`. This class combines the future computational reference of :class:`Action`

.. type:: ThreadSpawnFunction

   A function that takes a single argument of pointer to :class:`EThread` and returns :code:`void`. The argument will be the :class:`EThread` in which the function is executing.

.. class:: EventProcessor

   .. function:: EventType register_event_type(char const * name)

      Register an event type with the name :arg:`name`. The unique type index is returned.

   .. function:: Event * schedule_spawn(Continuation * c, EventType ev_type, int event = EVENT_IMMEDIATE, void * cookie = nullptr)

      When the :class:`EventProcessor` starts a thread of type :arg:`ev_type`, :arg:`c` will be
      called before any events are dispatched by the thread. The handler for :arg:`c` will be called
      with an event code of :arg:`event` and data pointer of :arg:`cookie`.

   .. function:: Event * schedule_spawn(void ( * f) (EThread * ), EventType ev_type)

      When the :class:`EventProcessor` starts a thread of type :arg:`ev_type` the function :arg:`f`
      will be called with a pointer to the :class:`EThread` instance which is starting.
