Packages

c

bigglue.platforms.instances

MapperPlatform

abstract class MapperPlatform[Input <: Identifiable[Input], Output <: Identifiable[Output]] extends UnaryPlatform[Input, Output]

Created by edmundlam on 8/8/17.

Linear Supertypes
UnaryPlatform[Input, Output], UnaryChecker[Input, Output], OutputStoreChecker[Output], InputStoreChecker[Input], ConfigChecker, Platform, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. MapperPlatform
  2. UnaryPlatform
  3. UnaryChecker
  4. OutputStoreChecker
  5. InputStoreChecker
  6. ConfigChecker
  7. Platform
  8. AnyRef
  9. Any
Implicitly
  1. by any2stringadd
  2. by StringFormat
  3. by Ensuring
  4. by ArrowAssoc
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new MapperPlatform()

Abstract Value Members

  1. abstract def wake(): Unit

    This function is called when the upstream connector lets the platform know that there's data to be computed.

    This function is called when the upstream connector lets the platform know that there's data to be computed.

    Definition Classes
    Platform

Concrete Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. def +(other: String): String
    Implicit
    This member is added by an implicit conversion from MapperPlatform[Input, Output] to any2stringadd[MapperPlatform[Input, Output]] performed by method any2stringadd in scala.Predef.
    Definition Classes
    any2stringadd
  4. def ->[B](y: B): (MapperPlatform[Input, Output], B)
    Implicit
    This member is added by an implicit conversion from MapperPlatform[Input, Output] to ArrowAssoc[MapperPlatform[Input, Output]] performed by method ArrowAssoc in scala.Predef.
    Definition Classes
    ArrowAssoc
    Annotations
    @inline()
  5. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  6. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  7. def check(conf: PipeConfig, inputMap: DataStore[Input], outputMap: DataStore[Output]): Unit

    Check routine for unary data processors

    Check routine for unary data processors

    conf

    the config file to check

    inputMap

    the input store to check

    outputMap

    the output store to check

    Definition Classes
    UnaryChecker
  8. def checkConfig(conf: PipeConfig): Unit

    Check that paths in configuration file relevant to host class are well-formed.

    Check that paths in configuration file relevant to host class are well-formed. Default does nothing.

    conf

    the config file to check

    Definition Classes
    ConfigChecker
  9. def checkInput(inputMap: DataStore[Input]): Unit

    Check that input store is compatible with the host class.

    Check that input store is compatible with the host class. Default does nothing.

    inputMap

    the input store to check

    Definition Classes
    InputStoreChecker
  10. def checkOutput(outputMap: DataStore[Output]): Unit

    Check that output store is compatible with the host class.

    Check that output store is compatible with the host class. Default does nothing.

    outputMap

    the output store to check

    Definition Classes
    OutputStoreChecker
  11. def clone(): AnyRef
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  12. var computationOpt: Option[Computation]

    What computation this platform ends up running.

    What computation this platform ends up running. (Mapper, Reducer, PairwiseComposer) This ends up being set in init() by both UnaryPlatform and BinaryPlatform.

    Definition Classes
    Platform
  13. def ensuring(cond: (MapperPlatform[Input, Output]) ⇒ Boolean, msg: ⇒ Any): MapperPlatform[Input, Output]
    Implicit
    This member is added by an implicit conversion from MapperPlatform[Input, Output] to Ensuring[MapperPlatform[Input, Output]] performed by method Ensuring in scala.Predef.
    Definition Classes
    Ensuring
  14. def ensuring(cond: (MapperPlatform[Input, Output]) ⇒ Boolean): MapperPlatform[Input, Output]
    Implicit
    This member is added by an implicit conversion from MapperPlatform[Input, Output] to Ensuring[MapperPlatform[Input, Output]] performed by method Ensuring in scala.Predef.
    Definition Classes
    Ensuring
  15. def ensuring(cond: Boolean, msg: ⇒ Any): MapperPlatform[Input, Output]
    Implicit
    This member is added by an implicit conversion from MapperPlatform[Input, Output] to Ensuring[MapperPlatform[Input, Output]] performed by method Ensuring in scala.Predef.
    Definition Classes
    Ensuring
  16. def ensuring(cond: Boolean): MapperPlatform[Input, Output]
    Implicit
    This member is added by an implicit conversion from MapperPlatform[Input, Output] to Ensuring[MapperPlatform[Input, Output]] performed by method Ensuring in scala.Predef.
    Definition Classes
    Ensuring
  17. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  18. def equals(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  19. var errorCuratorOpt: Option[ErrorCurator[Input]]
    Definition Classes
    UnaryPlatform
  20. def finalize(): Unit
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  21. def formatted(fmtstr: String): String
    Implicit
    This member is added by an implicit conversion from MapperPlatform[Input, Output] to StringFormat[MapperPlatform[Input, Output]] performed by method StringFormat in scala.Predef.
    Definition Classes
    StringFormat
    Annotations
    @inline()
  22. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
  23. def getErrorCurator(): ErrorCurator[Input]
    Definition Classes
    UnaryPlatform
  24. def getInputMap(): DataStore[Input]
    Definition Classes
    UnaryPlatform
  25. def getInputs(): Seq[Input]
    Definition Classes
    UnaryPlatform
  26. def getOutputMap(): DataStore[Output]
    Definition Classes
    UnaryPlatform
  27. def getProvenanceCurator(): ProvenanceCurator[Input, Output]
    Definition Classes
    UnaryPlatform
  28. def getUpstreamConnector(): Connector[Input]
    Definition Classes
    UnaryPlatform
  29. def getVersionCurator(): VersionCurator[Output]
    Definition Classes
    UnaryPlatform
  30. def hashCode(): Int
    Definition Classes
    AnyRef → Any
  31. def init(conf: PipeConfig, inputMap: DataStore[Input], outputMap: DataStore[Output], builder: PlatformBuilder, mapper: Mapper[Input, Output]): Unit
  32. def init(conf: PipeConfig, inputMap: DataStore[Input], outputMap: DataStore[Output], builder: PlatformBuilder): Unit

    This sets up the platform by connecting the Input Map and Output Map to the platform, as well as any other initialization that needs to be done.

    This sets up the platform by connecting the Input Map and Output Map to the platform, as well as any other initialization that needs to be done. In the case of the example, we use bigglue.platforms.instances.bigactors.BigActorUnaryPlatform.init This calls initConnector as well.

    conf

    The configuration file needed to initialize.

    inputMap

    The Map that data gets sent in from.

    outputMap

    The Map that data gets sent out to.

    builder

    The builder that created the platform. This was called with bigglue.computations.Mapper.init or bigglue.computations.Reducer.init

    Definition Classes
    UnaryPlatform
  33. def initConnector(conf: PipeConfig, builder: PlatformBuilder): Unit

    This sets up the connector that sends data down the pipeline.

    This sets up the connector that sends data down the pipeline. This initializes the connector with Connector.init, and then adds the platform to the connector with Connector.registerPlatform. As a default, we use bigglue.connectors.instances.IncrTrackerJobQueue.

    conf

    The configuration file needed to initialize.

    builder

    The builder that created the platform. This was called with bigglue.computations.Mapper.init or bigglue.computations.Reducer.init

    Definition Classes
    UnaryPlatform
  34. var inputMapOpt: Option[DataStore[Input]]
    Definition Classes
    UnaryPlatform
  35. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  36. var mapperOpt: Option[Mapper[Input, Output]]
  37. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  38. final def notify(): Unit
    Definition Classes
    AnyRef
  39. final def notifyAll(): Unit
    Definition Classes
    AnyRef
  40. var outputMapOpt: Option[DataStore[Output]]
    Definition Classes
    UnaryPlatform
  41. def persist(): Unit

    This is called by the computation's persist step.

    This is called by the computation's persist step. In particular, this tends to see how much of the data set actually needs to be re-sent. As of right now, it sends the entirety of the input map to the upstream connector, which handles which things to send/re-send down the pipeline, unless there is a reducer with nothing in the input file, or if there is nothing in the output file. If there is nothing in the output file, the program assumes that a new data store has been created and sends down all data.

    Definition Classes
    UnaryPlatform → Platform
  42. var provenanceCuratorOpt: Option[ProvenanceCurator[Input, Output]]
    Definition Classes
    UnaryPlatform
  43. def run(): Unit
    Definition Classes
    MapperPlatform → Platform
  44. def setComputation(computation: Computation): Platform

    This ends up being called in the init step of the Mapper, Reducer, or PairwiseComposer This links the actual computation and platform together.

    This ends up being called in the init step of the Mapper, Reducer, or PairwiseComposer This links the actual computation and platform together.

    computation

    The computation that is linked with the platform. (This is usually called through platform.setComputation(this))

    Definition Classes
    Platform
  45. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  46. def terminate(): Unit
    Definition Classes
    MapperPlatform → Platform
  47. def toString(): String
    Definition Classes
    AnyRef → Any
  48. def tryComputeThenStore(mapper: Mapper[Input, Output], input: Input, outputMap: DataStore[Output]): List[Output]
  49. def updatedVersion(ver: Option[String]): Boolean
    Definition Classes
    UnaryPlatform
  50. var upstreamConnectorOpt: Option[Connector[Input]]
    Definition Classes
    UnaryPlatform
  51. var versionCuratorOpt: Option[VersionCurator[Output]]
    Definition Classes
    UnaryPlatform
  52. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  53. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  54. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  55. def →[B](y: B): (MapperPlatform[Input, Output], B)
    Implicit
    This member is added by an implicit conversion from MapperPlatform[Input, Output] to ArrowAssoc[MapperPlatform[Input, Output]] performed by method ArrowAssoc in scala.Predef.
    Definition Classes
    ArrowAssoc

Inherited from UnaryPlatform[Input, Output]

Inherited from UnaryChecker[Input, Output]

Inherited from OutputStoreChecker[Output]

Inherited from InputStoreChecker[Input]

Inherited from ConfigChecker

Inherited from Platform

Inherited from AnyRef

Inherited from Any

Inherited by implicit conversion any2stringadd from MapperPlatform[Input, Output] to any2stringadd[MapperPlatform[Input, Output]]

Inherited by implicit conversion StringFormat from MapperPlatform[Input, Output] to StringFormat[MapperPlatform[Input, Output]]

Inherited by implicit conversion Ensuring from MapperPlatform[Input, Output] to Ensuring[MapperPlatform[Input, Output]]

Inherited by implicit conversion ArrowAssoc from MapperPlatform[Input, Output] to ArrowAssoc[MapperPlatform[Input, Output]]

Ungrouped