c

bigglue.connectors.instances

IncrTrackerJobQueue

class IncrTrackerJobQueue[Data <: Identifiable[Data]] extends JobQueue[Data]

This is a Connector that has a status map attached to it. As of right now, the status map is InMemDataMultiMap, along with a TextFileDataMap as a backup when the pipeline closes out since it is guaranteed that the file system is there on every system. However, eventually I would like to have this map be of the same medium as the Data Store chosen just in case that isn't actually a possibility and/or to save computation power if the status map is heavily used.

Data

The type of data that's being sent down the pipeline from this connector. In the case of the example, it's bigglue.examples.GitID for gitID, and bigglue.examples.GitCommitInfo for commitInfoMap. Needs to be of type Identifiable

Linear Supertypes
JobQueue[Data], Connector[Data], AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. IncrTrackerJobQueue
  2. JobQueue
  3. Connector
  4. AnyRef
  5. Any
Implicitly
  1. by any2stringadd
  2. by StringFormat
  3. by Ensuring
  4. by ArrowAssoc
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new IncrTrackerJobQueue()

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. def +(other: String): String
    Implicit
    This member is added by an implicit conversion from IncrTrackerJobQueue[Data] to any2stringadd[IncrTrackerJobQueue[Data]] performed by method any2stringadd in scala.Predef.
    Definition Classes
    any2stringadd
  4. def +>(connector: Connector[Data]): Connector[Data]
    Definition Classes
    Connector
  5. def ->[B](y: B): (IncrTrackerJobQueue[Data], B)
    Implicit
    This member is added by an implicit conversion from IncrTrackerJobQueue[Data] to ArrowAssoc[IncrTrackerJobQueue[Data]] performed by method ArrowAssoc in scala.Predef.
    Definition Classes
    ArrowAssoc
    Annotations
    @inline()
  6. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  7. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  8. def clone(): AnyRef
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  9. var downstreamConnectorOpt: Option[Connector[Data]]
    Definition Classes
    Connector
  10. def ensuring(cond: (IncrTrackerJobQueue[Data]) ⇒ Boolean, msg: ⇒ Any): IncrTrackerJobQueue[Data]
    Implicit
    This member is added by an implicit conversion from IncrTrackerJobQueue[Data] to Ensuring[IncrTrackerJobQueue[Data]] performed by method Ensuring in scala.Predef.
    Definition Classes
    Ensuring
  11. def ensuring(cond: (IncrTrackerJobQueue[Data]) ⇒ Boolean): IncrTrackerJobQueue[Data]
    Implicit
    This member is added by an implicit conversion from IncrTrackerJobQueue[Data] to Ensuring[IncrTrackerJobQueue[Data]] performed by method Ensuring in scala.Predef.
    Definition Classes
    Ensuring
  12. def ensuring(cond: Boolean, msg: ⇒ Any): IncrTrackerJobQueue[Data]
    Implicit
    This member is added by an implicit conversion from IncrTrackerJobQueue[Data] to Ensuring[IncrTrackerJobQueue[Data]] performed by method Ensuring in scala.Predef.
    Definition Classes
    Ensuring
  13. def ensuring(cond: Boolean): IncrTrackerJobQueue[Data]
    Implicit
    This member is added by an implicit conversion from IncrTrackerJobQueue[Data] to Ensuring[IncrTrackerJobQueue[Data]] performed by method Ensuring in scala.Predef.
    Definition Classes
    Ensuring
  14. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  15. def equals(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  16. def finalize(): Unit
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  17. def formatted(fmtstr: String): String
    Implicit
    This member is added by an implicit conversion from IncrTrackerJobQueue[Data] to StringFormat[IncrTrackerJobQueue[Data]] performed by method StringFormat in scala.Predef.
    Definition Classes
    StringFormat
    Annotations
    @inline()
  18. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
  19. def getDownstream(): Connector[Data]
    Definition Classes
    Connector
  20. def getStatusMap(): DataMultiMap[Status, Data]
  21. def getUpstream(): Connector[Data]
    Definition Classes
    Connector
  22. def hashCode(): Int
    Definition Classes
    AnyRef → Any
  23. def iMap: DataStore[Data]
  24. var iMapOpt: Option[DataStore[Data]]
  25. def init(conf: PipeConfig): Unit
    Definition Classes
    JobQueue → Connector
  26. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  27. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  28. final def notify(): Unit
    Definition Classes
    AnyRef
  29. final def notifyAll(): Unit
    Definition Classes
    AnyRef
  30. def persist(dataStore: DataStore[Data]): Unit

    This is the section of the code that figures out what data from the Input Data Store to actually send down.

    This is the section of the code that figures out what data from the Input Data Store to actually send down. To do this, it first reconstructs the status map from the TextFileDataMap backup of the status map and the version of the platform taken from registerPlatform. After reconstructing the status map, it then takes the data from the data store that needs to be resent (due to a status of being Not Done or Modified or of Error, and resends that information.

    dataStore

    The input data store.

    Definition Classes
    IncrTrackerJobQueue → Connector
  31. var queue: DataQueue[Data]
    Definition Classes
    JobQueue
  32. def registerDownstreamConnector(connector: Connector[Data]): Unit
    Definition Classes
    Connector
  33. def registerPlatform(platform: Platform): Unit

    This registers that platform used the by connector, and sets the version of this connector to the version of the platform.

    This registers that platform used the by connector, and sets the version of this connector to the version of the platform.

    platform

    The platform to be registered by the connector.

    Definition Classes
    IncrTrackerJobQueue → Connector
  34. def registerUpstreamConnector(connector: Connector[Data]): Unit
    Definition Classes
    Connector
  35. def reportUp(status: Status, data: Seq[Data]): Unit
    Definition Classes
    IncrTrackerJobQueue → JobQueue → Connector
  36. def reportUp(status: Status, data: Data): Unit
    Definition Classes
    Connector
  37. def retrieveUp(): Seq[Data]
    Definition Classes
    IncrTrackerJobQueue → JobQueue → Connector
  38. def sendDown(data: Seq[Data]): Unit
    Definition Classes
    IncrTrackerJobQueue → JobQueue → Connector
  39. def sendDown(data: Data): Unit
    Definition Classes
    Connector
  40. def sendDownModified(data: Seq[Data]): Unit
    Definition Classes
    IncrTrackerJobQueue → JobQueue → Connector
  41. def sendDownModified(data: Data): Unit
    Definition Classes
    Connector
  42. def signalDown(): Unit
    Definition Classes
    JobQueue → Connector
  43. def size(): Int
    Definition Classes
    JobQueue → Connector
  44. val statusMap: DataMultiMap[Status, Data]
  45. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  46. def terminate(): Unit
    Definition Classes
    JobQueue → Connector
  47. def textMap: TextFileDataMap
  48. var textMapOpt: Option[TextFileDataMap]
  49. def toString(): String
    Definition Classes
    AnyRef → Any
  50. var upstreamConnectorOpt: Option[Connector[Data]]
    Definition Classes
    Connector
  51. var ver: Option[String]
  52. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  53. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  54. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  55. def →[B](y: B): (IncrTrackerJobQueue[Data], B)
    Implicit
    This member is added by an implicit conversion from IncrTrackerJobQueue[Data] to ArrowAssoc[IncrTrackerJobQueue[Data]] performed by method ArrowAssoc in scala.Predef.
    Definition Classes
    ArrowAssoc

Inherited from JobQueue[Data]

Inherited from Connector[Data]

Inherited from AnyRef

Inherited from Any

Inherited by implicit conversion any2stringadd from IncrTrackerJobQueue[Data] to any2stringadd[IncrTrackerJobQueue[Data]]

Inherited by implicit conversion StringFormat from IncrTrackerJobQueue[Data] to StringFormat[IncrTrackerJobQueue[Data]]

Inherited by implicit conversion Ensuring from IncrTrackerJobQueue[Data] to Ensuring[IncrTrackerJobQueue[Data]]

Inherited by implicit conversion ArrowAssoc from IncrTrackerJobQueue[Data] to ArrowAssoc[IncrTrackerJobQueue[Data]]

Ungrouped